Hole data recovery method and device of database and storage medium

Abstract

The application discloses a kind of hollow data recycling method, device and storage medium of database, relate to database operation and maintenance field, which comprises: the non-system table of change data existing in database is regarded as to be recycled table;According to the associated information of associated metadata set of to-be-recycled table;According to each original data file path aggregation, obtain the reference metadata set corresponding to each original data file respectively, and according to the reference metadata set, obtain the hollow rate of corresponding original data file;Each original data file is arranged in hollow rate descending order, and according to descending order result, the target number of original data files is regarded as to-be-recycled file, to recycle hollow data by rewriting to-be-recycled file.The technical scheme of the embodiment of the application greatly improves the hollow data recycling efficiency, avoids the long write lock time of current to-be-recycled table, ensures the read-write operation of other business to the table, and improves the hollow data recycling rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of database operation and maintenance, and in particular to a method and device for recovering hollow data of a database and a storage medium. BACKGROUND

[0002] In current databases, multi-version storage of data is a common way to improve database concurrency. After a piece of data is deleted, only a mark is made, and the data marked for deletion is also stored in metadata information, that is, it is recorded in the metadata file corresponding to the original data which row of data is deleted, and in subsequent business, this row of data can be skipped.

[0003] Since the original data under the above processing method has not been truly deleted, it will still occupy corresponding physical resources, causing data to expand indefinitely. Accordingly, a hollow data recovery strategy is needed to release the physical resources occupied by the deleted data. In the prior art, for the recovery of hollow data in a data table, a data rewriting method based on a file system is usually used, that is, the file content is copied to a new location in order to skip the hollow area. After the copying is completed, all original data in the original storage resource is deleted, and then the copied data is written to the original storage resource in sequence to complete the recovery of the hollow data.

[0004] However, the above recovery method not only needs a large amount of physical resources as an intermediate carrier for copying data, but also has low hollow data recovery efficiency, and at the same time, it needs to occupy the write lock of the data table for a long time, which greatly affects the use of the current data table by other businesses and reduces the read-write convenience of the data table in the database. SUMMARY

[0005] The present application provides a method and device for recovering hollow data of a database, an electronic device and a storage medium to solve the problem of low efficiency of hollow data recovery of a data table.

[0006] According to another aspect of the present application, a method for recovering hollow data of a database is provided, comprising:

[0007] In response to obtaining a hollow data recovery instruction, a non-system table in the database that has changed data is taken as a table to be recovered;

[0008] According to the associated metadata set of the table to be recovered, associated information is obtained, wherein the associated information includes an original data file path;

[0009] According to each original data file path, the reference metadata set corresponding to each original data file is aggregated and obtained, and the hollow rate of the corresponding original data file is obtained according to the reference metadata set;

[0010] The original data files are arranged in descending order of the hole rates, and a target number of the original data files are taken as the to-be-recovered files according to the descending arrangement result, so as to recover the hole data by rewriting the to-be-recovered files.

[0011] The target number of the original data files are taken as the to-be-recovered files according to the descending arrangement result, including: the hole rates of the original data files are sequentially accumulated according to the descending arrangement result, until the maximum accumulated result less than or equal to a preset recovery threshold is obtained, the target number is determined according to the maximum accumulated result, and the target number of the original data files are taken as the to-be-recovered files.

[0012] The hole data is recovered by rewriting the to-be-recovered files, including: a rewriting execution transaction is started, and the deleted rows in the to-be-recovered files are deleted through the rewriting execution transaction, so as to generate a new file not including the hole data.

[0013] After the new file not including the hole data is generated, further comprising: obtaining to-be-rewritten data rows and to-be-reorganized data rows according to the reference metadata set corresponding to the to-be-recovered files; rewriting each to-be-rewritten data row, and configuring at least one rewritten data row after rewriting as rewritten metadata; reorganizing each to-be-reorganized data row, so as to configure a plurality of to-be-reorganized data rows as reorganized metadata based on the data row threshold; associating the new file with the rewritten metadata and the reorganized metadata, and submitting the rewriting execution transaction.

[0014] After the to-be-rewritten data rows and the to-be-reorganized data rows are obtained, further comprising: rewriting each to-be-rewritten data row, and recombining each rewritten data row after rewriting and the to-be-reorganized data rows in the same reference metadata as new metadata; associating the new file with the new metadata, and submitting the rewriting execution transaction.

[0015] After the new file is associated with the rewritten metadata and the reorganized metadata, and the rewriting execution transaction is submitted, further comprising: continuing to save the to-be-recovered files and the reference metadata set; obtaining an active transaction identifier of a current database, and marking expired metadata in the reference metadata set through the active transaction identifier; traversing the original data files in the to-be-recovered files through the expired metadata, so as to obtain expired original data files; deleting the expired metadata and the expired original data files.

[0016] According to another aspect of the present application, a database hole data recovery device is provided, comprising:

[0017] The to-be-recovered table obtaining module is configured to, in response to obtaining a hole data recovery instruction, take a non-system table in which changed data exists in the database as a to-be-recovered table.

[0018] The association information acquisition module is configured to acquire association information according to the association metadata set of the table to be recycled; wherein the association information comprises an original data file path;

[0019] The hole rate acquisition module is configured to aggregate the reference metadata set corresponding to each original data file according to each original data file path, and acquire the hole rate of the corresponding original data file according to the reference metadata set;

[0020] The sorting execution module is configured to arrange the original data files in descending order of hole rate, and arrange the target number of original data files as the table to be recycled according to the descending arrangement result, so as to recycle the hole data by rewriting the table to be recycled.

[0021] According to another aspect of the present application, an electronic device is provided, comprising at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the database hole data recycling method according to any embodiment of the present application.

[0022] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the database hole data recycling method according to any embodiment of the present application.

[0023] According to another aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the database hole data recycling method according to any embodiment of the present application.

[0024] The technical solution of the embodiment of the present application responds to the acquisition of the hole data recycling instruction, takes the non-system table with changed data in the database as the table to be recycled, acquires the association information according to the association metadata set of the table to be recycled, aggregates the reference metadata set corresponding to each original data file according to each original data file path, and acquires the hole rate of the corresponding original data file according to the reference metadata set, arranges the original data files in descending order of hole rate, and arranges the target number of original data files as the table to be recycled according to the descending arrangement result, so as to recycle the hole data by rewriting the table to be recycled. Therefore, not only is it avoided to configure a large number of physical resources as the intermediate carrier of the rewriting process, but also the hole data recycling efficiency is greatly improved, the long write lock time of the current table to be recycled is avoided, the read and write operations of other businesses on the table are ensured, and the hole data recycling rate is improved.

[0025] It is to be understood that the description of the background of the application is not an acknowledgement or admission that any of the information provided in the description of the background of the application is prior art to the application. The information in the description of the background of the application may contain ideas, concepts and / or discoveries not yet known to be prior art to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0027] Figure 1 is a flow chart of a database hole data recycling method according to an embodiment of the present application;

[0028] Figure 2 is a flow chart of another database hole data recycling method according to an embodiment of the present application;

[0029] Figure 3 is a flow chart of yet another database hole data recycling method according to an embodiment of the present application;

[0030] Figure 4 is a structural schematic diagram of a database hole data recycling device according to an embodiment of the present application;

[0031] Figure 5 is a structural schematic diagram of an electronic device for implementing a database hole data recycling method according to an embodiment of the present application;

[0032] In the above-mentioned embodiments of the present application, Figure 5 In the above-mentioned embodiments of the present application, 10 is an electronic device, 11 is a processor, 12 is a read-only memory (ROM), 13 is a random access memory (RAM), 14 is a bus, 15 is an input / output (I / O) interface, 16 is an input unit, 17 is an output unit, 18 is a storage unit, and 19 is a communication unit. DETAILED DESCRIPTION

[0033] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described accompanying drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Embodiment one

[0036] Figure 1 is a flowchart of a database hole data recovery method provided by the first embodiment of the present application. The present embodiment can be applied to the case where the hole rate of each original data file of the table to be recovered is calculated by referring to the metadata set. The method can be executed by a database hole data recovery device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device (for example, a server). As shown in Figure 1 , the method comprises:

[0037] S101, in response to obtaining a hole data recovery instruction, taking a non-system table with changed data in the database as a table to be recovered.

[0038] The hole data recovery instruction can be triggered automatically in a periodic or non-periodic manner, or manually triggered by an operation and maintenance personnel. After the electronic device obtains the hole data recovery instruction, it traverses each data table in the database and determines the type of each data table. The data table type includes a system table and a non-system table (for example, a business table). The data table type can be distinguished by the name prefix of the data table, or by the creator information of the data table, for example, the system table is automatically created by the system or created by the administrator in the background, and the non-system table is created by a normal user. For the system table, no hole data recovery is needed, and only the non-system table needs to be recovered.

[0039] For a non-system table, first, a Write Lock permission of the non-system table is acquired; wherein, the Write Lock is a core lock mechanism in a database for ensuring data consistency, and its function is to force exclusive access, that is, only one thread or transaction is allowed to perform a modification operation on a data table at the same time, so as to avoid concurrent write conflicts with other write requests; if the Write Lock permission is not acquired, the current request will enter a lock waiting queue until the Write Lock permission of the current non-system table is acquired.

[0040] After the Write Lock permission is acquired, it is judged whether the non-system table has data change phenomenon according to a transaction execution condition of the current system table; for example, according to a transaction log of the system table, it can be queried whether there is active transaction writing after the previous hole data recovery instruction execution is completed; if there is active transaction writing, it indicates that there is changed data, and if there is no active transaction writing, it indicates that there is no changed data.

[0041] In addition, the current transaction number can also be compared with a transaction number (i.e. historical transaction number) when the previous hole data recovery operation is executed. If the current transaction number is the same as the historical transaction number, it indicates that there is no data writing in the interval recovery time (i.e. interval time of acquiring the hole data recovery instruction twice) of the non-system table, and obviously, the hole data recovery is not needed; if the current transaction number is different from the historical transaction number, it indicates that there is data writing in the interval recovery time of the non-system table, and the hole data recovery is needed, and accordingly, the non-system table is taken as a to-be-recovered table, that is, the non-system table with changed data is taken as a to-be-recovered table.

[0042] S102, acquiring association information according to the associated metadata set of the to-be-recovered table; wherein, the association information includes an original data file path.

[0043] Metadata is information used to describe data properties of a data table, and is stored in the form of key-value pair, i.e. "Key (key name)-Value (key value)". For a non-system table, there are multiple metadata corresponding thereto, and the metadata can include technical metadata, business metadata and operational metadata, etc. In the implementation of the present application, the specific number and type of the metadata associated with the non-system table are not limited.

[0044] The technical metadata describes the table structure and technical attributes, and can include table definitions (for example, table name, storage location, database type, and creation time), field attributes (for example, field name, data type, and length), and constraint rules, and the like key names; the business metadata is associated with business logic, and can include business terms, business rules, and data lineage, and the like key names; and the operation metadata records the data processing process, and can include update frequency, data quality (for example, null value rate and verification rules), and access permission, and the like key names.

[0045] All the metadata associated with the table to be recovered collectively form an associated metadata set. Through the “Key” value of each associated metadata, all detailed metadata information (that is, associated information) associated with the “Key” value can be obtained, and the metadata stored through encoding is converted into a structured data form through deserialization processing to facilitate reading. The associated information includes the number of data rows (that is, Data Chunk (DC) rows), the sequence number of each data row (indicating the position of the data row in the sequence), the data length and the data position (indicating the starting offset or position index of the data row in the original data file), and the original data file path from which each data row is respectively derived, which indicates the source of the data. In addition, a deletion mark for marking the data deletion state is also included. Since the granularity of the deletion operation is the data row, the deletion mark is used to indicate which data rows have been logically deleted.

[0046] S103, aggregating and obtaining the reference metadata set corresponding to each original data file according to the original data file path, and obtaining the hole rate of the corresponding original data file according to the reference metadata set.

[0047] Since the metadata and the original data file have a many-to-many reference relationship, that is, one metadata can refer to multiple original data files, that is, multiple data rows in one metadata can be distributed in different original data files; one original data file can also be referenced by multiple metadata, that is, the data rows in one original data file can also be referenced by multiple metadata; therefore, after obtaining the original data file according to each associated metadata, the original data file path of each original data file is taken as the “Key” value, and all metadata (that is, reference metadata) referring to the file and the data row identifier referred to by each reference metadata are aggregated.

[0048] The reference metadata set corresponding to the original data file is a set including all reference metadata. By obtaining the reference metadata set, the complete reference relationship covering the current original data file is ensured, and omission of part of the metadata is avoided to cause deviation of the hole rate calculation result. In particular, in order to ensure the storage normativity of the metadata and the data file, each original data file can be configured to have a fixed size, that is, to be stored in a preset storage threshold (for example, 16M), and each metadata is stored with a fixed number of data rows, that is, to be stored in a data row threshold (for example, 5).

[0049] For example, assuming that there are 10 associated metadata for the table A to be recycled, and each associated metadata includes 2 data rows marked as deleted (that is, deleted rows), and each deleted row is located in a different original data file, a total of 20 original data files to be executed for the recycling operation can be obtained. As described in the above technical solution, for each original data file, the corresponding reference metadata set is obtained, that is, a total of 20 reference metadata sets are obtained, each reference metadata set includes one or more reference metadata according to the number of metadata referencing the current original data file. For each original data file, the total number of data rows in the reference metadata set is accumulated, and the number of deleted rows is also accumulated, and then the ratio of the total number of deleted rows to the total number of data rows is taken as the hole rate of the original data file.

[0050] S104, arrange the original data files in descending order of hole rate, and take a target number of original data files as the recycled files according to the descending arrangement result, to recycle the hole data by rewriting the recycled files.

[0051] Since rewriting too much file data at a time will cause too long single execution time, and then cause too long time holding the write lock of the current table to be recycled, affecting the business use of the current table to be recycled by other business operations, a proper number of original data files need to be taken as the recycled files. According to the descending arrangement mode, that is, arranging the original data files according to the order from large to small of the hole rate, and then taking a specified number (that is, a target number) of original data files as the recycled files, and then rewriting the recycled files, the long write lock time of the current table to be recycled is avoided, and a high hole data recycling rate is ensured.

[0052] Optionally, in the embodiment of the present application, the target number of original data files are arranged as the to-be-recovered files according to the descending arrangement result, comprising: according to the descending arrangement result, the hole rates of the original data files are sequentially accumulated until the maximum accumulated result less than or equal to the preset recovery threshold is obtained, and then the target number is determined according to the maximum accumulated result, so as to arrange the target number of original data files as the to-be-recovered files.

[0053] Specifically, when the target number is a fixed number, the rewrite recovery time of each original data file in the to-be-recovered table cannot be accurately estimated, at this time, the hole rates of each original data file are accumulated according to the preset recovery threshold until the maximum accumulated result less than or equal to the preset recovery threshold is obtained, and then the target number is determined according to the maximum accumulated result, so as to realize accurate prediction of the rewrite recovery time.

[0054] For example, the preset recovery threshold is 60%, the hole rates of the first five original data files are accumulated, and the accumulated result is 58%, and the hole rate of the sixth original data file is 5%. Obviously, if the hole rates of the first six original data files are accumulated, the accumulated result 63% will exceed the preset recovery threshold 60%, therefore, five is taken as the target number, and the first five original data files are rewritten and recovered, so that a reasonable target number is obtained, and the long write lock time of the current to-be-recovered table is further avoided, and a high hole data recovery rate is ensured.

[0055] The technical scheme of the embodiment of the present application responds to the obtained hole data recovery instruction, takes the non-system table with changed data in the database as the to-be-recovered table, obtains the association information according to the association metadata set of the to-be-recovered table, aggregates the reference metadata set corresponding to each original data file according to the path of each original data file, obtains the hole rate of the corresponding original data file according to the reference metadata set, arranges each original data file in descending order of the hole rate, and arranges the target number of original data files as the to-be-recovered files according to the descending arrangement result, so as to rewrite and recover the hole data of the to-be-recovered files. Therefore, not only the configuration of a large number of physical resources as intermediate carriers in the rewriting process is avoided, but also the hole data recovery efficiency is greatly improved, the long write lock time of the current to-be-recovered table is avoided, the read and write operations of other businesses on the table are ensured, and the hole data recovery rate is improved.

[0056] Embodiment two

[0057] Figure 2 is a flowchart of a hole data recovery method of a database provided by the second embodiment of the present application. The relationship between the present embodiment and the above-mentioned embodiments is that the hole data recovery actually includes metadata rewriting and original data file rewriting, as shown in Figure 2 The method specifically comprises:

[0058] S201, start a rewrite execution transaction, and delete the delete rows in the to-be-recycled file through the rewrite execution transaction to generate a new file that does not include the hole data.

[0059] The transaction refers to a set of indivisible operations in a database management system, and the operations in the set are executed successfully or failed and rolled back to the initial state, that is, back to the state before the to-be-recycled file and the metadata are rewritten, to ensure the atomicity, consistency and integrity of the data table rewriting data; the rewrite execution transaction refers to a transaction used to execute the data table rewriting operation.

[0060] After the original data file is read through the rewrite execution transaction, the data rows that are marked as deleted (that is, have a delete mark) are deleted, and a new file that does not include the row data is regenerated, that is, a new file that does not include the hole data is generated; in particular, the rewriting task of the original data file can be distributed to multiple computing nodes of the database for parallel execution, and then summarized after the rewriting is completed, so as to improve the rewriting efficiency of the original data file.

[0061] S202, according to the reference metadata set corresponding to the to-be-recycled file, obtain the to-be-rewritten data row and the to-be-reorganized data row.

[0062] According to the rewriting requirement of the to-be-recycled file, it is necessary to determine which metadata of the "Key" needs to be updated, because for a "Key", since one metadata corresponds to multiple data rows, part of the data rows need to be rewritten (that is, the to-be-rewritten data row) to generate new metadata, and part of the data rows do not need to be rewritten (that is, the to-be-reorganized data row) and can be kept as they are. The rewritten data row creates a new version of metadata to reflect the updated state.

[0063] Taking the user's business order management as an example, the "Key" can be the user ID (Identity document, identity card identification number), and each metadata corresponds to multiple orders of a user (that is, one data row records an order information). Assuming that initially, user A has 10 orders, and since each metadata includes at most five data rows (that is, the data row threshold is 5), the 10 orders of user A will be stored in two metadata, that is, metadata A1 includes orders 1-5, and metadata A2 includes orders 6-10. Assuming that orders 1, 3 and 7 have expired (for example, have been deleted), then orders 1, 3 and 7 (corresponding data rows) are to-be-rewritten data rows, and corresponding orders 2, 4, 5, 6, 8, 9 and 10 are not expired, so orders 2, 4, 5, 6, 8, 9 and 10 (corresponding data rows) do not need to be rewritten, that is, they are to-be-reorganized data rows.

[0064] S203, rewrite each to-be-rewritten data row, and configure at least one rewritten data row as rewritten metadata.

[0065] After rewriting orders 1, 3, and 7, new metadata, that is, rewritten metadata (for example, metadata A3) is generated. In particular, only part of the data rows in the reference metadata is recycled because not all data is expired (for example, order 2 is an active order), which leads to the splitting of metadata A1 and metadata A2, that is, metadata A1 contains mixed states (that is, order 1 needs to be rewritten, and order 2 does not need to be rewritten), and direct queries may lead to incorrect query results (for example, order 1 is invalid, but the metadata still points to the order). The new metadata A3 isolates such changes and avoids pollution of the original data.

[0066] S204, recombine each to-be-recombined data row to configure multiple to-be-recombined data rows as recombined metadata based on the data row threshold.

[0067] For data rows that do not need to be rewritten, that is, to-be-recombined data rows, data row recombination is directly performed. Since the data row threshold limits the maximum number of each metadata (for example, 5), orders 2, 4, 5, 6, and 8 are combined into recombined metadata A4, and orders 9 and 10 are combined into recombined metadata A5. In particular, recombined metadata A5 still has 3 available data row storage locations, which can be provided to to-be-recombined data rows of other reference metadata (for example, reference metadata of user B) to reduce the number of managed metadata.

[0068] In particular, an empty data (for example, empty data A) can also be generated as a pointer of the newly obtained metadata, used to redirect the new metadata, that is, rewritten metadata A3, recombined metadata A4, and recombined metadata A5. Accordingly, the new metadata actually includes empty data A, rewritten metadata A3, recombined metadata A4, and recombined metadata A5 when the rewritten execution transaction is committed.

[0069] S205, associate the new file with the rewritten metadata and the recombined metadata, and commit the rewritten execution transaction.

[0070] After the new file is generated, the path and data row mapping relationship of the new file and other associated information are recorded, and the new metadata is associated with the new metadata through the associated information. If the new metadata is written and the new file is generated successfully, the current rewritten execution transaction can be committed; if the new metadata fails to be written or the new file fails to be generated, the transaction is rolled back to the state before rewriting to ensure data consistency.

[0071] Optionally, in the embodiment of the present application, after obtaining the to-be-rewritten data rows and the to-be-recombined data rows, the method further comprises: rewriting each to-be-rewritten data row, and recombining each rewritten data row with the to-be-recombined data rows in the same reference metadata to obtain new metadata; associating the new file with the new metadata, and submitting the rewriting execution transaction.

[0072] Specifically, still taking the above technical solution as an example, while rewriting the to-be-rewritten data rows (i.e., the above orders 1, 3, and 7) to obtain rewritten data rows, the to-be-recombined data rows (i.e., orders 2, 4, 5, 6, 8, 9, and 10) are temporarily stored, and after the rewritten data rows of orders 1, 3, and 7 are obtained, the rewritten data rows and the to-be-recombined data rows are combined according to the original sequence, that is, the rewritten data row 1, the to-be-recombined data row 2, the rewritten data row 3, the to-be-recombined data row 4, and the to-be-recombined data row 5 are combined to form a new metadata A6, and the to-be-recombined data row 6, the rewritten data row 7, the to-be-recombined data row 8, the to-be-recombined data row 9, and the to-be-recombined data row 10 are combined to form a new metadata A7. Compared with the above technical solution, the data rows in each metadata still maintain the original sequence, and the number of metadata (only two metadata A6 and A7, and three metadata A3, A4, and A5 are required) is reduced, and the management complexity of the metadata is reduced.

[0073] The technical solution of the embodiment of the present application opens a rewriting execution transaction, deletes the deleted rows in the to-be-recovered file through the rewriting execution transaction, generates a new file that does not include hole data, ensures the atomicity, consistency, and integrity of the rewritten data of the data table, obtains the to-be-rewritten data rows and the to-be-recombined data rows according to the reference metadata set corresponding to the to-be-recovered file, obtains the rewritten metadata and the recombined metadata according to the to-be-rewritten data rows and the to-be-recombined data rows, and isolates the different state data rows contained in the metadata through the rewritten metadata and the recombined metadata, thereby avoiding the pollution of the original data.

[0074] Embodiment Three

[0075] Figure 3 is a flowchart of a hole data recovery method of a database provided by the third embodiment of the present application. The relationship between the present embodiment and the above embodiments is that, for the old version of the to-be-recovered file and the reference metadata set, after it is determined that there is no running transaction, the to-be-recovered file and the reference metadata set are deleted, as shown in Figure 3 The method specifically comprises the following steps.

[0076] S301, a rewriting execution transaction is opened, and a deleted row in the to-be-recovered file is deleted through the rewriting execution transaction to generate a new file that does not include hole data.

[0077] S302, acquire the data row to be rewritten and the data row to be reorganized according to the reference metadata set corresponding to the file to be recycled.

[0078] S303, rewrite each data row to be rewritten, and configure at least one rewritten data row after rewriting as rewritten metadata.

[0079] S304, reorganize each data row to be reorganized, so as to configure a plurality of data rows to be reorganized as reorganized metadata based on the data row threshold.

[0080] S305, associate the new file with the rewritten metadata and the reorganized metadata, and submit the rewritten execution transaction.

[0081] S306, continue to save the file to be recycled and the reference metadata set.

[0082] After rewriting, the original data file (i.e. the file to be recycled) and the metadata (i.e. the reference metadata set) of the old version still occupy storage space, but since other transactions may still continue to read the old version of the above data, if deleted at this time, it will cause business error, therefore, the file to be recycled and the reference metadata are first retained to ensure the normal operation of each business.

[0083] S307, acquire the active transaction identifier of the current database, and mark the expired metadata in the reference metadata set through the active transaction identifier.

[0084] Acquire a list of all running transactions in the current database, and record the identifier of each active transaction therein; compare the version number of the metadata with the active transaction identifier, and mark the metadata "Key" as expired if all active transactions do not reference it, that is, mark the metadata whose version is earlier than the start event of all active transactions as expired metadata.

[0085] S308, traverse the original data file in the file to be recycled through the expired metadata, to acquire the expired original data file.

[0086] Traverse the original data file path referenced by the "Key" of the expired metadata, and mark all original data files referenced only by the expired metadata as expired original data files.

[0087] S309, delete the expired metadata and the expired original data file.

[0088] Delete the expired metadata (for example, from the metadata storage engine) and the expired original data file (for example, from the file system or object storage) according to the marking result, so as to release the storage space.

[0089] The technical scheme of the embodiment of the application, after associating a new file with the rewrite metadata and the reorganization metadata and submitting a rewrite execution transaction, continues to save the to-be-reclaimed file and the reference metadata set to ensure that other transaction services continue to read the original data, then acquires the active transaction identifier of the current database and marks the expired metadata through the active transaction identifier, traverses the original data file path in the to-be-reclaimed file through the expired metadata to acquire the expired original data file, and finally deletes the expired metadata and the expired original data file, thereby realizing the timely reclaiming of the hollow data storage space.

[0090] Embodiment four

[0091] Figure 4 is a structural block diagram of a database hollow data reclaiming device provided by the fourth embodiment of the application, which specifically comprises:

[0092] The to-be-reclaimed table acquisition module 401 is configured to, in response to acquiring a hollow data reclaiming instruction, take a non-system table with changed data in the database as a to-be-reclaimed table.

[0093] The association information acquisition module 402 is configured to acquire association information according to the association metadata set of the to-be-reclaimed table; wherein the association information comprises an original data file path.

[0094] The hollow rate acquisition module 403 is configured to aggregate the reference metadata set corresponding to each original data file according to each original data file path, and acquire the hollow rate of the corresponding original data file according to the reference metadata set.

[0095] The sorting execution module 404 is configured to arrange the original data files in descending order of the hollow rate, and take a target number of original data files as to-be-reclaimed files according to the descending arrangement result, so as to reclaim the hollow data by rewriting the to-be-reclaimed files.

[0096] The technical scheme of the embodiment of the application, in response to acquiring a hollow data reclaiming instruction, takes a non-system table with changed data in the database as a to-be-reclaimed table; acquires association information according to the association metadata set of the to-be-reclaimed table; aggregates the reference metadata set corresponding to each original data file according to each original data file path, and acquires the hollow rate of the corresponding original data file according to the reference metadata set; arranges the original data files in descending order of the hollow rate, and takes a target number of original data files as to-be-reclaimed files according to the descending arrangement result, so as to reclaim the hollow data by rewriting the to-be-reclaimed files. Therefore, not only is it avoided to configure a large number of physical resources as intermediate carriers of the rewriting process, but also the hollow data reclaiming efficiency is greatly improved, which not only avoids occupying a long write lock time of the current to-be-reclaimed table, ensures the read-write operation of other services on the table, but also improves the hollow data reclaiming rate.

[0097] Optionally, the to-be-recovered table acquisition module 401 is specifically configured to sequentially accumulate the hole rates of the original data files according to the descending arrangement result until a maximum accumulated result less than or equal to a preset recovery threshold is obtained, and determine a target number according to the maximum accumulated result, so as to take the original data files of the target number as the to-be-recovered files.

[0098] Optionally, the sorting execution module 404 is specifically configured to start a rewrite execution transaction, and delete the delete rows in the to-be-recovered files through the rewrite execution transaction, to generate a new file not including hole data.

[0099] Optionally, the sorting execution module 404 is specifically further configured to acquire to-be-rewritten data rows and to-be-reorganized data rows according to the reference metadata set corresponding to the to-be-recovered files; rewrite each to-be-rewritten data row, and configure at least one rewritten data row after rewriting as rewritten metadata; reorganize each to-be-reorganized data row, to configure a plurality of to-be-reorganized data rows as reorganized metadata based on the data row threshold; associate the new file with the rewritten metadata and the reorganized metadata, and submit the rewrite execution transaction.

[0100] Optionally, the sorting execution module 404 is specifically further configured to rewrite each to-be-rewritten data row, and recombine each rewritten data row after rewriting and to-be-reorganized data rows in the same reference metadata as new metadata; associate the new file with the new metadata, and submit the rewrite execution transaction.

[0101] Optionally, the sorting execution module 404 is specifically further configured to continue to save the to-be-recovered files and the reference metadata set; acquire an active transaction identifier of a current database, and mark expired metadata in the reference metadata set through the active transaction identifier; traverse the original data files in the to-be-recovered files through the expired metadata, to acquire expired original data files; delete the expired metadata and the expired original data files.

[0102] The above-described apparatus can execute the database hole data recovery method provided by any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiment can be referred to the database hole data recovery method provided by any embodiment of the application.

[0103] Embodiment Five

[0104] Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0105] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method for recovering vacant data from a database.

[0108] In some embodiments, the hole data reclamation method of a database can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., a memory unit. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the heterogeneous hardware accelerator via a ROM and / or communication unit. When the computer program is loaded onto the RAM and executed by the processor, one or more steps of the hole data reclamation method of a database described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the hole data reclamation method of a database by way of other any suitable means, e.g., by way of firmware.

[0109] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0110] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0111] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide for interaction with a user, the systems and techniques described here can be implemented on a heterogeneous hardware accelerator having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the heterogeneous hardware accelerator. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0113] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0114] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0115] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, and this is not limited herein.

[0116] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of hole data reclamation of a database, characterized by, The method comprises the following steps: In response to obtaining a hole data recycling instruction, a non-system table in the database with changed data is taken as a table to be recycled; Obtain association information according to the association metadata set of the table to be recycled; wherein the association information comprises an original data file path; According to each original data file path, aggregate to obtain the reference metadata set corresponding to each original data file respectively, and obtain the hole rate of the corresponding original data file according to the reference metadata set; wherein the hole rate refers to the ratio of the total number of deleted rows to the total number of data rows in the reference metadata set; Arrange each original data file in descending order of hole rate, and take a target number of original data files as recycled files according to the descending arrangement result, to recycle hole data by rewriting the recycled files.

2. The hole data recovery method of a database according to claim 1, wherein, According to the descending arrangement result, the hole rate of each original data file is sequentially accumulated until the maximum accumulated result less than or equal to the preset recycling threshold is obtained, and the target number of original data files is determined as the recycled files according to the maximum accumulated result. The method comprises the following steps:

3. The hole data recovery method of a database according to claim 1, wherein, Start a rewrite execution transaction, and delete the deleted rows in the recycled files through the rewrite execution transaction to generate a new file that does not include hole data. After generating the new file that does not include hole data, the method further comprises the following steps:

4. The hole data recovery method of a database according to claim 3, wherein, According to the reference metadata set corresponding to the recycled files, obtain data rows to be rewritten and data rows to be reorganized; Rewrite each data row to be rewritten, and configure at least one rewritten data row as rewritten metadata; Reorganize each data row to be reorganized to configure multiple data rows to be reorganized as reorganized metadata based on a data row threshold; Associate the new file with the rewritten metadata and the reorganized metadata, and submit the rewrite execution transaction. After obtaining the data rows to be rewritten and the data rows to be reorganized, the method further comprises the following steps:

5. The hole data recovery method of a database according to claim 3, wherein, Rewrite each data row to be rewritten, and recombine each rewritten data row with the data rows to be reorganized in the same reference metadata to form new metadata; Associate the new file with the new metadata, and submit the rewrite execution transaction. After associating the new file with the rewritten metadata and the reorganized metadata, and submitting the rewrite execution transaction, the method further comprises the following steps:

6. The hole data recovery method of a database according to claim 4, wherein, Continue to save the recycled files and the reference metadata set; Obtain the active transaction identifier of the current database, and mark the expired metadata in the reference metadata set through the active transaction identifier; Through the expired metadata, traverse the original data files in the recycled files to obtain expired original data files; Delete the expired metadata and the expired original data files. The method comprises the following steps:

7. A hole data reclamation apparatus of a database, characterized by comprising: A recycled table obtaining module is configured to, in response to obtaining a hole data recycling instruction, take a non-system table in the database with changed data as a table to be recycled; ​ The association information acquisition module is configured to acquire association information according to the association metadata set of the table to be recycled; wherein the association information comprises an original data file path; The hole rate acquisition module is configured to aggregate the reference metadata set corresponding to each original data file according to each original data file path, and acquire the hole rate of the corresponding original data file according to the reference metadata set; wherein the hole rate refers to the ratio of the total number of deleted rows to the total number of data rows in the reference metadata set; The sorting execution module is configured to arrange each original data file in descending order of hole rate, and arrange a target number of original data files as the file to be recycled according to the descending arrangement result, so as to recycle the hole data by rewriting the file to be recycled.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the database hole data recycling method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the database hole data recycling method of any one of claims 1-6 when executed.

10. A computer program product comprising a computer program which, when executed by a processor, implements the database hole data recycling method of any one of claims 1-6.

Images