Garbage data recovery method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for garbage data recovery, which relate to the fields of computer storage and database technology. For example, the present application can be used in the field of distributed databases. The method includes: in response to satisfying the merge conditions of the LSM Tree, performing a merge operation on the LSM Tree; determining the first data volume of the invalid BLOB and the second data volume of the valid BLOB stored in each BLOB file; for different situations of the first data volume and the second data volume, respectively recovering garbage data by performing a hole punching operation and rewriting the valid BLOB to recover garbage data. The garbage data recovery method of the embodiments of the present application reduces the consumption of system I / O resources, effectively avoids the reduction of system performance, and improves the efficiency of garbage data recovery, and releases storage space in a timely manner.

Description

Technical Field

[0001] The present application relates to the fields of computer storage and database technology. Specifically, the present application relates to a garbage data recovery method, device, electronic device and computer-readable storage medium. Background Art

[0002] Log-Structured Merge-Tree (LSM Tree) is a data structure widely used to manage persistent data, especially for write-intensive application scenarios. However, when processing write operations, LSM Tree needs to periodically merge data from memory to disk due to its design, which will lead to write amplification problem, that is, the amount of data actually written to disk is much larger than the original data amount.

[0003] To solve this problem, the existing technology adopts the key-value separation method, that is, only the key is retained in the LSMTree and the value is stored elsewhere to reduce write amplification. However, this method increases the instantaneous demand for I / O resources during garbage collection, resulting in a decrease in system response time and throughput during garbage collection. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium, and computer program product for garbage data recovery, which can solve the problem of high I / O resource requirements in related technologies during garbage data recovery, resulting in reduced system performance. The technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, a method for garbage data recovery is provided, which is applied to a key-value separation storage system, wherein the storage system includes a first storage area and a second storage area, wherein the first storage area stores data in the form of a log structure merge tree LSM Tree, and the second storage area is used to store multiple binary large object BLOB files corresponding to the LSM Tree; the method includes:

[0006] In response to satisfying a merge condition of the LSM Tree, performing a merge operation on the LSM Tree; the LSM Tree stores at least one first key-value pair, and the key and value of each first key-value pair are respectively an identifier and a storage address of a BLOB in a BLOB file;

[0007] For each BLOB file in the second storage area, determine a first data amount of an invalid BLOB and a second data amount of a valid BLOB stored in the BLOB file; wherein the valid BLOB is a BLOB corresponding to a first key-value pair that is not deleted or overwritten in the LSM Tree after a merge operation is performed on the LSM Tree;

[0008] For each BLOB file, perform the following target operations:

[0009] If the first data volume of the BLOB file exceeds a first preset threshold, and the proportion of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is taken as a file to be punched and added to a recycling candidate list, so as to perform garbage data recovery on the file to be punched in the recycling candidate list through a punch hole operation;

[0010] If the ratio of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed a second preset threshold, the BLOB file is used as a reclaimed file to be rewritten, and the valid BLOB in the BLOB file is rewritten to obtain a new BLOB file.

[0011] According to a second aspect of an embodiment of the present application, a garbage data recovery device is provided, which is applied to a key-value separation storage system, wherein the storage system includes a first storage area and a second storage area, wherein the first storage area stores data in the form of a sampling log result merge tree LSM Tree, and the second storage area is used to store multiple binary large object BLOB files corresponding to the LSM Tree, and the device includes:

[0012] A merge operation module, configured to perform a merge operation on the LSM Tree in response to satisfying a merge condition of the LSM Tree; the LSM Tree stores at least one first key-value pair, wherein the key and the value of each first key-value pair are respectively an identifier and a storage address of a BLOB in a BLOB file;

[0013] a data volume determination module, configured to determine, for each BLOB file in the second storage area, a first data volume of an invalid BLOB and a second data volume of a valid BLOB stored in the BLOB file; wherein the valid BLOB is a BLOB corresponding to a first key-value pair that is not deleted or overwritten in the LSM Tree after a merge operation is performed on the LSM Tree;

[0014] The target operation execution module is used to execute the following target operations for each BLOB file:

[0015] If the first data volume of the BLOB file exceeds a first preset threshold, and the proportion of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is treated as a file to be recovered by punching holes and added to a recovery candidate list, so as to perform garbage data recovery on the files to be recovered by punching holes in the recovery candidate list through a punch hole operation; if the proportion of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold, the BLOB file is treated as a file to be rewritten and recovered, and the valid BLOB in the BLOB file is rewritten to obtain a new BLOB file.

[0016] According to a third aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0017] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0018] According to the fifth aspect of the embodiment of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When a processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device executes the steps of implementing the method provided in the first aspect.

[0019] The technical solution provided by the embodiment of the present application has the following beneficial effects: by performing a merge operation on the LSM Tree when the merge condition of the LSM Tree is met, the storage information of the LSM Tree is updated, and after the storage information of the LSM Tree is updated, the subsequent garbage data recovery step is triggered. When performing garbage data recovery, by analyzing each BLOB file in the second storage area, the first data volume of the invalid BLOB and the second data volume of the valid BLOB in each BLOB file are determined; by comparing the first data volume with the first preset threshold, if the proportion of the second data volume to the total data volume of the BLOB file to which it belongs exceeds the second preset threshold, the BLOB file is used as a file to be recovered by punching holes, and the file to be recovered by punching holes is added to the recovery candidate list, so that garbage data recovery is performed on the files to be recovered by punching holes in the recovery candidate list in the subsequent recovery through a punching operation; by using the file whose proportion of the second data volume of each BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold as a file to be rewritten, garbage data recovery is achieved by rewriting the valid BLOB in the BLOB file to a new BLOB file. The embodiments of the present application adopt different garbage data recovery methods for BLOB files according to the different situations of the data volume of valid BLOBs and the data volume of invalid BLOBs in BLOB files. When the data volume of invalid BLOBs is large and the data volume of valid BLOBs is also large, garbage data of BLOB files is recovered by punching operations, thereby reducing the consumption of I / O resources and solving the problem of excessive consumption of I / O resources leading to degradation of system performance in related technologies. When the data volume of valid BLOBs is small, garbage data recovery is performed by sampling and rewriting valid BLOBs, thereby improving the efficiency of garbage data recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0021] Figure 1 A schematic diagram of a system architecture for implementing garbage data recovery provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a garbage data recovery method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of another process of garbage data recovery provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a process for garbage data recovery by hole punching operation provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a data storage structure of a first storage area and a second storage area provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a data storage format of a BLOB file provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a flow chart of another garbage data recovery method provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the structure of a garbage data recovery device provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0031] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0033] First, several terms involved in this application are introduced and explained:

[0034] Log-Structured Merge Tree (LSM Tree) is a data structure commonly used in database systems. It converts random writes into sequential writes, greatly improving the performance of writing data to disk. The core of LSM Tree is to separate the write operation from the merge operation, and improve the write and query performance by writing data to log files and memory cache, and then performing merge operations regularly.

[0035] However, the merging process of LSM Tree brings about read amplification and write amplification problems. The read operation of LSM Tree needs to search for the data from the upper level to the lower level layer by layer. This process may involve more than one I / O, especially for range queries, where the read amplification problem is more serious. In addition, because all writes are sequential, deletions and updates are not in-place updates, so expired data will not be cleaned up, occupying a part of the space, resulting in space amplification problems.

[0036] When the LSM Tree's external memory space is insufficient, the files in the external memory that have orderly key values ​​need to be merged regularly. During this period, multiple files need to be read into the memory, deleted or updated key-value pairs need to be deleted, and all data needs to be merged and sorted before being written into a new file in the external memory. In addition, the merge operation will be performed between different levels and between upper and lower layers, and the write amplification problem is very serious.

[0037] WiscKey is a persistent key-value storage system optimized for solid-state drives (SSDs). It is built on LSM Tree and has a performance-oriented data layout. In order to reduce the write amplification problem caused by merging, Wisckey proposes key-value separation. The values ​​are stored in a circular log. The LSM Tree merge process does not delete invalid values ​​in the circular log, and garbage collection is required afterwards. In addition, Wisckey's circular log for storing values ​​can only perform garbage collection from the tail of the log. When there are many valid values ​​at the tail, the garbage collection process has a large write amplification problem and reduces the service life of the SSD.

[0038] At the same time, the garbage collection process of the key-value separation storage system also has the problem of read amplification. During the garbage collection process, it is not known whether the key-value pair in the ring log is valid, and the LSM Tree needs to be queried. If the key corresponding to the value is found, the value address is compared with the current address of the value in the ring log to see if it is consistent. The garbage collection operation is performed based on the result obtained from the LSM Tree, which increases the instantaneous use of system I / O resources.

[0039] The garbage data recovery method, device, electronic device, computer-readable storage medium and computer program product provided in this application are intended to solve the above technical problems of related technologies.

[0040] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0041] Figure 1 A schematic diagram of the system architecture for implementing garbage data recovery provided in an embodiment of the present application, wherein the terminal 101 can be a user terminal, the server 102 can be a database server, and the database server includes a key-value separation storage system, and the key-value separation storage system includes a first storage area and a second storage area, wherein the first storage area stores data in the form of an LSM Tree, and the second storage area is used to store the BLOB file corresponding to the LSM Tree. It should be understood that a piece of data includes two attributes, a key and a value. The key-value separation storage system in the embodiment of the present application stores the value with a larger amount of data separately from its corresponding key. Specifically, the LSM Tree stores data in the form of a key-value pair, and the LSM Tree includes a first key-value pair and a second key-value pair, wherein the value of the first key-value pair is the storage address of a binary large object BLOB, and the real BLOB is stored in the BLOB file, and the value of the first key-value pair in the LSM Tree, that is, the storage address of the BLOB, is used to reference the corresponding BLOB, and the value of the second key-value pair is the value of the data. That is, the embodiment of the present application does not separate the key-value pairs with a small amount of data.

[0042] In an embodiment of the present application, the user terminal continuously performs operations such as adding, deleting, and modifying data through write operations. When the amount of data in the LSM Tree reaches a certain condition, the server triggers a merge operation on the LSM Tree. The merge operation of the LSM Tree is used to delete key-value pairs, or overwrite the value corresponding to a key. It should be noted that when overwriting the value of a key-value pair, when the key-value pair is the first key-value pair, the storage address of the latest value corresponding to the key is used to overwrite the storage address corresponding to the key before; when the key-value pair is the second key-value pair, the latest value corresponding to the key is used to overwrite the value corresponding to the key before.

[0043] When the key-value pair is the first key-value pair, after the first key-value pair is deleted or the corresponding value of the key-value pair is overwritten, the BLOB corresponding to the key-value pair in the BLOB file is no longer referenced by the LSM Tree and becomes an invalid BLOB. The BLOB corresponding to the first key-value pair that has not been deleted or overwritten is still referenced and is a valid BLOB. Therefore, after performing a merge operation on the LSM Tree, each BLOB file may generate a certain number of invalid BLOBs, and garbage collection operations need to be performed.

[0044] In an embodiment of the present application, the garbage data recovery method is determined based on the data volume of the valid BLOB files and the data volume of the invalid BLOB files of each BLOB file. When the valid data volume is small, the garbage data recovery is performed by rewriting to prevent the second storage area from having a large number of files with small data volumes, which increases the difficulty of LSM Tree reference. It should be understood that when the garbage data is recovered by rewriting, since the valid BLOB is written to the new file, the storage address of the valid BLOB has changed, so it is necessary to update the storage address corresponding to the BLOB in the LSM Tree, which consumes I / O resources. Therefore, in an embodiment of the present application, when the invalid data volume is large and the valid data volume is also large, the garbage data recovery is performed by punching operations, and the valid data is not rewritten, thereby reducing the consumption of I / O resources.

[0045] In some embodiments, the terminal 101 is a device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. that can be used to access a database, but is not limited thereto. The terminal 101 is installed with an application that supports database access, which is not limited in the embodiments of the present application.

[0046] In some embodiments, the server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server 102 is used to operate the data in the first storage area and the second storage area according to the instructions of the terminal 101, and after the merging conditions of the LSM Tree are met, perform a merge operation on the LSM Tree and perform corresponding garbage data recovery.

[0047] The present application provides a method for garbage data recovery in an embodiment. The method is applied to a key-value separation storage system. The storage system includes a first storage area and a second storage area. The first storage area stores data in the form of a log structure merge tree LSM Tree, and the second storage area is used to store multiple binary large object BLOB files corresponding to the LSM Tree, such as Figure 2 As shown, the method includes S201-S203, wherein S203 includes S2031 and S2032.

[0048] S201. In response to satisfying the merge condition of the LSM Tree, perform a merge operation on the LSM Tree.

[0049] The garbage data recovery method in the embodiment of the present application can be applied in a key-value separation storage system, which includes a first storage area and a second storage area. The first storage area stores data in the form of an LSM Tree, and the second storage area is used to store multiple BLOB files corresponding to the LSM Tree, and the BLOB file is used to store BLOB data, and BLOB is a binary large object. BLOB is a data type used to store binary data, such as large files such as pictures, audio, video, and documents. The BLOB data type can store a large amount of data, and in the embodiment of the present application, data of the BLOB type is collectively referred to as BLOB.

[0050] In an embodiment of the present application, LSM Tree stores data in the form of key-value pairs, and the data includes at least one first key-value pair, and may also include multiple second key-value pairs, wherein the key and value of the first key-value pair are the identifier and storage address of a BLOB in a BLOB file, respectively. It should be understood that in the data field, one data corresponds to one key-value pair, wherein the key is the unique identifier of the value; the key and value of the second key-value pair may be the identifier and the data of data with a smaller data volume. In an embodiment of the present application, by setting the value of the first key-value pair to the storage address of a BLOB, the corresponding BLOB can be determined by the storage address, so the first key-value pair realizes the reference to the BLOB through the storage address. It should be understood that the amount of data at the storage address is significantly smaller than the amount of data of a BLOB data, so the embodiment of the present application reduces the storage pressure of the LSM Tree.

[0051] In the embodiment of the present application, the merge condition of the LSM Tree may be the same as the merge condition of the LSM Tree in the prior art. For example, when the amount of data in the memory reaches a threshold, when the amount of data in the MemTable (C0 layer) in the memory reaches a certain size, a merge operation is triggered to migrate the data to the SSTable on the disk. When the number of SSTables in a certain layer reaches a set threshold, a merge operation is triggered to merge multiple SSTables in the layer into a new SSTable and put it into a higher layer. When the merge condition of the LSM Tree is triggered, a merge operation is performed on the LSM Tree. The result of the merge operation is that the key-value pairs in the LSM Tree will be updated, thereby deleting some key-value pairs or overwriting the value of a key with a new value.

[0052] By performing a merge operation on the LSM Tree, the key-value pairs in the LSM Tree are updated, reducing the number of SSTables that need to be accessed during read operations. This is because the merged data is stored in fewer and larger SSTables, reducing the I / O operations when users read data.

[0053] S202: For each BLOB file in the second storage area, determine a first data volume of invalid BLOBs and a second data volume of valid BLOBs stored in the BLOB file.

[0054] In the embodiment of the present application, after performing the merge operation on the LSM Tree, for each BLOB in each BLOB file, the first key-value pair corresponding to the BLOB may be deleted, or the value in the first key-value pair may be overwritten with another storage address. In these two cases, there is no first key-value pair to reference the BLOB, and the BLOB has no use value. If it continues to exist in the second storage area, it will cause the space expansion problem of the second storage area. Therefore, it is necessary to perform garbage collection on the unreferenced BLOBs.

[0055] In an embodiment of the present application, for each BLOB file, based on the data volume of the valid BLOBs and the data volume of the invalid BLOBs in the BLOB file, it is determined whether to perform garbage data recovery on the BLOB file, and what garbage data recovery method to sample for garbage data recovery. Wherein, the valid BLOB is the BLOB corresponding to the first key-value pair that is not deleted or overwritten in the LSM Tree after the merge operation is performed on the LSM Tree, and the invalid BLOB is the BLOB that is deleted or overwritten in the LSMTree after the merge operation is performed on the LSM Tree. The data volume of the valid BLOB and the data volume of the invalid BLOB in each BLOB file may be determined based on the reference relationship between each first key-value pair in the LSM Tree after the merge operation, and the data volume of the invalid BLOB in each BLOB file is referred to as the first data volume corresponding to the BLOB file, and the data volume of the valid BLOB is referred to as the second data volume of the BLOB file.

[0056] After determining the first data volume and the second data volume of each BLOB file, step S2031 or S2032 is performed for each BLOB file.

[0057] S2031. If the first data volume of the BLOB file exceeds a first preset threshold, and the proportion of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is treated as a file to be punched and added to a recovery candidate list, so as to perform garbage data recovery on the file to be punched in the recovery candidate list through a punch hole operation.

[0058] In an embodiment of the present application, for each BLOB file, a first preset threshold and a second preset threshold are preset, wherein the first preset threshold is used to compare with the first data volume. When the first data volume is greater than the first preset threshold, it indicates that the invalid BLOB has a large data volume. The second preset threshold is used to compare with the ratio of the second data volume to the total data volume of the BLOB file. When the ratio of the second data volume to the total data volume of the BLOB file is greater than the second preset threshold, it indicates that the valid data volume in the BLOB file is large. As an optional embodiment, the first preset threshold can be set to 4MB, and the second preset threshold can be set to 20%.

[0059] The total data volume of each BLOB file can be stored in the meta information of the BLOB file, and the meta information can be stored in the meta information storage block of the BLOB file. The total data volume of each BLOB file can be the sum of the first data volume, the second data volume and the data volume recovered by the hole punching operation in the BLOB file.

[0060] In an embodiment of the present application, for each BLOB file, if the first data volume is greater than a first preset threshold, and the ratio of the second data volume to the BLOB file is greater than a second preset threshold, the BLOB file is added to a recycling candidate list. As an optional embodiment, the file identifier of the BLOB file can be added to the recycling candidate list to determine that the BLOB file corresponding to the file identifier is a file to be recycled by hole punching, and a hole punching operation is used to perform garbage data recovery on the files to be recycled by hole punching in the recycling candidate list.

[0061] The punch hole operation is a file system-level feature that allows you to create a "hole" at a specified location in an existing file, that is, to release disk space in a specific area of ​​the file, so that the area is still part of the file logically, but no longer occupies disk space physically. This operation is often used to optimize storage, especially when you need to delete a large amount of data in a file, by "punching a hole" in the file to free up space without affecting the total size of the file and the logical location of subsequent data. The punch hole operation is particularly useful in file systems that support sparse files, as it can reduce the disk space actually occupied by the file and improve storage efficiency.

[0062] In the embodiment of the present application, garbage data recovery is achieved by punching operations. Compared with the prior art of rewriting all BLOB files, the use of I / O resources will be greatly reduced, and the write amplification problem caused by the rewriting process will be further alleviated. At the same time, there is no need to write the storage address of the rewritten BLOB into the LSM tree, which reduces the access to the LSM tree, thereby having a smaller impact on the front-end business. Garbage data recovery by punching operations is more lightweight, the garbage data recovery process can be triggered more frequently, and the storage resources of the space will be released more timely.

[0063] S2032: If the ratio of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold, the BLOB file is used as a reclaimed file to be rewritten, and the valid BLOB in the BLOB file is rewritten to obtain a new BLOB file.

[0064] In an embodiment of the present application, for each BLOB file, the purpose of performing garbage data recovery on the BLOB file is to retain the valid BLOB in the BLOB file. Therefore, when the second data volume in the BLOB file is small, if the garbage data recovery is continued to be performed using the hole punching operation, when the first data volume is large, the garbage data recovery will take up more resources and time. Therefore, in an embodiment of the present application, when the ratio of the second data volume to the total data volume of the BLOB file does not exceed the second preset threshold, garbage data recovery is performed by rewriting, that is, the BLOB file is used as the file to be rewritten, and the valid BLOB file in the BLOB file is rewritten to obtain an information BLOB file.

[0065] As an optional embodiment, for each BLOB file, if the first data volume in the BLOB file is less than a first preset threshold and the ratio of the second data volume to the file data volume is greater than a second preset threshold, the BLOB file is not processed.

[0066] like Figure 3 As shown, Figure 3A flowchart of another garbage data recovery process provided for an embodiment of the present application. In which, the first storage area stores data in the form of an LSM Tree, and in response to the merge condition of the LSM Tree, a merge operation is performed on the LSM Tree. After the merge operation is completed, garbage data recovery is performed on the BLOB in the second storage area, specifically, the first data volume and the second data volume are obtained, and the ratio of the second data volume to the total data volume of the BLOB file is taken as m, and m is compared with a second preset threshold. If m does not exceed the second preset threshold, garbage data recovery is performed through a rewrite operation. If m exceeds the second preset threshold, the first data volume is taken as n, and n is compared with the first preset threshold. If the first data volume exceeds the first preset threshold, garbage data recovery is performed through a hole punching operation.

[0067] In the embodiment of the present application, by performing a merge operation on the LSM Tree when the merge condition of the LSM Tree is met, the storage information of the LSM Tree is updated, and after the storage information of the LSM Tree is updated, the subsequent garbage data recovery step is triggered. When performing garbage data recovery, each BLOB file in the second storage area is analyzed to determine the first data volume of the invalid BLOB and the second data volume of the valid BLOB in each BLOB file; by comparing the first data volume with the first preset threshold, if the proportion of the second data volume to the total data volume of the BLOB file to which it belongs exceeds the second preset threshold, the BLOB file is used as a file to be recovered by punching holes, and the file to be recovered by punching holes is added to the recovery candidate list, so that garbage data recovery is performed on the files to be recovered by punching holes in the recovery candidate list through subsequent punching operations; by using the files whose proportion of the second data volume of each BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold as files to be rewritten, garbage data recovery is achieved by rewriting the valid BLOB in the BLOB file to a new BLOB file. The embodiments of the present application adopt different garbage data recovery methods for BLOB files according to the different situations of the data volume of valid BLOBs and the data volume of invalid BLOBs in BLOB files. When the data volume of invalid BLOBs is large and the data volume of valid BLOBs is also large, garbage data of BLOB files is recovered by punching operations, thereby reducing the consumption of I / O resources and solving the problem of excessive consumption of I / O resources leading to degradation of system performance in related technologies. When the data volume of valid BLOBs is small, garbage data recovery is performed by sampling and rewriting valid BLOBs, thereby improving the efficiency of garbage data recovery.

[0068] As an optional embodiment of the present application, garbage data recovery is performed on the to-be-punched recovery files in the recovery candidate list through a punch hole operation, including:

[0069] At a preset time interval, periodically determine whether there are files to be recovered by punching holes in the recovery candidate list. If there are files to be recovered by punching holes, perform the following operations on each file to be recovered by punching holes:

[0070] Obtain the file to be recovered by hole punching, and generate a reference snapshot of the LSM Tree at the current moment corresponding to the file to be recovered by hole punching, where the reference snapshot is used to record each first key-value pair in the LSM Tree at the current moment;

[0071] Establishing a corresponding relationship between the reference snapshot and the file to be holed and recovered;

[0072] When all snapshots generated before the reference snapshot are deleted, garbage data is recovered on the file to be recovered by hole punching according to the reference snapshot through a hole punching operation.

[0073] In an embodiment of the present application, the key-value separation storage system may include a merge operation node and a data recovery node. The merge operation node is used to determine the file to be recovered by hole punching, and then add the file identifier of the file to be recovered by hole punching to a recovery candidate list. The data recovery node is used to periodically determine whether there is a file identifier in the recovery candidate list, so as to determine whether there is a file to be recovered by hole punching that needs to be recovered through garbage data recovery through hole punching operation.

[0074] In the embodiment of the present application, the snapshot is used to record the storage information of the LSM Tree at a certain moment. The snapshot is generally the storage information of the LSM Tree at the access moment generated for the user when the user accesses the database, so that multiple users at the same moment can access the data without affecting each other. When the user's access is over, the snapshot will be deleted. When performing a hole punching operation to delete invalid BLOBs on the file to be recovered, it is necessary to consider whether the BLOB is still in use in the historical snapshot. If the BLOB is deleted directly, it will affect the access of the user corresponding to the historical snapshot. In the embodiment of the present application, by judging whether all historical snapshots have been deleted, when all historical snapshots have been deleted, garbage data recovery is performed on the BLOB file.

[0075] Specific as Figure 4 As shown, Figure 4A flow chart of garbage data recovery by punching operation is provided for an embodiment of the present application. The recovery candidate list includes file identifiers of multiple files to be recovered by punching holes. The data recovery node periodically traverses the recovery candidate list. After determining that the file identifier "file 1" exists in the recovery candidate list, the corresponding BLOB file 1 to be recovered by punching holes is determined according to the file identifier, and a reference snapshot of the current moment is generated. Each snapshot includes a time attribute for recording the generation time of the corresponding snapshot. After the data recovery node generates a reference snapshot D for BLOB file 1, a corresponding relationship between BLOB file 1 and reference snapshot D is established. The snapshot list includes multiple snapshots, or includes snapshot information of multiple snapshots. It should be understood that the snapshots in the snapshot list may include snapshots generated before the generation time of reference snapshot D and snapshots generated after the generation time of reference snapshot D. For example, snapshot A and snapshot B are snapshots generated before the generation time of reference snapshot D, and snapshot C is a snapshot generated after the generation time of reference snapshot D. According to the time attributes of each snapshot, after determining that snapshot A and snapshot B are both deleted, garbage data recovery is performed on BLOB file 1 by punching holes.

[0076] As an optional embodiment of the present application, for all the files to be holed in the recycling candidate list, the same reference snapshot can be generated for all the files to be holed, and a corresponding relationship between the reference snapshot and each file to be holed is established.

[0077] In the art, the reference snapshot of the file to be recovered by hole punching can also be the oldest snapshot. It should be understood that from the process of obtaining the file to be recovered by hole punching to the actual garbage data recovery, the key-value separation storage system may be accessed by the user, that is, after generating the reference snapshot, multiple snapshots may be generated. Compared with these snapshots, the reference snapshot corresponding to the file to be recovered by hole punching is the oldest snapshot corresponding to the file to be recovered by hole punching.

[0078] In the embodiment of the present application, after obtaining the file to be recovered by hole punching, a corresponding reference snapshot is created to record the data information in the LSM Tree corresponding to the file to be recovered by hole punching, and when all historical snapshots generated before the generation time of the reference snapshot are deleted, garbage data of the file to be recovered by hole punching is recovered according to the reference snapshot, thereby preventing the problem of BLOB being deleted while being referenced by historical snapshots and affecting user access.

[0079] As an optional embodiment of the present application, each BLOB corresponds to a recycling mark, which is a first mark or a second mark. The first mark of each BLOB indicates that the BLOB has been recycled by punching a hole in the corresponding BLOB file, and the second mark of each BLOB indicates that the BLOB has not been recycled by punching a hole in the corresponding BLOB file.

[0080] Performing garbage data recovery on the to-be-drilled-hole-recovered file through a hole-punching operation according to the reference snapshot includes:

[0081] According to the second identifier in the file to be hole punched and recovered, determining each first BLOB in the file to be hole punched and recovered that has not been hole punched and recovered;

[0082] For each first BLOB, if the first key-value pair corresponding to the key of the first BLOB does not exist in the reference snapshot, or the first storage address corresponding to the key of the first BLOB in the reference snapshot is not the storage address of the first BLOB in the second storage area, the first BLOB is reclaimed through a hole punching operation, and the reclaim identifier corresponding to the first BLOB is updated to the first identifier.

[0083] In the embodiment of the present application, the recycling mark of each BLOB can be marked with 0 and 1, where 0 is the first mark, used to indicate that the BLOB has been recycled, and 1 is the second mark, used to indicate that the BLOB has not been recycled. Figure 5 As shown, Figure 5 A schematic diagram of a data storage structure of a first storage area and a second storage area provided in an embodiment of the present application. For each first key-value pair in the first storage area, k of the first key-value pair is the key, and v is a storage address in the second storage area. For each BLOB in the second storage area, the BLOB includes a regression identifier (i.e., the identifier in the figure), k, and binary data, wherein k is the key corresponding to the BLOB, and the binary data is the value corresponding to the BLOB. It should be understood that when a data is separated into a key value, the key and the storage address corresponding to the value constitute a first key-value pair, and the key, value, and recycling identifier constitute a BLOB. In the technical field of this invention, the value in the BLOB is also referred to as the BLOB. For distinction, the value is referred to as the binary data in the BLOB here. The BLOB also includes address information for indicating the storage address of the BLOB.

[0084] When performing garbage data recovery, taking the file 1 to be recovered by hole punching in the second storage area as an example, the file to be recovered by hole punching includes BLOB1, BLOB2, BLOB3 and BLOB4, among which, the BLOBs with the recovery mark as the second mark, i.e. 1, are BLOB2, BLOB3 and BLOB4. For BLOB2, the key of BLOB2 is 4, and there is no first key-value pair with key 4 in the reference snapshot, indicating that the first key-value pair corresponding to the key corresponding to BLOB2 has been deleted during the merge operation, so BLOB2 needs to be recovered by hole punching. When performing hole punching recovery, the hole punching recovery operation will change the recovery mark of BLOB2 to 0, that is, in an embodiment of the present application, the recovery mark can be set to 0 by the punching operation. For BLOB3, the key of BLOB3 is 1, the value of the first key-value pair with key 1 in the reference snapshot is a, and the storage address of BLOB3 is f. The storage address corresponding to the key of BLOB3 in the first key-value pair is not the storage address of BLOB3 in the second storage block, indicating that the key corresponding to BLOB3 is overwritten during the merge operation of LSM Tree, so BLOB3 needs to be reclaimed by punching holes, and the reclaim flag of BLOB3 is changed to 0. For BLOB4, the key of BLOB4 is 2, the value of the first key-value pair with key 2 in the reference snapshot is b, and the storage address of BLOB4 in the second storage area is also b, so BLOB4 is not reclaimed by punching holes.

[0085] In the embodiment of the present application, by setting a recycling mark in the BLOB, the first BLOBs that have not been recycled by hole punching are obtained through the second mark in the file to be recycled by hole punching, and by comparing the storage address of each first BLOB in the second storage area with the value of the key of the first BLOB in the first key-value pair in the reference snapshot, it is determined whether the first BLOB is still referenced by the reference snapshot. If it is not referenced by the reference snapshot, the first BLOB is recycled by hole punching, and the recycling mark of the first BLOB is changed to prevent repeated traversal of the BLOB that has been recycled by hole punching. The garbage data recycling by hole punching will not modify the storage address of each BLOB. Compared with the garbage data recycling by rewriting in the prior art, it reduces the consumption of I / resources and improves the performance of the system.

[0086] As an optional embodiment of the present application, the BLOB file is used as a rewritten recovery file, and a valid BLOB in the BLOB file is rewritten to obtain a new BLOB file, including:

[0087] Create a new BLOB file in the second storage area, write the valid BLOB in the to-be-rewritten recovery file into the new BLOB file, and update the storage address of the key corresponding to the valid BLOB in the LSM Tree according to the storage address of the valid BLOB in the new BLOB file;

[0088] After the storage address of the key corresponding to the valid BLOB in the LSM Tree is updated, if it is determined that all snapshots of the generated LSMTree are deleted, the BLOB file determined as the recovered file to be rewritten is deleted.

[0089] In the embodiment of the present application, after determining the recovery file to be rewritten, the valid BLOB in the recovery file to be rewritten is written into the new BLOB file. It should be understood that after the valid BLOB is written into the information file, the storage address of the valid BLOB changes. In order to enable the LSM Tree to continue to reference the BLOB, the storage address of the BLOB needs to be updated in the LSM Tree. Specifically, according to the key of the valid BLOB, the value of the first key-value pair can be changed to the storage address of the valid BLOB written into the new BLOB file, thereby ensuring the consistency of data storage.

[0090] In an embodiment of the present application, after the storage address of the key corresponding to the valid BLOB in the LSM Tree is updated, it is necessary to wait until all snapshots corresponding to the LSM Tree before the update time are deleted, and then delete the reclaimed file to be rewritten, so as to prevent the problem of deleting the invalid BLOB in the reclaimed file to be rewritten when it is still referenced by the historical snapshot, causing the historical snapshot to lose the reference.

[0091] As an optional embodiment, when rewriting a valid BLOB in a file to be rewritten, the valid BLOB may also be written into an existing BLOB file, thereby avoiding the problem of too many BLOB files in the system and making file management difficult due to newly created BLOB files.

[0092] As an optional embodiment of the present application, each BLOB includes an information header and an information body, wherein the recycling mark is recorded in the information header, and the information body records the description information of the BLOB, the description information includes the key of the BLOB and the amount of data stored in the BLOB, and the information body also records the storage data corresponding to the key of the BLOB; each BLOB is stored in at least one storage block in the BLOB file, and the information header, description information and storage data of the BLOB are stored in each storage block in sequence, wherein the information header of the BLOB is stored in the first storage block of the BLOB, and the data storage amount of each storage block in the BLOB file is a first preset value;

[0093] For each BLOB file, each first BLOB in the BLOB file is determined by:

[0094] Obtaining meta information of the BLOB file, where the meta information of the BLOB file is used to store a first preset value;

[0095] Determine the first storage block of the first BLOB of the BLOB file according to the meta information of the BLOB file;

[0096] For the first storage block of the BLOB in the determined BLOB file, if the recycling identifier recorded in the first storage block of the BLOB is the first identifier, then according to the first preset value, by traversing each storage block after the first storage block, determining the storage block with the second identifier recorded, and taking the first storage block as the first storage block of the next BLOB;

[0097] For the first storage block of the BLOB in the determined BLOB file, if the recycling identifier recorded in the first storage block of the BLOB is the second identifier, then based on the amount of data recorded in the first storage block of the BLOB and the first preset value, determine each storage block of the BLOB and the first storage block of the next BLOB of the BLOB.

[0098] In the embodiment of the present application, in the prior art, in the data file composed of the values ​​separated by the key value, the values ​​are arranged head to tail. For binary data, there are only 1 and 0 in the data. Since the principle of hole punching recovery is to fill 0 in the position of the original data, if the data storage format of the original data file is used, it is impossible to distinguish whether a group of data with all 0s is the data after hole punching recovery or the data with a value of 0. Therefore, in order to match the data recovery method of hole punching recovery, the embodiment of the present application sets a new data storage format for BLOB files.

[0099] like Figure 6 As shown, Figure 6 A schematic diagram of a data storage format of a BLOB file provided in an embodiment of the present application. For a BLOB file, the BLOB file includes multiple storage blocks, and the data volume of each storage block is a first preset value. The first preset value can be called the storage block size, which is used to store the file header data Blob FileHeader, BLOB data and meta information data meta blocks and file footer data footer. Different BLOB files can set different first preset values.

[0100] The file header data includes the format version information used by the file. Each BLOB includes an information header and an information body. The information header records the recycling mark corresponding to the BLOB. The information body includes description information and storage data. The description information records the key corresponding to the BLOB and the amount of storage data. The recycling mark, key, data amount and storage data in each BLOB are arranged according to preset rules. The meta-information data can record the necessary meta-information required for reading the file, such as the dictionary used by the compression algorithm and the first preset value. The file footer data can record the index information of the metadata segment.

[0101] Each BLOB in each BLOB file is aligned with a storage block as a unit, that is, if the BLOB is smaller than a first preset value, for example Figure 6 If the BLOB4 in the BLOB cannot fill a single storage block, the storage block is padded with zeros, and the padded part is called a fragment. If the amount of data in the BLOB exceeds a first preset threshold, for example Figure 6 BLOB2 in the example can be stored across multiple storage blocks and zero-padded at the end of the last storage block. If BLOB is an integer multiple of the first preset value, for example Figure 6 If BLOB1 is in the data block, then BLOB1 can be stored in the corresponding storage block.

[0102] In addition, the file header data is located in front of the BLOB position, so the file header data also needs to be aligned in storage blocks. The meta-information block and file footer information are at the end of the file and do not affect the query of each BLOB, so they do not need to be aligned in storage blocks.

[0103] In an embodiment of the present application, for each file to be punched and recovered, if the data stored in the file to be punched and recovered is not binary data, the recovery identifier can be set to a special character. When determining the unrecovered data in the file to be punched and recovered, the first identifier can be found by traversing each storage block in the file to be punched and recovered, and the unrecovered data in the file to be punched and recovered can be determined. However, since the data in the embodiment of the present application is binary data, the data is composed of 1 and 0, resulting in the recovery identifier can only be represented by 1 and 0. In the embodiment of the present application, the first identifier is represented by 0, and the second identifier is represented by 1. The data storage format of the new BLOB file set by the embodiment of the present application can determine each first BLOB data in sequence.

[0104] by Figure 6 For example, in the embodiment of the present application, when determining each first BLOB data of the file to be holed and recovered, the meta information of the BLOB file is first obtained, and the storage address of the first storage block for storing the first BLOB of the BLOB file is determined according to the first preset value in the meta information, such as Figure 6 As shown, the file header in the BLOB file occupies a preset number of storage blocks, 1 in the figure. According to the size of the first preset value in the meta information and the number of storage blocks occupied by the file header, the storage address of the first storage block of the first BLOB can be determined. Therefore, the embodiment of the present application can use the order of the storage blocks in the BLOB file to describe the storage address of the storage block. According to the storage address of the first storage block of the first BLOB, the corresponding storage block is found, and the recycling mark is determined in the first storage block of the first BLOB.

[0105] If the recycling mark in the first storage block of the first BLOB is 1, then the amount of storage data corresponding to the BLOB is determined according to the recycling mark, key, data volume and storage data arrangement rules in the BLOB, thereby determining the last storage block of the first BLOB. All data between the first storage block and the last storage block of the BLOB is the BLOB, and the first storage block of the next BLOB is the next storage block of the last storage block of the BLOB. Determine the recycling mark in the first storage block of the next BLOB. If the recycling mark is 0, then the storage blocks after the storage block are sequentially changed until the storage block with the recycling mark position of 1 is found. It should be understood that the storage block must be the first storage block of the first BLOB. Similarly, the first BLOB can be determined according to the recycling mark, key, data volume and storage data arrangement rules, and continue to traverse backward until each first BLOB is found.

[0106] If the recycling mark in the first storage block of the first BLOB is 0, it means that the first BLOB in the BLOB file is recycled by punching holes, then the storage blocks after the first storage block are traversed in sequence until a storage block with a recycling mark position of 1 is found. It should be understood that the storage block must be the first storage block of the first BLOB. Then, according to the recycling mark, key, data volume and arrangement rules of the stored data, the data volume of the stored data of the first BLOB can be determined, and the last storage block of the BLOB can be determined. The data corresponding to the first storage block and the last storage block of the BLOB and the storage blocks between the two are the first BLOB. And the first storage block of the next BLOB is the next storage block of the last storage block of the BLOB. Similarly, it can be determined whether the next BLOB is the first BLOB according to the next storage block. Follow the above steps until each first BLOB is found.

[0107] The above two ends respectively introduce the method of determining each first BLOB of the present application by taking the first BLOB as an unpunched BLOB and a punched BLOB as examples. It should be understood that when the first storage block of a punched BLOB1 appears, each storage block after the first storage block is traversed in sequence until a storage block with a recycling mark of 1 is found. When a non-punched BLOB2 is found, the last storage block of the BLOB can be determined according to the storage address of the first storage block of the BLOB and the data volume of the BLOB, and the first storage block of the next BLOB can be determined at the same time. Then, according to the recycling mark in the first storage block of the next BLOB, it can be determined whether the next BLOB is recycled by punching holes, and the subsequent first BLOBs can be determined in the corresponding manner of being recycled by punching holes or not being recycled by punching holes.

[0108] It should be understood that in the embodiment of the present application, there may be other BLOBs that have been recovered by punching holes between the determination of a BLOB that has been recovered by punching holes and the determination of the first BLOB after the BLOB.

[0109] As an arrangement rule among the recycling mark, key, data volume and stored data in each BLOB in the embodiment of the present application, the specific arrangement rule may be recycling mark, key data volume, key, data volume of stored data and stored data, wherein: the recycling mark is at a designated storage position in the information header, the recycling mark is determined according to the designated storage position, the key data volume, key, data volume of stored data and value are sequentially stored in the information body of the BLOB, and the key can be obtained by reading the corresponding data volume according to the key data volume, and the stored data can be obtained by reading the corresponding data volume according to the data volume of the stored data.

[0110] In the embodiment of the present application, a new data storage format of BLOB files is set to match the hole punching recovery operation, thereby realizing garbage data recovery through hole punching recovery.

[0111] As an optional embodiment of the present application, determining a first data amount of an invalid BLOB and a second data amount of a valid BLOB stored in the BLOB file includes:

[0112] Obtain a first record table, in which a third data amount of each BLOB file is recorded; for each BLOB file, the third data amount is the remaining data amount of the BLOB file after the last target operation is performed on the file, and the remaining data amount is the amount of data that is not recovered by hole punching after hole punching recovery is performed, or the data amount of the BLOB file after the BLOB file is rewritten;

[0113] Acquire the data amount of the valid BLOB in the BLOB file as the second data amount;

[0114] For each BLOB file, if the first record table includes a third data volume corresponding to the BLOB file, a difference between the third data volume of the BLOB file and the second data volume of the BLOB file is used as the data volume of the invalid BLOB in the BLOB file and as the first data volume;

[0115] For each BLOB file, if the first record table does not include the third data volume corresponding to the BLOB file, the difference between the total data volume of the BLOB file and the second data volume is used as the data volume of the invalid BLOB and as the first data volume; the total data volume of the BLOB file is the original data volume of the BLOB file;

[0116] The method further includes:

[0117] After the target operation is performed, the remaining data amount of each BLOB file in the first record table is updated to the remaining data amount of each BLOB file after the current target operation is performed on each BLOB file.

[0118] In an embodiment of the present application, for each BLOB file, the amount of data in the BLOB file that is still referenced by the LSM Tree after the LSM Tree is merged can be counted as the amount of data of the valid BLOB and the amount of data of the invalid BLOB. If the amount of data of the invalid BLOB in the BLOB file is to be counted, it is necessary to determine one by one whether each BLOB is recovered by hole punching, and whether the BLOB that is not recovered by hole punching is referenced. In order to improve the efficiency of obtaining the data amount of the invalid BLOB, a first record table is set to record some historical information of each BLOB file, so that the data amount of the invalid BLOB can be determined based on the historical information and the data amount of the valid BLOB.

[0119] Specifically, in the embodiment of the present application, the first record table records the remaining amount of work after the last target operation is performed on each BLOB file, and the remaining amount of work is used as the third data amount of the BLOB file. When the target operation is a hole punching recovery operation, the remaining amount of data is the amount of data that is not recovered by the hole punching recovery after the hole punching recovery is performed; if the target operation is a rewrite operation, the remaining amount of data is the amount of data of the rewritten BLOB file corresponding to the BLOB file.

[0120] It should be understood that for each BLOB file, the BLOB file may be a BLOB file after garbage collection is performed through a hole punching operation, or a BLOB file used to write a valid BLOB in a file to be rewritten, or a newly created BLOB file used to receive new BLOB data written by a user. When the BLOB file is a BLOB file after a hole punching operation, the first record table will record the amount of data that has not been hole punched and recovered after the BLOB hole punching operation. When the BLOB file is a BLOB file used to write a file to be rewritten, the first record table will record the remaining workload of the BLOB as the amount of data written in the BLOB file.

[0121] In an embodiment of the present application, for each BLOB file, it can be determined whether the file identifier exists in the first record table based on the file identifier of the BLOB file. If the file identifier exists, the difference between the third data amount corresponding to the file identifier and the data amount of the valid BLOB in the BLOB file is used as the data amount of the invalid BLOB, that is, the difference between the third data amount and the second data amount is used as the first data amount.

[0122] In an embodiment of the present application, for each BLOB file, if the file identifier does not exist in the first record table, it means that the BLOB file does not have the conditions for executing the target operation before, and is a BLOB file that has not executed the target operation. At this time, the difference between the total data amount of the BLOB file and the second data amount is used as the first data amount. As an optional embodiment, the total data amount of the BLOB file can be stored in the file header of the BLOB file, or it can be recorded in the first record table. The total data amount of each BLOB file includes the first data amount, the second data amount, and the data amount of the BLOB that has been punched and recovered. It should be understood that for the BLOB file that has not executed the target operation, the data amount of the BLOB that has been punched and recovered is 0.

[0123] In an embodiment of the present application, in order to maintain data consistency, after executing the target operation, the remaining data amount corresponding to each file identifier in the first record table is updated in a timely manner, so that the remaining data amount of each BLOB file is the remaining data amount of each BLOB file after each BLOB file executes the current target operation, thereby determining the first data amount of each subsequent BLOB file.

[0124] like Figure 7 As shown, Figure 7 A flow chart of another garbage data recovery method provided in an embodiment of the present application. It can be seen from the chart that after recovering garbage data through a hole punching operation and recovering garbage data through a rewrite operation, the corresponding remaining workload in the first record table will be updated.

[0125] As an optional embodiment of the present application, the BLOB file is taken as a file to be holed and recycled and added to the recycling candidate list, which also includes:

[0126] Determine the garbage data collection priority of the BLOB file according to the first data volume in the BLOB file and the total data volume of the BLOB file;

[0127] The garbage data of the files to be punched in the collection candidate list are collected through a punch hole operation, including:

[0128] Get the priority of each file to be recovered in the recovery candidate list;

[0129] The target operation is performed on each file to be holed and recovered in turn according to the priority of each file to be holed and recovered in the recovery candidate list.

[0130] In the embodiment of the present application, each of the files to be holed and recovered in the recovery candidate list has its own corresponding priority, which is used to indicate the order in which the hole-punching operation is performed on each of the files to be holed and recovered. The priority can be determined based on at least one of the first data volume, the second data volume, and the data volume of the BLOB that has been hole-punched and recovered. As an optional embodiment, for each file to be holed and recovered, the ratio of the first data volume to the total data volume of the file to be holed and recovered can be used as a hole-punching score, and the priority of the hole-punching operation of each file to be holed and recovered is determined based on the hole-punching score of each file to be holed and recovered.

[0131] In the embodiment of the present application, the total data volume of the files to be recovered by punching holes is obtained through each first data volume to be recovered, and the priority of each file to be recovered by punching holes is determined, so that the files to be recovered by punching holes with larger first data volumes are firstly recovered by punching holes, thereby releasing storage resources in time.

[0132] As an optional embodiment of the present application, the method further includes: determining a second data volume of each to-be-rewritten recycled file;

[0133] Creating at least one new BLOB file according to the second data volume of each of the to-be-rewritten and recovered files;

[0134] Write the valid BLOB in the reclaimed file to be rewritten into the new BLOB file, including:

[0135] According to the second data volume of each reclaimed file to be rewritten, the valid BLOBs of all reclaimed files to be rewritten are written into at least one new BLOB file, wherein one BLOB file stores at least one valid BLOB of one reclaimed file to be rewritten, and one valid BLOB of one reclaimed file to be rewritten is stored in one file.

[0136] In an embodiment of the present application, each newly created BLOB file is used to write at least one BLOB file. Therefore, when creating a BLOB file, it is necessary to determine the second data volume of each reclaimed file to be rewritten, so that the storage quantity of the newly created BLOB file is not less than the largest second data volume among the second data volumes in each reclaimed file to be rewritten.

[0137] In an embodiment of the present application, the valid BLOBs of multiple files to be rewritten and recovered can be written into a new BLOB file, avoiding the generation of a large number of BLOB files with small data volumes after multiple files to be rewritten and recovered are written into different BLOB files, which is inconvenient to manage and query data.

[0138] In the embodiment of the present application, each valid BLOB cannot be split, so a valid BLOB is stored in a BLOB file; each BLOB cannot be empty, so each BLOB file stores at least one valid BLOB of a reclaimed file to be rewritten.

[0139] The embodiment of the present application provides a garbage data recovery device, which is applied to a key-value separation storage system. The storage system includes a first storage area and a second storage area. The first storage area stores data in the form of a sampling log result merge tree LSMTree, and the second storage area is used to store multiple binary large object BLOB files corresponding to the LSM Tree, such as Figure 8 As shown, the garbage data recovery device may include: a merging operation module 801, a data volume determination module 802, and an execution target operation module 803, wherein:

[0140] A merge operation module 801 is used to perform a merge operation on the LSM Tree in response to satisfying a merge condition of the LSM Tree; the LSM Tree stores at least one first key-value pair, and the key and value of each first key-value pair are respectively an identifier and a storage address of a BLOB in a BLOB file;

[0141] The data volume determination module 802 is used to determine, for each BLOB file in the second storage area, a first data volume of an invalid BLOB and a second data volume of a valid BLOB stored in the BLOB file; wherein the valid BLOB is a BLOB corresponding to a first key-value pair that is not deleted or overwritten in the LSM Tree after the merge operation is performed on the LSMTree;

[0142] The target operation execution module 803 is used to execute the following target operations for each BLOB file:

[0143] If the first data volume of the BLOB file exceeds a first preset threshold, and the proportion of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is treated as a file to be recovered by punching holes and added to a recovery candidate list, so as to perform garbage data recovery on the file to be recovered by punching holes in the recovery candidate list through a punch hole operation; if the proportion of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold, the BLOB file is treated as a file to be rewritten and recovered, and the valid BLOB in the BLOB file is rewritten to obtain a new BLOB file.

[0144] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0145] As an optional embodiment of the present application, when the target operation execution module 803 is used to perform garbage data recovery on the to-be-punched recovery files in the recovery candidate list through the punchhole operation, it is specifically used to:

[0146] At a preset time interval, it is periodically determined whether there are files to be recovered by punching holes in the recovery candidate list. If there are files to be recovered by punching holes, the following operations are performed on each file to be recovered by punching holes:

[0147] Obtain the file to be recovered by hole punching, and generate a reference snapshot of the LSM Tree at the current moment corresponding to the file to be recovered by hole punching, wherein the reference snapshot is used to record each first key-value pair in the LSM Tree at the current moment;

[0148] Establishing a corresponding relationship between the reference snapshot and the file to be holed and recovered;

[0149] In the case that all snapshots generated before the reference snapshot are deleted, garbage data is recovered on the file to be hole-punched and recovered by a hole-punching operation according to the reference snapshot.

[0150] As an optional embodiment of the present application, each BLOB corresponds to a recycling mark, and the recycling mark is a first mark or a second mark. The first mark of each BLOB indicates that the BLOB has been recycled by punching a hole in the corresponding BLOB file, and the second mark of each BLOB indicates that the BLOB has not been recycled by punching a hole in the corresponding BLOB file.

[0151] When the target operation execution module is used to perform garbage data recovery on the to-be-hole-punched recovery file through a hole-punching operation according to the reference snapshot, it is specifically used to:

[0152] Determine, according to the second identifier in the file to be hole punched and recovered, each first BLOB in the file to be hole punched and recovered that has not been hole punched and recovered;

[0153] For each first BLOB, if the first key-value pair corresponding to the key of the first BLOB does not exist in the reference snapshot, or the first storage address corresponding to the key of the first BLOB in the reference snapshot is not the storage address of the first BLOB in the second storage area, the first BLOB is reclaimed through a hole punching operation, and the reclaim flag corresponding to the first BLOB is updated to the first flag.

[0154] As an optional embodiment of the present application, when the target operation execution module 803 is used to take the BLOB file as a reclaimed file to be rewritten, rewrite the valid BLOB in the BLOB file, and obtain a new BLOB file, it is specifically used to:

[0155] Create a new BLOB file in the second storage area, write the valid BLOB in the to-be-rewritten recovery file into the new BLOB file, and update the storage address of the key corresponding to the valid BLOB in the LSMTree according to the storage address of the valid BLOB in the new BLOB file;

[0156] After the storage address of the key corresponding to the valid BLOB in the LSM Tree is updated, if it is determined that all the snapshots of the generated LSMTree are deleted, the BLOB file determined as the recovered file to be rewritten is deleted.

[0157] As an optional embodiment of the present application, each BLOB includes an information header and an information body, the recycling mark is recorded in the information header, the information body records the description information of the BLOB, the description information includes the key of the BLOB and the amount of data stored in the BLOB, and the information body also records the storage data corresponding to the key of the BLOB; each BLOB is stored in at least one storage block in the BLOB file, and the information header, description information and storage data of the BLOB are stored in each storage block in sequence, wherein the information header of the BLOB is stored in the first storage block of the BLOB, and the data storage amount of each storage block in the BLOB file is a first preset value;

[0158] For each BLOB file, the target operation execution module determines each first BLOB in the BLOB file in the following manner:

[0159] Obtaining meta information of the BLOB file, where the meta information of the BLOB file is used to store the first preset value;

[0160] Determine the first storage block of the first BLOB of the BLOB file according to the meta information of the BLOB file;

[0161] For the first storage block of the BLOB in the determined BLOB file, if the recycling identifier recorded in the first storage block of the BLOB is the first identifier, then according to the first preset value, by traversing each storage block after the first storage block, determine the first storage block recorded with the second identifier, and use the first storage block as the first storage block of the next BLOB;

[0162] For the first storage block of the BLOB in the BLOB file that has been determined, if the recycling identifier recorded in the first storage block of the BLOB is the second identifier, then based on the amount of data recorded in the first storage block of the BLOB and the first preset value, determine each storage block of the BLOB and the first storage block of the next BLOB of the BLOB.

[0163] As an optional embodiment of the present application, when the data volume determination module is used to determine the first data volume of the invalid BLOB and the second data volume of the valid BLOB stored in the BLOB file, it is specifically used to:

[0164] Obtain a first record table, in which a third data amount of each BLOB file is recorded; for each BLOB file, the third data amount is the remaining data amount of the BLOB file after the last target operation is performed on the file, and the remaining data amount is the amount of data that is not recovered by hole punching after hole punching recovery is performed, or the data amount of the BLOB file after the BLOB file is rewritten;

[0165] Acquire the data amount of the valid BLOB in the BLOB file as the second data amount;

[0166] For each BLOB file, if the first record table includes a third data volume corresponding to the BLOB file, a difference between the third data volume of the BLOB file and the second data volume of the BLOB file is used as the data volume of the invalid BLOB in the BLOB file and as the first data volume;

[0167] For each BLOB file, if the first record table does not include the third data volume corresponding to the BLOB file, the difference between the total data volume of the BLOB file and the second data volume is used as the data volume of the invalid BLOB and as the first data volume; the total data volume of the BLOB file is the original data volume of the BLOB file;

[0168] The method further includes:

[0169] After the target operation is performed, the remaining data amount of each BLOB file in the first record table is updated to the remaining data amount of each BLOB file after the current target operation is performed on each BLOB file.

[0170] As an optional embodiment of the present application, before the target operation execution module is used to treat the BLOB file as a file to be holed and recycled and add it to the recycling candidate list, it is also used to:

[0171] Determining a garbage data collection priority of the BLOB file according to the second data amount in the BLOB file and the total data amount of the BLOB file;

[0172] The garbage data of the to-be-punched-reclaimed file in the recycling candidate list is recycled by the punch hole operation, including:

[0173] Obtain the priority of each file to be recovered in the recovery candidate list;

[0174] The target operation is performed on each of the files to be hole-punched and recycled in turn according to the priority of each of the files to be hole-punched and recycled in the recycling candidate list.

[0175] As an optional embodiment of the present application, the garbage data recovery device further includes a rewriting module, which is used to:

[0176] Determining a second data amount of each recovered file to be rewritten;

[0177] Creating at least one new BLOB file according to the second data volume of each of the to-be-rewritten and recovered files;

[0178] The valid BLOB in the reclaimed file to be rewritten is written into the new BLOB file, including:

[0179] According to the second data volume of each reclaimed file to be rewritten, the valid BLOBs of all reclaimed files to be rewritten are written into at least one new BLOB file, wherein one BLOB file stores at least one valid BLOB of one reclaimed file to be rewritten, and one valid BLOB of one reclaimed file to be rewritten is stored in one file.

[0180] In an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory. The processor executes the above-mentioned computer program to implement the steps of a garbage data recovery method. Compared with the related art, the following can be achieved: In the embodiment of the present application, different garbage data recovery methods are adopted for BLOB files according to different situations of the data amount of valid BLOBs and the data amount of invalid BLOBs in the BLOB files. When the data amount of the invalid BLOBs is large and the data amount of the valid BLOBs is also large, garbage data of the BLOB files is recovered by punching operations, which reduces the consumption of I / O resources and solves the problem of excessive consumption of I / O resources in related arts leading to degradation of system performance. When the data amount of the valid BLOBs is small, garbage data is recovered by sampling and rewriting the valid BLOBs, which improves the efficiency of garbage data recovery.

[0181] In an alternative embodiment, an electronic device is provided, such as Fig. 9 As shown, Fig. 9 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0182] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application I / On Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0183] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0184] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0185] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0186] Among them, electronic devices may include but are not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0187] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented. Compared with the related art, it can be achieved that: the embodiment of the present application adopts different garbage data recovery methods for BLOB files according to the different situations of the data volume of valid BLOBs and the data volume of invalid BLOBs in BLOB files. When the data volume of invalid BLOBs is large and the data volume of valid BLOBs is also large, garbage data of BLOB files is recovered by punching operations, which reduces the consumption of I / O resources, solves the problem of excessive consumption of I / O resources in related arts leading to decreased system performance, and improves the efficiency of garbage data recovery by sampling and rewriting valid BLOBs when the data volume of valid BLOBs is small.

[0188] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0189] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor. Compared with the related art, it can be achieved that: the embodiment of the present application adopts different garbage data recovery methods for BLOB files according to different situations of the data volume of valid BLOBs and the data volume of invalid BLOBs in BLOB files. When the data volume of invalid BLOBs is large and the data volume of valid BLOBs is also large, garbage data of BLOB files is recovered by punching holes, which reduces the consumption of I / O resources and solves the problem of excessive consumption of I / O resources in related art leading to decreased system performance. When the data volume of valid BLOBs is small, garbage data is recovered by sampling and rewriting valid BLOBs, which improves the efficiency of garbage data recovery.

[0190] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0191] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0192] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A garbage data recovery method, characterized in that: The method is applied to a key-value separation storage system, the storage system comprising a first storage area and a second storage area, the first storage area storing data in the form of a log structure merge tree LSM Tree, and the second storage area storing a plurality of binary large object BLOB files corresponding to the LSM Tree; the method comprising: In response to satisfying a merge condition of the LSM Tree, performing a merge operation on the LSM Tree; the LSM Tree stores at least one first key-value pair, and the key and value of each first key-value pair are respectively an identifier and a storage address of a BLOB in a BLOB file; For each BLOB file in the second storage area, determine a first data volume of an invalid BLOB and a second data volume of a valid BLOB stored in the BLOB file; wherein the valid BLOB is a BLOB corresponding to a first key-value pair that is not deleted or overwritten in the LSM Tree after a merge operation is performed on the LSM Tree; wherein each BLOB corresponds to a recycling identifier, the recycling identifier is a first identifier or a second identifier, the first identifier of each BLOB indicates that the BLOB has been recycled by punching a hole in the corresponding BLOB file, and the second identifier of each BLOB indicates that the BLOB has not been recycled by punching a hole in the corresponding BLOB file; For each BLOB file, perform the following target operations: If the first data volume of the BLOB file exceeds a first preset threshold, and the proportion of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is taken as a file to be punched and added to a recycling candidate list, so as to perform garbage data recovery on the file to be punched in the recycling candidate list through a punch hole operation; If the ratio of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed a second preset threshold, the BLOB file is used as a rewritten recovery file, and a valid BLOB in the BLOB file is rewritten to obtain a new BLOB file; The performing garbage data recovery on the to-be-punched recovery files in the recovery candidate list by punch hole operation includes: Obtain the file to be recovered by hole punching, and generate a reference snapshot of the LSM Tree at the current moment corresponding to the file to be recovered by hole punching, wherein the reference snapshot is used to record each first key-value pair in the LSM Tree at the current moment; Establishing a corresponding relationship between the reference snapshot and the file to be holed and recovered; When all snapshots generated before the reference snapshot are deleted, determining each first BLOB in the file to be holed and recovered that has not been holed and recovered according to the second identifier in the file to be holed and recovered; For each first BLOB, if the first key-value pair corresponding to the key of the first BLOB does not exist in the reference snapshot, or the first storage address corresponding to the key of the first BLOB in the reference snapshot is not the storage address of the first BLOB in the second storage area, the first BLOB is reclaimed through a hole punching operation, and the reclaim flag corresponding to the first BLOB is updated to the first flag.

2. The method according to claim 1, characterized in that: The performing garbage data recovery on the to-be-punched recovery files in the recovery candidate list by punch hole operation includes: At a preset time interval, it is periodically determined whether there are any files to be recovered by punching holes in the recovery candidate list. If there are any files to be recovered by punching holes, garbage data recovery is performed on the files to be recovered by punching holes in the recovery candidate list.

3. The method according to any one of claims 1 to 2, characterized in that: The method of taking the BLOB file as a rewritten recovery file and rewriting the valid BLOB in the BLOB file to obtain a new BLOB file includes: Create a new BLOB file in the second storage area, write the valid BLOB in the to-be-rewritten recovery file into the new BLOB file, and update the storage address of the key corresponding to the valid BLOB in the LSMTree according to the storage address of the valid BLOB in the new BLOB file; After the storage address of the key corresponding to the valid BLOB in the LSM Tree is updated, if it is determined that all snapshots of the generated LSM Tree are deleted, the BLOB file determined as the recovered file to be rewritten is deleted.

4. The method according to claim 1, characterized in that: Each BLOB includes an information header and an information body, the recycling mark is recorded in the information header, the description information of the BLOB is recorded in the information body, the description information includes the key of the BLOB and the amount of data stored in the BLOB, and the storage data corresponding to the key of the BLOB is also recorded in the information body; each BLOB is stored in at least one storage block in the BLOB file, and the information header, description information and storage data of the BLOB are stored in each storage block in sequence, wherein the information header of the BLOB is stored in the first storage block of the BLOB, and the data storage amount of each storage block in the BLOB file is a first preset value; For each BLOB file, each first BLOB in the BLOB file is determined in the following manner: Acquire meta information of the BLOB file, where the meta information of the BLOB file is used to store the first preset value; Determine, according to the meta information of the BLOB file, a first storage block of a first BLOB of the BLOB file; For the first storage block of the BLOB in the determined BLOB file, if the recycling identifier recorded in the first storage block of the BLOB is the first identifier, then according to the first preset value, by traversing each storage block after the first storage block, determine the first storage block with a second identifier recorded, and use the first storage block with the second identifier recorded as the first storage block of the next BLOB; For the first storage block of the BLOB in the determined BLOB file, if the recycling identifier recorded in the first storage block of the BLOB is the second identifier, then based on the amount of data recorded in the first storage block of the BLOB and the first preset value, determine each storage block of the BLOB and the first storage block of the next BLOB of the BLOB.

5. The method according to claim 4, characterized in that The determining of a first data volume of an invalid BLOB and a second data volume of a valid BLOB stored in the BLOB file includes: Obtain a first record table, wherein the first record table records a third data amount of each BLOB file; for each BLOB file, the third data amount is a remaining data amount of the BLOB file after the last target operation is performed on the file, and the remaining data amount is a data amount that is not recovered by hole punching after hole punching recovery is performed, or a data amount of a rewritten BLOB file corresponding to the BLOB file; Acquire the data amount of the valid BLOB in the BLOB file as the second data amount; For each BLOB file, if the first record table includes a third data volume corresponding to the BLOB file, a difference between the third data volume of the BLOB file and the second data volume of the BLOB file is used as the data volume of the invalid BLOB in the BLOB file and as the first data volume; For each BLOB file, if the first record table does not include the third data volume corresponding to the BLOB file, the difference between the total data volume of the BLOB file and the second data volume is used as the data volume of the invalid BLOB and as the first data volume; the total data volume of the BLOB file is the original data volume of the BLOB file; The method further comprises: After the target operation is performed, the remaining data amount of each BLOB file in the first record table is updated to the remaining data amount of each BLOB file after the current target operation is performed on each BLOB file.

6. The method according to claim 1, characterized in that The BLOB file is used as a file to be holed and recycled and added to a recycling candidate list, and before that, it also includes: Determining a garbage data collection priority of the BLOB file according to a second data amount in the BLOB file and a total data amount of the BLOB file; The performing garbage data recovery on the to-be-punched recovery files in the recovery candidate list by punch hole operation includes: Obtaining the priority of each to-be-punched and recycled file in the recycling candidate list; The target operation is performed on each file to be hole-punched and recycled in turn according to the priority of each file to be hole-punched and recycled in the recycling candidate list.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determining a second data amount of each recovered file to be rewritten; Creating at least one new BLOB file according to the second data volume of each of the to-be-rewritten and recovered files; The step of writing the valid BLOB in the to-be-rewritten recovery file into the new BLOB file comprises: According to the second data volume of each reclaimed file to be rewritten, the valid BLOBs of all reclaimed files to be rewritten are written into at least one new BLOB file, wherein one BLOB file stores at least one valid BLOB of one reclaimed file to be rewritten, and one valid BLOB of one reclaimed file to be rewritten is stored in one file.

8. A garbage data recovery device, characterized in that: The device is applied to a key-value separation storage system, the storage system includes a first storage area and a second storage area, the first storage area stores data in the form of a sampling log result merge tree LSM Tree, and the second storage area is used to store multiple binary large object BLOB files corresponding to the LSM Tree, the device includes: A merge operation module, configured to perform a merge operation on the LSM Tree in response to satisfying a merge condition of the LSM Tree; the LSM Tree stores at least one first key-value pair, wherein the key and the value of each first key-value pair are respectively an identifier and a storage address of a BLOB in a BLOB file; A data volume determination module is used to determine, for each BLOB file in the second storage area, a first data volume of an invalid BLOB and a second data volume of a valid BLOB stored in the BLOB file; wherein the valid BLOB is a BLOB corresponding to a first key-value pair that is not deleted or overwritten in the LSM Tree after the merging operation is performed on the LSMTree; wherein each BLOB corresponds to a recycling mark, the recycling mark is a first mark or a second mark, the first mark of each BLOB indicates that the BLOB has been recycled by punching a hole in the corresponding BLOB file, and the second mark of each BLOB indicates that the BLOB has not been recycled by punching a hole in the corresponding BLOB file; The target operation execution module is used to execute the following target operations for each BLOB file: If the first data volume of the BLOB file exceeds a first preset threshold, and the ratio of the second data volume to the total data volume of the BLOB file exceeds a second preset threshold, the BLOB file is taken as a file to be punched and recovered and added to a recovery candidate list, so as to perform garbage data recovery on the file to be punched in the recovery candidate list through a punch hole operation; if the ratio of the second data volume in the BLOB file to the total data volume of the BLOB file does not exceed the second preset threshold, the BLOB file is taken as a file to be rewritten and recovered, and a valid BLOB in the BLOB file is rewritten to obtain a new BLOB file; When the target execution operation module is used to perform garbage data recovery on the to-be-punched recovery files in the recovery candidate list through the punch hole operation, it is specifically used to: Obtain the file to be recovered by hole punching, and generate a reference snapshot of the LSM Tree at the current moment corresponding to the file to be recovered by hole punching, wherein the reference snapshot is used to record each first key-value pair in the LSM Tree at the current moment; Establishing a corresponding relationship between the reference snapshot and the file to be holed and recovered; When all snapshots generated before the reference snapshot are deleted, determining each first BLOB in the file to be holed and recovered that has not been holed and recovered according to the second identifier in the file to be holed and recovered; For each first BLOB, if the first key-value pair corresponding to the key of the first BLOB does not exist in the reference snapshot, or the first storage address corresponding to the key of the first BLOB in the reference snapshot is not the storage address of the first BLOB in the second storage area, the first BLOB is reclaimed through a hole punching operation, and the reclaim flag corresponding to the first BLOB is updated to the first flag.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images