A garbage data recycling method and device, electronic equipment and readable storage medium

Abstract

The application provides a garbage data recycling method and device, electronic equipment and readable storage medium. The method comprises: separately storing layout metadata of data records based on an LSM tree structure, wherein the layout metadata comprises a data amount of the data records and an identifier of a data file to which the data records belong; in response to a garbage data recycling request, traversing the LSM tree structure, calculating a garbage ratio of each data file; and performing a garbage data recycling operation on the data file according to the garbage ratio of the data file. In this way, without paying attention to problems such as a two-dimensional directory tree structure in a storage system and variability of data file sizes, the LSM tree structure stored is directly traversed, and the garbage ratio of the corresponding data file is calculated, which is simple, efficient and accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a garbage data recycling method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] In the existing distributed storage system, the storage structure of data records is generally an off-site update structure, that is, the update of data records is to store the updated data records to a new location instead of covering the old data records. This storage structure causes a lot of garbage data due to overwrite and deletion.

[0003] In order to recycle garbage data, it is necessary to calculate the garbage ratio of data files, but the two-dimensional directory tree structure in the distributed storage system makes the calculation of the garbage ratio more complex, which affects the recycling efficiency of garbage data. SUMMARY

[0004] The problem solved by the present application is that the calculation of the garbage ratio in the distributed storage system is complex, which affects the recycling efficiency of garbage data.

[0005] To solve the above problems, the first aspect of the present application provides a garbage data recycling method, comprising:

[0006] The layout metadata of the data records are stored separately based on the LSM tree structure, and the layout metadata includes the data amount of the data records and the identifier of the data file to which the data records belong;

[0007] In response to a garbage data recycling request, the LSM tree structure is traversed, and the garbage ratio of each data file is calculated;

[0008] The garbage data recycling operation is performed on the data file according to the garbage ratio of the data file.

[0009] The second aspect of the present application provides a garbage data recycling device, comprising:

[0010] The metadata storage module is configured to store the layout metadata of the data records separately based on the LSM tree structure, and the layout metadata includes the data amount of the data records and the identifier of the data file to which the data records belong;

[0011] The garbage ratio calculation module is configured to traverse the LSM tree structure in response to a garbage data recycling request, and calculate the garbage ratio of each data file;

[0012] The data recycling module is configured to perform the garbage data recycling operation on the data file according to the garbage ratio of the data file.

[0013] The third aspect of the present application provides an electronic device, comprising a memory and a processor;

[0014] the memory, configured to store a program;

[0015] the processor, coupled to the memory, configured to execute the program, so as to:

[0016] storing layout metadata of data records based on an LSM tree structure, the layout metadata containing data volume of the data records and identification of data files to which the data records belong;

[0017] in response to a garbage data recovery request, traversing the LSM tree structure, and calculating a garbage ratio of each data file;

[0018] performing a garbage data recovery operation on the data files according to the garbage ratio of the data files.

[0019] The fourth aspect of the present application provides a computer readable storage medium, having a computer program stored thereon, the program being executed by a processor to implement the garbage data recovery method described above.

[0020] In the present application, without paying attention to the two-dimensional directory tree structure in the storage system and the variability of the data file size, etc., the LSM tree structure stored is directly traversed, and the garbage ratio of the corresponding data file is calculated, which is simple, efficient and accurate.

[0021] In the present application, by combining the dump and merge operations of the LSM tree structure, the calculation of the effective data volume on the Data File is completed in the process of these operations, which reduces the complexity of the garbage ratio calculation. After the dump / merge is completed, only the LayoutFile (layout metadata file) extension header needs to be read to know the effective data volume of the Data File (data file).

[0022] In the present application, the information required for calculating the garbage ratio is decoupled from the IO path, which does not affect the IO performance.

[0023] In the present application, no additional index structure needs to be maintained, which reduces the metadata complexity and memory occupation. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 a flowchart of the garbage data recovery method according to an embodiment of the present application;

[0025] Figure 2 a flowchart of the garbage data recovery method S100 according to an embodiment of the present application;

[0026] Figure 3A structure block diagram of a garbage data recycling device according to an embodiment of the present application;

[0027] Figure 4 A structure block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and understandable, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be accurately conveyed to those skilled in the art.

[0029] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should be understood as their common meanings by those skilled in the art to which the present application belongs.

[0030] The existing distributed storage system is generally an append-based storage system, in which the storage structure of data records is generally a heterogeneous update structure. Among them, the index structure can generally select one of two strategies to handle updates, i.e., in-place update and non-in-place (i.e., heterogeneous) update. The in-place update structure directly covers the old record to store the new update; in contrast, the heterogeneous update structure always stores the update to a new location instead of covering the old entry.

[0031] In such a storage system, overwrite and deletion can cause a lot of garbage data. The recycling of these garbage data is of great significance, which can reduce the data amplification ratio of the storage system. An important step in the garbage recycling process is to calculate the garbage ratio, and then to schedule the recycling of garbage data according to the garbage ratio. For file storage systems, due to their two-dimensional directory tree structure, the variability of file size and the diversity of supported operations, the calculation of the garbage ratio will be relatively more complex, which affects the efficiency of garbage data recycling.

[0032] Among them, the data amplification ratio of the storage system is the ratio of the total amount of data in the entire storage system to the actual amount of valid data. With the update of data in the storage system, the data amplification ratio of the system will gradually increase.

[0033] In view of the above problems, the present application provides a new garbage data recycling scheme, which can solve the problem of too complex garbage ratio calculation by separately storing the layout metadata of data records in data files in the form of an LSM tree structure.

[0034] In order to facilitate understanding, the following terms that can be used are explained:

[0035] Garbage Collection: Garbage data formed by overwrite or delete in Append-based storage system is collected.

[0036] Layout Meta Data: Layout metadata, metadata information recording data storage location in file system.

[0037] DataFile: Logical file abstraction provided by distributed file system, used to store user data of file system.

[0038] LayoutFile: Logical file abstraction provided by distributed file system, used to store file layout metadata.

[0039] LSMTree: An index structure, generally used in Append system to store metadata. In this scheme, it is used to index Layout in file storage service.

[0040] LSMTree Dump: In the LSMTree index structure of file storage service, the index information stored in memory is persisted to the first layer LayoutFile.

[0041] LSMTree Compaction: In the LSM Tree index structure of file storage service, the first layer and the second layer of LayoutFile are merged to generate a new more compact LayoutFile.

[0042] The embodiment of the application provides a garbage data recovery method, which can be executed by a garbage data recovery device. The garbage data recovery device can be integrated in a Pad, a computer, a server, a computer, a server cluster, a data center and the like. As shown in the figure, it is a flow chart of the garbage data recovery method according to an embodiment of the application; wherein the garbage data recovery method comprises: Figure 1

[0043] S100, the layout metadata of the data record is stored separately based on the LSM tree structure, and the layout metadata contains the data amount of the data record and the identification of the data file to which the data record belongs;

[0044] In the application, the metadata of the data record contains many contents, and the layout metadata is part of the information in the metadata of the data record. The layout metadata is stored separately, that is, the layout metadata of the data record is stored separately from other information in the metadata.

[0045] ​In the present application, data is stored in a distributed storage system, and each time the data is updated, the old data is modified and stored as new data in the system, and the old data is retained. In fact, data is continuously written into the storage system in the form of a stream, and when a written stream reaches a threshold, it is sealed. The sealed stream is a data file in the present application, and the data file is a unit for calculating the garbage ratio. The data to be updated is considered to be the smallest granularity of the update in the present application, and is referred to as a data record. Each data file contains update information (updated data records) of one or more data records.

[0046] It can be understood that when the written stream reaches the threshold, it is sealed as a data file, and therefore, the total amount of data in each data file is consistent. In special cases, different storage strategies or sealing strategies can be set to obtain data files with different total amounts of data. In the present application, this is not limited.

[0047] In the present application, the storage location of the data record recorded in the layout metadata can be in the form of a pointer to the data record, or in the form of a way, or in other implementable ways. After each update of the data record, the newly added updated data record generates new layout metadata. In this way, the data record before the update becomes garbage data, and the index or pointer of the layout metadata points to the updated data record.

[0048] It should be noted that after each update of the data record, the newly added updated data record generates new layout metadata, which means that the old layout metadata also becomes garbage data. However, for the entire data record, the data amount is much larger than the data amount occupied by the layout metadata, and the layout metadata is stored separately in the LSM tree structure. Therefore, when garbage data is recycled or the garbage ratio of the data file is calculated, the garbage data in the layout metadata is ignored.

[0049] S200, in response to a garbage data recycling request, traversing the LSM tree structure to calculate the garbage ratio of each data file;

[0050] It can be understood that the layout metadata stored in the LSM tree structure includes the data amount of the data record and the identification of the data file to which the data record belongs. The latter can be considered as the data file associated with the layout metadata. By traversing the LSM tree structure, all layout metadata associated with the same data file and the effective data amount of each layout metadata record can be obtained, and the total amount of effective data of the same data file can be obtained, and the garbage ratio can be calculated.

[0051] S300, performing a garbage data recycling operation on the data file according to the garbage ratio of the data file.

[0052] In the present application, after receiving the layout metadata, the layout metadata is stored in the form of LSM tree structure, wherein each layout metadata records data records, data volume of the data records and data files to which the data records belong; after receiving the garbage data recovery request, the LSM tree structure is directly traversed to obtain all layout metadata associated with each data file or data volume of all valid data records associated with each data file; the garbage ratio of each data file is calculated through the valid data volume and the total data volume of each data file; and finally, the garbage data recovery operation is performed through the garbage ratio. In this way, without paying attention to the two-dimensional directory tree structure in the storage system and the variability of the data file size and other problems, the LSM tree structure is directly traversed to calculate the garbage ratio of the corresponding data file, which is simple, efficient and accurate.

[0053] Meanwhile, the layout metadata is stored separately as an LSM tree structure, which avoids maintaining additional complexity and resource occupation and does not affect the IO path.

[0054] It should be noted that after the garbage ratio of the data file is calculated, the garbage data recovery operation can be performed by first recovering the data file with the highest garbage ratio, or the garbage data recovery operation can be performed by dividing the set of data files with the highest garbage ratio and performing the garbage data recovery operation in units of the set. The specific recovery operation and scheduling are not described herein.

[0055] It should be noted that the LSM tree structure is also an append system, which has certain similarity with the distributed storage system, but they are not exactly the same. The specific differences are not described herein.

[0056] It should be noted that in the distributed system, a server or a process running on a server is a node, and nodes are interconnected through a network; each node can have multiple logical partitions, each of which is provided with a hard disk storage location and a memory storage location, and the LSM tree structure in the present application is stored in one of the logical partitions.

[0057] In one embodiment, the layout metadata of the LSM tree structure is stored in the form of LayoutFile (layout metadata file) (preset storage logic of LayoutFile (layout metadata file)), each LayoutFile (layout metadata file) contains at least one layout metadata, and the extension header of the LayoutFile (layout metadata file) contains the identifier of the data file associated with all layout metadata of the LayoutFile (layout metadata file) and the valid data volume in the data file.

[0058] In an embodiment, the LSM tree structure is traversed by traversing the extension header of all LayoutFiles of the LSM tree structure, counting the valid data amount of each data file in the data file based on the identification of the data file recorded in the extension header of each LayoutFile, and determining the total amount of valid data of the data file.

[0059] In the present application, a LayoutFile is an abstraction of a logical file, which can be regarded as a set of layout metadata grouped according to a preset strategy. The specific form of the LayoutFile can be a key-value interval, each key-value interval containing layout metadata corresponding to the range of the key-value interval. The specific form of the LayoutFile can also be an assignment interval, each assignment interval containing layout metadata corresponding to the range of the assignment interval, wherein each layout metadata is assigned a value according to a preset assignment rule. The specific form of the LayoutFile can also be other forms, which are not limited in the present application.

[0060] When the layout metadata is stored in the form of a LayoutFile (the storage logic of the preset LayoutFile), the LSM tree structure includes multiple layers, each layer including at least one (multiple) LayoutFile, and each LayoutFile including multiple layout metadata. For each LayoutFile, the data files associated with all layout metadata of the LayoutFile are obtained, and the valid data amount of each data file is obtained.

[0061] For example, a LayoutFile includes five layout metadata, A1M, B3M, C6N, D2N, and E3N. In the layout metadata A1M, A represents a data record, the number 1 represents the current update number of the data record, and M represents the associated data file. By counting the LayoutFile, the valid data amount (A+B) of the data file M and the valid data amount (C+D+E) of the data file N are obtained. A+B represents the sum of the data amount of the data record A and the data amount of the data record B.

[0062] In this embodiment, the layout metadata in the LayoutFile is counted and recorded in the extension header of the LayoutFile, so that when the LSM tree structure is traversed, each layout metadata does not need to be traversed, but only the LayoutFile needs to be traversed and the data file identifier and the corresponding valid data amount in the extension header are read. In this way, the counting of the valid data amount of the data file is divided into two parts, the valid data amount of each LayoutFile is counted first, and then the LayoutFile is traversed to obtain the valid data amount of all data files.

[0063] This splitting method can reduce the complexity of garbage ratio calculation on the one hand, and on the other hand, the counting process of the valid data amount of the LayoutFile is split out, so that it can be completed in advance during idle time, greatly improving the efficiency of the entire garbage ratio calculation. In this way, the entire garbage ratio calculation process is beneficially distributed by splitting, thereby improving the overall calculation efficiency.

[0064] In an embodiment, the layout metadata can also be directly stored in the LSM tree structure, and the valid data amount in the layout metadata and the associated data file are directly obtained when the LSM tree structure is traversed.

[0065] In an embodiment, in the calculation of the garbage ratio of each data file, the garbage ratio of the data file is calculated according to the total data amount and the total valid data amount of the data file.

[0066] The total data amount of the data file is determined in advance, and the total valid data amount of the data file is obtained by accumulating all the valid data amounts of the data file obtained by traversal. The garbage ratio of the data file can be obtained by subtracting the total valid data amount from the total data amount and dividing the difference by the total data amount, or by subtracting the quotient of the total valid data amount and the total data amount from 1.

[0067] As shown in FIG. 1, Figure 2 In an embodiment, the LSM tree structure adopts an N-layer data structure, where N is an integer greater than 1. The S100 separately stores the layout metadata of the data records based on the LSM tree structure, including:

[0068] S101, receiving the layout metadata of at least one data record, storing in the memory for accumulation;

[0069] S102, in the case that the amount of data in the memory reaches a first preset threshold, the layout metadata in the memory is transferred to the first layer of the LSM tree structure.

[0070] S103, according to a preset merging strategy, the LSM tree structure is merged layer by layer to obtain the Nth layer.

[0071] It can be understood that for the LSM tree structure, it has level numbers L0, L1, L2, …; the level numbers have a one-to-one correspondence with the N-layer data structure described above, but not the same number of corresponding relationship. For example, taking a 3-layer LSM tree structure as an example, it has level numbers L0, L1, L2, and the LSM tree structure is divided into the first layer, the second layer, and the third layer (the last layer, the Nth layer); wherein, the level number L0 corresponds to the first layer, the level number L1 corresponds to the second layer, and the level number L2 corresponds to the third layer.

[0072] In this application, the layout metadata in the memory is transferred to the first layer of the LSM tree structure, which can be specifically: when the layout metadata file in the current memory table exceeds the threshold value (which can be set to 1MB by default), a new memory table and a log file are established, and future operations are all based on the new memory table and the log file; the following operations are performed in the background: write the old memory table to SStable (the process is to first convert it into an immutable table, and then traverse and write); discard the old memory table; delete the old memory table and the log file; add the new SStable to Level0 (the first layer of the LSM tree structure).

[0073] In this application, SStable (Sorted String Table, Sorted string table) is a file of key / value string pairs sorted by key.

[0074] In an embodiment, the first preset threshold can be 1MB by default, or can be other numerical values, which can be determined according to actual conditions or experience, or can be determined by other means, which is not limited in this application.

[0075] In an embodiment, the number of layout metadata in the memory is accumulated, and in the case that the number reaches a preset threshold, the layout metadata in the memory is transferred to the first layer of the LSM tree structure.

[0076] In an embodiment, the S103, according to a preset merging strategy, the LSM tree structure is merged layer by layer to obtain the Nth layer, including:

[0077] In a case where the N-1th layer of the LSM tree structure meets a preset condition, the layout metadata of the N-1th layer and the current layout metadata of the Nth layer are merged, and the layout metadata of the N-1th layer and the current layout metadata of the Nth layer are emptied, and the Nth layer after being emptied is used to store the merged layout metadata.

[0078] It should be noted that the Nth layer of the LSM tree structure is a full layer, and the 1st layer to the N-1th layer are incremental layers.

[0079] In the present application, if the LSM tree structure is two layers, the layout metadata of the Nth layer is directly determined from the layout metadata of the 1st layer by using the above-mentioned merging manner.

[0080] In an embodiment, the S103 comprises: merging the LSM tree structure layer by layer according to a preset merging strategy to obtain the Nth layer, comprising:

[0081] In a case where the I-1th layer of the LSM tree structure meets a preset condition, the layout metadata of the I-1th layer is merged into the Ith layer of the LSM tree structure, and the layout metadata of the I-1th layer is emptied, I being an integer from 2 to N-1.

[0082] In the present application, the layout metadata of the I-1th layer is merged into the Ith layer of the LSM tree structure, which can be merged according to a preset strategy. Different preset strategies correspond to different preset conditions.

[0083] It should be noted that the above-mentioned merging process does not include the merging process of merging into the Nth layer and merging from the memory into the 1st layer. It can be understood that the merging process can also be applied to the merging process of merging into the Nth layer.

[0084] It should be noted that the above-mentioned merging process needs to correspond to an LSM tree structure of at least three layers, and the above-mentioned merging process does not exist in a case where the LSM tree structure is two layers.

[0085] In an embodiment, the preset strategy can be a Size-Tiered strategy, which ensures that the size of each SStable is similar, and limits the number of each SStable. As shown in the above figure, each SStable is limited to N, and when each SStable reaches N, a Compact operation is triggered to merge these SStables, and the merged result is written to the next layer to become a larger SStable.

[0086] In an embodiment, the preset policy can be a Leveled policy, which adopts a layered idea, and each layer limits the size of the total file. The Leveled is divided into multiple SStables of similar size at each layer, and the SStables at the same layer are globally ordered, and a Key has at most one record at each layer, and there is no redundant record.

[0087] In an embodiment, the preset policy can be other policies in addition to the above-described policies, and the present application does not limit this.

[0088] It should be noted that, in the process of layer-by-layer merging of the LSM tree structure according to the preset merging policy, if the layout metadata is directly stored in the LSM tree structure, the merging can be directly performed according to the above-mentioned merging limiting process; if the layout metadata of the LSM tree structure is stored in the form of a LayoutFile, after the merging according to the above-mentioned merging limiting process, the merged layout metadata needs to be converted into the form of a LayoutFile (and counted afterwards) and stored. Wherein, the storage of the layout metadata in the form of a LayoutFile can also be part of the preset policy.

[0089] In an embodiment, in the LSM tree structure adopting an N-layer data structure, the ratio of the maximum data amount of all data files associated with the layout metadata of the first layer to the maximum data amount of all data files associated with the layout metadata of the Nth layer is less than a preset ratio threshold.

[0090] In the present application, the maximum data amount of all data files associated with the layout metadata of the first layer is the sum of the data amounts of all data files associated with the layout metadata stored in the first layer; similarly, the maximum data amount of all data files associated with the layout metadata of the Nth layer is the sum of the data amounts of all data files associated with the layout metadata stored in the Nth layer.

[0091] For example, if the layout metadata stored in the first layer is A1M, B3M, C6N, D2N, and E3N, wherein in the layout metadata A1M, A represents a data record, the number 1 represents the current update number of the data record, and M represents the associated data file; the layout metadata stored in the first layer is associated with all data files M and N, and the corresponding maximum data amount is the sum of the data amounts of M and N.

[0092] It should be noted that in the N-layer data structure of the LSM tree structure, the layout metadata of the same layer data structure does not overlap, but the layout metadata of different layer data structures may overlap; and the layout metadata of the first layer data structure may overlap.

[0093] For example, the layout metadata stored in the Nth layer is A1M, B3M, C6N, D2N, and E3N; the layout metadata stored in the N-1th layer is A5P and B4P; then the layout metadata A1M overlaps with A5P, and the layout metadata B3M overlaps with B4P.

[0094] In combination with the above example, if the N-layer data structure of the LSM tree structure contains two or more layout metadata of the same data record, the layout metadata of the data record overlaps; in fact, if the same data record appears two or more times, the latest layout metadata data record is regarded as valid data, and the remaining layout metadata is regarded as garbage data. In combination with the above example, the layout metadata A1M and B3M data records are actually garbage data.

[0095] In the above case, traversing the LSM tree structure will regard the overlapping layout metadata garbage data as valid data for traversal, and the characteristics of this LSM tree structure may cause the total amount of valid data of the data file after traversal to be larger.

[0096] In the above case, due to the characteristics of the LSM tree structure, the storage space of the Nth layer is the largest, and the data calculated from the layout metadata of this layer is accurate data, and the layout metadata of the remaining layers of the LSM tree structure is possible error data; therefore, by controlling the ratio of the maximum data amount of the data file associated with the layout metadata of the remaining layers to the maximum data amount of the data file associated with the layout metadata of the Nth layer, the error of the garbage ratio of the calculated data file can be controlled.

[0097] For example, for a two-layer LSM tree structure, if the data amount of the layout metadata / LayoutFile (layout metadata file) of the first layer is controlled to be less than 5% of the second layer, the error of the garbage ratio of the Data File calculated by the above formula will also be less than 5%, and generally will be less than 2%-3%.

[0098] In an embodiment, when the data amount of the first layer LayoutFile (layout metadata file) approaches the threshold value, the data amount ratio of the first layer LayoutFile (layout metadata file) can be reduced by accelerating the merging from the first layer to the second layer, so as to reduce the calculation error.

[0099] It should be noted that for the LSM tree structure of 3 layers or more than 3 layers, the error of the final garbage ratio can also be controlled by the proportion of the maximum data amount of all data files associated with the layout metadata of the first layer to the maximum data amount of all data files associated with the layout metadata of the Nth layer being less than a preset proportion threshold.

[0100] It can be understood that the storage space occupied by each layer in the LSM tree structure, such as the first layer and the second layer, is generally increased by a consistent multiple in turn. Therefore, the proportion of the maximum data amount of the data files associated with the layout metadata of the remaining layers in the LSM tree structure to the maximum data amount of the data files associated with the layout metadata of the Nth layer can be converted into the proportion of the maximum data amount of all data files associated with the layout metadata of the first layer to the maximum data amount of all data files associated with the layout metadata of the Nth layer, so as to control the upper limit of the error of the final garbage ratio.

[0101] In this application, by combining the snapshotting and merging operations of the LSM tree structure, the calculation of the effective data amount on the Data File is completed in the process of these operations, the complexity of the garbage ratio calculation is reduced, and after the snapshotting / merging is completed, only the generated LayoutFile (layout metadata file) extension header needs to be read to know the effective data amount of the Data File (data file).

[0102] In this application, the information required for calculating the garbage ratio is decoupled from the IO path, and does not affect the IO performance.

[0103] In this application, no additional index structure needs to be maintained, which reduces the metadata complexity and memory occupation.

[0104] The embodiment of the present application provides a garbage data recycling device for executing the garbage data recycling method described above.

[0105] As shown in Figure 3 the garbage data recycling device comprises:

[0106] A metadata storage module 101 is configured to store layout metadata of data records based on an LSM tree structure, wherein the layout metadata comprises a data amount of the data records and an identifier of a data file to which the data records belong.

[0107] A garbage ratio calculation module 102 is configured to traverse the LSM tree structure and calculate a garbage ratio of each data file in response to a garbage data recycling request.

[0108] A data recycling module 103 is configured to perform a garbage data recycling operation on the data file according to the garbage ratio of the data file.

[0109] In an embodiment, the layout metadata of the LSM tree structure is stored in the form of LayoutFile (layout metadata file), each LayoutFile (layout metadata file) contains at least one layout metadata, and the extension header of the LayoutFile (layout metadata file) contains the identifier of the data file associated with all layout metadata of the LayoutFile (layout metadata file) and the amount of valid data in the data file.

[0110] In an embodiment, the garbage ratio calculation module 102 is further configured to traverse the extension headers of all LayoutFile (layout metadata file) of the LSM tree structure, count the amount of valid data in each data file recorded in the extension headers of each LayoutFile (layout metadata file) based on the identifier of the data file, and determine the total amount of valid data of the data file.

[0111] In an embodiment, the garbage ratio calculation module 102 is further configured to calculate the garbage ratio of the data file according to the total amount of data and the total amount of valid data of the data file.

[0112] In an embodiment, the LSM tree structure adopts an N-layer data structure, N is an integer greater than 1; the metadata storage module 101 is further configured to:

[0113] receive the layout metadata of at least one data record and accumulate in the memory; when the amount of data in the memory reaches a first preset threshold, transfer the layout metadata in the memory to the first layer of the LSM tree structure; and perform layer-by-layer merging on the LSM tree structure according to a preset merging strategy to obtain the Nth layer.

[0114] In an embodiment, the metadata storage module 101 is further configured to:

[0115] when the (N-1)th layer of the LSM tree structure satisfies a preset condition, merge the layout metadata of the (N-1)th layer and the current layout metadata of the Nth layer, and empty the layout metadata of the (N-1)th layer and the current layout metadata of the Nth layer, and the Nth layer after being emptied is used to store the merged layout metadata.

[0116] In an embodiment, the metadata storage module 101 is further configured to:

[0117] In a case where the I-1 layer of the LSM tree structure meets a preset condition, the layout metadata of the I-1 layer is merged into the I layer of the LSM tree structure, and the layout metadata of the I-1 layer is emptied, I being an integer containing 2 to N-1.

[0118] In an implementation, in the LSM tree structure employing the N-layer data structure, a ratio of a maximum data amount of all data files associated with the layout metadata of the 1st layer to a maximum data amount of all data files associated with the layout metadata of the Nth layer is less than a preset ratio threshold.

[0119] The garbage data recycling device provided by the above embodiments of the present application and the garbage data recycling method provided by the embodiments of the present application have the same beneficial effects as the method employed, run or implemented by the application program stored therein.

[0120] The internal functions and structures of the garbage data recycling device are described above, as shown in Figure 4 The garbage data recycling device can be implemented as an electronic device, including a memory 301 and a processor 303.

[0121] The memory 301 can be configured to store programs.

[0122] In addition, the memory 301 can also be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0123] The memory 301 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0124] The processor 303 is coupled to the memory 301 and is configured to execute programs in the memory 301 for:

[0125] The layout metadata of the data records is stored separately based on the LSM tree structure, and the layout metadata contains the data amount of the data records and the identification of the data files to which the data records belong.

[0126] In response to a garbage data recycling request, the LSM tree structure is traversed, and the garbage ratio of each data file is calculated.

[0127] Performing garbage data recycling operation on the data file according to the garbage ratio of the data file.

[0128] In an embodiment, the layout metadata of the LSM tree structure is stored in the form of LayoutFile (layout metadata file) (preset storage logic of LayoutFile (layout metadata file)), each LayoutFile (layout metadata file) contains at least one layout metadata, and the extension header of the LayoutFile (layout metadata file) contains the identifier of the data file associated with all layout metadata of the LayoutFile (layout metadata file) and the amount of valid data in the data file.

[0129] In an embodiment, the processor 303 is specifically configured to:

[0130] Traverse the extension headers of all LayoutFile (layout metadata files) of the LSM tree structure, count the amount of valid data in each data file recorded in the extension headers of various LayoutFile (layout metadata files) based on the identifier of the data file, and determine the total amount of valid data of the data file.

[0131] In an embodiment, the processor 303 is specifically configured to:

[0132] According to the total amount of data and the total amount of valid data of the data file, calculate the garbage ratio of the data file.

[0133] In an embodiment, the LSM tree structure adopts an N-layer data structure, N is an integer greater than 1; the processor 303 is specifically configured to:

[0134] Receive the layout metadata of at least one data record, accumulate in the memory; when the amount of data in the memory reaches a first preset threshold, transfer the layout metadata in the memory to the first layer of the LSM tree structure; according to a preset merging strategy, merge the LSM tree structure layer by layer to obtain the Nth layer.

[0135] In an embodiment, the processor 303 is specifically configured to:

[0136] When the N-1th layer of the LSM tree structure satisfies a preset condition, merge the layout metadata of the N-1th layer and the current layout metadata of the Nth layer, and empty the layout metadata of the N-1th layer and the current layout metadata of the Nth layer, and the Nth layer after being emptied is used to store the merged layout metadata.

[0137] In an embodiment, the processor 303 is specifically configured to:

[0138] In the case that the I-1th layer of the LSM tree structure meets the preset condition, the layout metadata of the I-1th layer is merged into the Ith layer of the LSM tree structure, and the layout metadata of the I-1th layer is emptied, I is an integer containing 2 to N-1.

[0139] In an embodiment, in the LSM tree structure adopting the N-layer data structure, the ratio of the maximum data amount of all data files associated with the layout metadata of the 1st layer to the maximum data amount of all data files associated with the layout metadata of the Nth layer is less than a preset ratio threshold.

[0140] In this application, Figure 4 only some components are shown schematically, and it does not mean that the electronic device only includes Figure 4 the components shown.

[0141] The electronic device provided by the embodiment has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0142] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0143] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0144] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions noted in the flow Figure 1 The functions noted in the flow

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow Figure 1 The functions noted in the flow Figure 1 The functions noted in the flow

[0146] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0147] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer readable media.

[0148] The present application also provides a computer readable storage medium corresponding to the garbage data recycling method provided by the foregoing embodiments, and a computer program (i.e., a program product) is stored on the computer readable storage medium. When the computer program is executed by a processor, the garbage data recycling method provided by any of the foregoing embodiments is executed.

[0149] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0150] The computer-readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0151] It should be noted that in the specification provided herein, a large number of specific details are explained. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known structures and techniques are not shown in detail in order not to obscure the understanding of the present specification.

[0152] It should also be noted that the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, product or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, product or device including the element.

[0153] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A garbage data recycling method, comprising: storing layout metadata of data records separately based on an LSM tree structure, the layout metadata containing data volume of the data records and identification of data files to which the data records belong; in response to a garbage data recycling request, traversing the LSM tree structure, calculating garbage ratio of each data file; performing garbage data recycling operation on the data files according to the garbage ratio of the data files; wherein the LSM tree structure is traversed by traversing extension headers of all layout metadata files of the LSM tree structure, counting valid data volume in each data file recorded in the extension headers of the layout metadata files based on the identification of the data file, and determining total valid data volume of the data file; in the calculation of the garbage ratio of each data file, the garbage ratio of the data file is calculated according to total data volume and total valid data volume of the data file.

2. The method of claim 1, wherein, the layout metadata of the LSM tree structure is stored according to a preset storage logic of layout metadata files, each layout metadata file contains at least one layout metadata, and the extension header of the layout metadata file contains identification of the data file associated with all layout metadata of the layout metadata file and valid data volume in the data file.

3. The method of claim 1, wherein, the LSM tree structure adopts an N-layer data structure, N being an integer greater than 1; the storing of the layout metadata of data records based on the LSM tree structure comprises: receiving the layout metadata of at least one data record, storing in a memory for accumulation; in the case that data volume in the memory reaches a first preset threshold, transferring the layout metadata in the memory to the first layer of the LSM tree structure; according to a preset merging strategy, merging the LSM tree structure layer by layer to obtain the Nth layer.

4. The method of claim 3, wherein, in the case that the Nth-1 layer of the LSM tree structure meets a preset condition, merging the layout metadata of the Nth-1 layer and the current layout metadata of the Nth layer, and emptying the layout metadata of the Nth-1 layer and the current layout metadata of the Nth layer, the Nth layer after being emptied being used to store the merged layout metadata. in the case that the Ith-1 layer of the LSM tree structure meets a preset condition, merging the layout metadata of the Ith-1 layer to the Ith layer of the LSM tree structure, and emptying the layout metadata of the Ith-1 layer, I containing integers from 2 to N-1.

5. The method of claim 3, wherein, in the LSM tree structure adopting the N-layer data structure, the ratio of maximum data volume of all data files associated with the layout metadata of the first layer to maximum data volume of all data files associated with the layout metadata of the Nth layer is less than a preset ratio threshold. 7.A garbage data recycling device, comprising:

6. The method of claim 1, wherein, ​ ​ a metadata storage module, configured to separately store layout metadata of data records based on an LSM tree structure, the layout metadata including data volume of the data records and identification of data files to which the data records belong; a garbage ratio calculation module, configured to, in response to a garbage data recovery request, traverse the LSM tree structure, and calculate garbage ratio of each data file; wherein the LSM tree structure is traversed by the following steps: traversing extension headers of all layout metadata files of the LSM tree structure, counting valid data volume of each data file recorded in the data file in the extension headers of the layout metadata files based on the identification of the data file, and determining total valid data volume of the data file; and in the calculation of the garbage ratio of each data file, the garbage ratio of the data file is calculated according to the total data volume and the total valid data volume of the data file; a data recovery module, configured to perform garbage data recovery operation on the data files according to the garbage ratio of the data files.

8. An electronic device comprising: a memory and a processor; the memory, configured to store a program; the processor, coupled to the memory, configured to execute the program, so as to: separately store layout metadata of data records based on an LSM tree structure, the layout metadata including data volume of the data records and identification of data files to which the data records belong; in response to a garbage data recovery request, traverse the LSM tree structure, and calculate garbage ratio of each data file; wherein the LSM tree structure is traversed by the following steps: traversing extension headers of all layout metadata files of the LSM tree structure, counting valid data volume of each data file recorded in the data file in the extension headers of the layout metadata files based on the identification of the data file, and determining total valid data volume of the data file; and in the calculation of the garbage ratio of each data file, the garbage ratio of the data file is calculated according to the total data volume and the total valid data volume of the data file; perform garbage data recovery operation on the data files according to the garbage ratio of the data files. 9.A computer readable storage medium, having stored thereon a computer program, the program being executed by a processor to implement the method of any one of claims 1-6.

Images