Data deduplication and management system based on log-structured merge tree

Abstract

The application discloses a data deduplication and management system based on a log-structured merge tree, which has two data management structures, namely, a log-structured merge tree (LSMT) structure and a value file table (vFT) structure. The LSMT structure is used for saving initial data of the system, performing a deduplication operation and processing operation requests of a user. The vFT structure is used for saving non-deduplicated data which is deleted by the LSMT structure but still referenced by deduplicated data. The LSMT performs twice deduplication. Once deduplication is performed when data on a memory is persisted to a disk. Twice deduplication is performed in a compaction process of the system on multiple ordered string tables (SSTable). The application reduces data storage cost and memory overhead through twice deduplication and storing a fingerprint index in the disk. The application provides efficient query and modification functions after deduplication through a DedupVal index mode and the vFT structure. Garbage collection of the vFT can further reduce disk space occupation and read amplification of the DedupVal.

Description

Technical Field

[0001] This invention relates to the field of data management, and in particular to a data deduplication and management system based on a log structure merging tree. Background Technology

[0002] Log-Structured Merge-Tree (LSMT) is a popular index structure for persistent key-value stores. It caches all write or modification operations in memory first, and then transforms them into sequential writes on disk, making full use of disk write bandwidth and providing good write performance.

[0003] Deduplication is a data compression technique that uses fingerprinting to identify identical data. By referencing already stored identical data, it achieves the effect of storing only one copy of the same data on the disk, reducing storage costs and network traffic. Compared to local compression, it can further reduce the amount of data that needs to be stored.

[0004] In the information age, the amount of data across all industries is experiencing explosive growth. Reports indicate that by 2025, the global data volume will reach 175 ZB, with the growth rate exceeding the rate of decline in hardware costs. Key-value stores are a crucial infrastructure, serving as the underlying storage engine for databases. Therefore, how to deduplicate data within these stores to reduce hardware costs while maintaining their original low latency and high throughput characteristics after deduplication is a pressing problem for those skilled in the art.

[0005] Existing deduplication methods designed for log structure merge trees perform global deduplication on all key-value pairs and physically separate keys and values. However, global deduplication requires storing an index of all unique values ​​in memory, resulting in significant storage and indexing overhead. Furthermore, key-value separation prevents coordinated management of keys and values, leading to substantial overhead from additional garbage collection of values, and hinders efficient query, update, and delete functions. Therefore, to save hardware costs by performing deduplication within the log structure tree while maintaining the original system's low latency and high throughput, a redesign based on the characteristics of the log structure tree is necessary. Summary of the Invention

[0006] The purpose of this invention is to solve the problems in the prior art and provide a data deduplication method and a data management system after deduplication that is more suitable for log structure merging tree.

[0007] The technical solution adopted by this invention to solve its technical problem is: to provide a data deduplication and management system based on a log structure merging tree, characterized in that the system data is KV data, which includes a key and a value, with the initial key being an OrgKey and the initial value being an OrgVal. The system has two data management structures:

[0008] The log structure merge tree (LSMT) is used to store the initial data of the system, perform deduplication operations, and process user operation requests, including write requests, query requests, and modification requests. The deduplication operation refers to taking multiple data with duplicate values, keeping one data as the undeducation data, and treating the other data as the deduplication data. The undeducation data is not processed. The value of the deduplication data is replaced with a reference to the undeducation data. The number of times the undeducation data is referenced is represented by the data reference count (DedupCnt).

[0009] The value file table vFT structure is used to store ordered string tables SSTables that were deleted during the compaction process of the log structure merge tree LSMT structure but whose file reference count DedupRefCnt is not zero. The file used by the value file table vFT structure to store data is the value ordered string table vSSTable. The file reference count DedupRefCnt of the ordered string table SSTable is the sum of the data reference counts DedupCnt of all data in the ordered string table SSTable that has not been deleted.

[0010] Preferably, in the log structure merge tree (LSMT) structure, the specific implementation of the deduplication operation is as follows:

[0011] The system checks if duplicate values ​​exist in the key-value pairs. If duplicate values ​​are detected, for multiple key-value pairs with duplicate values, one key-value pair is retained as the un-deleted data, and the others are deleted data. The un-deleted data is not processed. The value of the deleted data is replaced by the OrgVal value with the deleted value DedupVal, which contains the index information of the value of the un-deleted data.

[0012] The log structure merge tree LSMT also caches the relevant indexes of fingerprint information during the deduplication process, and the fingerprint information cache is maintained using the Least Recently Used (LRU) algorithm.

[0013] Preferably, the log structure merge tree LSMT performs two deduplications:

[0014] The first deduplication is performed when persisting the data in memory to the disk. Specifically, the system performs a first deduplication on the data in the immutable memory table im-memtable, stores the deduplicated data in the disk, generates an ordered string table SSTable, and stores the relevant fingerprint information in the generated ordered string table SSTable.

[0015] Secondary deduplication is performed during the compaction operation of multiple ordered string tables (SSTables) in the system. Specifically, a portion of the ordered string tables (SSTables) are read out, and based on the relevant fingerprint information stored within them, secondary deduplication is performed on the key-value pairs (KV data) between the multiple ordered string tables (SSTables).

[0016] Preferably, when the system processes a user's query request, if the query request is a point query request, it searches for the corresponding value in the log structure merge tree LSMT based on the key sent by the user; if the value read is OrgVal, it directly returns the value; if the value read is the deduplicated value DedupVal, it reads the value of the corresponding non-deduplicated data based on the index information.

[0017] Preferably, when the system processes a user's query request, if the query request is a range query request, it searches for multiple corresponding Values ​​in the log structure merge tree (LSMT) based on the Key range sent by the user; if the Value read is an OrgVal value, it directly returns the Value; if the Value read is a deduplicated value (DedupVal), it reads the corresponding undeduplicated data Value based on the index information; if multiple DedupVals are found, they are placed in the read request queue (RQ), and multi-threaded concurrent reading of the corresponding undeduplicated KV data Values ​​is performed.

[0018] Preferably, when the system processes a user's modification request, if the value to be modified is an OrgVal value, the modification method is determined based on the number of times the data is referenced (DedupCnt); if the value to be modified is a DedupVal value, the data is modified directly, and the number of times the data referenced by the non-deleted data before the modification (DedupCnt) is decremented by 1; the modification request is completed during the compaction process.

[0019] Preferably, modification requests include update requests and deletion requests.

[0020] Preferably, the value file table vFT is combined with the file reference count DedupRefCnt and the Least Recently Used K algorithm LRU-K to cache part of the value ordered string table vSSTable in memory, thereby accelerating data retrieval.

[0021] Preferably, the value file table vFT is garbage collected when its total size exceeds a preset upper limit. Specifically, this involves selecting the N value-ordered string tables vSSTables with the highest number of references (DedupRefCnt) for garbage collection, generating a new value-ordered string table vSSTable, and deleting the old value-ordered string tables vSSTable. The total size upper limit of the value file table vFT and N are both configurable values. This invention has the following beneficial effects:

[0022] 1. By deleting duplicate data, the amount of data is reduced, thereby reducing the number of compactions and thus reducing the blocking of front-end writes. This improves write performance while reducing the hardware cost of data storage.

[0023] 2. By storing the fingerprint information required for deduplication in SSTable, the index memory overhead of data deduplication technology is reduced;

[0024] 3. By using deduplication techniques during the compaction process, the amount of data that needs to be written is reduced, thereby reducing write amplification in the system based on the log structure merging tree;

[0025] 4. Through the indexing method and vFT structure of DedupVal, efficient query and modification functions after deduplication are provided; garbage collection of vFT can further reduce disk space usage and reduce read amplification of DedupVal.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0027] Figure 1 This is a system structure diagram of an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the two data deduplication processes in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram illustrating the processing of user query requests according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram illustrating the processing of user modification requests according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the structure of vFT according to an embodiment of the present invention. Detailed Implementation

[0032] See Figure 1 The diagram shown is a system structure diagram of an embodiment of the present invention, which includes two structures: Log Structure Merging Tree (LSMT) and Value File Table (vFT), used to manage files with different reference statuses; wherein, the files SSTable managed by the Log Structure Merging Tree (LSMT) and the files vSSTable managed by the Value File Table (vFT) use the same format.

[0033] The log structure merge tree LSMT's memory includes a buffer and a cache. The buffer includes a mutable memory table (memtable) and an immutable memory table (im-memtable), which handle user write and modification requests. The cache is used to accelerate user query requests. The cache also includes an LRU FPCache used during the deduplication process to cache the mapping from the value to its FP fingerprint (i.e., the FP can be quickly obtained through the value). The LRU FPCache can accelerate the deduplication process.

[0034] The memory for the value file table (vFT) is also divided into two parts: one part stores vFT metadata, including basic information such as the number of vSSTables; the other part is the vFT cache, which is used to cache some data and speed up the user's reading of vSSTables.

[0035] The arrows on the disk section from SSTable to vSSTable indicate the conversion condition of files from the log structure merge tree LSMT to the value file table vFT: that is, the data in SSTable is no longer owned by LSMT (reflected in RefCnt=0), but is still referenced by the data after deduplication (reflected in the number of times the file is referenced: DedupRefCnt>0).

[0036] See Figure 2 The diagram shown is a schematic diagram of two data deduplication processes according to an embodiment of the present invention.

[0037] The first deletion is performed on the im-memtable. Afterwards, the indexes from the FP to the OrgKey related to the im-memtable are stored in a persistent SSTable. The system persists the im-memtable as an SSTable on disk in the background. During SSTable creation, the key-value data within the table is deleted, and the relevant fingerprint information is stored in the SSTable. The im-memtable in LSMT cannot be modified by new requests.

[0038] The second deduplication occurs during the background compaction of multiple SSTables. This process involves deduplicating all selected SSTables, integrating all FP index entries within them. If the same FP is associated with multiple OrgKeys, it indicates multiple OrgVals exist; only one copy is retained, placed in the SSTable with the largest total reference count (DedupRefCnt).

[0039] The deduplication process caches the relevant indexes of FP information in memory, and the FP cache is maintained using the LRU algorithm. The deduplication process maintains an index cache of Values ​​to FPs, reducing the number of fingerprint calculations for OrgVal and accelerating the deduplication process. Specifically, the index cache determines the evicted data items when the cache is full based on the access frequency information of the indexed items.

[0040] The deduplication process detects multiple key-value pairs with the same value. One key-value pair is retained as the non-deduplicated data, and the others are treated as the deduplicated data. The non-deduplicated data is not processed. The value of the deduplicated data is replaced by the OrgVal value with the deduplicated value DedupVal, which is the index information of the value of the non-deduplicated data.

[0041] DedupVal stores index information, which can be used to find its OrgVal value. Specifically, the structure of DedupVal is: unique fn + OrgKey. Here, unique fn represents the globally unique sequence number of each SSTable or vSSTable, and OrgKey is the key of the non-duplicate data that has the same value as its value.

[0042] See Figure 3 The diagram shown is a schematic representation of how a user query request is processed according to an embodiment of the present invention.

[0043] Specifically, when processing a point query request, the system looks up the value based on the key:

[0044] First, the cache is searched. If the value corresponding to the key is found, it is returned directly; otherwise, the search continues in the SSTable. During the SSTable search, if the key is not found, the result indicating that the key does not exist is returned; otherwise, the value corresponding to the key is read. If the value is an OrgVal, it is returned directly; if it is a DedupVal, it is parsed, and the OrgKey stored in the corresponding file is searched. Specifically, parsing involves first parsing fn. If the parsed fn does not exist, the search is performed in the vFT based on the OrgKey. Because fn in DedupVal becomes invalid after the vSSTable is deleted, it needs to be recovered using the OrgKey. To reduce the number of vSSTables that need to be searched, the search is performed in descending order of fn. If fn exists, the OrgValue corresponding to the OrgKey is read directly from the corresponding file.

[0045] Specifically, for range query requests, all values ​​corresponding to the relevant keys are retrieved from the LSMT, all DedupVals are placed into the RQ, and each DedupVal is recovered from the LSMT or vFT using a separate thread. Understandably, to fully utilize multi-core processors and the read-only nature of SSTables, multiple threads are used to recover the DedupVals within the RQ.

[0046] See Figure 4 The diagram illustrates how a user modification request is processed according to an embodiment of the present invention. The modification request includes updating the value of a key-value pair or deleting the key-value pair. Specifically, the system processes the modification request during the compaction process of LSMT. At this time, it needs to decrement the reference count of the OrgKey at its corresponding OrgVal, check whether the reference count of the OrgKey is 1, and decide whether to delete it.

[0047] Specifically, when the system processes an update request, if the request is to update OrgVal, it checks the number of times its data is referenced (DedupCnt) to determine whether to keep it in the original SSTable; if the request is to update DedupVal, it updates it directly and decrements the number of times the data referenced by it before the update (DedupCnt) that has not been deleted by 1.

[0048] Specifically, when the system processes a deletion request, it writes a key-value pair to the LSMT, where the value indicates that the key has been deleted. During compaction, if OrgVal is deleted, the system decides whether to retain the original SSTable based on the data's reference status; if DedupVal is deleted, the reference count (DedupCnt) of the data that was referenced before its deletion but has not been deleted is decremented by 1.

[0049] See Figure 5 The diagram shown illustrates the vFT structure design of an embodiment of the present invention. The SSTables in the system are initially managed by LSMT. Due to the compaction process, some SSTables are deleted by LSMT. SSTables with non-zero DedupRefCnt values ​​are stored in the vFT. To distinguish it from the files in LSMT, its management file is called vSSTable (value Sorted String Table, storing ordered string tables of the original values), and both have the same format. The valid data in the vSSTable consists of key-value pairs that were deleted or modified by LSMT but are still referenced by some DedupVal values.

[0050] For the user's query request, the vSSTable does not have locality. To speed up the request completion, vFT combines DedupRefCnt and the LRU-K algorithm to cache part of the data in memory.

[0051] The metadata in the vFT stores the total size of all vSSTables, a pre-set maximum file size limit, and the number of references for each vSSTable. The vFT determines the timing of vcompaction based on the preset size limit. During vcompaction, several vSSTables with the most references are selected for garbage collection based on DedupRefCnt, generating new vSSTables to reduce the number of vSSTables that need to be read for queries. Specifically, the vFT sorts vSSTables by reference count, selects several of the most referenced vSSTables, reads them, merges and sorts them using a method similar to compaction, deletes unreferenced key-value pairs, and generates new vSSTables by combining the reference count and key sorting. The number of vSSTables selected for garbage collection can be manually configured or determined based on the current amount of free resources and the total size of the vFT. The newly generated vSSTables use the same global fn allocation method as the files in the LSMT, and each vSSTable has a unique fn.

[0052] As can be seen, the present invention provides a data deduplication and management system based on a log structure merging tree, which is a log structure merging tree-friendly data deduplication method. It combines the characteristics and limitations of log structure merging trees and designs a data deduplication method more suitable for log structure merging trees. It can simultaneously ensure efficient data writing, querying, updating and deleting operations, reduce storage costs, and reduce write amplification. It provides a log structure merging tree-friendly and comprehensive data deduplication method and a data management method after deduplication.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data deduplication and management system based on a log structure merging tree, characterized in that, The system uses key-value (KV) data, which includes a key and a value. The initial key is an OrgKey, and the initial value is an OrgVal. The system has two data management structures: The log structure merge tree (LSMT) is used to store the initial data of the system, perform deduplication operations, and process user operation requests, including write requests, query requests, and modification requests. The deduplication operation refers to taking multiple data with duplicate values, keeping one data as the undeducation data, and treating the other data as the deduplication data. The undeducation data is not processed. The value of the deduplication data is replaced with a reference to the undeducation data. The number of times the undeducation data is referenced is represented by the data reference count (DedupCnt). The value file table vFT structure is used to store the ordered string table SSTable that was deleted during the compaction process of the log structure merge tree LSMT structure but whose file reference count DedupRefCnt is not zero. The file used by the value file table vFT structure to store the data is the value ordered string table vSSTable. The file reference count DedupRefCnt of the ordered string table SSTable is the sum of the data reference count DedupCnt of all data in the ordered string table SSTable that has not been deleted. The log structure merge tree (LSMT) performs two deduplications: The first deduplication is performed when persisting the data in memory to the disk. Specifically, the system performs a first deduplication on the data in the immutable memory table im-memtable, stores the deduplicated data in the disk, generates an ordered string table SSTable, and stores the relevant fingerprint information in the generated ordered string table SSTable. Secondary deduplication is performed during the compaction operation of multiple ordered string tables (SSTables) in the system. Specifically, a portion of the ordered string tables (SSTables) are read out, and based on the relevant fingerprint information stored within them, secondary deduplication is performed on the key-value pairs (KV data) between the multiple ordered string tables (SSTables).

2. The data deduplication and management system based on a log structure merging tree according to claim 1, characterized in that, In the log structure merge tree (LSMT) structure, the specific implementation of deduplication is as follows: The system checks if duplicate values ​​exist in the key-value pairs. If duplicate values ​​are detected, for multiple key-value pairs with duplicate values, one key-value pair is retained as the un-deleted data, and the others are deleted data. The un-deleted data is not processed. The value of the deleted data is replaced by the OrgVal value with the deleted value DedupVal, which contains the index information of the value of the un-deleted data. The log structure merge tree LSMT also caches the relevant indexes of fingerprint information during the deduplication process, and the fingerprint information cache is maintained using the Least Recently Used (LRU) algorithm.

3. The data deduplication and management system based on a log structure merging tree according to claim 2, characterized in that, When the system processes a user's query request, if the query request is a point query request, it searches for the corresponding value in the log structure merge tree LSMT based on the key sent by the user; if the value read is OrgVal, it returns the value directly; if the value read is the deduplicated value DedupVal, it reads the value of the corresponding non-deduplicated data based on the index information.

4. A data deduplication and management system based on a log structure merging tree according to claim 2, characterized in that, When the system processes a user's query request, if the query request is a range query request, it searches for multiple corresponding Values ​​in the log structure merge tree (LSMT) based on the range of Keys sent by the user. If the Value read is an OrgVal, it returns the Value directly. If the Value read is a deduplicated DedupVal, it reads the corresponding undeduplicated Data Value based on the index information. If multiple DedupVals are found, they are placed in the read request queue (RQ), and the corresponding undeduplicated Key-Value data Values ​​are read concurrently using multiple threads.

5. A data deduplication and management system based on a log structure merging tree according to claim 2, characterized in that, When the system processes a user's modification request, if the value to be modified is an OrgVal value, the modification method is determined based on the number of times the data is referenced (DedupCnt). If the value to be modified is a DedupVal value, the data is modified directly, and the number of times the data referenced by the data that was not deleted before the modification (DedupCnt) is decremented by 1. The modification request is completed during the compaction process.

6. A data deduplication and management system based on a log structure merging tree according to claim 5, characterized in that, Modification requests include update requests and delete requests.

7. A data deduplication and management system based on a log structure merging tree according to claim 1, characterized in that, The value file table vFT, combined with the file reference count DedupRefCnt and the Least Recently Used K algorithm LRU-K, caches part of the value ordered string table vSSTable in memory to speed up data retrieval.

8. A data deduplication and management system based on a log structure merging tree according to claim 1, characterized in that, The value file table vFT is garbage collected when its total size exceeds a preset limit. Specifically, the N value sorted string tables vSSTable with the most references to the file DedupRefCnt are selected for garbage collection, a new value sorted string table vSSTable is generated, and the old value sorted string table vSSTable is deleted. The total size limit of the value file table vFT and N are both configurable values.

Images