A KV storage system, electronic device, and media based on LSM-Tree

Abstract

This invention discloses a key-value storage system, electronic device, and medium based on an LSM-Tree, comprising: volatile memory for caching newly written data and retrieving newly written data; persistent memory including multiple cuckoo filters for indexing global data; the multiple cuckoo filters include several cuckoo filters, each containing a hash table with several hash buckets, each hash bucket containing four hash slots, each hash slot being 64 bytes in size, where the high 32 bytes are used to store the fingerprint of the key in the key-value pair, and the low 32 bytes are used to store the file number of the corresponding key-value pair; and a disk using a single-level file structure to persist key-value pair data and store data logs and system logs; wherein, the single-level file structure merges the files with the highest degree of overlap in the next round of merging operations based on the degree of overlap between files.

Description

Technical Field

[0001] This invention relates to the field of data storage, and more particularly to a KV storage system, electronic device, and medium based on LSM-Tree. Background Technology

[0002] LSM-Tree-based key-value (KV) storage systems play an indispensable role in numerous foundational software projects and are widely used in large-scale, highly available, and highly complex systems such as databases, cloud computing, and e-commerce. Traditional LSM-Tree-based KV storage systems employ a Log Storage Tree (LSM-Tree) architecture to organize and manage data. Through a built-in write cache in memory, KV storage can write data to disk in large volumes, achieving excellent write throughput. To accelerate data reading and access, LSM-Tree KV storage typically uses Bloom filters. The storage system can determine whether a particular KV pair is included in a specific file within a limited number of probe operations using a Bloom filter.

[0003] However, today, this multi-layered file organization architecture and Bloom filters have become potential performance bottlenecks in certain scenarios. Its main problems are as follows:

[0004] 1) The multi-level log tree structure requires the managed files to be merged and flushed to disk periodically. The write amplification caused by this process will occupy disk bandwidth and processor resources to a great extent, thus affecting the read and write experience of front-end users.

[0005] 2) During the search process, Bloom filters may encounter false positive queries due to the mathematical properties of hash indexes. These queries may incorrectly search for some cold files, resulting in significant additional disk bandwidth overhead and high-tail latency. Previous research indicates that false positive queries can take 10-20 times longer than normal queries.

[0006] Meanwhile, persistent memory offers a completely new approach to solving the aforementioned problems. Compared to volatile memory, persistent memory offers larger storage capacity at the same price and the excellent characteristic of not losing data when power is off. Compared to traditional hard disks (HDDs, SSDs), persistent memory has faster read and write speeds and byte-addressable properties, making it a promising area for optimizing key-value storage. Previous research and work have focused on using persistent memory devices as a supplement to volatile memory or as a global index storage device, without carefully considering the characteristics and features of persistent memory devices. These approaches are either only suitable for small data samples, experiencing significant performance degradation after data size increases; or they have significant shortcomings in terms of unit data storage cost or failure restart speed, failing to meet the needs of industrial production environments. Summary of the Invention

[0007] In view of this, the present invention provides a KV storage system, electronic device, and medium based on LSM-Tree.

[0008] In a first aspect, embodiments of the present invention provide a key-value storage system based on an LSM-Tree, the system comprising:

[0009] Volatile memory is used to cache newly written data and to retrieve newly written data.

[0010] Persistent memory includes multiple cuckoo filters for indexing global data; each cuckoo filter contains a hash table with several hash buckets, each hash bucket with 4 hash slots, and each hash slot is 64 bytes in size, of which the high 32 bytes are used to store the fingerprint of the key in the key-value pair, and the low 32 bytes are used to store the file number of the corresponding key-value pair;

[0011] The disk uses a single-level file structure to persist key-value pair data and store data logs and system logs; wherein, the single-level file structure merges the files with the highest degree of overlap in the next round of merge operations based on the degree of overlap between the files.

[0012] Furthermore, the single-level file structure, based on the degree of overlap between files, merges the files with the highest degree of overlap in the next round of merging operations, including:

[0013] Files with a file validity rate lower than the first threshold are designated as seed files;

[0014] Calculate the overlap rate between each file and the seed file, sort them, and calculate the sum of the overlap rates of the top K files with the highest overlap rates for each seed file.

[0015] The maximum sum of overlap rates is obtained by sorting. When the maximum sum of overlap rates is greater than the second threshold, the seed file corresponding to the maximum sum of overlap rates and the top K files with the highest overlap rates are selected as candidate files and merged in the next round of merging operations.

[0016] Furthermore, the formula for calculating the overlap rate is as follows:

[0017]

[0018] In the formula, min(x) represents the minimum key value in SSTable file x, max(x) represents the maximum key value in SSTable file x, s represents the seed file, and f represents all other files except the seed file.

[0019] Furthermore, the persistent memory also includes: FFB + tree index;

[0020] When the storage system performs a write operation, if a hash collision occurs, the storage system directly stores the key-value pair data in the FFB+ tree.

[0021] Furthermore, the cuckoo filter includes:

[0022] Set a random maximum load factor threshold for each cuckoo filter. When the load factor of the cuckoo filter reaches the maximum load factor threshold, start the expansion process.

[0023] Furthermore, when the storage system fails and restarts, it includes:

[0024] Obtain the global maximum valid file index M and the candidate files C for each merge operation, and read the file index C corresponding to each candidate file. i ;

[0025] For each key-value pair of a file whose file number is greater than the maximum valid file number M, search the global index for that key-value pair and its corresponding file number m;

[0026] If the file sequence number m is greater than the maximum valid file sequence number M, then the log file can be rolled back to the original file sequence number C as needed. i If the file sequence number m is less than the maximum valid file sequence number M, then no update is needed.

[0027] Secondly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described LSM-Tree-based KV storage system.

[0028] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described LSM-Tree-based KV storage system.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a KV storage system based on LSM-Tree, which uses multiple cuckoo filters to replace Bloom filters to index global data, uses a single-level file structure to replace a multi-level file organization architecture to persist key-value pair data, and optimizes the file merging process, thereby reducing the write amplification and long-tail read latency problems in the original storage system, while improving the random read and write performance of the storage system, and is suitable for storage scenarios with high requirements for random writes and point queries. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of a KV storage system based on LSM-Tree provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the index slots in the cuckoo index and the B+ tree index slots provided in the embodiments of the present invention;

[0033] Figure 3 This is a schematic diagram of file merging using a single-layer file structure provided in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the fault recovery process provided in an embodiment of the present invention;

[0035] Figure 5 This is a comparison diagram between the single-layer file structure provided in this embodiment of the invention and the traditional multi-layer file structure of LSM-Tree;

[0036] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0039] like Figure 1 As shown, this invention provides a KV storage system based on an LSM-Tree, the system comprising:

[0040] Volatile memory is used to cache newly written data and quickly retrieve newly written data;

[0041] Persistent memory is used as a global index for KV storage system data, including a multi-cuckoo filter to replace the original Bloom filter, used to index global data and accelerate data query;

[0042] The disk uses a single-level file structure to persist key-value pair data and stores data logs as well as system logs for system failure restarts.

[0043] Furthermore, this invention designs a persistent multi-cuckoo filter index, which contains multiple cuckoo filters. Each filter contains a hash table with several hash buckets, and each bucket has four hash slots, such as... Figure 2 As shown, each slot is 8 bytes (64 bytes) in size, with the high 32 bytes used to store the fingerprint of the key in the key-value pair, and the low 32 bytes used to store the specific file number of the corresponding key-value pair. The initial number of cuckoo filters and the initial number of buckets in the table can be adjusted according to the needs of the production environment. With a persistent global index, it is no longer necessary to use the original multi-level file organization structure as an index; therefore, a single-level file organization structure can replace the original multi-level structure.

[0044] Among them, single-level file structure, Figure 5 This diagram compares a single-level file structure with the traditional LSM-Tree multi-level file structure. Compared to the traditional LSM-Tree which uses multiple levels to store files, the single-level file structure stores all data files within a single level. The traditional LSM-Tree multi-level file structure requires merging between two file levels, resulting in a large amount of data being merged and causing significant write amplification. This invention employs a single-level file organization architecture and uses a merging algorithm based on file overlap to control the amount of data during merging and reduce write amplification.

[0045] Next, the process of merging single-level file structures will be further explained:

[0046] The storage system maintains the validity rate of each file every time data is written or updated. When the validity rate of a file is lower than a first threshold t, the file will be identified as a seed file.

[0047] The storage system will then calculate the overlap ratio (OR) between each file and the seed file, using the following formula:

[0048]

[0049] In the formula, min(x) represents the minimum key value in SSTable file x, max(x) represents the maximum key value in SSTable file x, s represents the seed file, and f represents all other files except the seed file;

[0050] After obtaining the OR values ​​of each file and the seed file, sort them from highest to lowest according to the size of the OR, obtain the K files with the highest degree of overlap, and calculate the sum of their OR values ​​(SUM). OR ;

[0051] Subsequently, the sum of the OR values ​​(SUM) calculated by comparing each seed file and its overlapping files was obtained. OR And obtain the maximum SUM OR If the value is greater than the second threshold (i.e., a sufficient number of files have significant overlap with a file containing a large amount of expired data), then the corresponding seed file and the K files with the highest overlap will be selected as candidate files for merging in the next merge operation. After the merge is complete, the global index of the merged key-value pairs will be updated in persistent memory.

[0052] by Figure 3 For example, in a certain merge, there are seed files 1, 2, ..., m. Calculate and obtain the SUM of each seed file. OR Value. Taking seed file 1 as an example, the K files with the highest overlap with it (K is set to 3 here) are file 2, file 20, and file 7, with overlaps of 0.50, 0.25, and 0.05 with seed file 1, respectively. Calculate its SUM. OR The SUM is 0.50 + 0.25 + 0.05 = 0.80. Following this logic, the SUM of each seed file can be obtained. OR Value. Then the maximum SUM is obtained. ORIf the value is 1.1 and its corresponding seed file number is 2, then file number 2 and the three files with the highest overlap will be used as candidate files for merging in the next merge operation. This merging strategy can effectively reduce the amount of data involved in word merging, reduce write amplification during the merging process, avoid inefficient merging operations, and effectively reclaim disk space.

[0053] Furthermore, when using persistent multi-cuckoo filters to trigger hash collisions, the time spent resolving the collisions and the disk bandwidth overhead are too high. Specifically, when a hash collision occurs, the index cannot rely solely on the hash value to determine whether the queried or updated key-value pair is the desired one. Therefore, actual data reading from the disk is required, consuming significant disk bandwidth resources and resulting in high latency, impacting the user experience of front-end queries and writes. To address this issue, this invention introduces an FFB+ tree index specifically designed for persistent memory to carry the key-value pairs that trigger hash collisions and store the full set of key-value pair data. Specifically, when the storage system performs a write operation, if it finds an item with the same hash value in the target bucket (i.e., a hash collision occurs), the storage system directly stores the KV data in the FFB+ tree. Since only a small portion of hot data will be updated in a large-scale concentrated manner, meaning only a small portion of the data will be stored in the B+ tree and repeatedly updated, this method does not lead to an excessively large persistent B+ tree size, thus preventing a decrease in search performance.

[0054] Furthermore, the present invention also expands capacity using a random maximum load factor technique, including: setting a random maximum load factor threshold for each cuckoo filter, and starting expansion when the load factor of the cuckoo filter reaches the maximum load factor threshold.

[0055] It should be noted that when using persistent multi-cuckoo filters to store data, there is a high probability that each filter will start its resizing operation simultaneously within a short time segment, greatly consuming CPU resources. Therefore, a technique of different maximum load factors is introduced to ensure that each cuckoo filter has a random maximum load factor threshold within a given interval. Furthermore, the average load factor of the cuckoo hash algorithm is 0.94. This invention sets a floating factor f, and the maximum load factor of each cuckoo filter will be a value Lf within the range [0.94-f, 0.94+f]. When the filter's load factor is greater than or equal to Lf, the resizing operation begins. The random maximum load factor effectively splits the filter hash table resizing operations that were originally concentrated within a short time interval, distributing these operations evenly across different time periods, thereby smoothly utilizing CPU resources and preventing the hash table resizing operation from affecting users operating in the foreground.

[0056] Furthermore, when using a multi-persistent cuckoo filter as a global index, to ensure that the content of the KV storage system requires persistent memory devices to remain consistent with the content before the restart after a device failure, when the storage system restarts after a failure, the multi-cuckoo filter and single-level file structure include:

[0057] Obtain the global maximum valid file index M and the candidate files C for each merge operation, and read the file index C corresponding to each candidate file. i ;

[0058] For each key-value pair of a file whose file number is greater than the maximum valid file number M, search the global index for that key-value pair and its corresponding file number m;

[0059] If the file sequence number m is greater than the maximum valid file sequence number M, then the log file can be rolled back to the original file sequence number C as needed. i If the file sequence number m is less than the maximum valid file sequence number M, then no update is needed.

[0060] For example, such as Figure 4 As shown, the KV storage system crashed during a merge operation. Upon restarting, it read the candidate files from the previous merge operation as C1, C2, and C3, which were 3, 5, and 9 respectively, with a maximum file index M of 15. The system then searches for the file index corresponding to each key-value pair in files 3, 5, and 9 in the persistent global index. If this index is greater than 15, for example... Figure 4 If file number k3 corresponds to 16 and file number kn corresponds to 17, then these file numbers are corrected to their original values. This fault restart algorithm ensures the correctness and consistency of storage system data by efficiently deleting invalid data items and invalid files, thereby achieving the goal of fast fault restart.

[0061] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the LSM-Tree-based KV storage system as described above. Figure 6 The diagram shown illustrates a hardware structure of any data processing-capable device within the LSM-Tree-based KV storage system provided in this embodiment of the invention, except... Figure 6 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0062] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the LSM-Tree-based KV storage system as described above. The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0063] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0064] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A key-value storage system based on an LSM-Tree, characterized in that, The system includes: Volatile memory is used to cache newly written data and to retrieve newly written data. Persistent memory includes multiple cuckoo filters for indexing global data; each cuckoo filter contains a hash table with several hash buckets, each hash bucket with 4 hash slots, and each hash slot is 64 bytes in size, where the high 32 bytes are used to store the fingerprint (hash value) of the key in the key-value pair, and the low 32 bytes are used to store the file number of the corresponding key-value pair; The disk uses a single-level file structure to persist key-value pair data and store data logs and system logs; wherein, the single-level file structure merges the files with the highest degree of overlap in the next round of merging operations based on the degree of overlap between files; In this single-level file structure, the files with the highest degree of overlap are merged in the next round of merging operations, including: Files with a file validity rate lower than the first threshold are designated as seed files; Calculate the overlap rate between each file and the seed file, sort them, and calculate the sum of the overlap rates of the top K files with the highest overlap rates for each seed file. The maximum sum of overlap rates is obtained by sorting. When the maximum sum of overlap rates is greater than the second threshold, the seed file corresponding to the maximum sum of overlap rates and the top K files with the highest overlap rates are selected as candidate files and merged in the next round of merging operations.

2. The KV storage system based on LSM-Tree according to claim 1, characterized in that, The formula for calculating the overlap rate is as follows: ； In the formula, Indicates SSTable file The value of the smallest key in the data. Indicates SSTable file The value of the largest key in the array. Indicates a seed file. This refers to files other than the seed file.

3. The LSM-Tree-based KV storage system according to claim 1, characterized in that, The persistent memory also includes: FFB+ tree index; When the storage system performs a write operation, if a hash collision occurs, the storage system directly stores the key-value pair data in the FFB+ tree.

4. The LSM-Tree-based KV storage system according to claim 1, characterized in that, The docuckoo filter includes: Set a random maximum load factor threshold for each cuckoo filter. When the load factor of the cuckoo filter reaches the maximum load factor threshold, start the expansion process.

5. The LSM-Tree-based KV storage system according to claim 1, characterized in that, When the storage system restarts due to a failure, including: Get the largest valid file number globally And candidate files for each merge operation Read the file number corresponding to each candidate file. ; For each file sequence number greater than the maximum valid file sequence number The key-value pairs of the file are searched in the global index for the key-value pairs and their corresponding file numbers. ; If file sequence number Greater than the maximum valid file number Then, the log file can be rolled back to its original file sequence number as needed. If the file sequence number Less than the maximum valid file number If so, no update is needed.

6. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the LSM-Tree-based KV storage system according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the LSM-Tree-based KV storage system as described in any one of claims 1-5.

Images