LSM tree key value storage system based on nonvolatile memory and adopting self-adaptive partitioning strategy

Abstract

The invention discloses an LSM tree key value storage system based on a nonvolatile memory and adopting a self-adaptive partitioning strategy. The system comprises a storage layer and a partitioning layer, the storage layer is located on a solid state disk (SSD), the storage layer comprises layers from L1 to Ln, and each layer comprises a plurality of SSTtables; wherein the key ranges of the SSTable of each layer are not intersected; the partition layer comprises (1) a plurality of partitions spanning a DRAM (Dynamic Random Access Memory) and an NVM (Non-Volatile Memory) and used for replacing a Memtable to receive a written key value pair; (2) a B + tree located in the memory DRAM and used for indexing the partitions; according to the method, an L0 layer in a traditional LSM tree is replaced by a partition layer, data are directly persisted by adopting a pre-write log, and a flashing process is omitted to reduce write amplification; meanwhile, a self-adaptive partitioning strategy is designed, the number of partitions and the key range are dynamically adjusted, fine-grained compression is achieved, the situation that the amount of data compressed at a time is too large is avoided, and therefore write pause is reduced; and finally, by introducing a multi-thread compression strategy, data migration of the partition layer is accelerated, the migration efficiency is improved, and the write pause problem is further relieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of optimized key-value storage systems, and in particular relates to an LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy. Background Art

[0002] In modern data centers, persistent key-value storage systems are becoming increasingly important as foundational components supporting a wide range of core applications. They serve a wide range of scenarios requiring high-performance data access and processing, such as social networks, e-commerce platforms, and search engines. These applications often feature frequent writes, large data volumes, and strong real-time requirements, placing extremely high demands on storage systems' write throughput, latency control, and system stability. To cope with write-intensive workloads, mainstream key-value storage systems generally use the Logical Structured Structure (LSM) tree as their core index structure. LSM trees utilize a non-in-place update strategy to convert random write operations into sequential write operations, significantly improving write efficiency. Inserts, updates, and deletes are all performed sequentially—the system simply appends key-value pairs or special tags to the end of the file, avoiding complex in-place modifications and improving system maintainability and performance stability. To optimize read performance, LSM tree key-value storage systems typically perform periodic data compression to reorder the data.

[0003] With the continuous advancement of hardware technology, non-volatile memory (NVM) is gaining popularity. Compared to DRAM, NVM is persistent memory—data is not lost after a power outage. It has a larger storage capacity and lower energy consumption, while its bandwidth and access latency are similar to those of DRAM. Compared to traditional storage media such as solid-state drives (SSDs), NVM offers faster access speeds and lower latency, and it transcends block-level access restrictions to enable fine-grained data operations. These characteristics offer new possibilities for the design and optimization of storage systems. Reconstructing existing LSM tree key-value storage systems using NVM is expected to achieve higher throughput, more stable write performance, and lower latency.

[0004] NoveLSM (Sudarsun Kannan, Nitish Bhat, Ada Gavrilovska, Andrea Arpaci-Dusseau, and Remzi Arpaci-Dusseau. 2018. Redesigning LSMs for Nonvolatile Memory with NoveLSM. In 2018 USENIX Annual Technical Conference (USENIX ATC18). 993–1005) uses a design that alternates between DRAM and NVM memtables to receive key-value pairs. However, the large-capacity NVM memtable generates a large number of SSTables during the flush process, rapidly filling up the L0 layer. Due to the disordered data in the L0 layer, the entire L0 and L1 layers may be involved in the compaction process, resulting in a large amount of data and prolonged compaction time. When the DRAM and NVM memtables are full again, data cannot be flushed to the L0 layer in a timely manner, causing severe write stalls.

[0005] MatrixKV (Ting Yao, Yiwen Zhang, Jiguang Wan, Qiu Cui, Liu Tang, Hongjiang, Changsheng Xie, and Xubin He 2020. MatrixKV: Reducing Write Stalls and Write Amplification in LSM-tree Based KV Stores with Matrix Container in NVM. In 2020 USENIX Annual Technical Conference (USENIX ATC 20). 17–31) introduces matrix containers in non-volatile memory (NVM) to accommodate row tables and reduces write stalls through single-threaded fine-grained column compression. Key-value pairs must first be written to a write-ahead log and then inserted into a mutable memtable. When the mutable memtable is full, it is converted to an immutable memtable and serialized into a row table. This results in the overhead of a write-ahead log and the overhead of serializing the immutable memtable into a row table. At the same time, because column compression is processed by a single thread, data migration efficiency is low and write stalls are easily caused.

[0006] FlatLSM (Kewen He, Yujie An, Yijing Luo, Xiaoguang Liu, and Gang Wang. 2023. FlatLSM: Write-Optimized LSM-Tree for PM-Based KV Stores. ACM Transactions on Storage 19, 2(2023), 1–26.) uses PMTables located in non-volatile memory (NVM) to receive write requests, eliminating the overhead of pre-write logs. It also reduces write amplification by introducing key-value separation on solid-state drives (SSDs), and employs a multi-threaded mechanism for PMTable and L1 layer compression to alleviate write pauses. However, FlatLSM suffers from data redundancy between PMTables and cannot compress all data within the same key range in NVM to the L1 layer at once, resulting in low compression efficiency and a certain amount of write amplification.

[0007] RangeKV (Ling Zhan, Kai Lu, Zhilong Cheng, and Jiguang Wan. 2020. RangeKV: An efficient key-value store based on hybrid DRAM-NVM-SSD storage structure. IEEE Access 8(2020), 154518–154529.) uses a RangeTab structure to reorganize L0 data, alleviating write stalls by reducing the amount of data compressed at a time. RangeTab-0, RangeTab-1, and RangeTab-n are divided by key range, and the Immutable Memtable flushes data to the corresponding RangeTab based on the key range. When the data volume in a RangeTab reaches a threshold, it is merged with the L1 layer. However, RangeKV has the serialization overhead of a write-ahead log and the Immutable Memtable. Furthermore, because the number of RangeTabs and key ranges are fixed, heavy write pressure on a RangeTab can easily cause it to fill up, blocking writes to higher-level data layers and causing write stalls.

[0008] However, the above existing methods still fail to effectively eliminate the overhead caused by pre-write logs and flushing, resulting in write amplification problems. At the same time, the disorder of the L0 layer leads to a huge amount of compressed data between the L0 layer and the L1 layer, resulting in write pause problems; the efficiency of migrating data located in the non-volatile memory NVM to the solid-state drive SSD is low, resulting in write amplification and write pause problems. These problems make the effectiveness of improving random write performance and alleviating write amplification still limited. Therefore, how to design an efficient, low-latency, low-write amplification LSM tree key-value storage system based on non-volatile memory NVM is still one of the core issues of current research. Summary of the Invention

[0009] To overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide an LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy. The system replaces the L0 layer in the traditional LSM tree with a partition layer, adopts a pre-write log to directly persist data, omits the flushing process to reduce write amplification; at the same time, an adaptive partitioning strategy is designed to dynamically adjust the number of partitions and key range to achieve fine-grained compression, avoid excessively large amounts of data compressed in a single time, and thus reduce write pauses; finally, by introducing a multi-threaded compression strategy, data migration at the partition layer is accelerated, migration efficiency is improved, and the write pause problem is further alleviated.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] A LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy includes a storage layer and a partitioning layer;

[0012] The storage layer is located on the solid state drive SSD, and the storage layer includes L1 to L n Layers, each layer contains several SSTables; the key ranges of SSTables in the same layer do not intersect;

[0013] The partition layer includes:

[0014] (1) Multiple partitions spanning DRAM and NVM are used to replace the key-value pairs received by the Mutable Memtable. Each partition is responsible for maintaining a specific key range of key-value pairs. The key ranges of each partition do not overlap and are continuous. The key range of each partition overlaps with the key ranges of a certain number of SSTables in the L1 layer, i.e., overlapping SSTables.

[0015] (2) A B+ tree in DRAM, used to index the partitions;

[0016] When writing or reading a key-value pair, first use the B+ tree to find the partition whose key range contains the key of the key-value pair; then, write the key-value pair to the partition or start searching from the partition; if the number of overlapping SSTables in the L1 layer is greater than the preset threshold or less than the preset threshold, use the adaptive partitioning strategy to split or merge the partitions; when the number of key-value pairs written to the partition reaches the preset threshold, the partition is compressed to the L1 layer.

[0017] A multi-threaded compression strategy is adopted to compress the partition layer and the L1 layer in the storage layer.

[0018] The structure of the partition includes: Partition Node (PN), Mutable NVMTable, Meta Node (MN), Immutable NVMTable list of itself (NTI) and Immutable NVMTable list of otherpartitions (NTO); wherein, Partition Node (PN) is used to record the metadata of the partition, Mutable NVMTable is used to receive written key-value pairs, if the Mutable NVMTable is full, it will be converted into Immutable NVMTable and then added to the end of the Immutable NVMTable list of itself (NTI), when a partition is split or merged, the Immutable NVMTable list of itself (NTI) is converted into Immutable NVMTable list of otherpartitions (NTO); Meta Node (MN) is used to record key partition status information for partition recovery.

[0019] The Partition Node (PN) includes: (1) a 16-byte Start Key and a 16-byte EndKey, which together represent the key range of the partition; a 2-byte Start Key Size and a 2-byte End Key Size, which respectively represent the sizes of the start key and the end key; (2) four pointers, an 8-byte NVMTable Pointer, an 8-byte NTI Pointer, an 8-byte NTO Pointer, and an 8-byte MN Pointer; these four pointers point to the Mutable NVMTable, Immutable NVMTable list of itself (NTI), Immutable NVMTable list of other partitions (NTO), and Meta Node (MN) in the partition, respectively;

[0020] When a write or read request arrives, the B+ tree in DRAM is used to locate the partition whose key value range contains the requested key. Then, the information in the Partition Node (PN) is used to locate the Mutable NVMTable in this partition to write the key-value pair or start a query from it. The Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) are then queried.

[0021] The physical structure used by the Mutable NVMTable and Immutable NVMTable is NVMTable. The NVMTable includes a Minimum Key, a Maximum Key, a NextNVMTable Pointer, a jump table located in the memory DRAM, and an NVM Log located in the non-volatile memory NVM. Among them, the Minimum Key and Maximum Key together represent the key range of the NVMTable. The Next NVMTable Pointer links adjacent ImmutableNVMTables to form an Immutable NVMTable list. The jump table points to the key-value pairs of the NVM Log. The specific structure of the NVM Log includes a 28-byte header and multiple log entries, as follows:

[0022] The header records the metadata of the NVM Log, including: (1) a 4-byte Magic Number, which is used to indicate whether the NVM Log is in use. If the content is "USED", it means that the NVM Log is in use. If the content is "FREE", it means that it is invalid and the NVM Log is recycled; (2) an 8-byte File Size and an 8-byte Used Size. File Size and Used Size respectively indicate the capacity of the NVM Log and the size of the space used; (3) an 8-byte Next NVM Log Pointer, which is used to persist the link relationship between the NVM Logs of multiple Immutable NVMTables in the non-volatile memory NVM;

[0023] The Log Entry is used to store written key-value pairs. Each Log Entry includes a Key Size, a Key, a Value Size, and a Value. Key and Key Size represent the key and key size, respectively, while Value and ValueSize represent the value and value size, respectively. When a new key-value pair is written to a partition, the key-value pair is written in the form of an append write. The key-value pair is first added to the end of the NVM Log in the form of a Log Entry, and the Used Size field of the Header is immediately updated. Then, the corresponding index node is inserted into the skip list and the Minimum Key and Maximum Key fields are updated.

[0024] The Meta Node (MN) is used to record key partition status information required for partition recovery. During normal system operation, when the status of a partition changes, the Meta Node (MN) of the partition is modified accordingly. The structure of the Meta Node (MN) includes:

[0025] (1) A 4-byte Magic Number, used to indicate whether the Meta Node (MN) is in use. If the content is "USED", it means that the Meta Node (MN) is in use. If the content is "FREE", it means that the Meta Node (MN) is invalid and the partition layer recycles the Meta Node (MN);

[0026] (2) A 2-byte Start Key Size, a 16-byte Start Key, a 2-byte End Key Size, and a 16-byte End Key; the Start Key and End Key represent the key range of the partition, and the Start Key Size and End Key Size represent the size of the starting key and ending value, respectively;

[0027] (3) An 8-byte NVM Log Pointer, pointing to the NVM Log of the Mutable NVMTable in the partition;

[0028] (4) An 8-byte NVMLI Pointer and an 8-byte NVMLO Pointer; the NVMLI Pointer points to the NVMLog of the first Immutable NVMTable in the Immutable NVMTable list of itself (NTI) within the partition, and the NVMLO Pointer points to the NVM Log of the first Immutable NVMTable in the Immutable NVMTable list of other partitions (NTO) within the partition;

[0029] When a system failure occurs, it is recovered through the Meta Node (MN) and NVM Log located in the non-volatile memory NVM;

[0030] The Immutable NVMTable list of itself (NTI) consists of multiple Immutable NVMTables in the partition and is used to store key-value pairs within the partition key range. When the used storage space of the Mutable NVMTable reaches a preset threshold, the Mutable NVMTable is first converted to an Immutable NVMTable, and then the available compression thread compresses the Immutable NVMTable to the L1 layer. At the same time, newly written key-value pairs are continuously written to the newly allocated Mutable NVMTable at a high speed and then converted to the Immutable NVMTable. Finally, the Immutable NVMTable list of itself (NTI) is formed.

[0031] When a Mutable NVMTable is converted to an Immutable NVMTable, two situations occur: Case 1: If the Immutable NVMTable list of itself (NTI) is empty, the Immutable NVMTable will become the head of the Immutable NVMTable list of itself (NTI); therefore, the NTIPointer in the Partition Node (PN) will point to the Immutable NVMTable, and the NVMLI Pointer in the Meta Node (MN) will point to the NVM Log corresponding to the Immutable NVMTable; Case 2: If the Immutable NVMTable list of itself (NTI) is not empty, the Immutable NVMTable will be appended to the end of the Immutable NVMTable list of itself (NTI), which is achieved by updating the Next NVMTable Pointer and Next NVMLog Pointer of the current tail Immutable NVMTable; In addition, whenever a new Immutable NVMTable is formed, it is necessary to determine whether the partition needs to be split or merged. If splitting or merging is not required, a new Mutable NVMTable is allocated and the Partition is updated. The NVMTable Pointer in the Node (PN) points to the newly allocated Mutable NVMTable, and the NVM Log Pointer in the Meta Node (MN) is set to the NVM Log corresponding to the newly allocated Mutable NVMTable;

[0032] The Immutable NVMTable list of other partitions (NTO) is converted from the Immutable NVMTable list of itself (NTI) of other partitions after the other partitions are split or merged; the key range of the key-value pairs stored in the Immutable NVMTable list of other partitions (NTO) within the partition may exceed the key range of the partition;

[0033] The ImmutableNVMTable in the Immutable NVMTable list of other partitions (NTO) will be compressed to the L1 layer by the compression thread first.

[0034] When a system failure occurs, it is recovered through the Meta Node (MN) and NVMLog located in the non-volatile memory NVM. The recovery process includes:

[0035] (1) According to the NVM Log Pointer in the Meta Node (MN), scan the key-value pairs in the NVM Log and rebuild the corresponding jump table, Minimum Key and Maximum Key in the memory DRAM; this process restores the Mutable NVMTable; (2) According to the NVMLI Pointer in the Meta Node (MN) and the NextNVM Log Pointer in the NVM Log, scan the NVMLog list to rebuild the Immutable NVMTable list of itself (NTI); (3) The recovery process of the Immutable NVMTable list of other partitions (NTO) is the same as that of the Immutable NVMTable list of itself (NTI); (4) Rebuild the Partition Node (PN) based on the partition key range in the Meta Node (MN) and the restored Mutable Memtable, ImmutableNVMTable list of other partitions (NTO) and Immutable NVMTable list of itself (NTI), and finally restore the partition.

[0036] The implementation method of the adaptive partitioning strategy is as follows:

[0037] If a partition has a large number of overlapping SSTables, the partition will be split into two smaller partitions; when the overlapping SSTables between the partition and the L1 layer are small, the partition will be merged with the adjacent partition;

[0038] The process of the adaptive partitioning strategy is as follows: for any given target partition, whenever a MutableNVMTable is converted to an Immutable NVMTable and immediately added to the tail of the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of other partitions (NTO) is not empty, a new Mutable NVMTable is allocated to receive the new key-value pair; otherwise, decide whether to split or merge the partition; if splitting or merging is required, perform the corresponding operation; otherwise, create a new Mutable NVMTable to accommodate the new key-value pair.

[0039] The specific process of determining whether to split or merge the partition includes:

[0040] When the system is first started, the partition layer contains only one partition. As key-value pairs are continuously written, the number of partitions in the partition layer (denoted as n) increases. p ) gradually increases until it reaches the maximum value specified by the user (denoted as δ max ); according to n p The value of , the system operation process is divided into three stages; if 1≤n p ≤δ min , then it is in stage 1; if δ min <n p ≤δ mid , then it is in the second stage; if δ mid <n p ≤δ max , then it is in the third stage, where δ min and δ mid are two user-specified parameters;

[0041] Let n s Let βs i and βm i be the user-specified thresholds at stage i for deciding whether the target partition needs to be split or merged. Different rules are applied at each stage as follows:

[0042] Rule 1: This rule applies to the first stage when the system starts; βs 1 is usually set to a small value. s ≥βs 1, then the target partition needs to be split;

[0043] Rule 2: This rule applies in phase 2 when there is still enough non-volatile memory NVM storage space to accommodate new key-value pairs; a fixed threshold is used to decide whether to split or merge partitions; if n s≥βs 2, the partition needs to be split; otherwise, if n s ≤βm 2 and n p -1>δ min , then the partition needs to be merged, where βs 2>βm 2; constraint n p -1>δ min Make sure you are still in stage 2 after completing the partition merging process;

[0044] Rule 3: This rule applies to the third stage, in which the thresholds for splitting and merging are dynamically adjusted; according to the coverage ratio r c Decide whether to split or merge partitions, r c Defined as the ratio of the number of overlapping SSTables in the partition to the total number of SSTables in the L1 layer; if r c ≥βs 3 and n p +1≤δ max , then the partition needs to be split; otherwise, if r c ≤βm3, then the partition needs to be merged.

[0045] The partition splitting process is as follows:

[0046] For a target partition, first identify its overlapping SSTables in the L1 layer and select the largest key in the overlapping SSTable in the middle as the split key. Then create a new partition, update the key ranges of the two partitions based on the split key, and immediately adjust the B+ tree index.

[0047] Next, the original partition's Immutable NVMTable list of itself (NTI) is converted into an Immutable NVMTable list of other partitions (NTO) by updating the NTO Pointer in the PartitionNode (PN) of the two partitions.

[0048] Finally, the key range, NVMLI Pointer, and NVMLO Pointer in the two partition Meta Nodes (MN) are updated to persist the partition status according to their respective Partition Nodes (PN).

[0049] The process of partition merging is as follows:

[0050] For a target partition, first find an acceptable partition, which is defined as a partition whose Immutable NVMTable list of other partitions (NTO) is empty in the adjacent partition and whose number of overlapping SSTables is less than that of another adjacent partition. Then, expand the key range of the acceptable partition to the union of the two partition key ranges and update the B+ tree index.

[0051] Subsequently, the original partition's Immutable NVMTable list of itself (NTI) is converted to an Immutable NVMTable list of otherpartitions (NTO) by modifying the NTO Pointer in the acceptable partition's Partition Node (PN). Finally, the partition to be merged is deleted, and the Meta Node (MN) is updated according to the acceptable partition's Partition Node (PN) to persist the partition's status.

[0052] The implementation method of the multi-thread merging strategy includes:

[0053] Set up two priority queues to manage the compression priorities of Immutable NVMTable list of itself (NTI) and Immutable NVMTable list of other partitions (NTO). Each available thread selects an Immutable NVMTable list of itself (NTI) or ImmutableNVMTable list of other partitions (NTO) from the two priority queues for compression.

[0054] The two priority queues manage the compression priorities of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) as follows:

[0055] When a partition receives frequent writes, the number of Immutable NVMTables in its Immutable NVMTable list of itself (NTI) increases rapidly. When the number of Immutable NVMTables in the Immutable NVMTable list of itself (NTI) reaches the user-set threshold θ, the write speed of the partition is slowed down, and the Immutable NVMTable list of itself (NTI) is prioritized for compression. In order to prioritize the compression of the Immutable NVMTable list that was formed earlier and contains more data, a high-priority queue and a low-priority queue are designed to manage the compression priority of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue store a pair of (Immutable NVMTablelist of itself (NTI) address or Immutable NVMTable list of other partitions (NTO) address, Partition Node (PN) address), where the Immutable NVMTable list of itself (NTI) or the Immutable NVMTable list of other partitions (NTO) address is the same as the partition node (PN) address). Partitions (NTO) and Partition Node (PN) belong to the same partition;

[0056] Set the compaction priority of the Immutable NVMTable list of itself (NTI) and the ImmutableNVMTable list of other partitions (NTO) in the following way:

[0057] (1) When the partition is split or merged, the original partition's Immutable NVMTable list of itself (NTI) is converted to Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue are traversed to find the tuple containing the address of the Immutable NVMTable list of itself (NTI) and remove it. Then, a new tuple containing the address of the Immutable NVMTable list of other partitions (NTO) and the corresponding Partition Node (PN) address is created and added to the high-priority queue.

[0058] (2) When an ImmutableNVMTable is added to the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of itself (NTI) contains only one ImmutableNVMTable, a new two-tuple is created and added to the low-priority queue; otherwise, the existing two-tuple is searched, and if it is not in the high-priority queue, it is removed from the low-priority queue and added to the high-priority queue;

[0059] The available thread selects an Immutable NVMTable list of itself (NTI) or an Immutable NVMTable list of other partitions (NTO) from two priority queues for compression as follows:

[0060] Multiple threads are allocated for compression between the partition layer and the L1 layer. Each available thread sequentially traverses the high-priority queue, then the low-priority queue, and selects the appropriate Immutable NVMTable list of itself (NTI) or Immutable NVMTable list of other partitions (NTO) for compression. The rules are as follows:

[0061] (1) If the tuple stores the address of an Immutable NVMTable list of other partitions (NTO), check whether there is an SSTable in the L1 layer whose key range overlaps with the key range of the Immutable NVMTable list of other partitions (NTO) and is currently being compressed; if so, skip the tuple; otherwise, compress the Immutable NVMTable list of other partitions (NTO), and remove the tuple after compression is complete;

[0062] (2) If the tuple stores the address of an Immutable NVMTable list of itself (NTI), relative to the Immutable NVMTable list of other partitions (NTO), the Immutable NVMTable list of itself (NTI) must also meet a condition: find the corresponding Partition Node (PN) based on the tuple, and check whether the Immutable NVMTable list of other partitions (NTO) in the same partition is empty; if it is empty, compress the Immutable NVMTable list of itself (NTI); otherwise, skip the tuple.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The present invention proposes a partition structure to ensure the orderliness of the partition layer and ensure that key-value pairs with the same key are written to the same partition, thereby reducing unnecessary key-value pair rewriting during the compression process between the partition layer and the L1 layer, significantly reducing write amplification, which lays the foundation for subsequent single compression to compress all key-value pairs in the same range to the L1 layer. This solves the problem that FlatLSM cannot compress data within the same key range in the non-volatile memory NVM to the L1 layer at one time. At the same time, each partition uses a Mutable NVMTable to accept key-value pairs. The NVM Log of the Mutable NVMTable serves as the pre-write log and L0 layer in the traditional LSM tree, further avoiding the write amplification problem caused by flushing operations. This solves the pre-write log overhead of MatrixKV and RangeKV and the Immutable Memtable serialization overhead. Partition Node is designed to manage components within the partition. In order to enable the partition to recover in the event of a system crash, Meta Node (MN) is designed to persist the key data of the partition. Immutable NVMTable list of itself (NTI) is designed to cache the written key-value pairs. Design ImmutableNVMTable list of other partitions (NTO) to accept key-value pairs of other partitions.

[0065] (2) The present invention adopts an adaptive partitioning strategy. When the number of SSTables covered by the partition and the L1 layer is higher than a certain threshold or less than a certain threshold, the partition is split or merged. Three rules are proposed to make the amount of data compressed by each partition and the L1 layer relatively balanced, which realizes fine-grained compression between the partition layer and the L1 layer, effectively avoiding the problem of excessive data volume and long time consumption in a single compression, so that the non-volatile memory NVM space can be released in time and space can be made for newly written key-value pairs, thereby reducing write pauses. This solves the write pause problem caused by the large amount of compressed data in NoveLSM and the write pause problem caused by the large write pressure of RangeTab of RangeKV.

[0066] (3) The present invention introduces a multi-threaded compression strategy, sets two priority queues to manage the compression priority of the ImmutableNVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO), and introduces multi-threading to implement concurrent compression of the partition layer and the L1 layer, thereby accelerating the migration of data on the non-volatile memory NVM to the lower layer, improving the speed of releasing non-volatile memory NVM space, and further reducing write pauses, which solves the problem of low efficiency of single-threaded column compression of MatrixKV.

[0067] In summary, the present invention replaces the L0 layer on the traditional solid-state drive (SSD) with an ordered partitioning layer based on non-volatile memory (NVM), persists the written key-value pairs to different partitions according to a specific key range, ensures that operations with the same key always fall into the same partition, and enables each partition to have a pre-write log function, thereby significantly reducing write amplification. In order to alleviate the write pause caused by the compression of the L0 and L1 layers, the present invention proposes an adaptive partitioning strategy, dynamically adjusts the partition granularity according to the number of overlapping SSTables between the L1 layer and the partition layer, and combines it with a multi-threaded compression mechanism to coordinate multi-threaded parallel processing of compression tasks between the partition layer and the L1 layer through a dual-priority queue, thereby improving the speed of releasing non-volatile memory NVM space. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] FIG1( a ) is a diagram of the overall structure of PartitionKV of the present invention, FIG1( b ) is a diagram of the partition structure, and FIG1( c ) is a diagram of the NVMTable structure.

[0069] Figure 2 This is an example diagram of the PartitionKV partition splitting and merging of the present invention.

[0070] Figure 3 This is the write flow chart of PartitionKV of the present invention.

[0071] Figure 4 This is a flowchart of reading PartitionKV of the present invention.

[0072] Figure 5 This is a comparison chart of the performance of PartitionKV of the present invention and YCSB of other systems. DETAILED DESCRIPTION

[0073] The technical solution adopted by the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0074] This paper aims to provide a non-volatile memory-based LSM tree key-value storage system, PartitionKV, that uses an adaptive partitioning strategy. PartitionKV is a key-value storage system based on a three-tier hybrid storage architecture consisting of DRAM, NVM, and SSD. The overall structure is shown in Figure 1(a).

[0075] A LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy includes a storage layer and a partitioning layer;

[0076] The storage layer is located on the solid state drive SSD, and the storage layer includes L1 to L n Layers, each layer contains several SSTables; the key ranges of the SSTables in each layer do not intersect;

[0077] The partition layer mainly includes:

[0078] (1) Multiple partitions spanning DRAM and NVM are used to replace the key-value pairs received by the Mutable Memtable. Each partition is responsible for maintaining a specific key range of key-value pairs. The key ranges of each partition do not overlap and are continuous. The key range of each partition overlaps with the key ranges of a certain number of SSTables in the L1 layer, i.e., overlapping SSTables. These overlapping SSTables may participate in the merging of partitions and the L1 layer.

[0079] (2) A B+ tree in DRAM, used to index the partitions;

[0080] When writing or reading a key-value pair, first use the B+ tree to find the partition whose key range contains the key of the key-value pair; then, write the key-value pair to the partition or start searching from the partition; to ensure that the number of overlapping SSTables involved in compression is appropriate, the present invention adopts an adaptive partitioning strategy based on the number of covered SSTables to dynamically split or merge partitions. If the number of overlapping SSTables in the L1 layer is greater than a preset threshold or less than a preset threshold, the adaptive partitioning strategy is used to split or merge the partitions; when the number of key-value pairs written to the partition reaches the preset threshold, the partition is compressed to the L1 layer to release the non-volatile memory NVM space, thereby preventing the non-volatile memory NVM capacity from being exhausted and blocking subsequent key-value pair writing.

[0081] In order to speed up the migration of data from the partition layer to the L1 layer, the present invention adopts a multi-threaded compression strategy to compress the partition layer and the L1 layer in the storage layer. Multiple compression threads can concurrently compress the partition layer and the L1 layer.

[0082] As shown in Figure 1(b), the structure of the partition mainly includes: Partition Node (PN), MutableNVMTable, Meta Node (MN), Immutable NVMTable list of itself (NTI) and ImmutableNVMTable list of other partitions (NTO); among them, Partition Node (PN) is used to record the metadata of the partition, such as the key range of this partition and pointers to other components of this partition; Mutable NVMTable is used to receive written key-value pairs. If the Mutable NVMTable is full, it will be converted into Immutable NVMTable and then added to the end of Immutable NVMTable list of itself (NTI). When a partition is split or merged, ImmutableNVMTable list of itself (NTI) is converted into Immutable NVMTable list of other partitions (NTO); Meta Node (MN) is used to record key partition status information for partition recovery.

[0083] As shown in the upper part of Figure 1(b), the Partition Node (PN) includes: (1) a 16-byte StartKey and a 16-byte EndKey, which together represent the key range of the partition; a 2-byte StartKeySize and a 2-byte EndKeySize, which respectively represent the sizes of the start key and the end key; (2) four pointers, an 8-byte NVMTablePointer, an 8-byte NTIPointer, an 8-byte NTOPointer, and an 8-byte MNPointer; these four pointers point to the Mutable NVMTable, Immutable NVMTable list of itself (NTI), Immutable NVMTable list of other partitions (NTO), and Meta Node (MN) in the partition, respectively;

[0084] When a write or read request arrives, the partition whose key value range contains the requested key is first located through the B+ tree in the memory DRAM; then, the Mutable NVMTable in this partition is located using the information in the Partition Node (PN) to write the key-value pair or start the query from it, and then the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) are queried.

[0085] The physical structure adopted by the Mutable NVMTable and Immutable NVMTable is NVMTable; the NVMTable includes a Minimum Key, a Maximum Key, a NextNVMTable Pointer, a jump table located in the memory DRAM, and an NVM Log located in the non-volatile memory NVM, as shown in Figure 1(c). Among them, the Minimum Key and the Maximum Key together represent the key range of NVMTable, and the NextNVMTable Pointer links the adjacent Immutable NVMTable to eventually form an Immutable NVMTable list; the jump table points to the key-value pair of the NVM Log, as shown in the upper half of Figure 1(c). The specific structure of the NVM Log is shown in the lower half of Figure 1(c). As can be seen from the figure, the specific structure of the NVM Log includes a 28-byte Header and multiple Log Entries, as follows:

[0086] The Header records the metadata of the NVM Log, including: (1) a 4-byte Magic Number, which is used to indicate whether the NVM Log is in use. If the content is "USED", it means that the NVM Log is in use. If the content is "FREE", it means that it is invalid and the NVM Log needs to be recycled; (2) an 8-byte File Size and an 8-byte UsedSize. File Size and Used Size respectively indicate the capacity of the NVM Log and the size of the space used; (3) an 8-byte NextNVM Log Pointer, which is used to persist the link relationship between the NVM Logs of multiple ImmutableNVMTables in the non-volatile memory NVM. This design ensures that even if the PartitionKV crashes and causes the link relationship between the Immutable NVMTable list of itself (NTI) and the ImmutableNVMTable list of other partitions (NTO) maintained by the NextNVMTable Pointer to be lost, the pointer can still be used to reconstruct the Immutable NVMTable list of itself (NTI) and the ImmutableNVMTable list of other partitions (NTO). partitions(NTO).

[0087] The Log Entry is used to store written key-value pairs. Each Log Entry includes a Key Size, a Key, a Value Size, and a Value. Key and Key Size represent the key and key size, respectively, while Value and ValueSize represent the value and value size, respectively. It should be noted that when a new key-value pair is written to a partition, the key-value pair is written in the form of an append write. Specifically, the key-value pair is first added to the end of the NVM Log in the form of a Log Entry, and the Used Size field of the Header is immediately updated. When the Used Size field is updated, it indicates that the key-value pair has been successfully written to the NVM. Then, the corresponding index node is inserted into the skip list and the Minimum Key and Maximum Key fields are updated.

[0088] In order to ensure that the partition can be recovered when the Partition Node (PN) stored in the memory DRAM is lost due to a system crash, the present invention designs a Meta Node (MN) located in the non-volatile memory NVM as a persistent storage mechanism. The Meta Node (MN) is used to record key partition status information for partition recovery. During normal system operation, when the status of a partition changes, the Meta Node (MN) of the partition is modified accordingly, for example: the key range of the partition changes or a new Mutable NVMTable is created. A robust recovery mechanism is constructed by maintaining state tracking based on the non-volatile memory NVM to ensure that the partition can be recovered when the system crashes. The structure of the Meta Node (MN) is shown in the lower half of Figure 1(b), including:

[0089] (1) A 4-byte Magic Number, used to indicate whether the Meta Node (MN) is in use. If the content is "USED", it means that the Meta Node (MN) is in use. If the content is "FREE", it means that the Meta Node (MN) is invalid and the partition layer will recycle the Meta Node (MN);

[0090] (2) A 2-byte Start Key Size, a 16-byte Start Key, a 2-byte End Key Size, and a 16-byte End Key; the Start Key and End Key represent the key range of the partition, and the Start Key Size and End Key Size represent the size of the starting key and ending value, respectively;

[0091] (3) An 8-byte NVM Log Pointer, pointing to the NVM Log of the Mutable NVMTable in the partition;

[0092] (4) An 8-byte NVMLI Pointer and an 8-byte NVMLO Pointer; the NVMLI Pointer points to the NVMLog of the first Immutable NVMTable in the Immutable NVMTable list of itself (NTI) within the partition, and the NVMLO Pointer points to the NVM Log of the first Immutable NVMTable in the Immutable NVMTable list of other partitions (NTO) within the partition;

[0093] When a PartitionKV fails, the Partition Node (PN) in the partition, the skip list in the NVMTable, and the NextNVMTable Pointer are all located in DRAM. This information is lost. During recovery to the pre-failure state, the three pointers of the Meta Node (MN) and the NVM Log it points to are used for recovery. Specifically, the Mutable NVMTable, Immutable NVMTablelist of itself (NTI), and Immutable NVMTablelist of other partitions (NTO) are rebuilt using the NVM LogPointer, NVMLI Pointer, and NVMLO Pointer.

[0094] The Immutable NVMTable list of itself (NTI) is composed of multiple ImmutableNVMTables in this partition, which are used to store key-value pairs within the partition key range; when the storage space used by the Mutable NVMTable reaches the preset threshold, the Mutable NVMTable is first converted to an Immutable NVMTable, and then the available compression thread compresses the Immutable NVMTable to the L1 layer. However, because the compression of the partition and L1 layer involves a large number of I / O operations, its execution speed is relatively slow. At the same time, the newly written key-value pairs are continuously written to the newly allocated Mutable NVMTable at a higher speed, and then converted to the Immutable NVMTable; this will cause more and more ImmutableNVMTables to be accumulated and wait for compression, eventually forming the Immutable NVMTable list of itself (NTI);

[0095] When a Mutable NVMTable is converted to an Immutable NVMTable, two situations occur: Case 1: If the Immutable NVMTable list of itself (NTI) is empty, the Immutable NVMTable will become the head of the Immutable NVMTable list of itself (NTI); therefore, the NTIPointer in the Partition Node (PN) will point to the Immutable NVMTable, and the NVMLI Pointer in the Meta Node (MN) will point to the NVM Log corresponding to the Immutable NVMTable; Case 2: If the Immutable NVMTable list of itself (NTI) is not empty, the Immutable NVMTable will be appended to the end of the Immutable NVMTable list of itself (NTI), which is achieved by updating the Next NVMTable Pointer and Next NVMLog Pointer of the current tail Immutable NVMTable; In addition, whenever a new Immutable NVMTable is formed, it is necessary to determine whether the partition needs to be split or merged. If splitting or merging is not required, a new Mutable NVMTable is allocated and the Partition is updated. The NVMTable Pointer in the Node (PN) points to the newly allocated Mutable NVMTable, and the NVM Log Pointer in the Meta Node (MN) is set to the NVM Log corresponding to the newly allocated Mutable NVMTable;

[0096] The Immutable NVMTable list of other partitions (NTO) is converted from the Immutable NVMTable list of itself (NTI) of other partitions after the other partitions are split or merged; the key range of the key-value pairs stored in the Immutable NVMTable list of other partitions (NTO) within the partition may exceed the key range of the partition;

[0097] Compared with the Immutable NVMTable list of itself (NTI), the Immutable NVMTable in the Immutable NVMTablelist of other partitions (NTO) will be compressed to the L1 layer by the compression thread first.

[0098] The partition design allows for recovery in the event of a PartitionKV crash. When a PartitionKV failure occurs, recovery is performed through the Meta Node (MN) and NVM Log in the non-volatile memory (NVM). The recovery process includes:

[0099] (1) According to the NVM Log Pointer in the Meta Node (MN), scan the key-value pairs in the NVM Log and rebuild the corresponding jump table, Minimum Key and Maximum Key in the memory DRAM; this process restores the Mutable NVMTable; (2) According to the NVMLI Pointer in the Meta Node (MN) and the NextNVM Log Pointer in the NVM Log, scan the NVMLog list to rebuild the Immutable NVMTable list of itself (NTI); (3) The recovery process of the Immutable NVMTable list of other partitions (NTO) is the same as that of the Immutable NVMTable list of itself (NTI); (4) Rebuild the Partition Node (PN) based on the partition key range in the Meta Node (MN) and the restored Mutable Memtable, ImmutableNVMTable list of other partitions (NTO) and Immutable NVMTable list of itself (NTI), and finally restore the partition.

[0100] The implementation method of the adaptive partitioning strategy is as follows:

[0101] Dynamically split or merge partitions based on the number of overlapping SSTables to ensure that the appropriate amount of data participates in compaction. Specifically, if a partition has a large number of overlapping SSTables, the partition will be split into two smaller partitions; however, in practical applications, NVM has a fixed storage capacity. To ensure that there is enough free space to create a new partition when a split is required, when there are fewer overlapping SSTables between the partition and the L1 layer, the partition will be merged with the adjacent partition;

[0102] The overall process of the adaptive partitioning strategy is as follows: For any given target partition, whenever a MutableNVMTable is converted to an Immutable NVMTable and immediately added to the end of the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of other partitions (NTO) is not empty, a new Mutable NVMTable is allocated to receive the new key-value pair; otherwise, a decision is made as to whether to split or merge the partition; if a split or merge is required, the corresponding operation is performed; otherwise, a new Mutable NVMTable is created to accommodate the new key-value pair. Next, the present invention will demonstrate how these decisions are made and how partition splitting and merging are performed.

[0103] The specific process of determining whether to split or merge the partition includes:

[0104] When PartitionKV is first started, the partition layer contains only one partition. As key-value pairs are continuously written, the number of partitions in the partition layer (denoted as n) increases. p ) gradually increases until it reaches the maximum value specified by the user (denoted as δ max ); according to n p The value of PartitionKV is divided into three stages; specifically, if 1≤n p ≤δ min , then it is in stage 1; if δ min <n p ≤δ mid , then it is in the second stage; if δ mid <n p ≤δ max , then it is in the third stage, where δ min and δ mid are two user-specified parameters;

[0105] Let n s Let βs i and βm i be the user-specified thresholds at stage i for deciding whether the target partition needs to be split or merged. Different rules are applied at each stage as follows:

[0106] Rule 1: This rule applies to the first phase when PartitionKV starts. In this case, βs 1 is usually set to a small value so that the partition splits quickly. Specifically, if n s≥βs 1, the target partition needs to be split; this reduces the number of overlapping SSTables within each partition, thereby shortening compaction time and improving the system's write performance.

[0107] Rule 2: This rule applies in phase 2 when there is still enough non-volatile memory NVM storage space to accommodate new key-value pairs; a fixed threshold is used to decide whether to split or merge partitions; specifically, if n s ≥βs 2, the partition needs to be split; otherwise, if n s ≤βm 2 and n p -1>δ min , then the partition needs to be merged, where βs 2>βm2; constraint n p -1>δ min Make sure you are still in stage 2 after completing the partition merging process;

[0108] Rule 3: This rule applies to the third stage. As mentioned above, the non-volatile memory NVM has a fixed storage capacity, which means that the number of partitions in PartitionKV is limited. However, as data continues to migrate from the partition layer to the L1 layer, the number of SSTables in the L1 layer continues to grow. In order to achieve a relatively balanced distribution of overlapping SSTables between partitions, the thresholds for splitting and merging are dynamically adjusted in the third stage; specifically, according to the coverage ratio r c Decide whether to split or merge partitions, r c Defined as the ratio of the number of overlapping SSTables in the partition to the total number of SSTables in the L1 layer; if r c ≥βs3 and n p +1≤δ max , then the partition needs to be split; otherwise, if r c ≤βm 3, then the partition needs to be merged.

[0109] In short, this adaptive method ensures that the number of overlapping SSTables of all partitions does not differ too much, thereby avoiding excessive data involved in the compression of the Immutable NVMTable list of other partitions (NTO) or the Immutable NVMTable list of itself (NTI).

[0110] The partition splitting process is as follows:

[0111] For a target partition, first identify its overlapping SSTables in the L1 layer and select the largest key in the overlapping SSTable in the middle as the split key; then create a new partition, update the key ranges of the two partitions based on the split key, and immediately adjust the B+ tree index; Figure 2 As shown, assuming Figure 2 The key range of Partition 1 in is [0-100] and needs to be split. Its overlapping SSTables include SST1, SST2, SST3, SST4 and SST5, so the largest key 60 in SST3 is used as the split key. Then create a new partition Partition 1* with a key range of (60-100], and adjust the key range of Partition 1 to [0-60]. At the same time, update the index key of Partition 1 to 60, and create a new index key 100, pointing to the Partition Node (PN) of Partition 1*.

[0112] Next, the original partition's Immutable NVMTable list of itself (NTI) is converted into an Immutable NVMTable list of other partitions (NTO) by updating the NTO Pointer in the PartitionNode (PN) of the two partitions; Figure 2 As shown, the original Immutable NVMTable list of itself (NTI) of Partition 1 will be converted into the Immutable NVMTable list of other partitions (NTO) of Partition 1 and Partition 1*, and pointed to by the NTO Pointer of the Partition Node (PN) of these two partitions; at the same time, the NTI Pointer of Partition 1 is set to empty.

[0113] Finally, the key range, NVMLI Pointer, and NVMLO Pointer in the two partition Meta Nodes (MN) are updated to persist the partition status according to their respective Partition Nodes (PN).

[0114] As can be seen from the above steps, partition splitting does not redistribute the key-value pairs in the original Immutable NVMTable list of itself (NTI) to the new partition. Instead, the Immutable NVMTable list of itself (NTI) is converted into a shared Immutable NVMTable list of other partitions (NTO) for use by the two partitions after the split. This also causes the key-value pairs stored in the Immutable NVMTable list of other partitions (NTO) to exceed the key range of each partition. Subsequent key-value pairs written will be written to the corresponding partitions based on the key ranges of the two new partitions.

[0115] The process of partition merging is as follows:

[0116] For a target partition, first find an acceptable partition, which is defined as a partition whose Immutable NVMTable list of other partitions (NTO) is empty in the adjacent partition and whose number of overlapping SSTables is less than that of another adjacent partition. Then, expand the key range of the acceptable partition to the union of the two partition key ranges and update the B+ tree index.

[0117] Assumptions Figure 2 Partition 3 in the table needs to be merged. The Immutable NVMTablelist of other partitions(NTO) of its left neighbor Partition 2 is empty, covering 4 overlapping SSTables. The Immutable NVMTable list of other partitions(NTO) of its right neighbor Partition 4 is also empty, but only covers 3 overlapping SSTables. Therefore, Partition 4 is selected as an acceptable partition. Next, the key range of Partition 4 is expanded from (220-280] to (180-280], and the index key corresponding to Partition 3 is deleted.

[0118] Then, the original partition's Immutable NVMTable list of itself (NTI) is converted to an Immutable NVMTable list of otherpartitions (NTO) by modifying the NTO Pointer in the acceptable partition Partition Node (PN); Figure 2As shown, by pointing the NTOPointer in the Partition Node (PN) of Partition 4 to the Immutable NVMTable list of other partitions (NTO), the Immutable NVMTable list of itself (NTI) of the original Partition 3 is attached to the merged Partition 4 as the Immutable NVMTable list of otherpartitions (NTO).

[0119] Finally, the partition to be merged is deleted, and the MetaNode (MN) is updated according to the Partition Node (PN) of the acceptable partition to persist the partition status. From the above steps, it can be seen that partition merging only completes the transfer of key-value pairs by modifying the pointer of the acceptable partition.

[0120] It's important to note that neither partition splitting nor merging operations involve rewriting the key-value pairs within the partition; instead, they adjust the partition structure by modifying pointers. Therefore, the time overhead of these two operations is very small and almost negligible.

[0121] Once the non-volatile memory (NVM) capacity is exhausted, subsequent key-value pair writes will be blocked, resulting in severe write stalls. To more effectively alleviate write stalls, the present invention proposes a multi-threaded compression strategy that allows multiple threads to concurrently manage compression between the partition layer and the L1 layer. Specifically, the implementation method of the multi-threaded merging strategy includes:

[0122] Two priority queues are set to manage the compression priorities of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO). Each available thread can select an Immutable NVMTable list of itself (NTI) or an Immutable NVMTable list of other partitions (NTO) from the two priority queues for compression. As mentioned above, the adaptive partitioning strategy ensures that the number of overlapping SSTables involved in each compression is moderate. Since key-value pairs with the same key are written to the same partition, it can be inferred that the overlapping SSTables of a partition usually do not participate in the compression of the Immutable NVMTable list of itself (NTI) or the Immutable NVMTable list of other partitions (NTO) of other partitions. Therefore, through the coordination of the multi-threaded compression strategy and the adaptive partitioning, fine-grained and fast compression is achieved.

[0123] The two priority queues manage the compression priorities of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) as follows:

[0124] It should be noted that when a partition receives frequent writes, the number of Immutable NVMTables in its Immutable NVMTable list of itself (NTI) will increase rapidly, causing compression to slow down and write pauses. To alleviate this problem, when the number of Immutable NVMTables in the Immutable NVMTable list of itself (NTI) reaches a user-set threshold θ, the write speed of the partition is slowed down, and the Immutable NVMTable list of itself (NTI) is prioritized for compression. In order to prioritize the compression of the Immutable NVMTablelist that was formed earlier and contains more data, a high-priority queue and a low-priority queue are designed to manage the compression priority of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue store a pair of (Immutable NVMTable list of itself (NTI) address or Immutable NVMTable list of other partitions (NTO) address, Partition Node (PN) address), where the Immutable NVMTable list of itself (NTI) or Immutable NVMTablelist of other partitions (NTO) and the Partition Node (PN) belong to the same partition.

[0125] Set the compaction priority of the Immutable NVMTable list of itself (NTI) and the ImmutableNVMTable list of other partitions (NTO) in the following way:

[0126] (1) When a partition is split or merged, the original partition's Immutable NVMTable list of itself (NTI) is converted into an Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue are traversed to find the tuple containing the address of the Immutable NVMTable list of itself (NTI) and remove it. Then, a new tuple containing the address of the Immutable NVMTable list of other partitions (NTO) and the corresponding Partition Node (PN) address is created and added to the high-priority queue. This is because in the design, a partition can only be split or merged when the Immutable NVMTable list of other partitions (NTO) is empty. Therefore, all Immutable NVMTable lists of other partitions (NTO) should be compressed into the L1 layer first.

[0127] (2) When an ImmutableNVMTable is added to the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of itself (NTI) contains only one ImmutableNVMTable, a new two-tuple is created and added to the low-priority queue; otherwise, the existing two-tuple is searched, and if it is not in the high-priority queue, it is removed from the low-priority queue and added to the high-priority queue;

[0128] The available thread can select an Immutable NVMTable list of itself (NTI) or an Immutable NVMTable list of other partitions (NTO) from two priority queues for compression as follows:

[0129] Multiple threads are allocated for compression between the partition layer and the L1 layer. Each available thread sequentially traverses the high-priority queue, then the low-priority queue, and selects the appropriate Immutable NVMTable list of itself (NTI) or Immutable NVMTable list of other partitions (NTO) for compression. The rules are as follows:

[0130] (1) If the tuple stores the address of an Immutable NVMTable list of other partitions (NTO), check whether there is an SSTable in the L1 layer whose key range overlaps with the key range of the Immutable NVMTable list of other partitions (NTO) and is currently being compressed; if so, skip the tuple; otherwise, compress the Immutable NVMTable list of other partitions (NTO), and remove the tuple after compression is complete; because if multiple threads select the same SSTable for compression at the same time, the key ranges of the generated new SSTables will overlap, affecting subsequent compression and read operations.

[0131] (2) If the tuple stores the address of an Immutable NVMTable list of itself (NTI), then relative to the Immutable NVMTable list of other partitions (NTO), the Immutable NVMTable list of itself (NTI) must meet an additional condition: find the corresponding Partition Node (PN) based on the tuple, and check whether the Immutable NVMTable list of other partitions (NTO) in the same partition is empty; if it is empty, compress the Immutable NVMTable list of itself (NTI); otherwise, skip the tuple; this is because if the Immutable NVMTable list of other partitions (NTO) comes from a split partition, the key values ​​it contains are newer than the key-value pairs in the Immutable NVMTable list of itself (NTI). If the Immutable NVMTable list of other partitions (NTO) is not compressed first, the old data may overwrite the new data; in addition, compressing the Immutable NVMTable list of other partitions (NTO) first also helps to make partitions split or merge.

[0132] Experimental analysis

[0133] The Persistent Memory Development Kit (PMDK) is a set of libraries and tools designed for system administrators and application developers to simplify the management and access of persistent memory devices. This paper uses the PMDK and implements PartitionKV on RocksDB.

[0134] Next, we will briefly introduce its writing and reading processes. The flowcharts are as follows Figure 3 and Figure 4 As shown:

[0135] 1. Implementation plan

[0136] like Figure 3 As shown, the writing process of the LSM tree key value storage system (PartitionKV) is:

[0137] (1) First, index the B+ tree according to the key of the key-value pair, locate the corresponding partition, and then write the key-value pair into the Mutable NVMTable in the partition;

[0138] (2) When the Mutable NVMTable is full, the Mutable NVMTable will be converted into ImmutableNVMTable and appended to the end of the Immutable NVMTable list of itself (NTI); then the compression priority of the Immutable NVMTable list of itself (NTI) will be adjusted, and it will be checked whether the partition needs to be split or merged; if necessary, after completing the corresponding operation, the compression priority of the Immutable NVMTable list of other partitions (NTO) corresponding to the partition will also be adjusted; finally, a new Mutable NVMTable will be created to receive the key-value pairs written later.

[0139] like Figure 4 As shown, the reading process of the LSM tree key value storage system (PartitionKV) is:

[0140] The search priorities from high to low are: Mutable NVMTable, Immutable NVMTable list of itself (NTI), Immutable NVMTable list of other partitions (NTO), and from L1 layer to L nFirst, search the partition where the specified key is located through the B+ tree, and then search from the Mutable NVMTable of the partition; if not found, search from the tail to the head of the Immutable NVMTable list of itself (NTI), then search from the tail to the head of the Immutable NVMTable list of other partitions (NTO), and finally search from L1 to L n Search in the SSTable of the layer.

[0141] 2. Experimental Setup

[0142] All experiments were conducted on a Linux server equipped with two Intel(R) Xeon(R) Gold 6240 2.60GHz 18-core processors and 379GB of DRAM. The kernel version was Linux 5.4.0-42-generic, and the operating system was Ubuntu 20.04.6LTS. The experiments used a 1.84TB Intel SSD PE2KX020T8 solid-state drive (SSD) with the EXT4 file system and two 128GB Intel Optane non-volatile memory (NVM) banks with the EXT4-DAX file system.

[0143] This paper mainly compares the performance of PartitionKV with RocksDB, NoveLSM, MatrixKV, and FlatLSM. The experimental setup is briefly described below:

[0144] (1) RocksDB version is 9.1.0, memtable size is 64MB, subcompaction is enabled and 5 threads are used.

[0145] (2) NoveLSM uses 8GB non-volatile memory NVM Memtable and 64MB memory DRAM Memtable, working alternately.

[0146] (3) The non-volatile memory NVM size of MatrixKV is 8GB, and the Memtable size is 64MB.

[0147] (4) FlatLSM uses 8GB of non-volatile memory (NVM), which includes four 2GB PM logs. A flush operation is triggered when the number of Immutable PMTables reaches 2, and the number of threads processing compression is set to 5. It should be noted that the key-value separation strategy is not enabled when implementing FlatLSM in this paper.

[0148] (5) The non-volatile memory NVM size of PartitionKV is 8GB, and the NVM Log size is 64MB. Therefore, the number of NVMLogs is 8GB / 64MB=128. The number of compression threads between the partition layer and the L1 layer is set to 5, and the threshold parameter is set as follows: δ min =20,δ mid =40,δ max =60, βs1=2, βs2=10, βm2=4, βs3=0.033, βm3=0.008, θ=3.

[0149] 3. Experimental results

[0150] (1) To evaluate the read and write performance of PartitionKV, YCSB was used to conduct a comprehensive evaluation of the database performance under different workloads (YCSB-A to YCSB-F). The results are as follows: Figure 5 Under each workload, we first randomly write an 80GB dataset with a key size of 16B and a value size of 4096B to these databases, and then perform one million operations to measure their performance.

[0151] Experimental results show that in the Load, YCSB-A, and YCSB-F scenarios, PartitionKV has higher throughput than FlatLSM, MatrixKV, NoveLSM, and RocksDB. In the Load (i.e., random write) scenario, PartitionKV is 1.65 times, 2.01 times, 2.03 times, and 2.60 times faster than FlatLSM, MatrixKV, NoveLSM, and RocksDB, respectively. In the YCSB-B, YCSB-C, and YCSB-D scenarios, PartitionKV also exhibits higher throughput than MatrixKV, FlatLSM, NoveLSM, and RocksDB. However, in the YCSB-E (range query) scenario, PartitionKV's throughput is comparable to FlatLSM and MatrixKV, lower than RocksDB, but still higher than NoveLSM. These experimental results show that PartitionKV performs better in all scenarios except the range query scenario.

[0152] (2) To evaluate the write latency of PartitionKV, the maximum, average, P90, P99, and P999 latencies are calculated in the YCSB Load scenario, and the results are shown in Table 1. PartitionKV has the lowest maximum and average latencies, while its P90, P99, and P999 latencies are higher than FlatLSM.

[0153] Table 1 Random Write Latency

[0154] Delay (us) maximum average P90 P99 P999 RocksDB 257163.26 70.36 22.86 1093.63 2148.35 NoveLSM 33067892.73 54.41 26.06 40.99 5394.43 MatrixKV 101253.12 52.39 20.28 1087.15 1093.46 FlatLSM 51707379.71 43.88 10.68 13.53 16.61 PartitionKV 56202.58 22.60 12.71 582.66 588.63

[0155] (3) To evaluate PartitionKV's write amplification, we randomly wrote 80GB of key-value pairs to these databases, with a key size of 16B and a value size of 4KB. The results are shown in Table 2. As can be seen, PartitionKV has the lowest write amplification. The actual amount of data written by PartitionKV is 1.65 times, 1.31 times, 1.29 times, and 1.65 times less than that of FlatLSM, MatrixKV, NoveLSM, and RocksDB, respectively.

[0156] Table 2 Write Amplification

[0157] database Write Amplification Actual written data (GB) RocksDB 5.14 411.2 NoveLSM 4.01 320.6 MatrixKV 4.08 326.8 FlatLSM 5.15 411.7 PartitionKV 3.12 249.6

[0158] The important protection points of the present invention are:

[0159] Designing an ordered partitioning layer minimizes data rewriting during compression and incorporates a write-ahead log function, significantly reducing write amplification.

[0160] An adaptive partitioning strategy is proposed to ensure the appropriate amount of data involved in compression and significantly improve the space utilization of non-volatile memory (NVM).

[0161] A multi-threaded compression strategy is proposed to prioritize compressing old data and accelerate the release of non-volatile memory (NVM) space.

[0162] Through the collaboration of adaptive partitioning and multi-threaded compression, PartitionKV achieves fine-grained and efficient compression, thereby significantly reducing write pauses.

Claims

1. An LSM tree key-value storage system based on non-volatile memory and using an adaptive partitioning strategy, characterized in that: Includes storage layer and partition layer; The storage layer is located on the solid state drive SSD, and the storage layer includes L1 to L n Layers, each layer contains several SSTables; the key ranges of SSTables in the same layer do not intersect; The partition layer includes: (1) Multiple partitions spanning DRAM and NVM are used to replace the key-value pairs received by the Mutable Memtable. Each partition is responsible for maintaining a specific key range of key-value pairs. The key ranges of each partition do not overlap and are continuous. The key range of each partition overlaps with the key ranges of a certain number of SSTables in the L1 layer, i.e., overlapping SSTables. (2) A B+ tree in DRAM, used to index the partitions; When writing or reading a key-value pair, first use the B+ tree to find the partition whose key range contains the key of the key-value pair; then, write the key-value pair to the partition or start searching from the partition; if the number of overlapping SSTables in the L1 layer is greater than the preset threshold or less than the preset threshold, use the adaptive partitioning strategy to split or merge the partitions; when the number of key-value pairs written to the partition reaches the preset threshold, the partition is compressed to the L1 layer.

2. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 1, characterized in that: A multi-threaded compression strategy is adopted to compress the partition layer and the L1 layer in the storage layer.

3. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 1, characterized in that: The structure of the partition includes: Partition Node (PN), Mutable NVMTable, MetaNode (MN), Immutable NVMTable list of itself (NTI) and Immutable NVMTable list of other partitions (NTO); wherein, PartitionNode (PN) is used to record the metadata of the partition, MutableNVMTable is used to receive the written key-value pairs, if the Mutable NVMTable is full, it will be converted into ImmutableNVMTable and then added to the end of the Immutable NVMTable list of itself (NTI), when a partition is split or merged, the Immutable NVMTable list of itself (NTI) is converted into Immutable NVMTable list of other partitions (NTO); MetaNode (MN) is used to record key partition status information for partition recovery.

4. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 3, characterized in that: The Partition Node (PN) includes: (1) a 16-byte Start Key and a 16-byte EndKey, which together represent the key range of the partition; a 2-byte Start Key Size and a 2-byte EndKey Size, which respectively represent the sizes of the start key and the end key; (2) four pointers, an 8-byte NVMTablePointer, an 8-byte NTIPointer, an 8-byte NTO Pointer, and an 8-byte MNPointer; these four pointers point to the Mutable NVMTable, Immutable NVMTable list of itself (NTI), Immutable NVMTable list of other partitions (NTO), and MetaNode (MN) in the partition respectively; When a write or read request arrives, the partition whose key value range contains the requested key is first located through the B+ tree in the memory DRAM; then, the Mutable NVMTable in this partition is located using the information in the PartitionNode (PN) to write the key-value pair or start the query from it, and then the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) are queried; The physical structure used by the Mutable NVMTable and Immutable NVMTable is NVMTable. The NVMTable includes a Minimum Key, a Maximum Key, a NextNVMTable Pointer, a jump table located in the memory DRAM, and an NVM Log located in the non-volatile memory NVM. Among them, the Minimum Key and Maximum Key together represent the key range of the NVMTable. The Next NVMTable Pointer links adjacent ImmutableNVMTables to form an Immutable NVMTable list. The jump table points to the key-value pairs of the NVM Log. The specific structure of the NVM Log includes a 28-byte header and multiple log entries, as follows: The header records the metadata of the NVM Log, including: (1) a 4-byte Magic Number, which is used to indicate whether the NVM Log is in use. If the content is "USED", it means that the NVM Log is in use. If the content is "FREE", it means that it is invalid and the NVM Log is recycled; (2) an 8-byte File Size and an 8-byte Used Size. FileSize and Used Size respectively indicate the capacity of the NVM Log and the size of the space used; (3) an 8-byte Next NVM Log Pointer, which is used to persist the link relationship between the NVM Logs of multiple Immutable NVMTables in the non-volatile memory NVM; The Log Entry is used to store written key-value pairs. Each Log Entry includes a Key Size, a Key, a Value Size, and a Value. Key and Key Size represent the key and key size, respectively, while Value and Value Size represent the value and value size, respectively. When a new key-value pair is written to a partition, the key-value pair is written in the form of an append write. The key-value pair is first added to the end of the NVM Log in the form of a Log Entry, and the UsedSize field of the Header is immediately updated. Then, the corresponding index node is inserted into the skip list and the Minimum Key and Maximum Key fields are updated. The MetaNode (MN) is used to record key partition status information required for partition recovery. During normal system operation, when the status of a partition changes, the MetaNode (MN) of the partition is modified accordingly. The structure of the MetaNode (MN) includes: (1) A 4-byte Magic Number, used to indicate whether the Meta Node (MN) is in use. If the content is "USED", it means that the Meta Node (MN) is in use. If the content is "FREE", it means that the Meta Node (MN) is invalid and the partition layer recycles the Meta Node (MN); (2) A 2-byte Start Key Size, a 16-byte Start Key, a 2-byte End Key Size, and a 16-byte End Key; the Start Key and End Key represent the key range of the partition, and the Start Key Size and End Key Size represent the size of the starting key and ending value, respectively; (3) An 8-byte NVM Log Pointer, pointing to the NVM Log of the Mutable NVMTable in the partition; (4) An 8-byte NVMLIPointer and an 8-byte NVMLO Pointer; NVMLIPointer points to the NVM Log of the first Immutable NVMTable in the Immutable NVMTable list of itself (NTI) within the partition, and NVMLO Pointer points to the NVM Log of the first Immutable NVMTable in the Immutable NVMTable list of other partitions (NTO) within the partition; When a system failure occurs, it is recovered through the Meta Node (MN) and NVM Log located in the non-volatile memory NVM; The Immutable NVMTable list of itself (NTI) consists of multiple Immutable NVMTables in this partition and is used to store key-value pairs within the partition key range. When the storage space used by the Mutable NVMTable reaches a preset threshold, the Mutable NVMTable is first converted to an Immutable NVMTable, and then the available compression thread compresses the Immutable NVMTable to the L1 layer. At the same time, newly written key-value pairs are continuously written to the newly allocated Mutable NVMTable at a high speed and then converted to the Immutable NVMTable. Finally, the Immutable NVMTable list of itself (NTI) is formed. When a Mutable NVMTable is converted to an Immutable NVMTable, there are two situations: Case 1: If the Immutable NVMTable list of itself (NTI) is empty, the Immutable NVMTable will become the head of the ImmutableNVMTable list of itself (NTI); therefore, the NTIPointer in the PartitionNode (PN) will point to the Immutable NVMTable, and the NVMLIPointer in the MetaNode (MN) will point to the NVM Log corresponding to the Immutable NVMTable; Case 2: If the Immutable NVMTable list of itself (NTI) is not empty, the ImmutableNVMTable will be appended to the end of the Immutable NVMTable list of itself (NTI), which is achieved by updating the Next NVMTable Pointer and Next NVM Log Pointer of the current tail Immutable NVMTable; In addition, whenever a new Immutable NVMTable is formed, it is necessary to determine whether the partition needs to be split or merged. If splitting or merging is not required, a new Mutable NVMTable is allocated and the Partition is updated. The NVMTable Pointer in the Node (PN) points to the newly allocated Mutable NVMTable, and the NVMLog Pointer in the MetaNode (MN) is set to the NVM Log corresponding to the newly allocated Mutable NVMTable; The Immutable NVMTable list of other partitions (NTO) is converted from the Immutable NVMTable list of itself (NTI) of other partitions after the other partitions are split or merged; the key range of the key-value pairs stored in the Immutable NVMTable list of other partitions (NTO) in this partition may exceed the key range of this partition; The Immutable NVMTable in the Immutable NVMTable list of other partitions (NTO) will be compressed to the L1 layer by the compression thread first.

5. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 4, characterized in that: When a system failure occurs, it is recovered through the MetaNode (MN) and NVM Log located in the non-volatile memory NVM. The recovery process includes: (1) According to the NVM Log Pointer in MetaNode (MN), scan the key-value pairs in NVM Log and rebuild the corresponding jump table, Minimum Key and Maximum Key in memory DRAM; this process restores Mutable NVMTable; (2) According to NVMLIPointer in MetaNode (MN) and NextNVM Log Pointer in NVM Log, scan NVM Log list and rebuild Immutable NVMTable list of itself (NTI); (3) The recovery process of Immutable NVMTable list of other partitions (NTO) is the same as that of Immutable NVMTable list of itself (NTI); (4) Rebuild Partition Node (PN) based on the partition key range in MetaNode (MN) and the restored Mutable Memtable, Immutable NVMTablelist of other partitions (NTO) and Immutable NVMTable list of itself (NTI), and finally restore the partition.

6. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 1, characterized in that: The implementation method of the adaptive partitioning strategy is as follows: If a partition has a large number of overlapping SSTables, the partition will be split into two smaller partitions; when the overlapping SSTables between the partition and the L1 layer are small, the partition will be merged with the adjacent partition; The process of the adaptive partitioning strategy is as follows: for any given target partition, whenever a Mutable NVMTable is converted to an Immutable NVMTable and immediately added to the end of the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of other partitions (NTO) is not empty, a new Mutable NVMTable is allocated to receive the new key-value pair; otherwise, a decision is made as to whether to split or merge the partition; if splitting or merging is required, the corresponding operation is performed; Otherwise, create a new Mutable NVMTable to hold the new key-value pairs.

7. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 6, characterized in that: The specific process of determining whether to split or merge the partition includes: When the system is first started, the partition layer contains only one partition. As key-value pairs are continuously written, the number of partitions in the partition layer (denoted as n p ) gradually increases until it reaches the maximum value specified by the user (denoted as δ max ); according to n p The value of , the system operation process is divided into three stages; if 1≤n p ≤δ min , then it is in stage 1; if δ min <n p ≤δ mid , then it is in the second stage; if δ mid <n p ≤δ max , then it is in the third stage, where δ min and δ mid are two user-specified parameters; Let n s Let βs i and βm i be the user-specified thresholds at stage i for deciding whether the target partition needs to be split or merged. Different rules are applied at each stage as follows: Rule 1: This rule applies to the first stage when the system starts; βs 1 is usually set to a small value. s ≥βs 1, then the target partition needs to be split; Rule 2: This rule applies in phase 2 when there is still enough non-volatile memory NVM storage space to accommodate new key-value pairs; a fixed threshold is used to decide whether to split or merge partitions; if n s ≥βs 2, the partition needs to be split; otherwise, if n s ≤βm 2 and n p -1>δ min , then the partition needs to be merged, where βs 2>βm 2; constraint n p -1>δ min Make sure you are still in stage 2 after completing the partition merging process; Rule 3: This rule applies to the third stage, in which the thresholds for splitting and merging are dynamically adjusted; according to the coverage ratio r c Decide whether to split or merge partitions, r c Defined as the ratio of the number of overlapping SSTables in the partition to the total number of SSTables in the L1 layer; if r c ≥βs 3 and n p +1≤δ max , then the partition needs to be split; otherwise, if r c ≤βm3, then the partition needs to be merged.

8. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 7, characterized in that: The partition splitting process is as follows: For a target partition, first identify its overlapping SSTables in the L1 layer and select the largest key in the overlapping SSTable in the middle as the split key; Then create a new partition, update the key ranges of both partitions based on the split key, and immediately adjust the B+ tree index; Next, the original partition's Immutable NVMTable list of itself (NTI) is converted into an Immutable NVMTable list of other partitions (NTO) by updating the NTO Pointer in the Partition Node (PN) of the two partitions. Finally, the key range, NVMLIPointer, and NVMLO Pointer in the two partition MetaNodes (MN) are updated to persist the partition status according to their respective PartitionNodes (PN).

9. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 7, characterized in that: The process of partition merging is as follows: For a target partition, first find an acceptable partition, which is defined as: the partition whose ImmutableNVMTable list of otherpartitions (NTO) in the adjacent partition is empty and the number of overlapping SSTables is less than that of another adjacent partition; Subsequently, the key range of the acceptable partition is expanded to the union of the two partition key ranges, and the B+ tree index is updated; Subsequently, the original partition's Immutable NVMTable list of itself (NTI) is converted to an Immutable NVMTable list of other partitions (NTO) by modifying the NTO Pointer in the acceptable partition PartitionNode (PN); Finally, delete the partition to be merged and update the MetaNode (MN) according to the Partition Node (PN) of the acceptable partition to persist the partition status.

10. The LSM tree key-value storage system based on non-volatile memory and adopting an adaptive partitioning strategy according to claim 2, characterized in that: The implementation method of the multi-thread merging strategy includes: Set two priority queues to manage the compression priorities of Immutable NVMTable list of itself (NTI) and Immutable NVMTable list of other partitions (NTO). Each available thread selects an Immutable NVMTable list of itself (NTI) or Immutable NVMTable list of other partitions (NTO) from the two priority queues for compression. The two priority queues manage the compression priorities of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) as follows: When a partition receives frequent writes, the number of Immutable NVMTables in its Immutable NVMTable list of itself (NTI) will increase rapidly. When the number of Immutable NVMTables in the Immutable NVMTable list of itself (NTI) reaches the user-set threshold θ, the write speed of the partition is slowed down, and the Immutable NVMTable list of itself (NTI) is prioritized for compression. In order to prioritize the compression of the Immutable NVMTable list that is formed earlier and contains more data, a high-priority queue and a low-priority queue are designed to manage the compression priority of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue store a pair of (Immutable NVMTable list of itself (NTI) address or Immutable NVMTable list of other partitions (NTO) address, PartitionNode (PN) address), where the Immutable NVMTable list of itself (NTI) or Immutable NVMTable list ofotherpartitions(NTO) and PartitionNode(PN) belong to the same partition; The compaction priority of the Immutable NVMTable list of itself (NTI) and the Immutable NVMTable list of other partitions (NTO) is set in the following way: (1) When the partition is split or merged, the original partition's Immutable NVMTable list of itself (NTI) is converted to Immutable NVMTable list of other partitions (NTO). The high-priority queue and the low-priority queue are traversed to find the tuple containing the address of the Immutable NVMTable list of itself (NTI) and remove it. Then, a new tuple containing the address of the Immutable NVMTable list of other partitions (NTO) and the corresponding PartitionNode (PN) address is created and added to the high-priority queue. (2) When a new Immutable NVMTable is added to the Immutable NVMTable list of itself (NTI), if the Immutable NVMTable list of itself (NTI) only contains one Immutable NVMTable, create a new two-tuple and add it to the low priority queue; otherwise, find the existing two-tuple, and if it is not in the high priority queue, remove it from the low priority queue and add it to the high priority queue; The available thread selects an Immutable NVMTable list of itself (NTI) or an Immutable NVMTable list of other partitions (NTO) from two priority queues for compression as follows: Multiple threads are allocated for compression between the partition layer and the L1 layer. Each available thread sequentially traverses the high-priority queue, then the low-priority queue, and selects the appropriate Immutable NVMTable list of itself (NTI) or Immutable NVMTable list of other partitions (NTO) for compression. The rules are as follows: (1) If the tuple stores the address of an Immutable NVMTable list of other partitions (NTO), check whether there is an SSTable in the L1 layer whose key range overlaps with the key range of the Immutable NVMTable list of other partitions (NTO) and is currently being compressed; if so, skip the tuple; otherwise, compress the Immutable NVMTable list of other partitions (NTO), and remove the tuple after compression is complete; (2) If the tuple stores the address of an Immutable NVMTable list of itself (NTI), relative to the Immutable NVMTable list of other partitions (NTO), the Immutable NVMTable list of itself (NTI) must also meet a condition: find the corresponding Partition Node (PN) based on the tuple, and check whether the Immutable NVMTable list of other partitions (NTO) in the same partition is empty; if it is empty, compress the Immutable NVMTable list of itself (NTI); otherwise, skip the tuple.

Images