LSM-Tree key value data layout architecture based on partition namespace solid-state disk

Abstract

The invention relates to the technical field of key value storage systems, and particularly discloses an LSM-Tree key value data layout architecture based on a partition namespace solid-state disk. Comprising the following steps: S1, when a new SSTable of an L layer is written, searching a zone with an overlapped key; s2, sorting zones with overlapped keys in the L + 1 layer according to the residual space; s3, selecting the zone with the maximum residual space from the sorted zones for writing, and if no zone containing the overlapping key exists, writing the zone into a new zone; and S4, in the selected zone, if the residual effective SSTable is less than a set threshold value K, writing the residual effective SSTable into a new zone along with the Compaction operation. According to the architecture, the overlapping performance of SSTable data in the same hierarchy is fully considered, so that invalid data residues in the Zone are reduced, overlapping key data in the same Zone can be processed at a time, the data migration amount during garbage collection is reduced, and the double-write amplification effect caused by large-granularity management and multiple data rewriting is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of key-value storage systems, and in particular to an LSM-Tree key-value data layout architecture based on a partition namespace solid-state disk. Background Art

[0002] ZNS SSD (Zoned Namespace Solid State Drive) is a new type of storage device designed to optimize I / O operations and improve the overall performance of storage systems through innovative storage area management. ZNS SSD divides the storage space into multiple independent namespaces (Zones), each of which has the characteristic of linear writing, which means that data can only be written sequentially and not randomly. In high-performance storage systems, ZNS SSD can better cope with write-intensive and random read-write workloads due to its unique architectural design, making it an ideal choice for modern data centers and mission-critical applications. Unlike traditional SSDs, ZNS SSD has made important innovations in the garbage collection mechanism. Traditional SSDs usually rely on built-in controllers to manage garbage collection, while ZNS SSDs hand over this responsibility to the host-side software. In ZNS SSD, garbage collection is performed in units of hardware partitions (Zones). When part of the data in a zone fails or needs to be updated, the host software will select these zones that need to be garbage collected and migrate the valid data in them to other writable zones, and then reset the original zones to a writable state. By transferring garbage collection tasks to the host side, ZNS SSD can more flexibly manage data migration and storage resource utilization, further reducing unnecessary write operations and extending the service life of the device.

[0003] On the other hand, LSM-Tree (Log Structured Merge Tree) is an efficient index structure designed specifically for write-intensive workloads and is widely used in key-value storage systems. LSM-Tree forms a hierarchical tree structure by storing data in multiple SSTables (sorted string tables) in layers. As new data is written, the data is first stored in the MemTable in memory. When the MemTable reaches a certain size, it is persisted to disk to form a new SSTable. In order to maintain the order of data and improve query efficiency, LSM-Tree periodically performs merge sort (Compaction) operations to merge the upper SSTable into the lower layer. This process can reduce data fragmentation and optimize query performance.

[0004] In a key-value storage system using LSM-Tree, data is written in batches to immutable files called SSTables. Over time, the system periodically performs merge sort operations in the background to merge smaller, older SSTables into new, larger SSTables to keep the data in order and reduce the complexity of lookups. However, during the merge sort process, data is read and rewritten multiple times, which significantly increases the number of write operations, i.e., the write amplification problem. In ZNS SSD, data is usually managed in blocks. As data is continuously written, the data in some blocks may become invalid or outdated. At this time, the SSD needs to perform garbage collection operations to remove these invalid data from the blocks and make room for new data to be written. However, due to the large storage management granularity of ZNS SSD, when performing garbage collection, the data in the entire zone may need to be migrated to other locations to release continuous free space. This process also brings additional write operations, further increasing write amplification.

[0005] When the two interact, the write amplification effect is further amplified. For example, merge sort may concentrate a large amount of data into certain areas, which may then need to be garbage collected, resulting in the same data being migrated and written multiple times. This double write amplification effect not only increases the total write volume of the system, but also has an adverse impact on the life of the storage device, because the write life of the SSD is limited. Summary of the invention

[0006] In view of the above-mentioned problems in the prior art, the present invention provides an LSM-Tree key-value data layout architecture based on a partitioned namespace solid-state disk. Through this architecture, the write amplification problem faced by ZNS SSD in the key-value storage system can be effectively reduced, and the performance of the storage system and the overall durability of the device can be improved.

[0007] The technical solution adopted by the present invention is as follows: An LSM-Tree key-value data layout architecture based on a partitioned namespace solid state disk includes the following steps: S1. When writing a new L-layer SSTable, look for zones with overlapping keys. S2, sort the zones with overlapping keys in the L+1 layer according to the remaining space; S3: Select the zone with the largest remaining space from the sorted zones for writing. If there is no zone with overlapping keys, write to a new zone. S4. In the selected zone, if the remaining valid SSTables are less than the set threshold K, the remaining valid SSTables will be written into the new zone together with this compaction operation.

[0008] Preferably, data is placed in different zones based on overlapping keys in the SSTable.

[0009] Preferably, the compaction operation triggered at the L layer will select and merge the SSTables containing overlapping keys at the L+1 layer with those at the L layer.

[0010] Preferably, each time a new SSTable is written, it is written to a zone that has overlapping keys with it.

[0011] Preferably, in step S3, if a compaction operation is triggered at this time, when an SSTable is selected, and overlapping keys are contained in the same zone, the valid data that needs to be relocated during garbage collection in a zone is minimized.

[0012] Preferably, after the compaction operation is performed on the data in the zone in step S4, if the remaining valid SSTables are less than the set threshold K, the zone does not need to perform an additional garbage collection to relocate the data.

[0013] Preferably, the threshold value K set in step S4 is obtained by experiments, and the interval of setting the K threshold value is 10%-20%.

[0014] Preferably, ZNS SSD adopts a lifecycle-based area allocation algorithm when processing LSM-Tree data placement, placing SSTables that need to be updated and deleted quickly at the lower level of the LSM-Tree, and placing SSTables that are updated relatively late at the higher level of the LSM-Tree.

[0015] Preferably, ZenFS assigns a life cycle to each level of the LSM-Tree according to the update frequency.

[0016] Preferably, data in the same level have different survival times.

[0017] The present invention has the following characteristics and advantages: (1) The present invention intelligently allocates SSTables with overlapping keys to the same zone, achieving efficient coordination between the merge sort (Compaction) operation and the garbage collection (Garbage Collection) operation, and fully considering the overlap of SSTable data in the same level, thereby reducing the residual invalid data in the zone; (2) The present invention intelligently maps SSTables of different levels in the LSM-Tree to different zones of the ZNS SSD. Valid data is processed centrally, which reduces the amount of data migration during garbage collection, reduces the impact of data migration and write amplification, and thus reduces the damage of write amplification to system performance and device life, thereby improving the performance of the storage system and the overall durability of the device. (3) By reducing the number of write operations, the present invention significantly improves the overall performance of the storage system, especially when processing write-intensive loads, making it more stable and efficient.

[0018] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram of the merge sort process caused by the existing data placement algorithm; Figure 2 This is a diagram of the merge sort process after the present invention places SSTables related to overlapping keys in the same zone. DETAILED DESCRIPTION

[0020] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0021] An LSM-Tree key-value data layout architecture based on a partitioned namespace solid state disk is shown, including the following steps: S1. When writing a new L-layer SSTable, look for zones with overlapping keys. S2, sort the zones with overlapping keys in the L+1 layer according to the remaining space; S3: Select the zone with the largest remaining space from the sorted zones for writing. If there is no zone with overlapping keys, write to a new zone. S4. In the selected zone, if the remaining valid SSTables are less than the set threshold K, the remaining valid SSTables will be written into the new zone together with this compaction operation.

[0022] The existing ZNS SSD adopts a lifecycle-based zone allocation algorithm when processing LSM-Tree data placement, which places data with the same lifecycle in the same zone. ZenFS, the file system that comes with ZNS SSD, uses the following features of LSM-Tree to predict the lifecycle of each data. In the LSM-tree, SSTables that need to be updated and deleted quickly are located at lower levels, and SSTables that are updated relatively late are located at higher levels. ZenFS assigns a lifecycle to each level of the LSM-Tree based on the update frequency, predicts the lifecycle of the newly created SSTable based on the lifecycle to which the SSTable belongs, and places SSTables with the same lifecycle in the same zone so that they can be deleted together when compression is triggered in the future.

[0023] However, this method does not take into account that other irrelevant data in the same level may still remain in the corresponding zone after the compaction operation, resulting in the inability to reduce the amount of valid data remaining in the victim zone.

[0024] The merge sort process caused by the existing data placement algorithm is shown in the figure Figure 1 As shown, data is placed in LSM-Tree. It can be seen that four zones are divided, namely Zone0 (Level 1), Zone1 (Level 2), Zone2, and Zone3. Zone0 (Level 1) has SSTables 25-60, 83-90, 92-101, and 115-129, and Zone1 (Level 2) has SSTables 0-20, 21-28, 28-40, 85-93, 94-99, and 100-126. That is, SSTables that are updated and deleted quickly are located at a lower level, and SSTables that are updated relatively late are located at a higher level.

[0025] After the Compaction (merge sort) operation is performed on the SSTables in these selected zones, there are still invalid SSTables residing in Zone0 (Level 1). The invalid SSTables are 92-101, with a number of 1. New SSTables are formed from 25-60, 83-90, and 115-129 for writing. There are still many invalid SSTables residing in Zone1 (Level 2), including 85-93, 94-99, and 100-126.

[0026] At this time, if the garbage collection operation is triggered for Zone1 (Level 2), the level 2 SSTable retained in the zone needs to be written to the new Zone3. If the compaction operation and garbage collection operation are continued, the data in level 2 will be repeatedly written to the new zone, causing serious write amplification problems. This is mainly because ZenFS does not take into account the different survival times of data in the same level.

[0027] Therefore, we consider placing data into different zones based on overlapping keys in SSTable.

[0028] Since the compaction operation triggered at the L layer will select and merge the SSTables with overlapping keys at the L+1 layer, we write each new SSTable into the zone with overlapping keys.

[0029] The present invention places the SSTables related to overlapping keys in the same zone and then merges and sorts them. Figure 2 As shown, data is placed in LSM-Tree. It can be seen that four zones are divided, namely Zone0, Zone1, Zone2, and Zone3. Zone0 has SSTables 25-60, 83-90, 0-20, 21-28, and 28-40, and Zone1 has SSTables 92-101, 115-129, 85-93, 94-99, 100-126, ...

[0030] The SSTables in these selected areas are compacted (merge sorted) and garbage collected. The specific operation process is as follows: (1) When writing a new L-layer SSTable, look for zones with overlapping keys; (2) Sort the zones with overlapping keys in the L+1 layer according to the remaining space; (3) Select the zone with the largest remaining space from the sorted zones for writing. If there is no zone with overlapping keys, write to a new zone. (4) If a compaction operation is triggered at this time, when selecting SSTable, since the same zone contains overlapping keys, the valid data that needs to be moved during garbage collection in a zone can be minimized.

[0031] (5) In the selected zone, if the remaining valid SSTables are less than the set threshold K, the remaining valid SSTables are written into the new zone together with this compaction operation.

[0032] From this, we can see that the remaining valid SSTables in Zone0 are higher than the set threshold k. The k value is obtained through experiments. Because SSTable may be set to different values, the K threshold is set between 5% and 50% for experiments. The effect is best in the range of 10%-20%, and the write amplification rate is the lowest.

[0033] After the operation, the valid SSTables written initially are retained. The number of valid SSTables remaining in Zone 1 is small, less than the set threshold K, and less than the first experimental solution. The remaining valid SSTables are written to the new Zone 3 along with this compaction operation. At this time, Zone 1 does not need to perform an additional garbage collection to relocate data, reducing the increase in data migration caused by large-granularity management and the double write amplification effect caused by merge sorting and garbage collection.

[0034] From the above embodiments, it can be seen that the present invention is different from the existing Zone allocation method based on lifecycle prediction. The method of the present invention takes into account the overlap of SSTable data in the same level, thereby reducing the invalid data residue in the Zone. When the system triggers a Compaction operation, the overlapping key data in the same Zone can be processed at one time, avoiding the situation where these data are migrated multiple times in subsequent garbage collection. In this way, the amount of data migration during garbage collection is reduced, and the double write amplification effect caused by large-granularity management and multiple data rewrites is reduced.

[0035] The present invention combines the storage characteristics of ZNS SSD with the index structure of LSM-Tree to propose an SSTable data layout architecture based on overlapping keys, which significantly reduces the write amplification effect. Specifically, by using the hierarchical architecture of LSM-Tree, the present invention achieves efficient collaboration between the merge sort (Compaction) operation and the garbage collection (Garbage Collection) operation by intelligently allocating SSTables with overlapping keys to the same zone.

[0036] After adopting the present invention, the amount of data migration required for garbage collection operations is significantly reduced. At the same time, since valid data is processed centrally, the free space of the zone is used more efficiently, thereby extending the service life of the SSD. In addition, by reducing the number of write operations, the overall performance of the storage system is significantly improved, especially when processing write-intensive loads, it is more stable and efficient.

[0037] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An LSM-Tree key-value data layout architecture based on a partitioned namespace solid state disk, characterized in that: The following steps are involved: S1. When writing a new L-layer SSTable, look for zones with overlapping keys. S2, sort the zones with overlapping keys in the L+1 layer according to the remaining space; S3: Select the zone with the largest remaining space from the sorted zones for writing. If there is no zone with overlapping keys, write to a new zone. S4. In the selected zone, if the remaining valid SSTables are less than the set threshold K, the remaining valid SSTables will be written into the new zone together with this compaction operation.

2. The architecture according to claim 1, characterized in that Place data into different zones based on overlapping keys in SSTable.

3. The architecture according to claim 1, characterized in that The compaction operation triggered at the L layer will select and merge the SSTables with overlapping keys at the L+1 layer.

4. The architecture according to claim 1, characterized in that: Each time a new SSTable is written, it is written to a zone that has overlapping keys.

5. The architecture according to claim 1, characterized in that: In step S3, if the compaction operation is triggered at this time, when the SSTable is selected, and the same zone contains overlapping keys, the valid data that needs to be moved during garbage collection in a zone is minimized.

6. The architecture according to claim 1, characterized in that: After the compaction operation is performed on the data in the zone in step S4, if the remaining valid SSTables are less than the set threshold K, the zone does not need to perform an additional garbage collection to relocate the data.

7. The architecture according to claim 1, characterized in that: The threshold value K set in step S4 is obtained from experiments, and the interval of the K threshold value is set to 10%-20%.

8. The architecture according to claim 1, characterized in that: ZNS SSD uses a lifecycle-based area allocation algorithm when processing LSM-Tree data placement, placing SSTables that need to be updated and deleted quickly at the lower level of the LSM-Tree, and placing SSTables that are updated relatively late at the higher level of the LSM-Tree.

9. The architecture according to claim 1, characterized in that: ZenFS assigns a lifecycle to each level of the LSM-Tree based on the update frequency.

10. The architecture according to claim 1, characterized in that: Data in the same level have different survival times.

Images