A key-value separation LSM-tree garbage collection optimization method for ZNS SSDs

Abstract

The application belongs to the field of data storage, and particularly relates to a key-value separation LSM-tree garbage collection optimization method for ZNS SSD, and constructs a key-value separation LSM-tree including a segment manager, a partition manager and ZenFS. The storage space of the ZNS SSD is statically divided into two independent areas, which are respectively managed by the partition manager and the ZenFS. The value log managed by the ZenFS is stripped and is uniformly controlled by the partition manager. The partition manager realizes data placement, partition bitmap management, garbage collection and crash recovery. The application introduces an address remapping mechanism and a bitmap mechanism, effectively reduces the interference of the write-back and read-back of the key-value separation LSM-tree on the traditional solid state disk in the garbage collection process on the front-end read-write, avoids the inconsistency of data, and reduces the number of data disk landing times, reduces the total amount of data written to the disk, and improves the writing performance.

Description

Technical Field

[0001] This invention belongs to the field of data storage, and specifically relates to a key-value separation LSM-tree garbage collection optimization method for ZNS SSDs, which is suitable for building high-performance and high-reliability storage systems. Background Technology

[0002] In recent years, with the continuous expansion of cloud computing, the amount of data generated has exploded. This massive amount of data places higher demands on the timeliness, reliability, and capacity of storage systems, and traditional storage systems are finding it increasingly difficult to meet these needs, gradually revealing many shortcomings.

[0003] At the hardware level, traditional hard disk drives (HDDs), due to their slow read / write speeds, high power consumption, and limited durability, are no longer adequate for the performance requirements of current storage systems and are gradually being replaced by solid-state drives (SSDs) with faster read / write speeds, smaller size, and shock resistance. However, SSDs rely on a flash translation layer (FTL) during read / write operations to map logical addresses to physical addresses and are responsible for critical functions such as wear leveling, garbage collection, and bad block management. The FTL requires expensive DRAM to maintain the address mapping table and needs to reserve additional flash memory space to support bad block replacement and garbage collection operations, increasing system complexity and manufacturing costs, as well as leading to higher static power consumption. Furthermore, because upper-layer applications are unaware of the garbage collection process within the SSD, the long-tail latency issue caused by the FTL can easily lead to system performance instability.

[0004] At the software level, traditional relational databases exhibit limitations such as performance bottlenecks and insufficient scalability when handling large-scale data, making them ill-suited to the demands of modern applications for efficient data access. In contrast, non-relational databases, with their flexible data models, high availability, and excellent scalability, are widely used in massive data scenarios. Key-value stores, as a common type, are particularly suitable for write-intensive scenarios. Index structures, such as log-structured merged trees (LSM-trees), cache random writes through a memory component (Memtable) and then sequentially write them to persistent storage in batches after sorting, thus exhibiting excellent write performance. However, as data continues to be written, the background merging operation of LSM-trees can cause significant write amplification. To address this, some research has proposed a key-value separation architecture for LSM-trees, separating large-size value data from the index to alleviate write amplification. However, this solution still suffers from read-back and write-back operations caused by garbage collection of value log files, which may interfere with foreground requests and lead to performance fluctuations.

[0005] In conclusion, existing storage technologies have significant shortcomings at both the hardware and software levels, necessitating the exploration of a novel storage architecture and data management approach to address the ever-increasing demands for data storage and processing. Summary of the Invention

[0006] To address the performance bottlenecks caused by severe write amplification and prominent long-tail latency in existing technologies, this invention proposes a key-value separation LSM-tree garbage collection optimization method for ZNS SSDs, including:

[0007] A key-value split LSM-tree is constructed, which includes a segment manager, a partition manager, and ZenFS. The storage space of ZNS SSD is statically divided into two independent regions, which are managed by the partition manager and ZenFS respectively. The value log managed by ZenFS is stripped and transferred to the partition manager for unified management. The partition manager is responsible for data placement, partition bitmap management, garbage collection, and crash recovery.

[0008] The partition manager divides each partition within its ZNS SSD independent area into multiple pages according to a preset size, and writes data in alignment with the page size; each partition maintains a data validity bitmap, and each bit in the data validity bitmap corresponds to each page of the partition, which is used to indicate the validity of the data stored on that page;

[0009] Furthermore, the partition manager reclassifies all partitions within the ZNS SSD's independent region into the following four categories: active partitions, full partitions, GC partitions, and free partitions. Free partitions are partitions that have not been written to, as well as full partitions that have been reset after garbage collection. Active partitions are partitions that are currently receiving data written in the foreground. Full partitions are partitions that have been written with sufficient data and are no longer receiving new data. GC partitions are partitions that receive valid data migration during background garbage collection.

[0010] The key-value separation LSM-tree uses a selective key-value separation method to divide the written key-value pairs, including: if the value of the written key-value pair is less than a preset key-value separation threshold, then the key-value pair is divided into a small value key-value pair; if the value of the written key-value pair is not less than the preset key-value separation threshold, then the key-value pair is divided into a large value key-value pair.

[0011] The beneficial effects of this invention are:

[0012] This invention divides the ZNS SSD space into two parts: one managed by ZenFS and the other by a partition manager, with no interference between the two management spaces. The space managed by the partition manager stores value log data, i.e., the WAL portion of the LSM-tree, reducing the total amount of data written to the system. By managing the value log files independently, combined with a key-value separation strategy, the amount of merged write data to the LSM-tree is effectively reduced, lowering system write amplification. The remapping mechanism decouples the direct mapping relationship between value indexes and value data in the key-value separated LSM-tree, so that value log garbage collection only needs to modify the mapping relationship without writing back valid data; at the same time, partition bitmap management avoids readback overhead, significantly improving the efficiency of data validity determination during value log garbage collection. The two work together to solve the common readback and writeback problems in key-value separation, reducing the interference of garbage collection on the LSM-tree foreground in key-value separation. Attached Figure Description

[0013] Figure 1 This is a diagram of the overall architecture of the present invention;

[0014] Figure 2 This is a schematic diagram of the data layout of the partition manager on the ZNS SSD in this invention;

[0015] Figure 3 This is a schematic diagram of the median log garbage collection process in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention provides a key-value separation LSM-tree garbage collection optimization method for ZNS SSDs, such as... Figure 1 As shown, a key-value split LSM-tree including a segment manager, a partition manager, and ZenFS is constructed. The storage space of the ZNS SSD is statically divided into two independent regions, which are managed by the partition manager and ZenFS respectively, without interfering with each other. The value log managed by ZenFS is stripped and transferred to the partition manager for unified management. The partition manager realizes data placement, partition bitmap management, garbage collection, and crash recovery.

[0018] In some embodiments, the key-value separation LSM-tree uses a selective key-value separation method to divide the written key-value pairs, including: if the value in the written key-value pair is less than a preset key-value separation threshold, then the key-value pair is divided into a small value key-value pair; if the value in the written key-value pair is not less than the preset key-value separation threshold, then the key-value pair is divided into a large value key-value pair.

[0019] Furthermore, for small key-value pairs, they are directly stored in the LSM-tree managed by the ZenFS file system. For large key-value pairs, they are directly stored in the value log managed by the partition manager, and then the keys and their data indexes in the large key-value pairs are stored in the LSM-tree. Specifically, the data index uses an address remapping mechanism, meaning the data index stored in the LSM-tree is a logical address, rather than the physical address directly stored as in traditional key-value separation architectures. This mechanism decouples the direct mapping relationship from value index to value in the key-value separation LSM-tree, effectively reducing the write-back overhead of data index modifications during garbage collection. The segment manager allocates logical addresses, and the partition manager manages physical addresses. The mapping relationship between logical and physical addresses is maintained by a mapping table residing in memory.

[0020] Furthermore, in traditional key-value separation architectures, regardless of the size of the key-value pair, all data must be written to the write-ahead log (WAL), and large key-value pairs also need to be written to the value log (vLog), leading to storage write amplification. To address the redundancy in write operations between WAL and vLog in key-value separation architectures, this invention integrates the WAL of large key-value pairs in the LSM-tree into the key-value separation's vLog. Under this mechanism, large key-value pairs are no longer written independently to the WAL, but directly to the vLog, which then handles their crash recovery. Specifically, because this invention employs a selective key-value separation method, performing key-value separation only on large key-value pairs, the partition manager is responsible for uniformly managing and maintaining their WAL and vLog when writing large key-value pairs, ensuring data consistency and durability. Small key-value pairs are still managed by the LSM-tree and their persistence is achieved by writing to the LSM-tree's write-ahead log. This design effectively reduces space fragmentation caused by page alignment of key-value pairs in partitions, improves batch processing efficiency when deleting small values, and accelerates the reclamation of invalid space, thereby significantly improving the overall storage efficiency and maintainability of the system.

[0021] Furthermore, this embodiment of the invention chooses to perform the key-value separation operation during the Flush phase, rather than when writing key-value pairs to the MemTable. Specifically, during the writing process to the MemTable, metadata (including logical addresses) of the key-value pairs is added, without physically separating the keys and values. This design is primarily based on the following considerations: the MemTable typically stores frequently accessed system data, and memory access speed is significantly faster than that of solid-state drives. Delaying key-value separation to the Flush phase ensures that query requests for frequently accessed data can still be responded to quickly in memory, thereby effectively reducing the access latency of foreground queries and improving system performance.

[0022] For example, the key-value splitting LSM-tree performs key-value pair write operations based on a selective key-value splitting method, including:

[0023] S11. Determine the log and MemTable write operations based on the type of the key-value pairs being written, including:

[0024] If the key-value pair being written is a small key-value pair, the small key-value pair is first written to the pre-write log, then persisted to ZNS SSD via ZenFS, and finally the un-key-value separation mark is saved to the value data of the small key-value pair and written to the MemTable of the LSM-tree.

[0025] If the key-value pair being written is a large key-value pair, the segment manager first allocates a logical address based on the range of the key in the large key-value pair. This logical address includes the logical segment number and the offset within the segment. Then, the partition manager further allocates a physical address for the large key-value pair based on the logical address allocated by the segment manager. This physical address includes the partition number and the offset within the partition. The logical address and physical address are then written to a memory mapping table. Next, the large key-value pair and its metadata are directly persisted to the ZNS SSD, and the corresponding bit in the data validity bitmap is simultaneously marked as valid. Finally, the metadata is saved to the value data of the large key-value pair and written to the MemTable of the LSM-tree. The metadata here includes the key-value separation flag after key-value separation of the large key-value pair, and the logical address corresponding to the value in the separated large key-value pair.

[0026] S12. When MemTable reaches a preset threshold, switch MemTable to Immutable MemTable;

[0027] S13. The data in the Immutable MemTable is flushed to the ZNS SSD via ZenFS to form the SSTable. For large value key-value pairs, true key-value separation is performed, and only the metadata of the large value key-value pairs is stored in the SSTable. For small value key-value pairs, key-value separation is not performed, and the value data of the small value key-value pairs is directly stored in the SSTable. At this time, the value data of the small value key-value pairs includes the unseparated key-value marker and the actual value data.

[0028] S14. When the total size of the SSTables at a certain level in the LSM-tree exceeds the maximum capacity threshold, a compaction operation is triggered.

[0029] In this embodiment of the invention, when the obsolete key-value pairs are stored using key-value separation, the corresponding physical address needs to be found through the mapping table based on the logical address. Then, the bitmap of the corresponding physical address is marked as invalid, and the relevant entries in the mapping table are cleared.

[0030] In this embodiment of the invention, when the front-end receives a query request, the system will perform different processing based on the storage location of the queried data. If the queried data is stored in a Memtable, the actual value is directly parsed from the value data and then returned to the front-end. If the queried data is stored in an SSTable, it is processed according to the data type. If it is key-value separated data, i.e., large value key-value pairs, the logical address is parsed from the value data, then the physical address corresponding to the logical address is obtained by combining it with the mapping table, and finally the actual value corresponding to the physical address is read and returned to the front-end. If it is non-key-value separated data, i.e., small value key-value pairs, the actual value is directly parsed from the value data and returned to the front-end.

[0031] In some embodiments, such as Figure 2 As shown, the partition manager divides each partition within the independent area of ​​the ZNS SSD under its jurisdiction into multiple pages according to a preset size, and writes data in alignment according to the page size; each partition maintains a data validity bitmap, and each bit in the data validity bitmap corresponds to each page of the partition, which is used to indicate the validity of the data stored on that page.

[0032] Preferably, during initialization, each bit in the data validity bitmap of each partition is marked as invalid; when data is written to a page in a partition, the bit corresponding to that page in the data validity bitmap maintained by that partition is marked as valid; when data is deleted from a page in a partition, the bit corresponding to that page in the data validity bitmap maintained by that partition is marked as invalid.

[0033] Furthermore, the partition manager reclassifies all partitions within the ZNS SSD's independent area into the following four categories: active partitions, full partitions, GC partitions (garbage collection partitions), and free partitions. Free partitions are partitions that have not been written to. In particular, full partitions that have been reset after garbage collection are also free partitions. Active partitions are partitions that are currently receiving data written in the foreground. Full partitions are partitions that have been written with sufficient data and are no longer receiving new data. GC partitions are partitions that receive valid data migration during background garbage collection.

[0034] In some embodiments, within the ZNS SSD independent area under the partition manager, the management of various partitions includes:

[0035] During a write operation, if the key-value pairs in the written data are large key-value pairs, the segment manager divides them into different logical segments according to the range of the keys in the large key-value pairs. Each logical segment corresponds to an active partition, and the partition manager maps the written data to the corresponding active partition according to the logical segment. In addition, each logical segment is allocated a corresponding GC partition.

[0036] When an active partition is full or the remaining space is less than the size of the data to be written, the active partition becomes a full partition; at the same time, the GC partition of the corresponding logical segment is allocated as the new active partition. If there is no available GC partition in the corresponding logical segment, a free partition is allocated as the active partition.

[0037] When the proportion of invalid data generated by LSM-tree Compaction within a partition is greater than or equal to the preset garbage collection threshold, garbage collection is triggered.

[0038] During garbage collection, valid data in a full partition is migrated to the GC partition corresponding to its logical segment. If no GC partition is available, a free partition is allocated as the GC partition.

[0039] After garbage collection is complete, the filled partition is reset and becomes a free partition.

[0040] By dividing logical segments by key range and mapping the corresponding logical segment data to the corresponding partition, cross-partition access during range queries can be effectively reduced, thereby significantly improving query efficiency.

[0041] In some embodiments, such as Figure 2 As shown, within the storage area managed by the partition manager, the data layout within the partition is arranged in ascending order of key-value pairs. Each key-value pair consists of three parts: a header area, a data area, and a padding area. The header area includes:

[0042] Meta-information redundancy check code (CRC) is used to verify the integrity of key-value pairs of data to detect whether errors occur during data writing or migration.

[0043] Data length (Len) indicates the storage length of the current key-value pair in the partition, facilitating data retrieval during sequential scans and garbage collection;

[0044] The global sequence number (Seq), uniformly assigned by the LSM-tree, is used to represent the version order of key-value pairs and to determine the relationship between old and new data during merging, deletion, and crash recovery.

[0045] Logical address (LBA), assigned by the segment manager, consists of a logical segment number and an offset within the segment, and is used to restore the mapping table during crash recovery;

[0046] Operation type (Type) is used to distinguish whether the key-value pair is an insertion or deletion operation.

[0047] In addition, the data area stores the specific key-value pair content, that is...<Key-Size, Key, Value-Size, Value> The padding area ensures that written data is page-aligned. This design facilitates effective page-level management via bitmaps and also meets the page-alignment requirements of the underlying storage hardware.

[0048] In some embodiments, a background thread is awakened after compaction and executes a detection task. In each detection task, it iterates through all filled partitions and calculates the percentage of invalid data. When the percentage of invalid data in a filled partition is greater than or equal to a preset garbage collection threshold, it is recorded as a candidate partition. If multiple candidate partitions exist, the candidate partition with the largest amount of invalid data is selected for garbage collection to improve the space reclamation efficiency. If there is only one candidate partition, it is directly garbage collected. If no candidate partition exists, no garbage collection operation is triggered in this detection task. Figure 3 As shown, the process of performing garbage collection on candidate partitions includes:

[0049] S21. Sequentially scan the data validity bitmap maintained by the candidate partition. If the currently scanned bit is marked as valid, proceed to step S22; otherwise, skip and continue scanning the next bit.

[0050] S22. Read the Header information of the data on the page corresponding to the current bit, obtain the data length and logical address, then read the Data information according to the data length, migrate the Data information and Header information to the GC partition, and mark the corresponding position of the GC partition data validity bitmap as valid; finally, look up the mapping entry corresponding to the logical address in the mapping table, update the physical address in the mapping entry to the new physical address after migration, complete the address mapping relationship change of the value data, and ensure that subsequent access can correctly locate the latest data position;

[0051] S23. After all valid data in the candidate partition has been migrated, the candidate partition performs a reset operation to clear all data and becomes a free partition so that it can be reallocated for new write requests.

[0052] This example demonstrates how to remap value data during garbage collection by updating the mapping between logical and physical addresses in the mapping table. This allows for updating the value read path without modifying the addresses pointed to by keys in the LSM-tree, avoiding the interference of write-back during garbage collection on normal foreground writes in key-value LSM-tree architectures. Simultaneously, the system can quickly determine data validity by scanning the data bitmap corresponding to the partition, avoiding the read-back overhead of data validity determination in key-value LSM-tree garbage collection and eliminating potential data inconsistency issues during garbage collection in traditional key-value architectures.

[0053] In some embodiments, after an unexpected system crash, key-value data managed by ZenFS, i.e., small key-value pairs that are not separated from their keys, can be recovered using logs recorded in the pre-write log; while large key-value pairs managed by the partition manager, as well as the in-memory mapping table and data validity bitmap, need to be recovered by the partition manager when the system restarts.

[0054] Furthermore, the system crash recovery process includes:

[0055] Mapping relationship recovery: Since each data value carries a corresponding logical address in its header area when it is written to the persistent device, the header information of all data on all pages of the partition is scanned sequentially to obtain the logical address, and the logical address and its physical address are rewritten into the memory mapping table.

[0056] Memory Loss Data Recovery: During each flush operation, the system records the maximum global sequence number in the current Immutable MemTable to the LSM-tree's metadata Manifest file. This global sequence number represents the key-value state that was successfully persisted before the system crash. During system restart and recovery, it is only necessary to iterate through all key-value pairs in each partition of the ZNS SSD, filter out key-value pairs with a global sequence number greater than the maximum global sequence number recorded in the Manifest file, and rewrite them to the MemTable. This restores the memory component to its pre-crash state, ensuring no data loss.

[0057] Data Validity Bitmap Recovery: The system verifies the validity of values ​​in the value log by checking the LSM-tree and reconstructs the corresponding partition bitmap accordingly. Although this process involves reading the LSM-tree, it does not impact foreground performance because it is executed during system initialization before foreground request processing begins. To further reduce recovery latency, the system employs an asynchronous parallel execution mechanism to concurrently rebuild and verify the bitmaps of multiple partitions, thereby significantly reducing system restart time.

[0058] Partition type recovery: If the partition status is Free, it is determined to be a free partition; if the status is Full, it is determined to be a full partition; if the partition status is Active, it is first determined to be an active partition, and the logical address of its first data is read to determine the corresponding logical segment. If the same logical segment corresponds to two active partitions, the global sequence number of the first data of the two active partitions is read respectively. The active partition with the smaller global sequence number is the GC partition corresponding to the logical segment, and the active partition with the larger global sequence number is the real active partition of the logical segment.

[0059] In summary, the key-value split LSM-tree garbage collection method for ZNS SSDs significantly improves performance, reduces tail latency, and minimizes write amplification through a series of optimization measures. First, this invention employs a key-value split LSM-tree index structure, which exhibits significantly lower write amplification than traditional LSM-trees under update-intensive loads. Specifically, this invention integrates the LSM-tree's pre-write log into the value log, avoiding redundant log data writing; and a dedicated crash recovery mechanism enables the system to quickly recover to its pre-crash state, ensuring data consistency and high system availability. Second, this invention uses a remapping mechanism to avoid the readback overhead of data validity checks during value log garbage collection and a bitmap mechanism to avoid the writeback overhead of updating the value addresses of valid data during value log garbage collection, reducing interference with normal foreground read / write operations and achieving overall system performance improvement.

[0060] This invention employs a DRAM-ZNS SSD storage architecture, storing key data structures such as mapping tables and bitmaps in DRAM for efficient read and write operations, while the actual data and related metadata are stored on the persistent ZNS SSD. This invention borrows the address remapping mechanism design concept from FTL and optimizes key-value separation garbage collection using a bitmap mechanism, avoiding read-back and write-back overhead and significantly reducing system read / write amplification. Simultaneously, it integrates the pre-write log from LSM-tree into the key-value separation value log, ensuring system performance and reliability while reducing data redundancy and the total amount of data written to disk.

[0061] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A key-value separation LSM-tree garbage collection optimization method for ZNS SSDs, characterized in that, A key-value split LSM-tree is constructed, which includes a segment manager, a partition manager, and ZenFS. The storage space of ZNS SSD is statically divided into two independent regions, which are managed by the partition manager and ZenFS respectively. The value log managed by ZenFS is stripped and transferred to the partition manager for unified management. The partition manager is responsible for data placement, partition bitmap management, garbage collection, and crash recovery. The partition manager divides each partition within its ZNS SSD independent area into multiple pages according to a preset size, and writes data in alignment with the page size; each partition maintains a data validity bitmap, and each bit in the data validity bitmap corresponds to each page of the partition, which is used to indicate the validity of the data stored on that page; Furthermore, the partition manager reclassifies all partitions within the ZNS SSD's independent region into the following four categories: active partitions, full partitions, GC partitions, and free partitions. Free partitions are partitions that have not been written to, as well as partitions that have been reset from full partitions after garbage collection. Active partitions are partitions that are currently receiving data written in the foreground. Full partitions are partitions that have been written with sufficient data and are no longer receiving new data. GC partitions are partitions that receive valid data migration during background garbage collection. The key-value separation LSM-tree uses a selective key-value separation method to divide the written key-value pairs, including: if the value of the written key-value pair is less than a preset key-value separation threshold, then the key-value pair is divided into a small value key-value pair; if the value of the written key-value pair is not less than the preset key-value separation threshold, then the key-value pair is divided into a large value key-value pair.

2. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, Within the ZNS SSD's independent area managed by the partition manager, the management of various partitions includes: During a write operation, if the key-value pairs in the written data are large key-value pairs, the segment manager divides them into different logical segments according to the range of the keys in the large key-value pairs. Each logical segment corresponds to an active partition. The partition manager maps the written data to the corresponding active partition according to the logical segments. When an active partition is full or the remaining space is less than the size of the data to be written, the active partition becomes a full partition; at the same time, the GC partition of the corresponding logical segment is allocated as the new active partition. If there is no available GC partition in the corresponding logical segment, a free partition is allocated as the active partition. When the proportion of invalid data generated by LSM-tree Compaction within a partition is greater than or equal to the preset garbage collection threshold, garbage collection is triggered. During garbage collection, valid data in a full partition is migrated to the GC partition corresponding to its logical segment. If no GC partition is available, a free partition is allocated as the GC partition. After garbage collection is complete, the filled partition is reset and becomes a free partition.

3. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, During initialization, each bit in the data validity bitmap of each partition is marked as invalid. When data is written to a page on a partition, the bit corresponding to that page in the data validity bitmap maintained by that partition is marked as valid. When data is deleted from a page on a partition, the bit corresponding to that page in the data validity bitmap maintained by that partition is marked as invalid.

4. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, The key-value splitting LSM-tree performs key-value pair write operations based on a selective key-value splitting method, including: S11. Determine the log and MemTable write operations based on the type of the key-value pairs being written, including: If the key-value pair being written is a small key-value pair, the small key-value pair is first written to the pre-write log, then persisted to ZNS SSD via ZenFS, and finally the un-key-value separation mark is saved to the value data of the small key-value pair and written to the MemTable of the LSM-tree. If the key-value pair being written is a large key-value pair, the segment manager first allocates a logical address based on the range of the key in the large key-value pair. This logical address includes the logical segment number and the offset within the segment. Then, the partition manager further allocates a physical address for the large key-value pair based on the logical address allocated by the segment manager. This physical address includes the partition number and the offset within the partition. The logical address and physical address are then written to a memory mapping table. Next, the large key-value pair and its metadata are directly persisted to the ZNS SSD, and the corresponding bit in the data validity bitmap is simultaneously marked as valid. Finally, the metadata is saved to the value data of the large key-value pair and written to the MemTable of the LSM-tree. This metadata includes a key-value separation flag and a logical address. S12. When MemTable reaches a preset threshold, switch MemTable to Immutable MemTable; S13. The data in the Immutable MemTable is flushed into the ZNS SSD via ZenFS to form the SSTable; among them, the large value key-value pairs are truly key-value separated, and only the metadata of the large value key-value pairs is stored in the SSTable; for the small value key-value pairs, their value data is directly stored in the SSTable. S14. When the total size of the SSTables at a certain level in the LSM-tree exceeds the maximum capacity threshold, a compaction operation is triggered.

5. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, The background thread is awakened after the compaction is completed and executes the detection task. In each detection task, it traverses all full partitions and calculates the proportion of invalid data. When the proportion of invalid data in a full partition is greater than or equal to the preset garbage collection threshold, it is recorded as a candidate partition. If there are multiple candidate partitions, the candidate partition with the largest amount of invalid data is selected for garbage collection. If there is only one candidate partition, it is directly garbage collected. If there are no candidate partitions, garbage collection is not triggered in this detection task. The process of performing garbage collection on candidate partitions includes: S21. Sequentially scan the data validity bitmap maintained by the candidate partition. If the currently scanned bit is marked as valid, proceed to step S22; otherwise, skip and continue scanning the next bit. S22. Read the Header information of the data on the page corresponding to the current bit, obtain the data length and logical address, then read the Data information according to the data length, and migrate the Data information and Header information to the GC partition; finally, find the mapping entry corresponding to the logical address in the mapping table, and update the physical address in the mapping entry to the new physical address after migration. S23. After all valid data in the candidate partition has been migrated, the candidate partition performs a reset operation to clear all data and becomes a free partition.

6. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, The crash recovery process includes: Mapping relationship recovery: Sequentially scan the header information of all pages in the partition to obtain the logical address, and rewrite the logical address and its physical address into the memory mapping table; Data recovery from memory loss: Traverse all key-value pairs in each partition of the ZNS SSD, filter out key-value pairs with a global serial number greater than the maximum global serial number recorded in the Manifest file, and rewrite them to the Memtable; Data validity bitmap recovery: Look back at the LSM-tree to verify whether the value data in the value log is valid, and then reconstruct the data validity bitmap accordingly; Partition type recovery: If the partition status is free, it is determined to be a free partition; if the status is full, it is determined to be a full partition; if the partition status is active, it is first determined to be an active partition, and the logical address of its first data is read to determine the corresponding logical segment. If the same logical segment corresponds to two active partitions, the global sequence number of the first data of the two active partitions is read respectively, and the active partition with the smaller global sequence number is the GC partition corresponding to the logical segment.

7. The key-value separation LSM-tree garbage collection optimization method for ZNS SSDs according to claim 1, characterized in that, Within the storage area managed by the partition manager, the data layout within the partition is arranged in the order of large key-value pairs. Each large key-value pair consists of three parts: a header area, a data area, and a padding area. The header area includes metadata redundancy check code, data length, global sequence number, logical address, and operation type.

Images