Cold and hot hierarchical storage and SLC bypass management method based on storage class memory SSD and ZNS SSD

Abstract

The invention relates to the technical field of data storage, and provides a cold and hot hierarchical storage and SLC bypass management method based on a storage class memory SSD and a ZNS SSD, which comprises the following steps: step 1, system initialization and heterogeneous hierarchical mapping; 2, request receiving and read-write shunting are carried out; when an application layer initiates a key value operation request, the system firstly judges the type of the request: if the request is a write request, entering step 3; if yes, entering step 7; step 3, writing and solidifying data; 4, temperature judgment and SLC bypass judgment are carried out; step 5, overlap detection and target level judgment; step 6, executing heterogeneous medium writing; step 7, when a read request is processed, performing cascading search according to medium performance from high level to low level, and realizing read optimization of physical isolation; and step 8, ending: returning an operation result to the application layer. According to the invention, cold and hot hierarchical storage and SLC bypass management can be well carried out.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and more specifically, to a method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSDs and ZNSSSDs. Background Technology

[0002] With the explosion of data-intensive applications, storage engines based on LSM-Tree (Log Structure Merge Tree), such as RocksDB, have been widely adopted. ZNS (Zoned Namespace) SSDs have gained attention for their ability to eliminate write amplification caused by internal garbage collection (GC). However, the existing "RocksDB + ZNS" architecture still faces the following challenges: 1. Lack of full utilization of media differences in read paths: Mixing hot and cold data in QLC (four-level cell) results in significant read latency.

[0003] II. SLC Cache Bottleneck and Wear: Existing tiered solutions typically flush data blindly from memory into the SLC (Single-Level Cell) region before moving it down to the QLC. When handling high-frequency compaction at the L0 (Level-0) / L1 (Level-1) levels, the SLC faces enormous write pressure, easily becoming a performance bottleneck and rapidly depleting its erase / write lifespan.

[0004] 3. Severe Read / Write Interference: Although the data volume of SSTable files at the L0 and L1 levels is not large, their I / O activity is extremely intense due to the overlapping key range (L0) and frequent merge sorting, which seriously interferes with the reading performance of the underlying data and leads to an increase in long-tail latency.

[0005] 4. Cold data contamination of SLC: Not all sinking data is hot data. Forcing cold data to be written to SLC not only wastes valuable SLC space, but also increases the additional write amplification when migrating from SLC to QLC. Summary of the Invention

[0006] The present invention provides a method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSDs and ZNS SSDs. This method can absorb extremely hot data flow by solidifying the L0 / L1 levels of the LSM-Tree onto the storage-class memory SSD medium and utilizing its low latency characteristics. At the same time, an SLC bypass strategy is designed so that cold data can directly skip the SLC and enter the QLC during the compaction process, thereby improving system read and write performance, reducing SLC wear, and extending device life.

[0007] A method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSDs and ZNS SSDs according to the present invention includes the following steps: Step 1: System initialization and heterogeneous hierarchy mapping; Load the RocksDB storage engine configuration, initialize the ZenFS file system to recognize the partition characteristics of the ZNS SSD, and establish the following physical isolation hierarchical mapping rules: 1.1) Extremely hot layer mapping: Force the file paths of Level-0 and Level-1 layers of the log structure merge tree LSM-Tree to point to the mount point of the storage-class memory SSD; 1.2) Thermal layer mapping: Mapping the file storage target of Level-2 layer to the SLC cache area of ​​the partition namespace solid-state drive ZNSSSD; 1.3) Cold data layer mapping: Mapping file storage targets at Level-3 and below to the QLC main memory area of ​​ZNS SSD; Step 2: Receiving requests and separating read / write operations; When the application layer initiates a key-value operation request, the system first determines the request type: if it is a write request, proceed to step 3; if it is a read request, proceed to step 7. Step 3: Data writing and solidification; Step 4: Temperature determination and SLC bypass determination; When the amount of data in the Level-1 layer reaches a threshold and needs to be moved down, bypass decision logic is executed, including: 4.1) Obtain file age: Read the metadata of the SSTable file to be compressed, obtain the creation timestamp, and calculate the data cooling rate. ; 4.2) Comparison of hot and cold thresholds: If If the data exceeds the preset cold data threshold, or if the file is marked as originating from a large batch of imported data, it is considered cold data; otherwise, it is considered hot data. Step 5: Overlap detection and target hierarchy determination; 5.1) If the data to be compressed in Level-1 is determined to be hot data, or if it is cold data but there are files with overlapping key value ranges in Level-2, then follow the mapping rules in step 1 and set the compression target level to Level-2. 5.2) If the data to be compressed in Level-1 is determined to be cold data and there are no files with overlapping key value ranges in Level-2, then the SLC bypass strategy is activated to redirect the compression target level to Level-3. Step 6: Perform heterogeneous media writing; The compression thread performs the write operation based on the determined target level, and RocksDB sends write requests with different lifecycle hints to the underlying ZenFS. Step 7: When processing read requests, perform a cascading search based on media performance from high to low to achieve read optimization with physical isolation. Step 8, End: Return the operation result to the application layer.

[0008] Preferably, in step 1, the storage-class memory SSD is an Optane SSD.

[0009] Preferably, step 3 includes the following steps: 3.1) The data is first written to the RocksDB's in-memory table; 3.2) When the memory table is full, a disk flushing operation is triggered to generate a sorted string table SSTable file, which is then physically written to the Optane SSD; 3.3) When the number of files in Level-0 reaches the threshold, compression is triggered, and the merging and sorting of Level-0 to Level-1 layers is completed within the Optane SSD; As a preferred option, in 4.1), The calculation formula is: .

[0010] As a preferred option, step 6 specifically involves: 6.1) Writing to SLC Zone: If the target is Level-2, RocksDB sends a short lifecycle hint, and ZenFS writes it to the SLC cache of ZNS SSD; 6.2) Writing to QLC Zone: If the target is Level-3, RocksDB sends an extremely long lifecycle hint, and ZenFS writes it directly to the QLC main memory area of ​​the ZNS SSD.

[0011] Preferably, step 7 includes: Level 1 query: Query memory; Second-level query: If no match is found, query Level-0 and Level-1 levels; Level 3 query: If no match is found, query Level-2. Level 4 query: If no match is found, query Level-3 and below.

[0012] Preferably, the method further includes passive migration from SLC to QLC: Level-2 data that has been written to the SLC cache will only be migrated to the QLC main memory area when the SLC space is insufficient or after the next compression and cooling.

[0013] The beneficial effects of this invention are as follows: a) Significantly improved read performance: Optane SSDs utilize microsecond-level latency to handle L0 / L1 lookups, greatly reducing average read latency; at the same time, Optane SSDs offload frequent write jitter, reducing the overall P99 long-tail latency of the system.

[0014] b) Extend SLC device lifespan: By using the SLC bypass strategy, cold data is prevented from invalidally occupying the SLC erase / write cycle (P / ECycle), significantly reducing the amount of SLC written.

[0015] c) Reduce system write amplification: Cold data falls directly into the QLC, eliminating the invalid dwell time and secondary migration at the SLC layer in the traditional path.

[0016] d) Highly flexible architecture: The heterogeneous zone management based on heat and hierarchy can adaptively adjust the cache tiering strategy according to load changes, making it suitable for various business scenarios; e) Easy to integrate: This invention can be directly deployed on the RocksDB + ZenFS + ZNS SSD architecture, and is compatible with existing FEMU emulation platforms and standard interfaces such as Simple Copy (hardware-level data copy command under the NVMe protocol). Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSDs and ZNS SSDs, as described in this embodiment. Detailed Implementation

[0018] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0019] Example like Figure 1 As shown, this embodiment provides a method for tiered cold and hot storage and SLC bypass management based on storage-class memory SSDs and ZNS SSDs, which includes the following steps: Step 1: System initialization and heterogeneous hierarchy mapping; Load the RocksDB storage engine configuration, initialize the ZenFS file system to recognize the partition characteristics of the ZNS SSD, and establish the following physical isolation hierarchical mapping rules to guide the subsequent data persistence location: 1.1) Tier-0 mapping: Forces the file paths of Level-0 (level 0) and Level-1 (level 1) of the log structure merge tree LSM-Tree to point to the mount point of the storage-class memory SSD (Optane SSD); 1.2) Tier-1 mapping: Maps the file storage target of Level-2 (second layer) to the SLC cache zone of the ZNS SSD partition namespace. 1.3) Cold data layer mapping (Tier-2): Maps the file storage targets of Level-3 (the third layer) and below (Level-4, etc.) to the QLC main memory area of ​​ZNS SSD.

[0020] This embodiment is built on a heterogeneous storage system, which includes a host (running the RocksDB storage engine and the ZenFS file system) and two types of storage devices: The first type of non-volatile storage device uses low-latency, high-write-resistance storage-class memory solid-state drives (such as Optane SSDs) to store extremely hot data.

[0021] The second type of non-volatile storage device is a large-capacity zoned namespace solid-state drive (ZNS SSD). Its internal logical partitioning is divided into an SLC cache area (simulating single-cell flash memory performance for hot data) and a QLC main memory area (simulating four-cell flash memory performance for cold data).

[0022] Step 2: Receiving requests and separating read / write operations; When the application layer initiates a key-value operation request, the system first determines the request type: if it is a write request, proceed to step 3; if it is a read request, proceed to step 7.

[0023] Step 3: Data writing and solidification; Includes the following steps: 3.1) The data is first written to the RocksDB's in-memory table (MemTable); 3.2) When the memory table is full, a flush operation is triggered to generate a sorted string table (SSTable) file, which is then physically written to the Optane SSD. 3.3) When the number of files in Level-0 reaches a threshold, compression is triggered, and the merge sorting from Level-0 to Level-1 is completed within the Optane SSD. This process utilizes Optane's low latency characteristics to absorb the most frequent write jitter in the LSM-Tree.

[0024] Step 4: Temperature determination and SLC bypass determination; When the amount of data in the Level-1 layer reaches a threshold and needs to be moved down, bypass decision logic is executed, including: 4.1) Obtain file age: Read the metadata of the SSTable file to be compressed, obtain the creation timestamp, and calculate the data cooling rate. ; The calculation formula is: .

[0025] 4.2) Comparison of hot and cold thresholds: If If the data exceeds a preset cold data threshold (e.g., 3600 seconds, which is configurable), or if the file is marked as originating from a bulk data import (Ingest), it is considered cold data; otherwise, it is considered hot data.

[0026] Step 5: Overlap Detection and Target Level Determination; Based on the temperature determination results from Step 4 and the key value overlap between levels, the system further combines the structural characteristics of the LSM-Tree to make dynamic routing decisions for the data sinking path: 5.1) Path A (Standard Downward Path): If the data to be compressed in Level-1 is determined to be hot data, or if it is cold data but there are files with overlapping key ranges in Level-2, then follow the mapping rules in step 1 and set the compression target level to Level-2. 5.2) Path B (SLC Bypass): If the data to be compressed in Level-1 is determined to be cold data and there are no files with overlapping key-value ranges in Level-2, the SLC bypass strategy is activated, and the compression target level is redirected to Level-3.

[0027] Step 6: Perform heterogeneous media writing; The compression thread performs writes based on the determined target level. RocksDB sends write requests with different lifecycle hints to the underlying ZenFS: 6.1) Writing to SLC Zone: If the target is Level-2, RocksDB sends a short lifecycle hint, and ZenFS writes it to the SLC cache of ZNS SSD; this area has good performance and is suitable for handling hot data.

[0028] 6.2) Writing to QLC Zone: If the target is Level-3, RocksDB sends an extremely long lifecycle warning, and ZenFS writes it directly to the QLC main memory of the ZNS SSD. This avoids invalid writes of cold data to the SLC layer and reduces write amplification.

[0029] Passive migration from SLC to QLC: For Level-2 data already written to the SLC cache, it will only be migrated to the QLC main memory (Level-3+) when the SLC space is insufficient or after the next compression and the data becomes cold. This means that the SLC area always acts as a high-speed buffer between Optane and QLC, serving only truly warm data.

[0030] Step 7: When processing read requests, perform a cascading search based on media performance from high to low to optimize read performance through physical isolation. Level 1 query: Query memory (MemTable / BlockCache); Second-level query (Optane SSD): If no match is found, query Level-0 and Level-1 layers; since Optane carries the latest and most frequently accessed data, most read requests return at this layer, achieving microsecond-level latency; Level 3 query (SLC Zone): If no match is found, query Level-2; Level 4 query (QLC Zone): If no match is found, query Level-3 and below.

[0031] Step 8, End: Return the operation result to the application layer.

[0032] This embodiment presents a cold / hot tiered storage system based on storage-class memory SSDs and ZNS SSDs. Through physically isolated storage tier mapping and a temperature-sensing bypass compression mechanism, it effectively solves the problems of rapid SLC wear and severe read / write interference in traditional LSM-Tree architectures on ZNS SSDs. This solution simulates the low-latency characteristics of Optane SSDs to absorb high-frequency compression jitter, and significantly reduces invalid writes to the SLC region through a cold data bypass (SLC Bypass) strategy. While significantly improving the overall system read / write throughput and response speed, it effectively extends the lifespan of critical ZNS SSD components, demonstrating significant novelty, inventiveness, and broad industrial application value.

[0033] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for tiered cold and hot storage and SLC bypass management based on storage-class memory SSDs and ZNS SSDs, characterized in that: Includes the following steps: Step 1: System initialization and heterogeneous hierarchy mapping; Load the RocksDB storage engine configuration, initialize the ZenFS file system to recognize the partition characteristics of the ZNS SSD, and establish the following physical isolation hierarchical mapping rules: 1.1) Extremely hot layer mapping: Force the file paths of Level-0 and Level-1 layers of the log structure merge tree LSM-Tree to point to the mount point of the storage-class memory SSD; 1.2) Thermal layer mapping: Mapping the file storage target of Level-2 layer to the SLC cache area of ​​the ZNS SSD partition namespace; 1.3) Cold data layer mapping: Mapping file storage targets at Level-3 and below to the QLC main memory area of ​​ZNS SSD; Step 2: Receiving requests and separating read / write operations; When the application layer initiates a key-value operation request, the system first determines the request type: if it is a write request, proceed to step 3; if it is a read request, proceed to step 7. Step 3: Data writing and solidification; Step 4: Temperature determination and SLC bypass determination; When the amount of data in the Level-1 layer reaches a threshold and needs to be moved down, bypass decision logic is executed, including: 4.1) Obtain file age: Read the metadata of the SSTable file to be compressed, obtain the creation timestamp, and calculate the data cooling rate. ; 4.2) Comparison of hot and cold thresholds: If If the data exceeds the preset cold data threshold, or if the file is marked as originating from a large batch of imported data, it is considered cold data; otherwise, it is considered hot data. Step 5: Overlap detection and target hierarchy determination; 5.1) If the data to be compressed in Level-1 is determined to be hot data, or if it is cold data but there are files with overlapping key value ranges in Level-2, then follow the mapping rules in step 1 and set the compression target level to Level-2. 5.2) If the data to be compressed in Level-1 is determined to be cold data and there are no files with overlapping key value ranges in Level-2, then the SLC bypass strategy is activated to redirect the compression target level to Level-3. Step 6: Perform heterogeneous media writing; The compression thread performs the write operation based on the determined target level, and RocksDB sends write requests with different lifecycle hints to the underlying ZenFS. Step 7: When processing read requests, perform a cascading search based on media performance from high to low to achieve read optimization with physical isolation. Step 8, End: Return the operation result to the application layer.

2. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 1, characterized in that: In step 1, the storage-class memory SSD is an Optane SSD.

3. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 2, characterized in that: Step 3 includes the following steps: 3.1) The data is first written to the RocksDB's in-memory table; 3.2) When the memory table is full, a disk flushing operation is triggered to generate a sorted string table SSTable file, which is then physically written to the Optane SSD; 3.3) When the number of files in the Level-0 layer reaches the threshold, compression is triggered, and the merge sorting from the Level-0 layer to the Level-1 layer is completed inside the Optane SSD.

4. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 3, characterized in that: In 4.1), The calculation formula is: .

5. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 4, characterized in that: Step 6 specifically involves: 6.1) Writing to SLC Zone: If the target is Level-2, RocksDB sends a short-lifetime hint, and ZenFS writes it to the SLC cache of the ZNS SSD; 6.2) Writing to QLC Zone: If the target is Level-3, RocksDB sends an extremely long lifecycle hint, and ZenFS writes it directly to the QLC main memory area of ​​the ZNS SSD.

6. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 5, characterized in that: Step 7 includes: Level 1 query: Query memory; Second-level query: If no match is found, query Level-0 and Level-1 levels; Level 3 query: If no match is found, query Level-2. Level 4 query: If no match is found, query Level-3 and below.

7. The method for cold and hot tiered storage and SLC bypass management based on storage-class memory SSD and ZNS SSD according to claim 6, characterized in that: The method also includes passive migration from SLC to QLC: Level-2 data that has been written to the SLC cache will only be migrated to the QLC main memory area when the SLC space is insufficient or after the next compression and cooling.

Images