ZNS storage method and system based on SSTable fine-grained popularity perception and cross-regional dynamic wear leveling

Abstract

The invention relates to the technical field of storage systems, in particular to a ZNS storage method and system based on SSTable fine-grained heat perception and cross-regional dynamic wear leveling, and the method comprises the following steps: step 1, initializing a ZNS storage environment and constructing a global resource pool; step 2, writing request initiation and data popularity classification; step 3, performing Zone allocation based on types; 4, physical writing and metadata unified updating are carried out; and 5, garbage collection GC and wear balance circulation. According to the method, the problems of SSTable heat deflection, Zone stiffness and unbalanced wear in a ZenFS scene can be effectively solved.

Description

Technical Field

[0001] This invention relates to the field of storage system technology, and more specifically, to a ZNS storage method and system based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling. Background Technology

[0002] With the development of flash memory technology, ZNS (Zoned Namespace) SSDs have attracted much attention because they eliminate the garbage collection overhead and write amplification of traditional FTL (Flash Translation Layer). ZenFS, as middleware connecting RocksDB (a key-value store based on an LSM-Tree log structure merged tree structure) and ZNS SSDs, is responsible for mapping SSTable (Sorted String Table) files to physical Zones (partitions). However, the existing ZenFS architecture faces the following serious challenges when handling real-world workloads: 1) SSTable Internal Hotness Skew and Long-Tail Deadness Issues: Existing ZenFS typically assumes that data at the same LSM-Tree level has a similar lifecycle, thus writing the entire SSTable sequentially to the same Zone. However, due to key space locality, the same SSTable often contains a mixture of extremely hot data (frequently updated) and cold data (long-term unchanging). When 90% of the hot data in a Zone becomes invalid, the remaining 10% of cold data (long-tail data) remains valid, causing the Zone to be unable to be reset, becoming a "dead Zone".

[0003] 2) Write amplification caused by garbage collection: In order to reclaim the aforementioned "zombie zones," the system is forced to perform garbage collection (GC), migrating still valid cold data to new zones. This not only causes additional write amplification but also consumes valuable I / O bandwidth.

[0004] 3) Severe uneven wear: a) Localized wear caused by hot and cold partitioning: If hot and cold data are simply separated, the zone storing hot data will be frequently erased, while the zone storing cold data will be almost never erased. Over time, the hot zone will reach its lifespan limit first, rendering the entire disk unusable.

[0005] b) Wear and tear caused by zone role differences: In ZenFS, the Journal Zone primarily records metadata and WAL (Write-Ahead Log), and its write and erase frequency is much higher than that of the ordinary Data Zone. The existing static allocation mechanism can lead to premature wear and tear of the Journal Zone.

[0006] In summary, there is an urgent need for a ZNS storage method and system based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling to solve the above problems. Summary of the Invention

[0007] The present invention provides a ZNS storage method and system based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling, which can overcome some or more defects of the prior art.

[0008] A ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling according to the present invention includes the following steps: Step 1: Initialize the ZNS storage environment and build the global resource pool; Scan all physical partitions (Zones) of the ZNS SSD device, read the write / erase cycle count (P / E Cycle) of each Zone, and then construct a global free Zone pool sorted from low to high based on the P / E Cycle value. Logically, the resource pool is divided into three dynamic candidate intervals: Younger age range: The head region with the lowest P / E value; Healthy range: The P / E value is in the middle range; The aging region: the tail region with the highest P / E value; Step 2: Write the request initiation and fine-grained heat classification of SSTable data; When the upper-layer LSM-Tree storage engine initiates a write or compression request, the SSTable data enters the heat classifier. By analyzing the update frequency of the key, historical access count, or timestamp, the data block to be written is classified into fine-grained heat based on the extent granularity and marked as a hot data stream or a cold data stream. Step 3: Zone allocation based on type; After receiving the request, the Zone allocator executes a counterintuitive wear leveling strategy to allocate Zones based on the data classification results obtained from S2 and the specific attributes of the request: Path A: If the request type is log, the Zone allocator selects a Zone from the young state range of the global resource pool; Path B: If the request type is hot data, the Zone allocator selects a Zone from the healthy range; Path C: If the request type is cold data, the Zone allocator selects a Zone from the old state range; Step 4: Physical write and unified metadata update; Based on the allocation results, the data is written to the physical layer's log zone, hot zone, or cold zone, and the file system's metadata and mappings are updated. Step 5: Garbage Collection (GC) and Wear Leveling Cycle; The system runs a GC scanning thread in the background, which periodically maintains the Zones that are full. The thread scans closed Zones and identifies zombie Zones that have a valid data rate below the threshold and a long residence time. For zombie Zones, it triggers downgraded garbage collection and extracts the remaining valid data. Then, it resets the original Zone and returns it to the global resource pool.

[0009] As a preferred option, in step 2, the popularity classification is specifically as follows: If the determination result is high-frequency updated data, the data stream is marked as a hot data stream; If the determination result is low-frequency or static data, the data stream is marked as a cold data stream.

[0010] Preferably, in step 2, the marked data stream initiates a space request to the Zone allocator.

[0011] Preferably, in step 4, when updating the file system metadata and mapping, for SSTable files, their physical address mapping is recorded; a logical SSTable file may include hot data segments stored in hot zones and cold data segments stored in cold zones, achieving physical isolation within the file.

[0012] As a preferred option, in step 5, when determining a dead zone, the system calculates the effective data rate and average data dwell time of the zone. If the effective rate is less than 5% and the dwell time is long, it is determined to be a dead zone; if the conditions are not met, it is processed according to normal logic or ignored.

[0013] Preferably, in step 5, when extracting the remaining valid data, the data is forcibly marked as cold data and flowed back to the Zone allocator in step 3, where it is rewritten into the cold Zone of the old state region.

[0014] Preferably, in step 5, after the data migration is completed, a reset operation is performed on the original Zone; at this time, the P / ECycle count of the Zone increases.

[0015] As a preferred option, in step 5, when returning to the global resource pool, the reset Zone is reordered according to the new P / E value and returned to the global idle Zone pool.

[0016] This invention provides a ZNS storage system based on SSTable fine-grained heat sensing and cross-region dynamic wear leveling, which adopts the above-mentioned ZNS storage method based on SSTable fine-grained heat sensing and cross-region dynamic wear leveling.

[0017] This invention identifies the popularity of data segments within the SSTable at the ZenFS layer and physically separates hot and cold data, writing them to different Zones to eliminate "dead Zones." Simultaneously, it establishes a global dynamic wear leveling mechanism, rotating roles between the Journal Zone and Data Zone, and between hot and cold Zones, to achieve overall wear leveling. Specific beneficial effects are as follows: (1) Significantly reduce system write amplification: By separating hot and cold data at the extent granularity, the lifecycle of data within the zone storing hot data is highly consistent, enabling rapid overall failure and direct reset, avoiding the repeated invalid data movement that is necessary in traditional GC due to the presence of long-tail data.

[0018] (2) Achieving zero-overhead wear leveling: The role rotation strategy proposed in this invention differs from traditional static wear leveling, which requires additional I / O to move data. Instead, it directly utilizes the characteristics of the business write stream to balance the media lifespan. By allocating Zones nearing their lifespan limit to long-term inactive cold data, the overall lifespan of the ZNS SSD is maximized.

[0019] (3) Solving the problem of Zone "dead" and resource waste: The predictive long-tail recycling mechanism effectively solves the problem of storage space waste caused by a very small amount of data occupying the entire Zone. Through active degradation and eviction, it ensures the purity and availability of high-performance hot Zone resource pool.

[0020] (4) Strong architectural compatibility and flexible deployment: The main improvements of this invention are focused on the allocator and reclaimer of the file system (ZenFS), which can be directly integrated into the existing RocksDB and ZNS architectures without modifying the upper-layer application code. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling in an embodiment. Detailed Implementation

[0022] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0023] Example like Figure 1 As shown, this embodiment provides a ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling, which includes the following steps: Step 1: Initialize the ZNS storage environment and build the global resource pool; Scan and sort: Scan all physical partitions (Zones) of the ZNS SSD device and read the P / E Cycle (Program / Erase Cycle) count for each Zone.

[0024] Constructing a global resource pool: The system constructs a global free zone pool. This pool is not merely a simple linked list, but a queue strictly sorted from low to high based on P / E cycle values. Logically, the resource pool is divided into three dynamic candidate intervals: Young zone: The top region with the lowest P / E value (e.g., the top 10% of the P / E values). Healthy range: The region where the P / E value is in the middle range; Old region: The tail region with the highest P / E value.

[0025] Step 2: Write the request initiation and fine-grained heat classification of SSTable data; When the upper-layer LSM-Tree (Log-Structured Merge-Tree) storage engine initiates a write or compaction request, the SSTable data enters the Hotness Classifier. By analyzing the update frequency of the key, historical access count, or timestamp, the SSTable classifies the data block to be written into fine-grained hotness based on extent granularity and marks it as a hot data stream or a cold data stream.

[0026] The popularity categories are as follows: If the determination result is "yes" (i.e., data is updated frequently), the data stream is marked as a hot data stream.

[0027] If the determination result is "no" (i.e., low frequency or static data), the data stream is marked as a cold data stream.

[0028] Flow to Allocator: The marked data flows to the Zone Allocator to initiate a space request.

[0029] Step 3: Zone allocation based on type; After receiving the request, the Zone allocator executes a counterintuitive wear leveling strategy to allocate Zones based on the data classification results obtained from S2 and the specific attributes of the request: Path A (Log Data): If the request type is Journal / WAL, the Zone allocator selects a Zone from the young state range (lowest P / E) of the global resource pool; this is because the Write Ahead Log has an extremely high write / erase frequency and must be handled by the "young" Zone with the longest lifespan.

[0030] Path B (Hot Data): If the request type is Hot Data, the Zone allocator selects a Zone from the healthy range (medium P / E).

[0031] Path C (Cold Data): If the request type is Cold Data, the Zone allocator selects a Zone from the old state zone (highest P / E). This step takes advantage of the characteristic of cold data that it is "not erased for a long time after being written" to put the high wear Zone into a "sleep" state and prevent it from running out of life.

[0032] Step 4: Physical write and unified metadata update; Physical disk write-in: Data is written to the physical layer's log zone, hot zone, or cold zone according to the allocation results.

[0033] Update Metadata & Mapping: Regardless of the path data takes to disk, the process ultimately converges here. The system updates the file system's metadata manifest.

[0034] For SSTable (sorted string table) files, their physical address mappings are recorded. It is worth noting that a logical SSTable file may consist of hot data segments stored in hot zones and cold data segments stored in cold zones, achieving physical isolation within the file.

[0035] Step 5: Garbage Collection (GC) and Wear Leveling Cycle; The system runs a GC scanning thread in the background to periodically maintain the Zones that are full. Scanning and Dead Zone Detection: The thread scans closed Zones and identifies dead Zones with a valid data ratio below the threshold and a long dwell time. When identifying dead Zones, the system calculates the valid data ratio and the average data dwell time (Age) of the Zone. If the valid data ratio is < 5% and the dwell time is long, it is identified as a dead Zone; if the conditions are not met, it is processed according to normal logic or ignored.

[0036] Trigger Demotion GC: For Zones that are determined to be zombie, trigger Demotion GC.

[0037] Extraction and Backflow: The system extracts the remaining valid data. Crucially, this extracted data is forcibly marked as "cold data" and backflowed to the Zone allocator in step 3 (pointing to path C) along the "forced downgrade to cold" path in the flowchart. This means that this data will be rewritten into the cold Zone of the old state region, achieving data purification.

[0038] Zone Reset and Rearrangement: After data migration is complete, a reset (Reset / Erase) operation is performed on the original Zone. At this time, the P / E Cycle count of the Zone is increased (updating the P / E cycle count).

[0039] Returning to the resource pool: After resetting, the Zone is re-sorted based on the new P / E value and returned to the global idle Zone pool. For example, a Journal Zone that was originally young becomes "old" due to frequent erases and writes, and will automatically fall into the healthy or old zone next time, no longer undertaking high-frequency tasks, thus achieving dynamic wear leveling.

[0040] This embodiment provides a ZNS storage system based on SSTable fine-grained heat sensing and cross-region dynamic wear leveling, which adopts the above-mentioned ZNS storage method based on SSTable fine-grained heat sensing and cross-region dynamic wear leveling.

[0041] To address the hot data skew issue within SSTables, this implementation breaks away from the traditional ZenFS limitation of mapping the entire SSTable to contiguous zones. During the process of flushing or compacting the LSM-Tree to generate a new SSTable, the system continuously monitors the update frequency and access time of key-value pairs. Utilizing the extent mapping feature of the ZNS file system, the same SSTable file is physically fragmented for storage: identified hot data segments are written to a dedicated hot zone, and cold data segments are written to a dedicated cold zone. Logical file integrity is maintained through SSTable metadata (Manifest), thereby achieving hot and cold isolation of the physical space within the SSTable.

[0042] To address the issues of uneven wear between zones and excessively rapid wear in the Journal Zone caused by hot / cold partitioning, this embodiment constructs a dynamic role allocation model based on the write / erase cycle (P / E cycle). The system maintains a global pool of idle Zone resources sorted by P / E value, abandoning fixed Zone type divisions. When requesting a new Zone, a counterintuitive allocation strategy is executed: the Zone with the lowest P / E value (youngest) is allocated to the frequently erased Journal role; the Zone with the medium P / E value is allocated to the hot data role; and the Zone with the highest P / E value (oldest) is allocated to the cold data role. Utilizing the characteristic of cold data being "not erased for a long time," high-wear Zones enter a "retirement mode," achieving passive wear leveling without data migration.

[0043] To address the issue of a small amount of long-tail data hindering hot zone resets, this embodiment introduces a proactive yield-prediction GC mechanism. The system background periodically scans full hot zones, calculating their effective data rate and data residency time. When a hot zone's effective data rate is detected to be below a preset very low threshold (e.g., 5%) and the remaining data residency time exceeds the average lifespan of hot data, the remaining data is determined to be "cold data disguised as hot data (Stragglers)." The GC process no longer moves this data to a new hot zone; instead, it forcibly "downgrades" it and migrates it to a cold zone for storage, then immediately resets the original hot zone, reclaiming physical space with minimal write amplification.

[0044] In summary, the ZNS storage method based on SSTable fine-grained heat sensing and cross-region dynamic wear leveling provided by this invention effectively solves the problems of SSTable heat skew, Zone deadlock, and uneven wear in ZenFS scenarios through physical hot and cold separation within the SSTable, role rotation allocation based on P / E cycle, and predictive degradation reclamation of long-tail data. This method significantly extends the overall lifespan of ZNS SSDs while reducing system write amplification. Through strict process control, it achieves precise matching between data lifecycle and storage medium physical characteristics, demonstrating significant novelty and practical value.

[0045] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling, characterized in that: Includes the following steps: Step 1: Initialize the ZNS storage environment and build the global resource pool; Scan all physical partitions (Zones) of the ZNS SSD device, read the P / E cycle count for each Zone, and then construct a global pool of free Zones sorted from low to high based on the P / E cycle value. Logically, the resource pool is divided into three dynamic candidate intervals: Younger age range: The head region with the lowest P / E value; Healthy range: The P / E value is in the middle range; The aging region: the tail region with the highest P / E value; Step 2: Write the request initiation and fine-grained heat classification of SSTable data; When the upper-layer LSM-Tree storage engine initiates a write or compression request, the SSTable data enters the heat classifier. By analyzing the update frequency of the key, historical access count, or timestamp, the data block to be written is classified into fine-grained heat based on the extent granularity and marked as a hot data stream or a cold data stream. Step 3: Zone allocation based on type; After receiving the request, the Zone allocator executes a counterintuitive wear leveling strategy to allocate Zones based on the data classification results obtained from S2 and the specific attributes of the request: Path A: If the request type is log, the Zone allocator selects a Zone from the young state range of the global resource pool; Path B: If the request type is hot data, the Zone allocator selects a Zone from the healthy range; Path C: If the request type is cold data, the Zone allocator selects a Zone from the old state range; Step 4: Physical write and unified metadata update; Based on the allocation results, the data is written to the physical layer's log zone, hot zone, or cold zone, and the file system's metadata and mappings are updated. Step 5: Garbage Collection (GC) and Wear Leveling Cycle; The system runs a GC scanning thread in the background, which periodically maintains the Zones that are full. The thread scans closed Zones and identifies zombie Zones that have a valid data rate below the threshold and a long residence time. For zombie Zones, it triggers downgraded garbage collection and extracts the remaining valid data. Then, it resets the original Zone and returns it to the global resource pool.

2. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 1, characterized in that: In step 2, the popularity categories are specifically as follows: If the determination result is high-frequency updated data, the data stream is marked as a hot data stream; If the determination result is low-frequency or static data, the data stream is marked as a cold data stream.

3. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 2, characterized in that: In step 2, the marked data stream initiates a space request to the Zone allocator.

4. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 3, characterized in that: In step 4, when updating the file system metadata and mapping, for SSTable files, their physical address mapping is recorded; a logical SSTable file may include hot data segments stored in hot zones and cold data segments stored in cold zones, achieving physical isolation within the file.

5. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 4, characterized in that: In step 5, when identifying a dead zone, the system calculates the zone's effective data rate and average data dwell time. If the effective rate is less than 5% and the dwell time is long, it is determined to be a dead zone; if the conditions are not met, it is processed according to normal logic or ignored.

6. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 5, characterized in that: In step 5, when extracting the remaining valid data, the data is forcibly marked as cold data and flows back to the Zone allocator in step 3, where it is rewritten into the cold Zone of the old state region.

7. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 6, characterized in that: In step 5, after the data migration is completed, a reset operation is performed on the original Zone; at this time, the P / E Cycle count of the Zone increases.

8. The ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claim 7, characterized in that: In step 5, when returning to the global resource pool, the reset Zones are reordered according to the new P / E values ​​and returned to the global idle Zone pool.

9. A ZNS storage system based on SSTable fine-grained thermal sensing and cross-regional dynamic wear leveling, characterized in that: It adopts the ZNS storage method based on SSTable fine-grained heat sensing and cross-regional dynamic wear leveling as described in claims 1-8.

Images