An end-side out-of-order write rearrangement method for consumer-grade partitioned storage

Abstract

This invention discloses an edge-side out-of-order write reordering method for consumer-grade partitioned storage, belonging to the field of computer storage systems and flash memory management technology. Addressing the problem of degraded multi-threaded parallel write performance in consumer-grade partitioned storage due to kernel-forced sequential writes, and the insufficient capacity of the edge-side SRAM write buffer to meet large-scale out-of-order reordering requirements, this invention proposes a dynamic processing method with a two-layer reordering architecture. By dividing the high-performance SLC flash memory region in the device into page-level SLC and zone-level SLC as an extended write buffer, when processing write requests, the continuity and temperature label of the data in the SRAM buffer are judged; if the data is continuous and hot data, a direct flushing strategy is executed to the zone-level SLC; if out-of-order data occurs and the buffer is insufficient, a degradation strategy is executed to temporarily store the data in the page-level SLC; and continuity is dynamically restored when subsequent data is supplemented. This invention overcomes the limitations of small edge-side caches while releasing the parallel write potential of the device, significantly improving multi-threaded write IOPS.

Description

Technical Field

[0001] This invention relates to the field of computer storage systems and operating system flash memory management technology, specifically to an edge-side out-of-order write reordering method for consumer-grade zoned storage in multi-threaded concurrent scenarios. Background Technology

[0002] With high-density flash memory such as QLC becoming the mainstream solution for meeting the large-capacity requirements of consumer storage devices, the Zoned Storage interface has been widely adopted due to its advantages of effectively reducing write amplification and providing efficient parallel reads. However, in order to support efficient coarse-grained mapping and avoid in-situ updates of flash blocks, the zoned interface mandates that sequential writes must be strictly performed within each zone.

[0003] In existing operating system kernels, the Linux kernel employs a "zone write plugging" mechanism to address the mandatory sequential requirements of partitioned storage. This mechanism queues and merges concurrent requests destined for the same zone at the file system level, limiting a single zone to only one in-flight request at a time, thereby forcibly serializing out-of-order parallel operations at the kernel level.

[0004] However, the aforementioned kernel-forced serialization mechanism has significant technical flaws when handling the multi-threaded, fine-grained parallel write scenarios common in consumer devices. The strict queuing mechanism leads to severe lock contention, greatly diminishing the performance advantages of modern NVMe devices' multi-queue parallel processing. Although existing technologies have attempted to bypass kernel limitations by offloading the out-of-order reordering process to the device's built-in SRAM write buffer, the SRAM capacity of consumer devices is extremely small, typically only a few hundred KB. As the concurrent queue depth increases, this tiny buffer space is instantly exhausted by discrete data, preventing subsequent data from being pieced together into continuous segments. This leads to widespread out-of-order write failures and host I / O timeout retries on the device, not only needlessly consuming storage bandwidth and computing resources but also causing a precipitous drop in parallel write performance. Summary of the Invention

[0005] The technical problem this invention aims to solve is that existing consumer-grade partition storage suffers from severely limited parallel write performance and easily exhausted device buffers when handling multi-threaded concurrent writes due to the kernel's forced sequential write queuing and the device-side SRAM write buffer capacity being too small to handle large-scale out-of-order reordering tasks. This invention provides an end-side out-of-order write reordering method for consumer-grade partition storage.

[0006] The specific technical solution adopted in this invention is as follows: An edge-side out-of-order write reordering method for consumer-grade partitioned storage, characterized by the following steps: Step S1: Receive out-of-order write requests from the host side at the storage device end, and temporarily store the request data in the built-in SRAM write buffer for merging.

[0007] Step S2: Obtain the current device operating status information, which includes the data temperature label, SRAM capacity status, and the current write pointer of the extended buffer area.

[0008] Step S3: Determine the temperature label of the data; if it is cold data, directly write it to the page-level SLC area, and when the SLC free block is lower than the preset threshold, then batch sort the order and migrate it to the bottom-level QLC area; if it is hot data, proceed to the next step.

[0009] The page-level SLC region is defined as follows: the address mapping granularity of the data in this part of the SLC is one page, 4KB.

[0010] The area-level QLC region is defined as follows: the address mapping granularity of the data in this part of the QLC is a zone.

[0011] Step S4: Determine whether the hot data sequence in the SRAM write buffer meets the preset continuity condition; if the continuity condition is met, execute the direct write strategy: directly write the continuous hot data sequence to the area-level SLC region; if the continuity condition is not met, i.e. there is out-of-order missing data, proceed to the next step. The area-level SLC region is defined as follows: the address mapping granularity of the data in this part of the SLC region is a zone.

[0012] The preset continuity condition is that the data in the current SRAM write buffer can form a continuous sequence in strict increment of logical block address (LBA), and the starting logical address of the sequence can be seamlessly connected with the logical address corresponding to the current write pointer of the physical partition of the underlying flash memory.

[0013] Step S5: Further determine whether the SRAM write buffer has reached the capacity threshold; if the capacity threshold has not been reached, retain the data and wait for new data to be written to the SRAM; if the capacity threshold has been reached, execute a selective flushing strategy: retain some data in the write buffer and temporarily store out-of-order data that exceeds the buffer's maintenance capacity to the page-level SLC area. The selective flushing strategy is as follows: Calculate whether the total number of missing logical pages exceeds the total capacity of the SRAM write buffer; if it does not exceed, assuming the total number of missing logical pages is K, then temporarily store the K data with the largest logical page numbers in the SRAM write buffer into the page-level SLC region; if it exceeds, then temporarily store all data in the SRAM write buffer into the page-level SLC region. The mechanism of temporarily storing the K data with the largest logical page numbers into the page-level SLC region aims to free up space in the SRAM write buffer equal to K logical pages, so that the missing data blocks needed to fill the gaps can be fully accommodated later.

[0014] Step S6: When a write request to fill in the missing logic block is received subsequently, the continuity recovery strategy is executed: the data of the new request is written together with the out-of-order data associated with the page-level SLC area to the district-level SLC area to reconstruct physical continuity.

[0015] Furthermore, the page-level SLC region and the zone-level SLC region adopt a hybrid granularity management mode with a shared free block list on the physical medium, and maintain independent write pointers for the two regions respectively; in the allocation of the zone-level SLC, if the actual allocated space is not filled with a complete zone, the unwritten blocks are marked as reserved.

[0016] This invention abandons the traditional mechanisms that rely solely on kernel "queueing and serialization" or "SRAM micro-buffering" to handle out-of-order requests. Instead, it employs a dynamic scheduling strategy that pools the existing SLC region within the device into a "two-layer extended buffer." While ensuring the small cache doesn't overflow, the system intelligently identifies out-of-order gaps: efficiently writing directly when data is continuous, safely temporarily storing data gaps, and promptly restoring continuity when data is complete. This strategy completely removes the kernel's constraints on concurrent requests, enabling the storage device to receive multi-threaded random requests at full concurrency, improving multi-threaded write efficiency several times over, while simultaneously preventing device crashes caused by insufficient buffer size. Attached Figure Description

[0017] Figure 1 is a logic flowchart of the end-side out-of-order write reordering method in an embodiment of the present invention; Figure 2 is a schematic diagram of the existing partitioned storage I / O stack and zone write suspension mechanism; Figure 3 is a schematic diagram of the two-layer out-of-order write and rearrangement architecture in an embodiment of the present invention; Figure 4 is a schematic diagram of the workflow for selective write buffer flushing and dynamic space allocation in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] This invention proposes an edge-side out-of-order write reordering method for consumer-grade partitioned storage. This method embeds dynamic preordering logic combining SRAM and SLC dual-layer media within the flash translation layer (FTL), completely eliminating the single-zone concurrent write bottleneck of existing partitioned storage.

[0020] The present invention provides an edge-side out-of-order write reordering method for consumer-grade partitioned storage, comprising the following steps: Step S1: The storage device receives an out-of-order write request from the host side. Due to the sequential write limitation of partitioned storage, the device does not directly write the request to disk, but temporarily stores the requested data in the built-in SRAM write buffer to attempt sorting and merging.

[0021] Step S2: The system obtains the current operating status information of the device in real time, including: the data temperature tag transmitted by the host, which is used to distinguish between hot and cold data, the capacity status of SRAM, which determines whether it is full, and the current write pointer of the extended buffer area in the device.

[0022] Step S3: The system first determines the data temperature label. If it is determined to be "cold data," the system directly writes it to the page-level SLC (page SLC) area mapped at the page granularity. This batch of cold data will then reside in the page-level SLC until the total free blocks in the device's overall SLC fall below a preset threshold (threshold < 2). Only then will the system reorder the data and ultimately migrate it to the underlying, high-capacity zone-level QLC (zoned QLC) area mapped at the partition granularity. If it is determined to be "hot data," the system tends to keep this data in the SLC and proceeds to a rigorous continuity determination process.

[0023] Step S4: For "hot data", the system checks whether the data sequence in SRAM meets the continuity condition, that is, the LBA is continuously incremented and seamlessly connected with the write pointer of the current physical partition. If the determination is "yes", the direct flushing strategy is executed, and the continuous hot data sequence is directly written to the zoned SLC area mapped with partition granularity.

[0024] Step S5: If the continuity check is "No", meaning there is a gap caused by out-of-order missing data, the system will assess whether the SRAM write buffer has reached the capacity threshold. If the threshold has not been reached, the system will not perform a flush operation, allowing the data to reside and wait for new data to be written to the SRAM in order to naturally stitch the gap together; if the capacity threshold has been reached, the system will trigger a selective flush strategy and first calculate the "total number of missing logical pages" required to fill the current gap.

[0025] Step S6: The system compares the calculated "total number of missing logical pages" with the "total capacity of the SRAM write buffer": If the total number of missing logical pages does not exceed the total capacity of the SRAM write buffer: Assuming the total number of missing logical pages is K, the system will extract the K data with the largest logical page numbers from the SRAM and temporarily store them in the page-level SLC, thereby precisely freeing up K spaces to accommodate the missing data that arrives later, and then enter a waiting state.

[0026] If the total number of missing logical pages exceeds the total capacity of the SRAM write buffer, it means that the gap is too large. The system will directly store all the data in the SRAM write buffer into the page-level SLC, clear the buffer, and enter a waiting state.

[0027] Step S7: When the system is in a waiting state, upon receiving new write data that can fill in the previously missing logical blocks, the system executes a continuity recovery strategy. The system merges the newly arrived request data with the logically related out-of-order data previously temporarily stored in the page-level SLC and writes them together into the area-level SLC region, thereby reconstructing physical continuity.

[0028] To gain a deeper understanding of the technical problem this invention aims to solve, we first analyze the I / O stack, zone write suspension mechanism, and internal architecture of existing partitioned storage with reference to Figure 2. Figure 2 is a schematic diagram of the I / O stack and internal architecture of existing partitioned storage. In the block device layer on the host side, the existing Linux kernel employs a "zone write suspension mechanism" to meet the requirement that partitioned storage must write sequentially. This mechanism maintains a zwplug structure (e.g., zwplug_0, zwplug_1) for each region, which includes two states: "suspended" and "idle". When a bio request arrives, step ①, if the target region is in the "idle" state, the bio is converted into a request; steps ② and ③, if the target region is in the "suspended" state, subsequent bios will be intercepted and queued; step ④. Only when the write to the region is completed will the suspended request be converted into a request; steps ⑦ and ⑧, and sent to the driver layer's Submission Queue (SQ) and Completion Queue (CQ); steps ⑤ and ⑥. This kernel-forced serialization mechanism can lead to severe lock contention during multi-threaded writes. Furthermore, within the flash memory architecture on the right, the edge-side SRAM write buffer capacity is extremely small. When the kernel issues a large number of concurrent requests, this tiny buffer is easily exhausted, lacking efficient out-of-order data handling and management capabilities, which can easily trigger write crashes.

[0029] To address the aforementioned issues, this invention designs a two-layer reordering architecture. Referring to Figure 3, the core physical and logical architecture and data flow path of this invention are explained. Figure 3 is a schematic diagram of the two-layer out-of-order write reordering architecture in an embodiment of this invention. When faced with out-of-order requests from the host, such as L2, L3 -> L0, L1 -> L21-L24 -> L20 arriving sequentially, the data first enters the SRAM write cache to attempt merging, for example, merging L0 to L3. At this time, the system introduces a temperature detector: If the data is determined to be hot data (Y), and the data already forms a continuous sequence in the write cache, such as L0 to L3, it is directly flushed to the zone-level SLC in the SLC region via blue arrow A. If the data fails to form a continuous sequence before the write cache is exhausted, such as L21 to L24, with L20 missing, a degradation strategy is executed, and this out-of-order data is temporarily stored in the page-level SLC via blue arrow B. When the subsequent request to complete the missing data L20 arrives, the recovery mechanism is triggered: at the same time the new request L20 is written to the zone-level SLC, the operation of black arrow I is immediately triggered, migrating the associated out-of-order data L21-L24 temporarily stored in the page-level SLC to the zone-level SLC to reconstruct physical continuity.

[0030] If it is determined to be cold data (N), it is temporarily stored in the cold data area of ​​the page-level SLC. After the conditions are met, it is then migrated to the lower-level area-level QLC via black arrow II. Outdated data in the district-level SLC will also be migrated downwards according to the strategy, indicated by the green arrow. Throughout the process, the logical-physical mapping relationship between the "mapping table cache" in SRAM and the "mapping table" in SLC will be updated synchronously.

[0031] To efficiently manage data flow in the above architecture, this invention further designs a selective write buffer flushing and space allocation mechanism. See Figure 4 for a detailed explanation. Figure 4 is a schematic diagram of the workflow for selective write buffer flushing and dynamic space allocation. Multiple fragmented data segments may exist simultaneously in the SRAM write cache; the system determines the data's destination through a temperature selector. Selective write to the zone-level SLC: For partially contiguous data segments that are considered hot data, assuming the data in the write cache is L0, L1, L4, L5, the system writes the contiguous portions L0 and L1 into the zone-level SLC via red arrow ①. To prevent subsequent out-of-order data from overwriting, the unwritten sectors within this physical block are marked as "reserved blocks"; while the subsequent data L4 and L5, which are missing, remain in SRAM, as shown by red dashed line ②, awaiting completion. If data L2 and L3 arrive later, they are written directly into the zone-level SLC together.

[0032] Selective write to page-level SLC: When the write cache faces capacity limits and the missing data spans too large, assuming the data in the write cache is L7, L8, L11, L12, the data in the page-level SLC is L9, and the missing data is L6, L10, the system, via red arrow ③, strips the data with larger logical page numbers and selectively stores L11 and L12 temporarily in the page-level SLC, thereby freeing up much-needed merging space for SRAM.

[0033] Direct writing to page-level SLC: For data identified as "other" cold data by the temperature selector, such as L20, L21, L25, and L26, the current fault condition is disregarded, and the data is directly written to the page-level SLC via green arrow ①. Furthermore, the shared free list in the lower right corner of the diagram is the key space management mechanism of this invention. It uniformly provides physical block allocation for both region-level and page-level SLCs, enabling these two different granularity buffer regions to dynamically share physical resources, maximizing flash memory utilization.

[0034] Finally, the overall system workflow is driven by specific control logic. Refer to Figure 1 for a summary. Figure 1 is a flowchart of the end-side out-of-order write reordering method in this embodiment of the invention. The system receives the out-of-order write request (step S1) and obtains device operating status information (step S2); based on the temperature label and continuity conditions, it executes direct write to the region-level or page-level SLC (steps S3 and S4); when encountering out-of-order data and insufficient buffer, it triggers the selective write strategy steps S5 and S6 corresponding to Figure 4; and when subsequent new data requests arrive, it executes the continuity recovery strategy step S7 corresponding to Figure 3.

[0035] In summary, this invention, through the control logic shown in Figure 1, achieves the dual-layer extended buffer and selective flushing effects shown in Figures 3 and 4, thereby completely solving the problems of low parallel write performance and easy exhaustion of the small end-side buffer caused by the kernel forced serialization mechanism shown in Figure 2.

Claims

1. A method for edge-side out-of-order write reordering of consumer-grade partitioned storage, characterized in that, Includes the following steps: Step S1: Receive out-of-order write requests from the host side at the storage device end, and temporarily store the request data in the built-in SRAM write buffer for merging; Step S2: Obtain the current device operating status information, which includes the data temperature label, SRAM capacity status, and the current write pointer of the extended buffer area; Step S3: Determine the temperature label of the data; If it is cold data, it is directly written to the page-level SLC area. When the SLC free block is lower than the preset threshold, it is then sorted in batches and migrated to the underlying area-level QLC area. If it is hot data, proceed to the next step; Step S4: Determine whether the hot data sequence in the SRAM write buffer meets the preset continuity condition; If the continuity condition is met, execute the direct write strategy: directly write the continuous hot data sequence to the district-level SLC area; If the continuity condition is not met, i.e. there is a disordered missing sequence, proceed to the next step; Step S5: Further determine whether the SRAM write buffer has reached the capacity threshold; if it has not reached the capacity threshold, retain the data and wait for new data to be written to the SRAM; If the capacity threshold has been reached, a selective flushing strategy is executed: some data is retained in the write buffer, and out-of-order data that exceeds the buffer's maintenance capacity is temporarily stored in the page-level SLC area. Step S6: When a write request to fill in the missing logic block is received subsequently, the continuity recovery strategy is executed: the data of the new request is written together with the out-of-order data associated with the page-level SLC area to the district-level SLC area to reconstruct physical continuity.

2. The end-side out-of-order write reordering method for consumer-grade partitioned storage as described in claim 1, characterized in that, The page-level SLC region and the region-level SLC region adopt a hybrid granularity management mode with a shared free block list on the physical medium, and maintain independent write pointers for the two regions respectively. In the allocation of the area-level SLC, if the actual allocated space is not filled with the entire area, the unwritten blocks are marked as reserved.

3. The end-side out-of-order write reordering method for consumer-grade partitioned storage as described in claim 1, characterized in that, The preset continuity condition in step S4 is that the data in the current SRAM write buffer forms a continuous sequence in strict increment of logical block address, and the starting logical address of the continuous sequence is seamlessly connected with the logical address corresponding to the current write pointer of the physical partition of the underlying flash memory.

4. The end-side out-of-order write reordering method for consumer-grade partitioned storage as described in claim 1, characterized in that, The selective flushing strategy in step S5 is as follows: Calculate whether the total number of missing logical pages exceeds the total capacity of the SRAM write buffer; If the total number of missing logical pages is K, then the K data with the largest logical page number in the SRAM write buffer will be temporarily stored in the page-level SLC area. If the limit has been reached, all data in the SRAM write buffer will be temporarily stored in the page-level SLC region.

Images