A ZNS SSD data allocation method based on adaptive partition size configuration

Abstract

This invention belongs to the field of computer science, specifically relating to a ZNS SSD data allocation method based on adaptive partition size configuration. It includes setting four levels of partitions based on the logical space of the ZNS SSD and using four linked lists to manage these four levels of partitions respectively. If the written data is generated by an LSM tree flush operation, a partition with the highest parallelism is directly allocated to it. If the written data is generated by an LSM tree compression operation, a partition is allocated to the written data using a variable partition combination allocation method. A new ZNS command is designed, and a chip conflict-aware partition dynamic mapping method is used to realize the mapping of allocated partitions. This invention enables the system to reduce or even eliminate garbage collection overhead while achieving maximum device parallelism, greatly improving the device's write bandwidth.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to a ZNS SSD data allocation method based on adaptive partition size configuration. Background Technology

[0002] With the exponential growth of data, modern network-scale systems and applications place higher demands on the capacity and performance of storage systems. Traditional block flash-based SSDs, widely used in modern data processing, often require additional reserved space and cumbersome flash translation layers, which are detrimental to meeting these performance and capacity requirements. The emerging ZNS SSD presents a new storage interface that divides the entire logical address space into fixed-size partitions. Each partition must be written sequentially and cannot be overwritten; it must be reset before it can be rewritten. Simultaneously, ZNS SSDs allow the host to directly manage these partitions, minimizing garbage collection overhead and write amplification.

[0003] While ZNS is a good interface that draws a clear boundary between the host and flash firmware through partitioning, it lacks a hardware abstraction that explicitly leverages the internal parallelism of the SSD from the host. Therefore, much research has explored partition size and mapping methods in hopes of achieving a mapping scheme that optimizes device performance. Typically, a partition is converted into many flash blocks spanning multiple flash chips, each partition having a large writable capacity—this is the commonly used large partition. Larger partitions allow more flash chips to be mapped to them, resulting in higher internal parallelism, enabling parallel writing and erasing of multiple blocks and exhibiting high performance.

[0004] Unfortunately, this can prolong the latency of partition reclamation, thus blocking subsequent services. To address this issue, the concept of small partitions—partitions with smaller write capacities—is introduced. Small partitions typically map to flash blocks spanning only one or a few flash chips. Small partitions can solve the long latency problem of partition reclamation, but their drawback is also obvious: the fewer chips mapped, the lower the internal parallelism, leading to lower performance. Therefore, a fixed partition size often cannot simultaneously satisfy high parallelism and low reclamation latency. Thus, this invention proposes an adaptive variable partition size configuration and a corresponding data allocation method based on ZNS SSDs, enabling the system to fully utilize the device's full parallelism while reducing garbage collection latency and overhead. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a ZNS SSD data allocation method based on adaptive partition size configuration. By combining data access characteristics and the device's fully parallel write requirements, partitions of different sizes and parallelisms are matched to data with different characteristics, thereby achieving maximum device parallelism while reducing or even eliminating garbage collection overhead.

[0006] The specific plan includes the following steps:

[0007] S1. Based on the logical space of ZNS SSD, four levels of partitions are set up and four linked lists are used to manage the four levels of partitions respectively;

[0008] S2. If the written data is generated by flushing an LSM tree, then a partition with the maximum parallelism is directly allocated to it; if the written data is generated by compressing an LSM tree, then a partition is allocated to the written data using a variable partition combination allocation method.

[0009] S3. Design a new ZNS command to achieve partition mapping through a chip conflict-aware partition dynamic mapping method.

[0010] The beneficial effects of this invention are:

[0011] This invention designs a variable-size partition configuration based on ZNS SSD and proposes a corresponding data allocation method, enabling the system to reduce or even eliminate garbage collection overhead while achieving maximum device parallelism, thus greatly improving the device's write bandwidth. Attached Figure Description

[0012] Figure 1 This is a flowchart of the method of the present invention;

[0013] Figure 2 This is a schematic diagram illustrating the logical partition space management based on linked lists according to the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a ZNS SSD data allocation method based on adaptive partition size configuration, such as... Figure 1 As shown, it includes the following steps:

[0016] S1. Based on the logical space of ZNS SSD, four levels of partitions are set up and four linked lists are used to manage the four levels of partitions respectively.

[0017] Specifically, such as Figure 2 As shown, this invention defines SZ as the partition size that can be mapped to a parallel block group. First, the logical space of the ZNS SSD system is partitioned in units of SZ. That is, using partitions of size SZ as the base partitions, the logical space of the ZNS SSD is divided into multiple base partitions. In practical use, to achieve flexible and variable partition sizes, the base partitions are further divided and merged to obtain four levels of partitions. The size of a partition at the first level is equal to the size of one base partition; that is, a single base partition can be considered a first-level partition. The size of a partition at the second level is equal to the size of two base partitions; merging two base partitions yields a second-level partition. The size of a partition at the third level is equal to the size of four base partitions, and the size of a partition at the fourth level is equal to the size of eight base partitions. The number of partitions at each of the four levels is randomly initialized.

[0018] In this invention, the size of SZ is 64M. From the first level to the fourth level, the size of the corresponding partition increases by a factor of 2, namely 64M, 128M, 256M, and 512M respectively. At the same time, from the first level to the fourth level, the parallelism of the corresponding partition also increases by a factor of 2, with the 512M partition of the fourth level having the maximum parallelism of the device.

[0019] The logical space of the ZNS SSD system is divided into partitions of various sizes, managed using four types of linked lists. When a partition of a certain size is needed, the system first checks the linked list corresponding to that size to see if there is an allocable partition. If not, free partitions are merged or split. When a new partition of a certain size is allocated, it is added to the linked list corresponding to that size for unified management. When a partition of a certain size is reclaimed, it is removed from the linked list corresponding to that size.

[0020] S2. If the data to be written is a single file generated by the LSM tree flushing operation, then a partition with the maximum parallelism is directly allocated to it (i.e., a fourth-level partition is directly allocated); if the data to be written is multiple files generated by the LSM tree compression operation, then a partition is allocated to the data to be written using the variable partition combination allocation method.

[0021] Specifically, to ensure that the ZNS SSD system achieves maximum parallelism at every write operation, it is necessary to assign different write characteristics to the data based on the structural characteristics of the LSM tree. Regarding the data write characteristics, this invention primarily considers the refresh and compression operations of the LSM tree.

[0022] The LSM tree flush operation, which writes a file from memory to disk, typically generates only one SSTable file, which expires very quickly. Furthermore, the data within a file often expires simultaneously. Therefore, for this type of data generated by the LSM tree flush operation, it can be directly allocated to a partition with maximum parallelism to ensure that all chips in the device can be fully utilized during writing, while the extremely fast expiration will not incur any recycling overhead.

[0023] The number of files generated by LSM tree compression operations and their expiration times are uncertain. Therefore, the data generated by compression operations needs to be categorized based on the specific number of files generated each time. Since the number of files generated by a compression operation is unknown, and only by knowing the specific number of files generated can fully parallel and aware partitioning be performed, the compression results need to be predicted and statistically analyzed based on each actual compression process. Specifically, based on the number of files participating in compression each time and the number of expired key-value pairs that are merged and discarded, the specific number of files generated in this compression can be reasonably estimated.

[0024] Specifically, after obtaining the number of files generated in each compression operation, a variable partitioning allocation method is used to allocate partitions of the required size to the data files, ensuring that the write operation in this compression process achieves full device parallelism. This includes:

[0025] S21. Obtain the number of files generated by this LSM tree compression operation;

[0026] S22. Determine the required number of partitions and the size of each required partition based on the number of files; initialize the allocated set;

[0027] S23. Determine whether the number of partitions stored in the allocated set is equal to the number of partitions required. If yes, end the process and obtain all the partitions required to write data; otherwise, start a new round of allocation and execute step S24.

[0028] S24. To avoid conflicts between partitions during parallel writing, determine the partition size required for this round of allocation based on the device's maximum parallelism and the storage status in the allocated set;

[0029] S25. A partition is allocated using the partition space management method, and this partition does not conflict with any partitions stored in the allocated set; the partition allocated in this round is placed into the allocated set, and the process returns to step S23.

[0030] Specifically, a partition space management method based on the buddy algorithm is designed to achieve efficient allocation and reclamation of variable logical partitions. Step S25, the process of allocating a partition using the partition space management method, includes:

[0031] S251. Determine the target level corresponding to the required partition size and traverse the linked list corresponding to the target level. Determine whether there is an allocable partition (i.e., a free partition with a lifespan close to the data to be written) at the target level. If there is, allocate the partition and add it to the linked list corresponding to the target level. If there is no partition, proceed to step S252.

[0032] S252. Determine whether the sum of the sizes of all current free partitions is greater than the required partition size. If so, merge or split the free partitions to obtain the target level free partition, allocate the obtained free partition and add it to the linked list corresponding to the target level; otherwise, proceed to step S253.

[0033] Specifically, when the total size of all current free partitions is greater than the required partition size, there are two scenarios. The first is that there are free partitions at a higher level than the target level. In this case, the higher-level free partitions can be split to obtain the target level free partition. The second is that there are many free partitions at a lower level than the target level. In this case, these lower-level free partitions can be merged to obtain the target level free partition.

[0034] S253. The allocated partitions reclaimed through the space reclamation method are used as new free partitions. The free partitions are then merged or split to obtain free partitions at the target level. The obtained free partitions are allocated and added to the linked list corresponding to the target level.

[0035] Specifically, step S253, which involves reclaiming the allocated partition using a space reclamation method, includes:

[0036] S2531. Based on the required partition size, determine the target level corresponding to the required partition size, traverse each allocated partition in the linked list corresponding to the target level, reclaim the allocated partitions that meet the reclamation conditions and remove them from the linked list;

[0037] S2532. Determine if any allocated partitions at the target level have been reclaimed. If so, the reclamation ends. If not, traverse each allocated partition in the corresponding linked list of each of the other levels, reclaim the allocated partitions that meet the reclamation conditions, and remove them from the linked list.

[0038] The recycling condition is that the ratio of valid data size within an allocated partition to the total size of that allocated partition is less than 30%.

[0039] Specifically, this invention periodically determines whether the total size of all currently free partitions is less than the reclamation threshold. If so, it determines whether each allocated partition meets the reclamation conditions and reclaims the allocated partitions that meet the reclamation conditions as reserve partitions. This invention sets the reclamation threshold to 30% of the total space size. If the ratio of the total size of all currently free partitions to the total space size is less than 30%, then the allocated partitions need to be reclaimed.

[0040] In step S25, during the allocation process, when there are not enough free partitions to meet the allocation requirements, the reserve partitions are first extracted as new free partitions for merging or splitting. If there are still not enough free partitions, step S253 is executed to reclaim the allocated partitions in a timely manner through the space reclamation method.

[0041] S3. Design a new ZNS command to achieve partition mapping through a chip conflict-aware partition dynamic mapping method.

[0042] Specifically, to fully utilize the device's maximum parallelism, it's necessary to ensure that all chips are used at every write operation. Therefore, all chips and blocks on the device are first grouped according to the number of chips required for the base partition. The base partition needs to map four blocks from different chips; thus, every four chips form a parallel chip group. Blocks with the same offset within a parallel chip group form a parallel block group, where the size of each parallel block group is equal to the size of a base partition. Parallel chip groups do not conflict with each other, and parallel block groups in different parallel chip groups do not conflict with each other; however, parallel block groups in the same parallel chip group conflict with each other.

[0043] This invention proposes a new ZNS command, namely the Merge command. The specific format of the Merge command is (src, size), where src represents the starting logical address of the partition to be merged and mapped (i.e., the partition to be mapped), and size is the size of the partition to be mapped. The Merge command is specifically used for device-side partition mapping. When the host needs to allocate a partition of a certain size, it sends a Merge command to the device. The device, based on the starting logical address and size of the partition to be mapped passed in the Merge command, maps a corresponding number of non-conflicting parallel block groups to the partition to be mapped.

[0044] Specifically, to avoid conflicts between mapped parallel block groups, the process of dynamically mapping parallel block groups for partitions using the partition dynamic mapping method based on the Merge command is as follows:

[0045] S31. Confirm the number of partitions to be mapped according to the Merge command, and initialize the set of mapped chipsets;

[0046] S32. For each partition to be mapped, dynamically select a parallel chipset for mapping operation and add the selected parallel chipset to the set of mapped chipsets; when all partitions to be mapped are mapped, the process ends.

[0047] Specifically, in step 32, the process of dynamically selecting a parallel chipset for mapping any partition to be mapped includes:

[0048] S321. Confirm the number of parallel block groups z required for the partition to be mapped, and determine whether z is equal to the total number of parallel chipsets on the device. If yes, select all parallel chipsets on the device, add all parallel chipsets to the set of mapped chipsets, and then execute step S324. If not, execute step S322.

[0049] S322. Calculate the number of idle parallel block groups for each parallel chipset on the device side, sort all parallel chipsets in descending order according to the number of idle parallel block groups to obtain the first sequence, remove the parallel chipsets that exist in the set of mapped chipsets from the first sequence, and then execute step S323.

[0050] S323. Select the first z parallel chipsets and add these z parallel chipsets to the set of mapped chipsets;

[0051] S324. Map the idle parallel block group in the selected parallel chipset.

[0052] Specifically, before executing step S323, a prior judgment needs to be performed, including:

[0053] S3231. Determine whether the mapped chipset set contains all parallel chipsets on the device. If yes, the mapped chipset set needs to be cleared; otherwise, proceed to step S3232.

[0054] S3232. Determine whether the number of parallel chipsets contained in the first sequence is less than z. If so, the set of mapped chipsets needs to be cleared; otherwise, it remains unchanged.

[0055] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A ZNS SSD data allocation method based on adaptive partition size configuration, characterized in that, Includes the following steps: S1. Based on the logical space of ZNS SSD, four levels of partitions are set up and four linked lists are used to manage the four levels of partitions respectively; S2. If the written data is generated by flushing an LSM tree, then a partition with the maximum parallelism is directly allocated to it; if the written data is generated by compressing an LSM tree, then a partition is allocated to the written data using a variable partition combination allocation method. Step S2 allocates partitions for the write data using a variable partition combination allocation method, including: S21. Obtain the number of files generated by this LSM tree compression operation; S22. Determine the required number of partitions based on the number of files; initialize the allocated set; S23. Determine whether the number of partitions stored in the allocated set is equal to the number of partitions required. If yes, end the process and obtain all the partitions required to write data; otherwise, start a new round of allocation and execute step S24. S24. Determine the partition size required for this round of allocation based on the maximum parallelism of device writes and the storage status in the allocated set; S25. A partition is allocated using the partition space management method, and this partition does not conflict with any partitions stored in the allocated set; the partition allocated in this round is placed into the allocated set, and the process returns to step S23; S3. Design a new ZNS command to achieve partition mapping through a chip conflict-aware dynamic partition mapping method; The new ZNS command is a Merge command used for partition mapping on the device side. When the host needs to allocate a partition of a certain size, it sends a Merge command to the device side. The device side maps parallel block groups for the partition to be mapped according to the starting logical address and size of the partition to be mapped in the Merge command. The specific format of the Merge command is src,size; src represents the starting logical address of the partition to be mapped, and size is the size of the partition to be mapped. The process of implementing partition mapping based on the Merge command and using dynamic partition mapping includes: S31. Confirm the number of partitions to be mapped according to the Merge command, and initialize the set of mapped chipsets; S32. For each partition to be mapped, dynamically select a parallel chipset for mapping operation and add the selected parallel chipset to the set of mapped chipsets; when all partitions to be mapped are mapped, the process ends.

2. The ZNS SSD data allocation method based on adaptive partition size configuration according to claim 1, characterized in that, Step S1 specifically includes: First, divide the logical space of the ZNS SSD system into units based on the baseline partition; The baseline partition is classified into four levels of partitions. The size of the first level partition is equal to the size of 1 baseline partition, the size of the second level partition is equal to the size of 2 baseline partitions, the size of the third level partition is equal to the size of 4 baseline partitions, and the size of the fourth level partition is equal to the size of 8 baseline partitions. For each level of partition, when it is allocated, the partition is added to the linked list of the corresponding level; when it is reclaimed, the partition is removed from the linked list of the corresponding level.

3. The ZNS SSD data allocation method based on adaptive partition size configuration according to claim 1, characterized in that, Step S25, which involves allocating a partition using a partition space management method, includes the following steps: S251. Determine the target level corresponding to the required partition size and traverse the linked list corresponding to the target level. Check whether there is an allocable partition at the target level. If there is, allocate the partition and add it to the linked list corresponding to the target level. If not, proceed to step S252. S252. Determine whether the sum of the sizes of all current free partitions is greater than the required partition size. If so, merge or split the free partitions to obtain the target level free partition, allocate the obtained free partition and add it to the linked list corresponding to the target level; otherwise, proceed to step S253. S253. The allocated partitions reclaimed through the space reclamation method are used as new free partitions. The free partitions are merged or split to obtain free partitions at the target level. The obtained free partitions are allocated and added to the linked list corresponding to the target level.

4. The ZNS SSD data allocation method based on adaptive partition size configuration according to claim 3, characterized in that, Step S253, the process of reclaiming allocated partitions using space reclamation methods, includes: S2531. Based on the required partition size, determine the target level corresponding to the required partition size, traverse each allocated partition in the linked list corresponding to the target level, reclaim the allocated partitions that meet the reclamation conditions and remove them from the linked list; S2532. Determine if any allocated partitions at the target level have been reclaimed. If so, the reclamation ends. If not, traverse each allocated partition in the corresponding linked list of each of the other levels, reclaim the allocated partitions that meet the reclamation conditions, and remove them from the linked list. The recycling condition is that the ratio of valid data size within an allocated partition to the total size of that allocated partition is less than 30%.

5. The ZNS SSD data allocation method based on adaptive partition size configuration according to claim 1, step 32, the process of dynamically selecting a parallel chipset for mapping any partition to be mapped includes: S321. Confirm the number of parallel block groups z required for the partition to be mapped, and determine whether z is equal to the total number of parallel chipsets on the device. If yes, select all parallel chipsets on the device, add all parallel chipsets to the set of mapped chipsets, and then execute step S324. If not, execute step S322. S322. Calculate the number of idle parallel block groups for each parallel chipset on the device side, sort all parallel chipsets in descending order according to the number of idle parallel block groups to obtain the first sequence, remove the parallel chipsets that exist in the set of mapped chipsets from the first sequence, and then execute step S323. S323. Select the first z parallel chipsets and add these z parallel chipsets to the set of mapped chipsets; S324. Map the idle parallel block group in the selected parallel chipset.

Images