Dynamic hierarchical architecture for partition namespace solid-state disk

Abstract

The invention discloses a dynamic hierarchical architecture oriented to a partition namespace solid-state disk, which is characterized in that when a compression scheduling engine assigns a compression task, a capacity sensing module acquires the free space capacity of equipment, and determines weights of A, B and C categories for different hierarchies; then, in a priority judgment module, the priorities of the hierarchies are calculated according to the free space capacity and the scores of the hierarchies, and the hierarchies are enqueued according to the priorities of the hierarchies to form a priority queue; the hierarchies are sequentially taken out from the priority queue through a Victim selection module to serve as input of compression, and after the input hierarchies are determined, SST data, overlapped with the lower layer, of the input hierarchies are sent into a merging compression engine to be merged and sorted; and finally, outputting a result to a specified position through a data placement module. According to the dynamic hierarchical architecture oriented to the partition namespace solid-state disk, the effective data volume in the zone can be reduced by dynamically selecting a compact input hierarchy, so that the effective data migration overhead during garbage collection is reduced.

Description

Technical Field

[0001] The present invention relates to the field of ZNS storage technology, and in particular to a dynamic hierarchical architecture for a partition namespace solid state disk. Background Art

[0002] In the prior art, ZNS devices have been gradually deployed in the external storage of more and more data centers due to their fast access speed and unified interface compared to traditional devices. However, the sequential write characteristics of the ZNS interface greatly limit the performance of the device. The sequential write characteristics of the upper-level key-value storage application based on the log-structured merge tree perfectly match the limitations of the sequential write of the ZNS interface. At present, a large amount of existing work has alleviated the performance impact of the ZNS sequential write interface by using the log-structured merge tree. Improving the compression efficiency of the log-structured merge tree can fully exert the performance advantages of the ZNS interface.

[0003] Existing log structured merge tree compression generally selects the layer with a higher score as the input layer for compression, but does not consider the free space status of the current device. When the free space is insufficient, if a layer has a higher score but involves less data, it will still be preferred, but the layer involves less data, which leads to a significant increase in the effective data migration overhead during garbage collection. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic hierarchical architecture for partitioned namespace solid-state disks, which can reduce the amount of valid data in the zone by dynamically selecting the compaction input level to reduce the effective data migration overhead during garbage collection.

[0005] To achieve the above objectives, the present invention provides a dynamic hierarchical architecture for partition namespace SSDs. A capacity perception module, a priority judgment module, a victim selection module, and a data placement module are pre-set in the SSD. Whenever the compaction scheduling engine assigns a compaction task, the capacity perception module obtains the free space capacity of the device and determines the weights of the A, B, and C categories for different levels. Then, in the priority judgment module, the priority of the layer is calculated according to the free space capacity and the layer score, and each layer is queued according to the layer priority to form a priority queue; The Victim selection module sequentially takes out the layers from the priority queue as the input for compaction compression. After the input layer is determined, the SST data that overlaps with the lower layer is sent to the merge compression engine for merging and sorting. Finally, the results are output to the specified location through the data placement module.

[0006] Preferably, during the compaction process, invalid data volume is dynamically generated according to the free space. When the free space is insufficient, the lower layer is selected as the input in the priority calculation formula to generate more invalid data. Data within the same Level is placed in the same zone, and the zone is directly reset to complete GC.

[0007] Preferably, the capacity awareness module is responsible for obtaining the free space capacity Remain_Cap of the device to dynamically adjust the weight according to the remaining space; three critical values are proposed to characterize the space status of the current device: LESS, MEDIUM, LARGE. The three critical values are set as the ratios of the device's storage space: 0.3, 0.5, 0.7. The three critical values are closely related to the size of the device's storage space. The larger the device's storage space, the larger these three critical values are.

[0008] Preferably, the three critical values are dynamically adjusted according to the space size, and the space part of the priority calculation changes dynamically according to Remain_Cap. There are four cases between the three critical values: Remina_Cap < LESS, a large amount of invalid data needs to be recycled by GC, A = 3, B = 6, C = 9; LESS < Remina_Cap < MEDIUM, space needs to be recycled by GC, A = 2, B = 4, C = 6; MEDIUM < Remain_Cap < LARGE, A = 1, B = 2, C = 3; Remain_Cap > LARGE, A = 0.5, B = 1, C = 1.5.

[0009] Preferably, the priority judgment module calculates the priority of each level according to the relevant information of the remaining capacity and the scores of each level; the calculation formula is: , calculate the priority of; Among them, represents the weight, represents the degree of device idle, represents the score of the level; the levels in the LSM tree are divided into three categories: a, b, c, and the corresponding degrees of idle of these three categories of levels are A, B, C respectively, which are dynamically determined by the capacity awareness module according to the free space; finally, each level is queued according to the priority size.

[0010] Preferably, the victim selection module sequentially dequeues and selects levels from the priority queue as the input level for compaction compression. After determining the compression input level, the victim SST is selected from the level, and the overlapping SSTs are selected from the lower level and sent to the merge engine for merge sorting. After the merge sorting is completed and new data is generated, the previous data is invalidated, thereby generating invalid data.

[0011] Preferably, the data placement module places the sorted new data of the merge compression engine into the corresponding new output level. After migrating the valid data, the background garbage collection operation will also update the position of the data in the LSM tree through the data placement module. This position relationship is the mapping relationship between the logical position of the file system in the log structured merge tree level and the physical position on the ZNS device.

[0012] Therefore, the present invention adopts the above-mentioned dynamic hierarchical architecture for partitioned namespace solid-state disks, and proposes to use the compression mechanism in the log-structured merge tree to dynamically select the input level according to the current device free space and the level score to perform compression operations, thereby reducing the migration overhead of garbage collection. Compared with the traditional method of selecting the compressed input level according to the level score, it solves the problem of increased data migration overhead caused by ignoring the large amount of compressed data in the lower layer.

[0013] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a solid state disk internal module diagram of an embodiment of a dynamic hierarchical architecture for a partition namespace solid state disk of the present invention; Figure 2 It is a capacity perception module calculation flow chart of a dynamic hierarchical architecture embodiment for a partition namespace solid state disk of the present invention; Figure 3 It is a calculation flow chart of a priority determination module of an implementation scheme of a dynamic hierarchical architecture for a partition namespace solid state disk of the present invention. DETAILED DESCRIPTION

[0015] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0016] Embodiment 1

[0017] The present invention provides a dynamic hierarchical architecture for partition namespace solid state disks, such as Figure 1As shown in the figure, the capacity perception module, priority judgment module, Victim selection module and data placement module are pre-set in the SSD. Whenever the compaction scheduling engine assigns a compression task, the capacity perception module obtains the free space capacity of the device and determines the weights of categories A, B and C for different levels. Then, the priority of the level is calculated according to the free space capacity and the level score in the priority judgment module, and each level is queued according to the level priority to form a priority queue. The levels are taken out from the priority queue in turn by the Victim selection module as the input of compaction compression. After the input level is determined, the SST data that overlaps with the lower level is sent to the merge compression engine for merging and sorting. Finally, the result is output to the specified location through the data placement module.

[0018] The specific steps are: First, the compaction scheduler module queues each compaction task. When the system triggers a compaction, it takes a compaction job from the queue to execute the compaction.

[0019] Traditional LSM applications implement compaction by selecting the layer with a high score, but this does not take into account the free space status of the device, resulting in high GC overhead for ZNS devices. This embodiment improves the priority calculation formula and calculates the priority of each layer through the capacity-aware module and the priority judgment module.

[0020] After compaction starts, the capacity awareness module obtains the free space of the current device and sets the free space level according to different scenarios. The more free space there is, the less GC is needed. Therefore, the value of the idle degree in the formula should be reduced, because background GC will seriously affect the throughput of upper-layer applications. If there is less free space, GC is urgently needed to free up space. Therefore, minimizing GC overhead becomes the only way to improve system performance, so the value of the idle degree in the formula should be increased.

[0021] In addition, since the capacity threshold of the high-level layer is significantly greater than that of the low-level layer, data is transferred from the upper layer to the lower layer layer by layer, so the lower layer with larger capacity data has a greater probability of generating more invalid data. In this way, if enough space needs to be reclaimed, then when GC reclaims space, it will benefit more from migrating valid data in the zone where the high-level layer is located, and you can get less migration overhead, so the high-level It is lower than the lower level, so that the higher level can participate in compaction and generate enough invalid data.

[0022] Then the priority judgment module is based on the and The compaction priority of each layer is calculated, and these layers are queued to form a priority queue. After the priority of each layer is determined, the dequeue operation is performed to remove the layer to perform the compaction operation.

[0023] The Victim selection module selects the victim SSTable in the layer according to the dequeued level, then finds the SSTable with overlapping keys in the adjacent lower layer, and sends it to the Merge Engine module to perform the merge operation to generate a new SSTable.

[0024] Finally, the data placement module in the file system associated with the ZNS SSD allocates a zone for the newly generated SSTable.

[0025] In the process of compaction, invalid data is dynamically generated according to the free space. When the free space is insufficient, the priority calculation formula selects the lower layer as input to generate more invalid data. Since the current file system regards data in the same level as having the same lifespan, the data in the same level is placed in the same zone, and the zone is directly reset to complete GC.

[0026] Research and analysis show that the lifespan of SSTs in the same level is not exactly the same. Therefore, during garbage collection, the zone is not entirely filled with invalid data, but with a lot of valid data, resulting in a high migration overhead for valid data. In this embodiment, sufficient invalid data is generated as much as possible to achieve the prediction of the lifespan of data in the same level under ideal conditions, thereby reducing the impact of data migration on throughput.

[0027] Among them, the implementation and specific functions of the four modules are as follows: The capacity perception module is responsible for obtaining the free space capacity Remain_Cap of the device in order to dynamically adjust the weight according to the remaining space. Three critical values ​​are proposed to characterize the space status of the current device: LESS, MEDIUM, and LARGE. This is closely related to the space size of the device and is not fixed. The three critical values ​​are set to the ratio of the device storage space: 0.3, 0.5, and 0.7. The larger the device storage space, the larger the three critical values. For example, assuming that the storage space of the ZNS SSD is 1TB, the sizes of LESS, MEDIUM, and LARGE are: 300GB, 500GB, and 700GB respectively.

[0028] The three critical values are dynamically adjusted according to the space size. The space part of the priority calculation changes dynamically according to Remain_Cap. There are four cases between the three critical values, as Figure 2 shown below: (1) Remina_Cap < LESS, there is very little remaining free space, and a large amount of invalid data needs to be reclaimed by GC. A = 3, B = 6, C = 9; (2) LESS < Remina_Cap < MEDIUM, there is little remaining free space, and space needs to be reclaimed by GC. A = 2, B = 4, C = 6; (3) MEDIUM < Remain_Cap < LARGE, A = 1, B = 2, C = 3, there is more remaining space, and the need to reclaim space is small; (4) Remain_Cap > LARGE, A = 0.5, B = 1, C = 1.5, there is a large amount of remaining space, and almost no space needs to be reclaimed.

[0029] As Figure 3 shown, the priority judgment module calculates the priority of each level according to the relevant information of the remaining capacity and the scores of each level. The calculation formula is: , calculate priority. Among them, represents the weight, represents the free degree of the device, represents the score of the level. The levels in the LSM tree are divided into three categories: a, b, and c. For example, in a five-level LSM tree (L0 - L5): L0 and L1 are of category a, L2 and L3 are of category b, and L4 and L5 are of category c. Then the free degrees corresponding to these three categories of levels are A, B, and C respectively, which are dynamically determined by the capacity awareness module according to the free space. Finally, each level is queued according to the priority size.

[0030] The Victim selection module dequeues and selects levels from the priority queue in turn as the input levels for compaction. After determining the compaction input levels, it selects victim SSTs from these levels and selects overlapping SSTs from the lower levels and sends them to the merge engine for merge sorting. After the merge sorting ends and new data is generated, the previous data becomes invalid, thus generating invalid data.

[0031] The data placement module places the sorted new data from the merge compaction engine into the corresponding new output levels. In addition, after the background garbage collection operation migrates the valid data, it will also update the position of the data in the LSM tree through the data placement module. This position relationship is the mapping relationship between the logical position of the file system in the levels of the log-structured merge tree and the physical position on the ZNS device.

[0032] Therefore, the present invention adopts the above-mentioned dynamic hierarchical architecture for partition namespace solid-state disks, which can reduce the amount of valid data in the zone by dynamically selecting the compaction input level to reduce the effective data migration overhead during garbage collection.

[0033] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A dynamic hierarchical architecture for partitioned namespace solid state drives, characterized by: A capacity awareness module, a priority judgment module, a Victim selection module, and a data placement module are preset in the solid-state drive. Whenever the compaction scheduling engine dispatches a compaction task, the capacity awareness module obtains the free space capacity of the device and determines the weights of categories A, B, and C for different levels; Then, in the priority judgment module, the priority of each level is calculated based on the free space capacity and the score of the level, and each level is queued according to the level priority to form a priority queue; The Victim selection module sequentially takes out levels from the priority queue as the input for compaction. After determining the input level, the SST data overlapping with the lower layer is sent to the merge compaction engine for merge sorting; Finally, the result is output to the specified location through the data placement module.

2. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 1, characterized in that: During the compaction process, the amount of invalid data is dynamically generated according to the free space. When the free space is insufficient, the priority calculation formula selects the lower layer as the input to generate more invalid data. The data within the same Level is placed in the same zone, and the zone is directly reset to complete GC.

3. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 1, characterized in that: The capacity awareness module is responsible for obtaining the free space capacity Remain_Cap of the device to dynamically adjust the weights according to the remaining space; three critical values are proposed to describe the space condition of the current device: LESS, MEDIUM, LARGE. The three critical values are set as the ratios of the device storage space: 0.3, 0.5, 0.

7. The three critical values are closely related to the size of the device storage space. The larger the device storage space, the larger these three critical values are.

4. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 3, characterized in that: The three critical values are dynamically adjusted according to the space size, and the space part of the priority calculation changes dynamically according to Remain_Cap. There are four cases between the three critical values: Remina_Cap < LESS, a large amount of invalid data needs to be recycled by GC, A = 3, B = 6, C = 9; LESS < Remina_Cap < MEDIUM, space needs to be recycled by GC, A = 2, B = 4, C = 6; MEDIUM < Remain_Cap < LARGE, A = 1, B = 2, C = 3; Remain_Cap > LARGE, A = 0.5, B = 1, C = 1.

5.

5. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 1, characterized in that: The priority judgment module calculates the priority of each level based on the relevant information of the remaining capacity and the scores of each level; the calculation formula is: ,calculate Priority; in, represents the weight, Represents the idleness of the device. Represents the score of the level; divide the levels in the LSM tree into three categories: a, b, and c. The corresponding idleness of these three levels is A, B, and C, which is determined dynamically by the capacity perception module according to the free space; finally, each level is queued according to its priority.

6. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 1, characterized in that: The Victim selection module sequentially dequeues from the priority queue to select a level as the input level for compaction. After determining the compaction input level, a victim SST is selected from this level, and the overlapping SST is selected from the lower layer and sent to the merge engine for merge sorting. After the merge sorting ends and new data is generated, the previous data is invalidated, thus generating invalid data.

7. The dynamic hierarchical architecture for partitioned namespace solid state drives according to claim 1, characterized in that: The data placement module places the sorted new data of the merge compaction engine into the corresponding new output level. After the background garbage collection operation migrates the valid data, it will also update the position of the data in the LSM tree through the data placement module. This position relationship is the mapping relationship between the logical position of the file system in the log-structured merge tree level and the physical position on the ZNS device.

Images