A dynamic hierarchical architecture for partitioned namespace solid state disk

Abstract

The application discloses a kind of dynamic hierarchical architecture for partitioned namespace solid state disk, whenever compaction scheduling engine dispatches compression task, capacity-aware module obtains device free space capacity, and the weight of A, B, C category is decided for different level;Then in priority judgment module, according to the score of free space capacity, level, the priority of level is calculated, and according to level priority, each layer is queued, and priority queue is formed;Through Victim selection module, level is taken out from priority queue as the input of compaction compression in turn, after determining input level, it is sent into merging compression engine to merge sorting with the SST data of the lower layer overlapping;Finally, the result is output to the specified location by data placement module.The application adopts the above-mentioned dynamic hierarchical architecture for partitioned namespace solid state disk, and the amount of effective data in zone can be reduced by dynamically selecting compaction input level, to reduce the effective data migration overhead when garbage collection.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of ZNS storage, in particular to a dynamic hierarchical architecture for a partitioned namespace solid state disk. BACKGROUND

[0002] In the prior art, due to the characteristics of fast access and uniform interface compared with traditional devices, ZNS devices have been gradually deployed in more and more external storage of data centers. However, the sequential writing characteristics of the ZNS interface greatly limit the device performance. The sequential writing characteristics of the upper key value storage based on the log-structured merge tree perfectly match the sequential writing limitation of the ZNS interface. At present, a large number of existing works alleviate the performance impact caused by the ZNS sequential writing interface by using the log-structured merge tree, and improving the compression efficiency of the log-structured merge tree can fully exert the performance advantage of the ZNS interface.

[0003] The existing compression of the log-structured merge tree generally selects a higher level as the input level of compression, but does not consider the free space condition of the current device. When the free space is insufficient, if the level has a higher score but involves less data, the level will still be preferentially selected, but the involvement of less data in the level leads to a significant increase in the effective data migration overhead during garbage collection. SUMMARY

[0004] The purpose of the application is to provide a dynamic hierarchical architecture for a partitioned namespace solid state disk, which can reduce the amount of effective data in the zone by dynamically selecting the compaction input level, thereby reducing the effective data migration overhead during garbage collection.

[0005] To achieve the above purpose, the application provides a dynamic hierarchical architecture for a partitioned namespace solid state disk, which pre-provides a capacity awareness module, a priority judgment module, a Victim selection module and a data placement module in the solid state disk. Whenever the compaction scheduling engine dispatches a compression task, the capacity awareness module obtains the free space capacity of the device to determine the weights of the A, B and C categories for different levels;

[0006] Then, in the priority judgment module, the priority of the level is calculated according to the free space capacity and the score of the level, and each level is enqueued according to the priority of the level to form a priority queue;

[0007] The Victim selection module takes the levels from the priority queue as the input of compaction compression in sequence, determines the input level, and sends the SST data overlapping with the lower level of the input level to the merge compression engine for merge sorting after the input level is determined;

[0008] Finally, the data placement module outputs the result to the specified location.

[0009] Preferably, the amount of invalid data is dynamically generated according to the free space during the compaction compression process, when the free space is insufficient, the priority calculation formula selects the lower layer as the input to generate more invalid data, places the data in the same zone in the same level, and directly resets the zone to complete the GC.

[0010] Preferably, the capacity awareness module is responsible for obtaining the free space capacity Remain_Cap of the device, so as to dynamically adjust the weight according to the remaining space; three critical values are proposed to depict the current device space condition: LESS, MEDIUM, LARGE, the three critical values are set as the ratio of the device storage space: 0.3, 0.5, 0.7, the three critical values are closely related to the space size of the device, the larger the device storage space, the larger the three critical values.

[0011] Preferably, the three critical values are dynamically adjusted according to the space size, and the space part of the priority calculation is dynamically changed according to Remain_Cap, which is divided into four cases between the three critical values:

[0012] Remina_Cap<LESS, a large amount of invalid data needs to be recycled by GC, A=3, B=6, C=9;

[0013] LESS<Remina_Cap<MEDIUM, GC needs to recycle space, A=2, B=4, C=6;

[0014] MEDIUM<Remain_Cap<LARGE, A=1, B=2, C=3;

[0015] Remain_Cap>LARGE, A=0.5, B=1, C=1.5.

[0016] Preferably, the priority judgment module calculates the priority of each level according to the related information of the remaining capacity and the scores of each level; the calculation formula is: , the priority of is calculated;

[0017] wherein, represents the weight, represents the free degree of the device, represents the score of the level; the levels in the LSM tree are divided into a, b, c three types, and the free degrees corresponding to the three types of levels are A, B, C, which are dynamically determined by the capacity awareness module according to the free space; finally, each level is enqueued according to the priority size.

[0018] Preferably, the Victim Selection Module selects the level from the priority queue as the input level of the compaction compression in turn, determines the compression input level, selects the victim SST from the level, and selects the overlapping SST from the lower layer to send to the merge engine for sorting and merging, and after the new data is generated after the sorting and merging, the previous data is invalidated, thereby generating invalid data.

[0019] Preferably, the Data Placement Module places the sorted new data of the merge compression engine into the corresponding new output level, and the background garbage collection operation also updates the position of the data in the LSM tree through the Data Placement Module after migrating the valid data, and this position relationship is the mapping relationship between the logical position of the file system in the log-structured merge tree level and the physical position on the ZNS device.

[0020] Therefore, the application adopts the above-mentioned dynamic hierarchical architecture for the partitioned namespace solid state disk, and proposes to dynamically select the input level for compression operation according to the current device free space and the fraction of the level by using the compression mechanism in the log-structured merge tree, thereby reducing the migration overhead of garbage collection, and compared with the traditional selection of the compression input level according to the fraction of the level, the problem of increased data migration overhead caused by the neglect of large amount of compressed data in the lower layer is solved.

[0021] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a solid state disk internal module diagram of an embodiment of the dynamic hierarchical architecture for the partitioned namespace solid state disk of the application;

[0023] Figure 2 is a capacity-aware module calculation flowchart of an embodiment of the dynamic hierarchical architecture for the partitioned namespace solid state disk of the application;

[0024] Figure 3 is a priority judgment module calculation flowchart of an embodiment of the dynamic hierarchical architecture for the partitioned namespace solid state disk of the application. DETAILED DESCRIPTION

[0025] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments.

[0026] Embodiment One

[0027] The application provides a dynamic hierarchical architecture for a partitioned namespace solid state disk, as shown in Figure 1As shown, the capacity-aware module, priority judgement module, Victim selection module and data placement module are pre-configured in the solid state disk. When the compaction scheduling engine dispatches a compression task, the capacity-aware module obtains the device free space capacity, and determines the weight of A, B and C categories for different levels. Then, in the priority judgement module, the level priority is calculated according to the free space capacity and the score of the level, and each level is enqueued according to the level priority to form a priority queue. Through the Victim selection module, the levels are taken out from the priority queue in turn as the input of compaction compression. After the input level is determined, it is sent to the merge compression engine together with the SST data overlapping with the lower level to perform merge sorting. Finally, the result is output to the specified location through the data placement module.

[0028] The specific steps are as follows:

[0029] First, the compaction scheduler module enqueues each compaction task, and when the system triggers compaction, a compaction job is taken out from the queue for execution of compaction.

[0030] The traditional LSM application implements compaction from the level with a high score, but this does not consider the free space condition of the device, resulting in high GC overhead of the ZNS device. The embodiment improves the priority calculation formula, and calculates the priority of each level through the capacity-aware module and the priority judgement module.

[0031] After starting to execute compaction, the capacity-aware module obtains the free space condition of the current device, and sets the free degree according to different scenarios . The more the free space, the less the GC is needed, so the value of the free degree in the formula is reduced, because the background GC will seriously affect the throughput of the upper layer application. The less the free space, the more the GC is needed to release space, so reducing the GC overhead to the greatest extent becomes the only way to improve system performance, so the value of the free degree in the formula is increased.

[0032] In addition, since the capacity threshold of the high level is obviously larger than that of the low level, the data is transferred from the upper layer to the lower layer layer by layer, so the lower layer of the larger capacity data has a greater probability of generating more invalid data. In this way, if sufficient space needs to be recycled, the GC recycling space has greater benefits in migrating the valid data of the zone where the high level is located, and can obtain less migration overhead, so the priority of the high level is lower than that of the low level, so as to try to make the high level participate in compaction to generate sufficient invalid data.

[0033] Subsequently, the priority judgment module determines the compaction priority of each level according to the And The compaction priority of each level is calculated, the levels are enqueued to form a priority queue, and after determining the priority of each level, the dequeue operation is performed to take out the level to execute the compaction operation.

[0034] The victim selection module selects the victim SSTable in the level according to the dequeued level, and then finds the SSTable with overlapping keys in the adjacent lower level, and sends it to the Merge Engine module to perform the merge operation to generate a new SSTable.

[0035] Finally, the data placement module in the file system associated with the ZNS SSD allocates a zone for the newly generated SSTable.

[0036] During the compaction compression process, the amount of invalid data is dynamically generated according to the free space, and when the free space is insufficient, the priority calculation formula selects the lower layer as the input to generate more invalid data. Because the current file system considers the data in the same level as having the same life span, and places the data in the same level into the same zone, the zone is directly reset to complete GC.

[0037] Research and analysis show that the life span of SST in the same level is not completely the same, so during garbage collection, not all data in the zone is invalid, and there is a lot of valid data, resulting in a relatively high migration overhead of valid data. In this embodiment, sufficient invalid data is generated to achieve the ideal condition for predicting the life span of data in the same level as much as possible, thereby reducing the impact of data migration on throughput.

[0038] Among them, the implementation and specific functions of the four modules are as follows:

[0039] The capacity-aware module is responsible for obtaining the free space capacity Remain_Cap of the device, so as to dynamically adjust the weight according to the remaining space. Three critical values are proposed to characterize the current device space status: LESS, MEDIUM, LARGE, which are closely related to the size of the device space and are not fixed. The three critical values are set as the ratio of the device storage space: 0.3, 0.5, 0.7, the larger the device storage space, the larger the three critical values. For example, assuming that the storage space of the ZNS SSD is 1TB, the sizes of LESS, MEDIUM, and LARGE are 300GB, 500GB, and 700GB, respectively.

[0040] The three thresholds are dynamically adjusted according to the space size, and the space part of the priority calculation is dynamically changed according to Remain_Cap, which is divided into four cases between the three thresholds, as shown in Figure 2

[0041] (1) Remina_Cap < LESS, the remaining free space is very small, and a large amount of invalid data needs to be recovered by GC, A = 3, B = 6, C = 9;

[0042] (2) LESS < Remina_Cap < MEDIUM, the remaining free space is small, and the space needs to be recovered by GC, A = 2, B = 4, C = 6;

[0043] (3) MEDIUM < Remain_Cap < LARGE, A = 1, B = 2, C = 3, the remaining space is relatively large, and the demand for recovering space is small;

[0044] (4) Remain_Cap > LARGE, A = 0.5, B = 1, C = 1.5, there is a large amount of remaining space, and almost no space needs to be recovered.

[0045] As shown in Figure 3 , the priority judgment module calculates the priority of each level according to the related information of the remaining capacity and the scores of each level. The calculation formula is: , the priority of is calculated. Among them, represents the weight, represents the degree of free space of the device, represents the score of the level. The levels in the LSM tree are divided into a, b, and c three types, such as in a five-layer LSM tree (L0-L5): L0 and L1 are a type, L2 and L3 are b type, and L4 and L5 are c type. Then the free degrees of the three types of levels are A, B, and C respectively, which are dynamically determined by the capacity perception module according to the free space. Finally, each level is enqueued according to the priority size.

[0046] The victim selection module selects levels from the priority queue in turn to select levels as the input level of compaction compression, determines the compression input level, selects the victim SST from the level, and selects the overlapping SST from the lower layer to send to the merging engine for merging and sorting. After the merging and sorting is completed, the previous data is invalidated, thereby generating invalid data.

[0047] ​The data placement module places the sorted new data of the merge compression engine into the corresponding new output level, and in addition, after the migration of valid data, the background garbage collection operation also updates the position of the data in the LSM tree through the data placement module, which is the mapping relationship between the logical position of the file system in the log-structured merge tree level and the physical position on the ZNS device.

[0048] Therefore, the application adopts the above-mentioned dynamic hierarchical architecture of the partition-oriented namespace solid-state disk, which can reduce the amount of valid data in the zone by dynamically selecting the compaction input level, thereby reducing the valid data migration overhead during garbage collection.

[0049] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: the technical solutions of the present application can still be modified or replaced by the same, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A dynamic tiered architecture for a partitioned namespace solid state disk, characterized by: In the solid state disk, the capacity awareness module, the priority judgment module, the victim selection module and the data placement module are pre-set. When the merge compression module scheduling engine assigns a compression task, the capacity awareness module obtains the device free space capacity, and determines the weights of A, B and C for different levels; Then in the priority judgment module, the priority of the level is calculated according to the free space capacity and the score of the level, and each level is enqueued according to the level priority to form a priority queue; Through the victim selection module, the level is taken out from the priority queue as the input of the merge compression module compression in turn, and after the input level is determined, the SST data overlapping with the lower layer is sent to the merge compression engine for merging and sorting; Finally, the result is output to the specified location through the data placement module; The capacity awareness module is responsible for obtaining the free space capacity Remain_Cap of the device, so as to dynamically adjust the weight according to the remaining space; Three critical values are proposed to characterize the space status of the current device: LESS, MEDIUM, and LARGE. The three critical values are set as the ratio of the device storage space: 0.3, 0.5, and 0.

7. The three critical values are closely related to the size of the device storage space. The larger the device storage space, the larger the three critical values. The three critical values are dynamically adjusted according to the space size, and the space part of the priority calculation dynamically changes according to Remain_Cap, which is divided into four cases between the three critical values: Remina_Cap<LESS, a large amount of invalid data needs to be recovered by GC, A=3, B=6, C=9; LESS<Remina_Cap<MEDIUM, GC is needed to recover space, A=2, B=4, C=6; MEDIUM<Remain_Cap<LARGE, A=1, B=2, C=3; Remain_Cap>LARGE, A=0.5, B=1, C=1.

5. 2.The dynamic hierarchical architecture for partition-oriented namespace solid state disk according to claim 1, wherein: In the process of merge compression module compression, the amount of invalid data is dynamically generated according to the free space. When the free space is insufficient, the priority calculation formula selects the lower layer as the input to generate more invalid data. The data in the same level is placed in the same area, and the GC is completed by directly resetting the area. 3.The dynamic hierarchical architecture for partition-oriented namespace solid state disk according to claim 1, wherein: The priority judging module calculates the priority of each level according to the related information of the residual capacity and the scores of each level; the calculation formula is: , the priority of is calculated. wherein, representing a weight, representing the degree of idleness of the device, representing the score of the level; dividing the levels in the log-structured merge tree into three categories a, b, c, and the corresponding degrees of idleness of the three categories are A, B, C, which are dynamically determined by the capacity-aware module according to the free space; finally, the levels are enqueued according to the priority size.

4. The dynamic tiered architecture for a zone-oriented namespace solid state disk according to claim 1, wherein: The victim selection module selects the level as the input level of the merge compression module compression by dequeuing from the priority queue in turn. After determining the compression input level, the victim SST is selected from the level, and the overlapping SST is selected from the lower layer and sent to the merge engine for merging and sorting. After the new data is generated by the merge sorting, the previous data is invalidated, thereby generating invalid data.

5. The dynamic tiered architecture for a zone-oriented namespace solid state disk according to claim 1, wherein: The data placement module places the sorted new data of the merge compression engine into the corresponding new output level. After the garbage collection operation in the background migrates the valid data, it also updates the position of the data in the log-structured merge tree through the data placement module. This position relationship is the mapping relationship between the logical position of the file system in the log-structured merge tree level and the physical position on the partition namespace solid state disk device.

Images