LSM tree read optimization design method and system based on dynamic layer height

Abstract

The application provides a dynamic layer height-based LSM tree read optimization design method and system, comprising the following steps: S1: when a preset condition is triggered, a layer is added to ensure that the write performance is not affected when the write traffic is large, and a layer is reduced to improve the read performance when another preset condition is triggered; S2: the background merging type in the LSM tree-based key value storage is divided into ordinary merging and layer reduction merging, and is executed by different threads respectively, and the LSM tree architecture with the number of layers being dynamically adjusted according to the work load. The application introduces the method of dynamically adjusting the number of layers in the LSM tree-based key value storage, can automatically adjust the layer number parameter according to the current work load, achieves the real-time optimal performance, releases the user from the complicated work of manually adjusting the layer number parameter, and solves the read amplification problem caused by the LSM tree structure to a certain extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of key-value storage of LSM tree, in particular, to a design method and system for LSM tree read optimization based on dynamic layer height. BACKGROUND

[0002] LSM tree is a persistent data structure optimized for write performance. It converts random writes into sequential writes to fully utilize the bandwidth of underlying storage media. LevelDB is a classic key-value store based on LSM tree. The in-memory structure of LevelDB consists of a MemTable and an immutable MemTable, both of which are implemented as skip lists to support efficient insertion and query. LevelDB divides its on-disk structure into seven levels, each of which consists of many files. The capacity limit of each level increases by 10 times, and LevelDB guarantees that the key ranges of different files in each level do not overlap (except for the first level).

[0003] To store key-value pairs, LevelDB first inserts key-value pairs into the MemTable. When the MemTable reaches its capacity limit, it will be converted into an immutable MemTable and flushed to the first level as a file. When the number of files in the xth level reaches the limit, some files will be selected to be merged with files in the x+1th level.

[0004] To find a key-value pair, LevelDB searches the key in the MemTable, the immutable MemTable, and the files in the first to seventh levels in turn. Because only the files in the first level can have overlapping keys, all files in the first level whose key range contains the lookup key need to be searched; all files in the layers below the first layer only need to be searched one. For the lookup in the file, first read the index block and use its index information to locate the target data block. Then read the filter block, which is used to quickly determine whether the target data block contains the key to be looked up. Finally, if the filter block determines that it exists, read the data block for further lookup.

[0005] The multi-layer structure of LSM tree forces read requests to traverse each level from top to bottom, resulting in read amplification and affecting read performance. However, directly reducing the number of layers to two will cause the amount of data involved in merging to be too large, slowing down the merging progress and seriously blocking foreground read and write operations.

[0006] Therefore, how to design an efficient solution to the read amplification problem caused by the multi-layer structure of LSM tree without affecting write performance is crucial to improving the overall performance of key-value storage systems based on LSM tree.

[0007] Patent document CN108804625B discloses an optimization method, device and computer equipment of LSM tree, the method comprises: determining the target leaf node pointed to in the LSM tree, wherein the data has been deleted; adding a deletion mark for the target leaf node in the dynamic data of the LSM tree. However, the invention does not design a set of efficient solutions for the read amplification problem caused by the multi-layer structure of the LSM tree without affecting the write performance. SUMMARY

[0008] In view of the defects in the prior art, the purpose of the present application is to provide a LSM tree read optimization design method and system based on dynamic layer height.

[0009] According to the LSM tree read optimization design method based on dynamic layer height provided by the present application, the method comprises:

[0010] Step S1: when the write traffic is large, a layer is added to ensure that the write performance is not affected, and when the write traffic is small, a layer is reduced to improve the read performance;

[0011] Step S2: the background merging type in the key-value storage based on the LSM tree is divided into ordinary merging and layer reduction merging, and different threads are used for execution, and the LSM tree architecture with dynamic adjustment of the number of layers is adjusted according to the workload.

[0012] Preferably, in the step S1:

[0013] When the write traffic is large, and the second-to-last layer exceeds the capacity limit, a layer is added below the first layer before the ordinary merging starts, and the capacity limit of all layers below it is expanded by ten times to prevent the ordinary merging from blocking the foreground read-write operation;

[0014] When the write traffic is small, and the layer reduction merging flushes all data in the second-to-last layer into the last layer, the second-to-last layer is deleted.

[0015] Preferably, in the step S1:

[0016] The number of layers is changed by changing the data structure in the memory, and the files in the disk are not moved.

[0017] Preferably, in the step S2:

[0018] The ordinary merging flushes the data in the upper layer into the second-to-last layer, and whether to perform the layer adding operation is judged by detecting whether the second-to-last layer exceeds the capacity limit before the ordinary merging starts each time;

[0019] The layer reduction merging flushes the data in the second-to-last layer into the last layer, and whether to perform the layer reduction operation is judged by detecting whether the second-to-last layer still has data after the layer reduction merging ends each time.

[0020] Preferably, in the step S2:

[0021] The layer reduction merge and the normal merge are run in the background and are executed by different threads respectively.

[0022] According to the LSM tree read optimization design system based on dynamic layer height provided by the application, comprising:

[0023] Module M1: when the write traffic is large, a layer is added to ensure that the write performance is not affected when a preset condition is triggered, and when the write traffic is small, a layer is reduced to improve the read performance when another preset condition is triggered;

[0024] Module M2: the background merge types in the key value storage based on the LSM tree are divided into normal merge and layer reduction merge, and are executed by different threads respectively, and the LSM tree architecture with dynamic adjustment of the layer number is adjusted according to the workload.

[0025] Preferably, in the module M1:

[0026] When the write traffic is large, a layer is added below the first layer before the normal merge starts, and the capacity limit of all layers below it is expanded by ten times to prevent the normal merge from blocking the foreground read-write operation when the second-to-last layer exceeds the capacity limit.

[0027] When the write traffic is small, the second-to-last layer is deleted when all the data in the second-to-last layer is flushed into the last layer by the layer reduction merge.

[0028] Preferably, in the module M1:

[0029] The layer number change is realized by changing the data structure in the memory, and the files in the disk are not moved.

[0030] Preferably, in the module M2:

[0031] The normal merge flushes the data in the upper layer into the second-to-last layer, and whether the layer adding operation is performed is judged by detecting whether the second-to-last layer exceeds the capacity limit before the normal merge starts each time.

[0032] The layer reduction merge flushes the data in the second-to-last layer into the last layer, and whether the layer reduction operation is performed is judged by detecting whether the second-to-last layer still has data after the layer reduction merge ends each time.

[0033] Preferably, in the module M2:

[0034] The layer reduction merge and the normal merge are run in the background and are executed by different threads respectively.

[0035] Compared with the prior art, the application has the beneficial effects as follows:

[0036] The application introduces a method of dynamically adjusting the number of layers in the LSM tree-based key-value storage, which can automatically adjust the number of layers according to the current workload, so as to achieve real-time optimal performance, and liberate the user from the tedious work of manually adjusting the number of layers, and to a certain extent, solve the read amplification problem caused by the structure of the LSM tree. BRIEF DESCRIPTION OF DRAWINGS

[0037] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, which is to be read in conjunction with the accompanying drawings:

[0038] Figure 1 A schematic diagram of the layer adding operation in the application;

[0039] Figure 2 A schematic diagram of the layer reducing operation in the application;

[0040] Figure 3 A schematic diagram of the two kinds of merging methods in the application. DETAILED DESCRIPTION

[0041] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that, for those skilled in the art, without departing from the concept of the application, a number of changes and improvements can be made. These all belong to the protection scope of the application.

[0042] Example 1

[0043] The application provides an LSM tree read optimization design method based on dynamic layer height, including: 1. a design method for dynamically changing layer height according to write load, which increases a layer when the write traffic is large to ensure that the write performance is not affected, and reduces a layer when the write traffic is small to improve the read performance; 2. Subdivide the background merging into ordinary merging and layer-reducing merging, and execute them with different threads respectively. The application designs an LSM tree architecture that can dynamically adjust the number of layers according to the workload, which can greatly improve the read performance of the LSM tree-based key-value storage without sacrificing the write performance.

[0044] According to the LSM tree read optimization design method based on dynamic layer height provided by the application, as shown in Figures 1-3 , it comprises:

[0045] Step S1: When the write traffic is large and a preset condition is triggered, a layer is added to ensure that the write performance is not affected, and when the write traffic is small and another preset condition is triggered, a layer is reduced to improve the read performance;

[0046] Specifically, in the step S1:

[0047] When the write traffic is large, causing the second last layer to exceed the capacity limit, a layer is added below the first layer before the normal merge starts, and the capacity limit of all layers below it is expanded by ten times, preventing the normal merge from blocking foreground read-write operations.

[0048] When the write traffic is small, the delete layer merge flushes all data in the second last layer into the last layer, and the second last layer is deleted.

[0049] Specifically, in the step S1:

[0050] The number of layers is changed by changing the data structure in the memory, and the file in the disk is not moved.

[0051] Step S2: The background merge type in the LSM tree-based key-value storage is divided into normal merge and delete layer merge, and is executed by different threads respectively, and the number of layers of the LSM tree architecture is dynamically adjusted according to the workload.

[0052] Specifically, in the step S2:

[0053] The normal merge flushes the upper layer data into the second last layer, and detects whether the second last layer exceeds the capacity limit before each normal merge starts, and judges whether to perform the layer adding operation;

[0054] The delete layer merge flushes the data in the second last layer into the last layer, and detects whether the second last layer still has data after each delete layer merge ends, and judges whether to perform the layer deleting operation.

[0055] Specifically, in the step S2:

[0056] The delete layer merge and the normal merge run in the background and are executed by different threads respectively.

[0057] Embodiment 2:

[0058] Embodiment 2 is a preferred example of embodiment 1, to more specifically illustrate the present application.

[0059] The present application also provides a dynamic layer height-based LSM tree read optimization design system, which can be realized by executing the process steps of the dynamic layer height-based LSM tree read optimization design method, that is, those skilled in the art can understand the dynamic layer height-based LSM tree read optimization design method as a preferred embodiment of the dynamic layer height-based LSM tree read optimization design system.

[0060] According to the dynamic layer height-based LSM tree read optimization design system provided by the present application, it comprises:

[0061] Module M1: when the write traffic is large, a layer is added to ensure that the write performance is not affected when a preset condition is triggered, and when the write traffic is small, a layer is reduced to improve the read performance when another preset condition is triggered;

[0062] Specifically, in the module M1:

[0063] When the write traffic is large, causing the second-to-last layer to exceed the capacity limit, a layer is added below the first layer before the normal merge starts, and the capacity limit of all layers below it is expanded by ten times to prevent the normal merge from blocking the foreground read-write operation.

[0064] When the write traffic is small, the second-to-last layer is deleted when the reduce-layer merge flushes all data in the second-to-last layer to the last layer.

[0065] Specifically, in the module M1:

[0066] The number of layers is changed by changing the data structure in the memory, and the file in the disk is not moved.

[0067] Module M2: the background merge type in the LSM tree-based key-value storage is divided into normal merge and reduce-layer merge, and different threads are used to execute them, and the number of layers of the LSM tree architecture is dynamically adjusted according to the workload.

[0068] Specifically, in the module M2:

[0069] The normal merge flushes the data in the upper layer to the second-to-last layer, and detects whether the second-to-last layer exceeds the capacity limit before each normal merge starts to determine whether to perform the layer addition operation.

[0070] The reduce-layer merge flushes the data in the second-to-last layer to the last layer, and detects whether the second-to-last layer still has data after each reduce-layer merge ends to determine whether to perform the layer reduction operation.

[0071] Specifically, in the module M2:

[0072] The reduce-layer merge and the normal merge run in the background and are executed using different threads.

[0073] Embodiment 3:

[0074] Embodiment 3 is a preferred example of Embodiment 1, which more specifically illustrates the present application.

[0075] According to the LSM tree read optimization design method based on dynamic layer height provided by the present application, comprising:

[0076] The design method for dynamically changing the layer height according to the write load includes the following steps: if the current foreground write flow is too large, a layer is added below the first layer to ensure that the write performance is not reduced; if the current foreground write flow is not large, a layer is reduced to improve the read performance, and the second last layer is deleted;

[0077] The merge type is increased: the background merge type in the LSM tree-based key-value storage is subdivided into normal merge and layer reduction merge to facilitate layer reduction operation.

[0078] Preferably, the design method for dynamically changing the layer height according to the write load comprises:

[0079] When the second last layer exceeds the capacity limit due to large write flow, a layer is added below the first layer before the normal merge starts, and the capacity limit of all layers below it is expanded by ten times to prevent normal merge from blocking foreground read and write operations. When the write flow is small, the layer reduction merge flushes all data in the second last layer to the last layer, and the second last layer is deleted to reduce read amplification and improve read performance.

[0080] Preferably, the increase of the merge type comprises:

[0081] The execution method of layer reduction merge and normal merge is similar, except that layer reduction merge is specifically for the merge between the second last layer and the last layer, and in order to make the two merges not affect each other, two different threads are used to execute the corresponding merge.

[0082] The normal merge will continuously flush the data in the upper layer to the second last layer, and before each normal merge starts, it is detected whether the second last layer exceeds the capacity limit to determine whether to perform the layer addition operation.

[0083] The layer reduction merge will continuously flush the data in the second last layer to the last layer, and after each layer reduction merge ends, it is detected whether the second last layer still has data to determine whether to perform the layer reduction operation.

[0084] The merge method of layer reduction merge is similar to that of normal merge, except that layer reduction merge is specifically for the merge between the second last layer and the last layer.

[0085] Layer reduction merge and normal merge run in the background together and are executed by different threads.

[0086] The present application provides a design method for LSM tree read optimization based on dynamic layer height, including the following two aspects (in order to simplify the description, n represents the total number of layers in the following description):

[0087] I. The design method for dynamically changing the layer height according to the write load includes two operations:

[0088] 1. As shown in Figure 1 , the add - layer operation adds a new layer between L0 and L1 and increments n by 1. As a result, each layer below L0 becomes deeper (i.e., L x becomes L x+1 ), and the capacity is multiplied by 10. When write operations are intensive, adding a layer can significantly reduce the merge pressure. It should be noted that the change in the number of layers is achieved by changing the data structure in memory, and in fact, the files on disk are not moved.

[0089] 2. As shown in Figure 2 , the remove - layer operation deletes the empty L n-2 and decrements n by 1. The remove - layer operation is only valid when L n-2 does not contain key - value pairs and n is greater than 2. Deleting a layer can improve the performance of read operations.

[0090] II. As shown in Figure 3 , to adapt to the change in levels, two different merge methods need to be distinguished:

[0091] 1. Normal merge is the merge between L i and L i+1 (i >= 0 and i < n - 2). Normal merge is the merge method used in traditional LSM - trees (such as LevelDB), which can ensure that the capacity ratio of adjacent layers is approximately 1:10.

[0092] 2. Remove - layer merge is the merge between L n-2 and L n-1 . The remove - layer merge operates in the same way as normal merge, but it is special because the capacity of L n-1 is infinite. The remove - layer merge also serves as an indicator of whether n should change. When the foreground write operations are intensive and the remove - layer merge cannot flush all the data in L n-2 into L n-1 in time, L n-2 will exceed its capacity limit. Then, the add - layer operation will be triggered to add a layer so that write operations will not be blocked. When the amount of foreground write operations becomes less, the remove - layer merge can flush all the content in L n-2 into L n-1 in time, thus triggering the remove - layer operation and improving the read performance. After that, the remove - layer merge continues to perform merge operations between the new L n-2 and L n-1 and continues to trigger the remove - layer operation until n is reduced to 2.

[0093] In summary, the application can increase the layers when the write traffic is large and reduce the layers when the write traffic is small, so that the number of layers changes in real time according to the specific workload, to achieve real-time optimal read and write performance, and to solve the read amplification problem of the LSM tree to a certain extent.

[0094] Those skilled in the art know that, in addition to implementing the system and each device, module and unit thereof provided by the application in the form of pure computer readable program code, the system and each device, module and unit thereof provided by the application can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system and each device, module and unit thereof provided by the application can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures in the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules for implementing methods and structures in the hardware component.

[0095] The specific embodiments of the application are described above. It needs to be understood that the application is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. The embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily without conflict.

Claims

1. A read optimization design method for an LSM tree based on dynamic layer height, characterized in that, include: Step S1: When the write traffic is high and a preset condition is triggered, add a layer to ensure that the write performance is not affected; when the write traffic is low and another preset condition is triggered, remove a layer to improve the read performance. Step S2: Divide the background merge type in the LSM tree-based key-value store into normal merge and de-leveling merge, and execute them with different threads, dynamically adjusting the LSM tree architecture of the number of levels according to the workload.

2. The LSM tree read optimization design method based on dynamic layer height according to claim 1, characterized in that, In step S1: When the write traffic is large enough to cause the second-to-last layer to exceed its capacity limit, a new layer is added below the first layer before the normal merge begins, and the capacity limit of all layers below it is expanded tenfold to prevent the normal merge from blocking the foreground read and write operations. When the write traffic is low, the second-to-last layer is deleted when the layer reduction and merging process flushes all the data from the second-to-last layer to the last layer.

3. The LSM tree read optimization design method based on dynamic layer height according to claim 1, characterized in that, In step S1: The layer change is achieved by altering the data structure in memory; the files on the disk are not moved.

4. The LSM tree read optimization design method based on dynamic layer height according to claim 1, characterized in that, In step S2: Normal merging flushes the upper layer data to the second-to-last layer. Before each normal merging begins, it checks whether the second-to-last layer exceeds the capacity limit and determines whether to perform a layer addition operation. The layer reduction and merging process flushes the data from the second-to-last layer to the last layer. After each layer reduction and merging process, it checks whether there is still data in the second-to-last layer to determine whether to perform the layer reduction operation.

5. The LSM tree read optimization design method based on dynamic layer height according to claim 1, characterized in that, In step S2: Layer reduction merging and regular merging run in the background and are executed using different threads.

6. A read optimization design system for an LSM tree based on dynamic layer height, characterized in that, include: Module M1: When the write traffic is high and a preset condition is triggered, an additional layer is added to ensure that the write performance is not affected. When the write traffic is low and another preset condition is triggered, an additional layer is removed to improve the read performance. Module M2: Divides the background merge type in the LSM tree-based key-value store into normal merge and decrement merge, and executes them with different threads, dynamically adjusting the number of layers in the LSM tree architecture according to the workload.

7. The LSM tree read optimization design system based on dynamic layer height according to claim 6, characterized in that, In module M1: When the write traffic is large enough to cause the second-to-last layer to exceed its capacity limit, a new layer is added below the first layer before the normal merge begins, and the capacity limit of all layers below it is expanded tenfold to prevent the normal merge from blocking the foreground read and write operations. When the write traffic is low, the second-to-last layer is deleted when the layer reduction and merging process flushes all the data from the second-to-last layer to the last layer.

8. The LSM tree read optimization design system based on dynamic layer height according to claim 6, characterized in that, In module M1: The layer change is achieved by altering the data structure in memory; the files on the disk are not moved.

9. The LSM tree read optimization design system based on dynamic layer height according to claim 6, characterized in that, In module M2: Normal merging flushes the upper layer data to the second-to-last layer. Before each normal merging begins, it checks whether the second-to-last layer exceeds the capacity limit and determines whether to perform a layer addition operation. The layer reduction and merging process flushes the data from the second-to-last layer to the last layer. After each layer reduction and merging process, it checks whether there is still data in the second-to-last layer to determine whether to perform the layer reduction operation.

10. The LSM tree read optimization design system based on dynamic layer height according to claim 6, characterized in that, In module M2: Layer reduction merging and regular merging run in the background and are executed using different threads.

Images