Efficient compression method of LSM-tree based on reserved space

Abstract

The invention relates to the technical field of computer networks and computer storage, and particularly discloses an efficient compression method of an LSM-tree based on a reserved space, which is characterized in that a proper space is reserved in each layer of the LSM-tree, so that the data volume compressed each time is reduced, and the coordination of the compressed data volume and system resources is realized. On one hand, according to the method, after the space is reserved, the execution time of compression can be shortened, and therefore occupation of a CPU is reduced. And on the other hand, the capacity of the LSM-tree level is reduced by deploying the reserved space strategy, so that the number of key value pairs with non-overlapped key value ranges is correspondingly reduced. And in the merging process, the size of the rewritten data is correspondingly reduced, so that the write amplification is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of computer networks and computer storage, and in particular to an efficient compression method for LSM-tree based on reserved space. Background Art

[0002] With the rapid growth of global data, data storage systems not only need to be able to handle huge amounts of data, but also need to have efficient storage, indexing, and retrieval capabilities. Log-structured merge tree (LSM-tree for short) is a data structure for efficient storage and retrieval of data. It is mainly used to handle write and read operations in large-scale data storage systems. This is mainly due to the fact that LSM-tree converts random write requests into sequential write requests and periodically reorganizes the data layout in the memory through an operation called compression. Key-value storage usually uses LSM-tree as the storage engine to achieve high throughput write operations and competitive read performance.

[0003] When the size of the data stored in the i-th layer (0≤i≤6) of the LSM-tree reaches the capacity, the compression operation of this layer will be triggered. First, a file in the i-th layer will be selected and read into the memory; second, the files in the i+1th layer that overlap with the key value range of the file selected in the i-th layer are also selected and read into the memory; third, these selected files need to perform a merge sort operation, that is, the old versions of the key-value pairs with the same key are discarded, and then arranged in order according to the size of the key; finally, the sorted key-value pairs will be rewritten to the i+1th layer, and the original selected files are marked as obsolete and deleted from the memory.

[0004] From the execution mechanism of compression, it can be seen that it plays an important role in the key-value storage system based on LSM-tree, and has a significant impact on system performance and storage efficiency. However, compression will bring the following problems.

[0005] 1) Write amplification: During the compression process, the system needs to rewrite data to save valid data and remove invalid data. This results in the amount of data actually written to the storage medium being much larger than the original data amount;

[0006] 2) Long-term CPU usage: The compression operation needs to calculate whether the key-value pair is valid, which requires the continuous participation of the CPU. Long-term CPU usage will lead to system performance degradation, especially during the compression period, which may block other read and write requests and affect the system's response speed. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present invention provides an efficient compression method for LSM-tree based on reserved space. By reserving a suitable space size in each layer of the LSM-tree, the amount of data compressed each time is reduced, thereby achieving coordination between the amount of compressed data and system resources. On the one hand, after reserving space, the method can reduce the execution time of compression, thereby reducing the occupancy of the CPU. On the other hand, the method also effectively reduces write amplification, solving the problems mentioned in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: an efficient compression method of LSM-tree based on reserved space, comprising the following steps:

[0009] S1. Initialize the storage system and obtain the total number of LSM-tree layers and the capacity of each layer of the LSM-tree C. i ;

[0010] S2, determine whether to adopt the reserved space strategy, if it is necessary to adopt, execute the following steps, otherwise jump to step S5;

[0011] S3. Calculate the reservation rate r of each layer of LSM-tree i ;

[0012] S4, according to the reservation rate r obtained in step S3 i , calculate the new capacity C' of the reserved layer i ;

[0013] S5. If the reserved space strategy is adopted, the capacity of the layer is C' i , otherwise the capacity of the level remains C i ;

[0014] S6. According to the capacity of the layer, determine whether the data size of the layer is full. If it is full, execute the following steps; otherwise, do not perform additional operations;

[0015] S7. If the data size of the layer exceeds the capacity, that is, the data is full, a compression operation on the layer is triggered;

[0016] S8. Determine whether the data writing is completed. If so, complete the entire reserved space mechanism. Otherwise, continue to load data and jump to step S6 to determine whether a compression operation needs to be performed.

[0017] Preferably, in step S3, the LSM-tree is set to 7 layers, i.e., 0≤i≤6, and there is no need to reserve space in the last layer; the reservation rate of each layer is statically set to r0=0, r1=0.50, r2=0.25, r3=0.12, r4=0.06, r5=0.03, respectively.

[0018] Preferably, in step S3, the reservation rate r i The calculation formula is as follows:

[0019]

[0020] Among them, C i is the capacity of the i-th layer of the LSM-tree, R i is the size of the reserved space at the i-th level of the LSM-tree, and R i With C i The relationship satisfies 0≤R i <C i .

[0021] Preferably, in step S4, according to the reservation rate r obtained in step S3 i , deploy reserved space on each layer of the LSM-tree. After deploying the reserved space, the capacity of each layer increases from C i Changes to C' i ; New capacity C' i The calculation formula is as follows:

[0022] C′ i =(1-r i )·C i ;

[0023] Among them, C' i ≤C i .

[0024] Preferably, in step S7, if the reserved space strategy is adopted, when the amount of data in the LSM-tree level reaches C' i After that, the compression operation will be triggered; if the reserved space strategy is not adopted, the amount of data in the LSM-tree level must reach C i After that, compression of the layer is triggered.

[0025] The beneficial effects of the present invention are:

[0026] 1) Reduced write amplification: The compression operation of LSM-tree shows high write amplification mainly due to the edge merging problem. That is, the files selected for merging usually do not have completely overlapping key value ranges. Therefore, each compression will unnecessarily rewrite some non-overlapping data. The deployment of the reserved space strategy reduces the capacity of the LSM-tree hierarchy. Therefore, the number of key-value pairs with non-overlapping key value ranges is also reduced accordingly. During the merging process, the size of the rewritten data is also reduced accordingly, which effectively reduces write amplification.

[0027] 2) Reduced CPU usage: In order to reduce CPU usage, this paper reduces the execution time of the compression operation. After the reserved space policy is deployed, the compression operation is triggered in advance. At this time, the amount of data merged each time is effectively reduced. However, since the compression performed in the storage setting is relatively slow, which does not match the fast write operation in the memory, there will be write pauses and the compression time will increase. In order to coordinate the compression and write speeds, the reserved space policy ensures the match between the merged data size and system resources, thereby reducing the compression time. This effectively avoids long-term CPU usage. It is worth noting that the reserved space policy is implemented as application support for existing key-value storage, which only requires setting parameters without incurring additional overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is an architecture diagram of an efficient compression method of LSM-tree based on reserved space in the present invention;

[0029] Figure 2 The present invention is a flow chart of an efficient compression method of LSM-tree based on reserved space. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1-Figure 2 The present invention provides a technical solution: an efficient compression method of LSM-tree based on reserved space, wherein the reserved space reservation scheme is divided into three processes: calculation of reserved ratio, deployment of reserved space and triggering of reserved scheme. i (0≤r i ≤1) is the ratio of the reserved space to the allocated space of the i-th layer of the LSM-tree (i.e., the reservation rate), C i (MB) is the capacity of the i-th layer of the LSM-tree (also called the size of the allocated space), R i (MB) is the size of the reserved space at the i-th level of the LSM-tree. i With C i The relationship satisfies Formula 1.

[0032] 0≤R i <C i (Formula 1)

[0033] Calculation process of reservation ratio: First, we need to determine the size of the reserved space, where the reservation ratio r i It is expressed as Formula 2. At this time, the storage system sets a static reservation rate (it can also be adjusted dynamically according to the characteristics of the workload). Figure 1 As shown in the figure, the LSM-tree usually has 7 layers (i.e., 0≤i≤6), but the last layer does not trigger compression. Therefore, there is no need to reserve space in the last layer. The reservation rate of each layer is statically set to r0=0, r1=0.50, r2=0.25, r3=0.12, r4=0.06, r5=0.03.

[0034]

[0035] Deployment process of reserved space: After determining the reservation rate, the reserved space needs to be deployed on each layer of the LSM-tree. After the reserved space is deployed, the capacity of each layer increases from C i Changes to C' i Among them, C' i Satisfies formula 3. It can be seen that: C' i ≤C i At this point, the capacity of each layer is effectively reduced.

[0036] C' i =(1-r i )·C i (Formula 3)

[0037] Triggering process of the reservation scheme: When the amount of data stored in a layer of the LSM-tree reaches the capacity of the layer, the compression operation of the layer will be triggered. When the reserved space is not deployed, the amount of data in the LSM-tree layer must reach C i After the reserved space is deployed, when the amount of data in the LSM-tree layer reaches C' i The compression operation will be triggered after the data is compressed. This allows compression to be triggered in advance. Since the capacity is effectively reduced, the amount of data compressed each time is reduced, thus achieving coordination between the amount of compressed data and system resources.

[0038] An efficient compression method for LSM-tree based on reserved space, such as Figure 2 As shown, the implementation steps include the following:

[0039] (1) Initialize the storage system and obtain the total number of existing LSM-tree layers and the capacity of each layer of the LSM-tree C i ;

[0040] (2) Determine whether to adopt the reserved space strategy. If so, execute the following steps; otherwise, jump to step (5);

[0041] (3) Calculate the reservation rate r of each layer of the LSM-tree according to formula 2 i ;

[0042] (4) According to the reservation rate r obtained in step (3) i , calculate the new capacity C' of the reserved layer by formula 3 i ;

[0043] (5) If the reserved space strategy is adopted, the capacity of the layer is C' i , otherwise the capacity of the level remains C i ;

[0044] (6) Based on the capacity of the layer, determine whether the data size of the layer is full. If it is full, execute the following steps, otherwise no additional operations are performed;

[0045] (7) If the data size of the layer exceeds the capacity (i.e., the data is full), a compression operation on the layer is triggered;

[0046] (8) Determine whether the writing of data is completed. If it is completed, the entire reserved space mechanism is completed, otherwise continue to load data and jump to step (6) to determine whether a compression operation needs to be performed. At this point, the implementation steps of the efficient compression method based on LSM-tree with reserved space of the present invention are all completed.

[0047] In the embodiments provided by the present invention, it should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0048] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0049] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0050] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0051] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0052] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient compression method for LSM-tree based on reserved space, characterized in that: The steps include: S1. Initialize the storage system and obtain the total number of LSM-tree layers and the capacity of each layer of the LSM-tree C. i ; S2, determine whether to adopt the reserved space strategy, if it is necessary to adopt, execute the following steps, otherwise jump to step S5; S3. Calculate the reservation rate r of each layer of LSM-tree i ; S4, according to the reservation rate r obtained in step S3 i , calculate the new capacity C' of the reserved layer i ; S5. If the reserved space strategy is adopted, the capacity of the layer is C' i , otherwise the capacity of the level remains C i ; S6. According to the capacity of the layer, determine whether the data size of the layer is full. If it is full, execute the following steps; otherwise, do not perform additional operations; S7. If the data size of the layer exceeds the capacity, that is, the data is full, a compression operation on the layer is triggered; S8. Determine whether the data writing is completed. If so, complete the entire reserved space mechanism. Otherwise, continue to load data and jump to step S6 to determine whether a compression operation needs to be performed.

2. The efficient compression method of LSM-tree based on reserved space according to claim 1, characterized in that: In step S3, the LSM-tree is set to have 7 layers, i.e., 0≤i≤6, and there is no need to reserve space in the last layer; the reservation rate of each layer is statically set to r0=0, r1=0.50, r2=0.25, r3=0.12, r4=0.06, and r5=0.

03.

3. The efficient compression method of LSM-tree based on reserved space according to claim 1, characterized in that: In step S3, the reservation rate r i The calculation formula is as follows: Among them, C i is the capacity of the i-th layer of the LSM-tree, R i is the size of the reserved space at the i-th level of the LSM-tree, and R i With C i The relationship satisfies 0≤R i <C i .

4. The efficient compression method of LSM-tree based on reserved space according to claim 1, characterized in that: In step S4, according to the reservation rate r obtained in step S3 i , deploy reserved space on each layer of the LSM-tree. After deploying the reserved space, the capacity of each layer increases from C i Changes to C' i ; New capacity C' i The calculation formula is as follows: C’ i =(1-r i )·C i ; Among them, C' i ≤C i .

5. The efficient compression method of LSM-tree based on reserved space according to claim 1, characterized in that: In step S7, if the reserved space strategy is adopted, when the amount of data in the LSM-tree level reaches C' i After that, the compression operation will be triggered; if the reserved space strategy is not adopted, the amount of data in the LSM-tree level must reach C i After that, compression of the layer is triggered.

Images