Pre-Compaction method and system applied to LSM tree structure

Abstract

The invention discloses an advanced Compaction method and system applied to an LSM tree structure, and relates to the technical field of computer storage, and the method comprises the following steps: when the data volume stored in the i-th layer Li of an LSM tree reaches a threshold value, triggering Compaction operation; sSTable of a plurality of subsequent continuous Compaction operations to be executed in the Li + 1 layer of the (i + 1) th layer and the Li layer is selected to be carried out simultaneously in advance to the first Compaction operation; all the Compaction operation tasks are put into a task queue; and judging and processing according to the key range overlapping condition of the SSTable, and finally finishing the work of moving the SSTable from the Li layer to the Li + 1 layer by merging into a large Compaction operation or directly modifying hierarchical metadata. According to the invention, an advanced Compaction model is provided, so that the problem of additional read-write amplification caused by range overlapping in a continuous Compaction operation process is effectively relieved. Meanwhile, a file granularity assembly line scheme is added to optimize the proposed advanced Compaction model, and optimization of the storage system based on the LSM tree is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer storage technology, and in particular to an early compaction method and system applied to an LSM tree structure. Background Art

[0002] With the advent of the digital age, people are increasingly concerned about data storage and management. Traditional storage architectures are not effective in dealing with large amounts of high-frequency data writing. In this context, the LSM tree-based key-value storage system came into being. As an innovative data structure, the LSM tree structure provides an effective solution to the problems of traditional data storage.

[0003] LSM tree uses append-only storage to implement sequential write operations. Due to this feature, LSM tree provides excellent write performance. The main components of LSM tree include log files on disk, sorted string tables SSTable on disk, and mutable MemTable in memory. When new write and update requests arrive, they are first written to the log file on disk and then written to the MemTable in memory. When the current MemTable capacity reaches its threshold, a new MemTable and log file will be created to store subsequent write and update requests. In the background, the previous mutable MemTable will be converted into an immutable Immutable MemTable, and then the compaction thread will flush it to disk. A new SSTable will be generated in the L0 layer of the LSM tree, and the previous log file can be discarded. When the data refreshed from memory to the L0 layer reaches the threshold of the amount of data that the L0 layer can store, the SSTable in the L0 layer will be refreshed and merged into the L1 layer through the Compaction operation. The specific process of the Compaction operation is as follows: When L i When the data storage capacity of a layer reaches its threshold, it is first i Select an SSTable from the L i+1 Select all key ranges in the layer with L i The SSTables that intersect the key range of the SSTable are then read into memory for merge sorting to generate new files, which are then written back to disk.

[0004] Although the global order between SSTables can be maintained during the compaction operation of the LSM tree, the frequent rewrite operations during this period will cause the problem of read-write amplification. When the amount of data is very large, the compaction operation may also cause write pauses, resulting in reduced system performance. Many studies have also proposed some solutions, such as using key-value separation to reduce the amount of data read during the compaction operation, but this solution does not take into account the relationship between consecutive merge operations of adjacent layers. In the actual operation process, for consecutive merge operations of adjacent layers, if L i The key range selected by the layer is always the same as L i+1 If the key ranges of the layers overlap, the overlapping data will be read into memory multiple times for merge sorting and then written back to disk, resulting in unnecessary I / O amplification.

[0005] In summary, a strategy is needed that can ensure that the compaction operation can maintain the ordered state between SSTables while alleviating the additional read-write amplification problem caused by range overlap during continuous compaction operations. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art.

[0007] The technical solution adopted by the present invention to solve the technical problem is: to provide an early compaction method applied to an LSM tree structure, comprising the following steps:

[0008] When the i-th level L of the LSM tree i When the amount of stored data reaches its threshold, the Compaction operation is triggered;

[0009] Select the i+1th layer L i+1 With L i The SSTables of multiple consecutive compaction operations to be performed in the layer are read into the task queue;

[0010] After reading multiple SSTables into the task queue, it is determined whether to perform a large compaction operation or only modify the metadata to implement SSTable in L based on the overlap of different key ranges. i Layer and L i+1 Requests to move between tiers.

[0011] Preferably, the method determines whether to perform a large Compaction operation or only modify metadata to implement SSTable in L according to different key range overlap situations. i Layer and L i+1A request to move between tiers consists of the following steps:

[0012] The task queue is divided into two operation queues according to whether there is a key range overlap in SSTable. i Layer and L i+1 If there is no range overlap between SSTables in the layer, they are assigned to the first operation queue; if L i Layer and L i+1 If there is range overlap between SSTables in a layer, they are assigned to the second operation queue;

[0013] For the first operation queue, only modify L i Layer and L i+1 The metadata of the layer, then L i The SSTable of the layer is inserted into L i+1 For the second operation queue, multiple subsequent continuous compaction operation tasks are merged with the current compaction operation task into a large compaction operation task and executed together, so as to realize the SSTable from L i Layer and L i+1 Migration of layers.

[0014] Preferably, there are three situations in the second operation queue, and different processing is performed according to different situations, including:

[0015] For intersecting and disjoint relationships, find the newly generated SSTable in L after the Compaction operation task is executed. i+1 The location of the layer makes L i+1 The layer is globally ordered. SSTables that overlap with multiple compaction operations only need to be added to the queue once.

[0016] For the separated relationship, set an SSTable interval stop flag. When generating a new SSTable, the SSTable will no longer be written before the minimum key of the interval SSTable, and then a new SSTable will be created to accept the subsequent data writing.

[0017] Preferably, for the second operation queue, in order to speed up the write back speed of the SSTable, the write back of the previous SSTable file and the generation of the next SSTable file are performed simultaneously.

[0018] Preferably, the simultaneous writing back of the previous SSTable file and the generation of the next SSTable file is implemented by adopting a file granularity pipeline solution, which specifically includes the following steps:

[0019] Creation step: according to the task information returned in the task queue, create an iterator for the corresponding SSTable file in the task for subsequent traversal of related data, and create the SSTable file data structure in memory for receiving the subsequently generated ordered data blocks;

[0020] In the writing step, the data is traversed according to the iterator, the expired key-value pair information is deleted, the valid data is retained, and the valid data is organized into multiple data blocks and written into the created SSTable file data structure;

[0021] When the capacity of the SSTable to store data blocks reaches its threshold, the corresponding metadata is added to the SSTable and written back to the disk L i+1 Layer; while the previous SSTable is being written back to disk, the iterator continues to traverse the valid data and creates a new SSTable file data structure in memory to receive new valid data blocks; when all valid data is written back to L in order i+1 When the layer is created, the early compaction operation is completed.

[0022] The present invention further provides an early compaction system applied to an LSM tree structure, which is used to implement any of the above methods, including:

[0023] The trigger module is when the i-th layer L of the LSM tree i When the amount of stored data reaches its threshold, the Compaction operation is triggered;

[0024] Divide the module and select the i+1th layer L i+1 With L i The SSTables of multiple consecutive compaction operations to be performed in the layer are read into the task queue;

[0025] The compaction module reads multiple SSTables into the task queue and determines whether to perform a large compaction operation or only modify the metadata to implement SSTable in L based on the overlap of different key ranges. i Layer and L i+1 Requests to move between tiers.

[0026] The present invention has the following beneficial effects:

[0027] (1) The present invention effectively alleviates the I / O amplification problem caused by repeated reading and writing due to the overlapping ranges of continuous compaction operations. Through the present invention, the overlapping ranges only need to be read into the memory once and written back to the disk once to complete the compaction operation.

[0028] (2) The present invention speeds up the execution efficiency of compaction operations by using a file-granularity pipeline optimization solution. On the one hand, since the early compaction strategy may cause the writeback and generation of SSTables in memory to take a long time, the use of this optimization solution allows the writeback and generation operations to be executed in parallel in a pipeline manner, and the total time actually spent is only determined by the time spent on the generation of the SSTable operation; on the other hand, the optimization solution can be used regardless of the number of SSTables or the level to which the SSTable is finally written back, which means that the optimization solution has universal applicability.

[0029] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A method step diagram of an embodiment of the present invention;

[0031] Figure 2 is a detailed flow chart of an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of a second operation queue according to an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of a file granularity pipeline solution according to an embodiment of the present invention;

[0034] Figure 5 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The early compaction scheme applied to the LSM tree structure proposed in this invention: In the LSM tree, if L i When the layer reaches the SSTable storage capacity threshold, a compaction operation is triggered. After the trigger, the compaction operation is not performed directly, but the SSTable to be compacted is taken. These compaction operations are added to the task queue of the early compaction operation. When the length of the task queue reaches its threshold or L i Layer and L i+1 The SSTable ranges between layers no longer overlap, so this operation is no longer performed. The next step is to process the task queue. Since there may be two situations where the key ranges between SSTables overlap, the task queue is further divided into two operation queues for processing. The specific operations are as follows: (i) First operation queue: When L i Layer and Li+1 When there is no range overlap between SSTables in the layer, modify L directly i+1 The layer-level information metadata of the layer, and then insert the SSTable into L i+1 The corresponding position of the layer makes the layer globally ordered; (ii) The second operation queue: When L i Layer and L i+1 When there is a range overlap between SSTables in a layer, multiple compaction operations in the queue are combined into a large compaction operation and performed simultaneously. This large compaction operation is the process of merging and sorting all SSTables in the queue, and then writing them back to disk. It should be noted that when the merged and sorted SSTables are written back to disk, a file-granularity pipeline is used for write-back optimization. Based on the above-mentioned LSM tree early compaction scheme, the components and designs of the present invention are: compaction task queue. The queue is implemented using a linked list structure. Each node in the linked list stores a compaction task. In other words, each node stores L i Layer and L i+1 After the task queue is acquired, it needs to be divided into two specific operation queues according to the different situations of SSTable. Multiple consecutive compaction tasks with overlapping ranges will be merged into a large compaction task, and then the corresponding compaction task information and the request to write SSTable back to disk will be returned.

[0036] For details, see Figure 1 and Figure 2 The method step diagram and detailed flow chart of the embodiment of the present invention are shown, which include the following steps:

[0037] S101, when the i-th layer L of the LSM tree i When the amount of stored data reaches its threshold, the Compaction operation is triggered;

[0038] S102, select the i+1th layer L i+1 With L i The SSTables of multiple consecutive compaction operations to be performed in the layer are read into the task queue;

[0039] S103, after reading multiple SSTables into the task queue, determines whether to perform a large compaction operation or only modify the metadata to implement SSTable in L according to different key range overlap situations. i Layer and L i+1Requests to move between tiers.

[0040] like Figure 3 As shown in the figure, the second operation queue may have three situations: intersecting relationship, disjoint relationship and separated relationship. For intersecting and disjoint relationships, we only need to find the newly generated SSTable in L after the Compaction operation task is executed. i+1 The location of the layer makes L i+1 It is sufficient to have a global order in the layer. It should be noted that for SSTables that overlap with multiple compaction operations at the same time, it only needs to be added to the queue once. i+1 There are SSTable files that do not participate in the early compaction operation among the SSTable files selected for early compaction operation in the layer, so after the early compaction operation is completed, the new SSTable is written back to L i+1 When the layer is added, the new SSTable may overlap with the SSTable that does not participate in the early compaction operation, which will cause L i+1 The layer no longer maintains a globally ordered state and even data is lost. To address this problem, set an SSTable interval stop flag. When generating a new SSTable, no more data is written to the SSTable before the minimum key of the interval SSTable, and then a new SSTable is created to accept subsequent data writing. According to the scope of the Compaction operation, the subsequent data must be larger than the maximum key of the interval SSTable, so we ensure that L can still be maintained in the case of separation. i+1 The global order of layers.

[0041] Specifically, the process of the embodiment of the present invention is described by taking the intersection situation as an example. Figure 2 As shown. i SSTable1 of layer needs to be connected with L i+1 When the SSTable3 and SSTable4 of the layer are compacted, the compaction task is put into the queue, such as Figure 2 As shown in step 1 in ; then continue to search and find that the next compaction operation is required for SSTable2, SSTable4, and SSTable5, so the compaction task is also placed in the next node in the queue, such as Figure 2As shown in step 2 of . It can be found that at this time, for this intersection SSTable4, only one read and one write back are needed to complete the compaction operation. After all the overlapping tasks are placed in the task queue or the length of the task queue reaches its threshold, all compaction tasks are merged into a large compaction task for operation, that is, all valid data are merged and sorted to form a data block, such as Figure 2 As shown in step 3 of . Then write the generated data block to the SSTable file structure created previously, as shown Figure 2 As shown in step 4 in , the corresponding SSTable file metadata is added and written back to disk, as shown in Figure 2 As shown in step 5-1 in . While SSTable6 is being written back to disk, the next data block is also being generated, as shown in Figure 2 As shown in step 5-2 in . Figure 2 There are two steps 5 in FIG. 5 , which are used to illustrate that the two steps are executed in parallel.

[0042] See also Figure 4 As shown in the figure, it is a schematic diagram of the file granularity pipeline optimization solution of the present invention, which includes the following steps: In the process of executing the compaction operation, since all the SSTables written back are independent of each other, and since the time spent writing back the SSTable and generating the SSTable are of the same order of magnitude, the writing back of the previous SSTable file and the generation of the next SSTable file can be performed at the same time. Assuming that the time to generate the SSTable part is t1, the time to write back the SSTable part is t2, and there are a total of n SSTables in the task queue that need to be operated, Figure 4 It can be easily concluded that the total time required to complete the early compaction operation under this optimization scheme is n*t1+t2, and if this optimization scheme is not used, the total time required to complete is n*t1+n*t2. The above calculation of the total time shows that the use of this optimization scheme can effectively reduce the operation time and improve the efficiency of system operation.

[0043] To verify the effectiveness of the present invention, a verification experiment was conducted. The embodiment of the present invention runs Ubuntu 20.04 LTS on the Linux 5.11 kernel, equipped with an Inter(R) Core(TM) 3.20 GHz processor; the design is implemented based on LevelDB, and YCSB is used to generate data sets for experimental testing. In order to more intuitively evaluate the performance of the present invention, three systems, LevelDB, RocksDB and DiffKV, were used for comparison during the experiment. The MemTable of all systems participating in the experiment was set to 4MB, the SSTable file was set to 2MB, and a 500MB Block Cache was set. According to the experimental results, under the workload dominated by write operations, the throughput of the present invention is greatly improved compared with other systems, and the throughput is about 1.2-1.7 times, 1.1-1.5 times and 1.1-1.8 times of LevelDB, RocksDB and DiffKV respectively; under the workload dominated by read operations, the performance of the present invention can also be comparable to LevelDB, RocksDB and DiffKV.

[0044] See also Figure 5 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0045] Trigger module 501, trigger module, when the L of the LSM tree i When the amount of data stored in a layer reaches its threshold, a compaction operation is triggered;

[0046] Division module 502, select L i+1 Layer and L i The SSTables of multiple consecutive compaction operations in the layer are read into the task queue;

[0047] Compaction module 503 determines whether to perform a large compaction operation or only modify the metadata to implement SSTable in L according to the key range overlap of the SSTable read into the task queue. i Layer and L i+1 Requests to move between tiers.

[0048] It can be seen that the present invention proposes a model of early compaction, which effectively alleviates the additional read-write amplification problem caused by range overlap during continuous compaction operations, and adds a file granularity pipeline solution to optimize the proposed early compaction model, thereby optimizing the LSM tree-based storage system. It should be noted here that according to the characteristics of the read operation during execution, we perform traditional compaction operations on the lower-level SSTables, and only use the early compaction method proposed by the present invention at a higher level.

[0049] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An early compaction method applied to an LSM tree structure, characterized in that: The following steps are involved: When the i-th level L of the LSM tree i When the amount of stored data reaches its threshold, the Compaction operation is triggered; Select the i+1th layer L i+1 With L i The SSTables of multiple consecutive compaction operations to be performed in the layer are read into the task queue; After reading multiple SSTables into the task queue, it is determined whether to perform a large compaction operation or only modify the metadata to implement SSTable in L based on the overlap of different key ranges. i Layer and L i+1 Requests to move between tiers.

2. The early compaction method applied to the LSM tree structure according to claim 1 is characterized in that: The method determines whether to perform a large compaction operation or only modify the metadata to implement SSTable in L according to different key range overlap situations. i Layer and L i+1 A request to move between tiers consists of the following steps: The task queue is divided into two operation queues according to whether there is a key range overlap in SSTable. i Layer and L i+1 If there is no range overlap between SSTables in the layer, they are assigned to the first operation queue; if L i Layer and L i+1 If there is range overlap between SSTables in a layer, they are assigned to the second operation queue; For the first operation queue, only modify L i Layer and L i+1 The metadata of the layer, then L i The SSTable of the layer is inserted into L i+1 For the second operation queue, multiple subsequent continuous compaction operation tasks are merged with the current compaction operation task into a large compaction operation task and executed together, so as to realize the SSTable from L i Layer and L i+1 Migration of layers.

3. The early compaction method applied to the LSM tree structure according to claim 2 is characterized in that: There are three situations in the second operation queue, and different processing is performed according to different situations, including: For intersecting and disjoint relationships, find the newly generated SSTable in L after the Compaction operation task is executed. i+1 The location of the layer makes L i+1 The layer is globally ordered. SSTables that overlap with multiple compaction operations only need to be added to the queue once. For the separated relationship, set an SSTable interval stop flag. When generating a new SSTable, the SSTable will no longer be written before the minimum key of the interval SSTable, and then a new SSTable will be created to accept the subsequent data writing.

4. The early compaction method applied to the LSM tree structure according to claim 2, characterized in that: For the second operation queue, in order to speed up the write back of SSTable, the write back of the previous SSTable file and the generation of the next SSTable file are performed simultaneously.

5. The early compaction method applied to the LSM tree structure according to claim 4 is characterized in that: The simultaneous writing back of the previous SSTable file and the generation of the next SSTable file is implemented by adopting a file granularity pipeline solution, which specifically includes the following steps: Creation step: according to the task information returned in the task queue, create an iterator for the corresponding SSTable file in the task for subsequent traversal of related data, and create the SSTable file data structure in the memory for receiving the subsequently generated ordered data blocks; In the writing step, the data is traversed according to the iterator, the expired key-value pair information is deleted, the valid data is retained, and the valid data is organized into multiple data blocks and written into the created SSTable file data structure; When the capacity of the SSTable to store data blocks reaches its threshold, the corresponding metadata is added to the SSTable and written back to the disk L i+1 Layer; while the previous SSTable is being written back to disk, the iterator continues to traverse the valid data and creates a new SSTable file data structure in memory to receive new valid data blocks; when all valid data is written back to L in order i+1 When the layer is created, the early compaction operation is completed.

6. An early compaction system applied to an LSM tree structure, characterized in that: The method for implementing any one of claims 1 to 5 comprises: The trigger module is when the i-th layer L of the LSM tree i When the amount of stored data reaches its threshold, the Compaction operation is triggered; Divide the module and select the i+1th layer L i+1 With L i The SSTables of multiple consecutive compaction operations to be performed in the layer are read into the task queue; The compaction module reads multiple SSTables into the task queue and determines whether to perform a large compaction operation or only modify the metadata to achieve SSTable in L based on the overlap of different key ranges. i Layer and L i+1 Requests to move between tiers.

Images