Cold and hot data sensing LSM tree merging performance optimization method

Abstract

The invention discloses a cold and hot data sensing LSM tree merging performance optimization method, which comprises the following steps of: updating heat of a corresponding key range when a key value pair is written, and making preparation for calculating SSTable heat; during merging, calculating the popularity of the SSTable of the Li layer and selecting the SSTable with the highest popularity to participate in merging, so that outdated key value pairs are deleted as many as possible in each merging process; and after merging, a hot return-to-zero problem is considered to adapt to the condition that hot key value pairs in different time periods are different. According to the cold and hot data sensing LSM tree merging performance optimization method, the SSTable containing more hotspot key value pairs is selected into a memory to be merged by modifying a selection strategy for the SSTable during LSM tree merging, so that the amount of data written into a disk after merging is reduced, the overall performance of a system is improved, and the service life of the disk is also prolonged.

Description

Technical Field

[0001] The present invention relates to the technical field of computer database design, and in particular to a cold-hot data aware LSM tree merging performance optimization method. Background Art

[0002] With the development of big data technology, log structured merge (LSM) trees have been widely used in key-value-based databases. Noting that the sequential write speed of disks is faster than random write speed, the structure of LSM trees in memory is memtable, which is constructed from the data structure of red-black trees or skip lists. This can transform unordered key-value pairs into ordered ones, and then convert them into sequential writes to disk, thereby improving disk write performance. When writing to disk, key-value pair data is first organized into data blocks, and then multiple data blocks form a sorted string table (SSTable). These SSTables are stored in multiple levels on the disk in the form of files. When reading a key-value pair, the query will start from the highest level. If it is not found, the query will continue at the next level until it is found. This multi-level structure is conducive to improving the query efficiency of the database.

[0003] When the SSTable of a certain level Li exceeds the specified capacity, the LSM tree will use the merging process to randomly select an SSTable in the Li layer and merge it with the SSTables with overlapping key-value pairs in the next level Li+1. The principle of the merging process is similar to merging multiple ordered arrays into one ordered array. The newly generated SSTables will be rewritten to the disk, and the merged SSTables will be deleted. By merging, outdated and deleted key-value pairs are removed, thereby achieving the purpose of reducing the total capacity of the database.

[0004] like Figure 1 As shown in the figure, the generation and merging process of LSM tree SSTable. The existing LSM tree merging process continuously reads key-value pairs from disk into memory and writes valid key-value pairs back to disk, which leads to serious write amplification in the LSM tree merging process, affecting system performance and disk service life. Summary of the invention

[0005] The purpose of the present invention is to provide a cold and hot data aware LSM tree merge performance optimization method, so that as many SSTables participating in the merge as possible contain more invalid and deleted key-value pairs. By modifying the selection strategy of SSTables during LSM tree merge, SSTables containing more hot key-value pairs are selected into the memory for merging, which will reduce the amount of data written to the disk after the merge, improve the overall system performance, and extend the service life of the disk.

[0006] To achieve the above object, the present invention provides a cold-hot data-aware LSM tree merging performance optimization method, comprising the following steps:

[0007] When writing a key-value pair, update the heat of the corresponding key range to prepare for calculating the heat of the SSTable; when merging, calculate the heat of the Li layer SSTable and select the SSTable with the highest heat to participate in the merge, so that each merge process deletes as many outdated key-value pairs as possible;

[0008] After merging, the heat is reset to zero to adapt to the different situations of hot key-value pairs in different time periods.

[0009] Preferably, when a key-value pair is written to the LSM tree-based database, the bucket containing the key is incremented by one.

[0010] Preferably, the SSTable heat is represented by the sum of the values ​​of the key range buckets it contains;

[0011] Select the minimum key and maximum key contained in the SSTable in the SSTable to set the key range. The heat of the SSTable is represented by the sum of the values ​​of the key range buckets that overlap with its key range. The key value range of each bucket is set according to the optimal range obtained by testing under real workload. Preferably, when selecting the hottest SSTable to participate in the merge, the improved LSM tree SSTable selection strategy is adopted. The improved LSM tree SSTable selection strategy selects the hottest Li layer SSTable through heat perception.

[0012] Preferably, the heat zeroing includes clearing the value of the key range bucket every other day. One reason is to adapt to the scenario where the heat key range is different every day under the real workload, and to prevent the hot spot data from interfering with each other every day. The other reason is that the value of the key range bucket will always exceed the representation range of the computer if it increases continuously. If it exceeds the range, it will cause data representation errors. Regular clearing prevents heat comparison errors caused by the bucket exceeding the expression range.

[0013] Therefore, the present invention adopts the above-mentioned cold and hot data-aware LSM tree merge performance optimization method. Compared with equally selecting the SSTable of this level, this scheme proposes a cold and hot data-aware method to select the SSTable containing the most hot key-value pairs (these key-value pairs are frequently updated) for merging. This can reduce the amount of data written to the disk after the merge and reduce the write amplification of the LSM tree merge process. Therefore, the write performance of the LSM tree and the service life of the disk are improved. The technical scheme of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the process of generating and merging LSM tree SSTables;

[0015] Figure 2 The calculation method of SSTable heat;

[0016] Figure 3 The following is a flow chart of the LSM tree merge strategy based on cold and hot data perception. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0018] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0019] Embodiment 1

[0020] like Figure 3 As shown, the present invention provides a cold-hot data-aware LSM tree merging performance optimization method, comprising the following steps:

[0021] Step 1: Update the heat of the corresponding key range when writing a key-value pair to prepare for calculating the heat of the SSTable

[0022] When a key-value pair is written to a database based on an LSM tree, the bucket containing the key is incremented by one. Figure 2 As shown, the present invention first designs a key range bucket data structure to count the number of times a key-value pair of a key range is updated. When a put(key, value) is called in a LSM tree-based database (LevelDB, RocksDB, etc.), that is, when a key-value pair is written, the bucket containing the key is incremented by one. For example, Figure 2 The four buckets shown have key ranges of 0-100, 100-200, 200-300, and 300-400. The bucket with key 0-100 has a value of 34, which means that the upper layer has updated the data with database key 0-100 34 times. The other three buckets have been updated 28, 19, and 27 times, respectively.

[0023] Step 2: Calculate the heat of the Li layer SSTable during merging and select the SSTable with the highest heat to participate in the merge, so that each merge process deletes as many obsolete key-value pairs as possible. The heat of the SSTable is represented by the sum of the values ​​of the key range buckets it contains;

[0024] The minimum and maximum keys contained in the SSTable in the SSTable are selected to set the key range. The heat of the SSTable is represented by the sum of the values ​​of the key range buckets that overlap with its key range. The key value range of each bucket is set according to the optimal range obtained by testing under real workloads. When selecting the hottest SSTable to participate in the merge, the improved LSM tree SSTable selection strategy is adopted. The improved LSM tree SSTable selection strategy selects the hottest Li layer SSTable through heat perception.

[0025] The heat of an SSTable can be represented by the sum of the values ​​of the key range buckets it contains. For an SSTable with a key range of 10-152, its heat is composed of buckets with key ranges of 0-100 and 100-200, and its heat is 34+28=62. By this calculation method, the heats of SSTables with key ranges of 167-234, 257-299, and 336-390 are 47, 19, and 27, respectively. The size of the key value range of each bucket is the core of the design. If the key range is too large, the heat difference of SSTables at the same level will not be obvious. If the key range is too small, the number of buckets required will increase, increasing memory overhead.

[0026] Secondly, when the capacity of a certain level Li of the LSM tree exceeds the set threshold of that level, SSTable merge is triggered to delete obsolete key-value pairs. When selecting SSTable for merging, the SSTable of the Li level will no longer be randomly selected, but the SSTable with the highest heat of that level will be selected for merging based on the heat calculated by the key range bucket, because this SSTable contains more invalid key-value pairs that can be deleted during the merge process, so the data written back to the disk will be less in the end.

[0027] Step 3: Consider the problem of zeroing the heat after merging to adapt to the different situations of hot key-value pairs in different time periods

[0028] The value of the key range bucket will be cleared every other day. There are two purposes: one is to adapt to the scenario where the hot key range is different every day under real workload, so as to prevent the hot data from interfering with each other every day; the other is that the value of the key range bucket will always exceed the computer's representation range if it is continuously increased. For example, if it is represented by C++ int type, the maximum value of the bucket is 2147483647, which will cause data representation errors. Regular clearing can prevent heat comparison errors caused by buckets exceeding the expression range.

[0029] Therefore, the present invention adopts the above-mentioned hot and cold data-aware LSM tree merge performance optimization method to make more SSTables involved in the merge contain more invalid and deleted key-value pairs as much as possible. By modifying the selection strategy of SSTables during LSM tree merging, SSTables containing more hot key-value pairs are selected into the memory for merging, which will reduce the amount of data written to the disk after the merge, improve the overall performance of the system, and extend the service life of the disk.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A cold and hot data aware LSM tree merging performance optimization method, characterized in that: The following steps are involved: When writing a key-value pair, update the heat of the corresponding key range to prepare for calculating the heat of the SSTable; When merging, calculate the heat of the Li layer SSTable and select the hottest SSTable to participate in the merge, so that each merge process deletes as many obsolete key-value pairs as possible; After merging, the heat is reset to zero to adapt to the different situations of hot key-value pairs in different time periods.

2. According to the method for optimizing the performance of LSM tree merging based on cold and hot data perception in claim 1, it is characterized in that: When a key-value pair is written to an LSM-tree-based database, the bucket containing the key is incremented by one.

3. According to the method for optimizing the performance of LSM tree merging based on cold and hot data perception in claim 1, it is characterized in that: The heat of an SSTable is represented by the sum of the values ​​of the key range buckets it contains; Select the minimum key and maximum key contained in the SSTable in the SSTable to set the key range. The values ​​of the key range buckets whose SSTable heat overlaps with its key range are added together. Set the key value range of each bucket based on the optimal range obtained from tests under real workloads.

4. The method for optimizing the merging performance of LSM trees based on cold and hot data perception according to claim 1, characterized in that: When selecting the hottest SSTable to participate in the merge, the improved LSM tree SSTable selection strategy is adopted. The improved LSM tree SSTable selection strategy selects the hottest Li layer SSTable through heat perception.

5. The method for optimizing the merging performance of LSM trees based on cold and hot data perception according to claim 1, characterized in that: Heat zeroing includes clearing the value of the key range bucket every other day. One reason is to adapt to the scenario where the heat key range is different every day under real workload to prevent the interference of hot data every day. The other reason is that the value of the key range bucket will always exceed the representation range of the computer if it increases continuously. If it exceeds the range, it will cause data representation errors. Regular clearing prevents heat comparison errors caused by buckets exceeding the expression range.

Images