A cold-hot data-aware LSM tree merge performance optimization method

Abstract

The application discloses a cold and hot data sensing LSM tree merging performance optimization method, including the following steps: updating the hotness of the corresponding key range when writing key-value pairs, preparing for calculating the hotness of SSTable; calculating the hotness of Li layer SSTable and selecting the SSTable with the highest hotness to participate in merging during merging, so that as many obsolete key-value pairs as possible are deleted in each merging process; and considering the hotness zero problem after merging to adapt to different situations of hot key-value pairs in different time periods. The application adopts the cold and hot data sensing LSM tree merging performance optimization method, modifies the selection strategy of SSTables during LSM tree merging, lets SSTables containing more hot key-value pairs be selected into the memory for merging, so that the data amount written to the disk after merging is reduced, the overall performance of the system is improved, and the service life of the disk is prolonged.

Description

Technical Field

[0001] This invention relates to the field of computer database design technology, and in particular to a performance optimization method for LSM tree merging based on hot and cold data awareness. Background Technology

[0002] With the development of big data technology, Log Structured Merge (LSM) trees have been widely used in key-value databases. Noting the characteristic that sequential disk writes are faster than random writes, the LSM tree's in-memory structure is a memtable, constructed from a red-black tree or skip list data structure. This transforms unordered key-value pairs into ordered ones, allowing for sequential writing to disk and thus improving disk write performance. During disk writing, key-value pairs are first organized into data blocks, and then multiple blocks form a sorted string table (SSTable). These SSTables are stored as files at multiple levels on the disk. When reading a key-value pair, the search starts from the highest level; if not found, it continues searching at the next lower level until found. This multi-level structure helps improve database query efficiency.

[0003] When the number of SSTables at a certain level Li exceeds a specified capacity, the LSM tree will randomly select an SSTable from level Li and merge it with SSTables in the next level Li+1 that have overlapping key-value pairs. The merging process is similar to merging multiple sorted arrays into a single sorted array. The newly generated SSTables are rewritten to disk, while the merged SSTables are deleted. By merging to remove outdated and deleted key-value pairs, the total database size is reduced.

[0004] like Figure 1 The diagram illustrates the generation and merging process of an LSM tree SSTable. The existing LSM tree merging process continuously reads key-value pairs from disk into memory and writes valid key-value pairs back to disk. This results in severe write amplification during the LSM tree merging process, impacting system performance and disk lifespan. Summary of the Invention

[0005] The purpose of this invention is to provide a performance optimization method for LSM tree merging that is aware of hot and cold data, so that as many SSTables participating in the merge as possible contain more invalid and deleted key-value pairs. By modifying the selection strategy of SSTables during LSM tree merging, SSTables containing more hot key-value pairs are selected for merging in memory. This reduces the amount of data written to disk after merging, improves overall system performance, and extends the lifespan of the disk.

[0006] To achieve the above objectives, this invention provides a performance optimization method for LSM tree merging based on hot and cold data awareness, comprising the following steps:

[0007] When writing key-value pairs, update the popularity of the corresponding key range to prepare for calculating the popularity of SSTable; when merging, calculate the popularity of SSTable in the Li layer and select the SSTable with the highest popularity to participate in the merge, so that as many outdated key-value pairs as possible are deleted in each merge process;

[0008] The merged data is set to zero to accommodate different situations of hot key-value pairs at different times.

[0009] Preferably, when a key-value pair is written to an LSM-based database, the bucket containing that key is incremented by one.

[0010] Preferably, the SSTable heat is represented by the sum of the values ​​of the key range buckets it contains;

[0011] The minimum and maximum key ranges within an SSTable are selected. The SSTable's heat is represented by the sum of the values ​​of the key range buckets that overlap with its key range. The key value range for each bucket is set based on the optimal range obtained from testing under real workload conditions. Preferably, when selecting the SSTable with the highest heat for merging, an improved LSM tree SSTable selection strategy is used. This improved LSM tree SSTable selection strategy selects the SSTable of the Li layer with the highest heat through heat awareness.

[0012] Preferably, the hotness reset includes resetting the value of the key range bucket to zero every other day. This serves two purposes: first, it adapts to the scenario where the hotness key range varies every day under real workload, preventing interference between hot data from each day; second, the value of the key range bucket will eventually exceed the computer's representation range, which will lead to data representation errors. Resetting the bucket periodically prevents hotness comparison errors caused by the bucket exceeding the representation range.

[0013] Therefore, this invention employs the aforementioned hot / cold data-aware LSM tree merging performance optimization method. Compared to selecting all SSTables at the same level equally, this solution proposes a hot / cold data-aware method to select the SSTable containing the most hot key-value pairs (those frequently updated) for merging. This reduces the amount of data written to disk after merging and reduces write amplification during the LSM tree merging process. Thus, it improves the write performance of the LSM tree and the lifespan of the disk. The technical solution of this invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0014] Figure 1 This describes the process of generating and merging SSTables in an LSM tree.

[0015] Figure 2 This describes the calculation method for SSTable popularity.

[0016] Figure 3 This is a flowchart of the LSM tree merging strategy based on hot and cold data awareness. Detailed Implementation

[0017] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0019] Example 1

[0020] like Figure 3 As shown, this invention provides a performance optimization method for LSM tree merging based on hot and cold data awareness, comprising the following steps:

[0021] Step 1: When writing key-value pairs, update the heat index of the corresponding key range to prepare for calculating the SSTable heat index.

[0022] When a key-value pair is written to an LSM-based database, the bucket containing that key is incremented by one. For example... Figure 2 As shown, this invention first designs a key-range bucket data structure to count the number of times key-value pairs within a certain key range are updated. In LSM-based databases (LevelDB, RocksDB, etc.), when a put(key, value) operation is called (i.e., a key-value pair is written), the bucket containing that key is incremented by one. For example, Figure 2 The four buckets shown have key ranges of 0-100, 100-200, 200-300, and 300-400, respectively. The bucket with key 0-100 has a value of 34, indicating that the upper layer performed 34 update operations on the data with key 0-100 in the database. The other three buckets have update counts of 28, 19, and 27, respectively.

[0023] Step 2: During the merging process, calculate the heat of the SSTable in the Li layer and select the SSTable with the highest heat to participate in the merging. This ensures that as many outdated key-value pairs as possible are deleted in each merging process. The heat of the SSTable is represented by the sum of the values ​​of the buckets containing its key range.

[0024] The minimum and maximum keys within an SSTable are selected to define the key range. The SSTable's heat is represented by the sum of the values ​​of the key range buckets that overlap with its key range. The key value range for each bucket is set based on the optimal range obtained from testing under real workload conditions. When selecting the SSTable with the highest heat for merging, an improved LSM tree SSTable selection strategy is used. This improved LSM tree SSTable selection strategy selects the SSTable of the Li layer with the highest heat through heat awareness.

[0025] The popularity of an SSTable can be represented by the sum of the values ​​of the buckets containing its key ranges. For an SSTable with a key range of 10-152, its popularity is composed of buckets with key ranges of 0-100 and 100-200, resulting in a popularity of 34 + 28 = 62. Using the same calculation method, the popularity of SSTables with key ranges of 167-234, 257-299, and 336-390 are 47, 19, and 27, respectively. The size of the key value range for each bucket is crucial to the design. If the key range is too large, the difference in popularity calculated for SSTables at the same level will be insignificant. If the key range is too small, the number of buckets required will increase, increasing memory overhead.

[0026] Secondly, when the capacity of a certain level (Li) in the LSM tree exceeds a set threshold for that level, SSTable merging is triggered to remove outdated key-value pairs. Instead of randomly selecting SSTables for merging, the SSTable with the highest heat value at that level is chosen based on the heat value calculated from the key range buckets. This is because the SSTable contains more invalid key-value pairs that can be deleted during the merging process, resulting in less data being written back to disk.

[0027] Step 3: After merging, consider the issue of resetting the popularity to zero to accommodate different situations of hot key-value pairs at different time periods.

[0028] The key range bucket value will be reset to zero every other day. This serves two purposes: first, it adapts to real-world workloads where the hot key ranges differ daily, preventing interference between daily hot data; second, the continuously increasing key range bucket value will eventually exceed the computer's representation range. For example, using the C++ int type, the maximum bucket value is 2147483647, exceeding this value will lead to data representation errors. Resetting it periodically prevents errors in hot key comparisons caused by the bucket exceeding its representation range.

[0029] Therefore, this invention employs the aforementioned hot / cold data-aware LSM tree merging performance optimization method to ensure that as many SSTables participating in the merge as possible contain more invalid and deleted key-value pairs. By modifying the SSTables selection strategy during LSM tree merging, SSTables containing more hot key-value pairs are selected for merging in memory. This reduces the amount of data written to disk after merging, improves overall system performance, and extends disk lifespan.

[0030] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A performance optimization method for LSM tree merging based on hot and cold data awareness, characterized in that, Includes the following steps: S1. When writing key-value pairs, update the heat of the corresponding key range to prepare for calculating the SSTable heat. S1 includes: when a key-value pair is written to the LSM tree-based database, the bucket containing the key is incremented by one; S2. During the merging process, calculate the popularity of the SSTable in the Li layer and select the SSTable with the highest popularity to participate in the merging, so that as many outdated key-value pairs as possible are deleted in each merging process. The S2 includes: the heat of the SSTable is represented by the sum of the values ​​of the key range buckets it contains; Select the minimum and maximum keys contained in the SSTable to set the key range. The SSTable heat is represented by the sum of the values ​​of the key range buckets that overlap with its key range. Set the key value range for each bucket based on the optimal range obtained from testing under real workload. When selecting the SSTable with the highest popularity to participate in the merging, an improved LSM tree SSTable selection strategy is adopted. The improved LSM tree SSTable selection strategy selects the SSTable of the Li layer with the highest popularity through popularity perception. Secondly, when the capacity of a certain level of the LSM tree Li exceeds the set threshold for that level, SSTable merging is triggered to delete outdated key-value pairs. When selecting SSTables to merge, the SSTables of the Li level will no longer be randomly selected, but will be selected based on the heat calculated by the key range bucket, and the SSTable with the highest heat in that level will be selected to participate in the merging. S3. After merging, the popularity is reset to zero to adapt to different situations of hot key-value pairs in different time periods; The S3 includes: zeroing the heat index, which means that the value of the key range bucket will be cleared every other day. One reason is to adapt to the scenario where the heat index key range is different every day under real workload, and to prevent interference between hot data every day; the other reason is that the value of the key range bucket will always exceed the computer's representation range if it exceeds the range, which will lead to data representation errors. The timed clearing prevents the heat index comparison error caused by the bucket exceeding the representation range.

Images