File merging method based on data inclination

Abstract

The invention discloses a file merging method based on data inclination. According to the method, optimization of disk files is achieved through a merged file selection strategy based on read heat, an internal merging strategy based on HyperLogLog and a disk file merging algorithm based on inert update. According to the method, the file merging method of the LSM tree is redesigned on the basis of the inclination of the workloads, and compared with previous work, data cold and hot degree information in the workloads is fully utilized, and a disk file merging strategy based on the hot degree is designed. On a disk, cold read files are preferentially selected to be merged to a lower layer through read heat information, and meanwhile, the write amplification problem caused by merging operation to a system is relieved through an inert merging algorithm of disk files.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and relates to a file merging method based on data inclination. Background Art

[0002] Key-value storage systems store data in the form of key-value pairs and exhibit superior scalability compared to traditional relational databases. This system performs well in handling large-scale data storage and is commonly used in core applications such as cache systems and distributed file systems. Among them, the Log-Structured Merge Tree (LSM tree) [1] It is a popular implementation.

[0003] In a standard LSM tree structure, written data is first stored in an ordered memtable; when the memtable is full, it is converted into an immutable memtable, and then merged into the disk through a background process to form an SSTable file; on the disk, as the number of SSTables continues to increase, the background process will continue to merge files to maintain the order of the files.

[0004] As the amount of network data continues to grow, how to provide a high-performance storage engine to improve economic efficiency has become a key issue in infrastructure, and more and more scholars have begun to study adjustments to the structure of the LSM tree. However, existing research often simply modifies some components in the LSM, and the design often ignores the differences in data access frequency in actual workloads, that is, the uneven distribution of certain specific values ​​or keys in the data set, resulting in performance bottlenecks or unfairness when processing these values ​​or keys. For example, in the sales data of an e-commerce platform, there may be only a few popular products that are frequently queried, while other products are less frequently accessed. Currently, there is a lack of methods to integrate data access frequency information into the LSM tree system, which can effectively improve the storage efficiency of the system based on these data characteristics.

[0005] Pareto principle [2] , also known as the 80 / 20 rule, states that approximately 20% of the factors influence 80% of the results. In other words, only 20% of all variables are most important, while the remaining 80%, while numerous, have relatively little impact on the results. Real-life workloads often exhibit the Pareto effect, where only a small portion of data is frequently written and read, while the majority of data is only rarely updated or read. This reflects the imbalanced nature of workloads.

[0006] Current LSM optimization efforts typically focus on optimizing or adjusting one or more components of the LSM tree. These efforts ignore the data skew inherent in real-world workloads when merging files on disk, resulting in ineffective system performance improvements. Furthermore, no optimization of the on-disk file merging strategy is done based on these characteristics. To date, no research has incorporated data access frequency information into LSM system design.

[0007] Existing technologies and products often fail to take advantage of workload characteristics: actual workloads often share certain common characteristics, such as a strong skew in data access, where only a small amount of data is frequently read and updated, while most data is rarely read or written. Leveraging this workload characteristic to redesign the file selection strategy and merge algorithm in file merging can effectively improve system performance. Summary of the Invention

[0008] In order to solve the deficiencies in the prior art, the present invention aims to provide a file merging method based on data skewness.

[0009] This invention aims to design a new file merging strategy for a key-value storage system based on the data skewness of workloads. Specifically, the objectives of this invention can be divided into the following aspects:

[0010] How to estimate the heat value of data based on load skewness, and based on this, design a new disk file selection method to optimize the merging strategy and improve system performance.

[0011] How to redesign the file merge operation on the disk to reduce the write amplification problem of the LSM tree while avoiding affecting the query performance of the system.

[0012] In order to solve the above problems, the present invention proposes a disk file merging method based on data skew. First, a read heat value is maintained for all files on the disk, which reflects the number of times the file has been successfully queried. When a merge operation is required, the file with the lowest read heat is preferentially selected for merging. Secondly, a new file merging method is designed to reduce the write amplification problem caused by the merge operation while ensuring query performance. These improvements will significantly improve the performance of the key-value storage system and provide strong support for upper-level services.

[0013] Specifically, the present invention provides a file merging method based on data skewness, which optimizes disk files through a merge file selection strategy based on read popularity, an internal merge strategy based on HyperLogLog, and a disk file merging method based on lazy updates. The method includes the following steps:

[0014] Step 1: Maintain and update the read heat value of each file on the disk in memory, and maintain the HyperLogLog of each file in the L0 layer;

[0015] Step 2: Based on the read popularity values ​​of the files on the disk, select the file with the lowest read popularity to merge the files;

[0016] In step 3, if the file is located in the L0 layer, select additional files in the L0 layer that overlap with the file selected in step 2 and estimate the data volume ratio before and after the merge using HyperLogLog. If the ratio is less than the set threshold, all selected files are merged, and the resulting new file is still written to the L0 layer, exiting the process. If the ratio is greater than the set threshold, proceed to step 4.

[0017] Step 4: Apply the lazy update algorithm to merge the files merged in step 3, reduce the number of data rewrites by appending sub-data blocks, and complete the merge.

[0018] In step 1, the read heat value indicates the access frequency of the file, and the file with high access frequency is retained at the top level of the LSM tree;

[0019] The update of the read heat value is shown in the following formula:

[0020]

[0021] Among them hot r (t n ) indicates that file r is at time t i The heat value at time I r (t i ) indicates that file r is at time t i Whether it is accessed, t0 is an initial timestamp with an initial value of 0, which is incremented each time the system reads the file, and a is a smoothing parameter, which is set to a fixed value of 0.98 in this application.

[0022] In step 2, based on the heat values ​​maintained in the memory, the file with the lowest heat is selected for merging.

[0023] In step 3, the file on the layer to be merged is recorded as f i (i=0,1,2,…,n), calculate the data volume before and after the file merge based on HyperLogLog, and the data volume before the file merge is expressed as D pre =∑HLL(f i ), the amount of data after the files are merged is expressed as D new =HLL(∑f i );Calculate the ratio of the amount of data before and after merging p=D new / D pre,When the ratio p is less than the preset threshold, the files are merged and the merged files are stored in the original layer.

[0024] In step 3, the key value ranges of the layers to be merged partially overlap.

[0025] In step 3, if there is a lot of duplicate data in the L0 layer files, then in-place merging is more efficient because it reduces the need to move data to deeper levels of the LSM tree and avoids the write amplification problem. Write amplification is a situation where the amount of data generated during the merge process is greater than the actual amount of data, which usually occurs due to multiple copying and moving of data.

[0026] Write amplification has a significant impact on system performance. Since file merge operations require a large number of disk read and write operations, which occupy a large amount of disk bandwidth, the system's write operations may be blocked, affecting write performance. The merge operation proposed in the present invention can reduce the write amplification of file merge operations. When an internal merge or lazy merge is triggered, the amount of disk data written by the system is reduced: for internal merges, since there is no need to merge with L1 layer files, the amount of disk data read and write is reduced; for lazy merges, the file merge operation will be postponed, which can also reduce the amount of disk read and write.

[0027] In step 4:

[0028] Before merging: In the LSM tree, files need to be merged between layer Ln and layer Ln+1. The SSTable file in layer Ln contains key-value pairs 11, 22, and 33, while layer Ln+1 contains two SSTable files with key values ​​10, 12, 13 and 16, 20, 24 respectively.

[0029] Selection of files to be merged: Based on the heat recognition algorithm, select the file fc to be merged from layer Ln and determine the key value range that overlaps with the file Oi (i = 1, 2, 3, 4, ..., n) in layer Ln+1;

[0030] Division of sub-data blocks: The data in file fc will be divided into sub-data blocks d0, d1, ..., dn according to the overlapping range with Oi;

[0031] Lazy merge operation: For each file Oi in the Ln+1 layer, the system does not directly merge it with the data of fc. Instead, the system writes the sub-data blocks to Oi in an append-only manner. This method records the sub-data blocks in a linked list and updates the Bloom filter of Oi at the same time.

[0032] After the merger: Figure 2As shown in Figure 2, the original data of fc is appended to the corresponding file in layer Ln+1. For example, the key value 11 is appended to the SSTable containing 10, 12, and 13, while the key values ​​22 and 33 are appended to another SSTable containing 16, 20, and 24.

[0033] Conditions that trigger the actual merge: When an SSTable reaches a certain threshold due to additional sub-data blocks or the read cost is too high, the system will perform the actual merge operation, that is, sort all sub-data blocks together with the original data to form a new ordered SSTable;

[0034] Bloom filter parameter setting: In order to control the false positive rate of the Bloom filter, appropriate parameters are selected to accommodate more sub-data blocks and reduce the false positive problem caused by frequent updates;

[0035] Query process: When querying an SSTable, a Bloom filter is first used for quick screening, then all child data blocks are traversed in order from new to old, and finally the original data block is read to find the queried key value.

[0036] The key innovations of the present invention include:

[0037] Maintain HyperLogLog information for files at the L0 layer on the disk. For merge operations involving excessive duplicate data, internal merges of files are performed at the L0 layer.

[0038] The read popularity of files stored on disk is estimated. Based on this value, cold read files, that is, files with the lowest read popularity values, are preferentially selected for merging operations, while hot files with more duplicate data are merged in place.

[0039] A lazy merge algorithm based on disk files prevents the same file from being read and written multiple times by the merge operation, reducing the system's write amplification.

[0040] Based on the above method, the present invention also proposes a file merging system based on data skewness, comprising: a memory and a processor; a computer program is stored on the memory, and when the computer program is executed by the processor, the above method is implemented.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above method is implemented.

[0042] The present invention has the following advantages: It redesigns the file merging method for LSM trees based on workload skew. Compared to previous work, it fully utilizes the data hotness and coldness information in the workload and designs a heat-based disk file merging strategy. On disk, by reading heat information, cold read files are preferentially merged into the lower layer. At the same time, a lazy merge algorithm for disk files is used to alleviate the write amplification problem caused by the merge operation on the system.

[0043] The present invention and the prior art [6] The experimental comparison was conducted and the test recorded the amount of write amplification data of the system during the experiment. Under different loads, the present invention can compare with the existing technology. [6] Has lower write amplification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is an example diagram of the internal merging strategy based on HyperLogLog in the present invention.

[0046] Figure 2 This is an example diagram of the disk file merging algorithm based on lazy update of the present invention.

[0047] Figure 3 is a flow chart of the complete method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0049] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0050] LSM Tree: The LSM tree (Log-Structured Merge Tree) is a commonly used key-value storage structure suitable for scenarios with more writes than reads. It improves write performance by converting random writes into sequential writes. It organizes and manages data using a multi-layered structure on disk and maintains data order through background merge operations to improve read performance.

[0051] SSTable: SSTable (Sorted String Table) is a data storage format in an LSM tree. SSTables store key-value pairs in an ordered manner, using sparse indexes and block encoding to improve read efficiency. Each SSTable file typically contains multiple data blocks, where each block contains key-value pairs within a certain range. This design makes SSTables highly efficient in writing, querying, and compaction.

[0052] The present invention optimizes disk files based on the following strategies and algorithms, as follows:

[0053] Merge file selection strategy based on read popularity

[0054] Since the cost of querying on disk is higher than that on memory, in order to improve query efficiency, hot data should be maintained in the upper layer of the LSM tree on disk as much as possible to shorten the average access path length of the query.

[0055] The present invention optimizes file merging and query performance by maintaining the read heat value of each file on the disk in memory. Specifically, every time a query successfully reads a piece of data from a file on the disk, the read heat value of the file will be updated. The heat update formula is based on the exponential smoothing method. [3] The specific formula is as follows:

[0056]

[0057] Among them hot r (t n ) indicates that file r is at time t i The heat value at time I r (t i ) indicates that file r is at time t iWhether it has been accessed, t0 is an initial timestamp with an initial value of 0, which is automatically incremented each time the system reads a file, and a is a smoothing parameter, which is set to a fixed value of 0.98 in this embodiment. This formula can comprehensively take into account the historical access patterns of the data and estimate the heat value at a lower cost. In this formula, each file access will increase the heat value of the file, and as the number of accesses increases, the impact of historical accesses on the total heat value will gradually decay, so it can take into account historical access information and access information in the most recent time. Since the heat value calculation is statistically calculated at the file granularity, the time and space costs required to maintain this information are extremely low. When the system needs to perform a merge operation, it will select a file with the lowest read heat in the hot data maintenance layer, and select all files with overlapping ranges from the next layer for a merge operation. This merge file selection strategy can keep files that are accessed more frequently in the upper layer, thereby shortening the average query access path length.

[0058] Based on HyperLogLog [4] Internal merger strategy

[0059] In order to optimize the file merging operation on the L0 layer, the system also maintains an additional HyperLogLog for each file on the disk L0 [5] , which is used to approximate the amount of non-duplicate data contained in each file using a smaller space. Since the key value ranges of L0 layer files can overlap with each other, there may be a certain amount of duplicate data. When the L0 layer needs to perform a merge operation, it will first determine whether an internal merge can be performed. First, other files that have overlapping ranges with the selected files will be selected at the L0 layer. Assume that all the files selected in this operation are f i (i=0,1,2,…,n), firstly, the file data volume will be counted based on HyperLogLog. HyperLogLog is a probabilistic cardinality estimation algorithm used to quickly estimate the number of different elements in a set, with low memory consumption and computational complexity. pre =ΣHLL(f i ) and the amount of data after merging D new =HLL(Σf i ) and calculate the data volume ratio p=D before and after the merger new / D preIf the ratio p is less than the threshold set by the system, it means that there is a large amount of duplicate data in the selected files. In this case, the selected files will be merged in situ, and the merged files will still be stored in the L0 layer. In terms of threshold selection, a larger threshold will trigger more internal merges, and a smaller threshold will remove more duplicate data. In practice, the parameters should be selected according to the characteristics of the application. The more data is written repeatedly in the application, the lower the threshold can be selected. Since the merge operation does not require reading and writing overlapping files in the L1 layer, it can keep the hot data at the top layer while reducing the write amplification problem caused by the merge.

[0060] For example, in Figure 1 In the example, the key value ranges of the three files in the L0 layer overlap, so they are all selected for merging. After merging the HyperLogLogs corresponding to these three files and performing calculations, the amount of unique data obtained is relatively small compared to the total data volume. Therefore, it can be determined that there is a large amount of duplicate data. This time, an in-place merge at the L0 layer can be performed without affecting the files in the L1 layer. Figure 1 This shows the process of an L0 layer merge, in which three L0 layer files are merged into a larger L0 layer file.

[0061] Disk file merging algorithm based on lazy update

[0062] In order to maintain the orderliness of disk SSTable files, the LSM tree needs to continuously merge files. Since a single merge operation involves two layers of files, and the number of lower-layer files is usually several times that of upper-layer files, the merge operation will cause the same file to be read and written repeatedly, resulting in serious write amplification problems.

[0063] In order to alleviate the write amplification problem, the present invention designs a lazy file merging algorithm for disk files. The specific process of the algorithm is as follows: Figure 2 As shown:

[0064] When a layer on the disk needs to be merged, the file to be merged is first selected based on the above-mentioned "merge file selection strategy based on read heat". Here, the file to be merged is recorded as f c First, we still get all the files in the lower layer that have key value ranges overlapping with the file. Here, the files selected in the lower layer are recorded as O i (i=0,1,2,…,n). In the first step, f c The data is divided into d0, d1, d2...d n , where d i represents f c Neutralize O i Overlapping data is called sub-data blocks. Figure 2In, f c for l n Layer files, and l n+1 There is an overlap between the two files, which are called O0 and O1, where d0 contains the data

[11] and d1 contains the data [22,33]. i , does not actually merge the data, but writes its corresponding sub-data blocks in an appended manner, records them in the file in the form of a linked list, and uses the data in the sub-data blocks to update the file's Bloom filter data. After each file completes this operation, f c It can be deleted. At the same time, the system will calculate the query cost and file block merge cost of each file based on the cost model. Secondly, the query cost refers to the I / O overhead required to query data in this file and its corresponding sub-data blocks, and the merge cost refers to the I / O overhead required to merge the file and all its sub-data blocks into an ordered file. When the total amount of sub-data blocks appended to a file reaches the threshold set by the system, or the query cost of the file is too high due to too many sub-blocks, the file will be actually merged, that is, the original data and all sub-data blocks are read and sorted to create an SSTable file with ordered data.

[0065] In particular, in this design, in order to avoid the problem of excessive false positives caused by frequent updates of the Bloom filter, the parameters of the Bloom filter need to be set: assuming that the default size of each SSTable is S, the file size of kS (k>1) will be used to select the most appropriate Bloom filter parameters when creating the SSTable. The larger k is, the more sub-data blocks each file can receive. Specifically, in the Bloom filter, the most important parameter is the size of the Bloom filter. Given the error rate ε and the amount of data n, the optimal size of the Bloom filter is -1.44log2ε*n. In practice, ε is usually set to 1%, that is, the Bloom filter has a 1% probability of returning a false positive result. Secondly, assuming that there are n pieces of data in the current SSTable, the Bloom filter size will be set to -1.44log2ε*kn to achieve the best Bloom filter effect.

[0066] When querying a certain SSTable, the first step is to read the Bloom filter of the file for preliminary filtering. Secondly, all child data blocks are traversed from the new to the old, and finally the data blocks of the SSTable are read for search.

[0067] This lazy disk file merging algorithm significantly reduces the number of data reads and writes per SSTable, thereby mitigating write amplification issues across the entire system. By selecting less frequently accessed cold-read files for merging and updating the Bloom filter data for all files in real time, the negative impact of multiple sub-data blocks on query performance can be controlled.

[0068] Example 1

[0069] Using YCSB [5] The system was tested and compared with the present invention and MatrixKV [6] Write amplification.

[0070] First, 100GB of random data is written into the system, where each piece of data is 1KB in size and the data is evenly distributed.

[0071] We then tested workloads A, B, D, and F, which come with YCSB, with a total of 300 million requests. Workload C is read-only, and workload E is scan-intensive, so they were not included in the test. The settings for these four workloads are as follows:

[0072] Workload A: 50% read operations, 50% update operations

[0073] Workload B: 95% read operations, 5% update operations

[0074] Workload D: 95% read operations, 5% insert operations

[0075] Workload F: 50% read operations, 50% read-modify-write operations

[0076] During the operation, the overall write amplification of the system was statistically analyzed, and the following results were obtained:

[0077] Load A Load B Load D Load F MatrixKV 1128G 164G 242G 1251G The present invention 563G 81G 101G 515G

[0078] References

[0079] [1] LSM tree: https: / / zhuanlan.zhihu.com / p / 181498475

[0080] [2]Pareto principle: https: / / en.wikipedia.org / wiki / Pareto_principle

[0081] [3] Exponential smoothing method: https: / / wiki.mbalib.com / zh-tw / %E6%8C%87%E6%95%B0%E5%B9%B3%E6%BB%91%E6%B3%95

[0082] [4]HyperLogLog: https: / / en.wikipedia.org / wiki / HyperLogLog

[0083] [5]Yahoo! Cloud Serving Benchmark: https: / / github.com / brianfrankcooper / YCSB

[0084] [6]MatrixKV: Reducing Write Stalls and Write Amplification in LSM-treeBased KV Stores with Matrix Container in NVM: https: / / www.usenix.org / conference / atc20 / presentation / yao

[0085] As used in the present invention, the terms "comprise" and "include" are open expressions, that is, they include the contents specified in the present invention but do not exclude other aspects.

[0086] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A file merging method based on data skewness, characterized in that: The method optimizes disk files through a merge file selection strategy based on read popularity, an internal merge strategy based on HyperLogLog, and a disk file merge algorithm based on lazy update.

2. The method according to claim 1, wherein The method comprises the following steps: Step 1: Maintain and update the read heat value of each file on the disk in memory; Step 2: Based on the read popularity values ​​of the files on the disk, select the file with the lowest read popularity to merge; Step 3: Estimate the amount of non-duplicated data in each file on the disk layer to be merged, and merge the files based on the amount of non-duplicated data; Step 4: Apply the lazy update algorithm to merge the files merged in step 3, reduce the number of data rewrites by appending sub-data blocks, and complete the merge.

3. The file merging method according to claim 2, wherein: In step 1, the read heat value indicates the access frequency of the file, and the file with high access frequency is retained at the top level of the LSM tree; The update formula of the read heat value is as follows: Among them, hot r (t n ) indicates that file r is at time t i The heat value at time I r (t i ) indicates that file r is at time t i Whether it is accessed, t0 is an initial timestamp with an initial value of 0, which is incremented each time the system reads the file, and a is a smoothing parameter.

4. The file merging method according to claim 2, wherein: In step 2, the HyperLogLog of each file in the L0 layer is maintained at the same time.

5. The file merging method according to claim 4, wherein: In step 3, if the file is located in the L0 layer, it is necessary to select other files in the L0 layer that overlap with the file selected in step 2. The ratio of the data volume before and after the merge is estimated based on HyperLogLog. If the ratio is less than the set threshold, all selected files are merged, and the generated new file is still written to the L0 layer. If the ratio is greater than the set threshold, execute step 4.

6. The file merging method according to claim 2, wherein: In step 3, the file on the layer to be merged is recorded as f i (i=0,1,2,…,n), calculate the data volume before and after the file merge based on HyperLogLog, and the data volume before the file merge is expressed as D pre =∑HLL(f i ), the amount of data after the files are merged is expressed as D new =HLL(∑f i );Calculate the ratio of the amount of data before and after merging p=D new / D pre ,When the ratio p is less than the preset threshold, the files are merged and the merged files are stored in the original layer.

7. The file merging method according to claim 2, wherein: In step 3, the key value ranges of the layers to be merged partially overlap.

8. The file merging method according to claim 2, wherein: Step 4 further includes the following sub-steps: Step 4.1, selection of files to be merged: Based on the heat recognition algorithm, select the file fc to be merged from layer Ln and determine the key value range that overlaps with the file Oi (i = 1, 2, 3, 4, ..., n) in layer Ln+1; Step 4.2, division of sub-data blocks: the data in file fc will be divided into sub-data blocks d0, d1, ..., dn according to the overlapping range with file Oi; Step 4.3, lazy merge operation: For each file Oi in the Ln+1 layer, write the sub-data block to the file Oi in an append manner. This method records the sub-data block through a linked list and updates the Bloom filter of the file Oi at the same time.

9. The file merging method according to claim 8, wherein: In step 4, the conditions for triggering the actual merge are: when an SSTable reaches a threshold due to additional writing of sub-data blocks or the reading cost is too high, the actual merge operation is performed, that is, all sub-data blocks are sorted together with the original data to form a new ordered SSTable.

10. The file merging method according to claim 8, wherein: When querying an SSTable, first use the Bloom filter for quick screening, then traverse all child data blocks in order from new to old, and finally read the original data block to find the queried key value.

Images