Dynamically adaptive LSM (Log-structured merge) tree combination method and system

Abstract

The invention is suitable for the technical field of file processing and provides a dynamically adaptive LSM (Log-structured merge) tree combination method. The method comprises: dividing a key value interval into a plurality of nodes and organizing the nodes into a treelike structure, wherein each node corresponds to a key value interval and each key value interval contains a file corresponding to a key value interval range; dynamically adjusting the shape of a tree according to distribution of current data; when a newly written file exists, traversing the tree to look for an optimal node and putting the file into the node; and when the file is processed, performing Minor Compact processing on the interior of the node and executing Major Compact through a leaf node. The invention further correspondingly provides a dynamically adaptive LSM tree combination system realizing the method. Therefore, the dynamically adaptive LSM tree combination method and system can realize dynamic adaption to data distribution and improve data combination efficiency.

Description

technical field [0001] The invention relates to the technical field of file processing, in particular to a dynamically adaptable LSM tree merging method and system. Background technique [0002] Log-structuredmerge tree, also known as LSM tree, is a commonly used data organization method for NoSQL databases. It defers and batches changes to the index and efficiently migrates updates to disk in a merge-sort-like fashion. The node of the LSM tree is often a file in the specific implementation. The file is ordered internally and the files are disordered. When querying, it is necessary to query all files and merge the results of each file, resulting in low performance. Therefore, several files are generally Merge into one large file, reduce the number of files by merging files, and actually delete data to reduce the number of files involved in each query and improve data query efficiency, that is, Compact. The compact process reads multiple files and merges them and rewrites t...

AI Technical Summary

Problems solved by technology

[0010] (1) The data distribution is inconsistent. A line of data may be in one file or across multiple files. In the worst case, it may be in each file. In this way, the number of files involved in different key queries is different, and the query performance is not good. Stablize;

[0011] (2) A lot of I / O is wasted, a file may be merged multiple times, and it is repeatedly rewritten like a snowball

Then, since there is no guarantee how long it will take for deleted data to be cleared, a large amount of space may be wasted in deletion-intensive operations;

[0013] (4) During the merging process, the old files can only be deleted after the new files are written, which requires additional disk space

Its problem is that the distribution and number of Stripes in Level-1 are difficult to adjust dynamically. If the Stripes are too small, the files written from Level-0 will be too small; if the Stripes are too large, the Stripe interval is very large, and each Stripe will encounter the same problem as TieredCompact inside the Region

[0032] The existing technology cannot dynamically adapt to the distribution of data, and the organization of data cannot be dynamically adjusted efficiently according to the distribution of data. The efficiency of merging is low, and some useless merging is often done, occupying system resources

[0033] In summary, there are obviously inconveniences and defects in the actual use of the existing technology, so it is necessary to improve

Images