A two-stage merging method based on log-structured merging trees

Abstract

The invention relates to a log structure combination tree-based two-stage combination method. The method comprises the following steps of 1) selecting out a layer with most unreasonable spatial distribution in an open source system according to an imbalance score; 2) selecting out a target file in the layer with the most unreasonable spatial distribution according to a polling rule; 3) segmenting the target file into fragments according to lower layer files covering a same key value range, linking each fragment with the lower layer file of the corresponding key value range, and adding SliceLinks for each lower layer file; 4) checking a SliceLink quantity of each lower layer file, entering the step 2) if the SliceLink quantities of all the lower layer files do not exceed a preset threshold, and entering the step 5) until the SliceLink quantities of the lower layer files exceed the preset threshold; and 5) reading the lower layer files with the SliceLink quantities exceeding the preset threshold, and the target file having the corresponding key value range and linked with the fragments of the lower layer files into a memory of the open source system to perform combination to generate a new file, and writing the new file into a layer where the lower layer files are located. The method can be widely applied to the technical field of information storage.

Description

technical field [0001] The invention relates to a two-stage merging method based on a log structure merging tree, and belongs to the technical field of information storage. Background technique [0002] With the advent of the Web 2.0 era, the data scale is growing explosively. The traditional relational database has been difficult to meet the needs of the era of massive data storage, and the non-relational database with the characteristics of fast read and write speed, easy expansion, and low cost Databases are widely used. According to storage methods and storage content, non-relational databases can be divided into column storage, document storage, key-value storage, graph storage, and object storage. Among them, key-value storage is simple and very suitable for databases that do not involve too many data relationships and The business data of business relationships has been called the mainstream storage method in the business of major Internet companies. Key-value stora...

AI Technical Summary

Problems solved by technology

Since the merge operation is to push down the data, it is inevitable to read and write a file in the upper layer, and the only way to reduce the read and write amplification ratio is to start from the lower layer.

However, the high reading and writing amplification ratio of the log-structured merge tree will also lead to obvious access delay and jitter, which is not suitable for applications in non-durable devices, and often cannot provide high-quality storage services in practical applications.

[0004] To sum up, the existing data merging method of log-structured merging tree is a single-stage merging method, that is, selecting a small number of upper-level data and a lot of lower-level data for merging. This merging method is relatively simple, but it will cause serious writing problems. Amplification and storage service performance degradation issues

Images