Method and system for merging data based on log structure merging tree, equipment and medium

Abstract

The invention discloses a method and system for merging data based on a log structure merging tree, equipment and a medium. The method comprises the steps of dividing a disk into data blocks of a preset size, saving the data in the data blocks, and information of the data in index blocks; merging the current preset number of index blocks at preset intervals to form a new index block, and judging whether the effective data of the corresponding data block is smaller than a threshold value or not according to the information of the new index block; in response to the fact that the valid data of the multiple data blocks is smaller than a threshold value, combining the valid data of the multiple data blocks, and releasing the data blocks without the valid data; and in response to the fact that the valid data of only one data block is smaller than the threshold, merging the valid data of the data block into the data block with the lowest valid data except the data block. The data index is separated from the data, the data is stored in the pre-distributed data block, and the data is merged and recycled according to the index, so that the read-write data volume of the disk is reduced.

Description

technical field [0001] The present invention relates to the field of data processing, and more specifically refers to a method, system, computer equipment and readable medium for merging data based on a log-structured merging tree. Background technique [0002] LSM-Tree (Log-Structured Merge-Tree, Log-Structured Merge-Tree) was first proposed in a paper "The Log-Structured Merge-Tree (LSM-Tree)". It is not a product or system itself, but a Algorithms and ideas for problem solving. Based on this idea, multiple storage products have been implemented, such as NoSQL storage systems such as HBase and LevelDB. Its core idea is to convert random writes into sequential writes to improve write performance, and then periodically perform data consolidation (Compaction) to improve read performance. Based on this paper, a common implementation method is to layer the data into L0-Ln respectively, and then periodically merge the data of the L layer into the L+1 layer. This method will ge...

AI Technical Summary

Problems solved by technology

Based on this paper, a common implementation method is to layer the data into L0-Ln respectively, and then periodically merge the data of the L layer into the L+1 layer. This method will generate a large number of disk reads and write, and ultimately affect the performance of upper-layer applications

Images