Key value storage method and system oriented to LSM tree

Abstract

The invention provides a key value storage method and system oriented to an LSM tree. The method comprises the following steps: performing fine-grained division on disk levels, and setting a compaction strategy as follows: in a compaction task, all upper sub-levels participate in the task, and only one sub-level participates in the task at a lower layer, so as to reduce the proportion of lower-layer participation data to total participation data; divding the compaction task when the compaction task is executed, so that the number of files participating in the compaction task is reduced, and the degree of parallelism of the compaction is improved. The influence on the reading performance is reduced through a parallel reading algorithm, modeling is carried out through write amplification of the LSM tree, and a method for selecting parameters enabling write amplification to be minimized is provided.

Description

technical field [0001] The invention relates to computer storage technology, in particular to an LSM tree-oriented key-value storage method and storage system. Background technique [0002] Key-Value Store (Key-Value Store) stores data as a <key-value> collection, where the key acts as a unique identifier for the value. It does not support complex relational schemas like relational databases, but processes data through simple interfaces such as Put(k,v), Get(k), Update(k,v), and Delete(k). Due to its high performance, high scalability and other advantages, it plays an important role in today's network applications and distributed systems, and is widely used in graph databases, task queues, stream processing engines, application data caches, and event tracking systems. and other fields. [0003] LSM tree (Log-Structured Merge tree) is a storage engine widely used in key-value storage systems. It maintains a cache in memory. When a user writes a key-value pair, the da...

AI Technical Summary

Problems solved by technology

Excessive write amplification seriously affects the write performance of the LSM tree

However, LSM trees are often run in computers using SSDs, and frequent hard disk reads and writes will reduce the lifespan of SSDs.

In summary, for the LSM tree structure, write amplification is a very serious problem

On the other hand, when memory buffer and disk L 1 When the amount of data exceeds the threshold, the memory data cannot be flushed and must wait for L 1 Complete a compaction to free up a certain space for this layer before serving new write requests, resulting in write pauses, that is, a significant increase in phased write delays

Images