A key-value storage method based on log-structure merged tree

A log and key-value technology, applied in the input/output process of data processing, instruments, input/output to record carriers, etc., can solve the problems of speeding up writing, writing amplification, reducing the frequency of sorting and merging, and avoiding I/O O operation, reduce write amplification problems, and improve the effect of throughput

Active Publication Date: 2018-11-13
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This layer-by-layer flow mechanism leads to severe write amplification
The research of bLSM and PCP focuses on reducing the frequency of sorting and merging or speeding up writing, but ignores the layer-by-layer flow of key-value pairs that is the root cause of severe write amplification

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A key-value storage method based on log-structure merged tree
  • A key-value storage method based on log-structure merged tree
  • A key-value storage method based on log-structure merged tree

Examples

Experimental program
Comparison scheme
Effect test

example 1

[0034] Example 1 Key-value storage system based on log structure merge tree

[0035] 1) Judging the existence of key-value pair version based on Bloom Filter (BF)

[0036] In this method, if a specific key-value pair (kv) is to be jumped down, the first thing to ensure is that the skipped data storage component does not have any version of the key-value pair corresponding to the kv.

[0037] Two adjacent data storage components C i And data storage component C i+1 Take the key-value pairs after sorting and merging as an example. A typical LSM-tree-based key-value storage system places the key-value pairs in the data storage component C i+1 , This method tries to put it in the data storage component C i+M , M is greater than or equal to 1, and M is as large as possible.

[0038] This method uses a level-by-level judgment mechanism to judge whether it can jump to the data storage component C. i+M . You can skip to data storage component C i+M The condition is: data storage component C i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a key value storage method based on a log-structured merged tree. The method comprises: 1, setting a cache component in a memory for each data storage component, setting a Bloom filter for each cache component, and setting a Bloom filter for each data block of each SSTable; 2, carrying out jump judgment on each key value pair of current data blocks of to-be-merged data storage components Ci stage by stage by adopting the corresponding Bloom filter, wherein if the key value pair jumps to a C<i+M>, no version of key value pair corresponding to keys exists in the C<i+M> and each stage of component in the front of the C<i+M>, a B<i+M+1> comprises a corresponding version of key value pair and then the key value pair is set into a B<i+M>, and if the key value pair is jumped to a B<i+N>, no version of key value pair corresponding to keys in the B<i+N> and each stage of component in the front of the B<i+N> and a C<i+N> comprises a corresponding version of key value pairs; and 3, processing the to-be-merged key value pairs in the Bi by adopting a similarity method.

Description

Technical field [0001] The invention belongs to the technical field of computer software and relates to a key value storage method based on a log structure merge tree. Background technique [0002] In current data centers, key-value storage systems have become the core of large-scale data-intensive network applications. Many researches are devoted to designing high-performance and high-scalability key-value storage systems. Log-Structured Merge-tree (LSM-tree) is widely used in current emerging Internet applications because of its simultaneous support for incremental writing, low writing latency, and range-based scanning. However, the LSM-tree-based key-value storage system has serious write amplification. In order to improve the write performance of the system, bLSM adopts a replacement selection sort algorithm to reduce the frequency of sorting and merging (R.Sears and R. Ramakrishnan.bLSM: A General Purpose) Log Sturctured MergeTree.in SIGMOD'2012), PCP proposes a pipelined ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F3/0613G06F3/0643G06F3/0676
Inventor 岳银亮刘帆李宇哲王伟平
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap