High-performance and easy-to-expand key value storage method utilizing differential index mechanism

A kind of indexing and high-performance technology, which is applied in the field of high-performance and easy-to-extend key-value storage, can solve the problems of not being able to improve read and write performance at the same time and overall performance, and achieve the effect of ensuring consistency and low overhead

Active Publication Date: 2020-02-21
PINGCAP XINGCHEN (BEIJING) TECH CO LTD
View PDF5 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to propose a high-performance and easy-to-extend key-value storage method using a differentiated index mechanism to overcome the problems that the existing technology cannot simultaneously impro...

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High-performance and easy-to-expand key value storage method utilizing differential index mechanism
  • High-performance and easy-to-expand key value storage method utilizing differential index mechanism
  • High-performance and easy-to-expand key value storage method utilizing differential index mechanism

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] The present embodiment illustrates the detailed operation process and working method of the present invention by operating actual key-value pairs (key-value pairs). The system-related parameters are set as follows: the storage capacity of each partition is 40GB, and , the data size limit of the unordered layer of the hash index is 4GB. First, randomly write key-value data pairs with a total amount of 100GB to 1000GB into the invented key-value storage system, and then send 10M to 100M random read requests to the storage system where the data is written. At the beginning of the application, the key-value storage system is empty, and all data is written to one partition. attached figure 1 A schematic diagram of the high-performance and easily scalable key-value storage method using the differential index mechanism in this embodiment is given, including the differential index design based on the hash index, partial separation of key-value data pairs, and key-value range-b...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a high-performance and easy-to-expand key value storage method utilizing a differential index mechanism. The method is characterized by comprising the steps of differential index design, partial separation of key value data pairs, data dynamic partition management based on a key value range and range query optimization. Due to the fact that a two-layer storage architectureis adopted and the hash index is established for the hot data of the disorder layer, the system can fully utilize the locality principle of the data to accelerate access to the hot data, and low indexmemory overhead and efficient data query efficiency are guaranteed; in addition, parts of the key value data pairs are separately stored, so that the merging efficiency is greatly improved; secondly,dynamic partition management is carried out on data in different key value ranges, transverse horizontal expansion of key value storage is achieved, and it is guaranteed that efficient read-write performance can be obtained under large-scale data storage; and finally, merging strategy based on the number of files is designed, log files are read in parallel by adopting a multi-thread technology, and range query is accelerated in combination with a pre-reading mechanism.

Description

technical field [0001] The invention belongs to the technical field of computer storage systems, and in particular relates to using the locality of workload to construct a differentiated index mechanism in a key-value storage (KV Stores) system, and on this basis combined with data dynamic partitioning based on key-value ranges Management, implement high-performance and easy-to-extend key-value storage method. Background technique [0002] According to companies such as Google (Google) and Facebook (Facebook), in order to meet the storage and access requirements of massive unstructured data and overcome the shortcomings of traditional file storage in terms of scalability and performance, key-value storage (key- Value storage or KV storage for short) provides a good solution and has been widely adopted, such as Amazon's Dynamo, Baidu's Atlas and Google's LevelDB and other key-value storage systems. The above mainstream key-value storage systems all use log-structure merge tr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F16/2453G06F16/2455
CPCG06F16/2272G06F16/2453G06F16/24552Y02D10/00
Inventor 崔秋唐刘张强
Owner PINGCAP XINGCHEN (BEIJING) TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap