Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for reducing garbage collection and write amplification of key-value separation storage system

A separate storage and write amplification technology, applied in the direction of memory system, file system, file system type, etc., can solve the problems of performance not increasing but degrading, and achieve the effects of improving efficiency, reducing garbage collection operations, and reducing frequency

Active Publication Date: 2021-02-23
HUAZHONG UNIV OF SCI & TECH
View PDF14 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Wisckey also has another shortcoming. For key-value pairs with small values, the performance does not increase but decreases after the key-value separation, and the problem of space enlargement is obvious.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for reducing garbage collection and write amplification of key-value separation storage system
  • Method and system for reducing garbage collection and write amplification of key-value separation storage system
  • Method and system for reducing garbage collection and write amplification of key-value separation storage system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0061] A method for reducing write amplification and garbage collection in a key-value separation storage system such as figure 1 As shown, it includes memory, NVM and SSD. A first storage area and a second storage area are created in NVM. The first storage area manages data in the form of LSM-tree, and the second storage area manages data in the form of memory pool;

[0062] Methods include:

[0063] LSM-tree insertion steps:

[0064] Insert the key-value pair into the variable table in the memory; after the variable table becomes an immutable table, the stored key-value pair will be flushed to the first storage area;

[0065] In a key-value separation storage system such as levelDB, before the key-value pair is inserted into the LSM-tree, it will be inserted into the variable table in the memory, that is, figure 1 The memtable in the variable table is specifically a kind of jump table; when the variable table is full, it will become an immutable table, that is, figure 1 T...

Embodiment 2

[0100] A key-value separation storage system, including: memory, NVM, and SSD. A first storage area and a second storage area are created in the NVM. The first storage area uses LSM-tree to manage data, and the second storage area uses a memory pool. manage data in the form of

[0101] The key-value separation storage system also includes a data management module, which includes:

[0102] The LSM-tree insertion unit is used to insert the key-value pair into the variable table in the memory; after the variable table becomes an immutable table, the stored key-value pair will be flushed to the first storage area;

[0103] The memory pool insertion unit is used to insert the key-value pair into the second storage area, and form a new key with the key key of the key-value pair and its storage address addr in the second storage area The value pair , and use the LSM-tree insertion unit to insert the key-value pair ;

[0104] The key-value pair insertion unit is used to ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a system for reducing garbage collection and write amplification of a key-value separation storage system, and belongs to the field of computer storage, the systemcomprises a memory, an NVM and an SSD; and a first storage area and a second storage area in the NVM manage data according to an LSM-tree and a memory pool respectively; the method comprises the steps of judging whether the size of a to-be-inserted key value pair is greater than or equal to a first threshold value or not, if so, inserting the key value pair into the second storage area, and theninserting a key and a key value address into LSM-tree; otherwise, directly inserting the key value pair into LSM-tree; when the key value pair in the memory pool is deleted, marking the correspondingstorage unit as an idle memory unit; when the data volume in the second storage area is higher than a second threshold value, brushing down the key value pair with the least frequent access to the SSD, and updating the storage address of the key value pair; and setting a flag bit segment on the NVM, and identifying the validity of the key value pair of the data segment on the SSD. Garbage collection and write amplification of the key value storage system can be reduced, and the overall performance of the key value separation storage system is improved.

Description

technical field [0001] The invention belongs to the field of computer storage, and more specifically relates to a method and system for reducing garbage collection and write amplification of a key-value separation storage system. Background technique [0002] In recent years, with the rapid development of cloud computing, big data, Internet of Things, artificial intelligence and other information technologies and the extensive use of online shopping, online ordering, Weibo, and other software, the amount of data has exploded. How to effectively store and manage Massive data poses new challenges to existing computer system storage technologies. [0003] As we all know, the sequential writing of the disk is about 1000 times that of the random writing. The structure of LSM-tree (Log-StructuredMerge Tree, log structure tree) converts random writing into sequential writing, which greatly improves the performance of data writing to disk. However, the merging process of LSM-tree b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F12/02G06F16/13G06F16/18G06F16/901
CPCG06F12/0253G06F12/023G06F16/13G06F16/1847G06F16/9027G06F2212/1016G06F2212/1032G06F2212/1044
Inventor 陈俭喜冯丹郑梦丽马莉珍陈彧陈鑫宇张兴锐
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com