Writing and block granularity compressing and combining method and system of key value storage system based on OCSSD

Abstract

The invention discloses a writing and block granularity compressing and combining method and system of a key value storage system based on an OCSSD, and belongs to the field of computer storage. Through selective compression and combination of data block granularity, read-in and write-back of non-overlapping data blocks are avoided, and the problem of read-write amplification in the LSM-tree compression and combination process can be reduced; the data blocks of the same file are uniformly distributed in all channels of the OCSSD, so that the multi-channel parallel characteristic of the OCSSD can be explicitly utilized, and the throughput of a key value storage system is improved; by directly managing a bottom storage medium at a user layer and bypassing intermediate software layers such asa file system, consumption of host resources is reduced, and semantic features of application software are conveniently transmitted. Generally speaking, the problem of read-write amplification of thekey value storage system based on the LSM-tree can be effectively reduced, semantic isolation between upper-layer application software and bottom-layer storage equipment is eliminated, and the throughput of the whole key value of the system to operation is improved.

Description

technical field [0001] The invention belongs to the field of computer storage, and more specifically relates to a method and system for writing and block granularity compression and merging of an OCSSD-based key-value storage system. Background technique [0002] The application trend of large physical blocks of large-capacity solid state drives (SSD) and the development of 3D NAND Flash technology have brought opportunities and challenges to data storage management. Log-structured merge-tree (LSM-tree) key-value (key-value, KV) storage is optimized for random writes, providing efficient data insertion and deletion, and is widely used as a persistent storage engine in data center. LSM-tree KV storage directly uses SSD block devices, both of which have read / write amplification. What's worse is that the management of the host side and the device side are independent, and the problem of system read / write amplification will be further superimposed, seriously reducing the overa...

AI Technical Summary

Problems solved by technology

However, this method has the following defects: 1) When looking for a key-value pair in the lower-level file, it is necessary to read the upper-level file segment corresponding to the SliceLink first, and then read the content of the lower-level file itself if it is not hit, which introduces additional Small-grained read operations will lead to the read amplification problem of GET operations; 2) The number of overlapping file fragments between upper-level files and lower-level files is uncontrollable, and there is a hidden danger of excessive SliceLink metadata; 3) Relying on the file system, LSM-tree compression The merge operation, the GC operation of the file system, and the GC operation inside the SSD have functional redundancy. At the same time, the Key index of the LSM-tree and the namespace management of the file system also have functional redundancy.

Images