Distributed storage method based on KEY-VALUE pair

Abstract

The invention particularly relates to a distributed storage method based on a KEY-VALUE pair. The distributed storage method based on the KEY-VALUE pair comprises an operation receiving layer, a cache searching layer and an operation processing layer, wherein after receiving user data, the operation receiving layer of a server completes splicing of the key-value pair through internal data organization; a bottom layer is divided into different data levels for disk landing and is meanwhile responsible for analyzing and reading following data; since the access of the key value pair belongs to lightweight database type operation without metadata, and is appended to write in, time consumption brought by disk landing of random small files can be avoided. According to the distributed storage method based on the KEY-VALUE pair, the bottom layer storage is modified to be a key-value pair mode, so that the whole storage cluster can more quickly and efficiently process the small files, and IO loss caused by frequent addressing along with small file random write-in in a traditional distributed storage or local storage mode is avoided, and the storage efficiency under such scene is greatly improved.

Description

technical field [0001] The invention relates to the technical field of distributed storage, in particular to a distributed storage method based on KEY-VALUE key-value pairs. Background technique [0002] With the rapid development of network applications, the amount of network information data is increasing, and PB-level mass data storage is becoming more and more important. The traditional local disk storage method can no longer meet the requirements of large capacity, high reliability, high performance, and horizontal expansion of existing applications. In order to meet these new requirements, network-based distributed storage systems have received extensive attention. [0003] The data generated by various applications such as social networks, mobile communications, network video and audio, e-commerce, sensor networks, and scientific experiments can often generate tens of millions, billions, or even billions or tens of billions of massive small files. In many scenarios, ...

AI Technical Summary

Problems solved by technology

The traditional local disk storage method can no longer meet the requirements of large capacity, high reliability, high performance, and horizontal expansion of existing applications. In order to meet these new requirements, network-based distributed storage systems have received extensive attention.

[0003] The data generated by various applications such as social networks, mobile communications, network video and audio, e-commerce, sensor networks, and scientific experiments can often generate tens of millions, billions, or even billions or tens of billions of massive small files. In many scenarios, these small files are generally very small fragmented data. Traditional storage is very inefficient when processing such a large number of small file writes. This is because a large number of small files often bring a large number of random writes. When writing locally, the performance will be greatly reduced due to frequent disk addressing. The distributed storage that is becoming more and more popular nowadays will have lower performance due to inherent network delays, multi-copy settings, and metadata writing.

Images