File system embedded in horizontally expanded key-value storage system and file management method

Abstract

The invention discloses a file system embedded in a horizontally expanded key-value storage system and a file management method, belonging to the technical field of information storage. The file system includes: a memory layer, a first key-value storage layer and a second key-value storage layer; a memory layer It is used to cache externally written data, and write the data into the first key-value storage layer after reaching the preset writing amount; the first key-value storage layer is used to receive and store data from the memory layer through the first data table, and in When the first data table is full, write the content stored therein to the second key-value storage layer; the second key-value storage layer is used to receive and store data from the first key-value storage layer through the second data table; the first key-value The storage layer also includes a secondary index structure for recording the mapping relationship between the directory / file and the first data table, and the key value ranges between the first data tables belonging to the same directory are ordered. The invention can significantly improve the problems of read amplification, write amplification and space enlargement existing in the file system used for storing massive small files.

Description

technical field [0001] The invention belongs to the technical field of information storage, and more specifically relates to a file system embedded in a horizontally expanded key-value storage system and a file management method. Background technique [0002] In recent years, the rise and development of the Internet of Things, machine learning, and the Internet have brought massive amounts of small file data, typically data from e-commerce, social networking, and sensors. For massive small files, many platforms have proposed special file systems to optimize the storage performance of small files. The famous social networking site Facebook stores more than 260 billion pictures, and launched Haystack, which is customized and optimized for massive small pictures, and the average size of the pictures is only tens of KB. Taobao should be the largest C2C e-commerce website at present, storing more than 20 billion pictures, with an average size of only 15KB, and also launched Taob...

AI Technical Summary

Problems solved by technology

[0004] However, the log-structured merge tree is still a hierarchical structure, and the key-value storage system based on the log-structured merged tree still has the problems of read amplification and write amplification caused by the hierarchical structure. It will cause space enlargement, and the entire file system also has serious space enlargement problems

In general, the existing file system embedded in the key-value storage system has serious problems of read and write amplification and space enlargement when storing a large number of small files.

Images