File system embedded with transverse extension key value storage system and file management method

Abstract

The invention discloses a file system embedded with a transverse extension key value storage system and a file management method, and belongs to the technical field of information storage. The file system comprises a memory layer, a first key value storage layer and a second key value storage layer. The memory layer is used for caching external write-in data, and writing the data into the first key value storage layer after a preset write-in amount is reached; the first key value storage layer is used for receiving and storing the data from the memory layer through the first data table, and writing the content stored in the first key value storage layer into the second key value storage layer when the first data table is full. The second key value storage layer is used for receiving and storing the data from the first key value storage layer through the second data table. The first key value storage layer further comprises a secondary index structure used for recording the mapping relation between the directories / files and the first data tables, and the key value ranges among the first data tables belonging to the same directory are orderly. The problems of read amplification, write amplification and space amplification of a file system for storing massive small files can be remarkably improved.

Description

technical field [0001] The invention belongs to the technical field of information storage, and more specifically relates to a file system embedded in a horizontally expanded key-value storage system and a file management method. Background technique [0002] In recent years, the rise and development of the Internet of Things, machine learning, and the Internet have brought massive amounts of small file data, typically data from e-commerce, social networking, and sensors. For massive small files, many platforms have proposed special file systems to optimize the storage performance of small files. The famous social networking site Facebook stores more than 260 billion pictures, and launched Haystack, which is customized and optimized for massive small pictures, and the average size of the pictures is only tens of KB. Taobao should be the largest C2C e-commerce website at present, storing more than 20 billion pictures, with an average size of only 15KB, and also launched Taob...

AI Technical Summary

Problems solved by technology

[0004] However, the log-structured merge tree is still a hierarchical structure, and the key-value storage system based on the log-structured merged tree still has the problems of read amplification and write amplification caused by the hierarchical structure. It will cause space enlargement, and the entire file system also has serious space enlargement problems

In general, the existing file system embedded in the key-value storage system has serious problems of read and write amplification and space enlargement when storing a large number of small files.

Images