File storage method, file reading method and data storage system

Abstract

The invention provides a file storage method, a file reading method and a data storage system. The file storage method comprises the following steps: receiving storage requests of a plurality of filesassociated with the same object; storing the first key value pair of each file in the plurality of files in a first memory table of a memory; when the first memory table is full, transferring data inthe first memory table to a persistent memory and emptying the first memory table, wherein transferring the data in the first memory table to a persistent memory comprises: storing a second key valuepair of each of the plurality of files in an LSM tree-based first storage file in the persistent memory, and sequentially storing the third key value pair of each of the plurality of files in a plurality of storage units in the second storage file.

Description

technical field [0001] The present invention generally relates to the field of data storage, and more specifically, relates to a file storage method, a file reading method, and a data storage system based on an LSM (Log-Structured Merge) tree. Background technique [0002] LSM tree is a data storage structure, often used to build NoSQL key-value pairs (key-value) database. For example, currently known LevelDB, RocksDB, hbase, etc. are all key-value pair databases based on LSM trees. [0003] figure 1 A schematic diagram of an LSM tree-based data storage system 10 in the prior art is shown. Note that in figure 1 In , the RocksDB database is used as an example to show the names of various components, and the names of corresponding components in other databases may be the same or different. [0004] Such as figure 1 As shown in , the LSM tree-based data storage system 10 includes a memory 110 and a persistent storage 120 (such as a hard disk). In addition, although not sh...

AI Technical Summary

Problems solved by technology

It is friendly to read and write small files (such as text descriptions and comments less than 1kb), but relatively inefficient to read and write larger files (such as pictures or video files ranging from tens of kb to several Mb)

On the other hand, multiple files associated with a product are often stored and read simultaneously, and for each file, the data storage system 10 based on the LSM tree will search for all files according to the complete read operation process as described above. Requested file data without considering the relationship between these files, resulting in even lower reading efficiency

Images