Read-write solution for tens of millions of small file data

Abstract

The invention provides a read-write solution for tens of millions of small file data. According to the solution, the mode of creating large-block continuous disk space is used for storing a large number of small files when the small files are stored, namely, logically continuous data are stored on the continuous space of a disk array as far as possible; the disk space is divided into a plurality of blocks, and the size of each block is 64 KB. According to the basic thought, each small file can only be stored in the single block and cannot be stored by crossing two blocks, each folder can be provided with one or more blocks, the blocks are only used for storing the data of the corresponding folder, and each piece of file data is stored on the continuous disk space. Compared with the prior art, the logically continuous data are stored on the continuous space of each physical disk as far as possible, the cache technology is used for playing the role of a metadata server, the cache utilization rate is improved through simplified file information nodes, and thus the access performance of the small files is improved.

Description

technical field [0001] The invention relates to the technical field of computer applications, in particular to a reading and writing solution for tens of millions of small file data. Background technique [0002] At the current stage of reading and storage, small files are the most common data form for data access and use. Compared with the striping technology of large files, slicing is used to improve the concurrency of user access to files. Small files (≤64KB) are not conducive to striping. The traditional method is to store a single file in a single data server. However, when the number of small files reaches a certain level, a large number of repeated accesses to small files will bring performance burdens and I / O bottlenecks to the data server. The form of small files with high frequency is manifested, and in the information reading and storage of general users, there are more reading and storage of small files, so the research on the performance of high-frequency small...

AI Technical Summary

Problems solved by technology

[0004] (1) Due to the high access frequency of small files and the need to access the disk multiple times, the performance of disk I / O is low;

[0005] (2) Because the file is relatively small, it is easy to form file fragments and cause waste of disk space;

[0006] (3) It is easy to generate network delay when establishing a connection for each small file request, which reduces the reading rate of small files