A method and device for storing and accessing massive small files

Abstract

An embodiment of the application discloses a method for storing massive small files. The method comprises the steps of: writing sequence of received small files into a memory element; determining a file group to which the small files belong as well as a sequence of the small files in the file group according to starting address and capacity of the small files in the memory element, wherein the file group includes at least two small files; and taking an identification number of the file group and a sequence number of the small files as an index, and establishing a correspondence between the index and file names of the small files to realize storage. The embodiment of the application further discloses a method for accessing the massive small files, as well as a storing device and an accessing device for the massive small files, corresponding to the storing method and the accessing method. The methods and devices provided by the invention can compress directory indexing structure of the small files, save storage cost and improve storage and access efficiency of the small files.

Description

technical field [0001] The present application relates to the technical field of data storage and access, and in particular to a method for storing a large number of small files and its corresponding device, an access method and its corresponding device. Background technique [0002] With the development of information technology, all kinds of information are increasing rapidly, and individual files as the carriers of these information appear in large numbers, especially small files with not too large capacity. These small files can be as small as a few KB, and the large ones usually do not exceed 20MB. Common small files, such as: Weibo information, photos uploaded by users, emails, UGC data, etc. The development bottleneck brought about by the emergence of a large number of small files is the storage and access problem of small files. In the prior art, usually each small independent file is directly stored in a disk or a RAID disk array (Redundant Array of Independent Di...

AI Technical Summary

Problems solved by technology

Storage of a small number of small files on a disk or RAID disk array can still meet the application requirements, but when the number of small files stored increases to a certain scale (massive), the directory index structure of the storage system is very large and the number of nodes is large, and the files established for access Directory systems become bloated, increasing storage overhead

Moreover, the high-speed cache set up to improve the access speed will not be able to accommodate all directory indexes, and at least part of the directory index data will be transferred and stored to the disk area, which may result in multiple I / O operations to access a small file, thereby greatly reducing the The access rate of small files

In addition, since small files usually include metadata for describing small file attribute information (such as read and write times, access time, etc.), while storing massive small file data, additional storage space must be added for storing these pairs. User unnecessary metadata, resulting in waste of storage space

Images