Small file aggregation storage processing method

Abstract

The invention provides a small file aggregation storage processing method. The method comprises following steps: setting an information list in a cache, wherein the information list is used for recording the ino and size information of the small file; writing the small file to the cache of the aggregate object, and recording the ino information of the small file into the information list; when the aggregate object cache reaches the refresh condition, extracting the ino of the small file from the information list and recording the ino in an array; after the ino of all small files are recorded in the array, inserting the array into a preset position of the aggregate object. By means of the small file ino stored in object header, small files contained in the current object can be clearly indicated. By means of bitmap information of the object header, the data storage status of the current object can be indicated; during scanning, the cache utilization rate of the current object can be calculated according to the bitmap.

Description

technical field [0001] The invention relates to the field of small file processing, in particular to a small file aggregation storage processing method. Background technique [0002] With the continuous development of technologies such as cloud computing, cloud storage, and the Internet of Things, data is showing an explosive growth trend. In some scenarios with a large number of small files, such as social shopping sites, radio and television, and online videos, the system generates a large number of small files such as text, pictures, and music. These files have the following characteristics: large in number, and the size is generally below 1M. Fast flow, frequent operations (write once, read many times). By aggregating small files into a large aggregate file in a tightly aligned manner, and then storing them, small files can be stored efficiently, disk usage can be improved, and the number of Rados calls can be reduced during file reading and writing, reducing the need f...

AI Technical Summary

Problems solved by technology

However, when small files are stored in close alignment, multiple small files are stored on the same object, and the file information stored in the object cannot be determined when the file is read from the object

Since the disk space is not really released when the small file is deleted, it is necessary to count the disk usage of each object of the aggregated file, and determine whether to start the disk cleanup task according to the disk usage of the aggregated file. Therefore, it brings trouble to the disk management and cannot be timely Clear useless small files, wasting disk space

Images