Object storage small file processing method and device, equipment and storage medium

Abstract

The invention discloses an object storage small file processing method, device and equipment and a computer readable storage medium. The method comprises the steps that whether a small file containedin an object data writing request is smaller than or equal to a preset value or not is judged; if yes, whether the small file is written for the first time or not is judged, if yes, data in the smallfile is written into the aggregation file, and the identifier of the aggregation file, the length of the data and the position of the data in the aggregation file are stored in metadata of the small file; judging whether the capacity of the aggregation file exceeds a set capacity or not, if yes, writing the data into a new aggregation file, and if not, returning to execute the step of receiving the object writing request; writing the aggregated file reaching the set capacity into the OSD, and storing metadata into a database. According to the technical scheme disclosed by the invention, the plurality of small files are aggregated into one aggregated file, so that the writing frequency of the small files is reduced, and the load of frequent reading and writing of the small files on a storage system is effectively reduced.

Description

technical field [0001] The present application relates to the technical field of Ceph distributed file system, and more specifically, relates to a method, device, device and computer-readable storage medium for object storage and small file processing. Background technique [0002] Ceph is a distributed file system with excellent performance, strong reliability and scalability, which can provide three applications: file system, block storage and object storage. [0003] At present, for the object storage in the Ceph storage system, when writing small files, it is usually to write a single small file as a whole to the OSD (Object Storage Device, an object storage device, a kind of storage device related to the actual storage device). There is a logical concept of mapping relationship). However, as more and more data needs to be stored, a large number of small files will frequently call the write interface to write a large number of small files to the OSD, and the frequent wr...

AI Technical Summary

Problems solved by technology

However, as more and more data needs to be stored, a large number of small files will frequently call the write interface to write a large number of small files to the OSD, and the frequent writing of small files will affect the Ceph storage system. Causes a relatively large load, reduces small file writing performance, and increases OSD read and write pressure

Images