Distributed object storage system and method

Abstract

The invention discloses a distributed object storage system and method. The system comprises a main park system, at least one standby park system, a message queue and a data copying module, wherein the main park data service module is used for packaging a plurality of to-be-stored objects into files, storing the files into a distributed file system of a main park system, generating index information of the objects, and adding the index information into a distributed database and a message queue of the main park system; the data copying module is used for acquiring the index information from the message queue; and adding index information to a distributed database of the standby park system, searching the file stored in the distributed file system of the main park system according to the index information, and adding the file to the distributed file system of the standby park system. The technical problem that an existing distributed object storage system is not suitable for storing a large number of small objects is solved.

Description

technical field [0001] The present invention relates to the technical field of object storage, in particular to a distributed object storage system and method. Background technique [0002] Object storage is a technology often used in the Internet. Object storage is different from file storage in that objects are not organized in a hierarchical directory structure. Each object is located at the same level of a flat space called a storage pool, and each element of each level has a unique identifier in the storage system, and users access the container or object through this identifier. In object storage, a flat data organization structure is generally adopted and nested folders are discarded, so as to avoid maintaining a huge directory tree. [0003] Currently, object storage usually adopts a distributed storage method. However, distributed object storage has inherent shortcomings. If the object is stored as a separate file in the distributed file system (HDFS), due to the...

AI Technical Summary

Problems solved by technology

However, distributed object storage has inherent deficiencies

If the object is stored as a separate file in the distributed file system (HDFS), due to the memory limitation of the NameNode (NameNode), the number of stored files has an upper limit. A single object is usually small, and storing a large number of small files will be extremely difficult. Large waste of storage space in the entire distributed file system

In addition, most file systems currently have poor disaster recovery capabilities. In the event of an accident such as a power outage, data cannot be read at all, and objects are prone to loss.

Images