Capacity expansion method for gigabit-level object storage bucket

Abstract

The invention discloses a capacity expansion method for a gigabit-level object storage bucket. The capacity expansion method comprises that: a unified management platform comprising a virtual mapping layer, a metadata center, an entry component and an operation and maintenance component is constructed; and the control planes and the data planes stored by the heterogeneous objects are managed in a unified mode through the unified management platform, and smooth switching in the expansion period is achieved. The capacity expansion method specifically comprises that: 1) the scale risk of a single cluster and a single bucket is avoided, and horizontal capacity expansion can be realized naturally; the service is not perceived during capacity expansion, and the hot switching configuration directly takes effect; 2) the performance bottleneck of a single cluster and a single bucket does not exist, and the performance reliability of the object storage service is ensured by accessing a plurality of heterogeneous object storage clusters and a virtual mapping method; and 3) the performance bottleneck of a single cluster and a single bucket does not exist, and the performance reliability of the object storage service is ensured by accessing a plurality of heterogeneous object storage clusters and a virtual mapping method.

Description

technical field [0001] The present invention relates to the technical field of cloud storage, in particular to a method for expanding the capacity of a 100-billion-level object storage bucket. Background technique [0002] Due to the rapid development of big data and AI technology, higher requirements are placed on the underlying storage system: larger capacity, faster response speed, larger bandwidth, and the existing object storage single-cluster single-bucket data scale After exceeding 500-100 million, the performance drops sharply, and the metadata storage expands rapidly. At this time, the expansion in a single bucket brings great challenges. [0003] Most of the existing single-bucket expansion of object storage only supports expansion under a single cluster. Taking the two open source storage systems (SWIFT and CEPH) in the field of object storage as examples, we will briefly introduce the current single-bucket expansion method of object storage. [0004] The storage...

AI Technical Summary

Problems solved by technology

[0004] The storage engine of Swift open source object storage bucket is sqlite database. Once its index reaches about 500 million, its performance will degrade seriously. At this time, it is difficult to expand the capacity in a single cluster or single bucket

After the bucket index in the Ceph open source object storage bucket reaches the scale of 100 million, you can use the metadata index reorganization strategy, but it will cause a large amount of metadata balance, and once the single bucket reaches billions, the metadata expansion and reorganization will affect the system. The performance impact is quite challenging

[0005] Existing technologies often only consider the metadata storage scale in a single cluster and single bucket, which is very limited. How to achieve a scale of hundreds of billions in a single bucket of object storage and ensure no perception of business during expansion is an urgent need to solve The problem

Images