A data object storage method and device

Abstract

The invention discloses a data object storage method and device. The method comprises the steps of repeatedly judging whether the ratio, larger than a preset value, of the storage space occupied rate of one object storage devices to the average value of the storage space occupied rates of all object storage devices exists or not, according to a controlled copying algorithm under extensible hashing, determining the object storage devices corresponding to one collocation set again till the ratio, larger than a preset value, of the storage occupied rate of the object storage device to the average value of the storage space occupied rates of a second number of object storage devices does not exist; storing one data object and a copy thereof into the object storage devices except the object storage device with the ratio, larger than the preset value, of the storage space occupied rate to the storage space occupied rates of all the object storage devices, so that the storage space occupied rates between the stored object storage devices are kept close.

Description

technical field [0001] The invention relates to data storage technology, in particular to a data object storage method and device. Background technique [0002] The Ceph system is an open source distributed storage system for storing PB (abbreviation for petabyte, that is, 2 to the 50th power of bytes) level data using hundreds of storage devices. The Ceph system must balance the data and load distributed on these storage devices to improve resource utilization and maximize system performance, and the Ceph system needs to deal with system storage hardware expansion and storage hardware failure. In order to ensure the load balance of each storage device in the Ceph system, it is necessary to store new data and old data together in each storage device. However, the simple hash (Hash) distribution cannot effectively handle the change of the number of storage devices in the Ceph system, resulting in a large amount of data migration in the storage devices. In view of the above ...

AI Technical Summary

Problems solved by technology

[0006] The defect of the current CRUSH algorithm is that: since the mapping relationship between placement groups and object storage devices is completely based on a pseudo-random algorithm, in theory, when the number of placement groups and the number of object storage devices are large, each The number of placement groups corresponding to object storage devices tends to be the same, but in the actual production environment, the number of placement groups corresponding to each object storage device will have a large gap, and the maximum gap in the space usage of object storage devices is up to 20%

[0007] For example, for the Ceph system, by default, when the storage space occupancy rate of an object storage device reaches 85%, it is in a near full state, and when the occupancy rate reaches 95%, it is in a saturated (full) state, but the storage space of other object storage devices The space occupancy rate may be only 65% ​​or 75%. However, in this case, the storage device cluster in the Ceph system is close to being unable to store data, resulting in a lot of space waste

Images