Capacity balancing method and device of distributed storage system, program product and medium

Abstract

The invention discloses a capacity balancing method and device of a distributed storage system, a program product and a medium, and relates to the technical field of distributed storage, the method comprises the following steps: when the current number of iterations does not exceed the target number of iterations, determining a to-be-migrated storage device according to the capacity utilization rate of each storage device; wherein the capacity utilization rate is the quotient of the used space of each storage device and the device capacity; determining a to-be-migrated fragment group in the to-be-migrated storage device and a target storage device corresponding to the to-be-migrated fragment group; wherein the capacity utilization rate of the target storage device is lower than that of the to-be-migrated storage device; migrating the to-be-migrated fragment group to a target storage device; it can be seen that the capacity balancing method and device can balance the capacity in an iteration mode, the object fragments can be redistributed to the storage devices with the low capacity utilization rate from the storage devices with the high capacity utilization rate in each iteration process to achieve capacity balancing, and meanwhile the distributed consistency rule is not affected.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a capacity balancing method, device, program product and medium for a distributed storage system. Background Art

[0002] With the development of the Internet and the popularization of smart devices, traditional storage systems can no longer meet the needs of large-scale and diversified data storage. The rise of big data and cloud computing technologies has provided more application scenarios for distributed storage systems and promoted the development of distributed storage system technology. The design concept of distributed storage systems is based on scalability and fault tolerance. It adopts a distributed architecture, slices data and distributes it in the form of objects on multiple nodes, ensures the capacity of each node is balanced, and accesses and manages data through network connections, while providing a unified storage interface.

[0003] At present, distributed storage systems use mapping update algorithms to achieve capacity balancing. By monitoring the load and data distribution of storage nodes in the storage cluster, a balancing strategy is formulated in advance to determine which data needs to be migrated to achieve capacity balancing between storage nodes. However, the mapping update algorithm needs to frequently update the data mapping information, which makes the data migration cost high; and during the data migration process, data consistency problems may arise, especially when multiple storage nodes are migrating data at the same time, data consistency is difficult to ensure, and the mapping update algorithm needs to monitor and adjust the load and data distribution of the storage nodes, which will increase the management complexity of the system and its frequent data migration may introduce performance jitters in the storage system, resulting in unstable system response speed.

[0004] Therefore, how to provide a capacity balancing method for a distributed storage system to solve the problems of high data migration cost, difficulty in ensuring data consistency, high management complexity and possible introduction of performance jitter in the capacity balancing process and achieve balanced utilization of storage space is an urgent problem that needs to be solved today. Summary of the invention

[0005] The purpose of the present invention is to provide a capacity balancing method, device, computer program product and computer-readable storage medium for a distributed storage system, so as to balance the capacity utilization rate of each storage device through a capacity balancing iterative process and achieve balanced utilization of storage space.

[0006] In order to solve the above technical problems, the present invention provides a capacity balancing method for a distributed storage system, comprising: When the current number of iterations does not exceed the target number of iterations, the storage devices to be migrated are determined according to the capacity utilization rate of each storage device; wherein the capacity utilization rate is the quotient of the used space of each storage device and the device capacity, and the number of the storage devices to be migrated is 1; Determine a shard group to be migrated in the storage device to be migrated and a destination storage device corresponding to the shard group to be migrated; wherein the capacity utilization rate of the destination storage device is lower than the capacity utilization rate of the storage device to be migrated; Migrate the shard group to be migrated to the destination storage device, update the current iteration number and wait for the next iteration.

[0007] On the other hand, determining the storage device to be migrated according to the capacity usage rate of each storage device includes: The current storage device detects whether it is the storage device with the largest capacity usage rate among all the storage devices; wherein the current storage device is any of the storage devices; If so, the current storage device is determined to be the storage device to be migrated, and the step of determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated is performed.

[0008] On the other hand, before determining the storage device to be migrated according to the capacity usage rate of each storage device, the method further includes: The current storage device determines the target number of iterations according to the number of shards in the storage pool in the distributed storage system, its own device capacity and the total device capacity; wherein the own device capacity is the capacity of the shards in the storage pool stored in the current storage device, and the total device capacity is the capacity of the shards in the storage pool stored in all the storage devices; Determine whether the current number of iterations exceeds the target number of iterations; If not, the step of determining the storage device to be migrated according to the capacity usage rate of each storage device is executed.

[0009] On the other hand, the current storage device determines the target number of iterations according to the number of shards of the storage pool in the distributed storage system, the capacity of the current storage device and the capacity of all devices, including: The current storage device is pool_shard_count × (osd_size / sum(all_osd_sizes)) , calculate the target number of iterations; where, pool_shard_count Indicates the number of shards in the storage pool. osd_size Indicates the capacity of the device itself, all_osd_sizes Indicates the total device capacity.

[0010] On the other hand, the current storage device determines the target number of iterations according to the number of shards of the storage pool in the distributed storage system, the capacity of the current storage device and the capacity of all devices, including: The current storage device determines the original number of iterations based on the number of shards in the storage pool in the distributed storage system, its own device capacity, and the total device capacity; Determine the original number of iterations as the number of iterations to be adjusted, and determine the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted; The number of iterations to be adjusted is adjusted according to the operation complexity to obtain the target number of iterations.

[0011] On the other hand, determining the original number of iterations as the number of iterations to be adjusted, and determining the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted, includes: pass o ( k × OSDs × PGs × logn(PGs) ), calculate the operation complexity; wherein, k represents the number of iterations to be adjusted, OSDs represents a storage device of each of the distributed storage nodes, PGs Represents a shard group of each of the storage devices.

[0012] On the other hand, adjusting the number of iterations to be adjusted according to the operation complexity to obtain the target number of iterations includes: Determining whether the operation complexity exceeds a preset operation complexity threshold; If it exceeds the preset operation complexity threshold, the number of iterations to be adjusted is updated by the number of iterations to be adjusted - 1, and the step of determining the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device and the number of iterations to be adjusted is performed; If the preset running complexity threshold is not exceeded, the number of iterations to be adjusted is determined as the target number of iterations.

[0013] On the other hand, after migrating the shard group to be migrated to the destination storage device, the method further includes: The capacity utilization rate of each storage device is calculated and updated, and all the storage devices are sorted according to the size of the capacity utilization rate to obtain a storage device sorting queue.

[0014] On the other hand, the number of the shard groups to be migrated and the number of the destination storage devices are both 1.

[0015] On the other hand, the determining of the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated includes: According to the controlled replication rule under the scalable hash of the storage pool, the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated are determined.

[0016] On the other hand, the determining, according to the controlled replication rule under the scalable hash of the storage pool, the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated includes: Determine the shard group with the largest capacity in the storage device to be migrated as the shard group to be migrated; The destination storage device is determined according to the capacity utilization rate of each storage device and the controlled replication rule under the scalable hash of the storage pool where the shard group to be migrated is located; wherein the destination storage device is the storage device with the smallest capacity utilization rate among the storage devices corresponding to the storage pool where the shard group to be migrated is located.

[0017] On the other hand, before determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool, the method further includes: Determine whether the number of shards in the storage device to be migrated is greater than a first shard number threshold; If it is greater than the first shard number threshold, executing the step of determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool; Correspondingly, after determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool, the method further includes: Determine whether the number of shards in the destination storage device is less than a second shard number threshold; If it is less than the second shard number threshold, the steps of migrating the shard group to be migrated to the destination storage device, updating the current iteration number and waiting for the next iteration are performed.

[0018] The present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the capacity balancing method of the distributed storage system described above when executed by a processor.

[0019] The present invention also provides a capacity balancing device for a distributed storage system, comprising: Memory for storing computer programs; A processor is used to implement the steps of the capacity balancing method of the distributed storage system as described above when executing the computer program.

[0020] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the capacity balancing method of the distributed storage system as described above are implemented.

[0021] A capacity balancing method for a distributed storage system provided by the present invention comprises: when the current number of iterations does not exceed the target number of iterations, determining a storage device to be migrated according to the capacity utilization rate of each storage device; wherein the capacity utilization rate is the quotient of the used space of each storage device and the device capacity, and the number of storage devices to be migrated is 1; determining a shard group to be migrated in the storage device to be migrated and a destination storage device corresponding to the shard group to be migrated; wherein the capacity utilization rate of the destination storage device is lower than the capacity utilization rate of the storage device to be migrated; migrating the shard group to be migrated to the destination storage device, updating the current number of iterations and waiting for the next iteration.

[0022] It can be seen that the present invention can perform capacity balancing in an iterative manner by determining the storage device to be migrated according to the capacity utilization rate of each storage device when the current number of iterations does not exceed the target number of iterations, so that in each iteration, the object shards can be reallocated from the storage device with high capacity utilization rate to the storage device with low capacity utilization rate to achieve capacity balancing, while not affecting the distributed consistency rules, achieving balanced utilization of storage space, and solving the problems of high data migration cost, difficulty in ensuring data consistency, high management complexity, and possible introduction of performance jitter in the existing capacity balancing process. In addition, the present invention also provides a capacity balancing device, a computer program product, and a computer-readable storage medium for a distributed storage system, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0024] Figure 1 A flow chart of a capacity balancing method for a distributed storage system provided by an embodiment of the present invention; Figure 2A schematic diagram of the overall architecture of a distributed storage system provided by an embodiment of the present invention; Figure 3 A flow chart of another capacity balancing method for a distributed storage system provided by an embodiment of the present invention; Figure 4 A schematic flow chart of another capacity balancing method for a distributed storage system provided by an embodiment of the present invention; Figure 5 is a schematic diagram of capacity balancing logic in the related art; Figure 6 A schematic diagram of the capacity balancing logic of the capacity balancing method provided in an embodiment of the present invention; Figure 7 A simplified structural diagram of a capacity balancing device for a distributed storage system provided by an embodiment of the present invention; Figure 8 A schematic diagram of the specific structure of a capacity balancing device for a distributed storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] As data storage becomes larger, more diverse, and more expensive, the capacity balancing algorithm for distributed storage systems in related technologies cannot achieve optimal storage capacity utilization due to fault tolerance, the heterogeneity of storage devices, and specific constraints of data centers.

[0027] For example, in the related art, the distributed storage system adopts a mapping update algorithm to achieve capacity balance. By monitoring the load conditions and data distribution conditions of the storage nodes in the storage cluster, a balancing strategy is formulated in advance to determine which data needs to be migrated to achieve capacity balance between storage nodes. The data mapping information can be adjusted according to the load conditions and data distribution conditions of the storage nodes to achieve capacity balance between storage nodes. However, this mapping update algorithm has the following disadvantages: 1. High data migration cost: The mapping update algorithm needs to frequently update the data mapping information and perform data migration, which will result in a high cost of data migration, especially in large-scale storage systems; 2. Impact on the load of storage nodes: The data migration of the mapping update algorithm may affect the load of the storage node, especially under high load conditions, data migration may cause the performance of the storage node to degrade; 3. Data consistency is difficult to ensure: During the data migration process, data consistency problems may arise, especially when multiple storage nodes are migrating data at the same time, data consistency is difficult to ensure; 4. High management complexity: The mapping update algorithm needs to monitor and adjust the load and data distribution of the storage nodes, which will increase the management complexity of the system, especially in large-scale storage systems; 5. Performance jitter may be introduced: Frequent data migration may introduce performance jitter of the storage system, resulting in unstable system response speed.

[0028] Moreover, in a distributed storage system cluster, data writing and deleting operations may cause uneven data distribution among storage nodes. If data migration is not timely, storage space will be wasted.

[0029] Therefore, in order to solve the problems of uneven data distribution, unbalanced storage node capacity, untimely data migration, large amount of migrated data, high data migration cost in the capacity balancing process, impact on the load of storage nodes, difficulty in ensuring data consistency, high management complexity and possible introduction of performance jitter in distributed storage systems, the present invention proposes a capacity balancing iterative process, which reallocates object shards from storage devices with high capacity utilization to storage devices with low capacity utilization to achieve capacity balancing, and each iteration only migrates data for a shard group in one storage device to prevent excessive migration of data while not affecting the distributed consistency rules.

[0030] For details, please refer to Figure 1 , Figure 1 A flow chart of a capacity balancing method for a distributed storage system provided by an embodiment of the present invention. The method may include: Step 101: When the current number of iterations does not exceed the target number of iterations, determine the storage devices to be migrated based on the capacity utilization rate of each storage device; wherein the capacity utilization rate is the quotient of the used space of each storage device and the device capacity, and the number of storage devices to be migrated is 1.

[0031] It is understandable that the capacity balancing method of the distributed storage system provided in this embodiment can be applied to a distributed storage system (i.e., a distributed storage system cluster). Figure 2 The overall system architecture shown is based on the concepts of scalability and fault tolerance, and adopts a distributed architecture. The data is sliced ​​and stored in the form of objects in storage devices on multiple storage nodes (i.e., distributed storage nodes, such as nodes 1-3). The technology of data access and management is performed through network connections, and a unified storage interface is provided.

[0032] Accordingly, Figure 2 The distributed storage system shown may include multiple components, such as object gateway service, block device service, file system service, unified, self-controlled, scalable distributed storage consistency management system, storage pool, controllable, scalable, distributed data balanced placement algorithm, metadata and monitoring service cluster. In the storage pool, the client's data is stored by slicing, which is fixed to objects of a preset size (such as 4MB) by default. The objects are grouped, and the group id is the value of the modulo of Hash (object x) and the number of groups. Finally, the shard group is stored in the underlying storage device.

[0033] Among them, the above-mentioned object gateway service, block device service and file system service can be used to provide a unified storage interface to provide access interface services for distributed storage clients. The unified, self-controlled and scalable distributed storage consistency management system is the core component of the distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects, and each object has a unique identifier and related data. It is responsible for distributing objects on various storage nodes of the storage cluster and providing data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access. The distributed storage system consists of multiple storage nodes, each of which can contain multiple storage devices (such as hard disks and disks); the storage pool divides storage resources to form different pools to meet different storage requirements. The controllable, scalable and distributed data balancing placement algorithm component can evenly distribute data objects on various nodes of the system to avoid data hot spots and improve system performance. The metadata cluster is responsible for managing the metadata information of the file system, including the attributes and locations of files and directories; the monitoring service cluster is responsible for monitoring the status and configuration information of the cluster. Its technical principles include maintaining the status information, configuration information, health status, etc. of the cluster to ensure the consistency and availability of the cluster.

[0034] Correspondingly, in a distributed storage system, capacity balancing is an important component, which ensures that the data in the storage cluster is evenly distributed on each storage node to improve the performance and reliability of the system. Considering the distributed redundant architecture and storage pool strategy, in order to exceed the capacity balancing of distributed storage and reduce the impact of data migration in the balancing process on performance, the storage data cannot be placed on the storage device of any storage node. Uneven capacity distribution will lead to capacity deviation, similar to the principle of a wooden barrel. The capacity of a wooden barrel depends on the shortest piece of wood. In a distributed storage system, if the storage capacity of some nodes is overloaded or idle, the capacity utilization of the entire system will be limited, and the overall storage resources of the system cannot be fully utilized; in addition, due to the uneven storage capacity, data distribution may be uneven, increasing the delay of data access and reducing the overall performance of the system; therefore, unlike the related art, in order to avoid untimely balancing strategies, reduce migration costs, and reduce the impact on cluster performance, the capacity balancing method provided in this embodiment can be performed in real time in a dynamic process in an iterative manner, without being restricted by different strategies and business models.

[0035] It should be noted that the capacity balancing method provided in this embodiment can be applied to storage devices in a distributed storage system, that is, storage devices in each distributed storage node, such as storage devices can use their own OSD (Objectstorage daemon, object storage daemon) to complete the capacity balancing method. The capacity balancing method provided in this embodiment can be applied to other devices in a distributed storage system, such as distributed storage nodes or monitoring nodes, and this embodiment does not impose any restrictions on this.

[0036] Among them, the target number of iterations in this embodiment can be the number of iterations required for capacity balancing, that is, the number of capacity balancing iterations required each time capacity balancing is triggered; in this embodiment, by setting the target number of iterations, the optimal equilibrium state value to which capacity balancing converges can be quantified. The specific content of the target number of iterations in this step can be set by the designer according to practical scenarios and user needs, such as the target number of iterations can be a pre-set fixed value; in order to ensure the balancing effect of each capacity balancing, the target number of iterations can also be the number of iterations determined according to the number of shards in the storage pool in the distributed storage system, the capacity of the shards in the storage pool stored in all storage devices, and the capacity of the shards in the storage pool stored in each storage device. For example, when the method provided in this embodiment is applied to a storage device, each storage device can determine its corresponding target number of iterations according to the number of shards in the storage pool, the capacity of the shards in the storage pool stored in itself, and the capacity of the shards in the storage pool stored in all storage devices. When the method provided in this embodiment is applied to a distributed storage node, the current distributed storage node can use the maximum target iteration number among the target iteration numbers corresponding to the internal storage devices as the target iteration number corresponding to the current distributed storage node; the current distributed storage node is any distributed storage node in the distributed storage system. This embodiment does not impose any restrictions on this.

[0037] Correspondingly, this step may also include a process of obtaining and judging the target number of iterations. For example, when the method provided in this embodiment is applied to a storage device, the current storage device determines the target number of iterations based on the number of shards in the storage pool in the distributed storage system, its own device capacity, and the total device capacity; wherein the own device capacity is the capacity of the shards in the storage pool stored in the current storage device, and the total device capacity is the capacity of the shards in the storage pool stored in all storage devices; determine whether the current number of iterations exceeds the target number of iterations; if not, proceed to step 101; if so, end this process and wait for the next triggering of capacity balancing.

[0038] Furthermore, in order to improve the accuracy of the target number of iterations, the method provided in this embodiment can also use the operation complexity (i.e., runtime complexity) to adjust the target number of iterations to improve the balancing effect of each capacity balancing. For example, each storage device can determine its corresponding number of iterations to be adjusted based on the number of shards in the storage pool, the capacity of the shards in the storage pool stored by itself, and the capacity of the shards in the storage pool stored in all storage devices; determine the original number of iterations as the number of iterations to be adjusted, and determine the corresponding operation complexity of each storage device based on the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted; adjust the number of iterations to be adjusted based on the operation complexity to obtain the corresponding target number of iterations for each storage device.

[0039] It is understandable that the storage device to be migrated in this step can be the storage device for data migration required for this iteration. In this embodiment, the capacity utilization rate of each storage device in the distributed storage system (the quotient of the used space used_space_size and the device capacity device_size) is used to determine the storage device to be migrated, so that each iteration only migrates the data (i.e., the shard group to be migrated) in one storage device (i.e., the storage device to be migrated), preventing excessive migration of data and not affecting the distributed consistency rules.

[0040] Correspondingly, the specific method of determining the storage device to be migrated according to the capacity utilization rate of each storage device in this step can be set by the designer, such as directly using the storage device with the largest capacity utilization rate among all storage devices as the storage device to be migrated; for example, when the method provided in this embodiment is applied to storage devices, the current storage device can detect whether it is the storage device with the largest capacity utilization rate among all storage devices; wherein the current storage device is any storage device; if so, it is determined that the current storage device is the storage device to be migrated, and step 102 can be continued; if not, it can be determined that the current storage device is not the storage device to be migrated, and this process can be terminated to wait for the next iteration of this capacity balancing. In some embodiments, the optional storage device with the largest capacity utilization rate among all storage devices can also be used as the storage device to be migrated; wherein the number of storage devices corresponding to the storage pool corresponding to the optional storage device is greater than 1; that is, the storage device to be migrated in each iteration can be the storage device with the largest capacity utilization rate among the multiple storage devices corresponding to the storage pool, so as to reduce the situation where data migration cannot be performed during the iteration process. This embodiment does not impose any restrictions on this.

[0041] Step 102: Determine the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated; wherein the capacity usage rate of the destination storage device is lower than the capacity usage rate of the storage device to be migrated.

[0042] It is understandable that the shard group to be migrated in this step may be the shard group to be migrated in this iteration in the storage device to be migrated. The destination storage device in this embodiment may be the storage device to which the shard group to be migrated is to be migrated.

[0043] Correspondingly, the specific number of shard groups to be migrated and the destination storage device is not limited in this embodiment. For example, the number of shard groups to be migrated and the destination storage device can both be 1 to prevent excessive migration of data. The number of shard groups to be migrated can also be greater than 1, such as the two shard groups with the largest and second largest capacity (or number of shards) in the storage device to be migrated; correspondingly, the destination storage device can be 1 or greater than 1 to improve iteration efficiency. This embodiment does not impose any restrictions on this.

[0044] Correspondingly, the specific method for determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated in this step can be set by the designer according to the practical scenario and user needs. For example, the shard group with the largest capacity in the storage device to be migrated can be directly determined as the shard group to be migrated, and the storage device with the smallest capacity utilization rate can be determined as the destination storage device. In order to ensure that each data migration does not affect the CRUSH (Controlled Replication Under Scalable Hashing) rule of the storage pool, the CRUSH rule includes the type (such as disk and solid state drive, etc.) and location (such as rack and node, etc.) of the storage device, that is, each storage pool can have different replication strategies and data distribution rules, including the number of copies of the data and the location of the replication. In this step, the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated can be determined according to the CRUSH rule of the storage pool, so that the data migration does not affect the distributed consistency rule.

[0045] Correspondingly, the specific method for determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the CRUSH rule of the storage pool can be set by the designer. For example, the shard group with the largest capacity in the storage device to be migrated can be directly determined as the shard group to be migrated; the destination storage device is determined according to the capacity utilization rate of each storage device and the CRUSH rule of the storage pool where the shard group to be migrated is located; wherein the destination storage device is the storage device with the smallest capacity utilization rate among the storage devices corresponding to the storage pool where the shard group to be migrated is located. Correspondingly, the optional shard group with the largest capacity in the storage device to be migrated can also be used as the shard group to be migrated; wherein the number of storage devices corresponding to the storage pool corresponding to the optional shard group is greater than 1. This embodiment does not impose any restrictions on this.

[0046] Step 103: Migrate the shard group to be migrated to the destination storage device, update the current iteration number and wait for the next iteration.

[0047] It can be understood that in this step, the shard group to be migrated in the storage device to be migrated can be migrated to the destination storage device to complete the data migration of this iteration, and the current iteration number can be updated (such as the iteration number + 1), so as to wait for the next iteration to compare the updated current iteration number with the target iteration number, and continue to step 101 when the updated current iteration number does not exceed the target iteration number, so as to continue the data migration of the next iteration, thereby increasing the group shard count to the optimal state through multiple iterations, and eliminating the capacity utilization difference of the storage devices in the entire cluster. For example, a migration instruction can be triggered in this step, and the shard group to be migrated can be migrated from the storage device to be migrated to the destination storage device through the migration instruction.

[0048] Furthermore, in this embodiment, after migrating the shard group to be migrated to the destination storage device, the capacity utilization rate of each storage device can be calculated and updated, and all storage devices can be sorted according to the capacity utilization rate to obtain a storage device sorting queue to facilitate the determination of the storage device to be migrated in the next iteration.

[0049] In this embodiment, the embodiment of the present invention determines the storage device to be migrated according to the capacity utilization rate of each storage device when the current number of iterations does not exceed the target number of iterations, and can perform capacity balancing in an iterative manner, so that in each iteration process, the object shards can be reallocated from the storage device with high capacity utilization rate to the storage device with low capacity utilization rate to achieve capacity balancing, while not affecting the distributed consistency rules, achieving balanced utilization of storage space, and solving the problems of high data migration cost, difficulty in ensuring data consistency, high management complexity and possible introduction of performance jitter in the existing capacity balancing process.

[0050] Based on the above embodiments, in order to further improve the effect of capacity balancing, the embodiment of the present invention also provides another capacity balancing method for a distributed storage system, please refer to Figure 3 , the method comprising: Step 201: The current storage device determines the original number of iterations according to the number of shards of the storage pool in the distributed storage system, its own device capacity and the total device capacity.

[0051] The current storage device is any storage device, the own device capacity is the capacity of the shards in the storage pool stored in the current storage device, and the total device capacity is the capacity of the shards in the storage pool stored in all storage devices.

[0052] It is understandable that the capacity balancing method provided in this embodiment can be applied to storage devices in each distributed storage node in a distributed storage system. For example, the storage devices can use their own OSD (Object storage daemon) to complete the capacity balancing method.

[0053] Among them, for the current storage device in this step, the specific method of determining the original number of iterations according to the number of shards of the storage pool in the distributed storage system, its own device capacity and the total device capacity can be set by the designer. For example, the current storage device can be pool_shard_count × (osd_size / sum(all_osd_sizes)) , calculate the target number of iterations; where, pool_shard_count It can represent the number of shards in a storage pool, such as the total number of shards in all storage pools in a distributed storage system; osd_size Indicates the capacity of the device itself. all_osd_sizes Indicates the total device capacity. For example, the current storage device utilization pool_shard_count × (osd_size / sum(all_osd_sizes)) , get the corresponding calculation result, and use the integer part of the calculation result as the original number of iterations, or use the integer part of the calculation result + 1 as the original number of iterations.

[0054] Step 202: Determine the original number of iterations as the number of iterations to be adjusted, and determine the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted.

[0055] It is understandable that in this embodiment, the calculated operation complexity can be used to balance the balancing effect of each capacity balancing with the resources and time required for operation, which is beneficial to maintenance management. o ( k × OSDs × PGs × logn(PGs) ), calculate the running complexity; where, k It can represent the number of iterations to be adjusted. OSDs It can represent the storage devices of each distributed storage node (such as Figure 2 Storage nodes 1…n in PGs It can represent the shard groups of each storage device (such as Figure 2 groups 1…n in o (·) can represent the time complexity, and n can represent the number of shard groups of each storage device. The lower the operation complexity, the less time and memory space are required for each capacity balancing execution.

[0056] Step 203: According to the operation complexity, the number of iterations to be adjusted is adjusted to obtain a target number of iterations.

[0057] Among them, in this step, the current storage device can adjust the number of iterations to be adjusted according to the operation complexity to obtain the target number of iterations, so as to balance the balancing effect of each capacity balancing and the resources and time required for operation, which is conducive to maintenance and management. The specific method of adjusting the number of iterations to be adjusted according to the operation complexity in this step to obtain the target number of iterations can be set by the designer, such as the current storage device can determine whether the operation complexity exceeds the preset operation complexity threshold; if it exceeds the preset operation complexity threshold, the number of iterations to be adjusted is updated by the number of iterations to be adjusted-1, and enter step 202 to perform the step of determining the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device and the number of iterations to be adjusted; if it does not exceed the preset operation complexity threshold, the number of iterations to be adjusted is determined as the target number of iterations.

[0058] Step 204: Determine the current iteration number.

[0059] It is understandable that in this step, the current number of iterations in this capacity balancing may be determined. For example, in the first iteration, the current number of iterations may be determined to be 1.

[0060] Step 205: Determine whether the current number of iterations exceeds the target number of iterations; if not, proceed to step 206.

[0061] Among them, in this step, the current storage device can determine whether the current storage device has completed this capacity balancing by judging whether the current iteration number exceeds the target iteration number, so as to determine whether the current iteration number exceeds the target iteration number (such as Figure 4 When the maximum value in is reached, the process can end directly and wait for the next capacity balancing to be triggered.

[0062] Correspondingly, in this embodiment, the current storage device can determine whether the current number of iterations exceeds the target number of iterations at a preset time interval; it can also determine whether the current number of iterations exceeds the target number of iterations after the current storage device or other storage devices complete the data migration of this iteration. This embodiment does not impose any restrictions on this.

[0063] Step 206 : Determine whether it is the storage device with the highest capacity usage rate among all storage devices; if so, proceed to step 207 .

[0064] It can be understood that the current storage device determines whether it is the storage device with the largest capacity usage rate among all storage devices (ie, the storage device to be migrated, such as Figure 4 The source end in the example (i) determines whether data migration is required for this iteration itself, so that when the storage device is not the storage device with the largest capacity utilization rate among all storage devices, this iteration can be ended and the next iteration can be waited for.

[0065] Step 207: Determine the shard group with the largest capacity in the current storage device as the shard group to be migrated.

[0066] Among them, this embodiment is demonstrated by directly selecting the shard group with the largest capacity from the storage device to be migrated as the shard group to be migrated in each iteration. Other methods can also be used to determine the shard group to be migrated, such as selecting any shard group with a capacity greater than a capacity threshold as the shard group to be migrated, or using the optional shard group with the largest capacity in the current storage device as the shard group to be migrated; wherein the number of storage devices corresponding to the storage pool corresponding to the optional shard group is greater than 1. This embodiment does not impose any restrictions on this.

[0067] Step 208: Determine the destination storage device according to the capacity usage of each storage device and the controlled replication rule under the scalable hash of the storage pool where the shard group to be migrated is located.

[0068] The destination storage device is the storage device with the smallest capacity usage among the storage devices corresponding to the storage pool where the shard group to be migrated is located.

[0069] It can be understood that in this step, the current storage device can select the storage device with the smallest capacity utilization rate as the destination storage device from the storage devices that meet the CRUSH rule and to which the shard group to be migrated can be migrated based on the capacity utilization rate of each storage device and the CRUSH rule of the storage pool where the shard group to be migrated is located, so that the migration of the shard group to be migrated can meet the migration rule.

[0070] Accordingly, the migration rule may include not only the rule that data migration does not affect the CRUSH rule of the storage pool, but also the rule that optimizes the number of shards at the source and destination ends. For example, in some embodiments, before step 207, it may be determined whether the number of shards in the storage device to be migrated is greater than a first shard number threshold (e.g., Figure 4 If the number of shards on the source storage device is too large, the process proceeds to step 207; if the number of shards is not greater than the first shard number threshold, the current iteration can be terminated or the current capacity balancing can be terminated directly; accordingly, after step 208, it can be determined whether the number of shards on the destination storage device is less than the second shard number threshold (such as Figure 4 If the number of shards is less than the second shard number threshold, proceed to step 209; if the number of shards is not less than the second shard number threshold, reselect the destination storage device, such as selecting the storage device with the second smallest capacity utilization rate among the storage devices corresponding to the storage pool where the shard group to be migrated is located, or end this iteration.

[0071] Correspondingly, the migration rules may also include rules for reducing the differences in capacity utilization of all storage devices; for example, in some embodiments, after step 208, it can be determined whether the difference in capacity utilization between the storage device to be migrated and the destination storage device is greater than a difference threshold; if so, proceed to step 209; if not, end this iteration.

[0072] Step 209: Migrate the shard group to be migrated to the destination storage device.

[0073] It is understandable that in this step, the current storage device can migrate its own shard group to be migrated to the destination storage device to complete the data migration of this iteration, thereby increasing the group shard count to the optimal state through multiple iterations, while eliminating the capacity utilization differences of storage devices in the entire cluster, thereby optimizing the available storage space in all storage pools and the data distribution between available storage devices, and optimizing the read and write performance. For example, in this step, the current storage device can trigger a migration instruction, migrate the shard group to be migrated from the current storage device to the destination storage device through the migration instruction, and record the migration operation of the shard group to be migrated.

[0074] For example, if Figure 5 and Figure 6 As shown, in the related art, the object PG (Placement Group) group (i.e., shard group) of the storage pool is stored on the storage device through the shard consistency algorithm, without considering the capacity utilization of the storage device. When a storage device is full, it will cause space waste of other storage devices according to the above-mentioned barrel effect description; while in the embodiment of the present invention, according to the capacity utilization of each storage device, dynamic shard migration can be performed in each iteration, and the capacity of the storage device can be dynamically monitored to always maintain the optimal balance state.

[0075] Correspondingly, after this step, step 201 may be entered to calculate the updated target iteration number, that is, each iteration may use the latest target iteration number; that is, each iteration of capacity balancing may start from step 201 to update the target iteration number in real time. After this step, step 204 may also be entered to update the iteration number of the next iteration, that is, the target iteration number may use the iteration number calculated in the first iteration; that is, the second and subsequent iterations of capacity balancing may start from step 204, and this embodiment does not impose any restrictions on this.

[0076] Furthermore, in this step, after the current storage device migrates the shard group to be migrated to the destination storage device, it can also calculate and update the capacity utilization rate of each storage device, and sort all storage devices according to the capacity utilization rate to obtain a storage device sorting queue to facilitate the determination of the storage device to be migrated in the next iteration.

[0077] In this embodiment, the embodiment of the present invention adjusts the number of iterations to be adjusted according to the operation complexity to obtain the target number of iterations, which can balance the balancing effect of each capacity balancing with the resources and time required for calculation, and is beneficial to maintenance management.

[0078] Corresponding to the above method embodiment, an embodiment of the present invention further provides a capacity balancing device for a distributed storage system. The capacity balancing device for a distributed storage system described below and the capacity balancing method for a distributed storage system described above can refer to each other.

[0079] Please refer to Figure 7 , Figure 7 A simplified structural diagram of a capacity balancing device for a distributed storage system provided by an embodiment of the present invention. The capacity balancing device may include: A memory D1, for storing computer programs; The processor D2 is configured to implement the steps of the capacity balancing method for the distributed storage system provided by the above method embodiment when executing a computer program.

[0080] Accordingly, please refer to Figure 8 , Figure 8 A schematic diagram of the specific structure of a capacity balancing device of a distributed storage system provided in an embodiment of the present invention. The capacity balancing device 310 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the host. Furthermore, the central processor 322 can be configured to communicate with the storage medium 330 and execute a series of instruction operations in the storage medium 330 on the capacity balancing device 310.

[0081] The capacity balancing device 310 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Linux system.

[0082] Among them, the capacity balancing device 310 in this embodiment can be specifically a storage device in a distributed storage system, such as a storage device running an OSD; it can also be specifically a distributed storage node or a monitoring node in a distributed storage system, and this embodiment does not impose any restrictions on this.

[0083] The steps in the capacity balancing method of the distributed storage system described above can be implemented by the structure of the capacity balancing device of the distributed storage system.

[0084] Corresponding to the above method embodiment, the embodiment of the present invention further provides a computer program product. The computer program product described below and the capacity balancing method of a distributed storage system described above can refer to each other.

[0085] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the capacity balancing method for a distributed storage system provided in the above method embodiment.

[0086] Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium described below and the capacity balancing method of a distributed storage system described above can refer to each other.

[0087] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the capacity balancing method of a distributed storage system in the above method embodiment.

[0088] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0089] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device, computer program product, and computer-readable storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0090] The capacity balancing method, device, computer program product and computer-readable storage medium of a distributed storage system provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A capacity balancing method for a distributed storage system, characterized in that: include: When the current number of iterations does not exceed the target number of iterations, the storage devices to be migrated are determined according to the capacity utilization rate of each storage device; wherein the capacity utilization rate is the quotient of the used space of each storage device and the device capacity, and the number of the storage devices to be migrated is 1; Determine a shard group to be migrated in the storage device to be migrated and a destination storage device corresponding to the shard group to be migrated; wherein the capacity utilization rate of the destination storage device is lower than the capacity utilization rate of the storage device to be migrated; Migrate the shard group to be migrated to the destination storage device, update the current iteration number and wait for the next iteration.

2. The capacity balancing method of a distributed storage system according to claim 1, characterized in that: The step of determining the storage device to be migrated according to the capacity usage rate of each storage device includes: The current storage device detects whether it is the storage device with the largest capacity usage rate among all the storage devices; wherein the current storage device is any of the storage devices; If so, the current storage device is determined to be the storage device to be migrated, and the step of determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated is performed.

3. The capacity balancing method of a distributed storage system according to claim 1, characterized in that: Before determining the storage device to be migrated according to the capacity usage rate of each storage device, the method further includes: The current storage device determines the target number of iterations according to the number of shards in the storage pool in the distributed storage system, its own device capacity and the total device capacity; wherein the own device capacity is the capacity of the shards in the storage pool stored in the current storage device, and the total device capacity is the capacity of the shards in the storage pool stored in all the storage devices; Determine whether the current number of iterations exceeds the target number of iterations; If not, the step of determining the storage device to be migrated according to the capacity usage rate of each storage device is executed.

4. The capacity balancing method of a distributed storage system according to claim 3, characterized in that: The current storage device determines the target number of iterations according to the number of shards of the storage pool in the distributed storage system, the capacity of the current storage device and the capacity of all devices, including: The current storage device is pool_shard_count × (osd_size / sum(all_osd_sizes)) , calculate the target number of iterations; where, pool_shard_count Indicates the number of shards in the storage pool. osd_size Indicates the capacity of the device itself, all_osd_sizes Indicates the total device capacity.

5. The capacity balancing method of a distributed storage system according to claim 3, characterized in that: The current storage device determines the target number of iterations according to the number of shards of the storage pool in the distributed storage system, the capacity of the current storage device and the capacity of all devices, including: The current storage device determines the original number of iterations based on the number of shards in the storage pool in the distributed storage system, its own device capacity, and the total device capacity; Determine the original number of iterations as the number of iterations to be adjusted, and determine the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted; The number of iterations to be adjusted is adjusted according to the operation complexity to obtain the target number of iterations.

6. The capacity balancing method of a distributed storage system according to claim 5, characterized in that: The step of determining the original number of iterations as the number of iterations to be adjusted, and determining the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device, and the number of iterations to be adjusted, includes: pass o ( k × OSDs × PGs × logn(PGs) ), calculate the operation complexity; wherein, k represents the number of iterations to be adjusted, OSDs represents a storage device of each of the distributed storage nodes, PGs Represents a shard group of each of the storage devices.

7. The capacity balancing method of a distributed storage system according to claim 5, characterized in that: The step of adjusting the number of iterations to be adjusted according to the operation complexity to obtain the target number of iterations includes: Determining whether the operation complexity exceeds a preset operation complexity threshold; If it exceeds the preset operation complexity threshold, the number of iterations to be adjusted is updated by the number of iterations to be adjusted - 1, and the step of determining the operation complexity according to the storage devices of each distributed storage node in the distributed storage system, the shard groups of each storage device and the number of iterations to be adjusted is performed; If the preset running complexity threshold is not exceeded, the number of iterations to be adjusted is determined as the target number of iterations.

8. The capacity balancing method of a distributed storage system according to claim 1, characterized in that: After migrating the shard group to be migrated to the destination storage device, the method further includes: The capacity utilization rate of each storage device is calculated and updated, and all the storage devices are sorted according to the size of the capacity utilization rate to obtain a storage device sorting queue.

9. The capacity balancing method of a distributed storage system according to any one of claims 1 to 8, characterized in that: The number of the shard group to be migrated and the number of the destination storage device are both 1.

10. The capacity balancing method of a distributed storage system according to claim 9, characterized in that: The determining of the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated includes: According to the controlled replication rule under the scalable hash of the storage pool, the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated are determined.

11. The capacity balancing method of a distributed storage system according to claim 10, characterized in that: The step of determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool includes: Determine the shard group with the largest capacity in the storage device to be migrated as the shard group to be migrated; The destination storage device is determined according to the capacity utilization rate of each storage device and the controlled replication rule under the scalable hash of the storage pool where the shard group to be migrated is located; wherein the destination storage device is the storage device with the smallest capacity utilization rate among the storage devices corresponding to the storage pool where the shard group to be migrated is located.

12. The capacity balancing method of a distributed storage system according to claim 10, characterized in that: Before determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool, the method further includes: Determine whether the number of shards in the storage device to be migrated is greater than a first shard number threshold; If it is greater than the first shard number threshold, executing the step of determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool; Correspondingly, after determining the shard group to be migrated in the storage device to be migrated and the destination storage device corresponding to the shard group to be migrated according to the controlled replication rule under the scalable hash of the storage pool, the method further includes: Determine whether the number of shards in the destination storage device is less than a second shard number threshold; If it is less than the second shard number threshold, the steps of migrating the shard group to be migrated to the destination storage device, updating the current iteration number and waiting for the next iteration are performed.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the capacity balancing method of a distributed storage system as described in any one of claims 1 to 12 are implemented.

14. A capacity balancing device for a distributed storage system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the capacity balancing method for a distributed storage system as described in any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the capacity balancing method of a distributed storage system according to any one of claims 1 to 12 are implemented.

Images