Storage method, electronic device, medium and computer program product

Abstract

The embodiment of the invention discloses a storage method, electronic equipment, a medium and a computer program product, and the storage method comprises the steps: determining a first weight of each storage cluster through the available storage capacity of each storage cluster in at least two storage clusters; wherein the storage cluster is used for storing data blocks; the data block is obtained by processing original storage data based on an erasure code technology; the first weight is positively correlated with the available storage capacity; processing the first weight to obtain a second weight of each storage cluster; wherein the value change range of the second weight is smaller than the value change range of the first weight; based on the second weight, a target storage cluster is selected from the at least two storage clusters, and the data blocks are stored in the target storage cluster; wherein the probability that the target storage cluster is selected is linearly and positively correlated with the second weight of the target storage cluster. Through the method provided by the embodiment of the invention, the data storage amount of the storage cluster with relatively large available storage capacity can be reduced.

Description

Technical Field

[0001] This application belongs to the field of data storage and management technology, and in particular relates to a storage method, electronic device, medium and computer program product. Background Technology

[0002] Distributed storage systems typically determine the weight of storage resources (such as disks) based on their available storage capacity, and then select storage resources with higher weights to store data, such as directly storing data to the storage resource with the largest available storage capacity, in order to achieve dynamic balancing of storage resources. This storage method can lead to excessively large weight differences between storage resources, resulting in excessively large data storage volumes associated with storage resources with higher weights. For example, in expansion scenarios, new storage resources usually have large available storage capacity and high weights, causing a large amount of data to be concentrated on the new storage resource, resulting in excessive data storage volumes on the new storage resource. Summary of the Invention

[0003] This application provides a storage method, electronic device, medium, and computer program product that can reduce the weight differences between storage clusters and achieve balanced utilization of each storage cluster.

[0004] This application provides a storage method, the method comprising: A first weight is determined for each storage cluster based on the available storage capacity of each of at least two storage clusters; wherein the storage cluster is used to store data blocks; the data blocks are obtained by processing the original stored data based on erasure coding technology; and the first weight is positively correlated with the available storage capacity. The first weight is processed to obtain a second weight for each storage cluster; wherein the range of variation of the value of the second weight is smaller than the range of variation of the value of the first weight. Based on the second weight, a target storage cluster is selected from the at least two storage clusters, and the data block is stored in the target storage cluster; wherein the probability of the target storage cluster being selected is linearly positively correlated with the second weight of the target storage cluster.

[0005] In some embodiments, before selecting a target storage cluster from the at least two storage clusters based on the second weight and storing the data block in the target storage cluster, the method further includes: determining a first storage cluster with the largest available storage capacity and a second storage cluster with the smallest available storage capacity in each storage cluster; determining a first difference between the available storage capacity of the first storage cluster and the available storage capacity of the second storage cluster; the step of selecting a target storage cluster from the at least two storage clusters based on the second weight and storing the data block in the target storage cluster includes: if the first difference is less than a first threshold, selecting a target storage cluster from the at least two storage clusters based on the second weight and storing the data block in the target storage cluster.

[0006] As can be seen, by using the method given in this embodiment, when the first difference in available storage capacity between storage clusters is small, the adoption of a random storage strategy based on the second weight helps to ensure continuous optimization of data distribution balance.

[0007] In some embodiments, the method further includes: sorting the storage clusters according to the available storage capacity of each storage cluster to obtain a storage cluster sequence; determining a second difference between the available storage capacity of a third storage cluster and the available storage capacity of a fourth storage cluster in the storage cluster sequence; wherein the positive ranking of the third storage cluster in the storage cluster sequence is the same as the negative ranking of the fourth storage cluster in the storage cluster sequence; and performing data migration on the data in the third storage cluster and the storage cluster with the smaller available storage capacity in the fourth storage cluster when the absolute value of the second difference is greater than or equal to a first threshold.

[0008] It can be seen that by identifying storage cluster pairs with large differences in available storage capacity through the sorting mechanism (such as the third and fourth storage clusters), and triggering data migration operations when the second difference is greater than or equal to the first threshold, the problem of uneven available storage capacity in local storage clusters can be effectively identified and corrected. This avoids the performance degradation of storage clusters due to long-term imbalance in available storage capacity, thereby improving the stability and reliability of each storage cluster.

[0009] In some embodiments, the data migration between the third storage cluster and the fourth storage cluster with the smaller available storage capacity includes: migrating the data in the fourth storage cluster to the third storage cluster when the available storage capacity of the third storage cluster is greater than the available storage capacity of the fourth storage cluster.

[0010] It can be seen that selecting storage clusters with higher available storage capacity as data migration targets helps to quickly balance the available storage capacity among storage clusters. This data migration strategy can quickly alleviate storage imbalances while avoiding the processing burden of large-scale data migration, thus improving the controllability of the storage balancing process.

[0011] In some embodiments, processing the first weight to obtain a second weight for each storage cluster includes: normalizing the first weight to obtain a processed first weight; and determining a second weight for each storage cluster based on the processed first weight.

[0012] As can be seen, normalizing the first weight effectively compresses its value range, keeping the differences in the second weights across different storage clusters within a controllable range. This approach avoids the problem of excessively large differences between the first weights that might result from directly determining the first weight based on available storage capacity, thus improving the rationality and stability of data distribution and storage.

[0013] In some embodiments, determining the first weight of each storage cluster based on the available storage capacity of each of the at least two storage clusters includes: determining a target storage resource with the smallest available storage capacity in each of the at least two storage clusters; and determining the available storage capacity of the target storage resource in each storage cluster as the first weight of each storage cluster.

[0014] It can be seen that by selecting the storage resource with the smallest available storage capacity in each storage cluster to determine the first weight of the storage cluster, the actual carrying capacity of each storage resource in the storage cluster can be more accurately reflected. This avoids misjudgment caused by some storage resources with high availability masking the storage bottleneck of the storage cluster, and improves the rationality of weight allocation.

[0015] In some embodiments, the probability of the target storage cluster being selected is linearly positively correlated with the proportion of the second weight of the target storage cluster to the sum of the first weights.

[0016] It can be seen that by establishing a linear relationship between the storage probability and the proportion of the second weight in the sum of the first weight, it helps to further reduce the probability difference corresponding to each storage cluster, making the distribution of data blocks more uniform and achieving data storage balance.

[0017] This application provides an electronic device, which includes a processor and a memory for storing computer programs capable of running on the processor; wherein, The processor is used to run the computer program to perform any of the above-described storage methods.

[0018] This application provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-described storage methods.

[0019] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described storage methods.

[0020] This application provides a storage method, electronic device, medium, and computer program product. Based on the storage method provided in this embodiment, a second weight is obtained by processing a first weight, which can narrow the range of numerical variation between the second weights corresponding to each storage cluster. In the process of selecting a target storage cluster using the second weight, it helps to reduce the probability of data blocks being stored in a storage cluster with a large available storage capacity, preventing the storage cluster from being burdened with excessive data storage due to an excessively high weight. Especially in expansion scenarios, it helps to ensure that no individual storage cluster is overused during expansion, thus improving the load balancing capability of the storage system. Attached Figure Description

[0021] Figure 1 This is a flowchart of a storage method provided in an embodiment of this application; Figure 2 This is a flowchart of another storage method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a storage device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Distributed storage systems, simply put, distribute data across multiple independent devices, using multiple storage servers to share the storage load. This is a widely used cloud storage architecture. However, in distributed storage, issues such as disk failures and capacity expansion can easily lead to uneven data storage.

[0024] Currently, there are generally two solutions to address uneven storage. The first solution involves selecting storage resources based on metrics such as the file type and available storage capacity of the disk during data storage. This method typically assigns a weight to each storage resource based on its available capacity, and then determines the target storage resource for data storage. For example, it might directly select the storage resource with the largest available capacity to achieve dynamic data storage balance. The second solution involves adjusting and migrating based on available storage capacity. The adjustment targets are categorized as placement groups or disks. A placement group represents the logical unit above the disk, and it can extract data from its corresponding disk to achieve storage balance. Alternatively, when the adjustment target is a disk, the data to be migrated and the required storage capacity can be determined based on the disk's available storage capacity and the computing resources of the device hosting the disk, such as the Central Processing Unit (CPU) and memory, thus achieving data migration.

[0025] The two methods described above have the following drawbacks: Firstly, directly storing data to the storage resource with the largest available capacity based on its available storage capacity can lead to a massive data storage load on that resource, such as a disk, potentially causing disk failure. Secondly, assigning disk weights based on their available storage capacity can result in significant differences in weight between different disks. For instance, adding a new disk during expansion might have a very high weight, which could also lead to excessive data storage on disks with large available storage capacities.

[0026] Furthermore, the data migration methods described above typically carry certain risks. For instance, when migrating data while maintaining the weights of storage resources, the migration process can be quite complex when the weight differences between storage resources are small, resulting in a mismatch between the advantages and the complexity of data migration. If different weights are assigned to different storage resources, over-migration may occur if the data migration volume or storage resource weights are not updated in a timely manner.

[0027] To overcome the problems existing in related technologies, achieve balanced data storage, and ensure that each storage resource is effectively utilized, embodiments of this application provide a storage method, electronic device, medium, and computer program product. The storage method provided in this application can reduce the probability of data being stored in a storage cluster with a large available storage capacity, effectively reduce the amount of data stored on a single disk, and improve the overall performance of the storage cluster.

[0028] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0029] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other related elements in the method or apparatus that includes that element (e.g., steps in the method or units / modules in the apparatus; for example, units / modules in the apparatus may be portions of circuitry, processors, programs, or software, etc.).

[0030] The storage method provided in this application includes a series of steps, but the storage method provided in this application is not limited to the steps described. Similarly, the storage device provided in this application includes a series of modules, but the device provided in this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0031] This application provides a storage method, such as... Figure 1 As shown, Figure 1 A flowchart of a storage method is shown. Figure 1 The storage methods shown include: Step 101: Determine the first weight of each storage cluster by using the available storage capacity of each of the at least two storage clusters.

[0032] The storage cluster is used to store data blocks; the data blocks are obtained by processing the original stored data based on erasure coding technology; the first weight is positively correlated with the available storage capacity.

[0033] The storage cluster in this embodiment consists of two or more storage resources, such as disks and optical storage devices. The storage cluster can be used to store data blocks obtained through erasure coding. Specifically, the number of storage resources in the storage cluster can be pre-set based on the number of data blocks. Erasure coding is a technology used for data protection and recovery. In distributed storage, the original stored data is encoded into K data fragments and M parity fragments (K+M), each fragment corresponding to a data block, and each data block stored on a separate storage resource (such as a disk). Using erasure coding, the original stored data can be reconstructed from the remaining fragments even if some disks fail. For example, under an erasure coding strategy of K=4 and M=2, the storage system divides the original stored data into 4 data fragments and 2 parity fragments, generating a total of 6 fragments, and stores these 6 fragments on different disks.

[0034] When the number of storage resources is predetermined based on erasure coding, the storage cluster built based on this number of resources can be called a poolset. Specifically, when the erasure coding strategy is K+M, the original storage data is stored in K+M single-replica (single original storage data) storage pools, each corresponding to a disk (storage resource) on a single machine. These K+M single-replica storage pools are grouped together to form a poolset, i.e., a storage cluster. When storing data blocks, a storage cluster can be selected based on its weight.

[0035] After determining the number of storage resources in a storage cluster, a first weight for each storage cluster can be determined based on its available storage capacity. Here, the first weight of each storage cluster is positively correlated with its available storage capacity. Taking any storage cluster as an example, since a storage cluster typically includes at least two storage resources, the available storage capacity of a storage cluster can be determined based on the sum of the available storage capacities of each of its constituent storage resources.

[0036] Since the K+M data blocks obtained from erasure coding are typically stored across K+M storage resources, the available storage capacity of each storage resource in the storage cluster must be sufficient to meet the storage requirements of any one of the K+M data blocks. To ensure that each of the K+M data blocks can be stored in a storage resource, a first weight of the storage cluster can be determined based on the minimum available storage capacity of the storage resources in the storage cluster. In some embodiments, the first weight of each storage cluster is determined by the available storage capacity of each of at least two storage clusters. This includes: identifying the target storage resource with the minimum available storage capacity in each of the at least two storage clusters; and determining the available storage capacity of the target storage resource in each storage cluster as the first weight of each storage cluster.

[0037] The first weight is a parameter used to measure the relative load balancing among various storage clusters. Unlike related technologies that directly use available storage capacity to determine the probability of data storage to a storage cluster, this application embodiment can use the minimum available storage capacity of storage resources in each storage cluster as the first weight, more accurately reflecting the load bottleneck of the storage cluster. For example, if most disks in a storage cluster still have a lot of free space, but one disk is about to be full, the first weight of that storage cluster should be low to avoid continuing to allocate too much data to that storage cluster and prevent disk failure due to overload.

[0038] The target storage resource, acting as the bottleneck of the entire storage cluster, determines the cluster's maximum load capacity. Therefore, prioritizing the available storage capacity of the target storage resource allows for a more rational data distribution strategy and further improves storage stability and reliability. In practical applications, the available storage capacity of the target storage resource can also be expressed as a percentage of available storage capacity, meaning the proportion of available storage capacity within the target storage resource to its total storage capacity.

[0039] By identifying the target storage resource closest to full capacity in each storage cluster and using the available storage capacity of the target storage resource as the first weight of the storage cluster, the distribution of data storage can be controlled more precisely, effectively avoiding local resource overload.

[0040] Based on the above method, after determining the available storage capacity of each storage cluster, the first weight of each storage cluster can be determined based on a preset relationship and its available storage capacity. Alternatively, the available storage capacity of each storage cluster can be directly used as its first weight. For example, if each data block after erasure coding is stored on one disk in the storage cluster, the smallest percentage of available storage capacity among the K+M disks can be used as the first weight. Here, the percentage of available storage capacity represents the proportion of the disk's available storage capacity to its total disk storage capacity. It can be represented as ,in, It can be expressed as the smallest percentage of available storage capacity among the various storage resources in a storage cluster.

[0041] Once the first weight is determined, each piece of data (including data blocks obtained based on erasure coding) can be randomly stored in a storage cluster poolset based on the method given in formula (1). This probability is related to the first weight of each storage cluster. Based on formula (1), the probability of each storage cluster being selected for data storage is as follows: (1) in, This represents the probability that the i-th storage cluster is selected for data storage. This represents the sum of the first weights of all storage clusters. for It can be seen that, based on this method, the initial weight of each storage cluster may differ significantly. For example, in a capacity expansion scenario, the initial weight of the new storage cluster might be 100 (i.e., 100% available storage capacity), while the initial weight of other storage clusters might be 20 (i.e., 20% available storage capacity remaining), a difference of 5 times. In this case, each distinct piece of data will most likely be stored in the new storage cluster, resulting in a large amount of data being stored in the new cluster and increasing its storage pressure. To address this issue, the following steps can be performed.

[0042] Step 102: Process the first weight to obtain the second weight for each storage cluster.

[0043] The range of variation for the second weight is smaller than that for the first weight.

[0044] In this embodiment, after obtaining the first weight of each storage cluster, the first weight can be processed to narrow its data variation range, resulting in a second weight for each storage cluster. This second weight's data variation range is smaller than that of the first weight. Here, the numerical variation range can represent the span between the maximum and minimum values ​​of the weight, or the degree of weight concentration. For example, by processing the first weight of each storage cluster, the distribution of the second weight of each storage cluster can be made more concentrated than the distribution of the first weight.

[0045] For example, in the first weight corresponding to each storage cluster, the largest first weight is 100 and the smallest first weight is 10, so the range of the first weight is [10, 100]. By processing the first weight corresponding to each storage cluster, the range of the second weight of each storage cluster is made to be [20, 70] or [100, 160], which narrows the range of variation of the first weight.

[0046] In practical applications, the first weight can be linearly transformed, such as by processing the first weight through at least one of translation or scaling to obtain the second weight, or it can be based on a preset linear mapping relationship, such as mapping the range of the first weight from [min1, max1] to a new range [min2, max2] to obtain the second weight of each storage cluster.

[0047] Step 103: Based on the second weight, select the target storage cluster from at least two storage clusters and store the data block in the target storage cluster.

[0048] The probability of a target storage cluster being selected is linearly positively correlated with the second weight of the target storage cluster.

[0049] After obtaining the second weight, the probability of each storage cluster being selected can be determined by combining the method given in the above embodiments with the second weight. Based on the probability of each storage cluster being selected, data blocks are randomly stored in the target storage cluster. Here, after determining the target storage cluster, the data blocks processed based on erasure coding technology can be sequentially stored in each storage resource in the target storage cluster.

[0050] In this step, the probability of each storage cluster can be determined directly based on the second weight, for example, using the method in formula (1) to take the second weight as... The sum of the second weights of all storage clusters is obtained. This leads to the probability that the i-th storage cluster is selected for data (data block) storage. .

[0051] In some embodiments, when the range of the second weight is within the range of the first weight, such as when the maximum value of the second weight is less than the maximum value of the first weight and the minimum value of the second weight is greater than the minimum value of the first weight, the probability of the target storage cluster being selected is linearly positively correlated with the proportion of the second weight of the target storage cluster to the sum of the first weights. That is, based on the method given in formula (1), the second weight is used as... The sum of the first weights of all storage clusters is obtained. This leads to the probability that the i-th storage cluster is selected as the target storage cluster for data block storage. .

[0052] Since the second weight is linearly positively correlated with the probability of storing data blocks in the storage cluster, this means that when the weight of the target storage cluster is higher, the probability of the target storage cluster being selected is also higher, but there will be no exclusive phenomenon in extreme cases.

[0053] It can be seen that when the numerical range of the second weight falls within the numerical range of the first weight, the distribution of the second weight is more concentrated than that of the first weight, based on the sum of the first weights of all storage clusters. It is a relatively large value, therefore it is obtained by summing the second weight and the first weight of all storage clusters. Definite This will be more concentrated, meaning that the probability of each storage cluster being selected as the target storage cluster is more concentrated. This will help to further reduce the amount of data stored in storage clusters with large available storage capacity.

[0054] This application provides a storage method that processes the first weight of each storage cluster and narrows the range of the first weight to obtain a second weight. The probability of a target storage cluster being selected is determined based on the second weight. This helps to reduce the difference in the probability of each storage cluster being selected as the target storage cluster. The second weight can effectively control the load difference between storage clusters, which is beneficial to maintaining high storage performance and stability in scenarios such as expansion and disk failure.

[0055] In practical applications, steps 101 to 103 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor.

[0056] Based on the storage method given in the above embodiments, specifically, in some embodiments, before selecting a target storage cluster from at least two storage clusters based on a second weight and storing the data block in the target storage cluster, the method further includes: determining a first storage cluster with the largest available storage capacity and a second storage cluster with the smallest available storage capacity in each storage cluster; determining a first difference between the available storage capacity of the first storage cluster and the available storage capacity of the second storage cluster; the selection of a target storage cluster from at least two storage clusters and storage of the data block in the target storage cluster includes: if the first difference is less than a first threshold, selecting a target storage cluster from at least two storage clusters based on a second weight and storing the data block in the target storage cluster.

[0057] The first storage cluster refers to the storage cluster with the largest currently available storage capacity among multiple storage clusters, or the storage cluster with the highest percentage of available storage capacity. The first storage cluster still has a significant amount of available storage capacity available for use. Here, the percentage of available storage capacity represents the proportion of available storage capacity to the total storage capacity of the storage cluster. The second storage cluster refers to the storage cluster with the smallest currently available storage capacity, or the storage cluster with the lowest percentage of available storage capacity. The second storage cluster is closest to full capacity.

[0058] The first difference is used to measure the degree of difference in available storage capacity between the first and second storage clusters. It is calculated by subtracting the available storage capacity of the second storage cluster from the available storage capacity of the first cluster. For example, if the available storage capacity of the first cluster is 85% and the available storage capacity of the second cluster is 20%, then the first difference is 65%. The first difference reflects the degree of uneven distribution of available storage capacity.

[0059] Calculating the first difference allows for real-time understanding of the overall distribution of available storage capacity in the current storage cluster, thus determining whether further balancing measures are needed. If the first difference is large, such as being greater than or equal to the first threshold, it indicates a significant resource skew problem, which may affect the efficiency of subsequent data allocation and the stability of the storage system.

[0060] When the first difference is less than the first threshold, it means that the available storage capacity difference between the various storage clusters is small, and there is no need to immediately perform data migration or storage capacity adjustment operations on these storage clusters. The current weighting strategy can be used to select the target storage cluster based on the second weight of each storage cluster to realize the storage of data blocks.

[0061] In summary, the method presented in this embodiment can ensure that the load adjustment of the storage cluster is performed based on the second weight when the difference in available storage capacity of each storage cluster is small, thereby reducing the storage pressure on the disk.

[0062] When there is a significant difference in available storage capacity between storage clusters, dynamic adjustment of data block storage based on the second weight may not be able to solve the problem of uneven data distribution in a timely manner. In this case, in some embodiments, the above method further includes: sorting the storage clusters according to the available storage capacity of each storage cluster to obtain a storage cluster sequence; determining a second difference between the available storage capacity of the third storage cluster and the available storage capacity of the fourth storage cluster in the storage cluster sequence; wherein the positive ranking of the third storage cluster in the storage cluster sequence is the same as the negative ranking of the fourth storage cluster in the storage cluster sequence; and if the absolute value of the second difference is greater than or equal to a first threshold, performing data migration on the data in the storage cluster with the smaller available storage capacity between the third and fourth storage clusters.

[0063] To facilitate determining the differences in available storage capacity within each storage cluster, the clusters can be sorted in descending or ascending order based on their available storage capacity to obtain a sequence of storage clusters.

[0064] For example, suppose there are multiple storage clusters A, B, C, and D, each with available storage capacities of 100GB, 200GB, 300GB, and 400GB respectively. The sequence of storage clusters sorted from smallest to largest would be [A, B, C, D], and the sequence from largest to smallest would be [D, C, B, A]. Sort the storage clusters according to their available storage capacity, which helps to quickly identify uneven data distribution and provides a basis for subsequent data migration.

[0065] Forward ranking refers to the sequential position of the storage clusters in the sorted sequence, numbered from front to back. For example, in the sorted storage cluster sequence [A,B,C,D], A has a forward ranking of 1, B has a forward ranking of 2, and so on. Reverse ranking, on the other hand, refers to the sequential position of the storage clusters in the reverse sequence, numbered from back to front. For example, in the sorted storage cluster sequence [A,B,C,D], D has a reverse ranking of 1, C has a reverse ranking of 2, B has a reverse ranking of 3, and A has a reverse ranking of 4.

[0066] Based on forward and reverse rankings, the third and fourth storage clusters are selected from the storage cluster sequence. The forward and reverse rankings of the third and fourth storage clusters are mirror images of each other, forming a storage cluster pair. This pair ensures that these clusters are located at opposite ends of the storage cluster sequence, representing two extreme cases: smaller and larger capacities. By calculating the difference in available storage capacity between the third and fourth storage clusters (i.e., the second difference), the existence of significant data distribution unevenness can be assessed.

[0067] For example, in the above storage cluster sequence, if the third storage cluster is B (ranked 2 in the forward direction) and the fourth storage cluster is C (ranked 2 in the reverse direction), then the second difference is 300GB-200GB=100GB.

[0068] The selection method for the third and fourth storage clusters ensures that the third and fourth storage clusters are comparative in terms of capacity. This selection method helps to identify storage clusters in the storage system that have potential uneven data distribution.

[0069] When the absolute value of the second difference is greater than or equal to the first threshold, it indicates that there is a large difference in the available storage capacity between the third and fourth storage clusters. In this case, the storage cluster with the smaller available storage capacity between the third and fourth storage clusters can be selected as the data source, and some of the data in that storage cluster can be migrated to any storage cluster with an available storage capacity greater than the second threshold, thereby achieving a more uniform data distribution.

[0070] For example, if the first threshold is 50GB and the second difference is 100GB, data migration is triggered, migrating some data from storage cluster B to storage cluster C or storage cluster D, thereby narrowing the gap in available storage capacity between storage cluster B and storage cluster C or storage cluster D.

[0071] It can be seen that dynamic migration of data within the storage cluster can be achieved through the second difference. Simultaneously, during the process of storing data blocks to the storage cluster, the target storage cluster for data block storage can be randomly determined using the second weight, based on the method given in the above embodiments. That is, when the second difference is greater than or equal to the first threshold, data migration and data storage can be performed simultaneously based on the method given in the embodiments of this application.

[0072] In this embodiment, by sorting the storage clusters according to their available storage capacity and calculating the second difference under a specific ranking, it can be determined whether data migration is necessary. This method allows for the timely detection of uneven data distribution among storage clusters, enabling precise execution of data migration strategies and effectively improving the load balancing capability and operational efficiency of the storage system.

[0073] Furthermore, in order to ensure that the data in each storage cluster is migrated in a balanced manner and that the available storage capacity of each storage cluster is similar, in some embodiments, the above-mentioned data migration of data in the storage cluster with the smaller available storage capacity between the third and fourth storage clusters includes: when the available storage capacity of the third storage cluster is greater than that of the fourth storage cluster, migrating the data in the fourth storage cluster to the third storage cluster.

[0074] By comparing the available storage capacity of the third and fourth storage clusters, it's possible to identify which cluster is closer to full capacity. When the available storage capacity of either the third or fourth storage cluster is significantly lower than that of the other storage cluster in the cluster pair—for example, if the available storage capacity of the fourth storage cluster is less than that of the third—data can be migrated from the storage cluster with the smaller available storage capacity (the fourth storage cluster) to the storage cluster with the larger available storage capacity (the third storage cluster) to avoid disk overload.

[0075] To further improve the data migration speed, the above-mentioned migration of data from the fourth storage cluster to the third storage cluster includes: determining the data to be migrated in the fourth storage cluster based on the first difference; dividing the data to be migrated; and migrating the divided data to be migrated to the third storage cluster in parallel.

[0076] For example, suppose the second difference in available storage capacity between the third and fourth storage clusters is... The data size in the fourth storage cluster can be [size missing]. The data was migrated. Specifically, data of size [size missing] was migrated. The target data migration task is divided into N subtasks, and the N subtasks are processed in parallel to achieve parallel data transmission.

[0077] In practical applications, it is possible to set the data size to be migrated by each of the N subtasks to be equal, that is, each subtask is responsible for migrating a data size of... The data can be divided into N subtasks and P task groups, where 1 < P < N and P is an integer divisible by N. By migrating the data of the subtasks in each task group sequentially, i.e., by migrating the data in parallel through the P task groups, at most P subtasks can perform data migration at any given time.

[0078] In this embodiment, the third and fourth storage clusters can be storage clusters from any storage cluster pair. That is, for any storage cluster pair where the second difference is greater than or equal to the first threshold, data migration can be performed based on the method given in this embodiment. To prevent the processing pressure on the storage system caused by a large amount of data migration, the total number of storage clusters participating in the data migration can be set to be less than a preset percentage of the total number of storage clusters, such as the total number of storage clusters participating in the data migration being less than 10% of the total number of storage clusters.

[0079] When the storage clusters in the storage cluster pair are the first and last storage clusters in the storage cluster sequence, the third and fourth storage clusters in this embodiment can correspond to the first and second storage clusters in the above embodiment, and the second difference can be the first difference.

[0080] During the data migration process, the second difference between storage cluster pairs can be monitored in real time or at preset time intervals. When the second difference is less than the first threshold, the subtask is stopped, the data migration is stopped, and the data migration task queue is cleared. Based on the method given in the above embodiments, data blocks are stored based on the second weight of each storage cluster to achieve data balance for each storage cluster.

[0081] Based on the above embodiments where a target storage cluster is selected based on a second weight for data block storage, in some embodiments, the above-mentioned processing of the first weight to obtain a second weight for each storage cluster includes: normalizing the first weight to obtain a processed first weight; and determining a second weight for each storage cluster based on the processed first weight.

[0082] The normalization method can adjust the first weight of each storage cluster to a uniform and controllable numerical range. This prevents some storage clusters from being overloaded due to excessively high first weights, thus mitigating the risk of overload. In this embodiment, by incorporating the normalization method, the minimum and maximum available storage capacities, or percentages of the minimum and maximum available storage capacities, are introduced into the normalization formula as benchmarks. Combined with a preset threshold, this ensures that the difference between the maximum and minimum values ​​of the second weight obtained after processing does not exceed the preset threshold.

[0083] After normalization, a second weight for each storage cluster can be calculated based on the processed first weight. The second weight reflects the actual usage priority of the current storage cluster under the dynamic balancing mechanism. Since the normalized first weight is already within a controllable range, it can be adjusted as needed to make the second weight more accurately reflect the actual situation of each storage resource and facilitate probability calculation.

[0084] Based on the method given in this embodiment, formula (2) provides a way to determine the second weight based on the first weight. Method: (2) in, This is the second weight, which is the final weight of the storage cluster; This represents the percentage (first weight) of available storage capacity in the current storage cluster (poolset). For example, when the percentage of available storage capacity is 50%, The value is 50; This represents the percentage of minimum available storage capacity across all storage clusters. This represents the percentage of maximum available storage capacity across all storage clusters. This is a preset value used to control the difference between the maximum and minimum values ​​of the second weight. It can also be set as a preset ratio of the maximum difference percentage; It can ensure that the denominator of the fraction in formula (2) is not 0; b is a preset value, which can determine the magnitude of the second weight and the minimum value of the second weight. In order to facilitate the calculation of the difference between the second weights and to facilitate the determination of the range of numerical change of the maximum value of the second weight relative to the minimum value, b can be 100.

[0085] Based on formula (2), it can be seen that when the percentage of available storage capacity in each storage cluster is the same, and , If they are equal, the second weight is at its minimum, and the minimum value of the second weight is b. Otherwise, in all other cases, the maximum value of the second weight is at least greater than its minimum value. The reason is that, for the fraction in the parentheses of formula (2), when When the numerator of the fraction is at its maximum, the denominator is 1 greater than the numerator. When the value is 1, the maximum value of the second weight is equal to That is, the minimum difference between the maximum and minimum values ​​of the second weight. .when When the value is greater than 1, the value of the fraction increases and gradually approaches 1. At this point, the maximum difference between the maximum and minimum values ​​of the second weight is: , Therefore, the range of the difference between the second weights of each storage cluster is [min, max]. In practical applications, when the storage function of a storage cluster is suspended, the weights of that storage cluster can be left unadjusted, that is, the first weights of that storage cluster can be left unprocessed.

[0086] It can be seen that, based on the method given by formula (2), when b is 100, the value of the second weight of each storage cluster is distributed in [100, 100+threshold), so that the percentage between the maximum and minimum values ​​of the second weight is controlled below the threshold. This can effectively prevent the risk of excessive data storage in a storage cluster due to an excessively large first weight, and improve the overall performance of the storage cluster.

[0087] Based on the storage method given in the above embodiments, Figure 2 A schematic diagram of another storage method is shown, such as Figure 2 As shown, the storage method includes: Step 201: Collect the minimum available storage capacity for each storage cluster.

[0088] That is, the above embodiments provide the real-time collection of the available storage capacity of the target storage resources of each storage cluster.

[0089] Step 202: Overall weight adjustment.

[0090] Based on the method given in the above embodiments, a first weight of each storage cluster is determined according to the available storage capacity of the target storage resources of each storage cluster, and a second weight of each storage cluster is determined based on the first weight.

[0091] Next, step 201 is executed, which involves updating the minimum available storage capacity of each storage cluster based on a preset fixed time interval, and updating the first weight and the second weight; at the same time, step 203 is executed.

[0092] Step 203: Determine whether the difference in available storage capacity is greater than or equal to the first threshold.

[0093] Based on the method given in the above embodiments, the difference in available storage capacity can be a second difference. A storage cluster sequence is determined by the second weight of each storage cluster, and a second difference between storage cluster pairs is determined within the storage cluster sequence. The second difference is then used to determine whether the difference in available storage capacity within the storage cluster pair is greater than or equal to a first threshold.

[0094] If the difference in available storage capacity is greater than or equal to the first threshold, proceed to step 204; otherwise, proceed to step 205.

[0095] Step 204: Select a storage cluster for data migration.

[0096] Based on the method given in the above embodiments, data is migrated to the storage cluster with the smaller available storage capacity in the storage cluster pair.

[0097] At the same time, a second difference is checked based on a preset time interval. When the second difference (the difference in available storage capacity) is less than the first threshold, step 205 is executed.

[0098] Step 205: End the migration task.

[0099] This application provides a storage method that, by applying a secondary smoothing method to the weights of storage clusters, reduces the range of variation of the first weight to obtain a second weight. The second weight of each storage cluster is controlled within a set range, and a target storage cluster is selected based on the second weight. The probability of storing data blocks in each storage cluster is adjusted. This method, executed cyclically according to a preset period, ensures the dynamic balance of each storage cluster in the storage system. If the second difference is greater than or equal to a first threshold, the storage capacity of the data to be migrated is calculated, and the data is migrated in segments in parallel. The migration progress is continuously monitored during the migration process. When the second difference is less than the first threshold, the remaining migration tasks are stopped, and data block storage adjustment continues based on the second weight.

[0100] The storage method provided in this application can control the maximum difference percentage of the second weight below the threshold, preventing excessive weight from causing an excessive amount of data to be stored on the disks of the corresponding storage cluster, thus affecting the overall performance of the storage cluster. A balancing strategy that simultaneously adjusts weights and migrates data reasonably corrects both the data storage and migration processes. The data migration task employs a parallel queue and coordinated feedback mechanism to ensure balanced data storage and migration while preventing over-migration.

[0101] Those skilled in the art will understand that the order in which the steps are written in the above-described method of the specific implementation does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0102] Based on the storage method proposed in the foregoing embodiments, this application also provides a storage device. Figure 3 A schematic diagram of the structure of a storage device is shown, such as Figure 3 As shown, the storage device includes: The processing module 301 is used to determine a first weight of each storage cluster based on the available storage capacity of each of the at least two storage clusters; process the first weight to obtain a second weight of each storage cluster; wherein, the storage cluster is used to store data blocks; the data blocks are obtained by processing the original stored data based on erasure coding technology; the first weight is positively correlated with the available storage capacity; and the range of change of the value of the second weight is smaller than the range of change of the value of the first weight.

[0103] Storage module 302 is used to select a target storage cluster from at least two storage clusters based on a second weight, and store data blocks in the target storage cluster; wherein the probability of the target storage cluster being selected is linearly positively correlated with the second weight of the target storage cluster.

[0104] In practical applications, the processing module 301 and the storage module 302 can be implemented based on a processor and a communication device.

[0105] In some embodiments, before selecting a target storage cluster from at least two storage clusters based on a second weight and storing the data block to the target storage cluster, the processing module 301 is further configured to determine a first storage cluster with the largest available storage capacity and a second storage cluster with the smallest available storage capacity in each storage cluster; determine a first difference between the available storage capacity of the first storage cluster and the available storage capacity of the second storage cluster; the storage module 302 is specifically configured to, if the first difference is less than a first threshold, select a target storage cluster from at least two storage clusters based on the second weight and store the data block to the target storage cluster.

[0106] In some embodiments, the processing module 301 is further configured to sort the storage clusters according to the available storage capacity of each storage cluster to obtain a storage cluster sequence; determine a second difference between the available storage capacity of the third storage cluster and the available storage capacity of the fourth storage cluster in the storage cluster sequence; wherein the positive ranking of the third storage cluster in the storage cluster sequence is the same as the negative ranking of the fourth storage cluster in the storage cluster sequence; the storage module 302 is further configured to perform data migration on the data in the storage cluster with the smaller available storage capacity between the third storage cluster and the fourth storage cluster if the absolute value of the second difference is greater than or equal to a first threshold.

[0107] In some embodiments, the storage module 302 is specifically used to migrate data from the fourth storage cluster to the third storage cluster when the available storage capacity of the third storage cluster is greater than the available storage capacity of the fourth storage cluster.

[0108] In some embodiments, the processing module 301 is specifically used to normalize the first weight to obtain the processed first weight; and to determine the second weight of each storage cluster based on the processed first weight.

[0109] In some embodiments, the processing module 301 is specifically configured to: determine the target storage resource with the smallest available storage capacity in each of at least two storage clusters; and determine the available storage capacity of the target storage resource in each storage cluster as a first weight for each storage cluster.

[0110] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0111] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0112] This application also provides an electronic device. Figure 4 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 4 As shown, the electronic device 40 may include: Memory 401 is used to store executable instructions.

[0113] The processor 402 is configured to implement any of the above storage methods when executing executable instructions stored in the memory 401.

[0114] The processor 402 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0115] The aforementioned computer-readable storage medium or memory 401 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0116] This application embodiment further provides a computer storage medium, on which computer executable instructions are stored, which are used to implement any of the storage methods provided in the above embodiments.

[0117] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the storage methods provided in the above embodiments.

[0118] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0119] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0120] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0121] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0122] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0124] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A storage method characterized by comprising: The method comprises: determining a first weight of each storage cluster in the at least two storage clusters based on available storage capacity of the each storage cluster; wherein the storage cluster is used for storing a data block; the data block is obtained by processing original storage data based on an erasure code technology; the first weight is positively correlated with the available storage capacity; processing the first weight to obtain a second weight of the each storage cluster; wherein a numerical value variation range of the second weight is smaller than a numerical value variation range of the first weight; selecting a target storage cluster from the at least two storage clusters based on the second weight, and storing the data block into the target storage cluster; wherein a probability of being selected by the target storage cluster is linearly positively correlated with the second weight of the target storage cluster.

2. The method of claim 1, wherein, Before the step of selecting a target storage cluster from the at least two storage clusters based on the second weight, and storing the data block into the target storage cluster, the method further comprises: determining a first storage cluster with the largest available storage capacity and a second storage cluster with the smallest available storage capacity in the each storage cluster; determining a first difference between the available storage capacity of the first storage cluster and the available storage capacity of the second storage cluster; the step of selecting a target storage cluster from the at least two storage clusters based on the second weight, and storing the data block into the target storage cluster, comprises: in a case that the first difference is less than a first threshold, selecting a target storage cluster from the at least two storage clusters based on the second weight, and storing the data block into the target storage cluster.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: sorting the storage clusters according to the available storage capacity of the each storage cluster to obtain a storage cluster sequence; determining a second difference between the available storage capacity of a third storage cluster and the available storage capacity of a fourth storage cluster in the storage cluster sequence; wherein a forward ranking of the third storage cluster in the storage cluster sequence is the same as a reverse ranking of the fourth storage cluster in the storage cluster sequence; in a case that an absolute value of the second difference is greater than or equal to a first threshold, performing data migration on data in a storage cluster with smaller available storage capacity from the third storage cluster and the fourth storage cluster.

4. The method of claim 3, wherein, the step of performing data migration on data in a storage cluster with smaller available storage capacity from the third storage cluster and the fourth storage cluster, comprises: in a case that the available storage capacity of the third storage cluster is greater than the available storage capacity of the fourth storage cluster, migrating data in the fourth storage cluster to the third storage cluster.

5. The method of claim 1, wherein, the step of processing the first weight to obtain a second weight of the each storage cluster, comprises: performing normalization processing on the first weight to obtain a processed first weight; determining the second weight of the each storage cluster based on the processed first weight.

6. The method of claim 1, wherein, the step of determining a first weight of each storage cluster in the at least two storage clusters based on available storage capacity of the each storage cluster, comprises: determining a target storage resource with the least available storage capacity in each of the at least two storage clusters; determining the available storage capacity of the target storage resource in each of the storage clusters as a first weight of each of the storage clusters.

7. The method of claim 1, wherein, a probability of the target storage cluster being selected is linearly positively correlated with a proportion of the second weight of the target storage cluster in the sum of the first weights.

8. An electronic device, comprising: The electronic device includes a processor and a memory for storing a computer program capable of running on the processor; wherein, The processor is configured to run the computer program to perform the method of any one of claims 1 to 7.

9. A computer storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1 to 7. The computer program, when executed by the processor, implements the method of any one of claims 1 to 7.

Images