A Load Balancing Method and Device for Cooperative Repair of Encoded Data

Abstract

The present invention discloses a load-balanced collaborative repair method and device for encoded data, which relates to the technical field of data storage and recovery. The method includes encoding and collaboratively repairing the damaged disk data by using the collaborative repair algorithm BROR according to the characteristics of the damaged disk and the complete disk data; based on the result of the encoded collaborative repair, calculating the positions of the data to be transmitted by each disk by using the load-balanced algorithm BROR-LB to achieve data balance among nodes and complete the load-balanced collaborative repair of the encoded data. The present invention solves the problems of low data recovery efficiency and unbalanced system load in large-scale storage systems.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and recovery, and particularly to a method and device for collaborative repair of coded data with load balancing. Background Art

[0002] In modern data storage systems, especially in large-scale cloud data centers and distributed storage systems, the security and efficient recovery of data are core requirements. As a widely used technology, erasure codes can effectively improve the fault tolerance of storage systems. Especially when facing hard disk failures, they can ensure the integrity and accessibility of data. However, with the continuous growth of data volume and the expansion of the scale of storage systems, existing erasure code technologies face a series of challenges, especially in terms of data recovery efficiency and system load balancing.

[0003] Traditional erasure code technologies usually need to read a large amount of data from multiple intact disks to reconstruct lost data when one disk fails. This method performs well in small-scale systems, but in large-scale data centers, this data recovery strategy will cause huge I / O loads and network bandwidth consumption, seriously affecting the overall performance and response time of the system; In recent years, technologies such as RDOR have adopted collaborative repair algorithms, which can reduce the amount of data transmission by reusing duplicate data between disks. However, such methods have limitations in the number of parity disks. When the number of parity disks in a disk array is greater than two, methods such as RDOR will fail; In addition, the load during the data recovery process is usually unevenly distributed among different nodes. Some nodes may face excessive loads while other nodes are relatively idle, and this imbalance further exacerbates the performance bottleneck of the system.

[0004] Although existing technologies have tried to improve the recovery efficiency by optimizing the design and implementation of erasure codes, these technologies still fail to meet the need for rapid and balanced data recovery in large-scale environments. Summary of the Invention

[0005] Aiming at the above deficiencies in the prior art, a method and device for collaborative repair of coded data with load balancing provided by the present invention solve the problems of low data recovery efficiency and uneven system load in large-scale storage systems.

[0006] To achieve the above invention object, the technical solution adopted by the present invention is: A method for collaborative repair of coded data with load balancing, comprising the following steps:

[0007] S1: According to the characteristics of damaged disks and intact disk data, use the collaborative repair algorithm BROR to perform coded collaborative repair on the damaged disk data;

[0008] S2: Based on the coding collaborative repair result, use the load balancing algorithm BROR-LB to calculate the positions of the data to be transmitted on each disk, achieve data balance among nodes, and complete the load balancing of the coded data collaborative repair.

[0009] Further, for a disk array using the BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r parity rules to repair the damaged data; for the lost p - 1 pieces of data, the collaborative repair algorithm BROR will use the parity rule with a slope of i to recover pieces of data, so that the parity data generated by all r parity rules can participate in the data recovery. The formula is: , where p represents a coding parameter in the erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes participating in storage, and r represents the number of parity disks or nodes in the storage array.

[0010] Further, the use of the collaborative repair algorithm BROR for coding collaborative repair of damaged disk data includes the following sub-steps:

[0011] S11: Assume that the current disk array includes seven disks: Disk 0, Disk 1, Disk 2, Disk 3, Disk 4, Disk 5, and Disk 6, where Disk 0, Disk 1, Disk 2, Disk 3, and Disk 4 are used to store data, and Disk 5 and Disk 6 are used to store parity data;

[0012] S12: Use the BR code with the set parameters C(p = 7, n = 7, r = 2) to perform coding calculations on the information data, obtain two pieces of parity data, and store them on Disk 5 and Disk 6;

[0013] S13: Assume that the information data on Disk 0 is damaged. Use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0.

[0014] Further, the load balancing algorithm BROR-LB includes the following sub-steps:

[0015] S21: Assume p = 7, and its corresponding finite field GF(7) is represented as , where the primitive element is set to 3;

[0016] S22: Represent all elements in the finite field using the primitive element as ;

[0017] S23: Arrange all elements in the finite field to obtain a sequence ;

[0018] S24: Set the step size r to 3, and split the sequence according to the step size to obtain sets , and ;

[0019] S25: Use a parity check rule with a slope of 0 to perform data recovery, use a parity check rule with a slope of 1 to perform data recovery, and use a parity check rule with a slope of 2 to perform data recovery.

[0020] Another technical solution adopted by the present invention is: An apparatus for a load - balanced coded data collaborative repair method, characterized by comprising:

[0021] Single - disk failure repair module: When data loss occurs, the single - disk failure repair module first determines whether the number of damaged hard disks is 1. If so, it starts the collaborative repair algorithm BROR of the single - disk failure repair module to perform collaborative repair of the lost data;

[0022] Load - balancing module: The load - balancing module calculates the positions of the data to be transmitted by each disk based on the load - balancing algorithm BROR - LB to achieve data balance between nodes.

[0023] The beneficial effects of the present invention are: The method and apparatus of the present invention significantly improve the efficiency of data recovery and the stability of the system through advanced coding techniques and load - balancing strategies, and are particularly suitable for storage environments with large amounts of data and high availability requirements. The implementation of these technologies can not only reduce the risks brought by data damage, but also optimize resource utilization, improving the economic benefits and technical advantages of the system. Brief Description of the Drawings

[0024] Figure 1 is a flowchart of a load - balanced coded data collaborative repair method.

[0025] Figure 2 is a schematic diagram of the BR - code encoding storage principle provided by an exemplary embodiment of the present invention.

[0026] Figure 3 is a schematic diagram of the traditional single - disk repair strategy principle of the BR - code provided by an exemplary embodiment of the present invention.

[0027] Figure 4 It is a schematic diagram of the principle of single-disk data collaborative repair provided by an exemplary embodiment of the present invention.

[0028] Figure 5 It is a schematic diagram of the principle of unbalanced collaborative repair load when r = 2 provided by an exemplary embodiment of the present invention.

[0029] Figure 6 It is a schematic diagram of the principle of unbalanced collaborative repair load when r = 2 provided by an exemplary embodiment of the present invention.

[0030] Figure 7 It is the effect diagram of collaborative repair without using the load balancing algorithm provided by an exemplary embodiment of the present invention.

[0031] Figure 8 It is a schematic diagram of the data transmission volume of the collaborative repair algorithm in the single-disk repair scenario provided by an exemplary embodiment of the present invention.

[0032] Figure 9 It is a schematic diagram of the principle of disk data load balancing provided by an exemplary embodiment of the present invention.

[0033] Figure 10 It is a schematic diagram of the principle of balanced collaborative repair when r = 3 provided by an exemplary embodiment of the present invention.

[0034] Figure 11 It is a schematic diagram of the principle of optimized load balancing provided by an exemplary embodiment of the present invention.

[0035] Figure 12 It is a schematic diagram of the repair speed of the collaborative repair algorithm in the single-disk failure scenario provided by an exemplary embodiment of the present invention. Detailed implementation manners

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] Embodiment 1, as Figure 1 shown, a method for collaborative repair of encoded data with load balancing includes the following steps:

[0038] S1: According to the characteristics of the damaged disk and the complete disk data, use the collaborative repair algorithm BROR to perform encoded collaborative repair on the damaged disk data;

[0039] S2: Based on the results of the encoded collaborative repair, use the load balancing algorithm BROR-LB to calculate the positions of the data to be transmitted by each disk, realize data balance between nodes, and complete the collaborative repair of encoded data with load balancing.

[0040] In this embodiment, it involves the decoding and recovery process of BR codes. The BR (Blaum-Roth) code is a binary maximum distance separable (MDS) array code based on a binary quotient ring where , and p is a prime number. The encoding process of the BR code realizes the redundancy and recovery ability of data by using polynomial operations on these binary quotient rings. The construction of the BR code utilizes the ring . When p is a prime number, this code can recover all information bits from any k polynomials out of n polynomials, thus ensuring high data reliability.

[0041] The BR code array can be implemented by multiple data disks. Disks of the same size are arranged in an array to form a disk array storage system, as Figure 2 shown. Among these disks, some are information disks and the rest are parity check disks. The BR code operates on the data of the information disks through parity check rules with different slopes to obtain the parity check disk data, thereby protecting the information from disk failures. When a disk in the system fails, in order to maintain the reliability level of the system, it is necessary to reconstruct the data in the failed disk by reading the corresponding information and parity check data from all surviving disks, and the recovered data should be stored in the spare disk as soon as possible.

[0042] In the traditional BR code recovery scheme, if an information disk fails, the data of each erased bit can be recovered by reading the parity check bits of that row and the other information bits of that row. Specifically, when a certain information disk fails, the parity check bits of the corresponding row and the other information symbols in that row are used, and all the symbols on the failed disk can be recovered one by one through XOR operations. This recovery process only uses a single parity check column, resulting in low recovery efficiency, especially in large-scale data recovery, where the number of reads and the amount of calculation are large.

[0043] The traditional single-disk failure recovery strategy only uses a single parity check column for recovery. However, all data blocks are protected by multiple different parity check groups. The present invention will introduce the BROR algorithm that uses the data of multiple parity check disks for information recovery. It utilizes the information of all parity check data, thereby being able to achieve: (1) reducing the amount of data involved in hard disk read operations; (2) ensuring the load balancing characteristics of data transmission among all hard disks.

[0044] In the present invention, for a disk array using the BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r kinds of parity check rules to repair the damaged data; for the lost p - 1 copies of data, the collaborative repair algorithm BROR will use the parity check rule with slope i to recover a copy of data, so that the parity data generated by r parity rules can all participate in data recovery. The formula is: , where p represents an encoding parameter in erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes participating in storage, and r represents the number of parity disks or nodes in the storage array.

[0045] The collaborative repair algorithm BROR is used to perform encoding collaborative repair on the damaged disk data, including the following sub-steps:

[0046] S11: Assume that the current disk array includes seven disks: Disk 0, Disk 1, Disk 2, Disk 3, Disk 4, Disk 5, and Disk 6. Among them, Disk 0, Disk 1, Disk 2, Disk 3, and Disk 4 are used to store data, and Disk 5 and Disk 6 are used to store parity data, as Figure 3 shown;

[0047] S12: Use the BR code with the set parameters C(p = 7, n = 7, r = 2) to perform encoding calculation on the information data, obtain two copies of parity data, and store them on Disk 5 and Disk 6;

[0048] S13: Assume that the information data on Disk 0 is damaged. Use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0.

[0049] In this embodiment, it is assumed that the information data on Disk 0 is damaged. Therefore, it is necessary to recover the 6 copies of data on Disk 0 through the data of other disks . If the traditional data recovery strategy is used, the data of Disk 1, Disk 2, Disk 3, Disk 4, Disk 5, and Disk 6 will be used for exclusive OR operation, so as to recover Disk 0 using the parity rule with a slope of 0. Therefore, a total of copies of data are required. As Figure 4 shown, in this embodiment, is used to mark the lost data blocks, and is used to represent the data blocks used to recover the data.

[0050] Because for a single failure, the data can be recovered using only any single parity group, the traditional recovery strategy is just one of many solutions. InFigure 4 In the example, the traditional scheme can use a check group with a slope of 0 for recovery, or a check group with a slope of 1 for data recovery.

[0051] The collaborative repair strategy RDOR for RDP codes points out that if a check group with a slope of 0 is used for recovery , and a check group with a slope of 1 is used for recovery , then ultimately the information of 24 data blocks will be used to recover the data of disk 0 .

[0052] The specific method is as Figure 5 shown. Compared with the traditional recovery method that requires 16 data for recovery, BROR has a significant improvement in the amount of calculated data; and compared with the natural upper limit of the number of fault-tolerant disks of RDOR, BROR can be adapted to disk arrays with any number of checks, providing a collaborative repair strategy for it (as Figure 6 shown is the collaborative repair strategy of RDOR in the scenario of r = 3). This feature is particularly prominent in the distributed scenario. Since there are duplicates in the data blocks used for recovery in BROR, it can convert the network I / O or disk I / O for obtaining data in the traditional recovery method into faster memory data acquisition, thereby improving the data recovery rate. Even in a single-machine environment with a small difference in I / O rates, the reduced data transmission volume of BROR still has great significance.

[0053] Figure 7 shows the data sizes required for data repair in the single-disk failure scenario using the traditional decoding algorithm and the BROR algorithm respectively on the premise that the total data volume is 200MB. From the results in the figure, it can be seen that compared with the traditional decoding algorithm, BROR can reduce the data volume required for repair in the single-disk failure scenario by about 80 - 100MB, that is, 22% to 27%, and still maintain a similar optimization effect when r > 2.

[0054] In this embodiment, the check rule with a slope of 0 is used to recover the erased data in the first two rows of the matrix, and the check rule with a slope of 1 is used to recover the erased data in the last two rows of the matrix. This method is the most intuitive and can also achieve the principle of minimizing the data participating in the recovery. However, the amount of data used for data recovery in each disk is not the same. This problem is particularly prominent in the distributed environment. The nodes that have lost data need to request data from all the nodes where the complete data is located in the form of network I / O, and then perform data recovery locally; when the difference in the amount of data participating in data recovery among each node is too large, the data recovery efficiency often depends on the node with the largest data transmission volume, which will to a certain extent affect the effect of the BROR algorithm.

[0055] As Figure 8As shown, the data usage parameter is stored using a BR code of C(p = 7, n = 7, r = 3). Disks 4, 5, and 6 are parity data disks. In a distributed environment, assume that the data on disk 0 is lost, and the data is recovered using a parity rule with a slope of 0 The data is recovered using a parity rule with a slope of 1 The data is recovered using a parity rule with a slope of 2 For the data, disks 4 and 5 need to transfer five pieces of data, disks 1 and 6 need to transfer four pieces of data, and disks 3 and 2 need to transfer three and two pieces of data respectively. The uneven data load of each node will ultimately be reflected in the data recovery rate

[0056] Based on this phenomenon, the present invention proposes a load balancing algorithm BROR-LB for data transfer between nodes, as Figure 9 shown. This algorithm can ensure that the data participating in the recovery is evenly distributed among each node or disk to the greatest extent, thereby optimizing the effect of the BROR algorithm

[0057] In an embodiment of the present invention, based on the premise of p = 7, the load balancing of data recovery is actually equivalent to grouping. The BROR algorithm can maximize the data recovery efficiency only when the erased data is repaired with equal data amounts by different parity rules. Therefore, when p = 7 and r = 3, it needs to be divided into three sets of equal size , , . The goal of BROR-LB is how to evenly divide the integer set on the premise of ensuring data repairability

[0058] The load balancing algorithm BROR-LB includes the following sub-steps

[0059] S21: Assume p = 7, and its corresponding finite field GF(7) is represented as , where the primitive element is set to 3

[0060] S22: Represent all elements in the finite field using the primitive element as ;

[0061] S23: Arrange all elements in the finite field to obtain the sequence ;

[0062] S24: Set the step size r to 3, and split the sequence according to the step size to obtain the sets , and ;

[0063] S25: Perform data recovery using a verification rule with a slope of 0 on Perform data recovery using a verification rule with a slope of 1 on Perform data recovery using a verification rule with a slope of 2 on Perform data recovery. (This method is equivalent to performing data recovery using a row verification rule on Perform data recovery using a verification rule with a slope of 1 on Perform data recovery, and performing data recovery using a verification rule with a slope of 2 on Perform data recovery is equivalent).

[0064] After using BROR-LB for set partitioning, BROR can ensure that the recovered data is as load-balanced as possible. As Figure 10 shown, disks 0 to 6 all need to transfer 3 to 4 data copies for data recovery.

[0065] Using the BROR-LB algorithm can achieve a fully load-balanced effect for data in some cases. However, in more cases, there will be a situation where the data cannot be evenly distributed as in the above embodiments, that is, the 22 data blocks required for recovery cannot be evenly distributed among 6 nodes. The reason is that using the BROR-LB algorithm will result in , and three sets, that is, disks 3 and 4 transfer 3 data blocks, and the remaining disks transfer 4 data blocks. In response to this phenomenon, when performing data recovery in the present invention, the method of rotating each stripe data will be adopted to optimize the load balance. As Figure 11 shown, if stripe x uses a verification rule with a slope of 0 to perform recovery, such that disks 3 and 4 need to transfer 3 data blocks; then in stripe x + 1, a verification rule with a slope of 1 needs to be used to perform recovery. Ultimately, from a macroscopic perspective, the data required for recovery will be evenly distributed on each hard disk to the greatest extent.

[0066] In this embodiment, the effects of the traditional decoding algorithm, the BROR algorithm, and the BROR algorithm with a load balancing strategy are respectively tested in a single-machine environment in this scenario. Figure 12 Shows the fault repair speed of BROR in a single-machine scenario, that is, the decoding rate of the BR code during single-disk repair. As can be seen from the figure, the BROR algorithm has improved the decoding rate by about 10% to 19% compared with the traditional repair algorithm, and the optimization effect decreases as k increases; BROR-LB has a relatively obvious improvement compared with BROR, and the decoding rate maintains an improvement of about 20%. The reason is that the load balancing strategy makes the amount of data required to be transferred by each hard disk more uniform, reducing the tail latency effect.

[0067] Embodiment 2. A load - balanced coded data collaborative repair device, characterized by comprising:

[0068] Single - disk failure repair module: When data loss occurs, the single - disk failure repair module first determines whether the number of damaged hard disks is 1. If so, it starts the collaborative repair algorithm BROR of the single - disk failure repair module to perform collaborative repair of the lost data;

[0069] It calculates according to the coding parameter information to obtain the data volume that each disk needs to transmit, and based on whether load balancing is enabled, it calls the load - balancing module to calculate the positions where the data transmitted by each disk is located, and finally realizes the collaborative repair of the coded data; that is, according to the position of the failed disk and the relevant parameters of the used coding, it gives the positions of the surviving data blocks to be read during single - stripe collaborative repair, and reads the data into the cache.

[0070] Load - balancing module: The load - balancing module calculates the positions of the data that each disk needs to transmit based on the load - balancing algorithm BROR - LB to achieve data balance between nodes;

[0071] That is, according to the position of the failed disk and the relevant parameters of the used coding, it gives the positions of the surviving data blocks to be read during multi - stripe collaborative repair with load balancing, and reads the data into the cache.

[0072] The present invention provides a load - balanced coded data collaborative repair method and a corresponding device, aiming to solve the problems of low data recovery efficiency and unbalanced system load in large - scale storage systems. By adopting an improved BR code and a collaborative repair strategy, the present invention not only improves the speed of data recovery, but also optimizes the load distribution in the data center, and is particularly suitable for cloud storage environments that require high data reliability and fast recovery capabilities.

[0073] The collaborative repair method of the present invention includes encoding the data stored in each disk array using the BR code, so that the loss of any single - disk data can be recovered from the data in other disks. During the data recovery process, the data of multiple disks are all used to improve the recovery efficiency. By collaboratively distributing the recovery tasks among multiple disks, the present invention reduces the impact of single - point failures and at the same time reduces the load pressure on a single disk.

[0074] The present invention introduces a dynamic load - balancing strategy, which dynamically adjusts the allocation of data recovery tasks according to the current workload of each storage node. This strategy not only ensures the efficiency of the data recovery process, but also avoids the overall system performance degradation caused by overloading of some nodes. By evenly distributing the data recovery transmission volume of each node or disk, the present invention significantly improves the load - balancing characteristics of the erasure code during single - disk recovery.

[0075] To achieve these technical objectives, the present invention also provides a set of supporting devices, including but not limited to a single-disk failure repair module and a load balancing module. These modules work together to achieve fast data recovery and efficient operation of the entire system. The single-disk failure repair module is responsible for processing data encoded by the BR code, quickly locating and repairing damaged data; it calculates and recovers lost data through the collaborative repair algorithm proposed by the present invention, and improves the recovery efficiency in the single-disk failure scenario by reducing the amount of data participating in the repair process. The load balancing module, based on the location of the failed disk and the relevant parameters of the encoding used, gives the locations of the surviving data blocks to be read for data repair under the premise of load balancing, ensuring that all repair data is evenly distributed among different storage nodes and preventing any single node from becoming a bottleneck.

[0076] Those of ordinary skill in the art will realize that the embodiments described herein are for helping readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the invention.

Claims

1. A method for collaborative repair of encoded data with load balancing, characterized in that, It includes the following steps: S1: According to the damaged disk and the data characteristics of the complete disk, use the collaborative repair algorithm BROR to perform encoded collaborative repair on the damaged disk data; The encoded collaborative repair of the damaged disk data using the collaborative repair algorithm BROR includes the following sub-steps: S11: Assume that the current disk array includes seven disks: Disk 0, Disk 1, Disk 2, Disk 3, Disk 4, Disk 5, and Disk 6, where Disk 0, Disk 1, Disk 2, Disk 3, and Disk 4 are used to store data, and Disk 5 and Disk 6 are used to store parity data; S12: Use the BR code with the set parameters of C(p = 7, n = 7, r = 2) to perform encoding calculation on the information data, obtain two pieces of parity data, and store them on Disk 5 and Disk 6; S13: Assume that the information data on Disk 0 is damaged, and use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0, use the data to recover the on Disk 0; S2: Based on the results of the encoded collaborative repair, use the load balancing algorithm BROR-LB to calculate the positions of the data to be transmitted for each disk, achieve data balance between nodes, and complete the encoded data collaborative repair with load balancing; The load balancing algorithm BROR-LB includes the following sub-steps: S21: Assume p = 7, and its corresponding finite field GF(7) is expressed as , where the primitive element is set to 3; S22: Represent all elements in the finite field using a primitive element as ; S23: Permute all elements in the finite field to obtain a sequence ; S24: Set the step size r to 3, and split the sequence according to the step size to obtain the sets , and ; S25: Perform data recovery on using a check rule with a slope of 0, perform data recovery on using a check rule with a slope of 1, perform data recovery on using a check rule with a slope of 2.

2. The method for collaborative repair of encoded data with load balancing according to claim 1, wherein For a disk array using the BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r parity rules to repair the damaged data; for the lost p - 1 pieces of data, the collaborative repair algorithm BROR will use the parity rule with slope i to recover pieces of data, so that the parity data generated by all r parity rules can participate in data recovery. The formula is: , where p represents an encoding parameter in erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes participating in storage, and r represents the number of parity disks or nodes in the storage array.

3. An apparatus for a load-balanced coded data collaborative repair method according to any one of claims 1-2, characterized in that, It includes: Single-disk failure repair module: When data loss occurs, the single-disk failure repair module first determines whether the number of damaged hard disks is 1. If so, start the collaborative repair algorithm BROR of the single-disk failure repair module to perform collaborative repair of the lost data; Load balancing module: The load balancing module calculates the positions of the data to be transmitted for each disk based on the load balancing algorithm BROR-LB to achieve data balance between nodes.

Images