Coded data collaborative restoration method and device with balanced load

Abstract

The invention discloses a load-balanced coded data collaborative restoration method and device, and relates to the technical field of data storage and recovery, and the method comprises the steps: carrying out the coding collaborative restoration of damaged disk data through employing a collaborative restoration algorithm BROR according to the data features of a damaged disk and a complete disk; and on the basis of a coding collaborative repair result, calculating the position of data required to be transmitted by each disk by using a load balancing algorithm BROR-LB, so as to realize data balancing among the nodes and complete coding data collaborative repair of load balancing. According to the invention, the problems of low data recovery efficiency and unbalanced system load in a large-scale storage system are solved.

Description

Technical Field

[0001] The present invention relates to the field of data storage and recovery technology, and in particular to a load-balanced coded data collaborative repair method and device. Background Art

[0002] In modern data storage systems, especially in large-scale cloud data centers and distributed storage systems, data security and efficient recovery are core requirements. Erasure coding, as a widely used technology, can effectively improve the fault tolerance of storage systems, especially in the face of hard disk failures, to ensure data integrity and accessibility. However, with the continuous growth of data volume and the expansion of storage system scale, existing erasure coding technology faces a series of challenges, especially in terms of data recovery efficiency and system load balancing.

[0003] Traditional erasure coding technology usually requires reading a large amount of data from multiple complete disks to reconstruct the lost data when a disk fails. This method performs well in small-scale systems, but in large-scale data centers, this data recovery strategy will lead to huge I / O load and network bandwidth consumption, seriously affecting the overall performance and response time of the system; technologies such as RDOR that have emerged in recent years use collaborative repair algorithms to reduce the amount of data transmission by reusing duplicate data between disks. However, this type of method is limited by the limited number of check disks. When the number of check disks in the disk array is greater than two, methods such as RDOR will fail; in addition, the load during the data recovery process is usually unevenly distributed among different nodes. Some nodes may face excessively high loads, while other nodes are relatively idle. This imbalance further exacerbates the performance bottleneck of the system.

[0004] Although existing technologies attempt to improve recovery efficiency by optimizing the design and implementation of erasure codes, these technologies still fail to address the need for fast and balanced data recovery in large-scale environments. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a load-balanced coded data collaborative repair method and device that solves the problems of low data recovery efficiency and unbalanced system load in large-scale storage systems.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a load-balanced coded data collaborative repair method, comprising the following steps: S1: Based on the data characteristics of the damaged disk and the intact disk, the collaborative repair algorithm BROR is used to perform coding and collaborative repair on the damaged disk data; S2: Based on the results of the coding collaborative repair, the load balancing algorithm BROR-LB is used to calculate the location of the data that needs to be transmitted on each disk, achieve data balance between nodes, and complete load-balanced coding data collaborative repair.

[0007] Furthermore, for a disk array using BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r verification rules to repair the damaged data; for the p-1 lost data, the collaborative repair algorithm BROR will use the verification rule with slope i to restore the Therefore, the verification data generated by r verification rules can all participate in data recovery. The formula is: , p represents a coding parameter in erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes involved in storage, and r represents the number of check disks or nodes in the storage array.

[0008] Furthermore, the use of the collaborative repair algorithm BROR to perform coding collaborative repair on damaged disk data includes the following steps: S11: Assume that the current disk array includes seven disks: disk 0, disk 1, disk 2, disk, disk 4, disk, and disk 6, wherein disk 0, disk 1, disk 2, disk 3, and disk 4 are used to store data, and disk 5 and disk 6 are used to store verification data; S12: Use the BR code with set parameters C (p=7, n=7, r=2) to encode the information data, obtain two verification data and store them on disk 5 and disk 6; S13: Assume that the information data on disk 0 is damaged. Based on the collaborative repair algorithm BROR, use the data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 .

[0009] Furthermore, the load balancing algorithm BROR-LB includes the following sub-steps: S21: Assume p = 7, the corresponding finite field GF(7) is expressed as , where the primitive element is set to 3; S22: Based on the load balancing algorithm BROR-LB, all elements in the finite field are represented using primitive elements. ; S23: Arrange all elements in the finite field to obtain a sequence ; S24: Set the step length r to 3, and Split according to the step size to get the set , and ; S25: Use the verification rule with a slope of 0 for To recover data, use the calibration rule with a slope of 1. To recover the data, use the calibration rule with a slope of 2. Perform data recovery.

[0010] The technical solution also adopted by the present invention is: a device for a load-balanced coded data collaborative repair method, characterized in that it includes: Single disk fault repair module: When data loss occurs, the single disk fault repair module first determines whether the number of damaged hard disks is 1. If so, the collaborative repair algorithm BROR of the single disk fault repair module is started to collaboratively repair the lost data; Load balancing module: The load balancing module calculates the location of the data required to be transmitted on each disk based on the load balancing algorithm BROR-LB to achieve data balance between nodes.

[0011] The beneficial effects of the present invention are as follows: the method and device of the present invention significantly improve the efficiency of data recovery and the stability of the system through advanced coding technology and load balancing strategy, and are particularly suitable for storage environments with large data volumes and high availability requirements. The implementation of these technologies can not only reduce the risks caused by data corruption, but also optimize resource utilization and improve the economic benefits and technical advantages of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 The present invention is a flow chart of a load-balanced collaborative repair method for coded data.

[0013] Figure 2 It is a schematic diagram of the BR code encoding and storage principle provided by an exemplary embodiment of the present invention.

[0014] Figure 3 It is a schematic diagram of the principle of a traditional single-disk repair strategy for BR codes provided by an exemplary embodiment of the present invention.

[0015] Figure 4 It is a schematic diagram of the principle of collaborative repair of single disk data provided by an exemplary embodiment of the present invention.

[0016] Figure 5 It is a schematic diagram of the principle of collaborative repair of load imbalance when r=2 provided by an exemplary embodiment of the present invention.

[0017] Figure 6 It is a schematic diagram of the principle of collaborative repair of load imbalance when r=2 provided by an exemplary embodiment of the present invention.

[0018] Figure 7 It is a diagram showing the effect of collaborative repair without using a load balancing algorithm provided by an exemplary embodiment of the present invention.

[0019] Figure 8 It is a schematic diagram of data transmission volume of a collaborative repair algorithm provided by an exemplary embodiment of the present invention in a single disk repair scenario.

[0020] Fig. 9 It is a schematic diagram of the disk data load balancing principle provided by an exemplary embodiment of the present invention.

[0021] Fig.10 It is a schematic diagram of the principle of collaborative repair load balancing when r=3 provided by an exemplary embodiment of the present invention.

[0022] Fig.11 It is a schematic diagram of an optimized load balancing principle provided by an exemplary embodiment of the present invention.

[0023] Fig.12 It is a schematic diagram of the repair speed of a collaborative repair algorithm provided by an exemplary embodiment of the present invention in a single disk failure scenario. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0025] Embodiment 1, as Figure 1 As shown, a load-balanced coded data collaborative repair method includes the following steps: S1: Based on the data characteristics of the damaged disk and the intact disk, the collaborative repair algorithm BROR is used to perform coding and collaborative repair on the damaged disk data; S2: Based on the results of the coding collaborative repair, the load balancing algorithm BROR-LB is used to calculate the location of the data that needs to be transmitted on each disk, achieve data balance between nodes, and complete load-balanced coding data collaborative repair.

[0026] In this embodiment, the decoding recovery process of BR code is involved. BR (Blaum-Roth) code is a binary quotient ring based The binary maximum distance separable (MDS) array code is , p is a prime number. The encoding process of BR code realizes data redundancy and recovery capability by utilizing polynomial operations on these binary quotient rings. , when p is a prime number, this code can recover all information bits from any k polynomials among n polynomials, thus ensuring high data reliability.

[0027] BR code array can be realized by multiple data disks. Disks of the same size are arranged in an array to form a disk array storage system, such as Figure 2 As shown. Among these disks, some are information disks and the rest are parity disks. The BR code operates on the data of the information disk through parity check rules with different slopes to obtain the parity disk data, thereby protecting the information from the impact of disk failure. When a disk in the system fails, in order to maintain the reliability level of the system, it is necessary to reconstruct the data in the failed disk by reading the corresponding information and parity data from all surviving disks, and the recovered data should be stored in the spare disk as soon as possible.

[0028] In the traditional BR code recovery scheme, if an information disk fails, the data of each erased bit can be recovered by reading the check bit of the row and other information bits in the row. Specifically, when an information disk fails, all symbols on the failed disk can be recovered one by one through XOR operations using the check bit of the corresponding row and other information symbols in the row. This recovery process uses only a single parity column, resulting in low recovery efficiency, especially in large-scale data recovery, with a large number of reads and calculations.

[0029] Traditional single-disk failure recovery strategies use only a single parity column for recovery. However, all data blocks are protected by multiple parity groups. The present invention introduces a BROR algorithm that uses multiple parity disk data for information recovery. It uses the information of all parity data to achieve: (1) reduce the amount of data involved in hard disk read operations; (2) ensure the load balancing characteristics of data transmission between all hard disks.

[0030] In the present invention, for a disk array using BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r verification rules to repair the damaged data; for the p-1 lost data, the collaborative repair algorithm BROR will use the verification rule with slope i to restore the damaged data. Therefore, the verification data generated by r verification rules can all participate in data recovery. The formula is: , p represents a coding parameter in erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes involved in storage, and r represents the number of check disks or nodes in the storage array.

[0031] The method of using the collaborative repair algorithm BROR to perform coding collaborative repair on damaged disk data includes the following steps: S11: Assume that the current disk array includes seven disks: Disk 0, Disk 1, Disk 2, Disk 3, Disk 4, Disk 5, and Disk 6, where Disk 0, Disk 1, Disk 2, Disk 3, and Disk 4 are used to store data, and Disk 5 and Disk 6 are used to store check data. Figure 3 As shown; S12: Use the BR code with set parameters C (p=7, n=7, r=2) to encode the information data, obtain two verification data and store them on disk 5 and disk 6; S13: Assume that the information data on disk 0 is damaged. Based on the collaborative repair algorithm BROR, use the data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 .

[0032] In this example, it is assumed that the information data on disk 0 is damaged, so the 6 copies of data on disk 0 need to be restored using data from other disks. If the traditional data recovery strategy is used, the data of disk 1, disk 2, disk 3, disk 4, disk 5, and disk 6 will be used for XOR operation, so that disk 0 can be recovered using the verification rule with a slope of 0. Therefore, a total of If Figure 4 As shown, this embodiment uses To mark lost data blocks, use To represent the data block used to recover data.

[0033] Because for a single failure, data can be recovered using only any single checksum group, the traditional recovery strategy is only one of many options. Figure 4 In the example of FIG. 1 , the traditional solution can use a check group with a slope of 0 for recovery, or use a check group with a slope of 1 for data recovery.

[0034] The collaborative repair strategy RDOR for RDP code points out that if a check group with a slope of 0 is used to recover , use the calibration group with slope 1 to recover , then the information of 24 data blocks will be used to restore the disk 0 .

[0035] Specific methods such as Figure 5 Compared with the traditional recovery method that requires 16 copies of data for recovery, BROR has a significant improvement in the amount of data to be calculated; and compared with the natural upper limit of the number of fault-tolerant disks in RDOR, BROR can adapt to disk arrays with any number of verifications and provide them with a collaborative repair strategy (such as Figure 6 The figure shows the collaborative repair strategy of RDOR in the scenario of r=3). This feature is particularly prominent in distributed scenarios. Since the data blocks used for recovery in BROR are repeated, it can convert the network I / O or disk I / O used in traditional recovery methods to faster memory data acquisition, thereby improving the data recovery rate. Even in a single-machine environment with a small difference in I / O rates, the data transmission volume reduced by BROR is still of great significance.

[0036] Figure 7 The figure shows the data size required for data repair in a single disk failure scenario using the traditional decoding algorithm and the BROR algorithm, respectively, with a total data volume of 200MB. The results in the figure show that compared with the traditional decoding algorithm, BROR can reduce the amount of data required for repair in a single disk failure scenario by about 80-100MB, or 22% to 27%, and can still maintain a similar optimization effect when r>2.

[0037] In this embodiment, a check rule with a slope of 0 is used to recover the first two rows of erased data in the matrix, and a check rule with a slope of 1 is used to recover the last two rows of erased data in the matrix. This method is the most intuitive and can also achieve the principle of minimizing the amount of data involved in recovery. However, the amount of data used for data recovery in each disk is not the same. This problem is particularly prominent in a distributed environment. The node that loses data needs to request data from all nodes where complete data is located in the form of network I / O, and then recover the data locally; when the difference in the amount of data involved in data recovery in each node is too large, the efficiency of data recovery often depends on the node with the largest data transmission volume, which will affect the effect of the BROR algorithm to a certain extent.

[0038] like Figure 8 As shown in the figure, the data is stored using BR code with parameters C (p=7,n=7,r=3), and disk 4, disk 5, and disk 6 are check data disks. In a distributed environment, assume that the data on disk 0 is lost and is restored using the check rule with a slope of 0. The data is restored using the calibration rule with a slope of 1 The data is restored using the calibration rule with a slope of 2 If the data is stored in disk 4 and disk 5, disk 1 and disk 6 need to transfer five copies of data, disk 3 and disk 2 need to transfer three and two copies of data respectively. The imbalance of data load on each node will eventually be reflected in the data recovery rate.

[0039] Based on this phenomenon, the present invention proposes a load balancing algorithm BROR-LB for data transmission between nodes, such as Fig. 9 As shown in the figure, this algorithm can ensure that the data involved in recovery is evenly distributed in each node or disk to the greatest extent, thereby ensuring the optimization of the BROR algorithm.

[0040] In one embodiment of the present invention, based on the premise of p=7, the load balancing of data recovery is actually equivalent to The BROR algorithm can maximize the data recovery efficiency only when the erased data is repaired with equal data volume by different verification rules. Therefore, when p=7 and r=3, it needs to Divide into three equal-sized sets , , The goal of BROR-LB is to evenly partition the integer set while ensuring that the data is repairable.

[0041] The load balancing algorithm BROR-LB includes the following steps: S21: Assume p = 7, the corresponding finite field GF(7) is expressed as , where the primitive element is set to 3; S22: Based on the load balancing algorithm BROR-LB, all elements in the finite field are represented using primitive elements. ; S23: Arrange all elements in the finite field to obtain a sequence ; S24: Set the step length r to 3, and Split according to the step size to get the set , and ; S25: Use the verification rule with a slope of 0 for To recover data, use the calibration rule with a slope of 1. To recover the data, use the calibration rule with a slope of 2. To recover data. (This method is different from using row validation rules to To recover data, use the calibration rule with a slope of 1. Data recovery is equivalent to using a calibration rule with a slope of 2. data recovery is equivalent).

[0042] BROR after using BROR-LB to partition the set can ensure that the restored data is as load balanced as possible. Fig.10 As shown, disk 0 to disk 6 each need to transfer 3 to 4 copies of data for data recovery.

[0043] The use of the BROR-LB algorithm can achieve a completely balanced load on data in some cases, but in more cases, the data in the above embodiment cannot be evenly distributed, that is, the 22 data blocks required for recovery cannot be evenly distributed on the 6 nodes. The reason is that the use of the BROR-LB algorithm will result in , as well as Three sets, namely disk 3 and disk 4 transmit 3 data blocks, and the remaining disks transmit 4 data blocks. In view of this phenomenon, the present invention will optimize the load balancing by rotating each stripe data when performing data recovery. Fig.11 As shown, if the strip x uses the verification rule with a slope of 0 To restore, disk 3 and disk 4 need to transfer 3 data blocks; then in stripe x+1, a check rule with a slope of 1 is required to check Finally, from a macro perspective, the data required for recovery will be evenly distributed on each hard disk to the greatest extent possible.

[0044] In this embodiment, the effects of a traditional decoding algorithm, a BROR algorithm, and a BROR algorithm including a load balancing strategy in this scenario are tested in a single-machine environment. Fig.12 The fault repair speed of BROR in a single-machine scenario is shown, that is, the decoding rate of BR code when repairing a single disk. As can be seen from the figure, the decoding rate of the BROR algorithm is improved by about 10% to 19% compared with the traditional repair algorithm, and the optimization effect decreases as k increases; BROR-LB has a more obvious improvement compared with BROR, and the decoding rate remains about 20% higher. The reason is that the load balancing strategy makes the amount of data required to be transmitted by each hard disk more uniform, reducing the tail delay effect.

[0045] Embodiment 2, a load-balanced coded data collaborative repair device, characterized by comprising: Single disk fault repair module: When data loss occurs, the single disk fault repair module first determines whether the number of damaged hard disks is 1. If so, the collaborative repair algorithm BROR of the single disk fault repair module is started to collaboratively repair the lost data; It calculates based on the encoding parameter information to obtain the amount of data that each disk needs to transmit, and calls the load balancing module based on whether load balancing is enabled to calculate the location of the data transmitted by each disk, and finally realizes the collaborative repair of the encoded data; that is, based on the location of the failed disk and the relevant parameters of the encoding used, it gives the location of the surviving data block that needs to be read for single-stripe collaborative repair, and reads the data into the cache.

[0046] Load balancing module: The load balancing module calculates the location of the data required to be transmitted on each disk based on the load balancing algorithm BROR-LB to achieve data balance between nodes; That is, according to the location of the failed disk and the relevant parameters of the encoding used, the location of the surviving data block required to be read for load-balanced multi-strip collaborative repair is given, and the data is read into the cache.

[0047] The present invention provides a load-balanced coded data collaborative repair method and a corresponding device, aiming to solve the problems of low data recovery efficiency and unbalanced system load in large-scale storage systems. By adopting improved BR codes and collaborative repair strategies, the present invention not only improves the speed of data recovery, but also optimizes the load distribution of data centers, and is particularly suitable for cloud storage environments that require high data reliability and rapid recovery capabilities.

[0048] The collaborative repair method of the present invention includes using BR code to encode the data stored in each disk array, so that the loss of any single disk data can be recovered from the data in other disks. In the data recovery process, the data of multiple disks are fully used to improve the recovery efficiency. By collaboratively distributing the recovery tasks among multiple disks, the present invention reduces the impact of single point failures and alleviates the load pressure of a single disk.

[0049] The present invention introduces a dynamic load balancing strategy that dynamically adjusts the allocation of data recovery tasks according to the current workload of each storage node. This strategy not only ensures the efficiency of the data recovery process, but also avoids the degradation of overall system performance due to overload of some nodes. By evenly distributing the data recovery transmission volume of each node or disk, the present invention significantly improves the load balancing characteristics of erasure codes during single disk recovery.

[0050] To achieve these technical goals, the present invention also provides a set of supporting devices, including but not limited to a single-disk fault repair module and a load balancing module. These modules work together to achieve rapid data recovery and efficient operation of the entire system. The single-disk fault repair module is responsible for processing data encoded by BR code, quickly locating and repairing damaged data; it calculates and recovers lost data through the collaborative repair algorithm proposed by the present invention, and reduces the amount of data involved in the repair process, thereby improving the recovery efficiency in the single-disk failure scenario. The load balancing module gives the location of the surviving data blocks that need to be read for data repair under the premise of load balancing based on the location of the faulty disk and the relevant parameters of the encoding used, ensuring that all repair data is evenly distributed among different storage nodes to prevent any single node from becoming a bottleneck.

[0051] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.

Claims

1. A load-balanced collaborative repair method for coded data, characterized in that: The following steps are involved: S1: Based on the data characteristics of the damaged disk and the intact disk, the collaborative repair algorithm BROR is used to perform coding and collaborative repair on the damaged disk data; S2: Based on the results of the coding collaborative repair, the load balancing algorithm BROR-LB is used to calculate the location of the data that needs to be transmitted on each disk, achieve data balance between nodes, and complete load-balanced coding data collaborative repair.

2. A load-balanced coded data collaborative repair method according to claim 1, characterized in that: For a disk array using BR code (C(p,n,r)), when any disk data is damaged, the collaborative repair algorithm BROR will use all r verification rules to repair the damaged data; for the p-1 lost data, the collaborative repair algorithm BROR will use the verification rule with slope i to restore the Therefore, the verification data generated by r verification rules can all participate in data recovery. The formula is: , p represents a coding parameter in erasure code encoding and decoding, which is a prime number, n represents the number of disks or nodes involved in storage, and r represents the number of check disks or nodes in the storage array.

3. A load-balanced coded data collaborative repair method according to claim 2, characterized in that: The method of using the collaborative repair algorithm BROR to perform coding collaborative repair on damaged disk data includes the following steps: S11: Assume that the current disk array includes seven disks: disk 0, disk 1, disk 2, disk, disk 4, disk, and disk 6, wherein disk 0, disk 1, disk 2, disk 3, and disk 4 are used to store data, and disk 5 and disk 6 are used to store verification data; S12: Use the BR code with set parameters C (p=7, n=7, r=2) to encode the information data, obtain two verification data and store them on disk 5 and disk 6; S13: Assume that the information data on disk 0 is damaged. Based on the collaborative repair algorithm BROR, use the data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 , usage data Restore Disk 0 .

4. A load-balanced coded data collaborative repair method according to claim 3, characterized in that: The load balancing algorithm BROR-LB includes the following steps: S21: Assume p = 7, the corresponding finite field GF(7) is expressed as , where the primitive element is set to 3; S22: Based on the load balancing algorithm BROR-LB, all elements in the finite field are represented using primitive elements. ; S23: Arrange all elements in the finite field to obtain a sequence ; S24: Set the step length r to 3, and Split according to the step size to get the set , and ; S25: Use the verification rule with a slope of 0 for To recover data, use the calibration rule with a slope of 1. To recover the data, use the calibration rule with a slope of 2. Perform data recovery.

5. A device for the load-balanced coded data collaborative repair method according to any one of claims 1 to 4, characterized in that: include: Single disk fault repair module: When data loss occurs, the single disk fault repair module first determines whether the number of damaged hard disks is 1. If so, the collaborative repair algorithm BROR of the single disk fault repair module is started to collaboratively repair the lost data; Load balancing module: The load balancing module calculates the location of the data required to be transmitted on each disk based on the load balancing algorithm BROR-LB to achieve data balance between nodes.

Images