Node repair method and related device

Abstract

Embodiments of the present application provide a node repairing method and related equipment, which are used to solve the network transmission problem caused by repairing multiple nodes in a distributed storage system. The method comprises: acquiring all data nodes in the distributed storage system; wherein the all data nodes comprise fault nodes and surviving nodes; scheduling and grouping the surviving nodes, wherein the surviving nodes in any group are in pairs; performing a merge operation on the surviving nodes in the same group to transmit the node repairing data in the current surviving node to other surviving nodes in the same group to form a node repairing block; wherein the node repairing block is formed by the node repairing data of other surviving nodes and the node repairing data of the current surviving node; and transmitting the node repairing block in any group to the fault nodes to repair the fault nodes, so that the fault nodes complete fault repairing according to the node repairing block.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a node repair method and related equipment. Background Technology

[0002] Reed-Solomon codes (RS(k,m)) are the most widely used erasure codes in the erasure code family, especially in distributed storage systems.

[0003] To achieve high availability and low storage costs, erasure coding is widely used to replace replication. Compared to replication, erasure coding can reduce storage costs, but it also introduces higher repair overhead. For example, current multi-node repair algorithms are multiple single-node repair algorithms in parallel, which leads to two serious problems: unbalanced load among nodes and redundant block transmission resulting from multiple independent single-node repairs. When repairing multiple blocks within a stripe, the imbalance between the number of uploaded and downloaded blocks per node can easily cause repair time to depend on the heaviest-loaded node. In this case, network transmission overhead dominates in practical applications, and this overhead increases linearly with the number of faulty nodes, thus increasing repair time. Summary of the Invention

[0004] This application provides a node repair method and related equipment to solve the network transmission problems caused by repairing multiple nodes in a distributed storage system, so as to speed up the repair process as much as possible.

[0005] The first aspect of this application provides a node repair method applied to a distributed storage system, comprising:

[0006] Obtain all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and the surviving nodes are the nodes among all data nodes excluding the faulty nodes;

[0007] The surviving nodes are scheduled into groups, wherein the surviving nodes in any group exist in pairs;

[0008] The surviving nodes in the same group are merged to transfer the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block; wherein, the node repair block is formed by the node repair data of the other surviving nodes and the node repair data of the current surviving node;

[0009] The node repair block in any group is transmitted to the faulty node to repair the faulty node, so that the faulty node completes the fault repair according to the node repair block.

[0010] Optionally, the step of scheduling and grouping the surviving nodes includes:

[0011] Divide the k surviving nodes into Groups, in which any Each group includes two surviving nodes, where k is a positive integer;

[0012] The step of merging the surviving nodes in the same group to transmit the node repair data from the current surviving node to other surviving nodes in the same group includes:

[0013] The repair data of the first surviving node is transmitted to the second surviving node; wherein, the first surviving node is the one described above. The second surviving node is the surviving node in the second half of the same group, which is the surviving node in the group. Surviving nodes in the same group and located in the first half of the group.

[0014] Optionally, after transmitting the node repair data of the first surviving node to the second surviving node, the method further includes:

[0015] The Perform a merge operation on each group to obtain... Groups; wherein any group includes four of the surviving nodes;

[0016] Obtain the repair data of the third surviving node and the fourth surviving node, and determine the third weighting coefficient corresponding to the third surviving node and the fourth weighting coefficient corresponding to the fourth surviving node; wherein, the third surviving node and the fourth surviving node are respectively located in the... Surviving nodes in the same group and located in the latter half of the group;

[0017] The system determines the repair data of the fifth surviving node corresponding to the fifth surviving node based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the third weighting coefficient; and determines the repair data of the sixth surviving node corresponding to the sixth surviving node based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the fourth weighting coefficient; wherein, the fifth surviving node and the sixth surviving node are respectively in the... Surviving nodes in the same group and located in the first half of the group;

[0018] The repair data of the fifth surviving node is transmitted to the fifth surviving node, and the repair data of the sixth surviving node is transmitted to the sixth surviving node.

[0019] Optionally, the method further includes:

[0020] If half the number of surviving nodes in the same group is less than the number of faulty nodes, the node repair data of the surviving nodes in the first half of the current group is transmitted to the surviving nodes in the second half.

[0021] The node repair block includes:

[0022] The surviving nodes after transmitting the node repair data are merged to generate the node repair block; wherein, the node repair block contains all the node repair data obtained by the current surviving node.

[0023] Optionally, the method further includes:

[0024] If there exists that is not located in any The surviving nodes in each group are determined not to be located in any The surviving nodes in each group are the remaining surviving nodes, and all the remaining surviving nodes are merged into the remaining group;

[0025] The node repair block includes:

[0026] Determine the node repair data of the remaining surviving nodes in the remaining group and the weighting coefficients corresponding to the node repair data;

[0027] Based on the node repair data and weighting coefficients of the remaining surviving nodes, a node repair block corresponding to the remaining group is generated.

[0028] Optionally, if the number of remaining surviving nodes in the remaining group is less than the number of faulty nodes, and the faulty nodes include a first faulty node, a second faulty node, and a third faulty node, and the remaining surviving nodes include a first remaining surviving node and a second remaining surviving node, then generating a node repair block corresponding to the remaining group based on the node repair data and weighting coefficients of the remaining surviving nodes includes:

[0029] The first remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the first remaining weighting coefficient; wherein, the first remaining weighting coefficient is the weighting coefficient corresponding to the first faulty node;

[0030] The second remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the second remaining weighting coefficient; wherein, the second remaining weighting coefficient is the weighting coefficient corresponding to the second faulty node;

[0031] The third remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the third remaining weighting coefficient; wherein, the third remaining weighting coefficient is the weighting coefficient corresponding to the third faulty node;

[0032] The step of transmitting the node repair block from any group to the faulty node and repairing the faulty node includes:

[0033] The first remaining node repair block, the second remaining node repair block, or the third remaining node repair block are respectively transmitted to the first fault node, the second fault node, or the third fault node to repair the first fault node, the second fault node, or the third fault node respectively.

[0034] Optionally, the step of transmitting the node repair block from any group to the faulty node and repairing the faulty node includes:

[0035] Sort the surviving nodes in any group;

[0036] If the number of faulty nodes is m, the node repair blocks corresponding to the top m surviving nodes in any group are transmitted to the faulty nodes respectively, and the m faulty nodes are repaired; where m is a positive integer.

[0037] A second aspect of this application provides a node repair system applied to a distributed storage system, comprising:

[0038] An acquisition unit is used to acquire all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and the surviving nodes are the nodes among all data nodes excluding the faulty nodes;

[0039] A grouping unit is used to schedule and group the surviving nodes, wherein the surviving nodes in any group exist in pairs;

[0040] The merging unit is used to merge the surviving nodes in the same group to transfer the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block; wherein, the node repair block is formed by the node repair data of the other surviving nodes and the node repair data of the current surviving node.

[0041] A transmission unit is used to transmit the node repair block in any group to the faulty node to repair the faulty node, so that the faulty node can complete the fault repair according to the node repair block.

[0042] The second aspect of this application provides a method for performing the node repair method described in the first aspect.

[0043] A third aspect of this application provides a node repair apparatus, comprising:

[0044] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;

[0045] The memory is either a short-term storage memory or a persistent storage memory;

[0046] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the node repair method described in the first aspect.

[0047] A fourth aspect of this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the node repair method described in the first aspect.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The node repair method disclosed in this application first obtains all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and surviving nodes are the nodes among all data nodes excluding the faulty nodes; then, the surviving nodes are scheduled and grouped, wherein surviving nodes in any group exist in pairs; then, the surviving nodes in the same group are merged to transmit the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block; wherein, the node repair block is formed by the node repair data of other surviving nodes and the node repair data of the current surviving node; finally, the node repair block in any group is transmitted to the faulty node to repair the faulty node, so that the faulty node completes the fault repair according to the node repair block. Therefore, in the distributed storage system, faulty nodes can be effectively repaired through surviving nodes. At the same time, grouping and transmitting surviving nodes also facilitates reducing network transmission overhead during faulty node repair, thereby accelerating the repair speed and optimizing the multi-node repair process in the distributed storage system as much as possible. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1This is a schematic diagram of the encoding and decoding process of an existing RS(4,2) code;

[0051] Figure 2 This is a schematic diagram of a repair process for an existing RS(4,2) code;

[0052] Figure 3 This is a schematic diagram of the PPR repair process for an existing RS(4,2) code;

[0053] Figure 4 This is a schematic diagram of a load imbalance in an existing RS(6,2) code.

[0054] Figure 5 This is a schematic diagram of the transmission time slots of an existing RS(4,3) code;

[0055] Figure 6 This is a flowchart illustrating a node repair method disclosed in an embodiment of this application;

[0056] Figure 7 This is a flowchart illustrating another node repair method disclosed in an embodiment of this application;

[0057] Figure 8 This is a schematic diagram of a multi-node repair of an RS(6,3) code disclosed in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of a node repair system disclosed in an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of the structure of a node repair device disclosed in an embodiment of this application. Detailed Implementation

[0060] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0062] To facilitate understanding of the technical solutions in the embodiments of this application, the following describes the current mainstream methods for multi-node erasure coding fault repair optimization. It is easy to understand that current multi-node erasure coding fault repair optimization methods mainly execute repair schemes for multiple single nodes in parallel.

[0063] The following describes the mainstream node failure repair methods. It's important to note beforehand that erasure coding has the characteristic that when the number of failed data blocks in a stripe is less than or equal to m, the original data can always be reconstructed. Users can flexibly configure different parameters to achieve a balance between data reliability and redundancy.

[0064] For easier understanding and description, please refer to Figure 1 , Figure 1 This is a schematic diagram of the encoding and decoding process of an existing RS(4,2) code. For (k,m) erasure codes, a data object of size R is divided into k data blocks {C0, C1, ... C}. k-1 The size of each block is ,} Then according to the coding equation k data blocks are encoded into m parity blocks, c ij (0≤i≤m-1,0≤j≤k-1) represents the calculation of check block P. i Time C j The coefficients, and the encoding process can also be represented by the product of the (n, k) generator matrix G and the (k, 1) data column vector. It's easy to understand that for (k, m) erasure coding, it means that k data blocks are encoded into m parity blocks, and (k, m) represents a k-row, m-column submatrix during decoding. For ease of understanding and description, this will not be elaborated further. It's also easy to understand that i and j in this embodiment are both integers.

[0065] In this case, the decoding operation of the RS code can be performed by inverting the (k, k) submatrix of the generator matrix and multiplying it by the surviving data vector of (k, 1) to recover the faulty block. For example... Figure 1As shown, when two data blocks C0 and C1 fail, recovery involves two steps. First, the (4,4) submatrix corresponding to the row (2,3,4,5) of the remaining blocks in the generator matrix is ​​inverted. Second, the inverse matrix is ​​multiplied by the surviving block vectors C2, C3, P0, and P1. The product is the original data blocks C0 and C1. It's important to understand that in computer languages, values ​​are generally 0 as the starting value; therefore, 0 represents (1,0,0,0) in the first row.

[0066] For other existing technical solutions, please refer to Figure 2 , Figure 2 This is a schematic diagram of a repair process for an existing RS(4,2) code, where k=4 and m=2. Data block repair in a distributed storage cluster requires the repair node to retrieve k data blocks from k different surviving nodes, all of which are on the same stripe as the faulty data block. It's easy to understand that, assuming a block D... i A malfunction occurred, and It is a set of k surviving blocks. Therefore, a linear combination of these k blocks can be used to repair the block. , where a ij It is repair block D i Time D j The corresponding coefficient. In Figure 2 In the process, when block D0 fails, the four surviving nodes N1, N2, N3, and N4 transmit blocks D1, D2, D3, and D4 to node N6 respectively, and then, according to the decoding equation... The repair of the block took a total of 4 time slots. It is easy to understand that in this embodiment, i and j are both integers.

[0067] For other existing technical solutions, please refer to Figure 3 , Figure 3 This is a schematic diagram of a PPR repair process for an existing RS(4,2) code, where k=4 and m=2. The single-node repair algorithm PPR decomposes the entire repair process into multiple sub-processes to reduce the download traffic of the repair nodes. Figure 3 This demonstrates the repair process of the algorithm. First, nodes N2 and N4 receive block a sent by N1 and N3. 01 D1 and a 03 D3, node N2 calculates the intermediate block a by combining the received block with its own block. 01 D1+a 02 D2 then sends the intermediate block to node N4, and finally N4 completes the block calculation and sends it to the repair node N6, taking a total of 3 time slots.

[0068] For other existing technical solutions, please refer to Figure 4 , Figure 4This diagram illustrates a load imbalance in an existing RS(6,2) code, where k=6 and m=2. This existing scheme employs a parallel multi-node repair algorithm. This leads to two serious problems: load imbalance between nodes and redundant block transmission resulting from multiple independent single-node repairs. Figure 4 This indicates a load imbalance issue. When repairing two blocks within a stripe, the imbalance in the number of uploaded and downloaded blocks per node causes the repair time to depend on the heaviest-loaded node. Figure 4 The numbers corresponding to "upload" and "download" on the left represent the upload and download load during transmission in existing solutions. The numbers corresponding to "upload" and "download" on the right represent the upload and download load during transmission in the technical solution of this application.

[0069] There is also an existing technical solution; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the transmission time slots for an existing RS(4,3) code. That is, k = 4, m = 3. Wherein, Figure 5 This illustrates the problem of redundant block transmission caused by multiple independent single-node repairs. When transmitting data blocks, the original data blocks can be transmitted so that multiple nodes can repair multiple blocks simultaneously, reducing unnecessary block transmissions and thus reducing repair time. It should be noted that... Figure 5 The left side shows the existing technical solution. Figure 5 The technical solution of this application is shown on the right. (By...) Figure 5 It can also be seen that by transmitting the original data block to other nodes, those nodes can use this original data block to repair data blocks on multiple nodes. Figure 5 Logically, the left side uses 3 time slots, and the right side uses 2 time slots. Therefore, compared with existing technical solutions, the technical solution of this application can significantly reduce the transmission time.

[0070] To facilitate a detailed description of the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] Based on the above description, this application discloses a multi-node repair scheduling optimization method in a distributed erasure coding storage system. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating a node repair method disclosed in an embodiment of this application. It includes steps 601-604.

[0072] 601. Obtain all data nodes in the distributed storage system.

[0073] It should be noted beforehand that in the technical solution of this application, the algorithm divides the repair process into two main stages: the merging stage and the transmission stage. At the merging nodes, only surviving nodes transmit data blocks to each other, while at the transmission nodes, data blocks are transmitted to the faulty nodes (which can also be understood as repair nodes). For ease of understanding, this will be described in detail later.

[0074] In one embodiment, the distributed storage system can run on a metadata server. In the distributed storage system, if some data nodes fail, the system can retrieve all data nodes in the current system. Specifically, data block repair in the distributed storage cluster requires surviving nodes to retrieve k data blocks, all of which are on the same strip as the failed data block. Thus, all data nodes in the current distributed storage system that are on the same strip as the failed node can be retrieved. It is easy to understand that these data nodes include surviving nodes and failed nodes. Failed nodes are the data blocks that have failed, and surviving nodes are the data blocks that have not failed. For ease of understanding and description, this will not be elaborated further. For further understanding, please refer to... Figure 3 The specific description of the stripes is omitted here.

[0075] 602. The surviving nodes are grouped for scheduling.

[0076] Algorithms in distributed storage systems first identify all surviving nodes among the data nodes, and then group these surviving nodes for scheduling. In any given group, the surviving nodes exist in pairs. This can be understood as the number of surviving nodes in any group being even.

[0077] In one specific embodiment, the algorithm can divide the k surviving nodes into multiple parts for scheduling, with each part containing a power of 2 nodes. These parts can be executed in parallel. Each part can be understood as a group or set of nodes.

[0078] Specifically, if the set of surviving nodes is S and the set of failed nodes is M, then in dividing the k surviving nodes into p parts (or groups), It's easy to understand that k, p, and i are all natural numbers. This refers to the corresponding part or group, in which the number of surviving nodes is even.

[0079] 603. Merge the surviving nodes in the same group to transfer the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block.

[0080] After grouping the surviving nodes, a merging operation can be performed on the surviving nodes within the same group. This allows the node repair data from the current surviving node to be transmitted to other surviving nodes in the same group, forming a node repair block. See details in [link to documentation]. Figure 4 and Figure 5 The description of the transmitted data.

[0081] In one specific embodiment, the algorithm transmits original data blocks between surviving nodes in the same group. Specifically, nodes in the latter half of each group transmit node repair data to nodes in the former half. It's easy to understand that this node repair data can be interpreted as the node data of the surviving node; details will not be elaborated here. Based on this embodiment, in another specific embodiment, if the number of faulty nodes is greater than half the number of nodes in any group, the nodes in the former half of the group transmit additional blocks to the nodes in the latter half. Simultaneously, if the merge operation cannot be completed within the same time slot, the merge continues according to the above rules until the merge is complete. It's easy to understand that at the end of the merge phase, k surviving nodes are merged into the same group.

[0082] Thus, any surviving node can receive node repair data transmitted from other surviving nodes, thereby forming a node repair block with its own node repair data.

[0083] 604. Transmit the node repair block from any group to the faulty node to repair the faulty node so that the faulty node can complete the fault repair according to the node repair block.

[0084] After the merging phase is completed, the transmission phase can begin. In this phase, the node repair blocks from any group are transmitted to the faulty node to repair it, thus enabling the faulty node to complete the fault repair based on the node repair blocks.

[0085] In one specific embodiment, if the number of faulty nodes is m, then the first m nodes in any group can send node repair blocks to the m faulty nodes, thereby enabling the faulty nodes to complete the repair. It should be noted that the node repair blocks are sent in a one-to-one correspondence; specifically, the node repair block corresponding to the first surviving node is sent to the first faulty node, the node repair block corresponding to the second surviving node is sent to the second faulty node, and so on. Further details are omitted here.

[0086] The node repair method disclosed in this embodiment first obtains all data nodes in the distributed storage system. These data nodes include faulty nodes and surviving nodes, with surviving nodes being all data nodes excluding the faulty nodes. The surviving nodes are then grouped and scheduled, with each group consisting of pairs of surviving nodes. Next, the surviving nodes in the same group are merged to transmit the node repair data from the current surviving node to other surviving nodes in the same group, forming a node repair block. This node repair block is composed of the node repair data from other surviving nodes and the node repair data from the current surviving node. Finally, the node repair block from any group is transmitted to the faulty node for repair, allowing the faulty node to complete the fault repair based on the node repair block. Therefore, in a distributed storage system, faulty nodes can be effectively repaired using surviving nodes. Furthermore, grouping and transmitting surviving nodes reduces network transmission overhead during faulty node repair, thereby accelerating the repair speed and optimizing the multi-node repair process in a distributed storage system.

[0087] For a detailed description of a node repair method disclosed in this application, please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a flowchart illustrating another node repair method disclosed in an embodiment of this application. It includes steps 701-709.

[0088] 701. Obtain all data nodes in the distributed storage system.

[0089] In this embodiment, step 701 is the same as the aforementioned Figure 6 Step 601 is similar, and will not be elaborated here.

[0090] 702. Divide the k surviving nodes into The first surviving node repair data is transferred to the second surviving node.

[0091] Once all data nodes are acquired, the surviving nodes can be paired up and transferred to each other, transmitting the original data blocks. Specifically, the k surviving nodes are divided into... There are several groups, each with two surviving nodes. If at least one group exists, the surviving nodes in the latter half of each group transmit data to the surviving nodes in the former half to repair the data. It's easy to understand that k is a positive integer, and the first surviving node is the one in the latter half of the same group. The second surviving node is the one in the first half of the same group.

[0092] In one specific embodiment, for a detailed description of the merging and transmission stages in this embodiment, please refer to [link to relevant documentation]. Figure 8 , Figure 8This is a schematic diagram illustrating multi-node repair of an RS(6,3) code according to an embodiment of this application. Where k = 6, m = 3. The number of surviving nodes is 6, with N3 to N8 being surviving nodes and N0 to N2 being faulty nodes. For ease of understanding and description, these will not be elaborated further. Figure 8 As can be seen, the surviving nodes are divided into three groups: N3 and N4 are in one group, N5 and N6 are in another, and N7 and N8 are in yet another. In the first time slot, N4 transmits node repair data D4 to N3, N6 transmits node repair data D6 to N5, and N8 transmits node repair data D8 to N7. It's important to understand that in this step, the first surviving node can be understood as N4, N6, or N8, and the corresponding second surviving node is N3, N5, or N7. The repair data for the first surviving node corresponds to D4, D6, or D8.

[0093] 703. If half the number of surviving nodes in the same group is less than the number of faulty nodes, the node repair data of the surviving nodes in the first half of the current group shall be transferred to the surviving nodes in the second half.

[0094] When the number of surviving nodes in the same group is generally less than the number of faulty nodes, the node repair data of the surviving nodes in the first half of the current group is transmitted to the surviving nodes in the second half. It should be noted that this step is performed based on the determination of the relative numbers of faulty and surviving nodes.

[0095] In one specific embodiment, see [reference] Figure 8 Since there are 2 surviving nodes in the same group, half of them have a load factor of 1, while the number of failed nodes is 3, thus requiring this step. That is, the surviving nodes in the first half transmit node repair data to the surviving nodes in the second half. Specifically, the second surviving nodes N3, N5, or N7 transmit their corresponding node repair data D3, D5, or D7 to the first surviving nodes N4, N6, or N8. It's easy to understand that since the data transmission between N3 and N8 is between surviving nodes, the load factor for any surviving node is 1, allowing the process to be completed in the same time slot.

[0096] It should also be noted that the node repair data D3 to D8 in steps 702-703 of this embodiment can also be understood as the original data blocks.

[0097] 704. Perform a merge operation on each group to obtain... Groups.

[0098] Based on step 702 above, Each group is grouped into pairs. There are 4 groups, each with 4 surviving nodes. If there are surviving nodes that cannot form a group, proceed to step 705. If the surviving nodes can form a group, proceed to step 706.

[0099] In one specific embodiment, see [reference] Figure 8 Correspondingly, N3 to N6 can be grouped together, while N7 and N8 cannot be grouped together because the number of surviving nodes is 2. Therefore, N7 and N8 proceed to step 705, and the surviving nodes from N3 to N6 proceed to step 706.

[0100] 705. If there exists that is not located in any The surviving nodes in each group are determined not to be located in any The surviving nodes in each group are the remaining surviving nodes, and all remaining surviving nodes are merged into the remaining group.

[0101] When there exists that is not located in any If a node survives in a group, it is determined that it is not located in any group. The surviving nodes in each group are the remaining surviving nodes. Alternatively, all remaining surviving nodes can be merged into the remaining groups. In one embodiment, the remaining surviving nodes may not be merged into the remaining groups; details will not be elaborated here.

[0102] In one specific embodiment, see [reference] Figure 8 ,exist Figure 8 In this context, N7 and N8 are the remaining surviving nodes. The corresponding N3 to N6 are the surviving nodes in a group, which will not be elaborated here.

[0103] 706. Obtain the repair data of the third surviving node and the repair data of the fourth surviving node, and determine the third weighting coefficient corresponding to the third surviving node and the fourth weighting coefficient corresponding to the fourth surviving node.

[0104] After dividing the surviving nodes into groups of four, the surviving nodes in each group are defined as the third, fourth, fifth, and sixth surviving nodes. The third and fourth surviving nodes are those located in the latter half of the group, while the fifth and sixth surviving nodes are those located in the first half. Specifically, the repair data for the third and fourth surviving nodes is first obtained, and the third weighting coefficient for the third surviving node and the fourth weighting coefficient for the fourth surviving node are determined.

[0105] In one specific embodiment, see [reference] Figure 8Where N5 and N6 correspond to the third and fourth surviving nodes respectively, and N3 and N4 correspond to the fifth and sixth surviving nodes respectively, for ease of description, these will not be elaborated further, and will be explained using specific node numbers. Therefore, the data block for repairing data for the third surviving node is D5, and the data block for repairing data for the fourth surviving node is D6. Since the merging rule is that the surviving nodes in the latter half of each group transmit data to the surviving nodes in the former half, it is necessary to determine the third and fourth weighting coefficients corresponding to N5 and N6. Specifically, the third weighting coefficient is a. 05 and a 06 The fourth weighting coefficient is a 15 and a 16 .

[0106] 707. Determine the repair data of the fifth survivor node corresponding to the fifth survivor node based on the repair data of the third survivor node, the repair data of the fourth survivor node, and the third weighting coefficient, and determine the repair data of the sixth survivor node corresponding to the sixth survivor node based on the repair data of the third survivor node, the repair data of the fourth survivor node, and the fourth weighting coefficient, and transmit the repair data of the fifth survivor node to the fifth survivor node, and transmit the repair data of the sixth survivor node to the sixth survivor node.

[0107] Based on the above embodiments, the repair data for the fifth surviving node can be determined according to the repair data for the third surviving node, the repair data for the fourth surviving node, and the node repair data with the third weighting coefficient, wherein the repair data for the fifth surviving node corresponds to the fifth surviving node. Correspondingly, the repair data for the sixth surviving node can also be determined according to the repair data for the third surviving node, the repair data for the fourth surviving node, and the node repair data with the fourth weighting coefficient, wherein the repair data for the sixth surviving node corresponds to the sixth surviving node.

[0108] In one specific embodiment, see [reference] Figure 8 Specifically, the data corresponding to the fifth surviving node (N3) is a. 05 D5+a 06 D6, corresponding to the data of the sixth surviving node (N4) is a. 15 D5+a 16 D6.

[0109] Based on the above embodiments, in another specific embodiment, since the number of faulty nodes is greater than half the number of surviving nodes in the group (i.e., the number of faulty nodes is 3, and half the number of surviving nodes in the group is 2), the surviving nodes in the first half of the group need to transmit additional data blocks to the surviving nodes in the second half. For details, please refer to... Figure 8 Among them, N3 can transfer the data blocks corresponding to D3 and D4 to N5. That is, a 23D3+a 24 D4. Where, a 23 and a 24 These are all weighting coefficients; the specific values ​​of these weighting coefficients will not be elaborated here. It is easy to understand that in this step, in the second time slot (Timeslot 2), a... 05 D5+a 06 D6, a 15 D5+a 16 D6, a 23 D3+a 24 D4 can transmit simultaneously because each surviving node can upload and download at the same time. Since D7 and D8 meet the conditions for the transmission phase, they can proceed directly to transmission.

[0110] For specific transmission methods, please refer to [link / reference]. Figure 4 or Figure 5 The description.

[0111] 708. Merge any surviving nodes after the data has been repaired by the transmission nodes to generate a node repair block.

[0112] The surviving nodes that have transmitted node repair data are merged to generate a node repair block.

[0113] In one specific embodiment, based on step 707, and in combination with Figure 8 It can be seen that since the surviving nodes N3 to N6 are in the same group, the data of the node repair block corresponding to the surviving node N3 is... For the surviving node N4, its node repair block data is as follows: For the surviving node of N5, the data of its node repair block is as follows: It's not hard to understand that the 'a' in it... 0i a 1i or a 2i All of these are weighting coefficients, and the steps above have already explained this in detail. The specifics will not be repeated here, and there are no restrictions on the specific values ​​of the weighting coefficients.

[0114] Based on the above embodiments, in another specific embodiment, based on step 705, since nodes N7 and N8 are remaining surviving nodes, the corresponding node repair block for N7 or N8 is a. 07 D7+a 08 D8, a 17 D7+a 18 D8 and a 27 D7+a 28D8. It's easy to understand that since the number of faulty nodes is 3, and the number of surviving nodes in the groups of N7 and N8 is 2, and repairing each data block requires any k surviving data blocks (e.g., repairing D0 requires D3 to D8), three data blocks are needed for repair. In this embodiment, a... 27 D7+a 28 The data block D8 serves as the node repair block for node N2. It's not difficult to understand that a... 07 a 17 a 27 a 08 a 18 and a 28 All are weighted coefficients, i.e., a 07 and a 08 Let a be the first residual weighting coefficient described above. 17 and a 18 Let a be the second residual weighting coefficient described above. 27 and a 28 The third residual weighting coefficient described above is used, N7 ​​is the first residual surviving node, and N8 is the second residual surviving node. Further details are omitted here, and no restrictions are placed on the specific values ​​of the weighting coefficients.

[0115] It should be noted that the specific calculation method for the weighting coefficients can be found in [reference needed]. Figure 1 The specific description is not elaborated here.

[0116] 709. Sort the surviving nodes in any group. When the number of faulty nodes is m, transfer the node repair blocks corresponding to the top m surviving nodes in any group to the faulty nodes respectively, and repair the m faulty nodes.

[0117] Based on the above steps, when the number of faulty nodes is m, the node repair blocks corresponding to the top m surviving nodes in any group can be transmitted to the faulty nodes respectively, thereby repairing the m faulty nodes. The surviving nodes in any group can be sorted. It is easy to understand that in this embodiment, if the sorting of the relevant surviving nodes has already been determined above, the sorting action may not be performed in this step.

[0118] In one specific embodiment, see [reference] Figure 8 Since m=3, it is necessary to transmit the node repair blocks corresponding to nodes N3 to N5 in the same group to the faulty node. Specifically, based on step 708, the data of the node repair block corresponding to node N3 is as follows: The data in the node repair block corresponding to node N4 is: The data in the node repair block corresponding to node N5 is:

[0119] Therefore, the node repair block corresponding to node N3 is transmitted to the faulty node N0 to repair the data block D0; the node repair block corresponding to node N4 is transmitted to the faulty node N1 to repair the data block D1; and the node repair block corresponding to node N5 is transmitted to the faulty node N2 to repair the data block D2. It is easy to understand that since nodes N0 to N2 have not received any other data blocks (i.e., their load is 0), the relevant transmission steps can be executed synchronously in the third time slot (Timeslot 3), that is, the relevant node repair blocks are transmitted to nodes N0 to N2 respectively.

[0120] Since the surviving nodes N7 and N8 in the second and third time slots (i.e., Timeslot2 and Timeslot3) have not changed, nodes N7 and N8 send a during Timeslot2. 07 D7+a 08 D8 and a 17 D7+a 18 D8 did not change in Timeslot 3 because the download bandwidth of nodes N0, N1, and N2 was already full (as described above). Therefore, it could only wait for the next timeslot to send, that is, in the fourth timeslot (Timeslot 4), the relevant data block a was sent by node N7. 27 D7+a 28 D8 to N2.

[0121] In summary, this embodiment proposes a node repair method that optimizes multi-node repair scheduling in a general distributed erasure coding storage system, thereby solving the network transmission overhead problem caused by repairing multiple nodes in a distributed erasure coding storage system and accelerating the node repair speed. In other embodiments, the solution of this application can also be applied to several popular coding schemes such as XOR (Exclusive OR) codes or Reed-solomon codes, and no specific limitation is made here.

[0122] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0123] If the plan involves sensitive information (such as user information or corporate information), it should state that the collection, use, and processing of sensitive information must comply with the laws, regulations, and standards of the relevant countries and regions, and must be carried out with the permission or consent of the relevant entities (such as users or enterprises).

[0124] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a node repair system disclosed in an embodiment of this application.

[0125] Acquisition unit 901 is used to acquire all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and surviving nodes are all data nodes excluding faulty nodes;

[0126] Grouping unit 902 is used to schedule surviving nodes into groups, wherein surviving nodes in any group exist in pairs;

[0127] The merging unit 903 is used to merge surviving nodes in the same group to transfer the node repair data of the current surviving node to other surviving nodes in the same group to form a node repair block; wherein, the node repair block is formed by the node repair data of other surviving nodes and the node repair data of the current surviving node.

[0128] The transmission unit 904 is used to transmit the node repair block in any group to the faulty node to repair the faulty node so that the faulty node can complete the fault repair according to the node repair block.

[0129] For example, the system includes:

[0130] Grouping unit 902 is specifically used to divide the k surviving nodes into groups. Groups, in which any Each group contains two surviving nodes, where k is a positive integer;

[0131] Transmission unit 904 is specifically used to transmit the repair data of the first surviving node to the second surviving node; wherein, the first surviving node is... Among the surviving nodes in the same group and located in the latter half, the second surviving node is... Surviving nodes in the same group and located in the first half of the group.

[0132] For example, the system further includes: a determining unit 905;

[0133] Merging unit 903 is also used to merge Perform a merge operation on each group to obtain... There are groups; each group contains four surviving nodes.

[0134] The acquisition unit 901 is further configured to acquire the repair data of the third surviving node and the repair data of the fourth surviving node, and determine the third weighting coefficient corresponding to the third surviving node and the fourth weighting coefficient corresponding to the fourth surviving node; wherein, the third surviving node and the fourth surviving node are respectively in Surviving nodes in the same group and located in the latter half of the group;

[0135] The determining unit 905 is further configured to determine the repair data of the fifth surviving node corresponding to the fifth surviving node based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the third weighting coefficient, and to determine the repair data of the sixth surviving node corresponding to the sixth surviving node based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the fourth weighting coefficient; wherein, the fifth surviving node and the sixth surviving node are respectively in Surviving nodes in the same group and located in the first half of the group;

[0136] The transmission unit 904 is also used to transmit the repair data of the fifth surviving node to the fifth surviving node and transmit the repair data of the sixth surviving node to the sixth surviving node.

[0137] For example, the system also includes:

[0138] The transmission unit 904 is also used to transmit the node repair data of the surviving nodes in the first half of the current group to the surviving nodes in the second half when the number of surviving nodes in the same group is less than half of the number of faulty nodes.

[0139] The merging unit 903 is specifically used to merge any surviving node after transmitting node repair data to generate a node repair block; wherein, the node repair block contains all the node repair data obtained by the current surviving node.

[0140] For example, the system further includes: a generation unit 906;

[0141] Determining unit 905, specifically used when there is a location not located in any When determining the surviving nodes in a group, it is determined that they are not located in any The surviving nodes in each group are the remaining surviving nodes, and all remaining surviving nodes are merged into the remaining group;

[0142] The determining unit 905 is also used to determine the node repair data of the remaining surviving nodes in the remaining group and the weighting coefficients corresponding to the node repair data;

[0143] The generation unit 906 is used to generate a node repair block corresponding to the remaining group based on the node repair data and weighting coefficients of the remaining surviving nodes.

[0144] For example, if the number of remaining surviving nodes in the remaining group is less than the number of faulty nodes, and the faulty nodes include a first faulty node, a second faulty node, and a third faulty node, and the remaining surviving nodes include a first remaining surviving node and a second remaining surviving node, the system further includes: a repair unit 907.

[0145] The determining unit 905 is specifically used to determine the first remaining node repair block based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the first remaining weighting coefficient; wherein, the first remaining weighting coefficient is the weighting coefficient corresponding to the first fault node;

[0146] The determining unit 905 is further configured to determine the second remaining node repair block based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the second remaining weighting coefficient; wherein the second remaining weighting coefficient is the weighting coefficient corresponding to the second fault node;

[0147] The determining unit 905 is further configured to determine the third remaining node repair block based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the third remaining weighting coefficient; wherein the third remaining weighting coefficient is the weighting coefficient corresponding to the third fault node;

[0148] Repair unit 907 is used to transmit the first remaining node repair block, the second remaining node repair block, or the third remaining node repair block to the first fault node, the second fault node, or the third fault node, respectively, so as to repair the first fault node, the second fault node, or the third fault node, respectively.

[0149] For example, the system further includes: a sorting unit 908;

[0150] The sorting unit 908 is used to sort the surviving nodes in any group;

[0151] The transmission unit 904 is specifically used to transmit the node repair blocks corresponding to the top m surviving nodes in any group to the faulty nodes when the number of faulty nodes is m, so as to repair the m faulty nodes; where m is a positive integer.

[0152] Please refer to the following: Figure 10 A schematic diagram of a node repair device disclosed in this application includes:

[0153] Central processing unit 1001, memory 1005, input / output interface 1004, wired or wireless network interface 1003, and power supply 1002;

[0154] Memory 1005 is either a short-term storage memory or a persistent storage memory;

[0155] The central processing unit 1001 is configured to communicate with the memory 1005 and execute instructions stored in the memory 1005 to perform the aforementioned operations. Figure 6 or Figure 7 The node repair method in the illustrated embodiment.

[0156] This application also provides a chip system, characterized in that the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is used to run computer programs or instructions to perform the aforementioned... Figure 6 or Figure 7 The node repair method in the illustrated embodiment.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A node repair method, characterized in that, The method, applied to a distributed storage system, includes: Obtain all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and the surviving nodes are the nodes among all data nodes excluding the faulty nodes; The surviving nodes are scheduled into groups, wherein the surviving nodes in any group exist in pairs; The surviving nodes in the same group are merged to transfer the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block; wherein, the node repair block is formed by the node repair data of the other surviving nodes and the node repair data of the current surviving node; The node repair block in any group is transmitted to the faulty node to repair the faulty node, so that the faulty node completes the fault repair according to the node repair block; The step of transmitting the node repair block from any group to the faulty node and repairing the faulty node includes: Sort the surviving nodes in any group; If the number of faulty nodes is m, the node repair blocks corresponding to the top m surviving nodes in any group are transmitted to the faulty nodes respectively, and the m faulty nodes are repaired; where m is a positive integer.

2. The node repair method according to claim 1, characterized in that, The step of scheduling and grouping the surviving nodes includes: Divide the k surviving nodes into Groups, in which any Each group includes two surviving nodes, where k is a positive integer; The step of merging the surviving nodes in the same group to transmit the node repair data from the current surviving node to other surviving nodes in the same group includes: The repair data of the first surviving node is transmitted to the second surviving node; wherein, the first surviving node is the one described above. The second surviving node is the surviving node in the second half of the same group, which is the surviving node in the group. Surviving nodes in the same group and located in the first half of the group.

3. The node repair method according to claim 2, characterized in that, After transmitting the node repair data of the first surviving node to the second surviving node, the method further includes: The Perform a merge operation on each group to obtain... Groups; wherein any group includes four of the surviving nodes; Obtain the repair data of the third surviving node and the fourth surviving node, and determine the third weighting coefficient corresponding to the third surviving node and the fourth weighting coefficient corresponding to the fourth surviving node; wherein, the third surviving node and the fourth surviving node are respectively located in the... Surviving nodes in the same group and located in the latter half of the group; The repair data of the fifth surviving node, based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the third weighting coefficient, is determined, and the repair data of the sixth surviving node, based on the repair data of the third surviving node, the repair data of the fourth surviving node, and the fourth weighting coefficient, is determined; wherein, the fifth surviving node and the sixth surviving node are respectively in the... Surviving nodes in the same group and located in the first half of the group; The repair data of the fifth surviving node is transmitted to the fifth surviving node, and the repair data of the sixth surviving node is transmitted to the sixth surviving node.

4. The node repair method according to claim 2 or 3, characterized in that, The method further includes: If half the number of surviving nodes in the same group is less than the number of faulty nodes, the node repair data of the surviving nodes in the first half of the current group is transmitted to the surviving nodes in the second half. The node repair block includes: The surviving nodes after transmitting the node repair data are merged to generate the node repair block; wherein, the node repair block contains all the node repair data obtained by the current surviving node.

5. The node repair method according to claim 3, characterized in that, The method further includes: If there exists that is not located in any The surviving nodes in each group are determined not to be located in any The surviving nodes in each group are the remaining surviving nodes, and all the remaining surviving nodes are merged into the remaining group; The node repair block includes: Determine the node repair data of the remaining surviving nodes in the remaining group and the weighting coefficients corresponding to the node repair data; Based on the node repair data and weighting coefficients of the remaining surviving nodes, a node repair block corresponding to the remaining group is generated.

6. The node repair method according to claim 5, characterized in that, If the number of remaining surviving nodes in the remaining group is less than the number of faulty nodes, and the faulty nodes include a first faulty node, a second faulty node, and a third faulty node, and the remaining surviving nodes include a first remaining surviving node and a second remaining surviving node, then generating a node repair block corresponding to the remaining group based on the node repair data and weighting coefficients of the remaining surviving nodes includes: The first remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the first remaining weighting coefficient; wherein, the first remaining weighting coefficient is the weighting coefficient corresponding to the first faulty node; The second remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the second remaining weighting coefficient; wherein, the second remaining weighting coefficient is the weighting coefficient corresponding to the second faulty node; The third remaining node repair block is determined based on the node repair data of the first remaining surviving node, the node repair data of the second remaining surviving node, and the third remaining weighting coefficient; wherein, the third remaining weighting coefficient is the weighting coefficient corresponding to the third faulty node; The step of transmitting the node repair block from any group to the faulty node and repairing the faulty node includes: The first remaining node repair block, the second remaining node repair block, or the third remaining node repair block are respectively transmitted to the first fault node, the second fault node, or the third fault node to repair the first fault node, the second fault node, or the third fault node respectively.

7. A node repair system, characterized in that, Applied to a distributed storage system, the system includes: An acquisition unit is used to acquire all data nodes in the distributed storage system; wherein, all data nodes include faulty nodes and surviving nodes, and the surviving nodes are the nodes among all data nodes excluding the faulty nodes; A grouping unit is used to schedule and group the surviving nodes, wherein the surviving nodes in any group exist in pairs; The merging unit is used to merge the surviving nodes in the same group to transfer the node repair data of the current surviving node to other surviving nodes in the same group, forming a node repair block; wherein, the node repair block is formed by the node repair data of the other surviving nodes and the node repair data of the current surviving node. A transmission unit is configured to transmit the node repair block in any group to the faulty node to repair the faulty node, so that the faulty node completes the fault repair according to the node repair block. The system also includes: a sorting unit; A sorting unit is used to sort the surviving nodes in any set. The transmission unit is specifically used to transmit the node repair blocks corresponding to the top m surviving nodes in any group to the faulty nodes when the number of faulty nodes is m, so as to repair the m faulty nodes; where m is a positive integer.

8. A node repair device, characterized in that, The device includes: Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the node repair method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the node repair method as described in any one of claims 1 to 6.

Images