A node repair method based on erasure code and related equipment

Abstract

The embodiment of the present application provides a node repair method based on erasure code and related equipment, which is used to solve the problems of slow node repair speed of erasure code and lack of scheduling framework. The method of the embodiment of the present application includes: obtaining the data to be repaired uploaded by the user; determining the strip to be repaired where the fault block to be repaired is located, and selecting the help block; determining the repair stripe, and selecting the node to be requested; constructing a repair directed acyclic graph, setting the vertices and repair edges of the repair directed acyclic graph; performing mapping on each vertex in the mapping set according to the dynamic weight mapping algorithm, and recording the help node and the node to be requested with the smallest load after mapping; all target help nodes determined in the repair directed acyclic graph are allocated based on the link resource allocation algorithm of slicing, and slicing the target help block according to the target repair link to obtain the slice help block, so as to complete the node repair of the fault block to be repaired according to the slice help block and the target repair link.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to a node repair method based on erasure codes and related equipment. Background Art

[0002] Reed-Solomon codes (RS codes) are a type of forward error correction channel code, also known as erasure codes. Erasure codes have been widely used in large-scale distributed storage systems to improve data reliability with relatively low data redundancy. However, in the event of a full node failure, erasure codes require significant network transmission to recover data. However, existing research mostly assumes that each node is homogeneous, ignoring the heterogeneity of real-world data centers. In heterogeneous data center network (DCN) environments, existing research lacks a universal, efficient scheduling framework. Some existing repair algorithms only consider the rack architecture, minimizing cross-rack traffic while treating the cross-rack network as a homogeneous network. Alternatively, they only consider the network environment between racks while ignoring the network environment within a rack. This results in extremely low data recovery throughput in heterogeneous DCN environments with varying network bandwidth, latency, and data transmission paths. Summary of the Invention

[0003] The embodiments of the present application provide a node repair method and related equipment based on erasure codes, which are used to solve the problems of slow erasure code node repair speed and lack of a scheduling framework.

[0004] A first aspect of an embodiment of the present application provides a node repair method based on erasure coding, comprising:

[0005] Obtain the data to be repaired uploaded by the user, and divide the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired; wherein the m check blocks are obtained by encoding the k data blocks, k and m are positive integers, and k is not less than m; the set of the k data blocks and the m check blocks corresponding to the data to be repaired is determined as a stripe, and each data block is distributed on k+m different storage nodes;

[0006] Determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine a repair stripe, and select a node to be requested from the repair stripe; wherein, the faulty block to be repaired is used to describe the data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are the k+m-1 data blocks remaining in the stripe to be repaired except the faulty block, the storage node corresponding to the helper block is the helper node, the repair stripe is the stripe that does not contain the faulty block, the node to be requested is used to describe the storage node that does not contain the data block corresponding to the stripe to be repaired, and the node to be requested is used to store the repaired faulty block to be repaired;

[0007] Construct a repair directed acyclic graph, and set k+1 vertices and repair edges of the repair directed acyclic graph; wherein the first k vertices are used to represent the k helper nodes of the strip to be repaired, the last vertex is used to represent the faulty block to be repaired and the corresponding node to be requested, and the repair edge is used to represent the mapping set between any vertex and the node to be requested;

[0008] Perform mapping on each vertex in the mapping set according to a dynamic weight mapping algorithm, record the helping node and the node to be requested with the smallest load after mapping, and determine the helping node as a target helping node and the node to be requested as a target requesting node;

[0009] The slice-based link resource allocation algorithm repairs the target help blocks and target repair links of the target help nodes determined in all directed acyclic graphs, and slices the target help blocks according to the target repair links to obtain slice help blocks, so as to complete the node repair of the fault block to be repaired according to the slice help blocks and the target repair links; wherein, the target repair link is a non-overlapping data transmission link between all target help nodes and all target request nodes.

[0010] A second aspect of an embodiment of the present application provides a node repair system based on erasure codes, including:

[0011] an acquisition unit, configured to acquire data to be repaired uploaded by a user, and divide the data to be repaired into k data blocks and m check blocks based on an erasure code of the data to be repaired; wherein the m check blocks are obtained by encoding the k data blocks, k and m are positive integers, and k is not less than m; a set of the k data blocks and the m check blocks corresponding to the data to be repaired is determined as a stripe, and each data block is distributed across k+m different storage nodes;

[0012] A determination unit is configured to determine a stripe to be repaired where a faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine a repair stripe, and select a node to be requested from the repair stripe; wherein the faulty block to be repaired is used to describe a data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are k+m-1 data blocks remaining in the stripe to be repaired except the faulty block, the storage node corresponding to the helper block is a helper node, the repair stripe is the stripe that does not contain the faulty block, the node to be requested is used to describe a storage node that does not contain a data block corresponding to the stripe to be repaired, and the node to be requested is used to store the repaired faulty block to be repaired;

[0013] A setting unit is configured to construct a repair directed acyclic graph and set k+1 vertices and repair edges of the repair directed acyclic graph; wherein the first k vertices are used to represent the k helper nodes of the strip to be repaired, the last vertex is used to represent the faulty block to be repaired and the corresponding node to be requested, and the repair edge is used to represent a mapping set between any vertex and the node to be requested;

[0014] a recording unit, configured to perform mapping on each vertex in the mapping set according to a dynamic weight mapping algorithm, record the helping node and the node to be requested with the smallest load after mapping, and determine the helping node as a target helping node and the node to be requested as a target requesting node;

[0015] The determining unit 1202 is also used to repair the target helping blocks and target repair links of the target helping nodes determined in all directed acyclic graphs based on a slice-based link resource allocation algorithm, and slice the target helping blocks according to the target repair links to obtain sliced ​​helping blocks, so as to complete the node repair of the fault block to be repaired according to the slice helping blocks and the target repair links; wherein, the target repair link is a non-overlapping data transmission link between all target helping nodes and all target request nodes.

[0016] The erasure code-based node repair system provided in the second aspect of the embodiment of the present application is used to execute the erasure code-based node repair method described in the first aspect.

[0017] A third aspect of an embodiment of the present application provides a node repair device based on erasure codes, including:

[0018] CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply;

[0019] The memory is a transient storage memory or a persistent storage memory;

[0020] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the node repair method based on erasure code described in the first aspect.

[0021] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the node repair method based on erasure code described in the first aspect.

[0022] A fifth aspect of an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer, the computer executes the node repair method based on erasure code described in the first aspect.

[0023] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: through a node repair method based on erasure code disclosed in the embodiments of the present application, first obtain the data to be repaired uploaded by the user, and divide the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired; then determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine the repair stripe, and select the node to be requested from the repair stripe; then construct a repair directed acyclic graph, set k+1 vertices and repair edges of the repair directed acyclic graph; secondly, according to the dynamic weight mapping algorithm, Each vertex performs mapping, records the helping node and the node to be requested with the smallest load after mapping, and determines the helping node as the target helping node, and the node to be requested as the target requesting node; finally, the link resource allocation algorithm based on slicing repairs the target helping blocks and target repair links of all target helping nodes determined in the directed acyclic graph, and slices the target helping blocks according to the target repair links to obtain sliced ​​helping blocks, so as to complete the node repair of the faulty block to be repaired based on the sliced ​​helping blocks and the target repair links; wherein, the target repair link is a non-overlapping data transmission link between all target helping nodes and all target requesting nodes. Therefore, by calculating the repair link with the smallest load, the slow full-node repair speed of erasure codes in heterogeneous environments can be effectively solved, and the repair time can be reduced as much as possible. At the same time, by selecting the helping blocks to repair the faulty blocks, a general scheduling framework is implemented, which improves the feasibility of the solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0025] Figure 1This is a flowchart of an existing method for repairing a full-node failure in an erasure code;

[0026] Figure 2 A flowchart of a node repair method based on erasure coding disclosed in an embodiment of the present application;

[0027] Figure 3 A flowchart of another node repair method based on erasure codes disclosed in an embodiment of the present application;

[0028] Figure 4 A flowchart of another node repair method based on erasure codes disclosed in an embodiment of the present application;

[0029] Figure 5 The encoding and repair process in RS(2,2) disclosed in an embodiment of the present application;

[0030] Figure 6 This is an example diagram of an erasure code cluster repair disclosed in an embodiment of the present application;

[0031] Figure 7 This is an example diagram of selecting a helper and a requester disclosed in an embodiment of the present application;

[0032] Figure 8 A schematic diagram of two different scheduling orders for repairing a link disclosed in an embodiment of the present application;

[0033] Figure 9 A schematic diagram of a vertex mapping process based on DWM disclosed in an embodiment of the present application;

[0034] Figure 10 A schematic diagram of a transmission scheduling based on SLRA disclosed in an embodiment of the present application;

[0035] Figure 11 This is an example flow chart of a full-node repair framework disclosed in an embodiment of the present application;

[0036] Figure 12 This is a schematic diagram of the structure of a node repair system based on erasure codes disclosed in an embodiment of the present application;

[0037] Figure 13 This is a structural diagram of a node repair device based on erasure codes disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] See also Figure 1 , Figure 1 This is a flowchart of an existing erasure code full node fault repair method. It is not difficult to understand that, Figure 1 include Figure 1 (a), Figure 1 (b) Figure 1 (c) and Figure 1 (d).

[0040] in, Figure 1 (a) is a partial parallel repair (PPR) algorithm. Specifically, the surviving nodes in a stripe merge the data blocks into pairs (reducing the download load on the repair node) and then transfer them to other surviving nodes (repair nodes). Figure 1 (a) Specific examples of the PPR algorithm repair process under the erasure code RS(8,3), where k = 8 and m = 3. k describes the number of data blocks in the erasure code, and m describes the number of parity blocks corresponding to each data block.

[0041] Figure 1 (b) is an ECPipe (Erasure-Coded Pipeline) algorithm. Specifically, each block is divided into smaller packets, which are transmitted in units of packets and sent according to a linear relationship, such as A->B->C->D, similar to the pipeline technology used by CPUs to execute instructions. Figure 1 (b),An example of the algorithm is the repair process with k=4, where R represents the repair node.

[0042] Figure 1 (c) and Figure 1 (d) is a method for implementing node repair using the Repairboost framework. Specifically, the Repairboost framework abstracts the repair of a single block into a directed acyclic graph, and then maps the vertices of the graph to the surviving nodes, while keeping the mapped nodes with the minimum upload and download load (the number of data blocks). The leaf vertices of the graph are then constructed into a larger graph containing the source and the end point, and the maximum flow algorithm is used to select the transmission order of the nodes. Figure 1(c) is an example of mapping graph vertices to nodes. The PPR algorithm is selected to repair the block of RS(4,2). N2 and N1 are the leaf vertices of the graph after mapping. The numbers in red (and green) represent the upload and download load provided by the nodes before (and after) mapping. Figure 1 (d) shows an example of the maximum flow algorithm (k=4). The red arrow indicates the established maximum flow.

[0043] As can be seen from the above description, most existing research assumes that every node is homogeneous, ignoring the heterogeneity of real data centers. In heterogeneous DCN environments, existing research has yet to develop a universal, efficient scheduling framework. Some existing repair algorithms only consider the rack architecture, minimizing cross-rack traffic while treating cross-rack networks as homogeneous. Alternatively, they only consider the network environment between racks while ignoring the network environment within a rack.

[0044] Therefore, the embodiment of the present application designs a full-node repair framework for accelerating erasure codes in heterogeneous data center environments. Figures 2 to 4 Description of the illustrated embodiment.

[0045] See also Figure 2 , Figure 2 This is a flowchart of a node repair method based on erasure coding disclosed in an embodiment of the present application, including steps 201 to 205.

[0046] 201. Obtain the data to be repaired uploaded by the user, and divide the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired.

[0047] To repair data, users upload the data to be repaired. This data is then divided into k data blocks and m check blocks based on the erasure code configured for the data to be repaired. As you can see, m check blocks are encoded from k data blocks, where k and m are positive integers, and k is no less than m. The set of k data blocks and m check blocks corresponding to the data to be repaired is defined as a stripe, with each data block distributed across k+m different storage nodes.

[0048] In one specific embodiment, a user uploads a file to be repaired to a full-node repair framework in a heterogeneous DCN environment, i.e., the framework specifically described in this embodiment. The file data that currently needs to be repaired is the data to be repaired. In this embodiment, the data to be repaired can be divided into k data blocks {D0, D1, ..., D k-1}, at this time, the size of each data block is the ratio of the data size of the data to be repaired to the number of data blocks k. Correspondingly, the check block can be obtained by multiplying the data block and the coding coefficient, where the coding system can be calculated by a fixed formula. Thus, k data blocks are encoded into m check blocks. It is not difficult to understand that at this time k ≥ m. At the same time, it should also be understood that the collection of these k+m data blocks (check blocks can also be understood as a type of data block) is called a stripe, and each data block is distributed on k+m different storage nodes to ensure node-level reliability. For example, see Figure 5 , Figure 5 This is an encoding and repair process in RS(2,2) disclosed in an embodiment of the present application. Figure 5 (a) is the encoding process of RS(2,2), Figure 5 (b) is the repair process of RS(2,2). Figure 5 The encoding and repair operations of RS(2,2) codes are shown. For example, if D j (0≤j<k) (assuming D0 is faulty), it can be repaired from any k available blocks.

[0049] 202. Determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine the repair stripe, and select a node to be requested from the repair stripe.

[0050] Based on the stripe definition method in step 201, the stripe to be repaired where the faulty block to be repaired is located can be determined, and k helper blocks can be selected from the stripe to be repaired. A repair stripe can also be determined, and a node to be requested can be selected from the repair stripe. It is not difficult to understand that the faulty block to be repaired is used to describe the data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are the k+m-1 data blocks remaining in the stripe to be repaired except for the faulty block, the storage node corresponding to the helper block is the helper node, the repair stripe is the stripe that does not contain the faulty block, the node to be requested is used to describe the storage node that does not contain the data block corresponding to the stripe to be repaired, and the node to be requested is used to store the repaired faulty block to be repaired.

[0051] In one specific embodiment, the typical repair process of RS(k,m) code is as follows. First, k data blocks (called helper nodes / blocks) are selected from the remaining k+m-1 data blocks of the stripe where the faulty block to be repaired is located. Then, a node (called request / repair node) is selected from other surviving nodes that do not contain the stripe where the faulty block is located (i.e., the repair stripe). For ease of understanding, it is collectively referred to as the request node to store the repaired data block, i.e., the requested node described above. For a more intuitive understanding of the above description, please refer to Figure 6 , Figure 6This is an example diagram of an erasure code cluster repair disclosed in an embodiment of the present application. Figure 6 A cluster of 7 nodes with erasure codes is shown. The cluster contains three stripes of 12 blocks encoded by RS(2,2), where blocks of the same color belong to the same stripe. To repair a block in stripe 1, any two surviving blocks in the stripe (e.g., {D0,P0}, {D0,P1}, {P0,P1}) can be selected as helper nodes, where {P0,P1} is selected. Then, a node that does not contain the stripe is selected (e.g., N0 / N1 / N2, representing the i-th node in the cluster). Here, N6 is selected as the requesting node. It should be added that Figure 6 This cluster has seven nodes (servers). RS(2,2) encoding is deployed on the server. Typically, the k+m data blocks within a stripe are placed on different nodes. This ensures data recovery even if any m nodes fail. Each stripe color corresponds to an RS(2,2). Stripes are unrelated.

[0052] It should be noted that, in another feasible technical solution, the PPR algorithm can be used to repair block D0. First, node N5 aggregates blocks P0 and P1, and then sends the aggregated block to node N6 to complete the repair.

[0053] 203. Construct a repair directed acyclic graph and set the k+1 vertices and repair edges of the repair directed acyclic graph.

[0054] After defining the faulty and helper blocks, as well as their corresponding requesting and helping nodes, we can construct a full-node heterogeneous repair framework (HRF) for erasure codes in an accelerated heterogeneous DCN. The repair process for a single block is first abstracted as a repair directed acyclic graph (RDAG). This is denoted by G = (V, E), where V is the vertex set and E is the edge set. To facilitate further discussion, we define some terms as follows:

[0055] Parent / Child Vertices: For an edge (u, v) in an RDAG, u is called the parent vertex because it is the starting point of the directed edge. Similarly, v is called the child vertex. Furthermore, (u, v) represents vertex u pointing to vertex v. The first vertex is the parent, and the second vertex is the child.

[0056] In-degree / out-degree: The in-degree / out-degree of vertex v, denoted as ID(v) / OD(v) respectively, refers to the sum of the weights of all edges with vertex v as the end / start point. Mathematically defined as: ID(v) = |{e∈E,|e=(u,v)}, OD(v) = |{e∈E,|e=(v,u)}, where u is any vertex adjacent to v. e refers to the directed edge in the edge set pointed to by vertex v or u. e=(u,v): Vertex u points to vertex v, the parent vertex is u, and the child vertex is v. e=(v,u): Vertex v points to vertex u, the parent vertex is v, and the child vertex is u. Because it is a directed acyclic graph, it is impossible for both edges (u,v) and (v,u) to exist at the same time. If both edges exist at the same time, the graph is a cyclic graph.

[0057] Leaf Vertex: In RDAG, a leaf vertex is a vertex with no outgoing edges, meaning it has no child vertices. It is a vertex with an in-degree of 0 (i.e., no other vertex points to it).

[0058] Intermediate Vertex: In RDAG, an intermediate vertex is a vertex that has both a parent vertex and a child vertex. It is a vertex with both out-degree and in-degree non-zero.

[0059] For an RS(k,m) code, set k+1 vertices {v0,v1,…,v k-1 ,v k}, where the first k vertices {v0,v1,…,v k-1} represents the k surviving blocks (helper nodes) of the stripe, and the last vertex {v k} represents a failed block (requesting node). For multiple block failures, multiple RDAGs can be constructed, each representing the failed block in the stripe, with the remaining vertices representing the surviving blocks in the stripe. Edges are then constructed using different repair algorithms (such as traditional repair algorithms like CR, PPR, or ECPipe). An edge (u, v) in the graph indicates that all requested surviving blocks must be collected from the parent vertex of vertex u, that is, from the set of mappings between any vertex and the requested node. For ease of description, the above definition will not be further described.

[0060] 204. According to the dynamic weight mapping algorithm, mapping is performed on each vertex in the mapping set, and the helping node and the to-be-requested node with the smallest load after mapping are recorded. The helping node is determined as the target helping node, and the to-be-requested node is determined as the target requesting node.

[0061] After defining the mapping set between vertices and edges, mapping can be performed on each vertex in the mapping set according to the dynamic weight mapping algorithm, thereby recording the helping node and the node to be requested with the smallest load after mapping, and determining the helping node as the target helping node and the node to be requested as the target request node.

[0062] In one of the specific embodiments, the selection of helping nodes and requesting nodes in a strip is transformed into a vertex-to-node mapping problem in RDAG. After constructing the RDAG, HRF allocates repair tasks by mapping vertices to nodes to minimize the upload and download of the entire system. By proposing a dynamic weight mapping algorithm (DWM) based on a greedy approach, it can pursue a solution close to the optimal solution and return the solution within a limited time. Specifically, according to the DWM algorithm, let V2N(i,j) be a set of mappings from vertices to nodes, and V2N(i,j) is used to represent the vertex v of the i-th graph (RDAG graph). j Then, through the mapping calculation, we can record the helper node and the node to be requested with the smallest load after mapping (i.e., the load between the vertex and the node). Thus, the helper node with the smallest load is determined as the target helper node, and the node to be requested is the target request node.

[0063] Furthermore, the load between different vertices and nodes in the mapping set can be updated by looping the mapping step, so as to find the target help node and the target request node more accurately. Figure 4 The embodiment shown is described.

[0064] 205. The link resource allocation algorithm based on slicing repairs all target help blocks and target repair links of the target help nodes determined in the directed acyclic graph, and slices the target help blocks according to the target repair links to obtain slice help blocks, so as to complete the node repair of the fault block to be repaired according to the slice help blocks and the target repair links.

[0065] Based on step 204, the target helper blocks and target repair links of all target helper nodes identified in the repair directed acyclic graph (RDAG) can be allocated using a slice-based link resource allocation algorithm. The target helper blocks are then sliced ​​according to the target repair links to obtain sliced ​​helper blocks. This allows node repair of the faulty block to be repaired to be completed based on the sliced ​​helper blocks and target repair links. It is understood that the target repair link is a non-overlapping data transmission link between all target helper nodes and all target requesting nodes.

[0066] In one specific embodiment, the DWM algorithm selects a near-optimal helper and requester to reduce the node's maximum load. However, randomly arranging the node's transmission order can extend the repair time. The key challenge here is determining the node's transmission order. To fully utilize the node's network resources, a slicing-based link resource allocation algorithm (SLRA) is proposed. The data blocks corresponding to each link are sliced ​​according to the allocated transmission time. Specifically, the target helping block and target repair link corresponding to the target helping node (i.e., the transmission link between the helping node and the requesting node) are first determined. Then, the times of all links associated with leaf vertices in the RDAG are weighted and initialized. Because there are dependencies between vertices in the RDAG, SLRA sorts all links corresponding to leaf vertices in descending order of weight. It then adds links that do not compete (i.e., two links with different sending and receiving nodes) to the set of links to be sent. Finally, SLRA uses the minimum time in the set of links to be sent as a benchmark and assigns the minimum time to all other links to be sent. The data block is sliced ​​based on the time allocated to the link, resulting in sliced ​​helper blocks. For example, if the original transmission time between two nodes is 3 seconds and 2 seconds is allocated, the data block is sliced ​​into 2 / 3 of the original time, and the remaining portion waits for the next transmission. Thus, by continuously repeating this replenishment, the node repair of the faulty block to be repaired can be completed based on the cyclic slice helper blocks and the corresponding target repair link.

[0067] A node repair method based on erasure code disclosed in an embodiment of the present application first obtains the data to be repaired uploaded by the user, and divides the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired; then, the stripe to be repaired where the faulty block to be repaired is located is determined, and k helper blocks are selected from the stripe to be repaired; the repair stripe is determined, and the node to be requested is selected from the repair stripe; then, a repair directed acyclic graph is constructed, and k+1 vertices and repair edges of the repair directed acyclic graph are set; secondly, according to the dynamic weight mapping algorithm, mapping is performed on each vertex in the mapping set, and the mapping is recorded. Then, the helping node and the node to be requested with the smallest load are selected, and the helping node is determined as the target helping node, and the node to be requested is determined as the target requesting node; finally, the link resource allocation algorithm based on slicing is used to repair the target helping blocks and target repair links of the target helping nodes determined in the directed acyclic graph, and the target helping blocks are sliced ​​according to the target repair links to obtain sliced ​​helping blocks, so as to complete the node repair of the fault block to be repaired according to the sliced ​​helping blocks and the target repair links; wherein, the target repair link is a non-overlapping data transmission link between all target helping nodes and all target requesting nodes. Therefore, by calculating the repair link with the smallest load, the slow speed of full-node repair of erasure codes in heterogeneous environments can be effectively solved, and the repair time can be reduced as much as possible. At the same time, by selecting the helping blocks to repair the faulty blocks, a general scheduling framework is realized, which improves the feasibility of the solution.

[0068] Before determining the optimal helping and requesting nodes, it is necessary to determine the link transmission time between each node. For easier understanding, please refer to Figure 3 , Figure 3 This is a flowchart of another node repair method based on erasure coding disclosed in an embodiment of the present application, including steps 301 to 307.

[0069] 301. Divide the data to be repaired R into k data blocks, obtain the size corresponding to each data block, and encode the k data blocks into m check blocks according to the erasure code based on the coding equation.

[0070] In this embodiment, step 301 is Figure 2 The step 201 in the embodiment shown is similar. Specifically, the data to be repaired R is divided into k data blocks, and the size corresponding to each data block is obtained. Based on the coding equation, k data blocks are encoded into m check blocks according to the erasure code. It is not difficult to understand that the coding equation is D j is the jth data block, P i is the i-th check block, c ij Represents data block D j corresponds to the check block P i The coding coefficient of .

[0071] In one specific embodiment, the default data block size in Hadoop 1.x is 64MB, and the default data block size in Hadoop 2.x is 128MB. ij Can represent data block D j In the check block P i The coefficient in the calculation. In addition, k represents that a data of size R is divided into k data blocks, and each data block is If the number is not divisible, the last data block is padded with "0". Therefore, k ≥ m must be satisfied. Also, because k ≥ m, the system can tolerate up to m data block failures. Therefore, if k data blocks fail, the data is lost and cannot be recovered.

[0072] 302. Determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine the repair stripe, and select a node to be requested from the repair stripe.

[0073] In this embodiment, step 302 is Figure 2 Step 202 in the illustrated embodiment is similar and will not be described in detail here.

[0074] 303. Select two helping blocks in the stripe to be repaired, determine helping nodes corresponding to the two helping blocks, define a two-dimensional matrix, and determine the link bandwidth between the two helping nodes according to the two-dimensional matrix.

[0075] Before constructing the RDAG, it is noted that the selection of helper nodes and requester nodes determines the repair traffic of the system. However, most existing studies ignore heterogeneous links, and therefore cannot balance the upload and download repair traffic well in full-node repair. Therefore, two helper blocks can be selected in the strip to be repaired, and the helper nodes corresponding to the two helper blocks can be determined. A two-dimensional matrix b(i, j) is defined, and the link bandwidth between the two helper nodes is determined according to the two-dimensional matrix b(i, j). It is not difficult to understand that the two-dimensional matrix b(i, j) is used to represent N i With N j The link bandwidth between i Indicates the help node corresponding to the i-th help block, N j Indicates the help node corresponding to the j-th help block.

[0076] In one specific embodiment, among the helper nodes, any two helper nodes are selected, N i and N j Then, define b(i,j) as a two-dimensional matrix, where b(i,j) represents N i and N j The link bandwidth between them.

[0077] 304. Determine a two-dimensional weight matrix corresponding to the two-dimensional matrix, and determine a link transmission time between two helping nodes according to the two-dimensional weight matrix.

[0078] Then, based on step 303, a two-dimensional weight matrix corresponding to the two-dimensional matrix can be determined, and the link transmission time between the two helping nodes can be determined based on the two-dimensional weight matrix. It is not difficult to understand that the calculation method of w(i,j) is w(i,j) is used to represent the helping node N i With help node N j The transmission time of the link between them.

[0079] In one specific embodiment, a two-dimensional weight matrix w(i, j) is used to represent the transmission time (the inverse of the bandwidth) of different links, then As shown in Table 1, from node N i to N j The transmission time is w(i,j). For example, see Table 1,

[0080] Time <![CDATA[N0]]> <![CDATA[N1]]> <![CDATA[N2]]> <![CDATA[N3]]> <![CDATA[N4]]> <![CDATA[N5]]> <![CDATA[N6]]> <![CDATA[N0]]> - 1 4 2 1 2 3 <![CDATA[N1]]> 1 - 2 1 2 4 2 <![CDATA[N2]]> 4 2 - 3 3 5 3 <![CDATA[N3]]> 2 1 3 - 2 4 1 <![CDATA[N4]]> 1 2 3 2 - 5 3 <![CDATA[N5]]> 2 4 5 3 5 - 1 <![CDATA[N6]]> 3 2 3 1 3 1 -

[0081] In Table 1, w(0,2)=4, which means that node N0 needs 4 seconds of network transmission time to send a data block to node N2.

[0082] Therefore, for the convenience of understanding and the application of the subsequent DWM algorithm or SLRA algorithm, please refer to

[0083] Figure 7 and Figure 8 . Figure 7 This is an example diagram of selecting a helper and a requester disclosed in an embodiment of the present application;

[0084] Figure 8 This is a schematic diagram of two different scheduling orders for repairing a link disclosed in an embodiment of the present application.

[0085] Figure 7 Two examples of repairing three blocks using the CR algorithm (all faulty blocks are transmitted to the repairing node) are shown. In the first example ( Figure 7 a, randomly select helpers and requesters), helpers and requesters are randomly selected. Because w(6,4) plus w(5,4) equals 8 in Table 1, it takes 8s (N4) to complete the repair. In the second example ( Figure 7 b, Select the best helper and requester).,Based on time, the best helper and requester are selected,,and only 4s (N3) is needed to complete the repair.,Thus, reasonable selection of helpers and requesters can reduce,transmission time.

[0086] For example, the traditional method uses the default ordered list and adopts the polling mode for transmission scheduling. We noticed that this may increase the repair time. We first define a tuple (N i ,N j ) to represent the slave node N i N j Data transmission to the node. A multi-strip repair solution corresponds to an ordered list of tuples (N x ,N y ).

[0087] Figure 8 Shown Figure 7 Two different scheduling orders for repairing links in (b). Figure 8 (a) shows the execution of six repair links in a random order {(N1,N6),(N3,N6),(N5,N3),(N6,N3),(N1,N4),(N3,N4)}. The entire transmission phase lasts for 6 seconds. We found that when two tuples (N i1 ,N j1 ) and (N i2 ,N j2 ) i1 Equal to N i2 , or N j1 Equal to N j2 For example, (N3, N6) and (N3, N4). When contention occurs, if we can execute the slower link (the one that takes longer) in advance, the repair time can be reduced. Figure 8 (b) shows an optimal transmission order, which transmits (N3,N4) before (N3,N6), with a total repair time of 4 seconds.

[0088] It is not difficult to understand that the above is one of the feasible technical solutions for repairing faulty nodes. For other feasible technical solutions, please refer to Figure 4 The embodiment shown.

[0089] 305. Construct a repair directed acyclic graph and set k+1 vertices and repair edges of the repair directed acyclic graph.

[0090] 306. According to the dynamic weight mapping algorithm, perform mapping on each vertex in the mapping set, record the helping node and the to-be-requested node with the smallest load after mapping, and determine the helping node as the target helping node and the to-be-requested node as the target requesting node.

[0091] 307. The link resource allocation algorithm based on slicing repairs all target helper blocks and target repair links of the target helper nodes determined in the directed acyclic graph, and slices the target helper blocks according to the target repair links to obtain slice helper blocks, so as to complete the node repair of the fault block to be repaired according to the slice helper blocks and the target repair links.

[0092] In this embodiment, steps 305 to 307 are Figure 2 Steps 203 to 205 in the embodiment shown are similar and will not be described in detail here. Figure 4 A detailed description of the illustrated embodiments.

[0093] The node repair method based on erasure code disclosed in the embodiment of the present application effectively solves the problem that heterogeneous links cannot balance upload and download repair traffic in full-node repair, thereby improving the feasibility of the solution.

[0094] for Figure 2 For details on how to implement steps 203 to 205, please refer to Figure 4 , Figure 4 This is a flowchart of another node repair method based on erasure coding disclosed in an embodiment of the present application, including steps 401 to 412.

[0095] 401. For any fault block to be repaired, construct a repair directed acyclic graph.

[0096] Depend on Figure 2 In the description of step 203 in the illustrated embodiment, the repair process of a single faulty block can be abstracted into a RDAG, which is recorded as G = (V, E). It is easy to understand that multiple RDAGs can be constructed for repairing multiple faulty blocks, and the specific details are not limited here.

[0097] 402. According to the erasure code, k+1 vertices are set, and repair edges are constructed according to the node failure repair algorithm.

[0098] For an RS(k,m) code, set k+1 vertices {v0,v1,…,v k-1 ,v k}, where the first k vertices {v0,v1,…,v k-1} represents the k surviving blocks (helper nodes) of the stripe, and the last vertex {v k} represents the failed block (requesting node). For failures of multiple blocks, you can choose to construct multiple RDAGs, each of which represents the failed block of the stripe, and the remaining vertices represent the surviving blocks of the stripe. Then, different repair algorithms (such as traditional repair algorithms CR, PPR, or ECPipe) are used to construct the edge (u, v). The edge (u, v) in the graph indicates that all requested surviving blocks must be collected from the parent vertex of vertex u. It is not difficult to understand that from the above description, it can be seen that the repair edge E = (u, v) is used to represent the set of points from any vertex to the node to be requested.

[0099] 403. Combine the parent vertex and all child vertices in the pointing set corresponding to the repair edge with the decoding coefficients of the erasure code corresponding to the data to be repaired to obtain a mapping set.

[0100] According to the pointing set of step 402, the parent vertex and all child vertices in the pointing set corresponding to the repair edge can be combined with the decoding coefficients of the erasure code corresponding to the data to be repaired, thereby obtaining a mapping set.

[0101] In one specific embodiment, all requested surviving blocks collected from the parent vertex of vertex u are added to the local data (multiplied by the decoding coefficient), and the result is sent to vertex v. It is not difficult to understand that local data refers to the data blocks placed locally, that is, Figure 6 The data block on the middle node. The decoding coefficient is similar to the encoding coefficient c ij , except that the decoding coefficient is calculated by a fixed formula.

[0102] 404. Obtain all repair directed acyclic graphs (RDAGs) and set a set of all repair directed acyclic graphs (RDAGs) to input all mapping sets in the set into a dynamic weight mapping algorithm and output a mapping set from all vertices to the requested nodes.

[0103] Thus, through step 403, all repair directed acyclic graphs RDAGs are obtained, and a set δ of all repair directed acyclic graphs RDAGs is set, so that all mapping sets in the set δ are input into the dynamic weight mapping algorithm, and a mapping set V2N(i,j) of all vertices to the requested nodes is output. It is not difficult to understand that V2N(i,j) is used to represent the vertex v of the i-th repair directed acyclic graph in all RDAGs. j The node to be requested is mapped to, where j is an integer less than or equal to k.

[0104] 405. According to the dynamic weight mapping algorithm, set the stripe labels and minimum load parameters of all stripes, and mark all nodes with node identifiers.

[0105] Based on step 404, the stripe labels and minimum load parameters for all stripes are set according to the DWM algorithm, and all nodes are marked with node identifiers. It is easy to understand that all stripes include stripes to be repaired and repair stripes, and the minimum load parameter is used to calculate the parameter when the minimum load exists between the helping node and the requesting node.

[0106] In one specific embodiment, HRF first initializes sid to 0, representing the stripe index. It then traverses each vertex in each graph to perform the mapping. HRF then initializes the subsequent traversal queue q to an empty queue and sets minload and idx to record the receiving node with the minimum load. Here, sid represents the stripe ID, i.e., the stripe id. q represents the queue. minload is a parameter used to calculate the minimum load. idx is the ID or index of the labeled node.

[0107] 406. Determine the topological order according to the stripe label, traverse each vertex according to the topological order, the minimum load parameter and the node identifier, and map the leaf vertex to the node with the minimum upload load.

[0108] Then, the topological order is determined according to the stripe label, and each vertex is traversed according to the topological order, minimum load parameter and node identity, and the leaf vertex is mapped to the node with the smallest upload load.

[0109] In one specific embodiment, HRF determines the subsequent topological order based on the stripe id, then traverses each vertex in the topological order, and maps the leaf vertex to the node with the smallest upload load.

[0110] 407. After completing leaf vertex mapping, map the middle vertex and determine all mapped vertices and unmapped vertices, traverse all unmapped vertices, select the node with the smallest load after mapping as the next mapping node, and determine the current node load.

[0111] Based on step 407, after the leaf vertex mapping is completed, the intermediate vertices can be mapped and all mapped vertices and unmapped vertices can be determined. In this way, all unmapped vertices are traversed and the node with the smallest load after mapping is selected as the next mapping node to determine the current node load. It is not difficult to understand that mapped vertices are used to represent vertices that have been mapped, and unmapped vertices are used to represent vertices that have not been mapped.

[0112] In one specific embodiment, after completing the mapping of leaf vertices, HRF continues to map the intermediate vertices. v is the mapped vertex, and u is the unmapped vertex. HRF traverses the unmapped nodes in the strip and selects the node with the smallest load after mapping to perform the mapping. It is not difficult to understand that at this time, u and v generally refer to the vertex u and vertex v in the edge (u, v), and there is no restriction on who is the parent vertex, that is, u is not necessarily the parent vertex, and v is not necessarily the child vertex.

[0113] 408. Update the current node load corresponding to the next mapping node, return the current node load to the mapping set, and execute the next topological order step, so that after completing the mapping between vertices and nodes in all mapping sets according to the topological order, the helping node with the smallest node load is determined as the target helping node, and the corresponding node to be requested is determined as the target request node.

[0114] Based on step 407, the current node load corresponding to the next mapping node can be updated and returned to the mapping set. Thus, the next topological order step is executed. After the mapping between vertices and nodes in all mapping sets is completed according to the topological order, the helping node with the smallest node load is determined as the target helping node, and the corresponding node to be requested is determined as the target requesting node.

[0115] In one specific embodiment, after a mapping process, the HRF updates the node's upload and download loads. Finally, the algorithm returns the mapping set V2N(i, j). The next topological sequence step is then executed, mapping the next stripe. Thus, after all stripes are mapped, the helping node with the smallest node load is determined as the target helping node, and the corresponding requesting node is determined as the target requesting node.

[0116] To facilitate a general description of the DWM algorithm corresponding to steps 404 to 408, specifically, please refer to Table 2.

[0117]

[0118]

[0119] Let δ be the set of all RDAGs. Let V2N be the set of mappings from vertices to nodes, where V2N(i,j) represents the node to which the vertex of the i-th RDAG graph is mapped. u(i) and d(i) represent the upload and download loads of the node. It should be noted that u(i) only represents the node N of the i-th RDAG graph. iupload load. HRF first initializes sid to 0, indicating the index of the stripe (line 1). HRF traverses each vertex in each graph to perform mapping (line 2). HRF initializes q to an empty queue and sets minload and idx to record the receiving node with the smallest load (line 3). It traverses each vertex in topological order and maps the leaf vertex to the node with the smallest upload load, where represents the set of nodes where the stripe is located (lines 4-12). HRF continues to map the intermediate vertices (lines 12-33). v is the vertex that has been mapped and u is the unmapped vertex (line 22). HRF traverses the unmapped nodes in the stripe and selects the node with the smallest load after mapping to perform mapping (lines 20-29). Then, HRF updates the upload and download loads of the node (lines 30-32). Finally, the algorithm returns the mapping set V2N (line 35).

[0120] For further information, see Figure 9 , Figure 9 This is a schematic diagram of a DWM-based vertex mapping process disclosed in an embodiment of the present application. Figure 9 Shows how DWM maps vertices in multiple RDAGs to different nodes. Figure 7 Take the stripe distribution and w(i,j) function shown in Table 1 as examples. (u_{i}, d_{i}) represents the upload and download load of a node. Different colors represent different RDAGs. The DWM algorithm continuously maps vertices in the graph to different nodes and updates u_{i} and d_{i} in sequence.

[0121] First, HRF constructs three RDAGs for the three failed stripes using the CR algorithm (vertices of the same color in the graph belong to the same stripe). HRF then maps stripe 1. It maps leaf vertices v0 and v1 to N1 and N3, respectively, minimizing u(i) (step 1). It's easy to understand that these two leaf vertices are initially mapped randomly. Since vertex v2 has not yet been mapped, u(i) and d(i) cannot be updated. Next, the vertex can choose three nodes, N4, N5, and N6, for mapping. Mapping vertex v2 to node N6 minimizes the system load (d6 = 3); otherwise, the load increases (d4 = 4, d5 = 7). Therefore, HRF maps vertex v2 to node N6 (step 2). HRF maps the vertices of the remaining two graphs using the DWM algorithm (steps 3, 4, 5, and 6). After mapping the vertices of all three RDAGs, the optimal helpers and requesters are learned.

[0122] Furthermore, the yellow v0 and v1: because v0 needs to be mapped to the node with the lowest upload load, at this time the upload load of node 1 is 2, the upload load of node 3 is 1, and the upload loads of other nodes are all 0, so a node with an upload load of 0 can be randomly selected. The same applies to the green v0 and v1.

[0123] In DWM, a near-optimal helper and requester pair is selected to reduce the node's maximum load. However, randomly arranging the node's transmission order can extend the repair time. The key challenge here is determining the node's transmission order. To fully utilize the node's network resources, we propose the SLRA algorithm, which slices the data block corresponding to each link according to the allocated transmission time. See steps 409-412.

[0124] 409. Initialize the transmission time of all links associated with leaf vertices in the set δ to w(i, j). According to the link resource allocation algorithm of the slice and w(i, j), allocate the link weight z of the link between the two helping nodes.

[0125] After completing the DMW algorithm, the SLRA algorithm can be executed. Specifically, the transmission time of all links associated with leaf vertices in the set δ is initialized to w(i, j). Then, according to the link resource allocation algorithm of the slice and w(i, j), the link weight z of the link between the two helping nodes is allocated. It is not difficult to understand that w(i, j) is used to represent the helping node N i With help node N j The transmission time of the link between them. The w(i,j) can be found in Table 1.

[0126] In one specific embodiment, in SLRA, (N i ,N j ,z) is defined as the slave node N i To node N j Links assigned z time (corresponding weight). SLRA first initializes the time / weight of all links related to leaf vertices in RDAG to w(i,j). It is not difficult to understand that z represents the time / weight assigned by the SLRA algorithm to the link (N i ,N j ) allocates z time for sending data. The w(i,j) here is the same as the w(i,j) in Table 1, that is, the node N i to N j transmission time.

[0127] 410. Sort all links associated with leaf vertices in descending order of link weight, and place links without competing relationships in the set of links to be sent. Using the minimum transmission time in the set of links to be sent as a benchmark, determine the target helper block and target repair link that meet the minimum transmission time.

[0128] For easier understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram of a transmission scheduling based on SLRA disclosed in an embodiment of the present application. Figure 10 by Figure 7 Taking the RDAG mapped in and w(i,j) in Table 1 as an example, the transmission scheduling of SLRA is described. i ,N j ,z) represents node N i and node N j The link between them is allocated w time for transmission.

[0129] Since there are dependencies between vertices in RDAG, SLRA sorts all links corresponding to leaf vertices in descending order of z weight. Then, it adds links without contention (i.e., the sending and receiving nodes of the two links are different) to the set of links to be sent ( Figure 10 Step 1). Finally, SLRA uses the minimum transmission time (x) in the set of links to be sent as a benchmark to determine the target helper block and target repair link that meet the minimum x time.

[0130] 411. Allocate the minimum transmission time to other links to be sent in the set of links to be sent, so that other helping blocks of other helping nodes in other links to be sent are sliced ​​according to the minimum transmission time to obtain sliced ​​helping blocks.

[0131] Therefore, based on step 410, the minimum transmission time is allocated to other links to be sent in the set of links to be sent, so that other helping blocks of other helping nodes in other links to be sent are sliced ​​according to the minimum transmission time to obtain sliced ​​helping blocks.

[0132] In one specific embodiment, x time is allocated to all other links to be transmitted (step 2). The data block is sliced ​​according to the time allocated to the link. For example, if link (N5, N3) is allocated 2 seconds, the data block is sliced ​​into 2 / 3 of its original size, and the remaining portion is available for the next transmission.

[0133] 412. After repairing the target request node according to all target helping nodes in the links where no competition exists, delete the links where no competition exists, and update the transmission time of the link corresponding to the slice helping block according to the minimum transmission time to obtain the next transmission time of the updated link, and execute the link resource allocation algorithm of the next round of slices according to the next transmission time to realize the node repair of the fault block to be repaired.

[0134] Based on step 411, after repairing the target request node according to all target helping nodes in the link without competing relationship, the link without competing relationship is deleted, and the transmission time of the link corresponding to the slice helping block is updated according to the minimum transmission time to obtain the next transmission time of the updated link, and the link resource allocation algorithm of the next round of slices is executed according to the next transmission time to realize the node repair of the fault block to be repaired.

[0135] In one specific embodiment, the sent links are deleted from the RDAG, and the unsent links update the corresponding w and wait for the next transmission. The above steps are repeated until there are no vertices ( Figure 10 Specifically, Figure 9 Taking the RDAG mapped in and w(i,j) in Table 1 as an example, we describe how SLRA determines the sending order of nodes and how to use pipeline technology. Figure 10 As shown, SLRA first initializes the w values ​​for all links and sorts all links associated with leaf vertices in descending order of their w(i,j) weights. It then adds three non-contention links to the set of links to be sent: {(N5,N3,3), (N3,N4,2), (N1,N6,2)}. SLRA selects the smallest weight value, 2, from this set and assigns it to all links in the set. This means that links (N5,N3), (N3,N4), and (N1,N6) are all allocated 2 seconds to send data. Since links (N3,N4) and (N1,N6) have sufficient time to send their data chunks, the chunks are not fragmented. On the other hand, the chunk corresponding to link (N5,N3) is fragmented to 2 / 3 of its original size. Links (N3,N4) and (N1,N6) are then removed from the RDAG, and (N5,N3,3) is updated to (N5,N3,1). SLRA repeated the above steps and finally completed the repair task in 4 seconds.

[0136] To facilitate a general description of the SLRA algorithm corresponding to steps 409 to 412, specifically, please refer to Table 3.

[0137]

[0138]

[0139] First, it initializes the link weights of all graphs (line 1). Subsequently, it selects edges connected to leaf vertices in the graph and adds them to set Θ (lines 4-9), and sorts set Θ in descending order of z (line 12). Then, it removes conflicting links from set Θ (lines 14-19). Finally, it splits the data blocks corresponding to all links in set Θ into And transfer and update the graph (Lines 21-26). It is the size of each data block (usually 64MB or 128MB).

[0140] For further information, see Figure 11 , Figure 11 This is a flowchart illustrating a full-node repair framework disclosed in an embodiment of this application. By inputting the faulty block and the helper block into the HRF framework, the transmission queues between different nodes are screened using algorithms such as CR, PPR, or ECPipe within the HRF framework, thereby determining the final requesting node and helper node.

[0141] The node repair method based on erasure codes disclosed in the embodiments of this application can effectively reduce the full-node repair time of erasure codes in heterogeneous data center environments, accelerating the repair process. Furthermore, the method can be applied to different encodings, such as LRC, Butterfly, and RS, and to different repair algorithms, such as CR, PPR, or ECPipe. This enables full-node repair of erasure codes in accelerated heterogeneous DCNs, improving the feasibility of the solution.

[0142] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0143] See also Figure 12 , Figure 12 This is a structural diagram of a node repair system based on erasure codes disclosed in an embodiment of the present application.

[0144] The acquisition unit 1201 is configured to acquire the data to be repaired uploaded by the user and divide the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired. The m check blocks are obtained by encoding the k data blocks, k and m are positive integers, and k is not less than m. The set of k data blocks and m check blocks corresponding to the data to be repaired is determined as a stripe, and each data block is distributed on k+m different storage nodes.

[0145] Determining unit 1202 is configured to determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine the repair stripe, and select a node to be requested from the repair stripe; wherein the faulty block to be repaired is used to describe a data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are the k+m-1 data blocks remaining in the stripe to be repaired excluding the faulty block, the storage nodes corresponding to the helper blocks are helper nodes, the repair stripe is the stripe that does not contain the faulty block, the requester nodes are used to describe storage nodes that do not contain the data blocks corresponding to the stripe to be repaired, and the requester nodes are used to store the repaired faulty block to be repaired;

[0146] A setting unit 1203 is configured to construct a repair directed acyclic graph and set k+1 vertices and repair edges of the repair directed acyclic graph; wherein the first k vertices are used to represent the k helper nodes of the strip to be repaired, the last vertex is used to represent the faulty block to be repaired and the corresponding requested node, and the repair edge is used to represent the mapping set between any vertex and the requested node;

[0147] The recording unit 1204 is configured to perform mapping on each vertex in the mapping set according to the dynamic weight mapping algorithm, record the helping node and the to-be-requested node with the smallest load after mapping, and determine the helping node as the target helping node and the to-be-requested node as the target requesting node;

[0148] The determination unit 1202 is also used to determine the target helping blocks and target repair links of the target helping nodes determined in all repair directed acyclic graphs based on the slice-based link resource allocation algorithm, and slice the target helping blocks according to the target repair links to obtain sliced ​​helping blocks, so as to complete the node repair of the fault block to be repaired according to the slice helping blocks and the target repair links; wherein the target repair link is a non-overlapping data transmission link between all target helping nodes and all target request nodes.

[0149] See below Figure 13 The schematic diagram of the structure of a node repair device based on erasure code disclosed in an embodiment of the present application includes:

[0150] CPU 1301, memory 1305, input / output interface 1304, wired or wireless network interface 1303 and power supply 1302;

[0151] The memory 1305 is a temporary storage memory or a permanent storage memory;

[0152] The CPU 1301 is configured to communicate with the memory 1305 and execute the instructions in the memory 1305 to perform the aforementioned Figures 2 to 4 A node repair method based on erasure coding in any of the illustrated embodiments.

[0153] The embodiment of the present application also provides a chip system, characterized in that the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the aforementioned Figures 2 to 4 A node repair method based on erasure coding in any of the illustrated embodiments.

[0154] The embodiment of the present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the aforementioned Figures 2 to 4 A node repair method based on erasure coding in any of the illustrated embodiments.

[0155] The present application also provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the aforementioned Figures 2 to 4 A node repair method based on erasure coding in any of the illustrated embodiments.

[0156] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. A node repair method based on erasure code, characterized in that: The method comprises: Obtain the data to be repaired uploaded by the user, and divide the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired; wherein the m check blocks are obtained by encoding the k data blocks, k and m are positive integers, and k is not less than m; the set of the k data blocks and the m check blocks corresponding to the data to be repaired is determined as a stripe, and each data block is distributed on k+m different storage nodes; Determine the stripe to be repaired where the faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine a repair stripe, and select a node to be requested from the repair stripe; wherein, the faulty block to be repaired is used to describe the data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are the k+m-1 data blocks remaining in the stripe to be repaired except the faulty block, the storage node corresponding to the helper block is the helper node, the repair stripe is the stripe that does not contain the faulty block, the node to be requested is used to describe the storage node that does not contain the data block corresponding to the stripe to be repaired, and the node to be requested is used to store the repaired faulty block to be repaired; Construct a repair directed acyclic graph, and set k+1 vertices and repair edges of the repair directed acyclic graph; wherein the first k vertices are used to represent the k helper nodes of the strip to be repaired, the last vertex is used to represent the faulty block to be repaired and the corresponding node to be requested, and the repair edge is used to represent the mapping set between any vertex and the node to be requested; Perform mapping on each vertex in the mapping set according to a dynamic weight mapping algorithm, record the helping node and the node to be requested with the smallest load after mapping, and determine the helping node as a target helping node and the node to be requested as a target requesting node; The slice-based link resource allocation algorithm repairs the target help blocks and target repair links of the target help nodes determined in all directed acyclic graphs, and slices the target help blocks according to the target repair links to obtain slice help blocks, so as to complete the node repair of the fault block to be repaired according to the slice help blocks and the target repair links; wherein, the target repair link is a non-overlapping data transmission link between all target help nodes and all target request nodes.

2. The node repair method based on erasure code according to claim 1, characterized in that: The method of dividing the data to be repaired into k data blocks and m check blocks based on the erasure code of the data to be repaired includes: Divide the data to be repaired R into k data blocks and obtain the size corresponding to each data block Based on the coding equation, the k data blocks are encoded into the m check blocks according to the erasure code RS(k,m); wherein the coding equation is The D j is the jth data block, the P i is the i-th check block, the c ij Represents data block D j corresponds to the check block P i The coding coefficient of .

3. The node repair method based on erasure code according to claim 1, characterized in that: Before constructing and repairing the directed acyclic graph and setting k+1 vertices and repairing edges of the repaired directed acyclic graph, the method further includes: Selecting two helping blocks in the strip to be repaired, and determining helping nodes corresponding to the two helping blocks; Define a two-dimensional matrix b(i, j), and determine the link bandwidth between two helping nodes according to the two-dimensional matrix b(i, j); wherein the two-dimensional matrix b(i, j) is used to represent N i With N j The link bandwidth between the N i Indicates the help node corresponding to the i-th help block, the N j Indicates the help node corresponding to the j-th help block; Determine a two-dimensional weight matrix w(i, j) corresponding to the two-dimensional matrix b(i, j), and determine the link transmission time between two helping nodes based on the two-dimensional weight matrix w(i, j); wherein the calculation method of w(i, j) is as follows: The w(i,j) is used to represent the helper node N i With help node N j The transmission time of the link between them.

4. The node repair method based on erasure code according to claim 1, characterized in that: The step of constructing and repairing a directed acyclic graph and setting k+1 vertices and repairing edges of the repaired directed acyclic graph includes: For any faulty block to be repaired, construct the repair directed acyclic graph G = (V, E); wherein V is a vertex set used to describe all vertices on the repair directed acyclic graph, and E is an edge set used to describe all repair edges on the repair directed acyclic graph; According to the erasure code RS(k,m), k+1 vertices are set; wherein the k+1 vertices belong to the vertex set V={v0,v1,…,v k-1 ,v k }, V={v0,v1,…,v k-1 } vertices are used to represent the k helper nodes of the strip to be repaired, V = {v k } is used to represent the fault block to be repaired and the corresponding node to be requested; The repair edge E = (u, v) is constructed according to the node fault repair algorithm; wherein u is the parent vertex, used to represent the vertex on the repair edge; v is the child vertex, used to represent the vertex corresponding to the requested node pointed to by the parent vertex; the repair edge E = (u, v) is used to represent the set of directions between any vertex and the requested node; The parent vertex and all the child vertices in the pointing set corresponding to the repair edge E=(u, v) are combined with the decoding coefficients of the erasure code corresponding to the data to be repaired to obtain the mapping set.

5. The node repair method based on erasure code according to claim 4, characterized in that: The step of performing mapping on each vertex in the mapping set according to the dynamic weight mapping algorithm includes: Obtain all the repaired directed acyclic graphs RDAGs, and set a set δ of all the repaired directed acyclic graphs RDAGs; Input all mapping sets in the set δ into the dynamic weight mapping algorithm, and output the mapping set V2N(i, j) of all vertices to the requested node; wherein, V2N(i, j) is used to represent the vertex v of the i-th repaired directed acyclic graph in all RDAGs. j The node to be requested is mapped to, where j is an integer less than or equal to k.

6. The node repair method based on erasure code according to claim 4, characterized in that: The recording of the helping node and the node to be requested having the smallest load after mapping, and determining the helping node as a target helping node and the node to be requested as a target requesting node includes: According to the dynamic weight mapping algorithm, stripe labels and minimum load parameters are set for all stripes, and all nodes are marked with node identifiers; wherein all stripes include the stripe to be repaired and the repairing stripe, and the minimum load parameter is used to calculate the parameter when the minimum load exists between the helping node and the strip requesting node; Determine a topological order according to the stripe label, traverse each vertex according to the topological order, the minimum load parameter, and the node identifier, and map a leaf vertex to a node with the smallest upload load; wherein the leaf vertex is used to describe a vertex that is not pointed to in the parent vertex or child vertex; After the leaf vertex mapping is completed, the intermediate vertices are mapped, and all mapped vertices and unmapped vertices are determined; wherein the intermediate vertices are used to describe vertices that include both the parent vertex and the child vertex, the mapped vertices are used to describe vertices that have been mapped, and the unmapped vertices are used to describe vertices that are not mapped; Traversing all unmapped vertices, selecting the node with the smallest load after mapping as the next mapping node, and determining the current node load; wherein the current node load includes the upload load and download load between the unmapped vertex and the next mapping node; Update the current node load corresponding to the next mapping node, return the current node load to the mapping set V2N(i,j), and execute the next topological sequence step; After the mapping between vertices and nodes in all mapping sets is completed according to the topological order, the helping node with the minimum node load is determined as the target helping node, and the corresponding node to be requested is determined as the target requesting node.

7. The node repair method based on erasure code according to claim 6, characterized in that: The slicing-based link resource allocation algorithm combines the target helper blocks and target repair links of all target helper nodes determined in the repair directed acyclic graph, and slices the target helper blocks according to the target repair links to obtain sliced ​​helper blocks, including: Initialize the transmission time of all links associated with the leaf vertex in the set δ to w(i, j); wherein w(i, j) is used to represent the helper node N i With help node N j Transmission time of the link between them; According to the link resource allocation algorithm of the slice and w(i, j), the helper node N is allocated i With the help node N j The link weight z of the link between them; Sort all links associated with leaf vertices in descending order of the link weight z, and place links without contention in the set of links to be sent; wherein, the links without contention are used to describe the number of helper nodes N between any two links. i or help node N j different; Determining the target helping block and the target repair link that meet the minimum transmission time based on the minimum transmission time in the set of links to be sent; wherein the target helping block is used to describe the current helping node and the helping block corresponding to the current helping node, and the target repair link is used to describe the transmission link between the current helping node and the current helping node; The minimum transmission time is allocated to other links to be sent in the set of links to be sent, so that other helping blocks of other helping nodes in the other links to be sent are sliced ​​according to the minimum transmission time to obtain the sliced ​​helping blocks.

8. The node repair method based on erasure code according to claim 7, characterized in that: The completing the node repair of the fault block to be repaired according to the slice helper block and the target repair link includes: After repairing the target requesting node according to all target helping nodes in the link without contention, deleting the link without contention, and updating the transmission time of the link corresponding to the slice helping block according to the minimum transmission time to obtain the next transmission time of the updated link; The link resource allocation algorithm of the next round of slices is executed according to the next transmission time to realize the node repair of the fault block to be repaired.

9. A node repair system based on erasure code, characterized in that: The system comprises: an acquisition unit, configured to acquire data to be repaired uploaded by a user, and divide the data to be repaired into k data blocks and m check blocks based on an erasure code of the data to be repaired; wherein the m check blocks are obtained by encoding the k data blocks, k and m are positive integers, and k is not less than m; a set of the k data blocks and the m check blocks corresponding to the data to be repaired is determined as a stripe, and each data block is distributed across k+m different storage nodes; A determination unit is configured to determine a stripe to be repaired where a faulty block to be repaired is located, and select k helper blocks from the stripe to be repaired; determine a repair stripe, and select a node to be requested from the repair stripe; wherein the faulty block to be repaired is used to describe a data block that needs to be repaired in the data to be repaired, the stripe to be repaired is the stripe where the faulty block is located, the helper blocks are k+m-1 data blocks remaining in the stripe to be repaired except the faulty block, the storage node corresponding to the helper block is a helper node, the repair stripe is the stripe that does not contain the faulty block, the node to be requested is used to describe a storage node that does not contain a data block corresponding to the stripe to be repaired, and the node to be requested is used to store the repaired faulty block to be repaired; A setting unit is configured to construct a repair directed acyclic graph and set k+1 vertices and repair edges of the repair directed acyclic graph; wherein the first k vertices are used to represent the k helper nodes of the strip to be repaired, the last vertex is used to represent the faulty block to be repaired and the corresponding node to be requested, and the repair edge is used to represent a mapping set between any vertex and the node to be requested; a recording unit, configured to perform mapping on each vertex in the mapping set according to a dynamic weight mapping algorithm, record the helping node and the node to be requested with the smallest load after mapping, and determine the helping node as a target helping node and the node to be requested as a target requesting node; The determination unit is further used to repair the target helping blocks and target repair links of the target helping nodes determined in all directed acyclic graphs based on a slicing link resource allocation algorithm, and slice the target helping blocks according to the target repair links to obtain sliced ​​helping blocks, so as to complete the node repair of the fault block to be repaired according to the slice helping blocks and the target repair links; wherein the target repair link is a non-overlapping data transmission link between all target helping nodes and all target request nodes.

10. A node repair device based on erasure code, characterized in that: The device comprises: CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the node repair method based on erasure code according to any one of claims 1 to 8.

Images