Graph-based erasure code fast recovery method, system and readable storage medium

Abstract

The application provides a graph-based erasure code fast recovery method, a system and a readable storage medium, and the method comprises the following steps: S1, collecting system parameters and real-time load information of each node by a quantification method; S2, constructing two graph models, and the two graph models correspond to selection of a source node and selection of a decoding node in a recovery task respectively; S3, using a maximum flow algorithm to obtain an optimal solution in which available bandwidths of each node are just occupied, that is, a recovery scheduling scheme; S4, according to the recovery scheduling scheme, sending each recovery task from the selected source node to a corresponding decoding node for reading of a data block, and then writing data into a target node set in the system after decoding and recovery are completed on the decoding node. By putting the available bandwidths of each node into the graph model for scheduling, and based on the maximum flow algorithm, it is ensured that task scheduling of each node will not exceed the available bandwidth, higher-dimensional load balancing is achieved, and congestion and jamming phenomena are avoided.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a graph-based erasure coding fast recovery method, system, and readable storage medium. Background Technology

[0002] Distributed storage uses erasure coding to achieve data fault tolerance, but erasure coding requires significant network and disk I / O resources for data recovery. Basic erasure coding recovery requires acquiring several source data blocks and decoding them on the target node to recover the lost data blocks. The following methods are scheduling algorithms designed based on this recovery model.

[0003] Classic random scheduling methods, such as those implemented in HDFS and Ceph storage systems, are based on simple random strategies that randomly select the source data blocks and target nodes required for recovery. This method is simple to implement, but it is difficult to solve the problem of encountering busy nodes during the recovery process.

[0004] The Fast-PR scheduling algorithm is based on graph modeling and uses the maximum flow algorithm to ensure a load balance of the number of tasks received by each node. This method has better load balancing capabilities than the random algorithm in normal scenarios, but it is difficult to bypass these nodes when there are busy nodes.

[0005] The EC-Scheduler scheduling algorithm uses a simple greedy algorithm to select the source and target nodes with lower loads for data recovery based on the real-time load of each node. Although EC-Scheduler selects nodes with lower loads to avoid busy nodes, under the traditional recovery mode, the target node is under much greater pressure than the source node (it needs to receive k data blocks and perform decoding), which can easily lead to large tail latency when most target nodes are busy.

[0006] The three scheduling algorithms based on traditional recovery modes mentioned above all struggle to avoid busy nodes in high-load clusters during the recovery process, resulting in significant recovery tail delays and impacting system data reliability. Summary of the Invention

[0007] The purpose of this invention is to provide a graph-based erasure coding fast recovery method, system, and readable storage medium that can avoid busy nodes and accelerate data recovery under high system load.

[0008] To achieve the above objectives, this invention provides a graph-based erasure coding fast recovery method, comprising the following steps:

[0009] S1. Collect system parameters and real-time load information of each node through quantitative methods;

[0010] S2. Construct two graph models, which correspond to the selection of source nodes and decoding nodes in the recovery task, respectively.

[0011] S3. Use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, i.e., the recovery scheduling scheme.

[0012] S4. According to the recovery scheduling scheme, each recovery task is sent from the selected source node to the corresponding decoding node to read the data block. After the data is decoded and recovered on the decoding node, the data is written to the target node set by the system.

[0013] Optionally, the system parameters include erasure coding parameter information, number of nodes, basic bandwidth information, and source data information.

[0014] Optionally, the real-time load information of the node includes the node's available bandwidth.

[0015] Optionally, the graph model includes task nodes and source / decoding nodes, the path relationship between nodes represents the topological relationship between the recovery task and the source / decoding nodes, and the edge weight between the task node and the source / decoding node represents the required traffic size.

[0016] Optionally, the graph model further includes a virtual target node, where the edge weight between the virtual target node and the source node / decoding node represents the node's own available bandwidth capability.

[0017] Optionally, the maximum flow algorithm is the Ford-Fulkerson algorithm.

[0018] Based on the same inventive concept, this application also proposes a graph-based erasure coding fast recovery system, comprising:

[0019] The parameter collection module is configured to collect system parameters and real-time load information of each node through a quantification method.

[0020] The graph model building module is configured to build two graph models, which correspond to the selection of the source node and the selection of the decoding node in the recovery task, respectively.

[0021] The scheduling module is configured to use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, and thus obtain a recovery scheduling scheme.

[0022] The data recovery module is configured to send each recovery task from the selected source node to the corresponding decoding node to read data blocks according to the recovery scheduling scheme, and then write the data to the target node set by the system after the decoding recovery is completed on the decoding node.

[0023] Optionally, the system parameters include erasure coding parameter information, number of nodes, basic bandwidth information, and source data information.

[0024] Optionally, the real-time load information of the node includes the node's available bandwidth.

[0025] Based on the same inventive concept, this application also proposes a readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the graph-based erasure coding fast recovery method as described above.

[0026] In the graph-based erasure coding fast recovery method, system and readable storage medium provided by the present invention, the available bandwidth of each node is put into the graph model for scheduling, and the maximum flow algorithm is used to ensure that the task scheduling of each node will not exceed its available bandwidth, thereby achieving higher-dimensional load balancing and ensuring that no congestion occurs. Attached Figure Description

[0027] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0028] Figure 1 A flowchart illustrating a graph-based erasure coding fast recovery method according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of a graph model for source node selection provided in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of a graph model for selecting decoding nodes according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of a graph-based erasure coding fast recovery system provided in an embodiment of the present invention.

[0032] In the attached image:

[0033] T1, T2, T3, U1, U2, U3 - Task nodes; N1, N2, N3, N4, N5 - Source nodes; V1, V2, V3, V4, V5 - Decoding nodes; S0 - Virtual start node; DE - Virtual target node;

[0034] 100 - Parameter collection module; 200 - Graph model construction module; 300 - Scheduling module; 400 - Data recovery module. Detailed Implementation

[0035] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the purpose of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only used to complement the content disclosed in the specification, for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, if they are the same as or similar to the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0036] As used herein, the singular forms “a,” “an,” and “the” include plural objects unless otherwise expressly indicated. As used herein, the term “or” is generally used to include “and / or” unless otherwise expressly indicated. As used herein, the term “a number” is generally used to include “at least one” unless otherwise expressly indicated. As used herein, the term “at least two” is generally used to include “two or more” unless otherwise expressly indicated. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature.

[0037] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0038] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a graph-based erasure coding fast recovery method according to an embodiment of the present invention. This embodiment provides a graph-based erasure coding fast recovery method, including the following steps:

[0039] S1. Collect system parameters and real-time load information of each node through quantitative methods;

[0040] S2. Construct two graph models, which correspond to the selection of source nodes and decoding nodes in the recovery task, respectively.

[0041] S3. Use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, i.e., the recovery scheduling scheme.

[0042] S4. According to the recovery scheduling scheme, each recovery task is sent from the selected source node to the corresponding decoding node to read the data block. After the data is decoded and recovered on the decoding node, the data is written to the target node set by the system.

[0043] This invention achieves higher-dimensional load balancing and avoids congestion by incorporating the available bandwidth of each node into a graph model for scheduling and using the maximum flow algorithm to ensure that the task scheduling of each node does not exceed its available bandwidth.

[0044] First, step S1 is executed, which collects system parameters and real-time load information of each node using a quantification method. The system parameters include erasure coding parameters, the number of nodes, basic bandwidth information, and source data information. The real-time load information of each node includes its available bandwidth, which includes available uplink bandwidth and available downlink bandwidth.

[0045] Then, step S2 is executed to construct two graph models, which correspond to the selection of source nodes and decoding nodes in the recovery task, respectively. The maximum flow algorithm is then used to obtain an optimal solution in which the available bandwidth of each node is just fully utilized.

[0046] like Figure 2 As shown, the graph model selected for the source node includes task nodes (T1, T2, T3) and source nodes (N1, N2, N3, N4, N5). The path relationships between nodes represent the topological relationship between the recovery task and the source nodes. The edge weights between the task nodes and the source nodes represent the required traffic volume. S0 represents the virtual starting node of the graph model. Figure 2 In this diagram, the virtual starting node S0 is connected to each task node, with each edge having a weight of k, where k represents the number of data blocks in the erasure coding stripe. This is equivalent to the number of source data blocks required by a single recovery task, meaning each task needs to obtain at least k data blocks from different nodes. Each recovery task node is connected to each source node of its corresponding stripe, with each edge having a weight of 1, because each recovery task can only obtain one data block from each source node.

[0047] Similarly, please refer to Figure 3The graph model selected by the decoding nodes includes task nodes (U1, U2, U3) and decoding nodes (V1, V2, V3, V4, V5). The path relationships between nodes represent the topological relationship between the recovery tasks and the decoding nodes. S0 represents the virtual starting node of the graph model. The virtual starting node S0 is connected to each task node with an edge weight of k. This means that each decoding task needs to download k data from other nodes and transmit them to the corresponding decoding node. Each task node is connected to its selectable decoding node, and the edge weight between the task and the decoding node is also k, because decoding only needs to be completed on one node, and the traffic required for this single task on one decoding node is k.

[0048] Furthermore, the graph model also includes a virtual target node (DE), where the edge weights between the virtual target node (DE) and the source / decoding node represent the node's own available bandwidth capacity. For the source node, each source node is connected to the virtual target node (DE), and the edge weights represent the available uplink bandwidth of each node; for the decoding node, the edge weights between each decoding node and the virtual target node (DE) represent the node's own available downlink bandwidth.

[0049] Then execute S3, based on Figure 2 and Figure 3 An optimal solution can be obtained using a typical maximum flow algorithm such as the Ford-Fulkerson algorithm, which finds a recovery scheme that makes the available bandwidth of each node just fully utilized. Therefore, this scheduling ensures that busy nodes will only occupy a small amount of bandwidth, thus ensuring that recovery will not be blocked on these nodes.

[0050] Finally, S4 is executed, which sends each recovery task from the selected source node to the corresponding decoding node to read data blocks according to the recovery scheduling scheme. After the data is decoded and recovered on the decoding node, the data is written to the target node set by the system.

[0051] In this embodiment, Figure 2 and Figure 3 The flow of data between task nodes represents the execution method of the task. The source node connected to a task node represents the source node required for recovery, while the decoding node connected to a task node represents the decoding node to which the task is sent. If the decoding node is not the target node set by the system, the recovered data must be sent to the corresponding target node after decoding is completed.

[0052] Please refer to Figure 4 Based on the same inventive concept, embodiments of the present invention also provide a graph-based erasure coding fast recovery system, comprising:

[0053] The parameter collection module 100 is configured to collect system parameters and real-time load information of each node through a quantization method.

[0054] Graph model building module 200 is configured to build two graph models, which correspond to the selection of source nodes and the selection of decoding nodes in the recovery task, respectively.

[0055] The scheduling module 300 is configured to use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, i.e., a recovery scheduling scheme.

[0056] The data recovery module 400 is configured to send each recovery task from the selected source node to the corresponding decoding node to read data blocks according to the recovery scheduling scheme, and then write the data to the target node set by the system after the decoding recovery is completed on the decoding node.

[0057] Based on the same inventive concept, this application also proposes a readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the graph-based erasure coding fast recovery method described above in the above feature description.

[0058] The readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device, such as, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer programs described herein can be downloaded from the readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. Each computing / processing device's network adapter card or network interface receives the computer program from the network and forwards it for storage in a readable storage medium within the respective computing / processing device. The computer program used to perform the operations of this invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from a computer program. These electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present invention.

[0059] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer programs can also be stored in a readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the readable storage medium storing the computer program comprises an article of manufacture including instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0060] A computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executing on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0061] In summary, the embodiments of the present invention provide a graph-based erasure coding fast recovery method, system, and readable storage medium. By placing the available bandwidth of each node into a graph model for scheduling, and based on the maximum flow algorithm to ensure that the task scheduling of each node will not exceed its available bandwidth, a higher-dimensional load balancing is achieved, ensuring that no congestion or blocking occurs.

[0062] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A graph-based erasure coding fast recovery method, characterized in that, Includes the following steps: S1. Collect system parameters and real-time load information of each node through a quantitative method; wherein, the real-time load information of the node includes the available uplink bandwidth and available downlink bandwidth of each node. S2. Construct two graph models, which correspond to the selection of source nodes and decoding nodes in the recovery task, respectively; where... The graph model for source node selection includes a virtual start node, task nodes, source nodes, and virtual target nodes. The virtual start node is connected to each task node via an edge with weight k. Each task node is connected to the source node of its corresponding stripe via an edge with weight 1. Each source node is connected to the virtual target node via an edge with weight equal to the available uplink bandwidth of that source node. The edge weight represents the amount of traffic required by the node. The graph model selected by the decoding node includes a virtual start node, task nodes, decoding nodes, and a virtual target node. The virtual start node is connected to each task node via an edge with weight k. Each task node is connected to the decoding node via an edge with weight k. Each decoding node is connected to the virtual target node via an edge with weight equal to the available downlink bandwidth of that decoding node. Here, k is the number of data blocks in the erasure coding stripe. S3. Use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, i.e., the recovery scheduling scheme. S4. According to the recovery scheduling scheme, each recovery task is sent from the selected source node to the corresponding decoding node to read the data block. After the data is decoded and recovered on the decoding node, the data is written to the target node set by the system.

2. The graph-based erasure coding fast recovery method according to claim 1, characterized in that, The system parameters include erasure coding parameter information, number of nodes, basic bandwidth information, and source data information.

3. The graph-based erasure coding fast recovery method according to claim 1, characterized in that, In the graph model, the path relationship between the task node and the source node / decoding node represents the topological relationship between the recovery task and the source node / decoding node.

4. The graph-based erasure coding fast recovery method according to claim 1, characterized in that, The maximum flow algorithm is the Ford-Fulkerson algorithm.

5. A graph-based erasure coding fast recovery system, characterized in that, include: The parameter collection module is configured to collect system parameters and real-time load information of each node through a quantification method, wherein the real-time load information of the node includes the available uplink bandwidth and available downlink bandwidth of each node. The graph model construction module is configured to build two graph models, corresponding to the selection of source nodes and decoding nodes in the recovery task, respectively. The graph model for source node selection includes a virtual start node, task nodes, source nodes, and virtual target nodes. The virtual start node is connected to each task node via an edge with weight k. Each task node is connected to the source node of its corresponding stripe via an edge with weight 1. Each source node is connected to the virtual target node via an edge with weight equal to the available uplink bandwidth of that source node. The edge weight represents the amount of traffic required by the node. The graph model selected by the decoding node includes a virtual start node, task nodes, decoding nodes, and a virtual target node. The virtual start node is connected to each task node via an edge with weight k. Each task node is connected to the decoding node via an edge with weight k. Each decoding node is connected to the virtual target node via an edge with weight equal to the available downlink bandwidth of that decoding node. Here, k is the number of data blocks in the erasure coding stripe. The scheduling module is configured to use the maximum flow algorithm to obtain an optimal solution in which the available bandwidth of each node is just fully utilized, and thus obtain a recovery scheduling scheme. The data recovery module is configured to send each recovery task from the selected source node to the corresponding decoding node to read data blocks according to the recovery scheduling scheme, and then write the data to the target node set by the system after the decoding recovery is completed on the decoding node.

6. The graph-based erasure coding fast recovery system according to claim 5, characterized in that, The system parameters include erasure coding parameter information, number of nodes, basic bandwidth information, and source data information.

7. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the graph-based erasure coding fast recovery method according to any one of claims 1-4.

Images