Repair scheduling method in distributed storage system

Abstract

The embodiment of the invention provides a repair scheduling method in a distributed storage system, and the method comprises the steps: obtaining n repair tasks to be processed in a batch, the n repair tasks are determined by taking a principle of finding a regular subgraph from a source node graph, and the n repair tasks are determined according to the principle of finding the regular subgraph from the source node graph; the regular subgraph is used for ensuring that each survivor node can provide service for at least one repair task with the same number or similar number; constructing a replacement node graph G based on the n repair tasks, and finding target matching from the replacement node graph G; and respectively distributing a source node and a replacement node for each task in the n repair tasks at least according to the regular subgraph and the target matching, and completing data repair. According to the embodiment of the invention, balance can be ensured in the process of reading the source data from each node, so that the load of some nodes is prevented from being too high.

Description

Technical Field

[0001] The present application relates to the field of data storage, and specifically, an embodiment of the present application relates to a repair scheduling method in a distributed storage system. Background Art

[0002] In a distributed storage system based on erasure codes, when a storage node fails, it is necessary to recover the lost data blocks from other nodes that are working normally (i.e., surviving nodes). This process involves reading the necessary data blocks and check blocks from other nodes (i.e., the source nodes for this data repair) and reconstructing the lost data through a decoding algorithm. However, this process brings high disk input and output (I / O), network transmission, and decoding computational overhead. In particular, as the capacity of a single storage node increases and the number of storage devices in a distributed storage system increases, the repair time for failed data also increases significantly.

[0003] In a distributed storage system based on erasure codes, it is assumed that the system deploys a (k,m) MDS erasure code. From the perspective of system reliability, the "k+m" data blocks / check blocks in the same stripe are stored in "k+m" different nodes, with one block in each node. In most of the distributed storage systems based on erasure codes that are actually deployed, data blocks / check blocks are randomly distributed among nodes. When a system failure occurs, for each lost block: first, k surviving blocks are randomly selected from the same stripe; then these blocks are read from the nodes that store these k blocks (i.e., the selected source nodes); then the lost blocks are reconstructed; and finally, they are written to a randomly selected replacement node. During the reading and reconstruction process, if the traditional method of randomly selecting source nodes is adopted, some nodes may be frequently accessed, resulting in load imbalance. In the related art, data blocks and check blocks are randomly distributed, and source nodes are randomly selected during fault repair. In this case, some nodes may be frequently accessed as source nodes, resulting in excessive disk I / O and network loads on these nodes, while other nodes may be rarely accessed, thereby failing to fully utilize the resources of the distributed storage system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a repair scheduling method in a distributed storage system. The present invention proposes a new data fault repair scheduling method for application in a distributed system. When a node fails and is repaired, the method of packaging multiple strips with consecutive serial numbers into a batch is not adopted (that is, the principle of selecting multiple repair tasks for each batch provided by the related technology is to package multiple strips with consecutive serial numbers into a batch). Instead, some strips are selected from a large number of repair task queues to be repaired according to the load balancing principle (that is, multiple strips are selected as multiple repair tasks in a batch based on the principle of finding a regular subgraph in the source node graph), and these strips are packaged into a batch (that is, the method of obtaining n repair tasks to be processed in this batch provided by some embodiments of the present application) for processing. In this way, it can be ensured that the process of reading source data from each surviving node is balanced in the distributed system, so as to avoid excessive load on some nodes and better achieve load balancing of the distributed system.

[0005] In a first aspect, an embodiment of the present application provides a repair scheduling method in a distributed storage system, the method comprising: obtaining n repair tasks to be processed in this batch, wherein the n repair tasks are determined based on the principle of finding a regular subgraph from a source node graph, the degrees of source data vertices on the regular subgraph are equal, and the regular subgraph is used to ensure that each surviving node among multiple surviving nodes selected as source nodes can provide services for at least one repair task of the same number or a similar number; constructing an alternative node graph G based on the n repair tasks and finding a target match from the alternative node graph G, wherein the target match is used to ensure that different surviving nodes are assigned to provide services as much as possible for different tasks in the n repair tasks; at least according to the regular subgraph and the target match, respectively assigning a source node and an alternative node to each task in the n repair tasks to complete data repair, wherein both the alternative node graph and the source node graph are represented by a bipartite graph model.

[0006] Some embodiments of the present application select multiple repair tasks that need to be repaired in each batch based on the principle of ensuring the existence of a regular subgraph on the source node graph. Compared with the related art method of selecting multiple repair tasks in each batch only based on whether the serial numbers are continuous, the load balancing capability of each surviving node in the distributed system is significantly improved, and the access performance of the distributed system is further improved.

[0007] In some embodiments, the source node graph is a bipartite graph, which is used to model the selection method of the source node. In the source node graph, any repair task uses a task vertex T i Representation, each surviving node uses a source data vertex N j Characterizes that the edge (T i ,N j) represents the source data vertex N j Selected as the task vertex T i The source node.

[0008] Some embodiments of the present application simulate the process of selecting source nodes for multiple repair tasks by constructing a bipartite graph model. Specifically, the repair tasks and surviving nodes are respectively represented by corresponding vertices and the source data nodes selected for the corresponding repair tasks are represented by edges. In this way, by introducing algorithms related to the bipartite graph, it is possible to determine whether there is a regular subgraph in the graph, thereby improving the processing speed of achieving load balancing among all nodes selected as source nodes.

[0009] In some embodiments, if the network of the distributed storage system is isomorphic, the determination principle for finding a regular subgraph from the source node graph is: each source data vertex is connected to the same number of task vertices to ensure that the degree of each source data vertex is equal.

[0010] Some embodiments of the present application can achieve the following optimal load balancing by determining the degree of the source data vertices to confirm whether there is a regular subgraph in the corresponding source node graph: each repair task can read k surviving blocks for repair, and each surviving node selected as a source node can provide k blocks for k repair tasks.

[0011] In some embodiments, constructing the alternative node graph G based on the n repair tasks includes: obtaining a set of n repair tasks in the batch T = {T1, T2, ..., T n}, the selection method of the replacement node is modeled as a bipartite graph G = (T, A, N) to obtain the replacement node graph, where each T i ∈T, any repair task in the alternative node graph uses a task vertex T i Representation, each replacement node of the repair task adopts a replacement vertex N j Characterizes that the edge (T i ,N j ) represents the alternative vertex N j Selected as the task vertex T i Alternative node.

[0012] Some embodiments of the present application simulate the process of selecting alternative nodes for repair tasks by constructing a bipartite graph model, and then use the bipartite graph algorithm to better obtain an alternative node graph with perfect matching, thereby ensuring the load balancing of each alternative node when executing this batch of repair tasks, and further improving the speed of data reading in the distributed system.

[0013] In some embodiments, if the network of the distributed storage system is isomorphic, the existence of a target match in the alternative node graph G means finding a set of edges in the corresponding alternative node graph and this set of edges includes all alternative vertices in the graph, so that each alternative vertex is connected to exactly one task vertex.

[0014] Some embodiments of the present application can achieve the following optimal load balancing purpose by ensuring that target matching is achieved to determine alternative nodes for the n repair tasks in this batch: all n repair tasks are evenly distributed to n surviving nodes, and each surviving node executes one repair task.

[0015] In some embodiments, the method of obtaining the n repair tasks to be processed in this batch includes: initializing a set T; initializing a candidate task list candidate_tasks, wherein the candidate task list candidate_tasks contains all the repair tasks at the current moment; selecting a repair task one by one from the candidate task list candidate_taskscandidate_tasks, and repeatedly performing the following operations until the number of repair tasks in the set T reaches n or no repair task that meets the conditions can be found: adding the repair task t selected this time to the set T; adding the edges related to the repair task t to the corresponding source node graph S to obtain the updated source node graph, checking whether the updated source node graph contains the regular subgraph, if so, continuing to select the next repair task from the candidate task list candidate_tasks; if not, deleting the repair task t from the set T, and trying to select another repair task t' from the candidate task list candidate_taskscandidate_tasks; constructing an alternative node graph G based on the n repair tasks and finding a perfect match from the alternative node graph G, including: when the number of tasks in the set T reaches n, constructing the alternative node graph G based on all repair tasks in the set T; and finding the target match in the alternative node graph G.

[0016] Some embodiments of the present application confirm whether to use the selected repair task as the repair task of this batch by verifying one by one whether the repair task selected this time can make a regular subgraph exist on the source node graph. This can ensure that all repair tasks selected in this batch can achieve optimal load balancing on the source node, thereby improving the data reading and writing speed of the distributed system.

[0017] In some embodiments, the data repair is completed by assigning a source node and an alternative node to each of the n repair tasks at least based on the k regular generated subgraph and the target match, including: packaging the n repair tasks in the set T into a batch; assigning an alternative node and a source node to each repair task in the batch according to the regular subgraph, the target match and the bandwidth weight.

[0018] In order to improve the overall performance of heterogeneous networks, some embodiments of the present application also need to consider bandwidth weights when selecting the final source nodes and replacement nodes for each of the n repair tasks in this batch. This can further improve the network's data processing speed while ensuring that the distributed system tries its best to meet load balancing.

[0019] In some embodiments, the allocation of alternative nodes and source nodes for each repair task in the batch according to the regular subgraph, the target matching and the bandwidth weight includes: if the network of the distributed storage system is heterogeneous, then when allocating the alternative nodes and the source nodes, surviving nodes with larger bandwidth weights are preferentially selected as alternative nodes or source nodes, wherein the bandwidth weight is obtained by normalizing the bandwidth information of each surviving node collected in real time using the maximum and minimum normalization method.

[0020] Some embodiments of the present application also need to consider the bandwidth weights of the corresponding nodes when allocating source nodes and replacement nodes for multiple repair tasks in this batch, and then select nodes with richer bandwidth resources as source nodes or replacement nodes for repair tasks to improve data processing speed.

[0021] In a second aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement a method for processing a repair task in a distributed storage system as described in any one of the embodiments included in the first aspect.

[0022] In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method for processing repair tasks in a distributed storage system as described in any one of the embodiments of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 One of the flowcharts of the repair scheduling method in the distributed storage system provided in the embodiment of the present application;

[0025] Figure 2 A schematic diagram of k-regular generation of subgraphs and target matching provided in an embodiment of the present application;

[0026] Figure 3 The second flowchart of the repair scheduling method in the distributed storage system provided in the embodiment of the present application;

[0027] Figure 4 Flow chart 3 of the repair scheduling method in the distributed storage system provided in the embodiment of the present application;

[0028] Figure 5 A block diagram of a device for processing a repair task in a distributed system provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0031] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0032] Related vocabulary explanation:

[0033] Load imbalance: Since data blocks and check blocks are randomly distributed, and source nodes are randomly selected during fault repair, some nodes may be frequently accessed as source nodes, resulting in excessive disk I / O and network loads on these nodes, while other nodes may be rarely accessed, resulting in failure to fully utilize their resources.

[0034] Network bandwidth limitation: In a distributed storage system, the interconnection network between nodes is usually configured as 1Gbps or 10Gbps Ethernet, which is significantly lower than the I / O bandwidth of multiple disks aggregated within the node. The system also needs to reserve enough network bandwidth to serve the front-end online applications. Therefore, data repair work can only use limited network bandwidth. Even if a high-speed network is used, if the repair tasks are not balanced, the overall repair efficiency will still be affected.

[0035] Computing resource limitations: During the repair task execution, a decoding algorithm needs to be executed to reconstruct the lost data blocks. If certain nodes are frequently selected as source nodes or target nodes, the CPU computing power of these nodes may become a bottleneck, affecting the repair efficiency.

[0036] Heterogeneous system environment: In a heterogeneous storage system environment, the hardware configurations between nodes may be different. If the repair mechanism does not take this heterogeneity into account, the load imbalance problem may be further aggravated when processing repair tasks.

[0037] Dynamically changing workloads: Distributed storage systems not only have to handle backend fault repair tasks, but also have to deal with workloads generated by foreground online applications. These foreground applications may occupy a large amount of network bandwidth and computing resources, thus affecting the efficiency of data repair tasks.

[0038] When existing solutions process data recovery of failed nodes in a distributed storage system based on erasure codes, they usually randomly distribute data blocks and check blocks on different storage nodes. When a node failure is detected, all lost data blocks stored on the node are identified. Then, for each lost block, the system randomly selects k surviving blocks from the same stripe and randomly selects nodes storing these k blocks as source nodes, reads the data of these blocks, reconstructs the lost data blocks using a decoding algorithm, and writes the reconstructed lost data blocks to a randomly selected replacement node. In order to reduce the number of repairs, the system usually packages the tasks to be repaired with consecutive serial numbers into a batch for execution (i.e., the selection of batches to be processed provided by the relevant technology). n repair tasks); however, the inventors of the present application have found in their research that this method of selecting multiple repair tasks in each batch will lead to load imbalance. This is because the data blocks and check blocks are randomly distributed, and the source nodes are randomly selected during fault repair. Some nodes may be frequently accessed as source nodes, resulting in excessive disk I / O and network loads on these nodes, while other nodes may be rarely accessed. At the same time, the interconnection network between nodes is usually configured as 1Gbps or 10Gbps Ethernet, which is significantly lower than the I / O bandwidth of multiple disks aggregated within the node. Data repair work can only use limited network bandwidth, and decoding algorithms need to be executed during the repair process. Some nodes may become bottlenecks for computing resources because they are frequently used as source nodes or target nodes. In addition, differences in hardware configuration between nodes may cause repair tasks to further aggravate load imbalance, and the workload of foreground online applications may occupy a large amount of resources, affecting the execution efficiency of data repair tasks, and ultimately leading to low repair efficiency. The batch repair mechanism may cause some nodes to become bottlenecks, extending the overall repair time.

[0039] In other words, due to the above reasons, the batch repair mechanism of related technologies may cause certain nodes to become bottlenecks, thereby prolonging the overall repair time. This impact will be more significant when multiple nodes fail at the same time.

[0040] The embodiments of the present application relate to data fault repair technology in a distributed storage system. The technical solution solves problems such as load imbalance, network bandwidth limitation, and computing resource limitation in the data repair process by optimizing the strip selection, source node selection, and load balancing algorithm of multiple repair tasks in each batch, thereby improving the repair efficiency and the overall performance of the system. The technology provided by the embodiments of the present application can be applied to distributed storage systems such as Hadoop Distributed File System (HDFS). It is not difficult to understand that the embodiments of the present application are used in a large-scale data storage environment to ensure efficient scheduling and load balancing of repair tasks through batch processing algorithms and maximum matching algorithms, thereby improving the reliability and stability of the system.

[0041] Please see Figure 1 , Figure 1 A repair scheduling method in a distributed storage system provided for an embodiment of the present application comprises: S101, obtaining n repair tasks to be processed in this batch, wherein the n repair tasks are determined based on the principle of finding a regular subgraph from a source node graph, the degrees of source data vertices (i.e., surviving nodes selected as source nodes) on the regular subgraph are equal, and the regular subgraph is used to ensure that each surviving node among the multiple surviving nodes selected as source nodes can provide services for at least one repair task of the same number or a similar number; S102, constructing an alternative node graph G based on the n repair tasks and finding a target match from the alternative node graph G, wherein the target match is used to ensure that different surviving nodes are assigned to provide services as much as possible for different tasks among the n repair tasks; S103, at least according to the regular subgraph and the target match, respectively assigning a source node and an alternative node to each repair task among the n repair tasks to complete data repair, wherein both the alternative node graph and the source node graph are characterized by a bipartite graph model.

[0042] The implementation process of the above steps is described below by way of example.

[0043] In some embodiments of the present application, the source node graph is a bipartite graph, which is used to model the selection method of the source node. In the source node graph, any repair task uses a task vertex T i Representation, each source node of the repair task uses a source data vertex N j Characterizes that the edge (T i ,N j ) represents the source data vertex N j Selected as the task vertex T i For example, in some embodiments of the present application, if the network of the distributed storage system is isomorphic, the determination principle for finding a regular subgraph from the source node graph is: each source data vertex is connected to the same number of task vertices to ensure that the degree of each source data vertex is equal.

[0044] It should be noted that the regular subgraph determination principle of the embodiment of the present application can achieve the following technical objectives (assuming that each repair task requires k data blocks to complete the data repair of a missing data block): each repair task can read k surviving blocks for repair, and each storage node can provide k blocks for k tasks, where k is an integer greater than zero and k is the number of data blocks included in the data to be stored. It is not difficult to understand that this method can achieve a balanced task load on all source nodes when executing the n repair tasks of this batch, thereby improving the speed and efficiency of data processing.

[0045] In some embodiments of the present application, constructing the alternative node graph G based on the n repair tasks includes: obtaining a set of n repair tasks in the batch T = {T1, T2, ..., T n}, the selection method of the replacement node is modeled as a bipartite graph G = (T, A, N) to obtain the replacement node graph, where each T i ∈T, any repair task in the alternative node graph uses a task vertex T i Representation, each replacement node of the repair task adopts a replacement vertex N j Characterizes that the edge (T i ,N j ) represents the alternative vertex N j Selected as the task vertex T i For example, in some embodiments of the present application, if the network of the distributed storage system is isomorphic, the existence of a target match in the alternative node graph G means that a set of edges is found in the corresponding alternative node graph and the set of edges includes all alternative vertices in the graph, so that each alternative vertex is connected to exactly one task vertex.

[0046] It should be noted that some embodiments of the present application can achieve that all n repair tasks are evenly distributed to n surviving nodes by selecting alternative nodes for each of the n repair tasks in this batch by finding target matches. Each surviving node performs one repair task, thereby achieving load balancing among the alternative nodes and improving the speed and efficiency of data processing.

[0047] For example, in some embodiments of the present application, S101 obtains the n repair tasks to be processed in this batch, including: initializing a set T; initializing a candidate task list candidate_tasks, wherein the candidate task list candidate_tasks contains all the repair tasks at the current moment; selecting a repair task one by one from the candidate task list candidate_taskscandidate_tasks, and repeatedly performing the following operations until the number of repair tasks in the set T reaches n or no repair task that meets the conditions can be found: adding the repair task t selected this time to the set T; adding in the corresponding source node graph S The edges related to the repair task t obtain an updated source node graph, and check whether the updated source node graph contains the regular subgraph. If so, continue to select the next repair task from the candidate task list candidate_tasks; if not, delete the repair task t from the set T, and try to select another repair task t' from the candidate task list candidate_taskscandidate_tasks; the corresponding S102 exemplarily includes: when the number of tasks in the set T reaches n, construct the alternative node graph G based on all the repair tasks in the set T; find the target match in the alternative node graph G.

[0048] It should be noted that in order to improve the processing performance of heterogeneous networks, some embodiments of the present application need to consider the bandwidth characteristics of the corresponding nodes while allocating source nodes and replacement nodes for the n repair tasks in this batch according to regular subgraphs and target matching.

[0049] That is to say, in some embodiments of the present application, S103 allocates source nodes and substitute nodes to each of the n repair tasks at least based on the k regular generated subgraphs and the target matching to complete data repair, including: packaging the n repair tasks in the set T into a batch; allocating substitute nodes and source nodes to each repair task in the batch according to the regular subgraph, the target matching and the bandwidth weight. For example, in some embodiments of the present application, the allocation of substitute nodes and source nodes to each repair task in the batch according to the regular subgraph, the target matching and the bandwidth weight includes: if the network of the distributed storage system is heterogeneous, then when allocating the substitute nodes and the source nodes, the surviving nodes with larger bandwidth weights are preferentially selected as substitute nodes or source nodes, wherein the bandwidth weight is obtained based on the bandwidth information of each surviving node collected in real time and the bandwidth information is normalized using the maximum and minimum normalization method.

[0050] It is not difficult to understand that in order to improve the overall performance of the heterogeneous network, some embodiments of the present application also need to consider the bandwidth weight when selecting the final source node and replacement node for each repair task in the n repair tasks in this batch, so that the network's data processing speed can be further improved while ensuring that the distributed system tries its best to meet the load balancing condition. Some embodiments of the present application also need to consider the bandwidth weight of the corresponding node when allocating source nodes and replacement nodes for multiple repair tasks in this batch, and then select nodes with richer bandwidth resources as source nodes or replacement nodes for repair tasks to improve data processing speed.

[0051] Combine the following Figure 2-Figure 4 The process of obtaining the regular subgraph and the target matching is exemplified.

[0052] It should be noted that the embodiments of the present application use a bipartite graph model to characterize the mapping relationship between the source nodes and the replacement nodes of a batch of data to be repaired, thereby calculating the distribution of network traffic and repair load caused by this batch of repair tasks. For ease of understanding, some embodiments of the present application assume that the network of the distributed storage system is isomorphic, that is, the downstream and upstream bandwidths of all nodes are the same. Some embodiments of the present application further assume that only one node in the system fails, and the distributed storage system deploys (k, m) MDS erasure codes, which means that at most one block will be lost in each stripe. The repair of a lost block is called a repair task (represented by T). Some embodiments of the present application represent the set of n surviving nodes as N.

[0053] 1. Alternative Node Graph

[0054] Assume that some embodiments of the present application are given a set of n repair tasks in a batch T = {T1, T2, ..., T n}, the selection of alternative nodes is modeled as a bipartite graph G = (T, A, N). In graph G, each T i ∈T represents a task vertex, each N j ∈N represents a surviving vertex (or alternative vertex), and the edge (T i ,N j ) represents the surviving node N j Can be used as task T i Some embodiments of the present application refer to such a graph G as a replacement node graph. If there is a perfect match (or target match) in the graph G, all n repair tasks can be evenly distributed to the n surviving nodes, and each node performs one repair task. At this time, in the corresponding batch of repair processes, the downlink network traffic and decoding load between all surviving nodes are balanced.

[0055] For example, Figure 2are the replacement node graph and the source node graph in the storage system deploying (3,2) erasure code provided by some embodiments of the present application, Figure 2 The upper part is the replacement node graph, and the lower part is the source node graph. Figure 2 The top part illustrates an example G with a perfect match, where Figure 2 Each repair task T i There are three edges, for example, (T1, N1), (T1, N2), and (T1, N6), indicating that any of the surviving nodes N1, N2, and N6 can serve as a replacement node for task T1. Figure 2 In , the dashed lines represent possible choices and the black edges correspond to a perfect matching of an equilibrium schedule where each surviving node performs a repair task.

[0056] In a distributed storage system, each stripe contains "k+m" data blocks / check blocks, and two blocks in the same stripe cannot be stored on the same node. Therefore, there are "k+m" nodes in the system that store blocks in a stripe, while the remaining nodes do not store blocks in the stripe. When a node fails, each repaired data block cannot be stored on the same node as other blocks in the same stripe. Therefore, there are "n-(k+m-1)" nodes in the system that can serve as replacement nodes for the repaired data blocks, which means that each task vertex T i The degree is n-(k+m1).

[0057] The inventors of the present application have found in their research that for large-scale distributed storage systems, k+m is usually much smaller than n, so the graph G is a dense graph, and in most cases there is a perfect match in the graph G. It is worth noting that although there is a perfect match in the graph G in most cases, there are still very few cases where there is no perfect match. In this case, some embodiments of the present application can use a maximum matching algorithm and some optimization methods to improve the selection of alternative nodes.

[0058] 2. Source Node Graph

[0059] Similar to the replacement node graph, some embodiments of the present application define another bipartite graph S = (T, A, N), called the source node graph, which is used to model the selection scheme of the source node. i ∈T represents a task vertex, and each N j ∈N represents a source data vertex, and the edge (T i ,N j ) indicates N j It is T i The source node, that is, N j A data block is stored in T iThe corresponding faulty data blocks are in the same stripe and can participate in repairing the corresponding faulty data blocks. If a regular subgraph (or k-regular generated subgraph) is found in S, then each repair task can read k surviving blocks for repair, and each storage node can provide k blocks for k tasks, thereby achieving a balanced upstream repair traffic on each node.

[0060] For example, some embodiments of the present application provide Figure 2 The lower part of the diagram gives an example graph S. Take T1 as an example, it has four edges, namely (T1, N2), (T1, N3), (T1, N4) and (T1, N6). This means that each surviving node in N2, N3, N4 and N6 can provide data blocks for task T1 for fault data repair. Figure 2 In the figure, the black edge represents a 3-regular generated subgraph of S, which means that each task can receive 3 blocks for faulty data repair, and each surviving node contributes 3 blocks, achieving a balanced upstream repair traffic.

[0061] When a block fails, there are k+m-1 available blocks in the storage system to repair it. This shows that in S, each task vertex is connected to k+m-1 source data vertices. Therefore, the inventors of the present application found in the study that the selection of the source node is equivalent to finding a generated subgraph of S, so that all task vertices in the generated subgraph have the same degree k, and the degree of the source data vertex is equal to the number of blocks read from the corresponding source node. In order to balance the upstream traffic between the surviving nodes, some embodiments of the present application need to minimize the degree difference between the source data vertices. Obviously, if the degree of all source data vertices is k, a complete balance is achieved. However, due to the uneven data layout within each batch, it is difficult to find k regular subgraphs in S. Since the faulty nodes in the current storage system generally have a large storage capacity, the queue to be repaired is usually very long. Some embodiments of the present application can select n tasks from its sufficiently long queue of pending repair tasks so that the upstream traffic of the source node reaches (approximately) complete balance.

[0062] The following is an exemplary description of the process of obtaining n repair tasks selected from each batch in S101 and the data repair process recorded in S103.

[0063] For n repair tasks in a batch, the embodiments of the present application find a perfect match in the alternative node graph G and find k regular generated subgraphs in the source node graph S. Then, a replacement node and k source nodes can be assigned to each repair task, thereby ensuring that the repair traffic and decoding load of the batch are balanced among all surviving nodes. The following describes how some embodiments of the present application select n repair tasks and combine them into a batch to achieve a balance between the repair traffic and the decoding load.

[0064] Considering that finding a k-regular generated subgraph in S is more complicated than finding a perfect match in G, the strategy of some embodiments of the present application is to first select n repair tasks to ensure that these tasks can form a k-regular generated subgraph in S; then, build G based on these n tasks, and determine a perfect match for these tasks in this graph. It should be noted that it is not always possible to find a k-regular generated subgraph in S, and it is not always possible to find a perfect match in G. In order to solve this problem, some embodiments of the present application design a batch processing algorithm and use the maximum matching algorithm to balance the repair load as much as possible. Assume that the input is the current set of faulty data blocks, the alternative node graph G and the source node graph S of the storage system, an integer k, representing the number of source nodes required for each task, and the output is a task set T, so that there is a k-regular generated subgraph H in G or the repair traffic is as balanced as possible, the maximum match M in the alternative node graph G, and a list of alternative nodes and source nodes allocated according to H and M. The steps of the algorithm are as follows Figure 3 As shown, specifically including:

[0065] The first step is initialization: create a repair task set T and a candidate task list candidate tasks.

[0066] Initialize an empty set T to store the selected tasks. Initialize a candidate task list candidate_tasks, which contains all the tasks that currently need to be repaired.

[0067] Step 2: Select a task: Select a task t from candidate_tasks and add it to T. Update the source node graph S.

[0068] Select a task t from candidate_tasks, add it to T, and update the source node graph S, that is, add edges related to t. The edges related to t are the connections between the source nodes and task t in the source node graph.

[0069] The third step is to check the condition: whether there is a k-regular generated subgraph H in S.

[0070] Check whether the updated source node graph has a k-regular generated subgraph H.

[0071] Step 4: If H exists, continue to select the task; otherwise, backtrack to the task selection process.

[0072] Step 5: When T reaches n, construct an alternative node graph G based on T and find the maximum matching M in G.

[0073] Step 6: Package the tasks in T and execute the repair tasks.

[0074] That is, if H exists, continue to select the next task; if not, backtrack and try to select another task t'.

[0075] The above process is repeated until the number of tasks in T reaches n or no tasks that meet the conditions can be found.

[0076] It should be noted that the backtracking mechanism of some embodiments of the present application means: if a task that meets the conditions cannot be found in a certain step, then backtrack to the previous step, cancel the selection of the previous step, and try to select another candidate task. If all candidate tasks cannot meet the conditions, record the current best choice and continue to try other combinations. The maximum matching of some embodiments of the present application means: when the number of tasks in T reaches n, construct an alternative node graph G based on T. Use an existing efficient algorithm to find the maximum matching M in G. Check whether M is a perfect match. If M is a perfect match, it is marked as completely balanced; if M is not a perfect match, it is marked as approximately balanced. Task packaging and execution of some embodiments of the present application: The tasks in T are packaged into a batch. According to the maximum matching M and k regularity, a subgraph H is generated to assign alternative nodes and source nodes, and execute the repair tasks of the batch.

[0077] The following is an illustrative description of the load balancing of heterogeneous environments in some embodiments of the present application.

[0078] In an actual distributed storage system, priority is usually given to serving foreground online applications, so resources such as network bandwidth occupancy and storage I / O often change dynamically, resulting in a heterogeneous repair environment. Some embodiments of the present application can set weights in the alternative node graph and the source node graph, and expand the alternative node graph and the source node graph so that they can be applied to heterogeneous environments.

[0079] First, the bandwidth information of each node needs to be collected in real time. In order to make the bandwidth data in the same range, the maximum and minimum normalization method can be used to normalize the bandwidth. The bandwidth weight of each node is calculated using the following formula:

[0080]

[0081] bandwidth(N i ) is the original value of the node bandwidth, minbandwidth is the minimum bandwidth value of all nodes, and maxbandwidth is the maximum bandwidth value of all nodes. In a heterogeneous environment, each edge in the replacement node graph and the source node graph will carry the normalized bandwidth weight information.

[0082] It should be noted that in a distributed storage system, a single faulty node often has a huge storage capacity, resulting in a large amount of data waiting to be repaired. Therefore, in most batches, some embodiments of the present application can find a perfect match in G, and many batches can find k regular generated subgraphs in S. For the remaining batches, some embodiments of the present application can still balance their repair loads almost completely.

[0083] In summary, as shown in Figure 4, some embodiments of the present application provide a method for processing repair tasks in a distributed storage system, which exemplarily includes: the first step is to find a set T of n repair tasks through a batch processing algorithm; the second step is that there exists a k-regular generated subgraph H (or called k-regular generated subgraph H) in the source node graph S corresponding to the repair task T; the third step is to build an alternative node graph G based on the repair task T; the fourth step is to find the maximum match M in G; the fifth step is to allocate alternative nodes and source nodes to perform the repair task according to the match M and the subgraph H.

[0084] It is not difficult to understand that some embodiments of the present application bring a series of significant effects by introducing a new data fault repair scheduling method. The following is a detailed description of these effects:

[0085] 1. Load Balancing

[0086] Upstream traffic balance: Some embodiments of the present application ensure balanced distribution of upstream traffic during the repair process by selecting appropriate source nodes and replacement nodes. This avoids overloading of some source nodes, thereby improving the overall stability and reliability of the system.

[0087] Downlink traffic balance: Some embodiments of the present application select alternative nodes through the maximum matching algorithm to ensure balanced distribution of downlink traffic and decoding load. This avoids performance bottlenecks caused by excessive load on some alternative nodes, thereby improving the processing capacity and efficiency of the system.

[0088] 2. Improve repair efficiency

[0089] Non-continuous strip packing: Some embodiments of the present application select strips from a large number of task queues to be repaired for packing, rather than packing by continuous sequence numbers. This method improves the flexibility of selecting repair tasks, reduces unnecessary waiting time, and thus improves repair efficiency.

[0090] Batch processing algorithm: The batch processing algorithm of some embodiments of the present application can find a set of repair tasks so that the uplink repair traffic is completely balanced or the difference is minimal. This allows each repair batch to fully utilize system resources, thereby speeding up the repair process.

[0091] 3. Resource optimization

[0092] The bandwidth weight calculation of some embodiments of the present application uses the maximum and minimum normalization method to calculate the bandwidth weight of each node, ensuring that network bandwidth resources are fully utilized during the repair process. This not only improves data transmission efficiency, but also reduces delays caused by insufficient bandwidth.

[0093] Load balancing strategy: Some embodiments of the present application ensure load balancing of source nodes through a deterministic selection algorithm, avoiding overloading of some nodes due to excessive use, which enables more reasonable allocation and utilization of system resources.

[0094] 4. System stability

[0095] Avoid single point overload: Some embodiments of the present application use load balancing strategies to avoid failure or service interruption caused by excessive load on a single node, which improves the overall stability and reliability of the system.

[0096] Reducing network congestion: Some embodiments of the present application reduce the occurrence of network congestion by optimizing the balanced distribution of upstream and downstream traffic, thereby improving the network transmission efficiency and stability of the system.

[0097] 5. Scalability and adaptability

[0098] Flexible repair task selection In some embodiments of the present application, the non-continuous stripe packaging method makes the selection of repair tasks more flexible and can adapt to distributed storage systems of different scales and load conditions.

[0099] Strong adaptability: In some embodiments of the present application, through the above-mentioned batch processing algorithm and maximum matching algorithm, the system can automatically adjust the repair strategy under different load conditions, thereby adapting to various application scenarios.

[0100] To summarize, some embodiments of the present application bring about the following comprehensive effects: improving system performance: improving the overall performance and efficiency of the system through load balancing and resource optimization; enhancing system stability: reducing the risk of single-point overload and network congestion, and enhancing the stability and reliability of the system; improving user experience: improving the user experience by improving repair efficiency and system stability, and reducing the inconvenience caused by data loss or repair delays; reducing operation and maintenance costs: reducing the operation and maintenance costs and management complexity of the system by optimizing resource utilization and reducing the occurrence of failures; supporting large-scale applications: supporting the application of large-scale distributed storage systems through flexible task selection and highly adaptable repair strategies.

[0101] The improvements of some embodiments of the present application include: non-continuous strip packing: different from the traditional method of packing multiple strips of continuous serial numbers into a batch, the new method is to select some strips from a large number of task queues to be repaired and pack these strips into a batch for processing. This helps to manage repair tasks more flexibly and reduce repair delays; maximum and minimum normalization: use the maximum and minimum normalization method to calculate the weight of each node bandwidth to balance the workload of each node during the repair process; maximum matching and perfect matching: construct a graph based on the task set and find the maximum matching in the graph. If it is a perfect match, the downlink repair traffic and decoding load are completely balanced; if not, an approximate balance is achieved; bipartite graph model: a bipartite graph model and a batch processing algorithm are used to pack the repair tasks into batches, and the source node is selected to balance the uplink repair traffic between nodes. We use another bipartite graph model and a maximum matching algorithm to select alternative nodes to balance their downlink repair traffic and decoding load. Some embodiments of the present application are aimed at load balancing in heterogeneous environments: the maximum and minimum normalization method is used to calculate the weight of each node bandwidth, and the weight is added when constructing the source node graph and the alternative node graph to balance the workload of each node. The bipartite graph model and maximum matching algorithm of some embodiments of the present application: How to use the bipartite graph model and batch processing algorithm to find a set of repair tasks to ensure that the upstream traffic is completely balanced or the difference is minimal. How to use another bipartite graph model and maximum matching algorithm to select an alternative node to ensure that the downstream traffic and decoding load are completely balanced or approximately balanced.

[0102] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for processing a repair task in a distributed storage system as described in any one of the above embodiments can be implemented.

[0103] Please refer to Figure 5 , Figure 5 The apparatus for processing a repair task in a distributed system provided in an embodiment of the present application is shown. It should be understood that the apparatus is similar to the above-mentioned Figure 1 The provided method embodiment corresponds to the various steps involved in the above method embodiment, and the specific functions of the device can refer to the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the operating system of the device. The device for processing repair tasks in the distributed system includes: a batch of pending tasks selection module 501, an alternative node graph G processing module 502, and a source node and alternative node allocation processing module 503.

[0104] The task selection module 501 for this batch of pending tasks is configured to obtain n repair tasks for this batch of pending tasks, wherein the n repair tasks are determined based on the principle of finding a regular subgraph from the source node graph, the degrees of source data vertices on the regular subgraph are equal, and the regular subgraph is used to ensure that each surviving node can provide services for at least one repair task of the same number or a similar number.

[0105] The alternative node graph G processing module 502 is configured to construct an alternative node graph G based on the n repair tasks and find a target match from the alternative node graph G, wherein the target match is used to ensure that different surviving nodes are assigned to provide services as much as possible for different tasks among the n repair tasks.

[0106] The alternative node allocation processing module 503 is configured to allocate source nodes and alternative nodes to each of the n repair tasks at least based on the regular subgraph and the target matching to complete data repair, wherein the alternative node graph and the source node graph are both represented by a bipartite graph model.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0108] like Figure 6 As shown, some embodiments of the present application provide an electronic device 700, including a memory 710, a processor 720, and a computer program stored on the memory 710 and executable on the processor 720, wherein the processor 720 can implement the method for processing repair tasks in a distributed storage system as described in any one of the above embodiments when reading and executing the program through a bus 730.

[0109] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0110] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the present embodiment may be used to execute the instructions in the memory 710 to implement Figure 1 The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0111] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0113] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0116] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A repair scheduling method in a distributed storage system, characterized in that: The method comprises: Obtain n repair tasks to be processed in this batch, wherein the n repair tasks are determined based on the principle of finding a regular subgraph from a source node graph, the degrees of source data vertices on the regular subgraph are equal, and the regular subgraph is used to ensure that each of the multiple surviving nodes selected as source nodes can provide services for at least one repair task of the same number or a similar number; Constructing a replacement node graph G based on the n repair tasks and finding a target match from the replacement node graph G, wherein the target match is used to ensure that different surviving nodes are assigned to provide services as much as possible for different tasks in the n repair tasks; At least according to the regular subgraph and the target matching, a source node and a replacement node are respectively assigned to each of the n repair tasks to complete data repair; Wherein, the replacement node graph and the source node graph are both represented by a bipartite graph model.

2. The method according to claim 1, characterized in that The source node graph is a bipartite graph, which is used to model the selection method of the source node. In the source node graph, any repair task uses a task vertex T i Representation, each source node of the repair task uses a source data vertex N j Characterizes that the edge (T i ,N j ) represents the source data vertex N j Selected as the task vertex T i The source node.

3. The method according to claim 2, characterized in that If the network of the distributed storage system is isomorphic, the determination principle for finding a regular subgraph from the source node graph is: each source data vertex is connected to the same number of task vertices to ensure that the degree of each source data vertex is equal.

4. The method according to any one of claims 1 to 3, characterized in that The constructing of the alternative node graph G based on the n repair tasks includes: obtaining a set of n repair tasks in the batch T = {T1, T2, ..., T n }, the selection method of the replacement node is modeled as a bipartite graph G = (T, A, N) to obtain the replacement node graph, where each T i ∈T, any repair task in the alternative node graph uses a task vertex T i Representation, each replacement node of the repair task adopts a replacement vertex N j Characterizes that the edge (T i ,N j ) represents the alternative vertex N j Selected as the task vertex T i Alternative node.

5. The method according to claim 4, characterized in that If the network of the distributed storage system is isomorphic, the existence of a target match in the alternative node graph G means finding a set of edges in the corresponding alternative node graph and this set of edges includes all alternative vertices in the graph, so that each alternative vertex is connected to exactly one task vertex.

6. The method according to claim 1, characterized in that The step of obtaining the n repair tasks to be processed in this batch includes: Initialize the set T; Initialize a candidate task list candidate_tasks, wherein the candidate task list candidate_tasks contains all repair tasks at the current moment; Select one repair task from the candidate task list candidate_taskscandidate_tasks one by one, and repeat the following operations until the number of repair tasks in the set T reaches n or no repair task that meets the conditions can be found: Add the repair task t selected this time to the set T; Add the edges related to the repair task t to the corresponding source node graph S to obtain an updated source node graph, check whether the updated source node graph contains the regular subgraph, if so, continue to select the next repair task from the candidate task list candidate_tasks; if not, delete the repair task t from the set T, and try to select another repair task t' from the candidate task list candidate_taskscandidate_tasks; The constructing a replacement node graph G based on the n repair tasks and finding a perfect match from the replacement node graph G comprises: When the number of tasks in the set T reaches n, constructing the replacement node graph G based on all repair tasks in the set T; The target match is found in the alternative node graph G.

7. The method according to claim 6, characterized in that The step of at least generating a subgraph according to the k regularization and matching the target to respectively assign a source node and a replacement node to each of the n repair tasks to complete data repair includes: Packing the n repair tasks in the set T into a batch; Allocation of replacement nodes and source nodes to each repair task in the batch according to the regular subgraph, the target matching and the bandwidth weight.

8. The method according to claim 7, characterized in that The allocating a replacement node and a source node for each repair task in the batch according to the regular subgraph, the target matching and the bandwidth weight includes: If the network of the distributed storage system is heterogeneous, then when allocating the replacement node and the source node, a surviving node with a larger bandwidth weight is preferentially selected as the replacement node or the source node, wherein the bandwidth weight is obtained based on the bandwidth information of each surviving node collected in real time and normalized using the maximum and minimum normalization method.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method for processing a repair task in a distributed storage system as described in any one of claims 1 to 8 can be implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can implement the method for processing repair tasks in a distributed storage system as described in any one of claims 1 to 8.

Images