Node parallel repair method for distributed storage system based on regeneration code

Abstract

The invention belongs to the technical field of computer distributed storage, and particularly discloses a node parallel repair method for a distributed storage system based on regenerative codes, and the method comprises the steps: respectively initializing a parallel erasure coding directed acyclic graph pECDAG of each strip in a to-be-repaired cluster; dyeing vertexes in the initialized pECDAG on the basis of nodes in the to-be-repaired cluster to obtain an initial repairing scheme; the method comprises the steps that a repair scheme optimization process is repeatedly executed on an initial repair scheme until the maximum repair load of the repair scheme is reduced to the minimum, a final repair scheme is obtained, and the repair scheme optimization process comprises the steps that a strip which has the maximum influence on the repair load in a current repair scheme is judged, re-dyeing the top points in the pECDAG of the strip with the maximum influence on the repair load, wherein the top points are in the pECDAG of the strip with the maximum influence on the repair load; and repairing the to-be-repaired cluster based on the final repairing scheme.

Description

Technical Field

[0001] The present application belongs to the field of computer distributed storage technology, and more specifically, relates to a node parallel repair method for a distributed storage system based on a regeneration code. Background Art

[0002] Erasure codes have been widely used in actual distributed storage systems to provide fault tolerance. Compared with traditional backup technologies, they have lower redundant storage overhead. Among the many erasure codes, Reed-Solomon (RS) coding is the most common coding method and has been widely deployed in actual production environments. However, RS codes incur high repair costs, namely the network transmission bandwidth caused by repairing failed blocks. Since repairing failed blocks requires obtaining k surviving blocks from the system, the network bandwidth is increased. Therefore, some coding methods have been proposed to reduce the repair bandwidth, such as regenerating codes, locally repairable codes LRC (locally repairable codes) and piggybacking codes. In particular, minimum-storage regenerating (MSR) codes have been theoretically proven to be the optimal repair bandwidth, which can minimize the bandwidth for repairing a single node failure while maintaining the same minimum storage redundancy as RS codes.

[0003] Although MSR codes minimize the repair bandwidth in theory, they are still limited in actual repair performance because the amount of data that the node needs to obtain from other available nodes exceeds the amount of lost data. In other words, the bandwidth amplification phenomenon still exists. Currently, RSPipe has been studied to decompose the RS-coded repair operation into multiple partial repair sub-operations by parallelizing and load balancing the repair process. These sub-operations are executed in parallel on different nodes, and the partially repaired blocks are merged into the final decoded blocks, thereby improving the repair efficiency and reducing bandwidth consumption. However, the repair of RS coding satisfies the additive associativity of linear combinations, so the repair operation can be decomposed; in contrast, MSR coding has a different mathematical structure from RS coding. The repair of MSR coding requires solving a linear combination system and cannot be directly decomposed.

[0004] In the prior art, a pruning-based heuristic algorithm is used to schedule single-block parallel repair of MSR codes. However, the algorithm has the following shortcomings: 1. The pruning-based heuristic algorithm has no running time guarantee. It takes a long time to generate a scheduling strategy for parallel repair of MSR codes with large parameters, resulting in low repair efficiency; 2. The repair scheduling of a single block has already taken a long time, and further node repair will become more complicated, resulting in low repair efficiency; 3. In the node repair scenario, the simple superposition of single-block repair will cause load imbalance, and there is load optimization space between stripes, resulting in low repair efficiency. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of the present application is to provide a node parallel repair method for a distributed storage system based on regeneration codes, aiming to solve the problem of low repair efficiency of the existing node repair method.

[0006] To achieve the above objectives, in a first aspect, the present application provides a node parallel repair method for a distributed storage system based on a regeneration code, comprising: Initialize the parallel erasure-coding directed acyclic graph (pECDAG) of each stripe in the cluster to be repaired; Coloring the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired to obtain an initial repair plan; Repeating the repair scheme optimization process for the initial repair scheme until the maximum repair load of the repair scheme is reduced to a minimum, thereby obtaining a final repair scheme, wherein the repair scheme optimization process comprises: determining the band that has the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the band that has the greatest impact on the repair load; The cluster to be repaired is repaired based on the final repair solution.

[0007] This application associates each vertex with a color through pECDAG, and each color corresponds to a storage node, so that the node is responsible for generating or storing all sub-blocks related to the same color vertex, maximizing the affinity ratio, and then continuously adjusting the pECDAG coloring scheme of the stripe to minimize the maximum repair load of the repair scheme. By combining parallel repair within and between stripes, node repair load balancing is achieved, and node repair efficiency is improved, so that lost data can be repaired as quickly as possible, improving repair performance and system reliability.

[0008] According to a node parallel repair method for a distributed storage system based on a regeneration code provided by the present application, the parallel erasure coding directed acyclic graph pECDAG of each stripe in the cluster to be repaired is initialized separately, including: For the pECDAG of each strip in the cluster to be repaired, all leaf vertices are bound to the preset color of the node where the leaf vertex is located, and the root vertex is bound to the preset color of the hot standby node assigned to the root vertex, and all intermediate vertex values ​​are empty.

[0009] According to a node parallel repair method for a distributed storage system based on regeneration codes provided by the present application, the vertices in the initialized pECDAG are colored based on the nodes in the cluster to be repaired to obtain an initial repair plan, including: Topologically sorting the intermediate vertices in the initialized pECDAG; In topological sorting order, color the intermediate vertices with the colors of their child vertices.

[0010] According to a node parallel repair method for a distributed storage system based on a regenerating code provided by the present application, the step of coloring an intermediate vertex into the color of a child vertex of the intermediate vertex includes: If the middle vertex has sub-vertices of different colors, determine the first color with the lowest repair load or the smallest repair bandwidth after coloring, and color the middle vertex into the first color.

[0011] According to a node parallel repair method for a distributed storage system based on regeneration codes provided by the present application, the determination of the stripe with the greatest impact on the repair load in the current repair scheme includes: Determine the node with the largest repair load in the current repair scheme; Determine the stripe with the largest repair flow that contributes to the repair load of the node with the largest repair load, as the stripe with the greatest impact on the repair load.

[0012] According to a node parallel repair method for a distributed storage system based on a regeneration code provided by the present application, the method further includes: Initialize a global flow table, wherein the global flow table includes the number of sub-blocks sent and received by each node when the node is repaired; After each execution of the repair solution optimization process, the data in the global flow table is updated.

[0013] In a second aspect, the present application provides a node parallel repair device for a distributed storage system based on a regeneration code, comprising: An initialization module is used to initialize the parallel erasure coded directed acyclic graph pECDAG of each stripe in the cluster to be repaired; A coloring module, used to color the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired, to obtain an initial repair plan; An optimization module is used to repeatedly perform a repair scheme optimization process on the initial repair scheme until the maximum repair load of the repair scheme is reduced to a minimum, thereby obtaining a final repair scheme, wherein the repair scheme optimization process includes: determining the band with the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the band with the greatest impact on the repair load; A repair module is used to repair the cluster to be repaired based on the final repair solution.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the node parallel repair method for a distributed storage system based on a regeneration code as described in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the node parallel repair method for a distributed storage system based on a regeneration code described in the first aspect or any possible implementation of the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product, which, when running on a processor, enables the processor to execute the node parallel repair method for a distributed storage system based on a regeneration code as described in the first aspect or any possible implementation of the first aspect.

[0017] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0018] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: Through pECDAG, each vertex is associated with a color, and each color corresponds to a storage node, so that the node is responsible for generating or storing all sub-blocks related to the same color vertex, maximizing the affinity ratio. Then, by continuously adjusting the pECDAG coloring scheme of the stripe, the maximum repair load of the repair scheme is minimized. By combining parallel repair within and between stripes, node repair load balancing is achieved, and node repair efficiency is improved, so that lost data can be repaired as quickly as possible, improving repair performance and system reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a flowchart of a node parallel repair method for a distributed storage system based on regeneration codes provided in an embodiment of the present application; Figure 2 This is a flow chart of the intra-strip parallel repair dyeing method provided in an embodiment of the present application; Figure 3 It is a flow chart of a node repair method combining intra-strip and inter-strip parallelism provided in an embodiment of the present application; Figure 4 This is a schematic diagram of node parallel repair provided by an embodiment of the present application; Figure 5 It is a schematic diagram of a single-strip intra-parallel repair strategy generation process based on affinity provided by an embodiment of the present application; Figure 6 It is a schematic diagram of a node repair load adjustment optimization process based on intra-strip and inter-strip parallelism provided in an embodiment of the present application; Figure 7 It is a structural schematic diagram of a node parallel repair device for a distributed storage system based on a regeneration code provided in an embodiment of the present application; Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0023] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0024] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0025] First, the following contents are introduced: In a storage system based on Clay coding, for each stripe, all k data are encoded using Clay codes to generate nk check blocks. These n blocks form a stripe. When a block in the stripe is lost, the lost block can be repaired by using partial data from the remaining n-1 surviving blocks.

[0026] Next, combine Figure 1-Figure 6 The node parallel repair method for a distributed storage system based on a regeneration code provided in an embodiment of the present application is introduced.

[0027] Figure 1 is a flow chart of a node parallel repair method for a distributed storage system based on a regeneration code provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: Step 100, respectively initialize the parallel erasure coded directed acyclic graph pECDAG of each stripe in the cluster to be repaired; This application proposes a collaborative design for full-node repair that takes into account both intra-stripe and inter-stripe parallel repair. The main idea is: first, based on an affinity-based heuristic algorithm, an intra-stripe parallel repair plan is generated for each block to be repaired, and then the intra-stripe parallel repair plan of certain blocks is adjusted according to the aggregated repair traffic distribution.

[0028] Affinity,means that in a pECDAG, an intermediate vertex or root vertex shares color with some child vertices, and the ratio of the number of vertices with affinity to the total number of intermediate vertices and root vertices in pECDAG is defined as the affinity ratio.

[0029] pECDAG, which refers to a directed acyclic graph , describes an erasure coding operation, where V is the vertex set, E is the edge set, and a vertex , representing the sub-blocks stored in the vertices (for ) or intermediate results, Represents a sub-block It is used to calculate The input is a linear combination of is called The child nodes of .

[0030] pECDAG associates each vertex with a color, which corresponds to a storage node, making the node responsible for generating or storing all sub-blocks related to the same color vertex.

[0031] Optionally, the color corresponding to each node can be preset manually.

[0032] This application first creates and initializes pECDAG for each band in the cluster to be repaired to facilitate subsequent staining.

[0033] Step 110, coloring the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired to obtain an initial repair solution; For each stripe, an initial parallel repair plan is generated for each failed block by staining its corresponding pECDAG.

[0034] Specifically, a color may be preset in advance for each node in the cluster to be repaired, and then the vertices in the initialized pECDAG are colored based on these colors. After all vertices are colored, an initial repair plan is obtained.

[0035] Step 120, repeatedly performing the repair scheme optimization process on the initial repair scheme until the maximum repair load of the repair scheme is reduced to the minimum, and obtaining a final repair scheme, wherein the repair scheme optimization process includes: determining the band that has the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the band that has the greatest impact on the repair load; After the initial repair scheme is obtained, the final repair scheme is obtained by continuously determining the band with the greatest impact on the repair load in the current repair scheme and re-staining the vertices in the pECDAG of the band with the greatest impact on the repair load until the maximum repair load of the repair scheme is reduced to the minimum.

[0036] Step 130: Repair the cluster to be repaired based on the final repair solution.

[0037] The present application provides a node parallel repair method for a distributed storage system based on a regeneration code. Each vertex is associated with a color through pECDAG, and each color corresponds to a storage node, so that the node is responsible for generating or storing all sub-blocks related to the same color vertex, maximizing the affinity ratio, and then continuously adjusting the pECDAG coloring scheme of the stripe to minimize the maximum repair load of the repair scheme. By combining parallel repair within and between stripes, node repair load balancing is achieved, and node repair efficiency is improved, so that lost data can be repaired as quickly as possible, thereby improving repair performance and system reliability.

[0038] In some embodiments, step 100 specifically includes: Step 1001: for each strip of pECDAG in the cluster to be repaired, all leaf vertices are bound to the preset colors of the nodes where the leaf vertices are located, and the root vertex is bound to the preset color of the hot standby node assigned to the root vertex, and all intermediate vertex values ​​are empty.

[0039] First, the pECDAG corresponding to each strip can be constructed according to the given encoding method and parameters. The initial coloring is: all leaf vertices are bound to the color of the node they are located in, the root vertex is bound to the color of the standby node it is assigned to, and all intermediate vertex values ​​are empty, and no data transmission is generated.

[0040] In some embodiments, step 110 specifically includes: Step 1101, topologically sorting the intermediate vertices in the initialized pECDAG; Step 1102: Color the middle vertices in the colors of their child vertices in order of topological sorting.

[0041] Figure 2 : is a flow chart of the parallel repair dyeing method in a strip provided in an embodiment of the present application, such as Figure 2 As shown in the figure, after initializing pECDAG, all intermediate vertices are first topologically sorted. Topological sorting is a linear sorting algorithm for directed acyclic graphs. This sorting is for each directed edge u→ v , in the sorting results, vertex u will appear in the vertex v Before.

[0042] After topological sorting, the graph vertices are traversed and colored according to the topological sorting order. When coloring, only the color of its child vertices is considered, because only in this way can its affinity ratio increase.

[0043] Optionally, after the vertex is colored, the data in the flow table can be updated synchronously so that a decision can be made based on the current result when processing the next vertex. The flow table is used to record the data transmission generated by the current partial coloring result.

[0044] In some embodiments, coloring the intermediate vertex to the color of the child vertex of the intermediate vertex in step 1102 specifically includes: If the middle vertex has sub-vertices of different colors, determine the first color with the lowest repair load or the smallest repair bandwidth after coloring, and color the middle vertex with the first color.

[0045] If the middle vertex has sub-vertices of different colors, the set of corresponding colors of its sub-vertices can be statistically obtained, and the color that can minimize the maximum repair load or repair bandwidth is selected to color it.

[0046] In some embodiments, determining the stripe with the greatest impact on the repair load in the current repair solution in step 120 specifically includes: Step 1201, determining the node with the largest repair load in the current repair scheme; Step 1202, determine the stripe with the largest repair traffic that contributes to the repair load of the node with the largest repair load, as the stripe with the greatest impact on the repair load.

[0047] Figure 3 : is a flowchart of a node repair method combining intra-strip and inter-strip parallelism provided in an embodiment of the present application, such as Figure 3 As shown, for each stripe, the initial parallel repair plan is generated for each failed block by coloring its corresponding pECDAG, and a single-block parallel repair schedule is performed to determine the color of the intermediate nodes, while the color of the root node is selected from multiple hot standby nodes in a round-robin manner.

[0048] In each optimization process, first find the node with the largest repair load , then locate all the files stored in The block on it, find all its corresponding strips , calculated in each strip The repair traffic contributed to the maximum repair load, and find the stripe that contributes the most to the load , if there are multiple strips with the same contribution, one is randomly selected, followed by Regenerate the affinity-based intra-strip parallel repair plan. If the maximum load before and after the process can be reduced, continue to the next optimization process, otherwise, the optimization process ends.

[0049] There may be multiple nodes with the largest repair load, and the node with the largest repair load may have multiple stripes with the same and largest contribution. For example, the repair load of nodes N1 and N2 is 8, among which the contribution load of the largest contribution stripe S1 on N1 is 4, the contribution load of the largest contribution stripe S1 on N2 is 3, and the contribution load of S2 is 3, then S1 contribution load = 3+4=7, S2 contribution load = 3, and then sort by contribution load, that is, {S1, S2}, and then re-color in the order of S1, S2.

[0050] In some embodiments, the method further comprises: Step 140, initializing a global flow table, the global flow table including the number of sub-blocks sent and received by each node when the node is repaired; Step 150: after each execution of the repair solution optimization process, update the data in the global flow table.

[0051] Optionally, a global flow table may be maintained, which is a two-dimensional table that maintains the number of sub-blocks sent and received by each node during node repair.

[0052] Optionally, the global flow table may be initialized before being used.

[0053] When coloring the vertices in pECDAG, after each vertex is colored, the data in the global flow table can be synchronously updated so that a decision can be made based on the current result when processing the next vertex.

[0054] Each time the repair plan optimization process is executed, after determining the stripe with the greatest impact on the repair load in the current repair plan, the load of the stripe can be removed from the global flow table, and then an affinity-based intra-stripe parallel repair plan is regenerated for the stripe and the global flow table is updated. If the maximum load can be reduced, the next optimization process will continue, otherwise, the optimization process ends.

[0055] Figure 4 is a schematic diagram of node parallel repair provided by an embodiment of the present application, such as Figure 4 As shown, in one embodiment of the present application, ~ is a data node in the cluster, ~ It is a hot standby node in the cluster. ~ Three strips, now If a node fails, it needs to be repaired. The repair steps are as follows: 1a. Initialize the repair scheme and global flow table according to the encoding method and the sequence of blocks to be repaired of the failed node.

[0056] 2a. Traversal ~ Stripe, generate a single-stripe parallel repair plan for the failed blocks of the stripe according to steps 3a to 8a.

[0057] 3a. Generate the corresponding pECDAG of the repaired failed block of the stripe.

[0058] Figure 5 FIG. 1 is a schematic diagram of a single-strip intra-parallel repair strategy generation process based on affinity provided in an embodiment of the present application. Figure 5 Shown: Node ~ They have corresponding colors respectively, and the directed graph is Corresponding to pECDAG, the vertex to be dyed in pECDAG is ,in Block to be repaired of four sub-blocks.

[0059] 4a. Topological sorting of the vertices to be dyed in pECDAG is performed, and the result is .

[0060] 5a. Traverse the vertices to be colored in topological order and select a color for the vertex according to steps 6a to 7a.

[0061] like Figure 5 As shown, it is the vertex Assign color, assuming that the vertex Has been assigned to the node The corresponding color is only Towards A sub-block is sent (i.e. and is set to 1, and all other values ​​are 0).

[0062] 6a. Try to dye the current vertex with the color of its child vertices, and calculate the maximum repair load and repair bandwidth after dyeing.

[0063] like Figure 5 As shown, the vertex There are two sub-vertices: and ,in Assigned a representative The color, Assigned a representative There are two options for color: 1b. Try to Assigning Representatives After constructing the local flow table based on this coloring scheme, it is found that both the maximum repair load and the repair bandwidth remain unchanged.

[0064] 2b. Try to Assigning Representatives After constructing the local flow table based on this coloring scheme, it is found that the maximum repair load remains unchanged, but the repair bandwidth increases to 2. The reason is that under this choice, an additional The corresponding sub-block from Transfer to . 7a. Compare all choices and give priority to the one with the lowest repair load. If the repair loads are the same, choose the one with the smallest repair bandwidth. If both are the same, randomly choose a color.

[0065] Finally Assigned representatives color.

[0066] 8a. Update the flow table and select the next vertex to color it according to the topological sort.

[0067] 9a. After initializing the parallel repair scheme for all stripes, adjust the parallel repair scheme through multiple iterations based on the global flow table.

[0068] 10a. In each iteration, first find the node with the largest repair load .

[0069] Figure 6 is a schematic diagram of a node repair load adjustment optimization process based on intra-strip and inter-strip parallelism provided in an embodiment of the present application, such as Figure 6 As shown, the node with the largest repair load in the cluster is , whose load is 10 sub-blocks.

[0070] 11a. Find the stripe with the largest contribution on the node with the largest repair load.

[0071] like Figure 6 As shown, for the node The block with the largest load contribution is and , both sizes are 4 sub-blocks.

[0072] 12a. Sort by load contribution to get a stripe sequence .

[0073] 13a. Remove the bands in the sequence in the global flow table and re-stain the bands one by one in the order of contribution.

[0074] 14a. After coloring is completed, the global flow table is updated. If the maximum load can be reduced, the next adjustment is continued. Otherwise, the adjustment is withdrawn, the strategy generation is ended and the result is obtained.

[0075] like Figure 6 As shown, the maximum repair load is reduced to 8, the adjustment is successful, and the next adjustment is continued.

[0076] Figure 7 is a schematic diagram of the structure of a node parallel repair device for a distributed storage system based on a regeneration code provided in an embodiment of the present application, such as Figure 7 As shown, the system includes an initialization module 710, a dyeing module 720, an optimization module 730 and a repair module 740, wherein: An initialization module 710 is used to initialize the parallel erasure coded directed acyclic graph pECDAG of each stripe in the cluster to be repaired respectively; A coloring module 720 is used to color the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired to obtain an initial repair solution; The optimization module 730 is used to repeatedly perform the repair scheme optimization process on the initial repair scheme until the maximum repair load of the repair scheme is reduced to the minimum, and obtain the final repair scheme, the repair scheme optimization process comprising: determining the stripe with the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the stripe with the greatest impact on the repair load; The repair module 740 is used to repair the cluster to be repaired based on the final repair solution.

[0077] It should be understood that the above-mentioned system is used to execute the methods in the above-mentioned embodiments. The implementation principles and technical effects of the corresponding program modules in the system are similar to those described in the above-mentioned methods. The working process of the system can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0078] Based on the method in the above embodiment, Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, an embodiment of the present application provides an electronic device, which may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the node parallel repair method for the distributed storage system based on the regeneration code in the above embodiment.

[0079] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the node parallel repair method for a distributed storage system based on a regeneration code described in each embodiment of the present application.

[0080] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the node parallel repair method for a distributed storage system based on regeneration code in the above embodiment.

[0081] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the node parallel repair method for a distributed storage system based on regeneration code in the above embodiment.

[0082] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0083] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0084] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0085] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0086] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A node parallel repair method for a distributed storage system based on regeneration codes, characterized in that: include: Initialize the parallel erasure coding directed acyclic graph pECDAG of each stripe in the cluster to be repaired respectively; Coloring the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired to obtain an initial repair plan; Repeating the repair scheme optimization process for the initial repair scheme until the maximum repair load of the repair scheme is reduced to a minimum, thereby obtaining a final repair scheme, wherein the repair scheme optimization process comprises: determining the band that has the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the band that has the greatest impact on the repair load; The cluster to be repaired is repaired based on the final repair solution.

2. The node parallel repair method for a distributed storage system based on regeneration codes according to claim 1, characterized in that: Initializing the parallel erasure coded directed acyclic graph pECDAG of each stripe in the cluster to be repaired respectively includes: For the pECDAG of each strip in the cluster to be repaired, all leaf vertices are bound to the preset color of the node where the leaf vertex is located, and the root vertex is bound to the preset color of the hot standby node assigned to the root vertex, and all intermediate vertex values ​​are empty.

3. The node parallel repair method for a distributed storage system based on regeneration codes according to claim 2, characterized in that: The step of coloring the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired to obtain an initial repair solution includes: Topologically sorting the intermediate vertices in the initialized pECDAG; In topological sorting order, color the intermediate vertices with the colors of their child vertices.

4. The node parallel repair method for a distributed storage system based on regeneration codes according to claim 3, characterized in that: The step of coloring the intermediate vertex into the color of the child vertex of the intermediate vertex includes: If the middle vertex has sub-vertices of different colors, determine the first color with the lowest repair load or the smallest repair bandwidth after coloring, and color the middle vertex into the first color.

5. The node parallel repair method for a distributed storage system based on regeneration codes according to claim 1, characterized in that: The determining of the stripe with the greatest impact on the repair load in the current repair solution includes: Determine the node with the largest repair load in the current repair scheme; Determine the stripe with the largest repair flow that contributes to the repair load of the node with the largest repair load, as the stripe with the greatest impact on the repair load.

6. The node parallel repair method for a distributed storage system based on regeneration codes according to claim 1, characterized in that: The method further comprises: Initialize a global flow table, wherein the global flow table includes the number of sub-blocks sent and received by each node when the node is repaired; After each execution of the repair solution optimization process, the data in the global flow table is updated.

7. A node parallel repair device for a distributed storage system based on regeneration code, characterized in that: include: An initialization module is used to initialize the parallel erasure coded directed acyclic graph pECDAG of each stripe in the cluster to be repaired; A coloring module, used to color the vertices in the initialized pECDAG based on the nodes in the cluster to be repaired, to obtain an initial repair plan; An optimization module is used to repeatedly perform a repair scheme optimization process on the initial repair scheme until the maximum repair load of the repair scheme is reduced to a minimum, thereby obtaining a final repair scheme, wherein the repair scheme optimization process includes: determining the band with the greatest impact on the repair load in the current repair scheme, and re-coloring the vertices in the pECDAG of the band with the greatest impact on the repair load; A repair module is used to repair the cluster to be repaired based on the final repair solution.

8. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the node parallel repair method for a distributed storage system based on regeneration code as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is enabled to execute the node parallel repair method for a distributed storage system based on regeneration codes as described in any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a processor, the processor is enabled to execute the node parallel repair method for a distributed storage system based on regeneration codes as described in any one of claims 1 to 6.

Images