Data recovery method, distributed storage system, equipment, medium and product

Abstract

The invention discloses a data restoration method, a distributed storage system, equipment, a medium and a product, and relates to the technical field of data storage, to-be-stored data is obtained and is subjected to block processing to obtain at least one data block, then the stored data of the at least one data block is subjected to mapping processing to obtain a multivariate polynomial coefficient, and the multivariate polynomial coefficient is subjected to data restoration; and completing the work of storing the storage data to the storage node while constructing the minimum storage regeneration code. Data symbols (storage data) are mapped into coefficients of a multivariate polynomial instead of traditional matrix elements, and repair processing in the subsequent data repair process is facilitated. And reconstructing a multivariate polynomial from the polynomial coefficient of the help node by using a corresponding multivariate polynomial interpolation mode to complete restoration. And by utilizing the low complexity characteristic of a polynomial interpolation mode, the high calculation load of matrix inversion is avoided, the time complexity of the repair process is reduced to # imgabs0 #, and the repair efficiency is remarkably improved.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and particularly relates to a data repair method, a distributed storage system, a device, a medium, and a product. Background Art

[0002] In a distributed storage system, the minimal storage regenerating code (MSR Code) is used to repair the amount of data downloaded from helper nodes for a failed node corresponding to the stored data. The conventional MSR code uses matrix inversion operations, resulting in a high time complexity during the repair process of the failed node.

[0003] Therefore, how to reduce the time complexity is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a data repair method, a distributed storage system, a device, a medium, and a product to at least solve the problem of high time complexity in related technologies.

[0005] This application provides a data repair method, including: Obtain the data to be stored, and perform block processing on the data to be stored to obtain at least one data block; Perform mapping processing on the stored data of at least one data block to obtain multivariate polynomial coefficients, construct a minimal storage regenerating code, and determine the stored data corresponding to at least one storage node; If there is a failed node among at least one storage node, use the multivariate polynomial interpolation method corresponding to the minimal storage regenerating code to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes among at least one storage node to repair the stored data of the failed node.

[0006] This application also provides a distributed storage system, including a repair node and at least one storage node; The repair node is used to execute the steps of the above-mentioned data repair method to repair the stored data of the failed node in the storage node.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above data repair methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above data repair methods are implemented.

[0009] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of any of the above data repair methods.

[0010] Through the present application, on the one hand, in the encoding process of the minimum storage regenerating code, the data to be stored is obtained, divided into at least one data block, and then the stored data of at least one data block is mapped to obtain multivariate polynomial coefficients, so as to construct the minimum storage regenerating code while completing the work of storing the stored data to the storage node. It realizes mapping the data symbols (stored data) to the coefficients of the multivariate polynomial, rather than the traditional matrix elements, which is convenient for the repair process in subsequent data repair. On the other hand, if there is a failed node, the corresponding multivariate polynomial interpolation method is used to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes to complete the repair. The repair of failed nodes in the conventional technical solution usually relies on Gaussian elimination or matrix decomposition, and the time complexity of these methods is relatively high (usually ), and through the polynomial interpolation method in the present application, that is, using the low complexity characteristic of the polynomial interpolation method, the high computational load of matrix inversion is avoided, and the time complexity of the repair process is reduced to , significantly improving the repair efficiency. Therefore, it can solve the technical problem of relatively high time complexity in the repair process of the conventional technical solution, achieve the technical effect of mapping the data symbols (stored data) to the coefficients of the multivariate polynomial, combining the multivariate polynomial interpolation method, and realizing the efficient repair of failed nodes and reducing the time complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] Figure 1 It is a flowchart of a data repair method provided by an embodiment of the present application; Figure 2 It is a structural diagram of a distributed storage system provided by an embodiment of the present application; Figure 3 It is a structural diagram of a data repair device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0014] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0015] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0016] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the data repair method depends, the specific application environment architecture or specific hardware architecture will be described herein.

[0017] To improve data storage reliability, a distributed storage system disperses a large amount of data and stores it on multiple servers (i.e., nodes), which are connected through a network. Since these nodes are usually distributed in different geographical locations and are independent of each other, the risk of data loss caused by natural disasters or hardware failures can be effectively reduced. In a distributed storage system, information related to a data file (message) is dispersed and stored on each node in the network in a specific manner, enabling end-users to retrieve the stored data by accessing some of the nodes. A common practice is to use erasure codes, such as using a maximum distance separable code (MDS Code) like Reed-Solomon code (RS Code), that is, achieving the maximum error correction ability under a given redundancy. This method can reduce network congestion and enhance the resilience of the system in case of node failures. Let be the total size of the data file, measured by the number of symbols in a finite field (with size ). Specifically, if the initial data is divided into blocks, after MDS code encoding, the generated data blocks are respectively stored in Among the storage nodes, at this time the system can tolerate any node failures, and the data collector can repair the entire message by connecting to any surviving nodes (which can also be partial nodes of the surviving nodes, i.e., helper nodes).

[0018] In distributed storage, the effective repair corresponding to a failed node is measured by the repair bandwidth, that is, the total amount of data downloaded from the helper nodes to repair the failed node. In a distributed storage system, there are repair nodes and storage nodes, with at least one storage node. The repair nodes are located within the server, specifically, the controller within the server performs the repair. The original file is divided into data blocks, and data blocks are encoded to obtain storage nodes, and the original file is distributed to storage nodes for storage. In the conventional technical solution during the repair process, a generation matrix is generated based on the array code, that is: ; wherein, the linear array code of ( ) is called the MDS array code. If for any , there is , if , then the MDS array code is called systematic, where represents the gear matrix on . Obviously, when the number of sub-packets , the MDS array code is the traditional MDS code.

[0019] During the process of constructing the matrix, it is necessary to select elements that meet specific conditions to construct the generation matrix, which may be difficult to achieve in actual implementation, that is, to prove that any column blocks of the corresponding generation matrix are linearly independent based on the combinatorial nullstellensatz.

[0020] Data storage requires constructing a generation matrix , and repair requires solving the linear equation system , that is, solving out the number of helper nodes, where corresponds to a invertible sub-matrix of , and corresponds to the data downloaded from the helper nodes. Its complexity is obtained through matrix inversion or Gaussian elimination, and the time complexity is . The current matrix inversion operation has a relatively high time complexity during the repair process of failed nodes, but the repair time is relatively long. The data repair method provided by this application can solve the above technical problems.

[0021] Figure 1 A flowchart of a data repair method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: S11: Acquire data to be stored, and process the data to be stored into blocks to obtain at least one data block; S12: Mapping the storage data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code, and determining storage data corresponding to at least one storage node; S13: If there is a failed node in at least one storage node, a multivariate polynomial interpolation method corresponding to a minimum storage regeneration code is used to reconstruct a multivariate polynomial from polynomial coefficients of a helper node in at least one storage node to repair storage data of the failed node.

[0022] Specifically, the data to be stored is obtained based on the size of the original file. The data to be stored is processed into blocks to obtain at least one data block. The block processing method of the data block here is the same as the conventional method, or it can be different, which is not limited here and can be set according to actual conditions.

[0023] In step S12, the stored data of at least one data block is mapped to obtain multivariate polynomial coefficients. It should be noted that the conventional encoding process is to encode the stored data of at least one data block to generate The encoding process uses a linear combination method. Conventional technical solutions, for example: construct a sub-packet number of of MSR code, the total size of the data file to be stored The information data is stored in Symmetric Matrix That is, the upper triangular part of each matrix can be stored separately information symbols, the corresponding encoding matrix is ​​a The matrix ,in is a The matrix of is a Diagonal matrix, and satisfies: Any The rows are linearly independent; Any The rows are linearly independent; of The diagonal elements are different.

[0024] In this embodiment, the stored data is mapped to obtain the coefficients of the multivariate polynomial. For example, in the case of single variable polynomial coefficient encoding, the original data is divided into blocks: ; Construct a polynomial to map the data into polynomial coefficients: . Generate storage nodes. Here, select different assignment points , calculate the polynomial values, , and each storage node stores a pair ( ). represents the -th storage node, represents the assignment point corresponding to the -th storage node.

[0025] If it is a multivariate polynomial coefficient encoding, it is necessary to map the parameters of the stored data on the original matrix into the coefficients of the multivariate polynomial. The coefficients follow the permutation principle of the output variables, that is, they are invariant under the permutation of the output variables.

[0026] In some embodiments, when the multivariate polynomial is a ternary polynomial, the coefficients corresponding to the ternary polynomial include a first coefficient and a second coefficient; storing the stored data in at least one multivariate polynomial coefficient includes: Determine the number of coefficients of the first coefficient and the second coefficient; Store the stored data of at least one data block according to the number of coefficients corresponding to the first coefficient and the second coefficient respectively.

[0027] Specifically, taking the three-variable polynomial as an example, when , represents the tuple information corresponding to the variables of the multivariate polynomial, and the corresponding number of sub-packets is , and the total size of the data file of the data to be stored can be obtained as . The corresponding three-variable polynomial is: ; Among them, , , correspondingly represent the , -th power data of the variables corresponding to , ; it can be obtained that there are respectively different , corresponding to the first coefficient and the second coefficient. Thus, the polynomial different coefficients can be used to store the corresponding data symbol information.

[0028] , the storage node stores the polynomial as: ; Among them, Indicates that the first variable and the third variable are substituted After that, they are fixed variables, that is, storage nodes The fixed variables on , the second variable Remain. The corresponding coefficients , where , and .

[0029] It should be noted here that the coefficients corresponding to the variables of the same degree of the two polynomials are the same. Therefore, the stored data mapped to each storage node is the number of coefficients of the multiple coefficients corresponding to the multivariate polynomial.

[0030] In addition, regarding the equivalence of the coefficients of the polynomial, take the polynomial and are equivalent, where represents the polynomial corresponding to the fixed variables after substituting the second variable and the third variable for the storage node . The proof is as follows: After substituting the second variable and the third variable, it is the polynomial corresponding to the fixed variable Consider the expansions of the polynomials and , where: ; ; Because , it can be obtained that when , , that is, it is proved.

[0031] In the coefficient mapping process of the ternary polynomial provided in this embodiment, the parameters of the original parity-check matrix are mapped into the coefficients of the multivariate polynomial, so as to facilitate the subsequent combination with the multivariate polynomial interpolation algorithm to achieve efficient repair of the failed node.

[0032] In step S13, if there is a failed node in the storage nodes, it is necessary to reconstruct the multivariate polynomial from the polynomial coefficients of the helping nodes by using the multivariate polynomial interpolation method corresponding to the MSR code to repair the stored data of the failed node. The repair process here can repair all the data, or repair the corresponding partial data through the helping nodes, and combine the partial data corresponding to each helping node to obtain the stored data of the failed node.

[0033] In some embodiments, reconstructing a multivariate polynomial from the polynomial coefficients of the helping nodes in at least one storage node by using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the stored data of the failed node includes: Reconstruct the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper nodes using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the data to be stored; and determine the stored data of the failed node according to the data to be stored. Alternatively, reconstruct the multivariate polynomial coefficients of the corresponding helper nodes from the polynomial coefficients of the helper nodes using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the stored data of the failed node.

[0034] Specifically, one solution is to reconstruct the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper nodes using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the data to be stored; and determine the stored data of the failed node according to the data to be stored. Here, the entire total coefficients of the multivariate polynomial are reconstructed first, that is, all the data to be stored are reconstructed. Based on this data to be stored, the stored data of the failed node can be known. That is, download all the data from any surviving nodes among the remaining

[0035] (It should be noted that the helper nodes are some of the surviving nodes, and the helper nodes must be surviving nodes) to repair all the data.

[0036] In the repair process provided in this embodiment, the stored data of the failed node can be determined after all the original data is repaired first, or a part of the data downloaded from multiple helper nodes can be pieced together for repair, improving the flexibility and diversity of the repair.

[0037] Taking the repair of univariate polynomial coefficients as an example, assume that the failed node 3 (storing ) fails, and its repair target .

[0038] The traditional method uses the method of constructing a generation matrix, performs an invertible submatrix, and then downloads data from the helper nodes. In this application, the polynomial interpolation method is used to collect the information of the helper nodes, and any helper nodes (such as nodes 1, 2, 4) are selected.

[0039] Lagrange interpolation: Use three points to reconstruct : ; Among them, represents the th storage node, represents the The assignment points corresponding to the storage nodes Indicates the th storage node.

[0040] Calculate the value of the failed node and directly obtain the failed node .

[0041] Each term in the interpolation formula requires multiplications and additions, and the time complexity is .

[0042] Another example is the binary polynomial. Assume that the failed node stores the coefficients of the polynomial , that is, the coefficients of the polynomial . The following describes the repair process of the failed node. For the helper node , is the th helper node , and the helper node stores the coefficients of the polynomial . Therefore, the failed node can download the information from the helper node (this is a linear combination of the information on the node on ). There is the following theorem: The binary polynomial can be obtained by polynomial interpolation from the data , downloaded from the helper node set.

[0043] ; It is necessary to repair the coefficients , corresponding to this polynomial from the data set . Theoretically speaking, there are a total of different terms, and the helper node set provides different interpolation points, which are sufficient to interpolate the polynomial .

[0044] The following proves through simple linear algebra knowledge that according to the values of these assignment points and , it can be obtained that: ; Because , it is obvious that the above coefficient matrix is invertible (Vandermonde-type matrix), so the corresponding coefficients It can be repaired by polynomial interpolation. It should be noted that in the process of repairing the polynomial coefficients, the matrix inversion operation is not used, but the mature polynomial interpolation algorithm is used. The process is similar to the Lagrange interpolation of a single variable.

[0045] Polynomial It can be repaired by downloading information from the help node. , obviously we can get . There is a polynomial and The corresponding coefficients are the same, so the polynomial can be repaired get All coefficients of The information stored on the node The repair process is explained.

[0046] Through the embodiments of the present application, on the one hand, in the encoding process of the minimum storage regeneration code, the data to be stored is obtained, divided into blocks to obtain at least one data block, and then the stored data of at least one data block is mapped to obtain multivariate polynomial coefficients, so as to construct the minimum storage regeneration code while completing the work of storing the stored data to the storage node. The data symbols (stored data) are mapped to the coefficients of the multivariate polynomial instead of the traditional matrix elements, which is convenient for the repair process in the subsequent data repair process. On the other hand, if there is a failed node, the corresponding multivariate polynomial interpolation method is used to reconstruct the multivariate polynomial from the polynomial coefficients of the helper node to complete the repair. The repair of failed nodes in conventional technical solutions usually relies on Gaussian elimination or matrix decomposition, and these methods have high time complexity (usually 2000 s). ), this application uses the polynomial interpolation method, that is, the low complexity of the polynomial interpolation method is used to avoid the high computational load of matrix inversion and reduce the time complexity of the repair process to , significantly improving the repair efficiency. Therefore, the technical problem of high time complexity of the repair process in conventional technical solutions can be solved, and the data symbols (stored data) can be mapped to the coefficients of the multivariate polynomial, combined with the multivariate polynomial interpolation method, to achieve the technical effect of efficient repair of failed nodes and reduced time complexity.

[0047] In some embodiments, the construction process of the minimum storage regeneration code includes: At least one multivariate polynomial coefficient of a multivariate polynomial is obtained based on a finite field; wherein the power data of the output variable of the multivariate polynomial is greater than or equal to 0 and less than or equal to ; The tuple information corresponding to the variables representing the multivariate polynomial; the coefficients of the multivariate polynomial follow the permutation principle of the output variables; Store the stored data in at least one multivariate polynomial coefficient.

[0048] Specifically, obtain at least one multivariate polynomial coefficient of the multivariate polynomial based on a finite field, and the multivariate polynomial is Multivariate polynomial: ; Among them, for any , that is, the power data of the output variable of the multivariate polynomial is greater than or equal to 0 and less than or equal to ; there is , that is, the multivariate polynomial coefficient follows the permutation principle of the output variable. Note that is the n -th symmetric group, and its elements are all permutations; represents the tuple information corresponding to the variables of the multivariate polynomial.

[0049] Through the above - mentioned multivariate polynomial, it can be known that the stored data is stored in each multivariate polynomial coefficient to construct an MSR code.

[0050] In this embodiment, storing data in the coefficients of the multivariate polynomial to complete the construction of the MSR code means that, based on the multivariate polynomial, the coefficients of the multivariate polynomial calculated by assignment on each storage node are stored, which is convenient for subsequent repair of the stored data of the failed node.

[0051] In some embodiments, the number of sub - packets of the minimum storage regeneration code is determined by the total number of coefficients of the multivariate polynomial and the number of data blocks.

[0052] Specifically, the number of sub - packets is equal to the amount of symbol data stored in each node. For single - node repair, if the redundancy is regarded as a constant, the existing number of sub - packets or is of exponential level with respect to . Since the huge number of sub - packets directly leads to a large increase in the number of operations and the data scale, in most parameter cases, the storage complexity of the MSR code in the actual system is very high.

[0053] Denote the number of sub - packets as , to repair a certain failed node , where , the amount of symbols downloaded from the helper nodes is denoted as , the number of helper nodes is denoted as , and the repair bandwidth is denoted as . Then the repair bandwidth (also measured by symbols in a finite field ), that is, the average amount of data downloaded from each helping node is .

[0054] The conventional number of sub - packets is only at the linear level of the dimension of the MDS code , and the storage complexity is relatively low.

[0055] The embodiments of this application are determined based on the total coefficient of the multivariate polynomial and the number of data blocks, that is, the number of sub - packets is the data symbols stored on the nodes corresponding to each data block as the number of sub - packets.

[0056] In the determination process of the number of sub - packets provided in this embodiment, compared with the exponential - level number of sub - packets in the conventional technical solution, the number of sub - packets in this embodiment is only at the polynomial level. In a large - scale distributed storage system, this improvement can significantly save storage resources. In addition, the reduction of the number of sub - packets also reduces the complexity of data storage and management, making the system easier to expand and maintain. For a distributed system storing massive data, this optimization can significantly reduce the hardware cost and operation cost.

[0057] In some embodiments, the determination process of the total coefficient of the multivariate polynomial includes: Obtain the monotonic non - decreasing tuple data relationship of the multivariate polynomial; Determine the corresponding number of groups in the data relationship according to the method of inserting spaces to determine the number of data relationships; Determine the number of variables of the multivariate polynomial according to the variable permutation principle of the multivariate polynomial coefficient; where the permutation principle is invariant under the permutation of output variables; Determine the total coefficient of the multivariate polynomial according to the number of variables and the number of data relationships.

[0058] Specifically, combined with the above - variable polynomial formula, the monotonic non - decreasing tuple data relationship: ; Among them, because , assume the number of times the integer appears in is . Then there is . Let's denote , then , which can be converted into a commonly used technique in combinatorial counting - the method of inserting spaces for calculation, that is, choosing spaces from spaces to get groups, and the size of each group corresponds to , so The quantity of , the proof is completed. That is, according to the method of inserting spaces, the number of grouped quantities corresponding to its data relationship formula is ones, and thus the number of data relationship formulas is determined to be , and the number of variables is determined to be according to the permutation principle.

[0059] Due to the variable permutation principle of the coefficients of multivariate polynomials: , the polynomial has a total of different coefficients, that is, the total coefficients of the multivariate polynomial. Therefore, the size of the data symbols that can be stored by these coefficients is , that is, data symbols are stored on each node, corresponding to the number of sub - packets .

[0060] Furthermore, for example: , the storage node stores the coefficients corresponding to the polynomial , where , and .

[0061] The polynomial has a total of different coefficients, so it can be used to store data symbols.

[0062] The proof is as follows: Expand the polynomial , consider the coefficient of the term , denoted as , where . According to , we can get , where is the permutation mapping of . By analogy, there are a total of such coefficients.

[0063] The conventional number of sub - packets is still of exponential level with respect to , bringing a large storage complexity and computational cost. According to the above method, an MSR code with a smaller number of sub - packets can be obtained. When , consider an MSR code like ( ), and the number of sub - packets obtained is only , while the conventional technical solution requires sub - packets; When , consider an MSR code like ( ), and the number of sub - packets obtained is only , while the number of sub - packages required in the conventional technical solution .

[0064] The process of determining the total coefficient of the multivariate polynomial provided in this embodiment is used to determine the number of sub - packages, reducing the storage complexity while improving the repair efficiency of failed nodes.

[0065] In some embodiments, the number of helper nodes is determined by the total coefficient of the multivariate polynomial and the polynomial coefficient corresponding to the helper node.

[0066] Specifically, regarding the total coefficient of the multivariate polynomial, which is also the number of data to be repaired, for the data that each helper node can provide, it is determined by the polynomial coefficient corresponding to the helper node. Therefore, here, the number of helper nodes can be obtained by dividing the total coefficient of the multivariate polynomial by the polynomial coefficient corresponding to the helper node.

[0067] In some embodiments, the process of determining the number of helper nodes includes: Obtain the total coefficient of the multivariate polynomial; Obtain the polynomial coefficient of the first - target multivariate polynomial; Determine the number of helper nodes according to the total coefficient and the polynomial coefficient of the first - target multivariate polynomial.

[0068] It should be noted that the total coefficient of the multivariate polynomial is considered in view of downloading information from the helper nodes, and the polynomial coefficient of the first - target multivariate polynomial here is the information downloaded from the helper nodes. The number of this helper node is determined by dividing the total coefficient by the polynomial coefficient of the first - target multivariate polynomial.

[0069] For example: a certain multivariate polynomial can be composed of the information downloaded from the helper nodes (the polynomial coefficient of the first - target multivariate polynomial), that is: .

[0070] This multivariate polynomial Regarding the output variable of the variable the degree is Then the corresponding coefficients in total are ones, which is also the number of data to be repaired. And the number of corresponding coefficients is , that is, each helper node provides data, then the total amount of data provided is , and the corresponding formula is: , that is, the number of helper nodes is .

[0071] It should be noted that the above example only corresponds to a certain multivariate polynomial distance. For the total coefficient, the number of helper nodes is , and only the above needs to be explained .

[0072] The process of determining the number of helper nodes provided in this embodiment is only corresponding to in the case of the conventional technical solution. In the case of this application, it can achieve greater than . The more helper nodes, the smaller the amount of data downloaded from the helper nodes, reducing the network transmission overhead and repair time.

[0073] It should be noted that in the above embodiment, both the number of helper nodes and the number of sub-packets are known. Regarding the code rate, it is approximately , and it increases with the increase of . The code rate of the MSR code in the conventional technical solution is limited to nearby, with a low code rate and a high redundancy, making it difficult to meet the requirements of existing distributed storage systems for high code rates. The code rate of this application will be close to 1, supporting a higher code rate (i.e., a lower redundancy), and being able to better meet the needs of modern distributed storage systems for high code rates. Compared with the conventional technical solution, the code rate of this application is higher and the redundancy is lower, thus improving the storage efficiency. A high code rate means that the system requires less redundant information when storing data and has a higher storage efficiency, which is particularly important for scenarios sensitive to storage costs (such as cloud storage and big data analysis). In addition, it is superior to the conventional technical solution in terms of storage complexity, repair efficiency, and redundancy, and is more suitable for actual industrial applications. By degrading the multivariate polynomial interpolation to multi-layer univariate interpolation, the encoding and decoding processes are simplified, and the implementation difficulty is reduced. This design not only achieves the optimal performance in theory but also ensures the efficiency and feasibility in practical applications through specific implementation methods (such as polynomial interpolation algorithms), making this solution more competitive in large-scale distributed storage systems.

[0074] In some embodiments, regarding the quality of the MSR code construction, two preset conditions need to be met, that is, after constructing the minimum storage regeneration code and before repairing the stored data of the failed node, it further includes: In the case where the minimum storage regeneration code meets the repair bandwidth preset condition and the repair data preset condition, it enters the step of reconstructing the multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node by using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the stored data of the failed node.

[0075] It should be noted that the preset condition for the repair bandwidth is to illustrate that the constructed MSR code has the optimal repair property, that is, the repair bandwidth of any failed node can reach the cut-set bound. As mentioned in the above embodiments, the effective repair of a failed node is measured by the repair bandwidth. Meeting the preset condition for the repair bandwidth indicates that the optimal repair ability has been achieved, which also corresponds to an improvement in the repair efficiency. Here, the cut-set bound is a lower bound corresponding to the repair bandwidth (RepairBandwidth, RB) of the MDS array code. The MSR code that reaches this lower bound has the optimal repair property.

[0076] The preset condition for the repaired data is that the encoding based on the multivariate polynomial coefficients constructed has the MDS property, that is, the information of any number of nodes can be used to repair all the original data. Here, repairing all the original data refers to the corresponding data to be stored, which can be used as both a performance condition for verifying the MSR code and a repair step in the subsequent repair process. It can be set according to the actual situation in the subsequent repair process.

[0077] After the MSR in this embodiment meets the above two preset conditions, it ensures that the MSR code has the optimal repair property and the MDS property, improves the repair efficiency of subsequent failed nodes, and also improves the reliability of the repair process.

[0078] In some embodiments, the determination process for the minimum storage regeneration code to meet the preset condition for the repair bandwidth includes: Obtaining the first invariant information of the first target failed node and the second invariant information of the first target helper node; Substituting the first invariant information into the second variable of the multivariate polynomial of the first target helper node to obtain the first target multivariate polynomial; Substituting the second invariant information into the second variable of the multivariate polynomial of the first target failed node to obtain the second target multivariate polynomial; If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, then determine whether the polynomial coefficient of the first target multivariate polynomial is the same as the data volume downloaded by the first target helper node; If they are the same, it is determined that the repair bandwidth of the first target failed node reaches the cut-set bound, and it is determined that the minimum storage regeneration code meets the preset condition for the repair bandwidth.

[0079] Specifically, in this embodiment, the corresponding failed node can be downloaded from all helper nodes or from one helper node, which is not limited here. One failed node and one helper node are used for illustration.

[0080] Obtaining the first target failed node (the first target failed node stores the polynomial The first invariant information of and the first target helper node (the first target helper node stores a polynomial The second invariant information of .

[0081] Substitute the first invariant information into the second variable of the multivariate polynomial of the first target helper node to obtain the first target multivariate polynomial, that is .

[0082] Substitute the second invariant information into the second variable of the multivariate polynomial of the first target failed node to obtain the second target multivariate polynomial, that is .

[0083] If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, it is determined to be equivalent, that is, the failed node can download the polynomial from the helper node The corresponding coefficients (this is a linear combination of the information on the node in ), which is equivalent to The corresponding coefficients.

[0084] Further, it is necessary to determine whether the polynomial coefficients of the first target multivariate polynomial are the same as the data volume downloaded by the first target helper node. It should be noted that the data volume downloaded by the first target helper node is a certain value , and the setting of the number of sub-packets in this formula can be calculated based on the size of the number of sub-packets determined in the above embodiments. The number of helper nodes can be calculated based on the size corresponding to the determination of the number of helper nodes in the above embodiments. Finally, if the polynomial coefficients of the first target multivariate polynomial are the same as the data volume downloaded by the first target helper node, it is determined that the repair bandwidth of the first target failed node reaches the cut-set bound, and it is determined that the minimum storage regenerating code meets the repair bandwidth preset condition. At this time, the first target multivariate polynomial , corresponding to different numbers of coefficients , equal to , that is, reaching the cut-set bound.

[0085] The determination process of the minimum storage regenerating code meeting the repair bandwidth preset condition provided in this embodiment enables the MSR code to have optimal repair properties, ensuring that the repair bandwidth of any failed node reaches the cut-set bound. The repair bandwidth is minimized to reduce the occupancy of network bandwidth and accelerate the data repair speed during the repair process.

[0086] In some embodiments, the determination process of the minimum storage regenerating code meeting the repair data preset condition includes: Take the number of variables in the multivariate polynomial as the number of storage nodes; Interpolate based on the variable data corresponding to the current storage node itself and other storage nodes to generate the output variable corresponding to the current storage node and at least one corresponding variable; Merge the generated output variables and at least one variable corresponding to at least one storage node to obtain the target multivariate polynomial; If the multivariate polynomial is the same as the target multivariate polynomial, determine the data repair of any data block to obtain the data to be stored, and determine that the minimum storage regeneration code meets the preset conditions for repairing data.

[0087] Specifically, take the number of variables in the multivariate polynomial as the number of storage nodes. For example, there are variables , as , that is, corresponding to storage nodes.

[0088] Interpolate based on the variable data corresponding to the current storage node itself and other storage nodes to generate the output variable corresponding to the current storage node and at least one corresponding variable. Here, it is interpolated based on the variable data corresponding to each storage node itself and other storage nodes.

[0089] Furthermore, consider any two different integers , first interpolate to generate the corresponding output variable , which is generated by interpolating through the variable data of each storage node providing its own multivariate polynomial. Considering the symmetry between the coefficients of each multivariate polynomial, the data set can be obtained through the above data to repair . Then, continue to interpolate to generate the last variable, the penultimate variable, until the first variable is interpolated.

[0090] Merge the generated output variables and the data set corresponding to at least one variable to obtain the target multivariate polynomial. By comparing the multivariate polynomial with the target multivariate polynomial, if they are the same, determine the data repair of any data block to obtain the data to be stored, that is, the MSR code meets the preset conditions for repairing data. The comparison process here is the comparison of the coefficients of the multivariate polynomial and the relational expressions. As long as the original multivariate polynomial can be reconstructed, it means that the data of any data block in this embodiment can be repaired to obtain the data to be stored.

[0091] The determination process of the minimum storage regeneration code provided in this embodiment that meets the preset conditions for repairing data ensures that the information of any data block node can repair all the original data, making the constructed MSR coding have the MDS property, achieving the optimal fault tolerance and storage efficiency in theory, and improving the performance and reliability of the system in practical applications.

[0092] In some embodiments, interpolating to generate an output variable corresponding to the current storage node and at least one corresponding variable based on the variable data corresponding to the current storage node itself and other storage nodes includes: Substituting the current storage node into the first variable of the multivariate polynomial to interpolate and generate an output variable corresponding to the current storage node; Reducing the storage nodes one by one. During the reduction process, substituting the new current storage node into the first variable of the multivariate polynomial to interpolate and generate target variables corresponding to the new current storage node by reverse deduction starting from the last variable; wherein, the number of times of reducing the storage nodes is the same as the number of times of determining the target variables.

[0093] Given , then , the multivariate polynomial can be repaired from the information in the node set .

[0094] Recall that the degree of the polynomial with respect to the variable is less than or equal to , and the degree with respect to is less than or equal to . Consider any pairwise distinct integers ; The node provides the data ; The node provides the data ; That is, for any , the node provides the data .

[0095] According to the symmetry between the coefficients, the data set can be obtained from the above data because the degree of the polynomial with respect to is , so the can be repaired from the obtained data set. Here, the output variable corresponding to the current storage node has been interpolated and generated.

[0096] Reduce the storage nodes one by one. Substitute the new current storage node into the first variable of the multivariate polynomial to obtain target variables corresponding to the new current storage node by reverse deduction starting from the last variable.

[0097] Let the integer range over the set , and the data set can be obtained.

[0098] Since the degree of the polynomial with respect to is less than or equal to , and the number of data in the above dataset is , a polynomial can be interpolated.

[0099] Similarly, by taking the integer through the set , the dataset can be obtained.

[0100] Since the degree of the polynomial with respect to is , and the number of data in the above dataset is , a polynomial can be interpolated.

[0101] Repeating the above similar steps for , a polynomial can finally be obtained.

[0102] For easy understanding, taking a three-variable polynomial as an example, given , then , and the three-variable polynomial can be repaired from the information in the node set .

[0103] Proof: , from the polynomial storing the data in the node , can be obtained, that is, let . Similarly, can be obtained from the data in the node . Since the degree of the polynomial with respect to the variable is equal to 1, and (equivalent according to and ) can be used to interpolate . At this time, can be fixed, and is run through , and the data set can be obtained. Since with respect to the variable has a degree , and there are interpolation points, can be used to interpolate . Similarly, running through , it can be obtained through Interpolated to obtain .

[0104] Any of the encoded ones provided in this embodiment surviving nodes can repair all the original data, which degrades the multivariate polynomial interpolation to multi-layer univariate interpolation to reduce the proof difficulty. For each storage node, variables corresponding to the storage node can be interpolated and generated to reconstruct the current multivariate polynomial, laying a foundation for the subsequent reconstruction of the total multivariate polynomial.

[0105] Furthermore, the present application also provides a distributed storage system, including a repair node and at least one storage node; The repair node is used to execute the steps of the above data repair method to repair the stored data of the failed nodes in the storage node.

[0106] Figure 2 is a structural diagram of a distributed storage system provided by an embodiment of the present application. As Figure 2 shown, the repair node repairs the failed nodes corresponding to multiple storage nodes. The repair process can refer to the above embodiment and will not be elaborated here.

[0107] For the introduction of a distributed storage system provided by the present application, please refer to the above method embodiment. The present application will not elaborate here, and it has the same beneficial effects as the above data repair method.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0109] The embodiment of the present application also provides a data repair device, Figure 3 is a structural diagram of a data repair device provided by an embodiment of the present application. As Figure 3 shown, the device includes: An acquisition module 11, configured to acquire data to be stored and perform block processing on the data to be stored to obtain at least one data block; A mapping processing module 12, configured to perform mapping processing on the stored data of at least one data block to obtain multivariate polynomial coefficients, so as to construct a minimum storage regeneration code and determine the stored data corresponding to at least one storage node; A repair processing module 13, configured to, if there are failed nodes in at least one storage node, reconstruct a multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node by using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the stored data of the failed nodes.

[0110] For the description of the features in the corresponding embodiments of the data repair device, reference can be made to the relevant descriptions in the corresponding embodiments of the data repair method, which will not be elaborated here one by one.

[0111] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above embodiments of the data repair method.

[0112] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any one of the above embodiments of the data repair method when running.

[0113] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical disks and other various media that can store computer programs.

[0114] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above embodiments of the data repair method.

[0115] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any one of the above embodiments of the data repair method.

[0116] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0117] The above has introduced in detail a data repair method, a distributed storage system, a device, a medium, and a product provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and modifications can still be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A data repair method, characterized in that, Including: Obtain the data to be stored, and perform chunking processing on the data to be stored to obtain at least one data chunk; Perform mapping processing on the stored data of at least one data chunk to obtain multivariate polynomial coefficients, construct a minimum storage regeneration code, and determine the stored data corresponding to at least one storage node; If there is a failed node among at least one storage node, use the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node to repair the stored data of the failed node.

2. The data repair method according to claim 1, wherein The construction process of the minimum storage regeneration code includes: Obtain at least one multivariate polynomial coefficient of a multivariate polynomial based on a finite field; wherein, the power data of the output variable of the multivariate polynomial is greater than or equal to 0 and less than or equal to ; Tuple information corresponding to the variables characterizing the multivariate polynomial; the multivariate polynomial coefficients follow the permutation principle of the output variables; Store the stored data in at least one multivariate polynomial coefficient.

3. The data repair method according to claim 2, wherein The number of sub-packages of the minimum storage regeneration code is determined by the total number of coefficients of the multivariate polynomial and the number of data chunks.

4. The data repair method according to claim 3, wherein The determination process of the total number of coefficients of the multivariate polynomial includes: Obtain the monotonically non-decreasing data relational expression of the tuple; Determine the corresponding number of groups in the data relationship formula according to the method of inserting spaces to determine the number of data relationship formulas; Determine the number of variables of the multivariate polynomial according to the variable substitution principle of the multivariate polynomial coefficients; where the substitution principle is invariant under the substitution of output variables; Determine the total number of coefficients of the multivariate polynomial according to the number of variables and the number of data relationship formulas.

5. The data repair method according to claim 4, characterized in that, The number of helper nodes is determined by the total number of coefficients of the multivariate polynomial and the polynomial coefficients corresponding to the helper nodes.

6. The data repair method according to claim 5, wherein After constructing the minimum storage regeneration code and before repairing the stored data of the failed node, it further includes: When the minimum storage regeneration code meets the repair bandwidth preset condition and the repair data preset condition, enter the step of using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node to repair the stored data of the failed node.

7. The data repair method according to claim 6, wherein The determination process of the minimum storage regeneration code meeting the repair bandwidth preset condition includes: Obtain the first invariant information of the first target failed node and the second invariant information of the first target helper node; Substitute the first invariant information into the second variable of the multivariate polynomial of the first target helper node to obtain a first target multivariate polynomial; Substitute the second invariant information into the second variable of the multivariate polynomial of the first target failed node to obtain a second target multivariate polynomial; If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, determine whether the polynomial coefficients of the first target multivariate polynomial are the same as the data volume downloaded by the first target helper node; If they are the same, determine that the repair bandwidth of the first target failed node reaches the cut-set bound, and determine that the minimum storage regeneration code meets the repair bandwidth preset condition.

8. The data repair method according to claim 7, wherein The determination process of the number of helper nodes includes: Obtain the total number of coefficients of the multivariate polynomial; Obtain the polynomial coefficients of the first target multivariate polynomial; Determine the number of helper nodes according to the total number of coefficients and the polynomial coefficients of the first target multivariate polynomial.

9. The data repair method according to claim 6, wherein, The determination process of the minimum storage regeneration code meeting the repair data preset condition includes: Take the number of variables in the multivariate polynomial as the number of storage nodes; Interpolate based on the variable data corresponding to the current storage node itself and other storage nodes to generate the output variable corresponding to the current storage node and at least one corresponding variable; Merge the output variables and at least one variable corresponding to the generated at least one storage node to obtain a target multivariate polynomial; If the multivariate polynomial is the same as the target multivariate polynomial, determine the data repair of any data block to obtain the data to be stored, and determine that the minimum storage regeneration code meets the preset conditions for the repaired data.

10. The data repair method according to claim 9, characterized in that, Interpolating based on the variable data corresponding to the current storage node itself and other storage nodes to generate the output variable corresponding to the current storage node and at least one corresponding variable, includes: Based on the current storage node being substituted into the first variable of the multivariate polynomial to interpolate and generate the output variable corresponding to the current storage node; Reduce the storage nodes one by one. During the reduction process, based on the new current storage node being substituted into the first variable of the multivariate polynomial to interpolate and generate the target variable starting from the last variable in reverse order corresponding to the new current storage node; wherein, the number of times of reducing the storage nodes is the same as the number of times of determining the target variable.

11. The data repair method according to claim 1, wherein Using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node to repair the stored data of the failed node, includes: Using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper nodes to repair the data to be stored; and determine the stored data of the failed node according to the data to be stored; Or, using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the multivariate polynomial coefficients of the helper nodes from the polynomial coefficients of the helper nodes to repair the stored data of the failed node.

12. A distributed storage system, characterized in that, Includes a repair node and at least one storage node; The repair node is used to execute the steps of the data repair method described in any one of claims 1 to 11 above to repair the stored data of the failed node in the storage node.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the data repair method described in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by the processor, implements the steps of the data repair method described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the data repair method described in any one of claims 1 to 11.

Images