Data repair method, distributed storage system, device, medium and product

Abstract

The present application discloses a data repair method, distributed storage system, device, medium and product, which relate to the field of data storage technology. The method obtains the data to be stored, divides it into blocks to obtain at least one data block, and then maps the stored data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code while completing the work of storing the stored data to the storage node. The method realizes mapping the data symbols (stored data) to the coefficients of the multivariate polynomial rather than the traditional matrix elements, which facilitates the repair process in the subsequent data repair process. The multivariate polynomial is reconstructed from the polynomial coefficients of the helper node using the corresponding multivariate polynomial interpolation method to complete the repair. The low complexity characteristics of the polynomial interpolation method are utilized to avoid the high computational load of matrix inversion, reduce the time complexity of the repair process to #imgabs0#, and significantly improve the repair efficiency.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data repair method, distributed storage system, device, medium and product. Background Art

[0002] In distributed storage systems, Minimal Storage Regenerating Code (MSR Code) is used to repair failed nodes by downloading data from helper nodes. Conventional MSR codes use matrix inversion operations, resulting in high time complexity in the failed node repair process.

[0003] Therefore, how to reduce time complexity is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The present application provides a data repair method, distributed storage system, device, medium and product to at least solve the problem of high time complexity in related technologies.

[0005] This application provides a data repair method, including:

[0006] Acquire data to be stored, and divide the data to be stored into blocks to obtain at least one data block;

[0007] Mapping the stored data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code, and determining the stored data corresponding to at least one storage node;

[0008] If there is a failed node in at least one storage node, a multivariate polynomial interpolation method corresponding to the minimum storage regeneration code is used to reconstruct a multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node to repair the storage data of the failed node.

[0009] The present application also provides a distributed storage system, comprising a repair node and at least one storage node;

[0010] The repair node is used to execute the steps of the above-mentioned data repair method to repair the storage data of the failed node in the storage node.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data repair methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data repair methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data repair methods when executed by a processor.

[0014] Through the present application, on the one hand, in the encoding process of the minimum storage regeneration code, the data to be stored is obtained, divided into blocks to obtain at least one data block, and then the stored data of at least one data block is mapped to obtain multivariate polynomial coefficients, so as to construct the minimum storage regeneration code while completing the work of storing the stored data to the storage node. The data symbols (stored data) are mapped to the coefficients of the multivariate polynomial, rather than the traditional matrix elements, to facilitate the repair process in the subsequent data repair process. On the other hand, if there is a failed node, the corresponding multivariate polynomial interpolation method is used to reconstruct the multivariate polynomial from the polynomial coefficients of the help node to complete the repair. Conventional technical solutions usually rely on Gaussian elimination or matrix decomposition to repair failed nodes. These methods have high time complexity (usually ), this application uses the polynomial interpolation method, that is, the low complexity of the polynomial interpolation method is used to avoid the high computational load of matrix inversion and reduce the time complexity of the repair process to , significantly improving repair efficiency. Therefore, this approach solves the high time complexity of the repair process in conventional solutions. By mapping data symbols (stored data) to the coefficients of a multivariate polynomial and combining this with multivariate polynomial interpolation, it achieves the technical effect of efficiently repairing failed nodes and reducing time complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of a data repair method provided in an embodiment of the present application;

[0017] Figure 2 A structural diagram of a distributed storage system provided in an embodiment of the present application;

[0018] Figure 3 A structural diagram of a data repair device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0020] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0021] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0022] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data repair method depends, the specific application environment architecture or specific hardware architecture is described here.

[0023] In order to improve the reliability of data storage, distributed storage systems store massive amounts of data in multiple servers (i.e., nodes) and connect them through a network. Since these nodes are usually distributed in different geographical locations and are independent of each other, the risk of data loss due to natural disasters or hardware failures can be effectively reduced. In a distributed storage system, information related to data files (messages) is stored in a specific way on various nodes in the network, allowing end users to retrieve stored data by accessing some nodes. A common practice is to use erasure codes, such as Maximum Distance Separable Code (MDS Code) such as Reed-Solomon Code (RS Code), that is, to achieve maximum error correction capability under given redundancy. This method can reduce network congestion and enhance the resilience of the system in the event of node failure. Assume is the total size of the data file, in finite fields (Size is Specifically, if the initial data is divided into Block, through After the MDS code is encoded, the generated The data blocks are stored in storage nodes, the system can tolerate any If a node fails, the data collector can connect to any A surviving node (or some of the surviving nodes, i.e., helping nodes) is used to repair the entire message.

[0024] The effective repair of failed nodes in distributed storage is measured by the repair bandwidth, that is, the total amount of data downloaded from the helper nodes to repair the failed nodes. In a distributed storage system, there are repair nodes and storage nodes. There is at least one storage node. The repair node is located in the server, specifically the controller in the server to perform the repair. data blocks, Encode the data blocks storage nodes, distributing the original files to The conventional technical solution generates a matrix based on the array code during the repair process, namely:

[0025] ;

[0026] in,( ) is called an MDS array code. , both ,like , then the MDS array code is called systematic, where express superior The gear matrix, obviously, in the number of sub-packages When , the MDS array code is the traditional MDS code.

[0027] In the process of constructing a matrix, it is necessary to select elements that meet specific conditions to build the generator matrix. This may be difficult in actual implementation, that is, to prove that any column blocks of the corresponding generator matrix are linearly independent based on the combinatorial zero point theorem.

[0028] Data storage requires constructing a The generator matrix , repair the linear equations to be solved , the number of nodes that will be helped Solve it out, correspond One Invertible submatrix, Corresponding to the data downloaded from the helper node. Its complexity is obtained by matrix inversion or Gaussian elimination, and the time complexity is The current matrix inversion operation has a high time complexity for repairing failed nodes, and the repair time is long. The data repair method provided in this application can solve the above technical problems.

[0029] Figure 1 A flowchart of a data repair method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0030] S11: Acquire data to be stored, and divide the data to be stored into blocks to obtain at least one data block;

[0031] S12: Mapping the stored data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code, and determining the stored data corresponding to at least one storage node;

[0032] S13: If there is a failed node in at least one storage node, a multivariate polynomial interpolation method corresponding to the minimum storage regeneration code is used to reconstruct a multivariate polynomial from the polynomial coefficients of the helper nodes in at least one storage node to repair the storage data of the failed node.

[0033] Specifically, data to be stored is obtained based on the size of the original file. The data to be stored is divided into blocks to obtain at least one data block. The block processing method of the data block here is the same as the conventional method, or it can be different and is not limited here. It can be set according to actual conditions.

[0034] In step S12, the stored data of at least one data block is mapped to obtain the multivariate polynomial coefficients. It should be noted that the conventional encoding process is to encode the stored data of at least one data block to generate The encoding process uses a linear combination method. Conventional technical solutions, for example: construct a sub-packet number of of MSR code, the total size of the data file to be stored Information data is stored in Symmetric matrix That is, the upper triangular part of each matrix can be stored separately information symbols, and the corresponding encoding matrix is a Matrix ,in is a The matrix, is a Diagonal matrix, and satisfies: Any The rows are linearly independent; Any The rows are linearly independent; of The diagonal elements are different.

[0035] In this embodiment, the stored data is mapped and processed to obtain the coefficients of the multivariate polynomial. For example, in the case of single variable polynomial coefficient encoding, the original data is divided into blocks as follows: ; Construct a polynomial and map the data into polynomial coefficients: Generate storage nodes, select Different evaluation points , calculate the polynomial value, , each storage node stores a pair of ( ). Indicates the storage nodes, Indicates the The assignment points corresponding to the storage nodes.

[0036] If the multivariate polynomial coefficients are encoded, the parameters of the stored data in the original matrix need to be mapped to the coefficients of the multivariate polynomial. The coefficients follow the permutation principle of the output variables, that is, they remain unchanged under the permutation of the output variables.

[0037] In some embodiments, when the multivariate polynomial is a ternary polynomial, the coefficients corresponding to the ternary polynomial include a first coefficient and a second coefficient; and storing the stored data in at least one multivariate polynomial coefficient includes:

[0038] determining the number of coefficients of the first coefficient and the second coefficient;

[0039] The stored data of at least one data block are sorted according to the number of coefficients corresponding to the first coefficient and the second coefficient.

[0040] Specifically, taking a three-variable polynomial as an example, hour, The tuple information corresponding to the variable representing the multivariate polynomial, the corresponding number of subpackaging is , the total size of the data files to be stored can be obtained as The corresponding three-variable polynomial is:

[0041] ;

[0042] in, , 、 Corresponding representation 、 The variable 、 Power data; can be obtained separately different , corresponding to the first and second coefficients, the polynomial can be used Different coefficients are used to store the corresponding data symbol information.

[0043] , storage node The storage polynomial is:

[0044] ;

[0045] in, Indicates that the first and third variables are brought in The latter is a fixed variable, that is, the storage node The fixed variable on the second variable Keep. The corresponding coefficient ,in, ,and .

[0046] It should be noted here that the coefficients corresponding to the variables of the same degree in the two polynomials are the same. Therefore, the storage data mapped to each storage node is the number of coefficients corresponding to the multiple coefficients of the multivariate polynomial.

[0047] In addition, regarding the coefficient equivalence of polynomials, the polynomial and are equivalent, where Represents a storage node Correspondingly, the second and third variables are introduced The following is the polynomial corresponding to the fixed variable, which is proved as follows:

[0048] Consider the polynomial and The expansion of , where:

[0049] ;

[0050] ;

[0051] because , but appropriate hour, , and you will get the certificate.

[0052] The coefficient mapping process in the ternary polynomial provided in this embodiment maps the parameters of the original check matrix to the multivariate polynomial coefficients, so as to facilitate the subsequent combination with the multivariate polynomial interpolation algorithm to achieve efficient repair of failed nodes.

[0053] In step S13, if a failed node exists among the storage nodes, a multivariate polynomial interpolation method corresponding to the MSR code is used to reconstruct the multivariate polynomial from the polynomial coefficients of the helper nodes to repair the stored data of the failed node. This repair process can repair all the data or repair the corresponding partial data through the helper nodes. The partial data corresponding to each helper node is combined to obtain the stored data of the failed node.

[0054] In some embodiments, reconstructing a multivariate polynomial from polynomial coefficients of a helper node in at least one storage node using a multivariate polynomial interpolation method corresponding to a minimum storage regeneration code to repair stored data of a failed node includes:

[0055] Reconstruct the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper node using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the data to be stored; and determine the storage data of the failed node based on the data to be stored;

[0056] Alternatively, the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code is used to reconstruct the multivariate polynomial coefficients of the corresponding helping node from the polynomial coefficients of the helping node to repair the storage data of the failed node.

[0057] Specifically, one solution is to use the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper node to repair the data to be stored; and determine the storage data of the failed node based on the data to be stored. Here, the entire total coefficients of the multivariate polynomial are reconstructed first, that is, all the data to be stored are reconstructed, and the storage data of the failed node can be known based on the data to be stored. That is, using the remaining arbitrary All data is downloaded from the surviving nodes (it should be noted that the helping nodes are some of the surviving nodes, and the helping nodes must be surviving nodes) to repair all data.

[0058] Another approach is to use the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to reconstruct the multivariate polynomial coefficients of the helper node from the polynomial coefficients of the helper node to repair the stored data of the failed node. In other words, each helper node downloads a portion of the data and repairs the data on the failed node.

[0059] In the repair process provided by this embodiment, the original data can be completely repaired first and then the storage data of the failed node can be determined. Alternatively, part of the data can be downloaded from multiple helper nodes for splicing and repair, thereby improving the flexibility and diversity of the repair.

[0060] Taking the repair of single variable polynomial coefficient as an example, suppose the failed node 3 (storage ) fails, its repair target .

[0061] The traditional method uses the method of constructing a matrix to generate a reversible sub-matrix and then download the data from the helper node. In this application, the polynomial interpolation method is used to collect the helper node information. helper nodes (such as nodes 1, 2, and 4).

[0062] Lagrange interpolation: using three points Refactoring :

[0063] ;

[0064] in, Indicates the storage nodes, Indicates the The assignment points corresponding to the storage nodes, Indicates the storage nodes.

[0065] Calculate the failed node value and get the failed node directly .

[0066] Each term in the interpolation formula needs to be times multiplication and addition, the time complexity is .

[0067] For example, taking the bivariate polynomial as an example, assuming that the failed node The polynomial is stored in The corresponding coefficients, that is, the polynomial The corresponding coefficients are as follows. The repair process of failed nodes is described below. , For the Helper nodes , help node Stores polynomials The coefficient of , so the failure node Available from the Help node Upload and download information (This is the information on the node The linear combination of ). There is the following theorem:

[0068] Two-variable polynomial Data that can be downloaded from the help node set , Perform polynomial interpolation to obtain .

[0069] ;

[0070] Requires a dataset , Repair the coefficients corresponding to the polynomial Theoretically, A total of different items, and the help node set provides Different interpolation points are enough to interpolate the polynomial .

[0071] The following is a simple linear algebra proof, according to this Assignment point value as well as , we can get:

[0072] ;

[0073] because , obviously we can get the above The coefficient matrix of is invertible (Vandermonde type matrix), so the corresponding coefficients It can be repaired by polynomial interpolation. It should be noted that in the process of repairing the polynomial coefficients, matrix inversion is not used, but a mature polynomial interpolation algorithm is used. The process is similar to the Lagrange interpolation of a single variable.

[0074] Polynomial It can be repaired by downloading information from the help node, so , obviously we can get There are polynomials and The corresponding coefficients are the same, so the polynomial can be repaired get All coefficients of The information stored on the node The repair process is explained.

[0075] Through the embodiments of the present application, on the one hand, in the encoding process of the minimum storage regeneration code, the data to be stored is obtained, divided into blocks to obtain at least one data block, and then the stored data of at least one data block is mapped to obtain multivariate polynomial coefficients, so as to construct the minimum storage regeneration code while completing the work of storing the stored data to the storage node. The data symbols (stored data) are mapped to the coefficients of the multivariate polynomial, rather than the traditional matrix elements, to facilitate the repair process in the subsequent data repair process. On the other hand, if there is a failed node, the corresponding multivariate polynomial interpolation method is used to reconstruct the multivariate polynomial from the polynomial coefficients of the help node to complete the repair. Conventional technical solutions usually rely on Gaussian elimination or matrix decomposition to repair failed nodes. These methods have high time complexity (usually ), this application uses the polynomial interpolation method, that is, the low complexity of the polynomial interpolation method is used to avoid the high computational load of matrix inversion and reduce the time complexity of the repair process to , significantly improving repair efficiency. Therefore, this approach solves the high time complexity of the repair process in conventional solutions. By mapping data symbols (stored data) to the coefficients of a multivariate polynomial and combining this with multivariate polynomial interpolation, it achieves the technical effect of efficiently repairing failed nodes and reducing time complexity.

[0076] In some embodiments, the process of constructing a minimum storage regeneration code includes:

[0077] Obtain at least one multivariate polynomial coefficient of a multivariate polynomial based on a finite field; wherein the power data of the output variable of the multivariate polynomial is greater than or equal to 0 and less than or equal to ; The tuple information corresponding to the variables representing the multivariate polynomial; the coefficients of the multivariate polynomial follow the permutation principle of the output variables;

[0078] The storage data is stored in at least one multivariate polynomial coefficient.

[0079] Specifically, at least one multivariate polynomial coefficient of a multivariate polynomial is obtained based on a finite field, and the multivariate polynomial is Polynomial of one variable:

[0080] ;

[0081] Among them, for any , which is the output variable of the multivariate polynomial Power data Greater than or equal to 0, and less than or equal to ;have , that is, the coefficients of the multivariate polynomial follow the permutation principle of the output variable. Note that yes The element symmetry group, whose elements are replacement; Tuple information corresponding to the variables representing the multivariate polynomial.

[0082] Through the above The multivariate polynomial can be known, and the stored data is stored in each multivariate polynomial coefficient to construct the MSR code.

[0083] This embodiment provides a method of storing data in the coefficients of a multivariate polynomial to complete the construction of the MSR code. That is, based on the multivariate polynomial, the multivariate polynomial coefficients calculated and assigned on each storage node are stored for convenience in subsequent repair of the storage data of the failed node.

[0084] In some embodiments, the minimum number of subpackets for storing the regeneration code is determined by the total coefficients of the multivariate polynomial and the number of data blocks.

[0085] Specifically, the number of subpackets is equal to the amount of symbolic data stored in each node. For single-node repair, if the redundancy Considered as a constant, the number of existing MSR code subpackets or It's about Since the huge number of subpackets directly leads to a significant increase in the number of operations and data size, the storage complexity of MSR codes in practical systems is very high in most parameter cases.

[0086] The number of scoring packets is , to repair a failed node ,in , the amount of symbols downloaded from the helper node is recorded as , the number of helping nodes is recorded as , the repair bandwidth is recorded as Then the repair bandwidth corresponding to the MSR code mentioned above is (Also based on finite fields The average amount of data downloaded from each helping node is .

[0087] The conventional sub-packaging number is just the MDS code dimension The linear level of storage complexity is low.

[0088] The embodiment of the present application is determined based on the total coefficients of the multivariate polynomial and the number of data blocks, that is, the number of subpackets is the data symbols stored on the node corresponding to each data block as the number of subpackets.

[0089] The process for determining the number of subpackets provided in this embodiment reduces the number of subpackets to a polynomial level, compared to the exponential level used in conventional solutions. This improvement can significantly save storage resources in large-scale distributed storage systems. Furthermore, the reduction in the number of subpackets reduces the complexity of data storage and management, making the system easier to expand and maintain. For distributed systems storing massive amounts of data, this optimization can significantly reduce hardware and operating costs.

[0090] In some embodiments, the process of determining the total coefficients of the multivariate polynomial includes:

[0091] Get the monotone non-decreasing value of a multivariate polynomial The data relation of tuple;

[0092] Determine the number of corresponding groups in the data relational expression according to the interpolation method to determine the number of data relational expressions;

[0093] The number of variables in the multivariate polynomial is determined based on the variable substitution principle of the coefficients of the multivariate polynomial; wherein the substitution principle is that the output variable remains unchanged under substitution;

[0094] Determine the total coefficients of a multivariate polynomial based on the number of variables and the number of data relationships.

[0095] Specifically, in combination with the above For the formula of the polynomial, monotone non-decreasing The data relation of the tuple is: ;

[0096] Among them, because , assuming middle integer The number of occurrences is Then there is , so we can get . It is worth remembering ,So , which can be converted into a commonly used technique in combination counting - the interpolation method to calculate, that is, Choose from a space Empty, get groups, the size of each group corresponds to ,therefore The number of , the proof is complete. That is, according to the interpolation method, the number of groups corresponding to the data relation is determined as , which determines the number of data relations. , the number of variables is determined by the substitution principle as .

[0097] Due to the variable substitution principle of multivariate polynomial coefficients: , polynomial Total different coefficients, that is, the total coefficients of the multivariate polynomial, so these coefficients can be used to store data with a symbol size of , that is, each node stores data symbols, corresponding to the number of subpackets .

[0098] Further, for example: , storage node Store polynomials The corresponding coefficients are ,and .

[0099] Polynomial Total different coefficients, so it can be used to store data symbols.

[0100] The proof is as follows: Let the polynomial Expand and consider The coefficient of ,in ,according to , we can get ,here yes The permutation mapping, analogy can be obtained that the coefficients have indivual.

[0101] The conventional subcontracting number is still about The exponential level of , which brings greater storage complexity and computational cost. According to the above method, the MSR code with a smaller number of subpackets can be obtained. When considering Such MSR code ( ), the number of sub-packages is only , while conventional technical solutions require subcontracting ;

[0102] when When considering Such MSR code ( ), the number of subpackets obtained is only , while conventional technical solutions require subcontracting .

[0103] The process of determining the total coefficients of the multivariate polynomial provided in this embodiment is used to determine the number of subpackets, thereby reducing storage complexity and improving the efficiency of repairing failed nodes.

[0104] In some embodiments, the number of helper nodes is determined by the total coefficient of the multivariate polynomial and the polynomial coefficients corresponding to the helper nodes.

[0105] Specifically, the total coefficient of the multivariate polynomial is also the number of data that needs to be repaired. The data that each helping node can provide is determined by the polynomial coefficient corresponding to the helping node. Therefore, the number of helping nodes can be obtained here by the total coefficient of the multivariate polynomial / the polynomial coefficient corresponding to the helping node.

[0106] In some embodiments, the process of determining the number of helping nodes includes:

[0107] Get the total coefficients of a multivariate polynomial;

[0108] Obtaining polynomial coefficients of a first target multivariate polynomial;

[0109] The number of the helper nodes is determined according to the total coefficient and the polynomial coefficient of the first target multivariate polynomial.

[0110] It should be noted that the total coefficient of the multivariate polynomial takes into account that the multivariate polynomial needs to download information from the helper node. The polynomial coefficient of the first target multivariate polynomial here is the information downloaded from the helper node. The number of the helper node is determined based on the total coefficient / polynomial coefficient of the first target multivariate polynomial.

[0111] For example: a multivariate polynomial The information downloaded from the helper node (the polynomial coefficients of the first target multivariate polynomial) is: .

[0112] The multivariate polynomial Output variables about arguments The number of times is Then the corresponding coefficients are That is, the number of data that needs to be repaired. The corresponding number of coefficients is , that is, each helper node provides data, then the total amount of data provided is , the corresponding formula is: , that is, the number of helping nodes is .

[0113] It should be noted that the above example only corresponds to a certain multivariate polynomial distance. For the total coefficient, the number of its helping nodes is , the above only needs to explain The situation is sufficient.

[0114] The process of determining the number of helper nodes provided in this embodiment is only corresponding to the conventional technical solution. In this case, the application can achieve greater than In this case, the more helper nodes there are, the smaller the amount of data downloaded from the helper nodes, reducing network transmission overhead and repair time.

[0115] It should be noted that in the above embodiment, the number of helping nodes and the number of subpackets are known, and the bit rate is approximately ,along with The code rate of the MSR code in the conventional technical solution is limited to Nearly zero, the code rate is low and redundancy is excessive, making it difficult to meet the high code rate requirements of existing distributed storage systems. The code rate of this application approaches 1, supporting higher code rates (i.e., lower redundancy), which can better meet the high code rate requirements of modern distributed storage systems. Compared to conventional solutions, this application has a higher code rate and lower redundancy, thereby improving storage efficiency. A high code rate means that the system requires less redundant information when storing data, resulting in higher storage efficiency, which is particularly important in cost-sensitive storage scenarios such as cloud storage and big data analytics. Furthermore, this solution outperforms conventional solutions in terms of storage complexity, repair efficiency, and redundancy, making it more suitable for practical industrial applications. By reducing multivariate polynomial interpolation to multi-layer single-variable interpolation, the encoding and decoding processes are simplified, reducing implementation complexity. This design not only achieves optimal performance in theory, but also ensures high efficiency and feasibility in practical applications through specific implementation methods (such as the polynomial interpolation algorithm), making this solution more competitive in large-scale distributed storage systems.

[0116] In some embodiments, the quality of the MSR code construction requires two pre-set conditions, namely, after constructing the minimum storage regeneration code and before repairing the stored data of the failed node:

[0117] When the minimum storage regeneration code meets the preset conditions of the repair bandwidth and the preset conditions of the repair data, the step of reconstructing the multivariate polynomial from the polynomial coefficients of the helper node in at least one storage node using the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code is entered to repair the storage data of the failed node.

[0118] It should be noted that the preset repair bandwidth condition indicates that the constructed MSR code possesses optimal repair properties, meaning that the repair bandwidth of any failed node can reach the cut set bound. As mentioned in the above embodiments, effective repair of failed nodes is measured by repair bandwidth. Meeting the preset repair bandwidth condition indicates optimal repair capability, which in turn corresponds to improved repair efficiency. The cut set bound here corresponds to a lower bound on the repair bandwidth (RB) of the MDS array code. An MSR code that meets this lower bound has optimal repair properties.

[0119] The precondition for repairing data is that the constructed encoding based on multivariate polynomial coefficients has the MDS property, that is, any The information of each node can be used to repair all the original data. The repaired original data here refers to the corresponding data to be stored. This can be used as a performance condition for verifying the MSR code and as a repair step in the subsequent repair process. It can be set according to actual conditions in the subsequent repair process.

[0120] This embodiment provides that after the MSR meets the above two preset conditions, the MSR code is guaranteed to have the optimal repair property and the MDS property, thereby improving the repair efficiency of subsequent failed nodes and also improving the reliability of the repair process.

[0121] In some embodiments, the process of determining whether the minimum storage regeneration code satisfies a preset condition of the repair bandwidth includes:

[0122] Obtaining first invariant information of a first target failure node and second invariant information of a first target helping node;

[0123] Substituting the first invariant information into the second variable of the multivariate polynomial of the first target helper node to obtain the first target multivariate polynomial;

[0124] Substituting the second invariant information into the second variable of the multivariate polynomial of the first target failure node to obtain a second target multivariate polynomial;

[0125] If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, determining whether the polynomial coefficients of the first target multivariate polynomial are the same as the amount of data downloaded by the first target helping node;

[0126] If they are the same, it is determined that the repair bandwidth of the first target failed node reaches the cut set boundary, and it is determined that the minimum storage regeneration code meets the preset condition of the repair bandwidth.

[0127] Specifically, in this embodiment, the corresponding failed node can be downloaded from all the helper nodes or from one helper node, which is not limited here.

[0128] Get the first target failure node (first target failure node Stored polynomials ) first invariant information and the first target helper node (first target helper node Stored polynomials )'s second invariant information .

[0129] The first invariant information is brought into the second variable of the multivariate polynomial of the first target helper node to obtain the first target multivariate polynomial, that is, .

[0130] The second invariant information is introduced into the second variable of the multivariate polynomial of the first target failure node to obtain the second target multivariate polynomial, that is, .

[0131] If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, then the equivalence is determined, i.e., the failure node Available from the Help node Upload and download polynomials The corresponding coefficient (this is the information on the node linear combination of ), which is equivalent to The corresponding coefficient.

[0132] Further determine whether the polynomial coefficient of the first target multivariate polynomial is the same as the amount of data downloaded by the first target help node. It should be noted that the amount of data downloaded by the first target help node is a certain value. , according to the setting of the number of subpackets in the formula, the size of the subpacket number can be calculated based on the above embodiment, and the number of helper nodes can be calculated based on the size corresponding to the number of helper nodes determined in the above embodiment. Finally, if the polynomial coefficient of the first target multivariate polynomial is the same as the amount of data downloaded by the first target helper node, it is determined that the repair bandwidth of the first target failed node reaches the cut set boundary, and it is determined that the minimum storage regeneration code meets the preset conditions of the repair bandwidth. At this time, the first target multivariate polynomial , corresponding to different numbers of coefficients ,equal , that is, reaching the cut set boundary.

[0133] This embodiment provides a process for determining whether a minimum storage regenerating code satisfies the preset repair bandwidth requirements, enabling the MSR code to possess optimal repair properties, ensuring that the repair bandwidth for any failed node reaches the cut set bound. This minimizes the repair bandwidth, thereby reducing network bandwidth usage and accelerating data repair.

[0134] In some embodiments, the process of determining whether the minimum storage regeneration code satisfies a preset condition for repairing data includes:

[0135] The number of variables in the multivariate polynomial is used as the number of storage nodes;

[0136] Generate an output variable corresponding to the current storage node and at least one corresponding variable based on variable data interpolation of the current storage node itself and other storage nodes;

[0137] Merging the output variable corresponding to the at least one generated storage node and at least one variable to obtain a target multivariate polynomial;

[0138] If the multivariate polynomial is the same as the target multivariate polynomial, it is determined that the data of any data block is repaired to obtain the data to be stored, and it is determined that the minimum storage regeneration code meets the preset condition for repairing the data.

[0139] Specifically, the number of variables of the multivariate polynomial is used as the number of storage nodes. For example, if there are variables , as , which corresponds to storage nodes.

[0140] The output variable and at least one corresponding variable corresponding to the current storage node are generated based on the variable data interpolation of the current storage node itself and other storage nodes. Here, the variable data interpolation of each storage node itself and other storage nodes is generated.

[0141] Furthermore, here we consider any two different integers , first interpolate to generate the corresponding output variable , each storage node provides its own variable data of multivariate polynomial for interpolation generation. Considering the symmetry between the coefficients of each multivariate polynomial, the data set can be obtained through the above data to repair the Then, continue interpolation to generate The last variable, the second to last variable, until the interpolation reaches the first variable.

[0142] The generated output variable and the dataset corresponding to at least one variable are merged to obtain a target multivariate polynomial. The multivariate polynomial and the target multivariate polynomial are compared. If they are identical, it is determined that the data in any data block can be used to repair the data to be stored, i.e., the MSR code meets the preset conditions for data repair. This comparison involves comparing the coefficients of the multivariate polynomial and the relational expression. If the original multivariate polynomial can be reconstructed, the data in any data block of this embodiment can be used to repair the data to be stored.

[0143] The minimum storage regeneration code provided in this embodiment satisfies the determination process of the preset conditions for repairing data, ensuring that the information of the node of any data block can repair all the original data, so that the constructed MSR code has the MDS property, theoretically achieving the optimal fault tolerance and storage efficiency, and improving the performance and reliability of the system in practical applications.

[0144] In some embodiments, generating an output variable corresponding to the current storage node and at least one corresponding variable based on interpolation of variable data corresponding to the current storage node itself and other storage nodes includes:

[0145] Substitute the current storage node into the first variable of the multivariate polynomial and generate the output variable corresponding to the current storage node by interpolation;

[0146] Reduce storage nodes one by one, and in the process of reduction, bring the new current storage node into the first variable of the multivariate polynomial, and interpolate to generate the target variable corresponding to the new current storage node by reverse deduction starting from the last variable; wherein, the number of times the storage node is reduced is the same as the number of times the target variable is determined.

[0147] Known ,So , which can be represented by a node set Information in Repair multivariate polynomials .

[0148] Review of polynomials About variables The number of times is less than or equal to ,about The number of times is less than or equal to Consider any two different integers ;

[0149] node Provide data ;

[0150] node Provide data ;

[0151] That is, for any ,node Provide data .

[0152] According to the symmetry between the coefficients, the data set can be obtained from the above data , because the polynomial is about Number of times , so the obtained data set can be used to repair The output variable corresponding to the current storage node has been interpolated and generated here.

[0153] The storage nodes are reduced one by one, and the new current storage node is brought into the first variable of the multivariate polynomial to obtain the target variable corresponding to the new current storage node by reverse deduction starting from the last variable.

[0154] Integer Take a collection , you can get the data set .

[0155] Since the polynomial is about The number of times is less than or equal to , and the amount of data in the above dataset is , so we can interpolate to get the polynomial .

[0156] Similarly, the integer Take a collection , you can get the data set .

[0157] Since the polynomial is about Number of times , and the amount of data in the above dataset is , so we can interpolate to get the polynomial .

[0158] right Repeat the above similar steps and you can finally get the polynomial .

[0159] For the sake of convenience, taking the three-variable polynomial as an example, it is known that ,So , which can be represented by a node set Information in Repair the three-variable polynomial .

[0160] prove: , can be obtained from the node The polynomial corresponding to the data is stored in get , that is, . Similarly, you can use the node The data in Because the polynomial About variables The degree of is equal to 1, so it can be and (according to and Equivalent) interpolation is obtained At this point, you can fix ,Will Run all over , the available data set ,because About variables Number of times , and there are interpolation points, so we can Interpolation Similarly, Run all over , can be Interpolation .

[0161] The encoding provided in this embodiment is arbitrary The surviving nodes can repair all the original data, which reduces the multivariate polynomial interpolation to multi-layer single-variable interpolation to reduce the difficulty of proof. For each storage node, the variables corresponding to the storage node can be interpolated to reconstruct the current multivariate polynomial, laying the foundation for the subsequent reconstruction of the total multivariate polynomial.

[0162] Furthermore, the present application also provides a distributed storage system, comprising a repair node and at least one storage node;

[0163] The repair node is used to execute the steps of the above data repair method to repair the storage data of the failed node in the storage node.

[0164] Figure 2 A structural diagram of a distributed storage system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the repair node repairs the failed nodes corresponding to multiple storage nodes. The repair process can refer to the above embodiment and will not be described in detail here.

[0165] For an introduction to a distributed storage system provided by this application, please refer to the above method embodiment, which will not be repeated here. It has the same beneficial effects as the above data repair method.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0167] The embodiment of the present application also provides a data repair device, Figure 3 A structural diagram of a data repair device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device includes:

[0168] An acquisition module 11 is configured to acquire data to be stored and process the data to be stored into blocks to obtain at least one data block;

[0169] A mapping processing module 12 is configured to perform mapping processing on the storage data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code and determine the storage data corresponding to at least one storage node;

[0170] The repair processing module 13 is used to reconstruct a multivariate polynomial from the polynomial coefficients of the helper nodes in the at least one storage node using a multivariate polynomial interpolation method corresponding to the minimum storage regeneration code to repair the storage data of the failed node if there is a failed node in the at least one storage node.

[0171] The description of the features in the embodiment corresponding to the data repair device can refer to the relevant description of the embodiment corresponding to the data repair method, and will not be repeated here.

[0172] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data repair method embodiments.

[0173] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data repair method embodiments when run.

[0174] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0175] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned data repair method embodiments are implemented.

[0176] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data repair method embodiments are implemented.

[0177] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The above is a detailed introduction to a data repair method, distributed storage system, device, medium, and product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A data repair method, characterized in that: include: Acquire data to be stored, and divide the data to be stored into blocks to obtain at least one data block; Mapping the stored data of at least one data block to obtain multivariate polynomial coefficients to construct a minimum storage regeneration code, and determining the stored data corresponding to at least one storage node; If there is a failed node in at least one storage node, reconstructing a multivariate polynomial from polynomial coefficients of helper nodes in at least one storage node using a multivariate polynomial interpolation method corresponding to a minimum storage regeneration code to repair the stored data of the failed node; Correspondingly, after constructing the minimum storage regeneration code and before repairing the stored data of the failed node, the method further includes: When the minimum storage regenerating code satisfies a preset condition for repair bandwidth and a preset condition for repair data, the method proceeds to the step of reconstructing a multivariate polynomial from polynomial coefficients of a helper node in at least one storage node using a multivariate polynomial interpolation method corresponding to the minimum storage regenerating code to repair the stored data of the failed node; wherein the preset condition for repair bandwidth is that the repair bandwidth of any failed node reaches a cut set bound, which is a lower bound of the repair bandwidth of an array code of a maximal distance separable code; and the preset condition for repair data is that information from any data block can be used to repair the data to be stored.

2. The data repair method according to claim 1, characterized in that: The construction process of the minimum storage regeneration code includes: At least one multivariate polynomial coefficient of a multivariate polynomial is obtained based on a finite field; wherein the power data of the output variable of the multivariate polynomial is greater than or equal to 0 and less than or equal to ; Tuple information corresponding to variables representing a multivariate polynomial; the coefficients of the multivariate polynomial comply with the permutation principle of the output variables; The stored data is stored in at least one multivariate polynomial coefficient.

3. The data repair method according to claim 2, characterized in that: The minimum number of subpackets for storing the regenerated code is determined by the total coefficients of the multivariate polynomial and the number of data blocks.

4. The data repair method according to claim 3, characterized in that: The process of determining the total coefficients of the multivariate polynomial includes: Get the monotone non-decreasing value of the multivariate polynomial The data relation of tuple; Determine the number of groups corresponding to the data relational expression according to the interpolation method to determine the number of the data relational expressions; Determine the number of variables of the multivariate polynomial according to the variable substitution principle of the multivariate polynomial coefficients; wherein the substitution principle is that the output variable remains unchanged under substitution; The total coefficients of the multivariate polynomial are determined according to the number of variables and the quantity of the data relationship expressions.

5. The data repair method according to claim 4, characterized in that: The number of the helping nodes is determined by the total coefficient of the multivariate polynomial and the polynomial coefficients corresponding to the helping nodes.

6. The data repair method according to claim 1, wherein: The process of determining whether the minimum storage regeneration code meets the preset condition of the repair bandwidth includes: Obtaining first invariant information of a first target failure node and second invariant information of a first target helping node; Substituting the first invariant information into the second variable of the multivariate polynomial of the first target helper node to obtain a first target multivariate polynomial; Substituting the second invariant information into the second variable of the multivariate polynomial of the first target failure node to obtain a second target multivariate polynomial; If the polynomial coefficients of the first target multivariate polynomial and the second target multivariate polynomial are the same, determining whether the polynomial coefficients of the first target multivariate polynomial are the same as the amount of data downloaded by the first target helping node; If they are the same, it is determined that the repair bandwidth of the first target failed node reaches the cut set boundary, and it is determined that the minimum storage regeneration code meets the preset condition of the repair bandwidth.

7. The data repair method according to claim 6, characterized in that: The process of determining the number of helping nodes includes: Obtaining total coefficients of the multivariate polynomial; Obtaining polynomial coefficients of the first target multivariate polynomial; The number of the helping nodes is determined according to the total coefficient and the polynomial coefficient of the first target multivariate polynomial.

8. The data repair method according to claim 5, characterized in that: The process of determining whether the minimum storage regeneration code meets the preset conditions for repairing data includes: Using the number of variables in the multivariate polynomial as the number of storage nodes; Generate an output variable corresponding to the current storage node and at least one corresponding variable based on variable data interpolation of the current storage node itself and other storage nodes; Merging the output variable corresponding to the at least one generated storage node and at least one variable to obtain a target multivariate polynomial; If the multivariate polynomial is identical to the target multivariate polynomial, it is determined that the data of any data block repairs the data to be stored, and it is determined that the minimum storage regeneration code meets the preset condition for repairing the data.

9. The data repair method according to claim 8, characterized in that: Generating an output variable corresponding to the current storage node and at least one corresponding variable based on variable data interpolation of the current storage node itself and other storage nodes, including: Substituting the current storage node into the first variable of the multivariate polynomial, interpolating to generate an output variable corresponding to the current storage node; The storage nodes are reduced one by one, and during the reduction process, the new current storage node is brought into the first variable of the multivariate polynomial to generate the target variable corresponding to the new current storage node by interpolation and reversed from the last variable; wherein the number of times the storage nodes are reduced is the same as the number of times the target variable is determined.

10. The data repair method according to claim 1, wherein: Reconstructing a multivariate polynomial from polynomial coefficients of a helper node in at least one storage node using a multivariate polynomial interpolation method corresponding to a minimum storage regeneration code to repair the storage data of the failed node, including: Reconstructing the total coefficients of the multivariate polynomial from the polynomial coefficients of the helper node using a multivariate polynomial interpolation method corresponding to a minimum storage regeneration code to repair the data to be stored; and determining the storage data of the failed node based on the data to be stored; Alternatively, the multivariate polynomial interpolation method corresponding to the minimum storage regeneration code is used to reconstruct the multivariate polynomial coefficients of the corresponding helping node from the polynomial coefficients of the helping node to repair the storage data of the failed node.

11. A distributed storage system, characterized in that: including a repair node and at least one storage node; The repair node is used to execute the steps of the data repair method described in any one of claims 1 to 10 to repair the storage data of the failed node in the storage node.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data repair method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data repair method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data repair method according to any one of claims 1 to 10 are implemented.

Images