Data processing method based on erasure code and related apparatus

Abstract

The application relates to the computer technical field and discloses a data processing method based on erasure code and related devices. The application obtains a plurality of original data blocks; divides the plurality of original data blocks into two sequential groups and a plurality of cross-group data block groups; the original data block serial numbers in the sequential groups are continuous, each cross-group data block group contains the same number of original data blocks, and the cross-group data block serial numbers are discontinuous; a plurality of check blocks of the plurality of original data blocks are generated, the plurality of check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. Thus, in the case of occupying fewer hard disks, data recovery can be performed in a local scope in the data block recovery process, global data recovery is maximally reduced, reconstruction cost is effectively reduced, and higher fault tolerance is achieved under the same condition.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and related apparatus based on erasure coding. Background Technology

[0002] Erasure coding is a technology used to improve the reliability of data in cloud storage systems. Currently, cloud storage reliability technologies are divided into erasure coding and multi-replica technology. Because multi-replica technology consumes a lot of disk space in the era of big data, resulting in low disk utilization and frequent unrecoverable errors in disk arrays, erasure coding technology is even more needed to quickly reconstruct lost data.

[0003] Current erasure coding techniques are mainly divided into two categories: one is erasure coding based on Galois field operations; the other is erasure coding based on "XOR" operations. Galois field erasure coding is mainly Reed-Solomon (RS) code. RS erasure coding can be infinitely expanded when storage system resources are sufficient, and RS erasure coding has the Maximum Distance Separable (MDS) property. However, redundant data and original data in RS have a linear combination, which means that once relevant data needs to be recovered, information from all remaining data units must be collected. This leads to additional time and space overhead. RS erasure coding cannot guarantee data recovery within the largest possible group, relying more on global data recovery. Therefore, how to use erasure coding technology for data processing to reduce data repair time and disk usage is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a data processing method and related apparatus based on erasure coding, which solves the problem of how to use erasure coding technology for data processing, thereby reducing data repair time and disk usage.

[0005] In a first aspect, embodiments of this application provide a data processing method based on erasure coding, including:

[0006] Obtain multiple raw data blocks;

[0007] The multiple original data blocks are divided into two sequential groups and multiple cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive.

[0008] Multiple check blocks are generated from the multiple original data blocks. The multiple check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group.

[0009] In one possible implementation, if the plurality of original data blocks correspond to multiple cross-group data block group partitioning strategies, then one of the cross-group data block group partitioning strategies is selected to divide the plurality of original data blocks into multiple cross-group data block groups.

[0010] In one possible implementation, the method further includes:

[0011] Based on a pre-built recovery status indication table, it is determined whether the failed data block is recoverable. The recovery status indication table is used to record the distribution of failed data blocks and the corresponding recovery indication information. If the failed data block is recoverable, the recovery indication information is a recovery condition. If the failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable.

[0012] If the failed data block is recoverable, then the failed data block is reconstructed using multiple check blocks of the original data block.

[0013] In one possible implementation, constructing the recovery status indicator table specifically includes:

[0014] Construct a sample of failed data blocks;

[0015] For each type of failed data block distribution sample, a determinant coefficient matrix is ​​constructed based on the unfailed data blocks to reconstruct the failed data block distribution sample;

[0016] Given that the determinant of the determinant coefficient matrix is ​​not zero, solve for the coefficient relationships in the determinant coefficient matrix.

[0017] Construct a correspondence between the distribution samples of the failed data blocks and the coefficient relationships.

[0018] In one possible implementation, the two global check blocks include a first global check block and a second global check block. In the matrix of row and column coefficients, the first global check block corresponds to the coefficients in the nth row of the determinant coefficient matrix, and the second global check block corresponds to the coefficients in the (n+1)th row of the determinant coefficient matrix, wherein the determinant coefficient matrix includes a total of n+1 rows of coefficients; and each original data block corresponds to one coefficient in one row of coefficients.

[0019] If there are 4 failed data blocks, and these failed data blocks are sequentially designated as the first failed data block, the second failed data block, the third failed data block, and the fourth failed data block, and in the determinant coefficient matrix, the first failed data block corresponds to the first coefficient, the second failed data block corresponds to the second coefficient, the third failed data block corresponds to the third coefficient, and the fourth failed data block corresponds to the fourth coefficient, then:

[0020] If all the failed data blocks are original data blocks, then the corresponding coefficient relationships are as follows: the first coefficient is not equal to the third coefficient, the second coefficient is not equal to the fourth coefficient, and the sum of the first coefficient and the third coefficient is not equal to the sum of the second coefficient and the fourth coefficient.

[0021] If a failed data block includes one cross-group check block and three original data blocks, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed.

[0022] If the failed data block includes one cross-group check block and three original data blocks, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: the first coefficient is not equal to zero, the second coefficient is not equal to zero, and the difference between the third coefficient and the first coefficient is not zero.

[0023] If the failed data block includes one global check block and three original data blocks, the corresponding coefficient relationship is: the difference between the third coefficient and the first coefficient is not zero, and the first coefficient is not equal to the third coefficient.

[0024] If a failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as either the failed cross-group check block or the failed global check block, then the failed data block cannot be reconstructed.

[0025] If the failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as the failed cross-group check block and the failed global check block, then the corresponding coefficient relationship is: the first coefficient is not zero;

[0026] If the failed data block includes two cross-group check blocks and two original data blocks, then the corresponding coefficient relationship is: the first coefficient is not equal to the second coefficient, and the first coefficient and the second coefficient are not zero;

[0027] If a failed data block includes two global check blocks and two original data blocks, and the two failed original data blocks are in the same group as one non-failed cross-group check block, then the failed data block cannot be reconstructed.

[0028] If the failed data block includes 2 global check blocks and 2 original data blocks, and the 2 failed original data blocks are in the same group as the 2 unfailed cross-group check blocks, then the corresponding coefficient relationship is: coefficients are unconstrained.

[0029] If the failed data block includes two cross-group check blocks, one global check block, and one original data block, then the corresponding coefficient relationship is: the first coefficient is not zero;

[0030] If a failed data block includes two global check blocks, one cross-group check block, and one original data block, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed.

[0031] If the failed data block includes 2 global check blocks, 1 cross-group check block, and 1 original data block, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: coefficients are unconstrained.

[0032] In one possible implementation, the minimum number of data blocks required to reconstruct a single failed data block from the plurality of original data blocks is the number of data blocks included in a single cross-group data block group.

[0033] In one possible implementation, the maximum number of failed data blocks allowed for the plurality of original data blocks is the total number of check blocks.

[0034] Secondly, embodiments of this application provide a data processing apparatus based on erasure coding, the apparatus comprising:

[0035] The acquisition module is used to acquire multiple raw data blocks;

[0036] The grouping module is used to divide the multiple raw data blocks into two sequential groups and multiple cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive.

[0037] The verification block determination module is used to generate multiple verification blocks for the multiple original data blocks. The multiple verification blocks include two global verification blocks, sequential group verification blocks corresponding to each sequential group, and cross-group verification blocks corresponding to each cross-group data block group.

[0038] In one possible implementation, the grouping module is further configured to select one of the cross-group data block grouping strategies and divide the multiple original data blocks into multiple cross-group data block groups if the multiple original data blocks correspond to multiple cross-group data block grouping strategies.

[0039] In one possible implementation, the device further includes:

[0040] The recovery determination module is used to determine whether a failed data block is recoverable based on a pre-built recovery status indication table. The recovery status indication table is used to record the distribution of failed data blocks and the corresponding recovery indication information. If a failed data block is recoverable, the recovery indication information is a recovery condition. If a failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable.

[0041] The reconstruction module is used to reconstruct the failed data block using multiple check blocks of the original data block if the failed data block is recoverable.

[0042] In one possible implementation, the device further includes:

[0043] The sample construction module is used to construct samples of failed data blocks;

[0044] The matrix building module is used to build a determinant coefficient matrix for reconstructing the distribution sample of the failed data blocks based on the unfailed data blocks for each type of failed data block distribution sample.

[0045] The calculation module is used to solve the coefficient relationships in the determinant coefficient matrix under the premise that the determinant of the determinant coefficient matrix is ​​not zero.

[0046] The correspondence construction module is used to construct the correspondence between the distribution samples of the failed data blocks and the coefficient relationships.

[0047] Thirdly, embodiments of this application provide an electronic device, including:

[0048] processor;

[0049] Memory used to store the processor's executable instructions;

[0050] The processor is configured to execute the instructions to implement any erasure coding-based data processing method provided in the first aspect above.

[0051] Fourthly, embodiments of this application also provide a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps of any erasure coding-based data processing method provided in the first aspect above.

[0052] Fifthly, embodiments of this application provide a computer program product, including a computer program that is executed by a processor to implement the steps of any erasure coding-based data processing method provided in the first aspect above.

[0053] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0054] The erasure coding-based data processing method provided in this application involves acquiring multiple raw data blocks; dividing these raw data blocks into two sequential groups and multiple cross-group data block groups; wherein the sequence numbers of the raw data blocks in the sequential groups are consecutive, and each cross-group data block group contains the same number of raw data blocks with non-consecutive data block sequence numbers; and generating multiple check blocks for the multiple raw data blocks, including two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. By dividing the raw data blocks into multiple cross-group data block groups, the number of cross-group data block groups and the number of data blocks included in each cross-group data block group can be flexibly adjusted. Data block sharing is also possible among the multiple cross-group data block groups. During data block recovery, data recovery can be performed within a local scope, minimizing the need for global data recovery, effectively reducing reconstruction costs, network bandwidth, and computation time. This better ensures the reliability and integrity of data during cloud computing storage. Simultaneously, it offers higher fault tolerance under the same conditions, occupies as little hard disk space as possible, and saves hard disk space.

[0055] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A structural diagram of an RS code provided in an embodiment of this application;

[0058] Figure 2 A structural diagram of an LRC erasure coding scheme provided in an embodiment of this application;

[0059] Figure 3 A flowchart illustrating a data processing method based on erasure coding, provided for an embodiment of this application;

[0060] Figure 4 A cross-group erasure coding structure diagram of the tuple (6,2,2,2) provided in the embodiments of this application;

[0061] Figure 5A cross-group erasure coding structure diagram of the tuple (6,2,2,3) provided in the embodiments of this application;

[0062] Figure 6 A flowchart illustrating the method for constructing a recovery status indicator table provided in an embodiment of this application;

[0063] Figure 7 A schematic diagram of the structure of the erasure coding-based data processing device provided in the embodiments of this application;

[0064] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0066] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0067] (1) In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0068] (2) "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A alone, A and B at the same time, and B alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0069] (3) A server is a service for a terminal. The services include providing resources to the terminal and storing terminal data. A server is a counterpart to the application installed on the terminal and works in conjunction with the application on the terminal.

[0070] (4) A terminal can refer to either a software application (APP) or a client. It has a visual display interface that allows interaction with the user; it corresponds to a server and provides local services to the client. For software applications, except for some applications that run only locally, they are generally installed on ordinary client terminals and need to cooperate with the server to run. With the development of the Internet, commonly used applications include email clients for sending and receiving emails, and instant messaging clients. For these types of applications, corresponding servers and service programs are needed on the network to provide services such as database services and configuration parameter services. Therefore, a specific communication connection needs to be established between the client terminal and the server to ensure the normal operation of the application.

[0071] (5) Data fragments: The smallest coded unit formed by the system dividing the original user data.

[0072] (6) Parity fragments: Data information outside the data block, auxiliary data information generated to recover the data block.

[0073] (7) Local check block: It is only linearly related to the grouped data blocks.

[0074] (8) Global check block: It has a linear correlation with all data blocks.

[0075] (9) MDS code: Maximum distance separable code, a linear encoding method that satisfies the Singleton boundary. Compared with other codes, it has the lowest storage overhead while maintaining the same fault tolerance.

[0076] (10) Strip: A redundant set consisting of multiple data blocks and their corresponding parity blocks. If a certain number of coded blocks are lost, they can be regenerated by performing operations on the remaining coded blocks in the strip.

[0077] (11) Storage overhead: The total number of initial data blocks and parity blocks.

[0078] (12) Fault tolerance rate: When n blocks of a coded block are lost, these n blocks of data may be any combination of data blocks, local parity blocks, and global parity blocks. The fault tolerance rate is theoretically the ratio of the number of reconstructed combinations to the total number of combinations.

[0079] (13) Fault tolerance: The maximum number of coded blocks that a stripe can theoretically tolerate failing. Assuming that the fault tolerance of an erasure code is n, then this erasure code can and can only reconstruct the failed block in the case of no more than n coded blocks failing (theoretically repairable).

[0080] (14) Reconstruction overhead: Data reconstruction is the number of codes read from the stripe.

[0081] The number of any elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinction only and has no limiting meaning.

[0082] Erasure coding is a technology used to improve the reliability of data in cloud storage systems. Currently, cloud storage reliability technologies are divided into erasure coding and multi-replica technology. Because multi-replica technology consumes a lot of disk space in the era of big data, resulting in low disk utilization and frequent unrecoverable errors in disk arrays, erasure coding technology is needed to quickly reconstruct lost data. Therefore, many fields are paying close attention to improving erasure coding technology, hoping to optimize data recovery time while reducing disk usage.

[0083] Current erasure coding techniques are mainly divided into two categories: one is erasure coding based on Galois field operations; the other is erasure coding based on "XOR" operations.

[0084] Among them, Galois erasure coding primarily uses Reed-Solomon (RS) codes. RS erasure coding can be infinitely expanded when storage system resources are sufficient, and it possesses the maximum distance separable (MDS) property. However, in RS, both redundant and original data undergo linear combinations, meaning that to recover relevant data, all remaining data units must be collected, resulting in additional time and space overhead. Local Reconstruction (LRC) erasure coding, on the other hand, does not perform complete linear combinations of original and redundant data; only partial linear combinations of relevant data are performed. LRC can reduce data retrieval, further improving performance in both time and space.

[0085] RS coding requires all data for recovery during decoding, which often takes a long time due to bandwidth and disk I / O limitations. To address this, LRC coding was proposed. This method can recover data using only a small amount of related missing and redundant data. Erasure coding cannot guarantee data recovery within a single block, relying more on global data recovery. Therefore, how to use erasure coding technology for data processing to reduce data recovery time and disk usage is an urgent problem to be solved.

[0086] In view of this, this application provides a data processing method and related apparatus based on erasure coding, which is used to solve the problem of how to use erasure coding technology for data processing, thereby reducing data repair time and disk usage.

[0087] The inventive concept of this invention is as follows: This application obtains multiple raw data blocks; divides the multiple raw data blocks into two sequential groups and multiple cross-group data block groups; the sequence numbers of the raw data blocks in the sequential groups are consecutive, and each cross-group data block group contains the same number of raw data blocks, but the sequence numbers of the data blocks in the cross-group data block groups are not consecutive; multiple check blocks are generated for the multiple raw data blocks, including two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. Therefore, by dividing multiple raw data blocks into multiple cross-group data block groups, not only can the number of cross-group data block groups and the number of data blocks included in each cross-group data block group be flexibly adjusted, but data block sharing can also be performed between multiple cross-group data block groups. During data block recovery, data recovery can be performed within a local scope, minimizing the need for global data recovery and effectively reducing reconstruction costs. Simultaneously, it has higher fault tolerance under the same conditions, occupies as little hard disk space as possible, and saves hard disk space.

[0088] After introducing the inventive concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit it. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0089] The method provided in this application is applicable to access control in scenarios where multiple clients share cloud disks, and is also applicable to cloud disk services provided based on distributed block storage cluster technology. In currently common distributed block storage systems, the method provided in this application can be directly applied to manage client access permissions, resolving potential data conflict issues when multiple clients share data. Furthermore, the method provided in this application ensures the consistency of client permissions within the cluster, guaranteeing that the failure of a small number of nodes in the storage cluster will not affect client access to services, thereby improving the availability and reliability of storage services.

[0090] The method provided in this application is also applicable to cloud storage systems, primarily used in situations where a single point of failure occurs in an offline cloud storage system, enabling rapid data recovery via cross-group MLRC. Currently, systems typically employ RS (Replica Set) algorithms or a 3-replica configuration. However, compared to traditional RS, LRC, and 3-replica algorithms, the newly proposed MLRC (Multi-Step Local Reconstruction Codes) algorithm can recover data quickly while reducing hard drive usage. This ensures the reliability of data in the cloud storage system and provides a user-friendly experience for upper-layer PASS (Platform as a Service) services.

[0091] Of course, the methods provided in this application embodiment are not limited to the application scenarios described above, and can also be used in other possible application scenarios. This application embodiment does not impose any limitations. To further illustrate the technical solutions provided in this application embodiment, a detailed description is provided below in conjunction with the accompanying drawings and specific embodiments. Although this application embodiment provides method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-creative labor. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in this application embodiment.

[0092] To facilitate understanding of the erasure coding-based data processing method provided in the embodiments of this application by those skilled in the art, RS codes and LRC codes will be introduced below with reference to the accompanying drawings.

[0093] refer to Figure 1 This is a structural diagram of an RS code provided in an embodiment of this application. Figure 1 As shown, assume the original data of the RS code is divided into 6 original data blocks and 4 parity blocks, with each parity block linearly related to the 6 original data blocks. This means that if one original data block is lost, all the remaining original data blocks must be collected to recover the lost block.

[0094] The purpose of LRC erasure coding is to reduce the resources required for data recovery; therefore, LRC only requires a portion of the original data blocks during the data recovery process. For example... Figure 2 As shown, the original data is divided into 6 original data blocks, generating 4 check blocks. Two of these check blocks, Q0 and Q1, are globally correlated with the 6 original data blocks. The remaining two check blocks, P0 and P1, are sequential grouped check blocks formed by dividing the original data blocks into two groups of equal size. Each group of original data blocks corresponds to one sequential grouped check block. For ease of description, the original data blocks are denoted as (d0, d1, d2, d3, d4, d5). The sequential grouped check block P0 is calculated from (d0, d1, d2), and the sequential grouped check block P1 is calculated from (d3, d4, d5).

[0095] To understand this, we can define an LRC tuple (k, l, r), where k is the number of original data blocks, l is the number of sequential check blocks (SPCs), each SPC includes k / l of original data blocks, and r is the number of global check blocks. There are a total of n = k + l + r data segments, therefore, compared to the original data segments, we need to increase the number of data segments by... Double the storage space.

[0096] Next, the number of parity blocks should be chosen to give erasure coding the MR (Maximally Recoverable) property. The MR property is explained as follows: For example, given an LRC tuple (6,2,2), a sequential parity block has 3 original data blocks, allowing a maximum of 4 original data blocks to be corrupted. However, this does not yet possess the MR property because the 4 corrupted original data blocks cannot be allowed to appear arbitrarily. For example, if d0, d1, d2, and p0 are all corrupted, the system cannot recover the original data because there are only two global parity blocks, making it impossible to solve for three unknowns. A system with the MR property can handle the loss of any parity block up to the upper limit r+1. Constructing the equation for the failed original data blocks is not difficult. The difficulty lies mainly in constructing an equation with the MR property that can recover any missing original data block with an upper limit of r+1 corrupted data.

[0097] Therefore, the tuple (6,2,2) can recover any 3 missing original data blocks, and has an 86% chance of handling 4 missing original data blocks. If data fragments are missing, quickly checking whether recovery is possible is a pressing issue. If the sequential check block is available, and at least one original data block has been erased, swap the sequential check block and the original data blocks. After the swap, the original data blocks are available, but the sequential check block is unavailable. Perform the corresponding operation for all grouping cases, and then test whether the original data blocks and the global check block are available. If erasing no more than the global check block, use an algorithm to determine if data recovery is possible.

[0098] Based on the example of the tuple (6,2,2) above, some general properties can be observed. Let (k,l,r) represent this:

[0099] (1) A single raw data block is erased and can be recovered by k / l sequentially grouped check blocks.

[0100] (2) Any original data block with a maximum number of erases of r+1 can be recovered by the system, which has MR properties.

[0101] Based on the above two points, we can further deduce the lower bound of the check block data:

[0102] For any (n,k) linear codes, where k is the original data block and nk is the parity data block, the following properties must be satisfied: nk≥l+r

[0103] (1) The system can recover any r+1 data blocks that are erased.

[0104] (2) Single data block erasure can be recovered by using k / l data blocks.

[0105] At this point, the lowest lower bound of the lowest data check block of the LRC is l+r.

[0106] From this, we can learn about the relevant knowledge and some properties of RS codes and LRC codes. Based on the above knowledge, it is easier to understand the data processing method based on erasure codes provided in the following embodiments of this application.

[0107] refer to Figure 3 This is a flowchart illustrating a data processing method based on erasure coding provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0108] In step 301, multiple raw data blocks are acquired.

[0109] In step 302, multiple raw data blocks are divided into two sequential groups and multiple cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive.

[0110] In step 303, multiple check blocks are generated for multiple original data blocks. The multiple check blocks include two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group.

[0111] For example, such as Figure 4 As shown, in the case of LRC grouping, another dimension of information can be added. This dimension mainly combines the data within the group for grouping. For example, when the LRC tuple is (6,2,2), (6,2,2) represents 6 original data blocks, 2 global check blocks, and 2 sequential grouping check blocks. Two cross-group check blocks P2 and P3 are introduced between groups, where P2 is generated from d0, d2, and d4, and P3 is generated from d1, d3, and d5. After generation, the tuple becomes (6,2,2,2). The tuple (6,2,2,2) represents 6 original data blocks d0, d2, d3, and d4. 1、 d2, d3, d4, d5, 2 global check blocks Q0 and Q1, 2 sequential group check blocks P0 and P1, and 2 cross-group check blocks P2 and P3.

[0112] In one possible implementation, if multiple original data blocks correspond to multiple cross-group data block group partitioning strategies, then one of the cross-group data block group partitioning strategies is selected to divide the multiple original data blocks into multiple cross-group data block groups.

[0113] For example, in addition to Figure 4 In addition to the grouping shown, another strategy can be used for grouping, such as... Figure 5As shown, each group spans two original data blocks. For example, the LRC tuple (6,2,2,3) represents 6 original data blocks d0, d2, d3, d4, d5, d6, d7, d8, d9, d1, d2, d3 ... 1、 d2, d3, d4, d5; 2 global check blocks Q0 and Q1; 2 sequential group check blocks P0 and P1; 3 cross-group check blocks P2, P3, P4. Among them, P2 is generated from d0 and d3, P3 is generated from d1 and d4, and P4 is generated from d2 and d5.

[0114] In one possible implementation, the erasure coding-based data processing method provided in this application can determine whether a failed data block is recoverable based on a pre-built recovery status indication table. The recovery status indication table is used to record the distribution of failed data blocks and the corresponding recovery indication information. If the failed data block is recoverable, the recovery indication information is a recovery condition. If the failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable. If the failed data block is recoverable, the failed data block is reconstructed using multiple check blocks of the original data block.

[0115] In one possible implementation, constructing a recovery status indicator table can be performed as follows: Figure 6 The steps shown are as follows:

[0116] In step 601, a sample of failed data blocks is constructed;

[0117] In step 602, for each type of failed data block distribution sample, a determinant coefficient matrix is ​​established based on the unfailed data blocks to reconstruct the failed data block distribution sample;

[0118] In step 603, based on the premise that the determinant of the determinant coefficient matrix is ​​not zero, the coefficient relationship in the determinant coefficient matrix is ​​solved;

[0119] In step 604, a correspondence is established between the distribution samples of failed data blocks and the coefficient relationships.

[0120] For example, taking 6 original data blocks and the tuple (6,2,2,2) as an example, we will discuss the specific process of constructing a recovery status indicator table when there are 4 failed data blocks. Where d0, d... 1、 d2, d3, d4, and d5 represent 6 original data blocks, Q0 and Q1 represent 2 global check blocks, P0 and P1 represent 2 sequential group check blocks, and P2 and P3 represent 2 cross-group check blocks. When all original data blocks are valid, the corresponding determinant coefficient matrix (1) can be obtained as follows:

[0121]

[0122] The following analysis addresses several scenarios where four data blocks fail:

[0123] (1) All four failed data blocks are original data blocks. Assuming that the four original data blocks d0, d1, d2, and d3 are failed, the determinant coefficient matrix (2) can be expressed as:

[0124]

[0125] If we want to theoretically reconstruct the system under the failure modes of 4 data blocks, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix must not be 0, which can be expressed as formula (3):

[0126]

[0127] Simplifying, we get formula (4):

[0128] (a0-a2)(a1-a3)(a0+a2-(a1+a3))≠0 (4)

[0129] The coefficient relationships in the determinant coefficient matrix are obtained as follows: a0≠a2, a1≠a3, a0+a2≠a1+a3. Therefore, the correspondence between the original data blocks and the coefficient relationships can be constructed for the four failed data blocks.

[0130] (2) The four failed data blocks consist of three original data blocks and one check block.

[0131] 1. If the failed check block is a cross-group check block and the three failed original data blocks are in the same group as the failed cross-group data block, then two equations and three unknowns can be obtained. However, the two equations cannot solve for the three unknowns, so theoretically it is not reconstructable.

[0132] 2. If the failed check block is a cross-group check block and the three failed original data blocks are not in the same group as the failed cross-group data block, assuming the failed data blocks are d0, d1, d2, P2, then the determinant coefficient matrix (5) can be expressed as:

[0133]

[0134] If we want to theoretically reconstruct the system under the failure modes of 4 data blocks, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix must not be 0, which can be expressed as formula (6):

[0135]

[0136] Simplifying, we get formula (7):

[0137] a0a1(a2-a0)≠0 (7)

[0138] The coefficient relationships in the determinant coefficient matrix are obtained as follows: a0≠0, a1≠0, a2-a0≠0. Therefore, the correspondence between the four failed data blocks (three original data blocks and one check block) and the coefficient relationships can be constructed.

[0139] 3. If the failed check block is a global check block, assuming the failed data blocks are d0, d1, d2, Q0, then the determinant coefficient matrix (8) can be represented as:

[0140]

[0141] If we want to theoretically reconstruct the system under the failure modes of 4 data blocks, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix must not be 0, which can be expressed as formula (9):

[0142]

[0143] Simplifying, we get formula (10):

[0144] (a2-a0)≠0, a0≠a2 (10)

[0145] The coefficient relationship in the determinant coefficient matrix is ​​obtained as follows: (a2-a0)≠0, a0≠a2. Therefore, the correspondence between the four failed data blocks (three original data blocks and one check block) and the coefficient relationship can be constructed.

[0146] (3) The four failed data blocks consist of two original data blocks and two check blocks.

[0147] 1. If the failed check block is a cross-group check block and a global check block, and the two failed original data blocks and one of the check blocks are in the same group, assuming the failed data blocks are d0, d2, P2, Q1, then if we want to theoretically reconstruct the data blocks under the four-data-block failure mode, the column vector where the data blocks are located needs to have a solution, that is, the determinant of the determinant coefficient matrix is ​​not 0, which can be expressed as formula (11):

[0148]

[0149] Theoretically, the failed data block is not reconstructable.

[0150] 2. If the failed check block consists of one cross-group check block and one global check block, and the two failed original data blocks come from two cross-group check blocks respectively, assuming the failed data blocks are d0, d1, P2, Q1, then if we want to theoretically reconstruct the data blocks under the four-data-block failure mode, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix must not be 0, so -a0≠0, i.e., a0≠0. Therefore, we can construct the correspondence between the four failed data blocks as two original data blocks and two check blocks and the coefficient relationship.

[0151] 3. If the failed check blocks are two cross-group check blocks, assuming the failed data blocks are d0, d1, P2, and P3, then if we want to theoretically reconstruct the data blocks under the four-block failure mode, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix must not be 0, which can be expressed as formula (12):

[0152] a0a1(a1-a0)≠0 (12)

[0153] The coefficient relationship in the determinant coefficient matrix is ​​obtained: a0≠a1≠0. Therefore, the correspondence between the four failed data blocks, two original data blocks, and two check blocks and the coefficient relationship can be constructed.

[0154] 4. If the failed check blocks are two global check blocks, and the two failed original data blocks are in the same group as one of the unfailed cross-group check blocks, assuming the failed data blocks are d0, d2, Q0, Q1, then if we want to theoretically reconstruct the data blocks under the four-block failure mode, the column vector containing the data blocks needs to have a solution, that is, the determinant of the determinant coefficient matrix is ​​not 0, which can be expressed as formula (13):

[0155]

[0156] Therefore, in theory, invalid data blocks cannot be reconstructed.

[0157] 5. If the failed check blocks are two global check blocks, and the two failed check blocks come from two cross-group check blocks respectively, assuming the failed data blocks are d0, d1, Q0, Q1, the final determinant is 1, so data recovery is possible in this case. And this is independent of the chosen coefficients.

[0158] (4) The four failed data blocks consist of one original data block and three check blocks.

[0159] 1. If the three failed check blocks include two cross-group check blocks and one global check block, and assuming the failed data blocks are d0, P2, P3, Q0, we can find that a0≠0. Therefore, we can construct a correspondence between the four failed data blocks, one original data block, and three check blocks and the coefficient relationship.

[0160] 2. If the three failed check blocks include two global check blocks and one cross-group check block, and one failed original data block comes from the failed cross-group check block. Assuming the failed data blocks are d0, P2, Q0, Q1, and their determinant is always equal to 0, theoretically, reconstruction is not possible.

[0161] 3. If the three failed check blocks include two global check blocks and one cross-group check block, and one failed original data block is not in the failed cross-group check block. Assuming the failed data blocks are d0, P3, Q0, Q1, then the determinant is always equal to -1, theoretically allowing for reconstruction, and this is independent of the chosen coefficients.

[0162] (5) The four failed data blocks are four check blocks.

[0163] All check blocks can be obtained by re-encoding the original data blocks, regardless of the selected coefficients.

[0164] Therefore, based on the above method, a correspondence between the distribution samples of failed data blocks and the coefficient relationships can be constructed to obtain a recovery status indicator table. The above example is only one scheme when there are 6 original data blocks and 2 cross-group check blocks. The calculation method for other failure schemes is the same as the above scheme. When calculating the 4-fault tolerance, other combinations of failed check blocks can also be selected, and the final result data can be calculated by combining all check blocks.

[0165] In one possible implementation, the two global check blocks include a first global check block and a second global check block. In the matrix of row and column coefficients, the first global check block corresponds to the coefficients in the nth row of the determinant coefficient matrix, and the second global check block corresponds to the coefficients in the (n+1)th row of the determinant coefficient matrix, wherein the determinant coefficient matrix includes a total of n+1 rows of coefficients; and each original data block corresponds to one coefficient in one row of coefficients.

[0166] If there are 4 failed data blocks, and these failed data blocks are sequentially designated as the first failed data block, the second failed data block, the third failed data block, and the fourth failed data block, and in the determinant coefficient matrix, the first failed data block corresponds to the first coefficient, the second failed data block corresponds to the second coefficient, the third failed data block corresponds to the third coefficient, and the fourth failed data block corresponds to the fourth coefficient, then:

[0167] If all the failed data blocks are original data blocks, then the corresponding coefficient relationships are as follows: the first coefficient is not equal to the third coefficient, the second coefficient is not equal to the fourth coefficient, and the sum of the first coefficient and the third coefficient is not equal to the sum of the second coefficient and the fourth coefficient.

[0168] If a failed data block includes one cross-group check block and three original data blocks, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed.

[0169] If the failed data block includes one cross-group check block and three original data blocks, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: the first coefficient is not equal to zero, the second coefficient is not equal to zero, and the difference between the third coefficient and the first coefficient is not zero.

[0170] If the failed data block includes one global check block and three original data blocks, the corresponding coefficient relationship is: the difference between the third coefficient and the first coefficient is not zero, and the first coefficient is not equal to the third coefficient.

[0171] If a failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as either the failed cross-group check block or the failed global check block, then the failed data block cannot be reconstructed.

[0172] If the failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as the failed cross-group check block and the failed global check block, then the corresponding coefficient relationship is: the first coefficient is not zero;

[0173] If the failed data block includes two cross-group check blocks and two original data blocks, then the corresponding coefficient relationship is: the first coefficient is not equal to the second coefficient, and the first coefficient and the second coefficient are not zero;

[0174] If a failed data block contains two global check blocks and two original data blocks, and the two failed original data blocks are in the same group as one unfailed cross-group check block, then the failed data block cannot be reconstructed.

[0175] If the failed data block includes 2 global check blocks and 2 original data blocks, and the 2 failed original data blocks are in the same group as the 2 unfailed cross-group check blocks, then the corresponding coefficient relationship is: coefficients are unconstrained.

[0176] If the failed data block includes two cross-group check blocks, one global check block, and one original data block, then the corresponding coefficient relationship is: the first coefficient is not zero;

[0177] If a failed data block contains two global check blocks, one cross-group check block, and one original data block, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed.

[0178] If the failed data block includes 2 global check blocks, 1 cross-group check block, and 1 original data block, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: coefficients are unconstrained.

[0179] For example, based on the previous description of selecting four failed data blocks when there are 6 original data blocks and 2 cross-group check blocks, the distribution of failed data blocks and the corresponding recovery indication information are shown in Table 1:

[0180] Table 1

[0181]

[0182]

[0183]

[0184] Other schemes are analyzed similarly to this method. By combining all check blocks, the final distribution of failed data blocks and the corresponding recovery indication information can be obtained. Therefore, when all coefficient matrices are satisfied, the MLRC erasure coding data processing method provided in this application can better adapt to the case of 4-tolerance errors.

[0185] In one possible implementation, after calculation and analysis, a tuple (k, l, r, m) can be obtained, where k represents k original data blocks, l represents l global check blocks, r represents r sequential group check blocks, and m represents m cross-group check blocks. To compare different cross-group check blocks, cases with 10 and 12 original data blocks can be selected respectively. When there are 10 original data blocks, there can be 2 or 5 cross-group check blocks. When there are 12 original data blocks, there can be 2, 3, 4, or 6 cross-group check blocks. The storage overhead and fault tolerance overhead for each case are calculated and analyzed using the same method as described above, as shown in Table 2.

[0186] Table 2

[0187]

[0188]

[0189] Therefore, for the same original data block, the more cross-group check blocks there are, and the more cross-group data block groups the original data block is divided into, the higher the fault tolerance rate and the highest fault tolerance level.

[0190] In one possible implementation, the minimum number of data blocks required to reconstruct a single failed data block from multiple original data blocks is the number of data blocks included in a single cross-group data block group. As shown in Table 2, the minimum number of data blocks required to reconstruct a single failed data block is the number of data blocks included in a single cross-group data block group, i.e., the number of data blocks included in a cross-group check block.

[0191] In one possible implementation, the maximum number of failed data blocks allowed across multiple original data blocks is equal to the total number of check blocks. As shown in Table 2, the highest fault tolerance element represents the maximum number of failed data blocks allowed across multiple original data blocks, which is the same as the corresponding total number of check blocks in Table 2.

[0192] In one possible implementation, the data processing method based on MLRC erasure coding provided in this application can also be compared longitudinally with other data processing methods based on erasure coding, still using (10,2,2,2) for comparison, as shown in Table 3:

[0193] Table 3

[0194] Encoding scheme Storage overhead Tolerance 4 error rate Highest fault tolerance Minimal Reconstruction of a Single Block RS(10,4) 14 100％ 4 10 LRC(10,2,2) 14 86％ 4 5 ESRC(10,2,2) 14 93％ 4 6 SHEC(10,5,4) 14 83％ 5 4 MLRC(10,2,2,2) 16 100％ 6 5

[0195] As can be seen from Table 3, although the data processing method based on MLRC erasure coding provided in this application has increased storage overhead, it only shows a significant improvement in fault tolerance rate, maximum fault tolerance, and minimum single-block reconstruction compared to other data processing methods based on erasure coding.

[0196] Based on the foregoing description, this application discloses a data processing method based on erasure coding. This method involves acquiring multiple raw data blocks; dividing these raw data blocks into two sequential groups and multiple cross-group data block groups; ensuring that the sequence numbers of the raw data blocks in the sequential groups are consecutive, while each cross-group data block group contains the same number of raw data blocks with non-consecutive sequence numbers; and generating multiple check blocks for the raw data blocks, including two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. By dividing the raw data blocks into multiple cross-group data block groups, the number of cross-group data block groups and the number of data blocks included in each cross-group data block group can be flexibly adjusted. Data block sharing is also possible among the multiple cross-group data block groups. During data block recovery, data recovery can be performed within a local scope, minimizing the need for global data recovery, effectively reducing reconstruction costs, network bandwidth, and computation time. This better ensures the reliability and integrity of data during cloud computing storage. Simultaneously, it offers higher fault tolerance under the same conditions, minimizing disk space usage.

[0197] like Figure 7 As shown, based on the same inventive concept as the erasure coding-based data processing method described above, this application also provides an erasure coding-based data processing apparatus, including: an acquisition module 701, a grouping module 702, and a check block determination module 703, wherein:

[0198] Acquisition module 701 is used to acquire multiple raw data blocks;

[0199] Grouping module 702 is used to divide the plurality of original data blocks into two sequential groups and a plurality of cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive.

[0200] The verification block determination module 703 is used to generate multiple verification blocks for the multiple original data blocks. The multiple verification blocks include two global verification blocks, sequential group verification blocks corresponding to each sequential group, and cross-group verification blocks corresponding to each cross-group data block group.

[0201] In one possible implementation, the grouping module 702 is further configured to select one of the cross-group data block grouping strategies and divide the multiple original data blocks into multiple cross-group data block groups if the multiple original data blocks correspond to multiple cross-group data block grouping strategies.

[0202] In one possible implementation, the device further includes:

[0203] The recovery determination module is used to determine whether a failed data block is recoverable based on a pre-built recovery status indication table. The recovery status indication table is used to record the distribution of failed data blocks and the corresponding recovery indication information. If a failed data block is recoverable, the recovery indication information is a recovery condition. If a failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable.

[0204] The reconstruction module is used to reconstruct the failed data block using multiple check blocks of the original data block if the failed data block is recoverable.

[0205] In one possible implementation, the apparatus for constructing the recovery status indication table further includes:

[0206] The sample construction module is used to construct samples of failed data blocks;

[0207] The matrix building module is used to build a determinant coefficient matrix for reconstructing the distribution sample of the failed data blocks based on the unfailed data blocks for each type of failed data block distribution sample.

[0208] The calculation module is used to solve the coefficient relationships in the determinant coefficient matrix under the premise that the determinant of the determinant coefficient matrix is ​​not zero.

[0209] The correspondence construction module is used to construct the correspondence between the distribution samples of the failed data blocks and the coefficient relationships.

[0210] The erasure coding-based data processing apparatus provided in this application adopts the same inventive concept as the erasure coding-based data processing method described above, and can achieve the same beneficial effects, so it will not be described again here.

[0211] Based on the same inventive concept as the erasure coding-based data processing method described above, this application also provides an electronic device. Referring below... Figure 8To describe an electronic device 800 according to this embodiment of the present application. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0212] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose electronic device. The components of the electronic device 800 may include, but are not limited to: at least one processor 801, at least one memory 802, and a bus 803 connecting different system components (including memory 802 and processor 801).

[0213] Bus 803 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the various bus structures.

[0214] The memory 802 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include read-only memory (ROM) 8023.

[0215] The memory 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0216] Electronic device 800 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. As shown, network adapter 806 communicates with other modules used in electronic device 800 via bus 803. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0217] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 802 including instructions, which can be executed by a processor 801 to complete the aforementioned attacker threat assessment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0218] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor 801, implements any of the erasure coding-based data processing methods provided in this application.

[0219] In an exemplary embodiment, various aspects of the erasure coding-based data processing method provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the erasure coding-based data processing method according to the various exemplary embodiments of this application described above.

[0220] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0221] The program product for the erasure coding-based data processing method according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0222] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0223] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0224] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0225] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0226] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0227] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0228] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0229] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0230] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0231] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0232] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method based on erasure coding, characterized in that, The method includes: Obtain multiple raw data blocks; The multiple original data blocks are divided into two sequential groups and multiple cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive. Multiple check blocks are generated from the multiple original data blocks, including two global check blocks, sequential group check blocks corresponding to each sequential group, and cross-group check blocks corresponding to each cross-group data block group. The method further includes: Based on a pre-built recovery status indication table, it is determined whether the failed data block is recoverable. The recovery status indication table is used to record the distribution of failed data blocks and the corresponding recovery indication information. If the failed data block is recoverable, the recovery indication information is a recovery condition. If the failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable. If the failed data block is recoverable, the failed data block is reconstructed using multiple check blocks of the original data block. The construction of the recovery status indication table specifically includes: constructing a sample of failed data blocks; for each type of failed data block distribution sample, establishing a determinant coefficient matrix based on unfailed data blocks to reconstruct the sample of failed data block distribution; solving the coefficient relationship in the determinant coefficient matrix under the premise that the determinant of the determinant coefficient matrix is ​​not zero; and constructing the correspondence between the sample of failed data block distribution and the coefficient relationship. The two global verification blocks include a first global verification block and a second global verification block. In the matrix of row and column coefficients, the first global verification block corresponds to the coefficients in the nth row of the determinant coefficient matrix, and the second global verification block corresponds to the coefficients in the (n+1)th row of the determinant coefficient matrix. The determinant coefficient matrix includes a total of n+1 rows of coefficients. Each original data block corresponds to one coefficient in one row of coefficients. If there are 4 failed data blocks, and these failed data blocks are sequentially designated as the first failed data block, the second failed data block, the third failed data block, and the fourth failed data block, and in the determinant coefficient matrix, the first failed data block corresponds to the first coefficient, the second failed data block corresponds to the second coefficient, the third failed data block corresponds to the third coefficient, and the fourth failed data block corresponds to the fourth coefficient, then: If all the failed data blocks are original data blocks, then the corresponding coefficient relationships are as follows: the first coefficient is not equal to the third coefficient, the second coefficient is not equal to the fourth coefficient, and the sum of the first coefficient and the third coefficient is not equal to the sum of the second coefficient and the fourth coefficient. If a failed data block includes one cross-group check block and three original data blocks, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes one cross-group check block and three original data blocks, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: the first coefficient is not equal to zero, the second coefficient is not equal to zero, and the difference between the third coefficient and the first coefficient is not zero. If the failed data block includes one global check block and three original data blocks, the corresponding coefficient relationship is: the difference between the third coefficient and the first coefficient is not zero, and the first coefficient is not equal to the third coefficient. If a failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as either the failed cross-group check block or the failed global check block, then the failed data block cannot be reconstructed. If the failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as the failed cross-group check block and the failed global check block, then the corresponding coefficient relationship is: the first coefficient is not zero; If the failed data block includes two cross-group check blocks and two original data blocks, then the corresponding coefficient relationship is: the first coefficient is not equal to the second coefficient, and the first coefficient and the second coefficient are not zero; If a failed data block includes two global check blocks and two original data blocks, and the two failed original data blocks are in the same group as one non-failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes 2 global check blocks and 2 original data blocks, and the 2 failed original data blocks are in the same group as the 2 unfailed cross-group check blocks, then the corresponding coefficient relationship is: coefficients are unconstrained. If the failed data block includes two cross-group check blocks, one global check block, and one original data block, then the corresponding coefficient relationship is: the first coefficient is not zero; If a failed data block includes two global check blocks, one cross-group check block, and one original data block, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes 2 global check blocks, 1 cross-group check block, and 1 original data block, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: coefficients are unconstrained.

2. The method according to claim 1, characterized in that, If the multiple original data blocks correspond to multiple cross-group data block grouping strategies, then one of the cross-group data block grouping strategies is selected to divide the multiple original data blocks into multiple cross-group data block groups.

3. The method according to any one of claims 1-2, characterized in that, The minimum number of data blocks required to reconstruct a single failed data block from the plurality of original data blocks is the number of data blocks included in a single cross-group data block group.

4. The method according to any one of claims 1-2, characterized in that, The maximum number of invalid data blocks allowed among the multiple original data blocks is the total number of check blocks.

5. A data processing device based on erasure coding, characterized in that, The device includes: The acquisition module is used to acquire multiple raw data blocks; The grouping module is used to divide the multiple raw data blocks into two sequential groups and multiple cross-group data block groups; the original data block numbers in the sequential groups are consecutive, and each cross-group data block group contains the same number of original data blocks, and the data block numbers in the cross-group data block groups are not consecutive. The verification block determination module is used to generate multiple verification blocks for the multiple original data blocks. The multiple verification blocks include two global verification blocks, sequential group verification blocks corresponding to each sequential group, and cross-group verification blocks corresponding to each cross-group data block group. The device further includes: A recovery determination module is used to determine whether a failed data block is recoverable based on a pre-built recovery status indication table. The recovery status indication table records the distribution of failed data blocks and the corresponding recovery indication information. If a failed data block is recoverable, the recovery indication information is a recovery condition; if the failed data block is not recoverable, the recovery indication information indicates that it is not reconstructable. A reconstruction module is used to reconstruct the failed data block using multiple check blocks of the original data block if the failed data block is recoverable. The device for constructing the recovery status indication table further includes: a sample construction module for constructing failed data block samples; a matrix establishment module for constructing a determinant coefficient matrix for reconstructing the failed data block distribution samples based on unfailed data blocks for each type of failed data block distribution sample; a calculation module for solving the coefficient relationships in the determinant coefficient matrix under the premise that the determinant of the determinant coefficient matrix is ​​not zero; and a correspondence construction module for constructing the correspondence between the failed data block distribution samples and the coefficient relationships. The two global verification blocks include a first global verification block and a second global verification block. In the matrix of row and column coefficients, the first global verification block corresponds to the coefficients in the nth row of the determinant coefficient matrix, and the second global verification block corresponds to the coefficients in the (n+1)th row of the determinant coefficient matrix. The determinant coefficient matrix includes a total of n+1 rows of coefficients. Each original data block corresponds to one coefficient in one row of coefficients. If there are 4 failed data blocks, and these failed data blocks are sequentially designated as the first failed data block, the second failed data block, the third failed data block, and the fourth failed data block, and in the determinant coefficient matrix, the first failed data block corresponds to the first coefficient, the second failed data block corresponds to the second coefficient, the third failed data block corresponds to the third coefficient, and the fourth failed data block corresponds to the fourth coefficient, then: If all the failed data blocks are original data blocks, then the corresponding coefficient relationships are as follows: the first coefficient is not equal to the third coefficient, the second coefficient is not equal to the fourth coefficient, and the sum of the first coefficient and the third coefficient is not equal to the sum of the second coefficient and the fourth coefficient. If a failed data block includes one cross-group check block and three original data blocks, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes one cross-group check block and three original data blocks, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: the first coefficient is not equal to zero, the second coefficient is not equal to zero, and the difference between the third coefficient and the first coefficient is not zero. If the failed data block includes one global check block and three original data blocks, the corresponding coefficient relationship is: the difference between the third coefficient and the first coefficient is not zero, and the first coefficient is not equal to the third coefficient. If a failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as either the failed cross-group check block or the failed global check block, then the failed data block cannot be reconstructed. If the failed data block includes one cross-group check block, one global check block, and two original data blocks, and the two failed original data blocks are in the same group as the failed cross-group check block and the failed global check block, then the corresponding coefficient relationship is: the first coefficient is not zero; If the failed data block includes two cross-group check blocks and two original data blocks, then the corresponding coefficient relationship is: the first coefficient is not equal to the second coefficient, and the first coefficient and the second coefficient are not zero; If a failed data block includes two global check blocks and two original data blocks, and the two failed original data blocks are in the same group as one non-failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes 2 global check blocks and 2 original data blocks, and the 2 failed original data blocks are in the same group as the 2 unfailed cross-group check blocks, then the corresponding coefficient relationship is: coefficients are unconstrained. If the failed data block includes two cross-group check blocks, one global check block, and one original data block, then the corresponding coefficient relationship is: the first coefficient is not zero; If a failed data block includes two global check blocks, one cross-group check block, and one original data block, and the failed original data block is in the same group as the failed cross-group check block, then the failed data block cannot be reconstructed. If the failed data block includes 2 global check blocks, 1 cross-group check block, and 1 original data block, and the failed original data block is not in the same group as the failed cross-group check block, then the corresponding coefficient relationship is: coefficients are unconstrained.

6. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the steps of the erasure coding-based data processing method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the steps of the erasure coding-based data processing method as described in any one of claims 1-4.

Images