Data block coding method, data block recovery method, data block coding device, data block recovery device and electronic equipment
The data block coding method enhances LRC by dividing blocks into groups with cross-blocks and constructing Cauchy matrices to generate parity blocks, enabling efficient and flexible recovery across multiple groups, reducing recovery latency.
Patent Information
- Application Number
- CN202410056900.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-15
- Publication Date
- 2025-07-15
AI Technical Summary
In the existing Local Repair Code (LRC) scheme, the lost data blocks can only be partially restored through one group, resulting in the problem of excessive recovery delay.
Group the original data blocks, generate cross blocks, and construct global check blocks and local check blocks based on the Cauchy matrix, providing multiple local recovery modes, allowing cross blocks to recover in multiple packets.
Through multiple local recovery modes, recovery delay is reduced and data recovery flexibility and efficiency are improved.
Smart Images
Figure CN120320780A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data storage, and more particularly, to a data block encoding method, a data block recovery method, an apparatus, and an electronic device. Background Art
[0002] Erasure Code is an encoding fault tolerance technology. Its basic principle is to fragment the stored data, and generate k + m pieces of data from k pieces of original data through a certain parity calculation method, and the original data can be restored through any k pieces of the k + m pieces of data. In this way, even if some data is lost, the system can still restore the original data. Due to the characteristics of Erasure Code that it occupies less overall storage space of the cluster while ensuring reliability, it is widely used in almost all distributed storage systems. However, compared with the traditional triple replication, Erasure Code has the problem of read amplification during data reconstruction.
[0003] Local Repair Code (LRC) is a local parity erasure code method. Its core idea is: divide the data blocks into multiple groups, and divide the parity blocks into global parity blocks and local parity blocks. When the number of lost data blocks does not exceed the number of local parity blocks within the group, data recovery can be performed through other readable data blocks within the group where the data block is located, which can reduce the problem of read amplification.
[0004] Currently, LRC has been increasingly widely used in various storage systems, but the problem existing in various LRC schemes is that the lost data blocks can only be locally recovered through one group, and the local recovery mode is single, which is likely to cause too long recovery delay. Summary of the Invention
[0005] The present application provides a data block encoding method, a data block recovery method, an apparatus, and an electronic device to implement multiple local recovery modes and avoid too long recovery delay.
[0006] In a first aspect, the present application provides a data block encoding method, including:
[0007] Dividing a plurality of original data blocks to obtain a plurality of groups, where at least one cross block is included in the plurality of groups, and the cross block is the same original data block in the plurality of groups;
[0008] Constructing a Cauchy matrix based on the number of the original data blocks and the number of the groups;
[0009] Generate global check blocks for each group based on the Cauchy matrix and the multiple original data blocks, generate local check blocks for each group based on the Cauchy matrix, the multiple original data blocks, a preset cross-block coefficient, and the cross blocks, and add the global check blocks and local check blocks of each group to each group, where the sum of the global check blocks of the multiple groups constitutes a new global check block.
[0010] In one implementation, the constructing the Cauchy matrix based on the number of the original data blocks and the number of the groups includes:
[0011] Construct the Cauchy matrix with the number of groups plus 1 as the number of rows and the number of original data blocks as the number of columns.
[0012] In one implementation, the generating global check blocks for each group based on the Cauchy matrix and the multiple original data blocks includes:
[0013] For the i-th group, multiply the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks to obtain the global check block of the i-th group, where i ranges from 1 to the number of groups.
[0014] In one implementation, the generating local check blocks for each group based on the Cauchy matrix, the multiple original data blocks, a preset cross-block coefficient, and the cross blocks includes:
[0015] Sum the data of each row of the Cauchy matrix by column to obtain the Cauchy matrix sum value corresponding to each column, and the Cauchy matrix sum value corresponding to each column corresponds to each original data block;
[0016] For the i-th group, determine the local check block of the i-th group in the following manner, where i ranges from 1 to the number of groups:
[0017] Multiply the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks, add the sum of the Cauchy matrix sum values corresponding to the i-th group multiplied by the sum of the target original data blocks corresponding to the i-th group, and then add the sum of the cross blocks multiplied by the i-th cross-block coefficient to obtain the local check block of the i-th group;
[0018] Where the target original data blocks corresponding to the i-th group are part or all of the original data blocks included in the i-th group, the target original data blocks corresponding to the multiple groups include all the original data blocks and the target original data blocks corresponding to each group are not repeated; the Cauchy matrix sum values corresponding to the i-th group refer to the Cauchy matrix sum values corresponding to the target original data blocks corresponding to the i-th group; the sum of the cross-block coefficients is zero.
[0019] Second aspect, the present application provides a method for recovering data blocks, including:
[0020] Determine the lost target data block; wherein, the target data block is a data block in a group obtained by encoding a plurality of original data blocks using the method described in the first aspect;
[0021] If the target data block is any cross block, use the data blocks in any group to which the cross block belongs to recover the target data block.
[0022] In one implementation, it further includes:
[0023] When using the data blocks in any group to which the cross block belongs to recover the target data block, if the recovery is not completed within a preset time, re-use the data blocks in other groups to which the cross block belongs to recover the target data block.
[0024] In one implementation, it further includes:
[0025] If the target data block is any data block other than the cross block, recover the target data block based on the data blocks in the group to which the target data block belongs.
[0026] In one implementation, it further includes:
[0027] If the target data block is any n data blocks, where n ranges from 2 to the number of groups plus 1, recover the target data block based on the data blocks in each group.
[0028] Third aspect, the present application provides a data block encoding device, including:
[0029] A grouping module for dividing a plurality of original data blocks into a plurality of groups, where at least one cross block is included in the plurality of groups, and the cross block is the same original data block in the plurality of groups;
[0030] A construction module for constructing a Cauchy matrix based on the number of original data blocks and the number of groups;
[0031] An encoding module for generating global check blocks for each group based on the Cauchy matrix and the plurality of original data blocks, generating local check blocks for each group based on the Cauchy matrix, the plurality of original data blocks, a preset cross block coefficient, and the cross block, and adding the global check blocks and local check blocks of each group to each group, wherein the sum of the global check blocks of the plurality of groups constitutes a new global check block.
[0032] In one implementation, the construction module is used for:
[0033] Construct the Cauchy matrix with the number of the groups plus one as the number of rows and the number of the original data blocks as the number of columns.
[0034] In one implementation, the encoding module is configured to:
[0035] For the i-th group, multiply the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks to obtain the global check block of the i-th group, where i ranges from 1 to the number of the groups.
[0036] In one implementation, the encoding module is configured to:
[0037] Sum the data of each row of the Cauchy matrix column by column to obtain the Cauchy matrix sum value corresponding to each column, and the Cauchy matrix sum value corresponding to each column corresponds to each original data block;
[0038] For the i-th group, determine the local check block of the i-th group in the following manner, where i ranges from 1 to the number of the groups:
[0039] Multiply the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks, add the sum of the Cauchy matrix sum values corresponding to the i-th group multiplied by the sum of the target original data blocks corresponding to the i-th group, and then add the sum of the i-th cross block coefficient multiplied by the sum of each cross block to obtain the local check block of the i-th group;
[0040] Wherein, the target original data block corresponding to the i-th group is part or all of the original data blocks included in the i-th group, the target original data blocks corresponding to the multiple groups include all the original data blocks and the target original data blocks corresponding to each group do not repeat; the Cauchy matrix sum values corresponding to the i-th group refer to the Cauchy matrix sum values corresponding to the target original data blocks corresponding to the i-th group; the sum of each cross block coefficient is zero.
[0041] In a fourth aspect, the present application provides a data block recovery device, including:
[0042] A determination module, configured to determine a lost target data block; wherein, the target data block is a data block in a group obtained by encoding multiple original data blocks by using the method described in the first aspect;
[0043] A recovery module, configured to, if the target data block is any cross block, recover the target data block by using the data blocks in any group to which the cross block belongs.
[0044] In one implementation, the recovery module is configured to:
[0045] When recovering the target data block using the data blocks in any group to which the cross block belongs, if the recovery is not completed within the preset time, the data blocks in other groups to which the cross block belongs are used again to recover the target data block.
[0046] In one implementation, the recovery module is configured to:
[0047] If the target data block is any data block other than the cross block, the target data block is recovered based on the data blocks in the group to which the target data block belongs.
[0048] In one implementation, the recovery module is configured to:
[0049] If the target data block is any n data blocks, where n ranges from 2 to the number of groups plus 1, the target data block is recovered based on the data blocks in each group.
[0050] In a fifth aspect, the present application provides an electronic device, including: a memory and a processor;
[0051] The memory is used to store a computer program;
[0052] The processor is used to execute the computer program stored in the memory, and when the computer program runs, the processor executes the method described in the first aspect or the second aspect.
[0053] In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the processor executes the method described in the first aspect or the second aspect.
[0054] In a seventh aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.
[0055] In the data block encoding method, data block recovery method, device, and electronic device provided by the present application, the original data blocks are grouped, and some of the original data blocks are divided into cross blocks, that is, the same data blocks in each group. Then, a Cauchy matrix is constructed based on the number of groups and the number of original data blocks, and further local check blocks and global check blocks of each group are generated. In the case of loss of cross blocks, recovery can be performed using the data in any group, providing more flexible options for data recovery and avoiding problems such as high recovery latency caused by a single local recovery mode. Description of the Drawings
[0056] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0057] Figure 1 It is a schematic diagram of a data block in an LRC mode;
[0058] Figure 2 It is a schematic diagram of a data block in another LRC mode;
[0059] Figure 3 It is a schematic diagram of a data block in an LRC mode provided by an embodiment of the present application;
[0060] Figure 4 It is a schematic flowchart of a data block encoding method provided by an embodiment of the present application;
[0061] Figure 5 It is Example 1 of a data block grouping provided by an embodiment of the present application;
[0062] Figure 6 It is Example 2 of a data block grouping provided by an embodiment of the present application;
[0063] Figure 7 It is a schematic flowchart of a data block recovery method provided by an embodiment of the present application;
[0064] Figure 8 It is a schematic structural diagram of a data block encoding device provided by an embodiment of the present application;
[0065] Figure 9 It is a schematic structural diagram of a data block recovery device provided by an embodiment of the present application;
[0066] Figure 10 It is a schematic block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners
[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application.
[0068] Figure 1 It is a schematic diagram of a data block in an LRC mode. AsFigure 1 As shown, each group includes multiple original data blocks. For example, Group 1 includes original data blocks D 1,1 to D 1,r , and Group g includes original data blocks D g,1 to D g,r . Each group has a local check block. For example, the local check block L1 in Group 1 and the local check block L g in Group g. The global check blocks are stored separately, such as global check blocks G1 to G g .
[0069] Figure 2 is a schematic diagram of data blocks in another LRC mode. As Figure 2 shown, each group includes multiple original data blocks. For example, Group 1 includes original data blocks D 1,1 to D 1,r , and Group g includes original data blocks D g,1 to D g,r . Each group has a local check block and a global check block. For example, the local check block L1 and the global check block G1 in Group 1, and the local check block L g and the global check block G g in Group g.
[0070] Regardless of which of the above LRC modes, each data block has only one local recovery mode. For example Figure 1 or Figure 2 if D 1,1 is lost, only other data blocks in Group 1 can be used to perform local recovery, and the local recovery mode is single. The problem that may be caused by this single local recovery mode is that if there are slow disks among other data blocks in Group 1, then even if local recovery is used, the data recovery latency is relatively long.
[0071] In view of this, in the embodiments of the present application, a data block encoding method is proposed, providing a new LRC mode, dividing some original data blocks into each group, that is, each group contains the same original data blocks. For example Figure 3The three original data blocks shown at the center of the figure are included in Group 1, Group 2 to Group l. For example, the data blocks corresponding to the disks prone to failure are divided into cross blocks, that is, the same data blocks in each group, and then the corresponding local parity blocks and global parity blocks are generated. In this way, if a cross block is lost, there are g local recovery modes, where g is the number of groups, providing more flexible choices for data recovery and avoiding problems such as high recovery latency caused by a single local recovery mode.
[0072] Figure 4 It is a schematic flow chart of a data block encoding method provided by an embodiment of the present application. As Figure 3 shown, the method includes:
[0073] S401. Divide a plurality of original data blocks to obtain a plurality of groups, where at least one cross block is included in the plurality of groups, and the cross block is the same original data block in the plurality of groups.
[0074] In a distributed system, data is stored in the form of data blocks. All the data to be stored in a file to be stored or a storage request of a user will be decomposed into a plurality of original data blocks to be stored. The plurality of original data blocks obtained by decomposition can be stored on different nodes in the distributed system or on different disks of the same node device.
[0075] In the embodiment of the present application, in order to implement multiple local recovery modes, when grouping the original data blocks, some original data blocks are divided into each group so that at least one same original data block, that is, a cross block, is included in each group. The cross block can be an original data block prone to failure, that is, prone to loss.
[0076] Optionally, when grouping, the plurality of original data blocks can be grouped without repetition first. At this time, at least one original data block is included in each group, but the original data blocks in different groups do not repeat. Then, the original data blocks serving as cross blocks are determined from the original data blocks. In addition to being divided into its original group, the cross block is also divided into other groups other than its original group. For example, the number of original data blocks is 5, which are D1, D2, D3, D4, and D5 respectively. These 5 original data blocks are divided into two groups. For example, D1, D2, and D3 are divided into the first group, and D4 and D5 are divided into the second group. Then, D1 among these 5 original data blocks is determined as the cross block, and the cross block D1 is divided into the second group. In this way, two groups are finally obtained, one group includes D1, D2, and D3, and the other group includes D1, D4, and D5.
[0077] S402. Construct a Cauchy matrix based on the number of original data blocks and the number of groups.
[0078] Exemplarily, taking the number of original data blocks as k and the number of groups as r, a Cauchy matrix is constructed with the number of groups plus 1 as the number of rows and the number of original data blocks as the number of columns, that is, a (r + 1) * k Cauchy matrix is constructed.
[0079] S403. Generate global check blocks for each group based on the Cauchy matrix and multiple original data blocks, generate local check blocks for each group based on the Cauchy matrix, multiple original data blocks, a preset cross-block coefficient, and cross blocks, and add the global check blocks and local check blocks of each group to each group. Among them, the sum of the global check blocks of multiple groups constitutes a new global check block.
[0080] In the implementation of this application, corresponding global check blocks and local check blocks are generated for each group, and the global check blocks and local check blocks are added to the corresponding groups. Thus, if any database in each group is lost, it can be restored through other databases in the group. For cross blocks, since cross blocks belong to multiple groups, cross blocks can be restored through the data in any group. And the sum of the global check blocks of multiple groups in the embodiment of this application constitutes a new global check block, so it is equivalent to having r + 1 global check blocks. Therefore, data recovery can be achieved when the number of lost data blocks in all groups is less than or equal to r + 1.
[0081] The following explains the global check blocks and local check blocks.
[0082] For the i-th group, multiply the i-th row of the Cauchy matrix by the vector formed by multiple original data blocks to obtain the global check block of the i-th group, where the value of i ranges from 1 to the number of groups r. That is, the operation is performed according to the following formula:
[0083]
[0084] Among them, G i represents the global check block of the i-th group, D j represents the j-th original data block, a i,j represents the value of the i-th row and j-th column of the Cauchy matrix, and k is the number of original data blocks.
[0085] Sum the data of each row of the Cauchy matrix by column to obtain the Cauchy matrix sum value corresponding to each column, and the Cauchy matrix sum value corresponding to each column corresponds to each original data block. That is, the operation is performed according to the following formula:
[0086]
[0087] Among them, a i,jDenotes the value of the i-th row and j-th column of the Cauchy matrix, k is the number of original data blocks, r is the number of groups, and d j Denotes the sum of the Cauchy matrix corresponding to the j-th column of the Cauchy matrix, d j Corresponds to D j Correspondingly.
[0088] For the i-th group, determine the local check block of the i-th group in the following manner, where the value range of i is from 1 to the number of groups r:
[0089] Multiply the i-th row of the Cauchy matrix by the vector composed of multiple original data blocks, add the sum of the products of the sum of the Cauchy matrices corresponding to the i-th group and the corresponding target original data blocks in the i-th group, and then add the sum of the products of the i-th cross-block coefficient and each cross-block to obtain the local check block of the i-th group; among them, the target original data blocks corresponding to the i-th group are part or all of the original data blocks included in the i-th group, and the target original data blocks corresponding to multiple groups include all the original data blocks and the target original data blocks corresponding to each group are non-repetitive. For example, according to the grouping method in the foregoing embodiment, first perform non-repetitive grouping on multiple original data blocks, then determine the cross-blocks, and divide the cross-blocks into each group. In this case, the target original data blocks corresponding to the i-th group are the original data blocks in each group after non-repetitive grouping, that is, the original data blocks unique to each group before adding the cross-blocks to each group; the sum of the Cauchy matrices corresponding to each target original data block in the i-th group refers to the sum of the Cauchy matrices corresponding to each target original data block in the i-th group; the sum of each cross-block coefficient is zero. That is, the operation is performed according to the following formula:
[0090]
[0091] Among them, L i Denotes the local check block of the i-th group, D j Denotes the j-th original data block, a i,j Denotes the value of the i-th row and j-th column of the Cauchy matrix, and k is the number of original data blocks.
[0092] M i Denotes the number of target original data blocks corresponding to the i-th group, that is, the number of target original data blocks included in each group is M1, M2,..., M r ; (M1 + M2 +... + M r = k),
[0093] The number of cross-blocks is C, C ≥ 1, v i Denotes the number of the cross-block, 1 ≤ v i ≤ k. x i Denotes the preset cross-block coefficient, x iSatisfy that under the operation domain, the sum of r x i is 0, that is For example, under GF(2^8), r can take the following values: when r is even, x i is 1; when r is odd, x r = 3, x r-1 = 2, and the remaining x i is 1.
[0094] The following is illustrated with specific examples.
[0095] Taking the number of original data blocks as 5 and the number of groups as 2 as an example, as Figure 5 shown, one group includes original data blocks D1, D2, and D3, and the other group includes original data blocks D1, D4, and D5, where D1 is the cross block.
[0096] Construct a 3*5 Cauchy matrix as follows:
[0097]
[0098] Sum the rows of the Cauchy matrix by column to obtain the Cauchy matrix sum value d:
[0099]
[0100] Calculate the global check block and local check block according to the following formula:
[0101]
[0102]
[0103]
[0104] where the values of i are 1 and 2 respectively.
[0105] The LRC coding matrix corresponding to the above operation formula is as follows:
[0106]
[0107] It should be noted that in the above formula, the target original data blocks corresponding to the first group are D1, D2, and D3, and the target original data blocks corresponding to the second group are D4 and D5 are used as examples. If the target original data blocks of the two groups are different from the above examples, for example, the situation of the original data blocks unique to each group before adding the cross block is different from the above examples, the target original data blocks corresponding to the first group are D2 and D3, and the target original data blocks corresponding to the second group are D1, D4, and D5, then only the part of the Cauchy matrix sum value multiplied by the target original data blocks in the above formula needs to be adjusted accordingly.
[0108] In this embodiment, the data blocks included in the two final groups are: {D1, D2, D3, L1, G1}, {D1, D4, D5, L2, G2}; when a single data block is lost, it can be recovered through the other data blocks in the group where the lost data block is located. If D1 is lost, it can be recovered through the other data blocks in any of the two groups. Moreover, the sum of the two local parity blocks is equivalent to a global parity block, which can ensure that any three database losses can be recovered.
[0109] Taking the number of original data blocks as 9 and the number of groups as 3 as an example, as Figure 6 shown, one group includes the original data blocks D1, D2, D3, D4, and D7, another group includes the original data blocks D1, D4, D5, D6, and D7, and yet another group includes the original data blocks D1, D4, D7, D8, and D9. D1, D4, and D7 are cross blocks.
[0110] Construct a 4*9 Cauchy matrix as follows:
[0111]
[0112] Sum the rows of the Cauchy matrix by column to obtain the Cauchy matrix sum value d:
[0113] 1 ≤ j ≤ 9
[0114] Calculate the global parity block and local parity blocks according to the following formula:
[0115]
[0116]
[0117]
[0118]
[0119] The LRC encoding matrix corresponding to the above operation formula is as follows:
[0120]
[0121] It should be noted that in the above formula, the target original data blocks corresponding to the first group are D1, D2, and D3, the target original data blocks corresponding to the second group are D4, D5, and D6, and the target original data blocks corresponding to the third group are D7, D8, and D9 for illustration. If the target original data blocks of the three groups are different from the above example, for example, the situation of the original data blocks unique to each group before adding the cross blocks is different from the above example, the target original data blocks corresponding to the first group are D1, D2, and D3, the target original data blocks corresponding to the second group are D4 and D5, and the target original data blocks corresponding to the third group are D6, D7, D8, and D9, then the part of the Cauchy matrix and the value multiplied by the target original data block in the above formula can be adjusted accordingly.
[0122] In this embodiment, the data blocks included in the final three groups are: {D1, D4, D7, D2, D3, L1, G1}, {D1, D4, D7, D5, D6, L2, G2}, {D1, D4, D7, D8, D9, L3, G3}; when a single data block is lost, it can be recovered through other data blocks in the group where the lost data block is located. If any one of D1, D4, or D7 is lost, it can be recovered through other data blocks in any one of the three groups. Moreover, the sum of the three local parity blocks is equivalent to a global parity block, so that any four databases can be recovered even if they are lost.
[0123] It should be noted that the description of addition or summation involved in the embodiments of the present application can be implemented by exclusive OR operation.
[0124] Based on the above data block encoding, an embodiment of the present application also provides a data block recovery method. Figure 7 It is a flowchart of a data block recovery method provided by an embodiment of the present application. As Figure 7 shown, the method includes:
[0125] S701. Determine the lost target data block.
[0126] Among them, the target data block is a data block in the group obtained by encoding multiple original data blocks using the method of the foregoing embodiment.
[0127] S702. If the target data block is any cross block, use the data blocks in any one of the groups to which the cross block belongs to recover the target data block.
[0128] When the method of the foregoing embodiment is used to group the original data blocks and generate the global check block and the local check block, since the cross block belongs to multiple groups, in the case where the cross block is lost, any data block in a group can be used for recovery, that is, there are multiple local recovery modes, which can be flexibly selected in the application. For example, in the case where a certain group has a slow disk, other groups can be selected for data recovery.
[0129] In practical applications, a timeout can be set for data recovery. Optionally, when using the data block in any group to which the cross block belongs to recover the target data block, if the recovery is not completed within the preset time, the data block in other groups to which the cross block belongs is used again to recover the target data block. Thus, it is possible to avoid too long recovery delay and improve the recovery efficiency.
[0130] Optionally, if the target data block is any data block other than the cross block, the target data block is recovered based on the data blocks in the group to which the target data block belongs.
[0131] Optionally, if the target data block is any n data blocks, where n ranges from 2 to the number of groups plus 1, the target data block is recovered based on the data blocks in each group.
[0132] The target data block can be an original data block, a local check block, or a global check block.
[0133] It should be noted that the local check block and the global check block are obtained based on the formula in the foregoing embodiment. Therefore, in data recovery, the operations can be performed with reference to the foregoing formula.
[0134] Figure 8 It is a schematic structural diagram of a data block encoding device provided by an embodiment of the present application. As Figure 8 shown, the data block encoding device 800 includes:
[0135] A grouping module 801, configured to divide a plurality of original data blocks into a plurality of groups, where at least one cross block is included in the plurality of groups, and the cross block is the same original data block in the plurality of groups;
[0136] A construction module 802, configured to construct a Cauchy matrix based on the number of original data blocks and the number of groups;
[0137] An encoding module 803, configured to generate a global check block for each group based on the Cauchy matrix and a plurality of original data blocks, generate a local check block for each group based on the Cauchy matrix, a plurality of original data blocks, a preset cross block coefficient, and the cross block, and add the global check block and the local check block of each group to each group, where the sum of the global check blocks of the plurality of groups constitutes a new global check block.
[0138] In one implementation, the construction module 802 is configured to:
[0139] Construct a Cauchy matrix with the number of groups plus 1 as the number of rows and the number of original data blocks as the number of columns.
[0140] In one implementation, the encoding module 803 is configured to:
[0141] For the i-th group, multiply the i-th row of the Cauchy matrix by the vector formed by multiple original data blocks to obtain the global check block of the i-th group, where the value of i ranges from 1 to the number of groups.
[0142] In one implementation, the encoding module 803 is configured to:
[0143] Sum the data of each row of the Cauchy matrix by column to obtain the Cauchy matrix sum value corresponding to each column, and the Cauchy matrix sum value corresponding to each column corresponds to each original data block;
[0144] For the i-th group, determine the local check block of the i-th group in the following manner, where the value of i ranges from 1 to the number of groups:
[0145] Multiply the i-th row of the Cauchy matrix by the vector formed by multiple original data blocks, add the sum of the Cauchy matrix sum values corresponding to the i-th group multiplied by the sum of the target original data blocks corresponding to the i-th group, and then add the product of the i-th cross-block coefficient and the sum of each cross-block to obtain the local check block of the i-th group;
[0146] Wherein, the target original data blocks corresponding to the i-th group are part or all of the original data blocks included in the i-th group, the target original data blocks corresponding to multiple groups include all the original data blocks and the target original data blocks corresponding to each group do not repeat; the Cauchy matrix sum values corresponding to the i-th group refer to the Cauchy matrix sum values corresponding to the target original data blocks corresponding to the i-th group; the sum of each cross-block coefficient is zero.
[0147] The device of the embodiments of the present application can be used to execute the data block encoding method in the foregoing embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.
[0148] Figure 9 It is a schematic structural diagram of a data block recovery device provided by the embodiments of the present application. As Figure 9 shown, the data block recovery device 900 includes:
[0149] A determination module 901, configured to determine the lost target data block; wherein, the target data block is a data block in the group obtained by encoding multiple original data blocks using the method of the first aspect;
[0150] A recovery module 902, configured to, if the target data block is any cross block, recover the target data block by using a data block in any group to which the cross block belongs.
[0151] In one implementation, the recovery module 902 is configured to:
[0152] When recovering the target data block by using a data block in any group to which the cross block belongs, if the recovery is not completed within a preset time, the target data block is recovered again by using a data block in another group to which the cross block belongs.
[0153] In one implementation, the recovery module 902 is configured to:
[0154] If the target data block is any data block other than the cross block, the target data block is recovered based on the data blocks in the group to which the target data block belongs.
[0155] In one implementation, the recovery module 902 is configured to:
[0156] If the target data block is any n data blocks, where n ranges from 2 to the number of groups plus 1, the target data block is recovered based on the data blocks in each group.
[0157] The device according to the embodiments of the present application can be used to execute the data block recovery method in the foregoing embodiments, and the implementation principles and technical effects are similar, which will not be elaborated here.
[0158] Figure 10 It is a schematic block diagram of an electronic device provided by the embodiments of the present application. As Figure 10 shown, the electronic device 1000 may include at least one processor 1001, configured to implement the method for reading multi-copy data provided by the embodiments of the present application.
[0159] Optionally, the electronic device 1000 further includes at least one memory 1002, configured to store program instructions and / or data. The memory 1002 is coupled to the processor 1001. The coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, and is used for information interaction between devices, units or modules. The processor 1001 may cooperate with the memory 1002. The processor 1001 may execute the program instructions stored in the memory 1002. At least one of the at least one memories may be included in the processor.
[0160] Optionally, the electronic device 1000 further includes a communication interface 1003 for communicating with other devices via a transmission medium, so that the electronic device 1000 can communicate with other devices. The communication interface 1003 can be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of implementing the transceiver function. The processor 1001 can use the communication interface 1003 to transmit and receive data and / or information, and is used to implement the method provided in the embodiments of the present application. For specific details, refer to the detailed description in the foregoing embodiments, and details are not described herein again.
[0161] In the embodiments of the present application, the specific connection medium between the above-mentioned processor 1001, memory 1002, and communication interface 1003 is not limited. In the embodiments of the present application Figure 10 it is assumed that the processor 1001, memory 1002, and communication interface 1003 are connected via a bus 1004. The bus 1004 is Figure 10 represented by a thick line in [the figure]. The connection manners between other components are only for illustrative purposes and are not limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 10 only a thick line is used to represent it in [the figure], but it does not mean that there is only one bus or one type of bus.
[0162] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or by instructions in software form. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.
[0163] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0164] The present application also provides a computer-readable storage medium storing a computer program (which may also be referred to as code or instructions). When the computer program is run, it causes the computer to execute the method in any of the foregoing embodiments.
[0165] The present application also provides a computer program product, including a computer program that, when executed by a processor, implements the method in any of the foregoing embodiments.
[0166] The terms "unit", "module", etc. used in this specification may be used to represent computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution.
[0167] Those of ordinary skill in the art will realize that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented in either electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends upon the particular application and design constraints of the technical solution. Skilled artisans may implement the described functions in different ways for each particular application, but such implementation should not be regarded as exceeding the scope of this application. In several embodiments provided in this application, it should be understood that the disclosed apparatus, device, and method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.
[0168] The unit described as a separate component may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0169] In addition, in each embodiment of this application, the various functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.
[0170] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid-state disk (SSD)), etc.
[0171] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods according to the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.
[0172] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.
[0173] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims described above.
Claims
1. A data block encoding method, characterized in that, including: dividing multiple original data blocks to obtain multiple groups, where at least one cross block is included in the multiple groups, and the cross block is the same original data block in the multiple groups; constructing a Cauchy matrix based on the number of the original data blocks and the number of the groups; generating a global check block for each group based on the Cauchy matrix and the multiple original data blocks, generating a local check block for each group based on the Cauchy matrix, the multiple original data blocks, a preset cross block coefficient, and the cross block, and adding the global check block and the local check block of each group to each group, where the sum of the global check blocks of the multiple groups constitutes a new global check block.
2. The method according to claim 1, characterized in that, The constructing the Cauchy matrix based on the number of the original data blocks and the number of the groups includes: constructing the Cauchy matrix with the number of the groups plus 1 as the number of rows and the number of the original data blocks as the number of columns.
3. The method according to claim 2, wherein The generating the global check block for each group based on the Cauchy matrix and the multiple original data blocks includes: for the i-th group, multiplying the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks to obtain the global check block of the i-th group, where the value of i ranges from 1 to the number of the groups.
4. The method according to claim 2, wherein The generating the local check block for each group based on the Cauchy matrix, the multiple original data blocks, the preset cross block coefficient, and the cross block includes: summing the data of each row of the Cauchy matrix by column to obtain the Cauchy matrix sum value corresponding to each column, and the Cauchy matrix sum value corresponding to each column corresponds to each original data block; for the i-th group, determining the local check block of the i-th group in the following manner, where the value of i ranges from 1 to the number of the groups: multiplying the i-th row of the Cauchy matrix by the vector formed by the multiple original data blocks, adding the sum of the Cauchy matrix sum values corresponding to the i-th group multiplied by the sum of the target original data blocks corresponding to the i-th group, and adding the sum of the i-th cross block coefficient multiplied by the sum of the cross blocks to obtain the local check block of the i-th group; wherein, the target original data block corresponding to the i-th group is part or all of the original data blocks included in the i-th group, the target original data blocks corresponding to the multiple groups include all the original data blocks and the target original data blocks corresponding to each group do not repeat; the Cauchy matrix sum values corresponding to the i-th group refer to the Cauchy matrix sum values corresponding to the target original data blocks corresponding to the i-th group; the sum of the cross block coefficients is zero.
5. A data block recovery method, characterized in that including: determining a lost target data block; wherein, the target data block is a data block in the group obtained by encoding multiple original data blocks by using the method described in any one of claims 1-4; if the target data block is any cross block, using the data block in any group to which the cross block belongs to recover the target data block.
6. The method according to claim 5, characterized in that, further including: when using the data block in any group to which the cross block belongs to recover the target data block, if the recovery is not completed within a preset time, reusing the data block in other groups to which the cross block belongs to recover the target data block.
7. The method according to claim 5, wherein further including: If the target data block is any data block other than the cross block, the target data block is recovered based on the data blocks in the group to which the target data block belongs.
8. The method according to claim 5, wherein It further includes: If the target data block is any n data blocks, where n ranges from 2 to the number of groups plus 1, the target data block is recovered based on the data blocks in each group.
9. A data block encoding device, characterized in that, It includes: A grouping module, configured to divide a plurality of original data blocks to obtain a plurality of groups, where at least one cross block is included in the plurality of groups, and the cross block is the same original data block in the plurality of groups; A construction module, configured to construct a Cauchy matrix based on the number of the original data blocks and the number of the groups; An encoding module, configured to generate a global check block for each group based on the Cauchy matrix and the plurality of original data blocks, generate a local check block for each group based on the Cauchy matrix, the plurality of original data blocks, a preset cross block coefficient, and the cross block, and add the global check block and the local check block of each group to each group, where the sum of the global check blocks of the plurality of groups constitutes a new global check block.
10. A data block recovery device, characterized in that, It includes: A determination module, configured to determine a lost target data block; where the target data block is a data block in a group obtained by encoding a plurality of original data blocks by using the method according to any one of claims 1-4; A recovery module, configured to, if the target data block is any cross block, recover the target data block by using the data blocks in any group to which the cross block belongs.
11. An electronic device, characterized in that, It includes: A memory and a processor; The memory is used to store a computer program; The processor is used to execute the computer program stored in the memory, and when the computer program runs, the processor executes the method according to any one of claims 1-8.
12. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1-8.
13. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-8.