Data processing method and device, equipment and storage medium
By grouping and XORing the data blocks and parity blocks of the Hitchhiker code, the problem of high bandwidth consumption during parity block recovery is solved, achieving more efficient data recovery.
Patent Information
- Application Number
- CN202511071741.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-21
AI Technical Summary
Hitchhiker codes consume excessive bandwidth during parity block recovery, especially when the parity block fails and the entire data block needs to be read.
The data block is divided into a first sub-data block and a second sub-data block, and encoded based on multiple different encoding functions to generate first and second check blocks. Then, the first sub-data block and the check block are divided into groups and XORed to generate intermediate and third check blocks, ensuring that the intermediate data block does not depend on a specific first check block.
During data block and parity block recovery, it is not necessary to read all data blocks, which reduces bandwidth consumption and improves the stability and recovery efficiency of the storage system.
Smart Images
Figure CN120994439A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data security storage technology, and in particular to data processing methods, apparatus, devices and storage media. Background Technology
[0002] Hitchhiker codes are erasure codes (ECs) based on MDS (Maximum Distance Separable) codes, primarily used to ensure the reliability of distributed storage systems and disk arrays. Compared to traditional RS (Reed-Solomon) codes, Hitchhiker codes can significantly reduce the bandwidth consumption required to recover from the failure of a single data block. However, when a parity block fails, decoding and recovering a single failed parity block using Hitchhiker codes still requires reading the entire data block, resulting in relatively high bandwidth consumption. Summary of the Invention
[0003] This application provides a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium to at least solve the problem of high bandwidth consumption during check block recovery in related technologies.
[0004] This application provides a data processing method, including:
[0005] Obtain target data to be stored, the target data comprising k data blocks, each of the data blocks being divided into a first sub-data block and a second sub-data block;
[0006] Based on multiple different encoding functions, the first sub-data block and the second sub-data block are encoded respectively to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and the second check blocks obtained based on the same encoding function are corresponding to each other.
[0007] The first sub-data block and the first check block are divided into m-1 groups, and the elements in each group are XORed to obtain m-1 intermediate data blocks.
[0008] The m-1 intermediate data blocks are XORed with the m-1 second check blocks to obtain m-1 third check blocks. Among them, for one of the second check blocks, the intermediate data block XORed with the second check block is not based on the first check block corresponding to the second check block.
[0009] The third verification block and the second verification block without XOR operation are used as the verification block of the second sub-data block, and the first verification block is used as the verification block of the first sub-data block. In the event of data loss, data recovery is performed based on the verification blocks.
[0010] This application also provides a data processing apparatus, including:
[0011] The data acquisition module is used to acquire target data to be stored, the target data including k data blocks, each of the data blocks being divided into a first sub-data block and a second sub-data block;
[0012] The first encoding module is used to encode the first sub-data block and the second sub-data block respectively based on multiple different encoding functions to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and second check blocks obtained based on the same encoding function correspond to each other.
[0013] The second encoding module is used to divide the first sub-data block and the first check block into m-1 groups, and perform an XOR operation on the elements in each group to obtain m-1 intermediate data blocks.
[0014] The third encoding module is used to perform an XOR operation on the m-1 intermediate data blocks and the m-1 second check blocks to obtain m-1 third check blocks. Wherein, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block.
[0015] The recovery module is used to use the third verification block and the second verification block without XOR operation as the verification block of the second sub-data block, and to use the first verification block as the verification block of the first sub-data block, and to perform data recovery based on the verification blocks in the event of data loss.
[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above data processing methods.
[0017] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.
[0018] In some embodiments of this application, after obtaining the target data to be stored, each data block in the target data is divided into a first sub-data block and a second sub-data block. Based on multiple different encoding functions, the first sub-data block and the second sub-data block are encoded respectively to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first sub-data block and the first check blocks are divided into m-1 groups, and the elements in each group are XORed to obtain m-1 intermediate data blocks. The m-1 intermediate data blocks are XORed with the m-1 second check blocks to obtain m-1 third check blocks. In this case, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block. Thus, on the one hand, the encoding process considers the first check block, meaning that not only the first sub-data block is grouped, but also the first check block is grouped. On the other hand, when generating the third check block, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block. Based on these two aspects, at least part of the final third check block includes the first check block, and within the same third check block, it does not include the first and second check blocks obtained based on the same encoding function. Therefore, when recovering a single data block and check block, it is not necessary to read all the first and second sub-data blocks, thereby reducing bandwidth consumption and solving the problem of high bandwidth consumption during check block recovery in some technologies. Attached Figure Description
[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A flowchart illustrating a data processing method provided for some embodiments of this application;
[0021] Figure 2 Schematic diagrams of a data processing apparatus provided for some embodiments of this application;
[0022] Figure 3 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation
[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0024] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0025] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0026] In the Hitchhiker code-based encoding and decoding algorithm, the original data to be saved is divided into k data blocks, and each data block is logically divided into two sub-data blocks. m represents the number of redundant parity blocks added after encoding, and n represents the total number of blocks after encoding, i.e., n = k + m. k, m, and n are positive integers.
[0027] For ease of understanding, assume k = 20, m = 5, and n = 25. Table 1 shows the encoding stripe structure of Hitchhiker codes in some techniques.
[0028] Table 1. Encoding Strip Structure
[0029]
[0030] In Table 1, a i and b i This represents the first and second sub-data blocks obtained after logically partitioning the i-th data block, where i is an integer between 0 and k. Five different MDS encoding functions f1 to f5 are used to encode the first sub-data blocks a1 to a2 respectively. 20 Second sub-data block b1~b 20 After encoding, the first sub-data block a1 to a2 can be obtained. 20 The corresponding five first check blocks f1(a) to f5(a), and the resulting second sub-data blocks b1 to b 20 The corresponding five second check blocks f1(b) to f5(b).
[0031] The first sub-data block is divided into 4 groups (i.e., m-1 groups), each group containing 5 elements (i.e., m-1 groups). We can take the first sub-data blocks (a1 to a5) as the first group and perform an XOR operation on the first sub-data blocks in each group, resulting in four XOR results. For example, we can take the first sub-data blocks a1 to a5 as the first group, and the first sub-data blocks a6 to a5 as the second group. 10 As the second group, performing an XOR operation on the first sub-data blocks a1 to a5 in the first group yields the XOR result. For the first sub-data block a6~a in the second group 10 Performing an XOR operation yields the result of the XOR operation.
[0032] By performing a one-to-one XOR operation on the results of the four XOR operations and then on the four second parity blocks f2(b) to f5(b), four third parity blocks can be obtained. For example, the XOR operation results... By performing an XOR operation with the second parity block f2(b), the third parity block can be obtained. The result of the XOR operation The third check block can be obtained by performing an XOR operation with the second check block f3(b).
[0033] First sub-data block a1~a 20 Together with the first check block f1(a) to f5(a), they can form the first logical stripe. The second sub-data block b1 to b... 20 Four third check blocks and a second check block f1(b) that has not undergone XOR operation can constitute a logical second stripe.
[0034] In actual data storage, the first and second sub-data blocks of each data block are stored as a whole. For example, the first sub-data block a1 and the second sub-data block b1 are stored as a whole, and the first sub-data block a2 and the second sub-data block b2 are stored as a whole. If one of the two sub-data blocks stored as a whole is lost, the other sub-data block will also be lost. For example, if a1 is lost, b1 will also be lost.
[0035] Similarly, parity blocks encoded using the same encoding function are also saved as a whole. For example, the first parity block f1(a) and the second parity block f1(b) are saved as a whole, and the first parity block f2(a) and the third parity block f1(b) are saved as a whole. Save them as a single unit. If one of the two check blocks saved as a single unit is lost, the other check block will also be lost.
[0036] Sub-data blocks and check blocks, as a whole, can be stored on different nodes of a distributed storage system. For example, the first sub-data block a1 and the second sub-data block b1 are stored on node 1, the first sub-data block a2 and the second sub-data block b2 are stored on node 2, the first check block f1(a) and the second check block f1(b) are stored on node 3, and the first check block f2(a) and the third check block... The data is stored in node 4. This way, if data is lost in some nodes, the lost data can be recovered based on the data that was not lost in other nodes. For example, if the first sub-data block a5 and the second sub-data block b5 are lost, the first sub-data block a5 and the second sub-data block b5 can be recovered using the following steps:
[0037] 1) Based on the second sub-data blocks b1~b4 and b6~b in the second stripe 20 And the second check block f1(b), recover the second sub-data block b5;
[0038] 2) Based on the second sub-data block b1~b 20 Determine the second check block f2(b);
[0039] 3) Based on the third check block The first sub-data block a1 to a4 and the second parity block f2(b) in the first stripe can be used to recover the first sub-data block a5.
[0040] For example, if the first parity block f1(a) and the second parity block f1(b) are lost, the first parity block f1(a) and the second parity block f1(b) can be recovered through the following steps:
[0041] 1) Read the first sub-data block a1~a in the first stripe. 20 And based on the first sub-data block a1~a 20 Restore the first check block f1(a);
[0042] 2) Read the second sub-data block b1~b in the second stripe. 20 And based on the second sub-data block b1~b 20 Restore the second check block f1(b).
[0043] As can be seen from the data recovery process described above, when recovering data blocks based on the coded stripe structure shown in Table 1, it is not necessary to read all data blocks in both stripes, thus reducing bandwidth consumption. However, when recovering the parity block, it is necessary to read all data blocks in both stripes, resulting in high bandwidth consumption during parity block recovery.
[0044] In view of this, this application provides a data processing method that can solve the problem of high data consumption during check block recovery. The data processing method can be applied to electronic devices. Electronic devices may include, but are not limited to, tablet computers, laptops, desktop computers, servers, etc. (See also...) Figure 1 This is a flowchart illustrating a data processing method provided in some embodiments of this application. Figure 1 In this context, the data processing method includes the following steps:
[0045] Step S101: Obtain the target data to be stored. The target data includes k data blocks, and each data block is divided into a first sub-data block and a second sub-data block.
[0046] Step S102: Based on multiple different encoding functions, the first sub-data block and the second sub-data block are encoded respectively to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and the second check blocks obtained based on the same encoding function are corresponding to each other.
[0047] Step S103: Divide the first sub-data block and the first check block into m-1 groups, and perform an XOR operation on the elements in each group to obtain m-1 intermediate data blocks.
[0048] Step S104: Perform an XOR operation on m-1 intermediate data blocks and m-1 second check blocks to obtain m-1 third check blocks. For one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block.
[0049] Step S105: The third check block and the second check block that has not undergone XOR operation are used as the check block of the second sub-data block, and the first check block is used as the check block of the first sub-data block. In the event of data loss, data recovery is performed based on the check blocks.
[0050] For ease of understanding, assume k = 20, m = 5, and n = 25. Table 2 shows the encoding stripe structure of Hitchhiker codes provided in some embodiments of this application.
[0051] Table 2. Coded strip structure
[0052]
[0053] The data block partitioning in Table 2 is similar to that in Table 1, and will not be repeated here. The following details the main differences between the schemes shown in Table 1 and Table 2.
[0054] Table 1 shows the 20 first sub-data blocks a1 to a2. 20 Divide into 4 groups (i.e., m-1 groups), each group containing 5 elements (i.e., m-1 groups). (1 element), and then perform an XOR operation on the elements in each group to get 4 XOR operation results.
[0055] In the scheme of this application, all elements in the first strip (including the first sub-data block a1 to a) are... 20 The first check block (f1(a) to f5(a)) is divided into 4 groups (i.e., m-1 groups). Simply put, based on the scheme shown in Table 1, each group obtained by the division only contains the first sub-data block, while in the scheme of this application, at least some of the groups obtained by the division also include the first check block.
[0056] In this embodiment, dividing the first sub-data block and the first check block into m-1 groups may include:
[0057] right Round down to the nearest integer and use the resulting value as the number of the first element;
[0058] right Round up to the nearest integer and use the resulting value as the number of the second element;
[0059] The first sub-data block and the first check block are divided into m-1 groups according to at least one of the first element quantity and the second element quantity, where n is the sum of m and k.
[0060] Specifically, taking k=20, m=5, and n=25 as an example. The first element obtained by rounding down is 6. The number of elements obtained by rounding up is 7. This means that each group can contain either 6 or 7 elements. By performing an XOR operation on the elements in each group and then XORing the result (the intermediate data block) with the second check block, the data size of each third check block is more uniform. Consequently, during data recovery, the bandwidth consumption when reading data from each node is more even, thus improving the stability of the storage system.
[0061] In other embodiments, considering the different bandwidths supported by different storage nodes, the maximum bandwidth supported by each storage node can be considered when dividing the number of elements in each group. For example, assuming that the third parity block A1 needs to be stored in storage node S1 and the third parity block A2 needs to be stored in storage node S2, if the bandwidth supported by storage node S1 is greater than the bandwidth supported by storage node S2, then the group used to generate the third parity block A1 can have a relatively larger number of elements (e.g., 7), and the group used to generate the third parity block A2 can have a relatively smaller number of elements (e.g., 6). This further improves the stability of the storage system.
[0062] Of course, it is understandable that in actual grouping, if issues such as relatively uniform bandwidth consumption are not considered, it is not necessary to group according to the restrictions on the number of the first and second elements. For example, the first group can include 3 elements, and the second group can include 10 elements. This application does not impose a restriction on the number of elements in each group.
[0063] Each group can contain all first sub-data blocks, or a subset of first sub-data blocks and a subset of first checksum blocks. For example, in Table 2, the first group contains 7 first sub-data blocks a1 to a7, and the second group contains 6 first sub-data blocks a6 to a7. 13 The third group includes five first sub-data blocks a. 14 ~a 18 The fourth group includes one first check block f5(a) and two first sub-data blocks a. 19 ~a 20 And four first parity blocks f1(a) to f4(a). After performing an XOR operation on the elements in each group, four intermediate data blocks (i.e., the XOR operation results) can be obtained. After performing a one-to-one XOR operation on the four intermediate data blocks and the four second parity blocks f2(b) to f5(b), the four third parity blocks shown in Table 2 can be obtained.
[0064]
[0065] Specifically, when performing an XOR operation between the intermediate data block and the second parity block, the following principle can be followed: for one of the second parity blocks, the intermediate data block XORed with the second parity block is not obtained based on the first parity block corresponding to the second parity block. For example, in Table 2, the intermediate data block XORed with the second parity block f4(b) is... This intermediate data block does not include the first parity block f4(a) corresponding to the second parity block f4(b). Similarly, the intermediate data block that performs an XOR operation with the second parity block f5(b) is... This intermediate data block does not include the first check block f5(a) corresponding to the second check block f5(b).
[0066] Based on the coded stripe structure shown in Table 2, when recovering a single data block and a parity block, it is not necessary to read all data blocks from both stripes. This reduces bandwidth consumption during parity block recovery. For ease of understanding, different recovery scenarios are explained below.
[0067] 1) Recover a single data block.
[0068] Specifically, a single data block comprises a set of corresponding first and second sub-data blocks. For example, first sub-data block a1 and second sub-data block b1 form one data block, and first sub-data block a2 and second sub-data block b2 form another data block.
[0069] In some embodiments, if one of the target data blocks is lost out of k data blocks, data recovery can be performed according to the following steps:
[0070] Based on the second sub-data blocks of other data blocks besides the target data block and the second parity block that has not undergone XOR operation, the second sub-data block of the target data block is recovered;
[0071] Find the first intermediate data block obtained from the first sub-data block of the target data block, and the third check block obtained from the first intermediate data block;
[0072] Based on the found third check block, the recovered second sub-data block and the unlost second sub-data block, the unlost first sub-data block and the first check block, the first sub-data block of the target data block is recovered.
[0073] For ease of understanding, the data block in Table 2 (a) is used as an example. 20 b 20 Let's take data block (a) as an example. 20 b 20 If lost, you can recover it by following these steps:
[0074] 11) Based on the second sub-data block b1~b in the second stripe 19 The second sub-data block b is obtained by solving the second parity block f1(b) that has not undergone XOR operation. 20 .
[0075] 12) Search based on a 20 The first intermediate data block obtained (i.e., including a) 20 The first intermediate data block), and the third check block obtained based on the first intermediate data block (i.e., the third check block including the first intermediate data block). Specifically, in Table 1, based on a 20 The first intermediate data block obtained is and based on the first intermediate data block The obtained third check block is
[0076] 13) Due to the second sub-data block b 20 It has been recovered, therefore, it can be based on the second sub-data block b1~b in the second stripe. 20The second check block f5(b) is calculated. Also, because the first check blocks f1(a)~f4(a) and the first sub-data block a... 19 Since nothing was lost, the first check block f1(a) to f4(a) and the first sub-data block a can be retrieved. 19 Substitute the calculated second check block f5(b) into the third check block. The first sub-data block a is calculated. 20 .
[0077] In this way, the data block (a) can be completed. 20 b 20 The recovery of the data blocks. As can be seen from steps 11) to 13), during the solution process, only a portion of the first sub-data block was obtained, and not all sub-data blocks in the two stripes were obtained. Therefore, the bandwidth consumption can be reduced during the data block recovery process.
[0078] 2) Restore the second parity block and the corresponding first parity block that have not undergone XOR operation.
[0079] In some embodiments, if the second parity block and the corresponding first parity block that have not undergone XOR operation are lost, data recovery can be performed according to the following steps:
[0080] Based on the second sub-data blocks of each data block, recover the second parity block that has not undergone XOR operation;
[0081] Locate the second intermediate data block obtained based on the lost first check block, and the third check block obtained based on the second intermediate data block;
[0082] Based on the second sub-data blocks of each data block, the found third check block, the first sub-data blocks that were not lost, and the first check block, the lost first check block is recovered.
[0083] Specifically, taking Table 2 as an example, the second check block without XOR operation and the corresponding first check block are (f1(a), f1(b)). If the check block (f1(a), f1(b)) is lost, it can be recovered using the following steps:
[0084] 21) Based on the second sub-data block b1~b in the second stripe 20 The second check block f1(b) is obtained by solving the problem.
[0085] 22) Locate the second intermediate data block (i.e., the intermediate data block including the first check block f1(a)) obtained based on the first check block f1(a) and the third check block obtained based on the second intermediate data block. Specifically, in Table 2, the second intermediate data block obtained based on the first check block f1(a) is... And the third check block obtained based on the second intermediate data block is
[0086] 23) Based on the second sub-data block b1~b in the second stripe 20 The second check block f5(b) is obtained by solving. The second check block f5(b), the first check blocks f2(a) to f4(a), and the first sub-data block a are then combined. 19 a 20 Substitute into the third check block The first check block f1(a) can then be obtained.
[0087] Thus, the check block (f1(a), f1(b)) can be obtained. According to steps 21) to 23), only a portion of the first sub-data block was obtained during the solution process, and not all sub-data blocks in the two stripes were obtained. Therefore, bandwidth consumption can be reduced during the recovery process of the check block.
[0088] 3) Restore the third check block and the first check block corresponding to the third check block.
[0089] Specifically, for one of the target third verification blocks, the target third verification block corresponds to the target first verification block, which is the first verification block corresponding to the second verification block used to generate the target third verification block. See Table 2 for an example. Third Verification Block It is generated based on the second check block f4(b), therefore the third check block The corresponding first check block is f4(a).
[0090] In some embodiments, in the event that the target third check block and the target first check block are lost, data recovery can be performed based on the following steps:
[0091] Based on the second sub-data blocks of each data block, determine the second check block used to generate the target third check block;
[0092] Based on the determined second check block, the first sub-data block that was not lost, and the first check block, recover the target third check block;
[0093] Find the third check block obtained based on the target first check block, and use the found third check block as the intermediate third check block;
[0094] Based on the intermediate third check block, restore the target first check block.
[0095] The process of restoring the target first verification block based on the intermediate third verification block can include:
[0096] Find the second check block used to generate the intermediate third check block, and use the found second check block as the intermediate second check block;
[0097] Calculate the value of the intermediate second check block based on the second sub-data block of each data block;
[0098] Based on the value of the intermediate second check block, the first sub-data block that was not lost, and the first check block, the target first check block is recovered.
[0099] Specifically, take the third verification block in Table 2 as an example. Taking the first check block f5(a) as an example. In the third check block... If the first check block f5(a) is lost, it can be recovered by following these steps:
[0100] 31) Based on the second sub-data block b1~b in the second stripe 20 Solving for the second check block f5(b) yields the solution.
[0101] 32) For the second check block f5(b) and the first sub-data block a 19 a 20 By performing an XOR operation with the first parity block f1(a) to f4(a), the third parity block can be recovered.
[0102] 33) Locate the intermediate third check block obtained based on the first check block f5(a).
[0103] 34) Locate the block used to generate the intermediate third checksum. The second check block f4(b).
[0104] 35) Based on the second sub-data block b1~b in the second stripe 20 Solving for the second check block f4(b) yields the solution.
[0105] 36) The second check block f4(b) and the first sub-data block a 14 ~a 18 Substitute into the middle third check block Then the first check block f5(a) can be obtained.
[0106] As can be seen from steps 31) to 36), during the solution process, only a portion of the first sub-data block was obtained, and not all sub-data blocks in the two stripes were obtained. Therefore, bandwidth consumption can be reduced during the recovery process of the check block.
[0107] 4) Recover multiple data blocks and multiple check blocks.
[0108] Specifically, in the event of the loss of multiple data blocks and multiple check blocks, data recovery can be performed by following these steps:
[0109] Based on the first sub-data block and the first check block that were not lost, recover the lost first sub-data block and the first check block;
[0110] Based on the unlost second check block, the unlost and recovered first sub-data block and the first check block, recover the lost second check block;
[0111] Based on the second check block that was not lost and recovered, and the second sub-data block that was not lost, the lost second sub-data block is recovered.
[0112] Specifically, based on the first sub-data block and the first check block that were not lost, multiple systems of equations are established and solved to recover the lost first sub-data block and the first check block.
[0113] Based on the unlost second check block, the unlost and recovered first sub-data block and the first check block, multiple systems of equations are established and solved to recover the lost second check block.
[0114] Based on the unlost and recovered second check block and the unlost second sub-data block, multiple systems of equations are established and solved to recover the lost second sub-data block.
[0115] In the technical solution of this application, when recovering a single data block or parity block, it is not necessary to read all sub-data blocks in two stripes, thus reducing bandwidth consumption during data recovery. Furthermore, while reducing bandwidth consumption, it also supports the recovery of multiple data blocks and parity blocks when they are lost, making it more adaptable.
[0116] In summary, in the technical solutions of some embodiments of this application, after obtaining the target data to be stored, each data block in the target data is divided into a first sub-data block and a second sub-data block. Based on multiple different encoding functions, the first sub-data block and the second sub-data block are encoded respectively to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first sub-data block and the first check blocks are divided into m-1 groups, and the elements in each group are XORed to obtain m-1 intermediate data blocks. The m-1 intermediate data blocks are XORed with the m-1 second check blocks to obtain m-1 third check blocks. In this case, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block. Thus, on the one hand, the encoding process considers the first check block, meaning that not only the first sub-data block is grouped, but also the first check block is grouped. On the other hand, when generating the third check block, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block. Based on these two aspects, at least part of the final third check block includes the first check block, and within the same third check block, it does not include the first and second check blocks obtained based on the same encoding function. Therefore, when recovering a single data block and check block, it is not necessary to read all the first and second sub-data blocks, thereby reducing bandwidth consumption and solving the problem of high bandwidth consumption during check block recovery in some technologies.
[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0118] See also Figure 2 This is a schematic diagram of a data processing apparatus provided in some embodiments of this application. Figure 2 In this context, the data processing device includes:
[0119] The data acquisition module 201 is used to acquire the target data to be stored. The target data includes k data blocks, and each data block is divided into a first sub-data block and a second sub-data block.
[0120] The first encoding module 202 is used to encode the first sub-data block and the second sub-data block respectively based on multiple different encoding functions to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and the second check blocks obtained based on the same encoding function are corresponding to each other.
[0121] The second encoding module 203 is used to divide the first sub-data block and the first check block into m-1 groups, and perform an XOR operation on the elements in each group to obtain m-1 intermediate data blocks;
[0122] The third encoding module 204 is used to perform an XOR operation on m-1 intermediate data blocks and m-1 second check blocks to obtain m-1 third check blocks. Wherein, for one of the second check blocks, the intermediate data block XORed with the second check block is not based on the first check block corresponding to the second check block.
[0123] The recovery module 205 is used to use the third check block and the second check block that has not been XORed as the check block of the second sub-data block, and to use the first check block as the check block of the first sub-data block, and to perform data recovery based on the check blocks in the event of data loss.
[0124] In some embodiments, the second encoding module 203 is specifically used for:
[0125] right Round down to the nearest integer and use the resulting value as the number of the first element;
[0126] right Round up to the nearest integer and use the resulting value as the number of the second element;
[0127] The first sub-data block and the first check block are divided into m-1 groups according to at least one of the first element quantity and the second element quantity, where n is the sum of m and k.
[0128] In some embodiments, the recovery module 205 is specifically used for:
[0129] If one of the target data blocks is lost in k data blocks, the second sub-data block of the target data block is recovered based on the second sub-data blocks of the other data blocks outside the target data block and the second check block that has not been XORed.
[0130] Find the first intermediate data block obtained from the first sub-data block of the target data block, and the third check block obtained from the first intermediate data block;
[0131] Based on the found third check block, the recovered second sub-data block and the unlost second sub-data block, the unlost first sub-data block and the first check block, the first sub-data block of the target data block is recovered.
[0132] In some embodiments, the recovery module 205 is specifically used for:
[0133] If the second parity block that has not undergone XOR operation and the corresponding first parity block are lost, the second parity block that has not undergone XOR operation is recovered based on the second sub-data blocks of each data block.
[0134] Locate the second intermediate data block obtained based on the lost first check block, and the third check block obtained based on the second intermediate data block;
[0135] Based on the second sub-data blocks of each data block, the found third check block, the first sub-data blocks that were not lost, and the first check block, the lost first check block is recovered.
[0136] In some embodiments, for one of the target third verification blocks, the target third verification block corresponds to the target first verification block, and the target first verification block is the first verification block corresponding to the second verification block used to generate the target third verification block; the recovery module 205 is further configured to:
[0137] If the target third check block and the target first check block are lost, then the second check block used to generate the target third check block is determined based on the second sub-data blocks of each data block.
[0138] Based on the determined second check block, the first sub-data block that was not lost, and the first check block, recover the target third check block;
[0139] Find the third check block obtained based on the target first check block, and use the found third check block as the intermediate third check block;
[0140] Based on the intermediate third check block, restore the target first check block.
[0141] In some embodiments, the recovery module 205 is specifically used for:
[0142] Find the second check block used to generate the intermediate third check block, and use the found second check block as the intermediate second check block;
[0143] Calculate the value of the intermediate second check block based on the second sub-data block of each data block;
[0144] Based on the value of the intermediate second check block, the first sub-data block that was not lost, and the first check block, the target first check block is recovered.
[0145] In some embodiments, the recovery module 205 is specifically used for:
[0146] If multiple data blocks and multiple check blocks are lost, the lost first sub-data block and first check block are recovered based on the first sub-data block and the first check block that were not lost.
[0147] Based on the unlost second check block, the unlost and recovered first sub-data block and the first check block, recover the lost second check block;
[0148] Based on the second check block that was not lost and recovered, and the second sub-data block that was not lost, the lost second sub-data block is recovered.
[0149] See also Figure 3 Embodiments of this application also provide an electronic device, including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.
[0150] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.
[0151] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0152] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.
[0153] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.
[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0155] The data processing method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A data processing method, characterized in that, The method includes: Obtain target data to be stored, the target data comprising k data blocks, each of the data blocks being divided into a first sub-data block and a second sub-data block; Based on multiple different encoding functions, the first sub-data block and the second sub-data block are encoded respectively to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and the second check blocks obtained based on the same encoding function are corresponding to each other. The first sub-data block and the first check block are divided into m-1 groups, and the elements in each group are XORed to obtain m-1 intermediate data blocks. The m-1 intermediate data blocks are XORed with the m-1 second check blocks to obtain m-1 third check blocks. Among them, for one of the second check blocks, the intermediate data block XORed with the second check block is not based on the first check block corresponding to the second check block. The third verification block and the second verification block without XOR operation are used as the verification block of the second sub-data block, and the first verification block is used as the verification block of the first sub-data block. In the event of data loss, data recovery is performed based on the verification blocks.
2. The method according to claim 1, characterized in that, The step of dividing the first sub-data block and the first check block into m-1 groups includes: right Round down to the nearest integer and use the resulting value as the number of the first element; right Round up to the nearest integer and use the resulting value as the number of the second element; The first sub-data block and the first check block are divided into m-1 groups according to at least one of the first element quantity and the second element quantity, where n is the sum of m and k.
3. The method according to claim 1 or 2, characterized in that, The data recovery based on the verification block in the event of data loss includes: If one of the target data blocks is lost among the k data blocks, the second sub-data block of the target data block is recovered based on the second sub-data blocks of the other data blocks besides the target data block and the second check block that has not been XORed. Find the first intermediate data block obtained based on the first sub-data block of the target data block, and the third check block obtained based on the first intermediate data block; Based on the found third verification block, the recovered second sub-data block and the unlost second sub-data block, the unlost first sub-data block and the first verification block, the first sub-data block of the target data block is recovered.
4. The method according to claim 1 or 2, characterized in that, The data recovery based on the verification block in the event of data loss includes: If the second parity block that has not undergone XOR operation and the corresponding first parity block are lost, the second parity block that has not undergone XOR operation is recovered based on the second sub-data blocks of each data block. Locate the second intermediate data block obtained based on the lost first check block, and the third check block obtained based on the second intermediate data block; Based on the second sub-data blocks of each data block, the found third check block, the first sub-data blocks that were not lost, and the first check block, the lost first check block is recovered.
5. The method according to claim 1 or 2, characterized in that, For one of the target third verification blocks, the target third verification block corresponds to the target first verification block, and the target first verification block is the first verification block corresponding to the second verification block used to generate the target third verification block; the method further includes: If the target third check block and the target first check block are lost, then a second check block for generating the target third check block is determined based on the second sub-data blocks of each data block. Based on the determined second check block, the unlost first sub-data block, and the first check block, the target third check block is recovered; Find the third check block obtained based on the target first check block, and use the found third check block as the intermediate third check block; Based on the intermediate third verification block, the target first verification block is restored.
6. The method according to claim 5, characterized in that, The step of restoring the target first verification block based on the intermediate third verification block includes: Find the second check block used to generate the intermediate third check block, and use the found second check block as the intermediate second check block; The value of the intermediate second check block is calculated based on the second sub-data blocks of each data block; Based on the value of the intermediate second check block, the first sub-data block that was not lost, and the first check block, the target first check block is recovered.
7. The method according to claim 1 or 2, characterized in that, The data recovery based on the verification block in the event of data loss includes: If multiple data blocks and multiple check blocks are lost, the lost first sub-data block and first check block are recovered based on the first sub-data block and the first check block that were not lost. Based on the unlost second check block, the unlost and recovered first sub-data block and the first check block, recover the lost second check block; Based on the second check block that was not lost and recovered, and the second sub-data block that was not lost, the lost second sub-data block is recovered.
8. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire target data to be stored, the target data including k data blocks, each of the data blocks being divided into a first sub-data block and a second sub-data block; The first encoding module is used to encode the first sub-data block and the second sub-data block respectively based on multiple different encoding functions to obtain m first check blocks corresponding to the first sub-data block and m second check blocks corresponding to the second sub-data block. The first check blocks and second check blocks obtained based on the same encoding function correspond to each other. The second encoding module is used to divide the first sub-data block and the first check block into m-1 groups, and perform an XOR operation on the elements in each group to obtain m-1 intermediate data blocks. The third encoding module is used to perform an XOR operation on the m-1 intermediate data blocks and the m-1 second check blocks to obtain m-1 third check blocks. Wherein, for one of the second check blocks, the intermediate data block XORed with the second check block is not obtained based on the first check block corresponding to the second check block. The recovery module is used to use the third verification block and the second verification block without XOR operation as the verification block of the second sub-data block, and to use the first verification block as the verification block of the first sub-data block, and to perform data recovery based on the verification blocks in the event of data loss.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 7.