Encoding method based on erasure code, distributed system, device and storage medium

Abstract

One or more embodiments of the specification provide an erasure code-based encoding method, a distributed system, a device and a storage medium. The method comprises: obtaining K data blocks of to-be-processed data; dividing the K data blocks into L groups of data groups; wherein each group of data groups comprises at least K encoding parameters corresponding to the K data blocks respectively; wherein K and L are integers greater than 1; for each group of data groups, using at least part of the data blocks in other groups of data groups and the encoding parameters corresponding to the at least part of the data blocks in the group of data groups to generate a partial check block of the group of data groups; wherein the data groups and the partial check blocks are stored in data nodes of the distributed system. The embodiment guarantees the security of user data while reducing the amplification of IOPS required for data recovery when data is lost or abnormal.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computer data processing, and in particular, to an encoding method based on erasure code, a distributed system, a device and a storage medium. BACKGROUND

[0002] The storage scale of a distributed system is becoming larger and larger, and the distributed system is a loosely coupled system formed by multiple data nodes interconnected through communication lines.

[0003] Erasure code is an error correction technology, and its basic principle is to divide the stored data into fragments, and generate k+m pieces of data from k pieces of original data obtained by fragmentation through certain check calculation, and restore the original data from any k pieces of data out of k+m pieces. In this way, even if part of the data is lost, the distributed system can still restore the original data.

[0004] Erasure code technology can minimize the storage overhead of the system while ensuring data reliability, so in order to reduce the storage space occupied by data in the distributed system, erasure code technology is applied to the data storage of the distributed system.

[0005] However, based on the existing erasure code technology, when some data is lost, the system needs to read enough data to recover the data required by the user, which will not only cause the amplification of IOPS (Input / Output Per Second), but also consume a large amount of bandwidth during data repair. SUMMARY

[0006] Therefore, one or more embodiments of the present specification provide an encoding method based on erasure code, a distributed system, a device and a storage medium.

[0007] To achieve the above object, the technical solutions of one or more embodiments of the present specification are as follows:

[0008] According to a first aspect of one or more embodiments of the present specification, an encoding method based on erasure code is provided, and the method comprises:

[0009] obtaining K data blocks of data to be processed;

[0010] dividing the K data blocks into L groups of data; wherein each group of data includes at least K encoding parameters corresponding to the K data blocks respectively; wherein K and L are integers greater than 1;

[0011] For each group of data sets, a partial check block of the group of data sets is generated by encoding at least part of data blocks in other groups of data sets and encoding parameters corresponding to the at least part of data blocks in the group of data sets; wherein the data sets and the partial check blocks are stored in data nodes of a distributed system.

[0012] Optionally, the method further comprises:

[0013] For each group of data sets, a global check block of the group of data sets is generated according to the K data blocks and the K encoding parameters; or, in the case that one or more partial check blocks of the group of data sets are generated by using all data blocks in other groups of data sets, the global check block of the group of data sets is generated according to the data blocks in the group of data sets and the corresponding encoding parameters, and the one or more partial check blocks.

[0014] Optionally, the method further comprises:

[0015] After the global check block is generated, L groups of encoding groups corresponding to the L groups of data sets are obtained; each of the encoding groups includes data blocks of one of the groups of data sets, the partial check block and the global check block.

[0016] The L groups of encoding groups are stored in data nodes of a distributed system.

[0017] Optionally, the distributed system is arranged in a multi-AZ environment, and each AZ includes one or more data nodes.

[0018] Each AZ corresponds to at least one group of encoding groups, and the at least one group of encoding groups is stored in one or more data nodes in the AZ.

[0019] Optionally, the method further comprises:

[0020] If a target data block stored in a data node in an AZ that has an exception needs to be read, a target partial check block generated by using the target data block and other data blocks used to generate the target partial check block are read from data nodes in other AZs.

[0021] The target data block is recovered by using the target partial check block and the other data blocks used to generate the target partial check block.

[0022] Optionally, for each group of data sets, if at least two partial check blocks are generated, the partial data blocks used to generate the at least two partial check blocks are different.

[0023] Optionally, the number of partial check blocks in the data sets is in a multiple relationship with the number of data blocks in the data sets.

[0024] The number of partial data blocks in the other data groups used by the partial check block is determined according to the multiple relationship.

[0025] Optionally, the partial check block is determined according to a sum of products of the partial data blocks and corresponding encoding parameters, respectively; and / or

[0026] The global check block is determined according to a sum of products of the K data blocks and the K encoding parameters, respectively.

[0027] Optionally, the number of data groups is determined according to the number of AZs; and / or, the number of global check blocks is in a multiple relationship with the number of AZs used to store the encoding groups.

[0028] Optionally, if at least two global check blocks are required to be generated, the data groups include at least two parameter groups, the parameter group includes K encoding parameters corresponding to the K data blocks, respectively; and the encoding parameters in different parameter groups are different.

[0029] Optionally, the method is applied to a client in a distributed system; and / or, the distributed system has a plurality of data nodes, and the method is applied in at least one data node.

[0030] According to a second aspect of one or more embodiments of the present specification, a distributed system is provided, including a plurality of data nodes, the data nodes are in communication connection with a client, and the data nodes or the client are configured to:

[0031] Obtain K data blocks of data to be processed;

[0032] Divide the K data blocks into L groups of data groups; wherein each group of data groups includes at least K encoding parameters corresponding to the K data blocks, respectively; wherein K and L are integers greater than 1;

[0033] For each group of data groups, at least part of the data blocks in other data groups and the encoding parameters corresponding to the at least part of the data blocks in the group of data groups are used for encoding to generate a partial check block of the group of data groups; wherein the data groups and the partial check blocks thereof are stored in the data nodes of the distributed system.

[0034] According to a third aspect of one or more embodiments of the present specification, a computer device is provided, including:

[0035] A processor;

[0036] A memory for storing processor executable instructions;

[0037] The processor implements the method of any one of the first aspect by running the executable instructions.

[0038] According to a fourth aspect of one or more embodiments of the present specification, a computer readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method according to any one of the first aspect.

[0039] The method for encoding based on erasure code, the distributed system, the computer device and the storage medium provided by one or more embodiments of the present specification can divide the K data blocks obtained by acquiring the data shards to be processed into L data groups, and then generate the partial check blocks of each data group by using at least part of the data blocks in other data groups when acquiring the partial check blocks of each data group. The data group and the partial check blocks thereof can be stored in the data nodes of the distributed system. While ensuring the security of user data, when data recovery of an abnormal data block is needed, the partial check blocks generated by using the abnormal data block in other data groups and the other data blocks used to generate the partial check blocks can be used to reduce the number of blocks that need to be read, thereby reducing the amplification of IOPS for data recovery when data is lost or abnormal, reducing the bandwidth loss during data recovery, and improving the efficiency of data recovery. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 FIG. 1 is a structural schematic diagram of a distributed system provided by an example embodiment.

[0041] Figure 2 FIG. 4 is a schematic diagram of an encoding group obtained by an encoding method based on erasure code in the related art.

[0042] Figure 3 FIG. 7 is a structural schematic diagram of a distributed system provided by an example embodiment and arranged in a multi-AZ environment.

[0043] Figure 4 FIG. 8 is a flowchart of an encoding method based on erasure code provided by an example embodiment.

[0044] Figure 5 FIG. 11 is a schematic diagram of an encoding group obtained by using the encoding method based on erasure code provided by an embodiment of the present specification.

[0045] Figure 6 FIG. 13 is a schematic diagram of an encoding group stored in a data node in an AZ provided by an example embodiment.

[0046] Figure 7 FIG. 16 is a structural schematic diagram of a computer device provided by an example embodiment. DETAILED DESCRIPTION

[0047] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description herein relates to the drawings, in which like references indicate like elements, unless otherwise indicated. The following exemplary embodiments described are not meant to represent all embodiments consistent with one or more embodiments of the present description. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the present description as detailed in the appended claims.

[0048] It should be noted that the order of the steps of the methods in other embodiments is not necessarily the order as shown and described in the present description. In some other embodiments, the steps included in the methods can be more or less than described in the present description. Furthermore, a single step described in the present description can be broken down into multiple steps in other embodiments; and multiple steps described in the present description can be combined into a single step in other embodiments.

[0049] The storage size of a distributed system is becoming larger and larger. Please refer to Figure 1 A distributed system is a loosely coupled system composed of multiple data nodes interconnected through communication lines. Each data node is a computer device capable of independently processing a certain transaction.

[0050] Erasure Code is an error correction technique. Its basic principle is to divide the stored data into fragments, and generate k+m pieces of data from k pieces of original data through certain check calculation, and restore the original data from any k pieces of data. Thus, even if some data is lost, the distributed system can still restore the original data.

[0051] Erasure Code technology can minimize the storage overhead of the system while ensuring data reliability. Therefore, in order to reduce the storage space occupied by data in the distributed system, Erasure Code technology is applied to the data storage of the distributed system. However, based on the existing Erasure Code technology, when some data is lost, the system needs to read enough data to restore the data required by the user. On the one hand, it will lead to the amplification of IOPS (Input / Output Per Second), and on the other hand, it will also consume a large amount of bandwidth during data repair.

[0052] Taking LRC erasure code as an example, LRC (Locally Repairable Codes) is a local check encoding method, and its core idea is that the parity block is divided into global parity block and local reconstruction parity block, and the fault recovery can be calculated in groups. LRC encoding is generally represented as LRC (k, z, g), where k represents the total number of data blocks, z represents the number of local reconstruction parity blocks, and g represents the number of global parity blocks. As shown in Figure 2 LRC (12, 3, 6) erasure code is taken as an example, which represents 12 data blocks, 3 local reconstruction parity blocks and 6 global parity blocks. Figure 2 The global parity blocks P1-P6 are generated by encoding 12 data blocks A1-A4, B1-B4 and C1-C4, and the local reconstruction parity block L1 is generated by XORing the data blocks A1-A4, the global parity blocks P1 and P2, the local reconstruction parity block L2 is generated by XORing the data blocks B1-B4, the global parity blocks P3 and P4, and the local reconstruction parity block L3 is generated by XORing the data blocks C1-C4, the global parity blocks P5 and P6. Among them, the data blocks A1-A4, the global parity blocks P1 and P2, and the local reconstruction parity block L1 constitute a first encoding group, the data blocks B1-B4, the global parity blocks P3 and P4, and the local reconstruction parity block L2 constitute a second encoding group, and the data blocks C1-C4, the global parity blocks P5 and P6, and the local reconstruction parity block L3 constitute a third encoding group. When one of the data nodes (such as the data node for storing the first encoding group) fails, the user needs to read the data block A1, and needs to read 12 data blocks B1-B4, C1-C4 and P3-P6 from other data nodes to recover the data on the data block A1. On the one hand, it will cause the amplification of IOPS (Input / Output Per Second), and on the other hand, it will consume a large amount of bandwidth during data repair. With the increase of the number of data copies in the erasure code stripe, this problem becomes increasingly serious.

[0053] Based on this, the encoding method based on erasure code provided by the embodiments of the present specification can divide the K data blocks obtained by acquiring the data shards to be processed into L data groups, and then generate the local reconstruction parity block of each data group by using at least part of the data blocks in other data groups, which can guarantee the security of user data and can use the local reconstruction parity block generated by using the abnormal data block in other data groups and other data blocks to generate the local reconstruction parity block when data recovery of the abnormal data block is needed, without Figure 2The number of blocks that need to be read is reduced, thus reducing the amplification of IOPS for data recovery when data is lost or abnormal, and reducing the bandwidth loss during data recovery, which is beneficial to improve the efficiency of data recovery.

[0054] In an exemplary application scenario, as users increasingly demand data reliability, the related art supports storage in multiple AZ (Availability Zones), that is, the distributed system is deployed in a multi-AZ environment. AZ (Availability Zones) refers to a physical area in the same region with independent power and network. The multi-AZ (Availability Zones) scheme is to disperse data to multiple separate data centers and ensure that data is still available when a single AZ encounters a machine room or network device failure. When the distributed system is deployed in a multi-AZ environment, the distributed system includes a plurality of data nodes, and each AZ includes one or more data nodes of the distributed system; in one example, please refer to Figure 3 , a distributed system with 7 data nodes is deployed in an environment with 3 AZs, in which AZ0 includes 2 data nodes, AZ1 includes 1 data node, and AZ2 includes 3 data nodes.

[0055] In order to reduce the storage space occupation of data under multi-AZ storage, erasure code technology is applied to multi-AZ data storage. When a single AZ encounters a machine room or network device failure, the system can obtain data from other machine rooms and use erasure code technology to recover abnormal data to provide to users. In the erasure code stripe as shown in Figure 2 , data blocks A1-A4, global check blocks P1, P2, and partial check block L1 form a first encoding group, data blocks B1-B4, global check blocks P3, P4, and partial check block L2 form a second encoding group, data blocks C1-C4, global check blocks P5, P6, and partial check block L3 form a third encoding group, and the three encoding groups are stored in the data nodes of three different AZs, that is, each AZ is used to store one of the encoding groups, which can be part or all of the data nodes in each AZ that store the encoding group, such as in Figure 3 , the first encoding group is stored in the data nodes of AZ0, the second encoding group is stored in the data nodes of AZ1, and the third encoding group is stored in the data nodes of AZ2. When AZ0 fails, if data block A1 needs to be read, B1-B4, C1-C4, P3-P6, a total of 12 data blocks, need to be read from other AZs to recover the data on data block A1, which causes a large amplification of IOPS.

[0056] Based on this, the encoding method based on erasure code provided by the embodiments of the present specification can be used to divide the K data blocks obtained by obtaining the data shards to be processed into L data groups, and then when obtaining the partial check blocks of each data group, at least part of the data blocks in other data groups can be used to generate the partial check blocks of the data group, such as storing the L data groups and their partial check blocks in L different AZ data nodes respectively; when any AZ is abnormal, if we want to read the target data block stored in the abnormal AZ, we can use the partial check blocks generated by using the target data block in other AZs and other partial data blocks to recover the abnormal data block, reducing the total number of data blocks or check blocks that need to be read when data recovery is performed, thereby reducing the bandwidth consumption and IOPS when data repair is performed.

[0057] Among them, the encoding method based on erasure code provided by the embodiments of the present specification can be applied to the client in the distributed system; and / or, the distributed system has a plurality of data nodes, and the encoding method based on erasure code can also be applied in at least one data node.

[0058] Next, the encoding method based on erasure code provided by the embodiments of the present specification will be described. Please refer to Figure 4 , the method comprises:

[0059] In step S101, K data blocks of data to be processed are obtained.

[0060] In step S102, the K data blocks are divided into L data groups; wherein each data group includes at least K encoding parameters corresponding to the K data blocks respectively; wherein K and L are integers greater than 1.

[0061] In step S103, for each data group, at least part of the data blocks in other data groups and the encoding parameters corresponding to the at least part of the data blocks in the data group are used for encoding to generate the partial check blocks of the data group; wherein the data group and its partial check blocks are stored in the data nodes of the distributed system.

[0062] It can be understood that the embodiments of the present specification do not make any limitation on the specific type of the to-be-processed data, and can be specifically set according to actual application scenarios. For example, the to-be-processed data can be a data request, which can be further divided into K sub-requests, each of which is a data block. For example, the to-be-processed data can also be a transaction data to be stored, which can be processed by fragmentation to obtain K data blocks. For example, the to-be-processed data can also be user registration data, which can be processed by fragmentation to obtain K data blocks. Wherein, K is an integer greater than 1.

[0063] After obtaining the K data blocks of the to-be-processed data, the K data blocks can be divided into L data groups, wherein L is an integer greater than 1. For example, in a distributed system with a plurality of data nodes, the number of data groups to which the K data blocks can be divided can be determined according to the number of data nodes, for example, one data node is used to store an encoding group composed of data blocks of a data group, partial check blocks and global check blocks, and the present embodiment does not make any limitation.

[0064] For example, in a multi-AZ environment, the number of data groups can also be determined according to the number of AZs; assuming that one AZ is used to store an encoding group composed of data blocks of a data group, partial check blocks and global check blocks, the number of data groups can be set to be less than or equal to the number of AZs. In one example, assuming that there are 3 AZs in the multi-AZ environment, namely AZ0, AZ1 and AZ2, the K data blocks can be divided into at most 3 groups, and after generating the partial check blocks and the global check blocks of each data group, an encoding group is obtained by combining the data blocks, the partial check blocks and the global check blocks of the data group, and there are 3 encoding groups in total, assuming that they are encoding group 1, encoding group 2 and encoding group 3, respectively. Encoding group 1 can be stored in the data nodes of AZ0, encoding group 2 can be stored in the data nodes of AZ1, and encoding group 3 can be stored in the data nodes of AZ2.

[0065] In some embodiments, in the L data groups obtained by division, each data group includes at least K encoding parameters corresponding to the K data blocks, respectively. The K encoding parameters can be used to generate partial check blocks and global check blocks in combination with the data blocks.

[0066] In a possible implementation, for the generation of the global check block, taking the generation of one global check block as an example, for each group of data groups, one global check block of the group of encoded data groups can be generated according to the K data blocks and the K encoding parameters. In the case where at least two global check blocks need to be generated, each group of data groups can include at least two parameter groups, the parameter groups including K encoding parameters respectively corresponding to the K data blocks, and the encoding parameters in different parameter groups being different, so that at least two global check blocks of the group of encoded data groups can be generated according to the K data blocks and the at least two different parameter groups respectively.

[0067] In an exemplary application scenario, in a multi-AZ environment, it is assumed that the data nodes in one AZ are used to store one encoding group, the encoding group including data blocks of one group of data, partial check blocks, and global check blocks, and each encoding group has at least one global check block, that is, the number of global check blocks can be in a multiple relationship with the number of AZs used to store the encoding group. Each AZ storing one group of encoding groups can correspond to one or more global check blocks, and each group of data groups can include one or more groups of parameter groups, the parameter groups including K encoding parameters respectively corresponding to the K data blocks, and the encoding parameters in different parameter groups being different, so that different global check blocks of the group of data can be generated based on the K data blocks and different parameter groups.

[0068] For the specific encoding manner of the global check block, the embodiment does not make any limitation thereon, and the specific setting can be made according to the actual application scenario. In one example, the global check block can be determined according to the sum of products of the K data blocks and the K encoding parameters in the group of data respectively. In another example, the global check block can be determined according to the sum of ratios of the K data blocks and the K encoding parameters in the group of data respectively.

[0069] For example, taking the determination of the global check block according to the sum of products of the K data blocks and the K encoding parameters in the group of data respectively as an example for description: Figure 5 The to-be-processed data is divided into 12 data blocks, and then the 12 data blocks are divided into three groups of data a, b, and c, respectively a={A1, A2, A3, A4}, b={B1, B2, B3, B4}, and c={C1, C2, C3, C4}. In a multi-AZ environment, please refer to Figure 6For example, assuming that data group a is stored in AZ0, data group b is stored in AZ1, and data group c is stored in AZ2, and that each AZ storing a data group corresponds to a global check block, each data group includes a parameter group including 12 encoding parameters corresponding to 12 data blocks, for example, data group a includes 4 data blocks and a parameter group Fx={x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12}, data group b includes 4 data blocks and a parameter group Fy={y1, y2, y3, y4, y5, y6, y7, y8, y9, y10, y11, y12}, and data group c includes 4 data blocks and a parameter group Fz={z1, z2, z3, z4, z5, z6, z7, z8, z9, z10, z11, z12}.

[0070] The global check block P1 corresponding to data group a is P1=Fx(A 1, A2, A3, A4, B 1, B2, B3, B 4, C1, C2, C3, C4) = x1*A1+x2*A2+x3*A3+x4*A4+x5*B1+x6*B2+x7*B3+x8*B4+x9*C1+x10*C2+x11*C3+x12*C4.

[0071] The global check block P2 corresponding to data group b is P2=Fy(A 1, A2, A3, A4, B 1, B2, B3, B 4, C1, C2, C3, C4) = y1*A1+y2*A2+y3*A3+y4*A4+y5*B1+y6*B2+y7*B3+y8*B4+y9*C1+y10*C2+y11*C3+y12*C4.

[0072] The global check block P3 corresponding to data group c is P3=Fz(A 1, A2, A3, A4, B 1, B2, B3, B 4, C1, C2, C3, C4) = z1*A1+z2*A2+z3*A3+z4*A4+z5*B1+z6*B2+z7*B3+z8*B4+z9*C1+z10*C2+z11*C3+z12*C4.

[0073] In one possible implementation, with respect to the generation of the partial check blocks, for each group of data sets, a partial check block of the group of data sets can be generated by encoding at least some data blocks in other data sets and encoding parameters corresponding to the at least some data blocks in the group of data sets. The number of partial check blocks in the group of data sets is in a multiple relationship with the number of data blocks in the group of data sets. The number of data blocks in the other data sets used by the partial check blocks can be determined according to the multiple relationship. For example, the number of data blocks in the other data sets used by the partial check blocks is equal to the number of the multiple. In one example, assuming that the number of data blocks in the group of data sets is 8, the number of partial check blocks is 2, and the multiple between the number of data blocks and the number of partial check blocks is 4, the number of data blocks in the other data sets used by the partial check blocks can be 4.

[0074] For the group of data sets, the other data sets used to generate the partial check blocks in the group of data sets can be all data sets or part of the data sets except the group of data sets.

[0075] In addition, in order to further reduce the total number of data blocks and partial check blocks to be read during subsequent data recovery, the number of partial check blocks can be less than or equal to the number of data blocks in the group of data sets. In one example, assuming that the number of data blocks in the group of data sets is 4, the number of partial check blocks can be 2 or 4. In another example, assuming that the number of data blocks in the group of data sets is 6, the number of partial check blocks can be 2, 3, or 6.

[0076] In some embodiments, for each group of data sets, if at least two partial check blocks are generated, the partial data blocks of the at least two partial check blocks are different from each other, so that each data block has a corresponding partial check block, and thus the lost data block can be recovered according to the partial check block corresponding to the lost data block during subsequent data block loss or AZ abnormality of the group of data sets. In one example, assuming that there are two groups of data sets a and b, the number of data blocks in the group of data sets is 4, the number of partial check blocks is 2, and the multiple between the number of data blocks and the number of partial check blocks is 2, the first partial check block of the group of data sets a can be generated by using two data blocks in the group of data sets b and encoding parameters corresponding to the two data blocks in the group of data sets a, and the second partial check block of the group of data sets a can be generated by using two data blocks different from the two data blocks in the group of data sets b and encoding parameters corresponding to the two data blocks in the group of data sets a.

[0077] This embodiment does not impose any restrictions on the specific encoding method of the partial check block, and it can be set according to the actual application scenario. In one example, the partial check block can be determined based on the sum of the products of the partial data block and its corresponding encoding parameter. In another example, the partial check block can be determined based on the sum of the ratios of the partial data block and its corresponding encoding parameter.

[0078] For example, please refer to Figure 5 as well as Figure 6 Assuming each data group has 4 data blocks and 2 partial check blocks, the ratio between the number of data blocks and the number of partial check blocks in a data group is 2. A partial check block can be generated using 2 data blocks from other data groups. Assuming these other data groups are all data groups other than the current data group, then the two partial check blocks L of data group a... 1-1 L 1-2 It can be generated using two data blocks from data group b and two data blocks from data group c, respectively, and two partial check blocks L are generated. 1-1 L 1-2 The data blocks used are different, for example, L 1-1 =x5*B1+x6*B2+x9*C1+x10*C2,L 1-2 = x7*B3+x8*B4+x11*C3+x12*C4, which can be understood as generating partial check block L. 1-1 It is not limited to using data blocks B1, B2, C1, C2; it can also use B3, B4, C3, C4, or even B1, B2, C3, C4, as long as some check blocks L are used. 1-1 The partial check block L1-2 can be generated using different data blocks. This embodiment does not impose any restrictions on this, so that each data block has a related partial check block. In this way, when a data block is lost, the lost data block can be recovered based on the partial check block related to the lost data block and other data blocks used to generate that partial check block.

[0079] Similarly, the two parts of data group b, check block L 2-1 L 2-2 It can be generated using two data blocks from data group a and two data blocks from data group c, respectively, with partial check block L. 2-1 and partial check block L 2-2 Use different data blocks to generate, for example, use data blocks A1, A2, C1, and C2 to generate a partial check block L. 2-1 L 2-1 = y1*A1+y2*A2+y9*C1+y10*C2; Use data blocks A3, A4, C3, and C4 to generate partial check block L.2-2 , L 2-2 = y3*A3 + y4*A4 + y11*C3 + y12*C4. Similarly, two partial check blocks L 3-1 , L 3-2 of the data group c can be generated using two data blocks in the data group a and two data blocks in the data group b respectively, the partial check block L 3-1 and the partial check block L 3-2 are generated using different data blocks, for example, the partial check block L 3-1 , L 3-1 = z1*A1 + z2*A2 + z5*B1 + z6*B2; the partial check block L 3-2 , L 3-2 = z3*A3 + z4*A4 + z7*B3 + z8*B4.

[0080] In some possible embodiments, for the data group, if the other data groups used to generate the partial check blocks in the data group are all the data groups except the current data group, and one or more partial check blocks in the current data group are generated using all the data blocks in the other data groups, then the one or more partial check blocks in the current data group already contain the information of all the data blocks in the other data groups, and thus the global check block of the current data group can be generated according to the data blocks in the current data group and the corresponding encoding parameters, and the one or more partial check blocks.

[0081] In one example, referring to Figure 5 and Figure 6 , the global check block P1 corresponding to the data group a is P1 = x1*A1 + x2*A2 + x3*A3 + x4*A4 + L 1-1 + L 1-2 ; the global check block P2 corresponding to the data group b is P2 = y5*B1 + y6*B2 + y7*B3 + y8*B4 + L 2-1 + L 2-2 ; and the global check block P3 corresponding to the data group c is P3 = z9*C1 + z10*C2 + z11*C3 + z12*C4 + L 3-1 + L 3-2 .

[0082] In some embodiments, in the case that each data group includes one set of parameter groups, the encoding parameters in the set of parameter groups can be used to generate the partial check blocks of the data group; in the case that each data group includes at least two sets of parameter groups, one set of parameter groups can be selected to generate the partial check blocks of the data group, or at least two partial check blocks of the data group can be generated according to the two sets of parameter groups respectively.

[0083] In some embodiments, after generating the partial check block and the global check block of each data group, L groups of encoding groups corresponding to the L groups of data groups can be obtained, each of the encoding groups including the data block, the partial check block and the global check block of one of the data groups; the L groups of encoding groups correspond to the L groups of data groups one-to-one, and then the L groups of encoding groups can be stored in the data nodes of the distributed system.

[0084] For example, the distributed system is arranged in a multi-AZ environment, each AZ including one or more data nodes; wherein each AZ corresponds to at least one group of encoding groups, and the at least one group of encoding groups is stored in one or more data nodes in the AZ. Please refer to Figure 6 , in Figure 5 the embodiments, Figure 5 the embodiments shown can obtain three groups of encoding groups, the first group of encoding groups {A1, A2, A3, A4, L 1-1 , L 1-2 , P1} is stored in the data nodes of AZ0, the second group of encoding groups {B1, B2, B3, B4, L 2-1 , L 2-2 , P2} is stored in the data nodes of AZ1, and the third group of encoding groups {C1, C2, C3, C4, L 3-1 , L 3-2 , P3} is stored in the data nodes of AZ2.

[0085] In some embodiments, when a data block in the same AZ is lost, it can be recovered using the data blocks, the partial check block and the global check block in the same AZ. For example Figure 5 and Figure 6 in the embodiments shown, P1 can be calculated from the data blocks A1-A4 and the partial check blocks L 1-1 and L 1-2 , when the data block A1 is lost, A1 can be calculated based on the global check block P1, the partial check blocks L 1-1 , L 1-2 , the data blocks A2, A3, A4 and A1-A4 and the encoding parameters corresponding to A1-A4, i.e. P1, L 1-1 , L 1-2 , A2, A3, A4 and A1-A4, the encoding parameters (x1, x2, x3, x4) are known items, and A1 is an unknown item, then

[0086] In some embodiments, when any AZ is abnormal, if a target data block stored in the data node of the abnormal AZ needs to be read, a target partial check block generated by using the target data block is read from the data node of another AZ, and other data blocks used to generate the target partial check block are read; then the target data block is recovered by using the target partial check block and the other data blocks used to generate the target partial check block. In this embodiment, since the partial check block is generated by using at least part of data blocks of other data groups, the total number of data blocks or partial check blocks that need to be read during data recovery is reduced, thereby reducing the bandwidth consumption and IOPS during data repair, and also facilitating to improve the data recovery efficiency.

[0087] For example, such as Figure 5 and Figure 6 In the described embodiment, assuming that AZ0 is abnormal, if data block A1 or A2 needs to be read, the partial check block L 2-1 , L 3-1 and other data blocks C1, C2, B1 and B2 used to generate the partial check block L 2-1 , L 3-1 are recovered, then y1*A1+y2*A2=L 2-1 -y9*C1-y10*C2; y1*A1+y2*A2=L 3-1 -z5*B1-z6*B2; only 2 partial check blocks and 4 data blocks need to be read from other AZs to recover the data block A1 or A2, the number of blocks to be read is reduced, the IOPS amplification during AZ failure is reduced, and the data recovery efficiency is improved.

[0088] Correspondingly, the specification also provides a distributed system comprising a plurality of data nodes, wherein the data nodes are in communication connection with a client, and the data nodes or the client are configured to:

[0089] obtain K data blocks of to-be-processed data;

[0090] divide the K data blocks into L data groups; wherein each data group comprises at least K encoding parameters corresponding to the K data blocks, respectively; wherein K and L are integers greater than 1;

[0091] for each data group, encode at least part of data blocks in other data groups and the encoding parameters in the data group corresponding to the at least part of data blocks to generate a partial check block of the data group; wherein the data group and the partial check block thereof are stored in a data node of the distributed system.

[0092] Optionally, the data node or the client is configured to: for each group of data groups, generate a global check block of the group of data groups according to the K data blocks and the K encoding parameters; or, in the case that one or more partial check blocks of the group of data groups are generated using all data blocks in other groups of data, generate the global check block of the group of data groups according to the data blocks in the group of data groups and the corresponding encoding parameters, and the one or more partial check blocks.

[0093] Optionally, the data node or the client is configured to: after generating the global check block, obtain L groups of encoding groups; each group of the encoding groups comprises data blocks of one group of the data groups, the partial check blocks and the global check block; store the L groups of encoding groups in data nodes of the distributed system.

[0094] Optionally, the distributed system is deployed in a multi-AZ environment, and each AZ comprises one or more data nodes.

[0095] Optionally, each AZ corresponds to at least one group of encoding groups, and the at least one group of encoding groups is stored in one or more data nodes in the AZ.

[0096] Optionally, the data node or the client is configured to: if a target data block stored in a data node in an AZ that has an exception needs to be read, read a target partial check block generated using the target data block and other data blocks used to generate the target partial check block from data nodes in other AZs; and recover the target data block using the target partial check block and the other data blocks used to generate the target partial check block.

[0097] Optionally, for each group of data groups, if at least two partial check blocks are generated, the partial data blocks used to generate the at least two partial check blocks are different.

[0098] Optionally, the number of partial check blocks in the group of data groups is in a multiple relationship with the number of data blocks in the group of data groups; and the number of partial data blocks in the other groups of data used by the partial check blocks is determined according to the multiple relationship.

[0099] Optionally, the partial check blocks are determined according to sums of products of the partial data blocks and corresponding encoding parameters respectively; and / or the global check block is determined according to a sum of products of the K data blocks and the K encoding parameters respectively.

[0100] Optionally, the number of groups of data groups is determined according to the number of AZs; and / or the number of global check blocks is in a multiple relationship with the number of AZs used to store the encoding groups.

[0101] Optionally, if at least two global check blocks need to be generated, the data set comprises at least two parameter sets, and the parameter sets comprise K encoding parameters corresponding to the K data blocks respectively; the encoding parameters in different parameter sets are different.

[0102] Correspondingly, refer to Figure 7 The embodiments of the present specification also provide a computer device 20, comprising:

[0103] a processor 21;

[0104] a memory 22 for storing executable instructions of the processor 21;

[0105] The processor 21 realizes the method of any one of the above by running the executable instructions.

[0106] The processor 21 executes the executable instructions included in the memory 22, and the processor 21 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0107] The storage 22 stores executable instructions of the frequency method, and can include at least one type of storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or a DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. Also, the computer device 20 can cooperate with a network storage device performing a storage function of the storage through a network connection. The storage 22 can be an internal storage unit of the computer device 20, such as a hard disk or a memory of the computer device 20. The storage 22 can also be an external storage device of the computer device 20, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 20. Further, the storage 22 can include both the internal storage unit and the external storage device of the computer device 20. The storage 22 is used to store executable instructions and other programs and data required by the device. The storage 22 can also be used to temporarily store data that has been output or will be output.

[0108] As an example, Figure 7 The computer device 20 can further include a network interface 23 to perform data exchange with other devices through the network interface.

[0109] The various embodiments described herein can be implemented using computer readable media, for example, computer software, hardware, or any combination thereof. For hardware implementation, the embodiments described herein can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices designed to perform the functions described herein, or a combination thereof. For software implementation, the embodiments, such as processes or functions, can be implemented with separate software modules, which allow at least one function or operation to be performed. The software code can be implemented by a software application (or program) written in any suitable programming language to be executed by a controller.

[0110] In an example embodiment, a non-transitory computer-readable storage medium including instructions, for example, a memory including instructions, is also provided, which can be executed by a processor of an apparatus to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0111] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform the above method.

[0112] The system, apparatus, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0113] In a typical configuration, a computer includes one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0114] The memory can include non-persistent memory in computer readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or Flash memory. The memory is an example of computer readable media. The computer readable media is a tangible computer readable media that excludes transitory computer readable media.

[0115] The computer readable media includes non-transitory and transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory computer readable media, such as modulated data signals and carrier waves.

[0116] It is also to be noted that the terms "comprising", "including", and "having" or variations thereof herein, are intended to be open-ended terms that specify the presence of the stated elements but do not preclude the presence of additional elements. It is also to be noted that the term "if' as used herein, is intended to mean "when" or "if the condition is met" and therefore indicates that the events happening after it are dependent on the event of interest occurring or the condition being met.

[0117] The above description of the preferred embodiments of the present application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present application to the precise form described, and many modifications, equivalents and variations are possible in light of these teachings. It is intended that the scope of the present application be limited not with this detailed description, but rather by the claims appended hereto.

[0118] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0119] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are used only to distinguish one piece of information from another, but do not imply a relationship between the information. For example, without departing from the scope of one or more embodiments, first information can be termed second information, and similarly, second information can be termed first information. The word "if' as used herein means "when" or "if the condition is met" depending on the context.

[0120] The foregoing description of one or more embodiments of the present application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present application to the precise form disclosed, and modifications, equivalent, and variations are possible in light of these teachings. It is intended that the scope of the present application be limited not with this detailed description but rather by the claims appended hereto.

Claims

1. A method for encoding based on erasure code, the method comprising: obtaining K data blocks to be processed; dividing the K data blocks into L groups of data blocks; wherein each group of data blocks comprises at least K encoding parameters corresponding to the K data blocks respectively; wherein K and L are integers greater than 1; for each group of data blocks, using at least part of the data blocks in other groups of data blocks and the encoding parameters corresponding to the at least part of the data blocks in the group of data blocks to generate partial check blocks of the group of data blocks; wherein the data blocks and the partial check blocks are stored in data nodes of a distributed system, and when any target data block in the at least part of the data blocks is lost, the target data block is recovered using the partial check blocks of the group of data blocks and other data blocks in the at least part of the data blocks except the target data block. 2.The method of claim 1, further comprising: for each group of data blocks, generating a global check block of the group of data blocks according to the K data blocks and the K encoding parameters; or, in the case of generating one or more partial check blocks of the group of data blocks using all data blocks in other groups of data blocks, generating a global check block of the group of data blocks according to the data blocks in the group of data blocks and the corresponding encoding parameters and the one or more partial check blocks. 3.The method of claim 2, further comprising: after generating the global check block, obtaining L groups of encoding groups corresponding to the L groups of data blocks; each of the encoding groups comprises the data blocks, the partial check blocks and the global check block of one of the data blocks; storing the L groups of encoding groups in data nodes of a distributed system.

4. The method of claim 3, the distributed system layout is in a multi-AZ environment, each AZ includes one or more data nodes, wherein, The AZs are availability zones; wherein each AZ corresponds to at least one group of encoding groups stored in one or more data nodes in the AZ. 5.The method of claim 4, further comprising: if a target data block stored in a data node in an abnormal AZ needs to be read, reading a target partial check block generated using the target data block and other data blocks used to generate the target partial check block from data nodes in other AZs; recovering the target data block using the target partial check block and the other data blocks used to generate the target partial check block. 6.The method of claim 1, for each group of data blocks, if at least two partial check blocks are generated, the partial data blocks used to generate the at least two partial check blocks are different. 7.The method of claim 1, the number of partial check blocks in the data blocks and the number of data blocks in the data blocks are in a multiple relationship; wherein the number of partial data blocks in the other groups of data blocks used by the partial check blocks is determined according to the multiple relationship. 8.The method of claim 4, the number of data blocks is determined according to the number of AZs; and / or the number of global check blocks and the number of AZs used to store the encoding groups are in a multiple relationship.

9. The method of claim 2, wherein if at least two global parity blocks are needed to be generated, the data groups comprise at least two parameter groups, and the parameter groups comprise K encoding parameters corresponding to the K data blocks respectively; and the encoding parameters in different parameter groups are different.

10. The method of any one of claims 1 to 9, wherein the method is applied to a client in a distributed system; and / or the distributed system has a plurality of data nodes, and the method is applied in at least one data node.

11. A distributed system comprising a plurality of data nodes, wherein the data nodes are communicatively connected to a client, and the data nodes or the client are configured to: obtain K data blocks of data to be processed; and generate, for each data group, a partial parity block of the data group using at least part of the K data blocks and encoding parameters corresponding to the at least part of the K data blocks in the data group; wherein K and L are integers greater than 1; and wherein the data groups and the partial parity blocks of the data groups are stored in the data nodes of the distributed system, and when any target data block of the at least part of the K data blocks is lost, the target data block is recovered using the partial parity block of the data group and the other data blocks of the at least part of the K data blocks excluding the target data block. The AZ is an availability zone. dividing the K data blocks into L groups of data blocks; wherein, 13. A computer device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the method of any one of claims 1 to 10 by executing the executable instructions.

14. A computer-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 10.

12. The distributed system of claim 11, the distributed system laid out in a multi-AZ environment, each AZ comprising one or more data nodes, wherein, ​ ​ ​ ​ ​ ​

Images