Coding method, device, computing equipment and computer storage medium

Abstract

The present application discloses a coding method, apparatus, computing device and computer storage medium, and relates to the field of data processing technology. The method is implemented based on a pre-constructed first erasure code and multiple second erasure codes, and includes: generating multiple data slices according to the data to be encoded, and dividing the multiple data slices into multiple data slice groups; using the first erasure code, globally encoding the multiple data slices to generate global verification slices; using multiple second erasure codes to locally encode at least a part of the multiple data slice groups, respectively, to generate a part of local verification slices; based on a part of the local verification slices, generating another part of the local verification slices. In the above manner, the present application can achieve fast encoding and also help to improve the probability of decoding.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a coding method, device, computing equipment and computer storage medium. Background Art

[0002] Erasure coding is a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations.

[0003] LRC erasure code is a type of coding that comprehensively considers decoding bandwidth, redundancy, and error correction capability in distributed storage systems. The LRC encoding and decoding method has been applied to distributed storage systems to a certain extent. Based on different system designs, the parameters of LRC encoding and decoding are different.

[0004] However, the inventors discovered during the process of implementing the present application that, in the existing LRC encoding and decoding method, the encoding and decoding performance is relatively poor, and the probability of repair after a fragment failure cannot be maximized. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an encoding method, an apparatus, a computing device and a computer storage medium, as well as a decoding method, an apparatus, a computing device and a computer storage medium, so as to solve the problems existing in the above-mentioned prior art that the encoding and decoding performance is poor and the probability of repair after a fragment failure cannot be maximized.

[0006] According to one aspect of the present application, a coding method is provided. The method is implemented based on a pre-constructed first erasure code and a plurality of second erasure codes, and includes:

[0007] Generate multiple data slices according to the data to be encoded, and divide the multiple data slices into multiple data slice groups;

[0008] Using the first erasure code, globally encode multiple data shards to generate global check shards;

[0009] At least a portion of the data shard groups in the plurality of data shard groups are locally encoded using a plurality of second erasure codes to generate a portion of local check shards; and another portion of local check shards is generated based on the portion of local check shards.

[0010] Optionally, the method further includes: storing multiple data shards, local check shards and global check shards in different storage nodes respectively.

[0011] Optionally, generating multiple data slices according to the data to be encoded further includes: splitting the data to be encoded into m sub-data to be encoded, and writing the m sub-data to be encoded into the m data slices respectively.

[0012] Optionally, using the first erasure code to globally encode the multiple data shards to generate the global check shards further includes:

[0013] Use the first erasure code to globally encode the m data shards to obtain g global verification data, and write the g-1 global verification data into the g-1 global verification shards respectively;

[0014] The method also includes: writing one global check data other than g-1 global check data as intermediate data into the lth local check shard, where l is the number of data shard groups.

[0015] Optionally, locally encoding at least a portion of the data shard groups in the plurality of data shard groups using a plurality of second erasure codes to generate a portion of local check shards further comprises:

[0016] For the i-th data shard group, 0<i<l, the i-th data shard group is locally encoded using the i-th second erasure code to obtain the i-th local verification data, and the i-th local verification data is written into the i-th local verification shard.

[0017] Optionally, generating another part of the local check slices based on a part of the local check slices further includes: for the lth data slice group, obtaining the lth local check data based on the data written by the 1st to the lth local check slices, and writing the lth local check data into the lth local check slice.

[0018] According to another aspect of the present application, a decoding method is provided. The method is implemented based on a pre-constructed first erasure code and a plurality of second erasure codes, and includes:

[0019] Obtain a first shard sequence number list to be loaded and a second shard sequence number list to be repaired, and load the first shard according to the first shard sequence number list; use the first erasure code to globally repair the data shards and / or global check shards contained in the second shard according to the data shards and / or global check shards contained in the first shard, and obtain repaired data shards and / or global check shards; update the first shard sequence number list and the second shard sequence number list;

[0020] Traverse all data shard groups. If any data shard group has a local check shard included in the second shard, use the second erasure code of the data shard group to partially repair the local check shard according to the first shard to obtain a repaired local check shard; update the first shard sequence number list and the second shard sequence number list.

[0021] Optionally, using the first error correction code, globally repairing the data slices and / or the global check slices included in the second slice according to the data slices and / or the global check slices included in the first slice is specifically as follows:

[0022] When the total number of data shards and / or global check shards contained in the first shard meets the repair condition, the data shards and / or global check shards contained in the second shard are globally repaired according to the data shards and / or global check shards contained in the first shard.

[0023] Optionally, before globally repairing the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard by using the first erasure code, the method further includes:

[0024] If it is determined based on the first shard sequence number list and the second shard sequence number list that any data shard group has one and only one data shard or local check shard included in the second shard, then the one data shard or local check shard included in the second shard is partially repaired to obtain a repaired data shard or local check shard; the first shard sequence number list and the second shard sequence number list are updated.

[0025] Optionally, before globally repairing the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard by using the first erasure code, the method further includes:

[0026] If it is determined according to the first shard sequence number list that the 1st to the 1st local check shards are all included in the first shard, then the intermediate data is calculated according to the local check data in the 1st to the 1st local check shards, and the intermediate data is written into the 1st local check shard;

[0027] Using the first error correction code, globally repairing the data slices and / or the global check slices included in the second slice according to the data slices and / or the global check slices included in the first slice is specifically as follows:

[0028] Using the first erasure code, globally repair the data shards and / or global check shards included in the second shard according to the lth local check shard, the data shards and / or global check shards included in the first shard; wherein m is the number of data shards and l is the number of data shard groups.

[0029] Optionally, after performing the global repair, the method further includes: calculating local verification data of the lth data shard group, and writing the local verification data into the lth local verification shard.

[0030] Optionally, the method also includes: pre-constructing a decision tree; obtaining a first shard sequence number list to be loaded and a second shard sequence number list to be repaired further includes: obtaining a second shard sequence number list to be repaired; traversing the decision tree according to the second shard sequence number list to obtain a first shard sequence number list to be loaded.

[0031] Optionally, the method also includes: if there is at least one first shard that has not been loaded successfully, updating the second shard sequence number list according to the sequence number of at least one first shard that has not been loaded successfully, traversing the decision tree according to the updated second shard sequence number list to obtain an updated first shard sequence number list, and loading the first shard according to the updated first shard sequence number list; repeating this step until all the first shards corresponding to the updated first shard sequence number list are loaded successfully.

[0032] According to another aspect of the present application, there is provided an encoding device, comprising:

[0033] A partitioning module, adapted to generate a plurality of data slices according to the data to be encoded, and to partition the plurality of data slices into a plurality of data slice groups;

[0034] A global encoding module, adapted to globally encode the plurality of data shards using a first erasure code to generate a global check shard;

[0035] The local encoding module is suitable for using multiple second erasure codes to locally encode at least a part of the multiple data shard groups respectively to generate a part of local check shards; and generating another part of local check shards based on the part of the local check shards.

[0036] In an optional manner, the device also includes a storage module, which is suitable for storing multiple data shards, local check shards and global check shards in different storage nodes respectively.

[0037] In an optional manner, the division module is further adapted to: split the data to be encoded into m sub-data to be encoded, and write the m sub-data to be encoded into m data slices respectively.

[0038] In an optional manner, the global encoding module is further adapted to: globally encode the m data shards using the first erasure code to obtain g global check data, and write the g-1 global check data into the g-1 global check shards respectively;

[0039] The device also includes: a data writing module, adapted to write one global check data other than g-1 global check data as intermediate data into the lth local check shard, where l is the number of data shard groups.

[0040] In an optional manner, the local encoding module is further adapted to: for the i-th data shard group, 0<i<l, locally encode the i-th data shard group using the i-th second erasure code to obtain the i-th local verification data, and write the i-th local verification data into the i-th local verification shard.

[0041] In an optional manner, the local encoding module is further adapted to:

[0042] For the lth data shard group, the lth local verification data is obtained according to the data written into the 1st to lth local verification shards, and the lth local verification data is written into the lth local verification data shard.

[0043] According to another aspect of the present application, a decoding device is provided, including:

[0044] A global repair module is adapted to obtain a first shard sequence number list to be loaded and a second shard sequence number list to be repaired, and load the first shard according to the first shard sequence number list; use the first erasure code to perform global repair on the data shards and / or global check shards included in the second shard according to the data shards and / or global check shards included in the first shard, to obtain repaired data shards and / or global check shards; and update the first shard sequence number list and the second shard sequence number list;

[0045] The local repair module is suitable for traversing all data shard groups. If any data shard group has a local check shard contained in the second shard, the local check shard is locally repaired according to the first shard using the second erasure code of the data shard group to obtain a repaired local check shard; the first shard sequence number list and the second shard sequence number list are updated.

[0046] In an optional manner, the global repair module is further adapted to:

[0047] When the total number of data shards and / or global check shards contained in the first shard meets the repair condition, the data shards and / or global check shards contained in the second shard are globally repaired according to the data shards and / or global check shards contained in the first shard.

[0048] In an optional manner, the local repair module is further adapted to: before globally repairing the data shards and / or global check shards included in the second shard using the first erasure code according to the data shards and / or global check shards included in the first shard, if it is determined according to the first shard sequence number list and the second shard sequence number list that any data shard group has only one data shard or local check shard included in the second shard, then locally repairing the one data shard or local check shard included in the second shard to obtain a repaired data shard or local check shard; and updating the first shard sequence number list and the second shard sequence number list.

[0049] In an optional manner, the global repair module is further adapted to: if it is determined according to the first shard sequence number list that the 1st to the lth local check shards are all included in the first shard, then intermediate data is calculated based on the local check data in the 1st to the lth local check shards, and the intermediate data is written into the lth local check shard; using the first erasure code, globally repairing the data shards and / or the global check shards included in the second shard according to the lth local check shard, the data shards and / or the global check shards included in the first shard; wherein m is the number of data shards, and l is the number of data shard groups.

[0050] In an optional manner, the device further includes: a local calculation module, adapted to calculate local verification data of the lth data shard group, and write the local verification data into the lth local verification shard.

[0051] In an optional manner, the device further comprises: a construction module, adapted to pre-construct a decision tree;

[0052] The global repair module is further adapted to: obtain a list of second shard sequence numbers to be repaired; and traverse the decision tree according to the list of second shard sequence numbers to obtain a list of first shard sequence numbers to be loaded.

[0053] In an optional manner, the global repair module is further adapted to: if there is at least one first fragment that has not been loaded successfully, updating the second fragment sequence number list according to the sequence number of at least one first fragment that has not been loaded successfully, traversing the decision tree according to the updated second fragment sequence number list to obtain an updated first fragment sequence number list, and loading the first fragment according to the updated first fragment sequence number list; repeating this step until all the first fragments corresponding to the updated first fragment sequence number list are loaded successfully.

[0054] According to another aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0055] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above encoding method and / or decoding method.

[0056] According to another aspect of the present application, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned encoding method and / or decoding method.

[0057] According to the encoding method, apparatus, computing device and computer storage medium of the present application, by constructing a global erasure code and multiple local erasure codes, multiple data shards are encoded using the global erasure code to obtain multiple global check shards, local erasure codes are used to encode part of the data shard group to obtain part of the local check data, and the part of the local check data is used to calculate to obtain another part of the local check data, fast encoding can be achieved, and it also helps to improve the probability of decoding; further, by making one of the global check data and one of the local check data share memory, memory space can be saved.

[0058] According to the decoding method, apparatus, computing device and computer storage medium of the present application, by first performing global repair and then performing local repair, the probability of successful data repair can be improved, and the performance of data decoding and repair can be improved; further, data can be encoded and decoded in stripes, and when there are damaged slices, a pre-constructed decision tree can be used to quickly decide whether it can be repaired and how to load other slices for optimal repair, which can improve the efficiency of decoding; secondly, when all local check slices are not damaged, the data stored in all local check slices are used to calculate an intermediate data as global check data, and the global check data is used for global repair, which can reduce the number of slices required to be loaded for repair, thereby greatly improving the probability of data repair; in addition, by first performing local repair when a data group has only one data slice or local check slice, the efficiency of data decoding can be improved.

[0059] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0061] Figure 1 A flowchart of the encoding method provided by an embodiment of the present application is shown;

[0062] Figure 2 A flowchart of an encoding method provided by another embodiment of the present application is shown;

[0063] Figure 3 A schematic diagram of an encoding method provided by another embodiment of the present application is shown;

[0064] Figure 4 A flowchart of a decoding method provided by another embodiment of the present application is shown;

[0065] Figure 5 A flowchart of a decoding method provided by another embodiment of the present application is shown;

[0066] Figure 6 A schematic diagram of a decision tree in another embodiment of the present application is shown;

[0067] Figure 7 A schematic diagram showing a decoding method in another embodiment of the present application is shown;

[0068] Figure 8 A structural diagram of a coding and decoding system in another embodiment of the present application is shown;

[0069] Fig. 9 A schematic diagram of the structure of an encoding device provided in another embodiment of the present application is shown;

[0070] Fig.10 A schematic diagram showing the structure of a decoding device provided by another embodiment of the present application is shown;

[0071] Fig.11 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0072] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0073] First, the terms involved in one or more embodiments of the present application are explained.

[0074] Erasure coding: It is a data protection method that divides data into fragments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations.

[0075] Sharding: refers to the process of breaking up a large data table into smaller tables that are distributed across multiple nodes.

[0076] Striping: A method of dividing continuous data into data blocks of the same size and writing each piece of data to different disks in the array. LRC striping shards are divided into three categories: data shards, local checksum shards, and global checksum shards.

[0077] Data sharding: Split the data to be encoded into multiple sub-data, and fill the multiple sub-data obtained by the splitting into each shard respectively to obtain each data shard.

[0078] Data slice group: A data slice group contains some data slices corresponding to the data to be encoded.

[0079] Global encoding: refers to encoding all data shards corresponding to the data to be encoded. The global verification data obtained by the global encoding is associated with the data stored in all data shards.

[0080] Local coding: refers to encoding the partial data slices corresponding to the data to be encoded. The local verification data obtained by the local coding is associated with the data stored in the partial data slices.

[0081] First erasure code: an erasure code used to globally encode all data shards.

[0082] Second erasure code: an erasure code used to locally encode a group of data shards.

[0083] Shard number: the number of each data shard, local checksum shard, and global checksum shard in the stripe.

[0084] The first shard: contains the shards required for repair selected according to the policy, as well as the shards that were successfully repaired during the repair process.

[0085] Second shard: contains known damaged data shards, global checksum shards and / or local checksum shards.

[0086] LRC: Local Reconstruction Code, that is, local reconstruction code. LRC coding with coding parameters (m, l, g) means dividing m data blocks into l groups, each group has m / l data blocks and a local check block, and g global check blocks are generated by all data blocks.

[0087] RS code: Reed-Solomon code is a forward error-corrected channel code that is used to correct the effective polynomial generated by over-sampled data. The encoding process first finds redundancy in these polynomials at multiple points before transmitting or storing them. This sampling of the polynomial beyond the necessary value makes the polynomial overdetermined (over-defined). When the receiver correctly receives enough points, it can recover the original polynomial, even if many points on the received polynomial are corrupted by noise. Reed-Solomon codes are now frequently used in distributed storage systems to reduce the cost of data storage while ensuring data reliability.

[0088] Figure 1 A flowchart of an encoding method provided in an embodiment of the present application is shown. The method is implemented based on a pre-constructed first erasure code and a plurality of second erasure codes. The method is applied to any device with computing capabilities, such as Figure 1 As shown, the method comprises the following steps:

[0089] Step S110: generating a plurality of data slices according to the data to be encoded, and dividing the plurality of data slices into a plurality of data slice groups.

[0090] The data to be encoded is divided into a plurality of data slices of the same size, and the plurality of data slices are divided into a plurality of data slice groups, wherein the data slices contained in each data slice group are non-repetitive.

[0091] Step S120: globally encode multiple data shards using the first erasure code to generate global check shards.

[0092] The first erasure code is also a global erasure code, which is generated in advance by a preset algorithm. The first erasure code is used to globally encode all data shards to obtain global verification shards.

[0093] Step S130, locally encode at least a portion of the data shard groups in the plurality of data shard groups using a plurality of second erasure codes to generate a portion of local check shards; and generate another portion of local check shards based on the portion of local check shards.

[0094] The second erasure code is a local erasure code, and multiple local erasure codes are generated in advance by a preset coding algorithm. For at least a part of the multiple data shard groups, the corresponding second erasure code is used for local encoding to generate a part of local check shards; and then another part of local check shards is generated based on the generated part of local check shards, and another part of local check shards is generated specifically based on the part of local check shards and the global check shards.

[0095] For example, there are 2 data shard groups in total. The first erasure code is used to locally encode the first data shard group to generate the first local check shard, and then the second local check shard is generated based on the first local check shard and the global check shard.

[0096] According to the encoding method of the embodiment of the present application, by constructing a global erasure code and multiple local erasure codes, the global erasure code is used to encode multiple data slices to obtain multiple global check slices, and the local erasure code is used to encode part of the data slice group to obtain part of the local check data, and the part of the local check data is used to calculate to obtain another part of the local check data. In the above manner, the efficiency of the encoding process can be improved, and it also helps to improve the probability of successful decoding.

[0097] Figure 2 A flowchart of an encoding method provided by another embodiment of the present application is shown. The method is implemented based on a pre-constructed first erasure code and a plurality of second erasure codes. The method is applied to any device with computing capability. Figure 2 As shown, the method comprises the following steps:

[0098] Step S210: construct a first erasure code and a plurality of second erasure codes.

[0099] First, the strip coding method of LRC with coding parameters (m, l, g) is introduced as follows:

[0100] Based on a Vandermonde determinant of [m+g+1, m], an elementary row transformation is performed to transform the square matrix of [m, m] in the Vandermonde determinant into a unit matrix, and the encoding matrix V is obtained after the transformation; wherein m represents the number of data slices, g represents the number of global check slices, and l represents the number of local check slices;

[0101] Get the encoding vector of g global check slices, i.e. g i =V m+i+1 , where 1≤i≤g, V m+i+1 is the m+i+1th row of the encoding matrix;

[0102] The local check code vector is obtained from the m+1th row of the code matrix V. Specifically, the m+1th row of the code matrix V is split according to the number of local check groups, where the i-th local check code vector l i =V m+1 [i*(m / l):(i+1)(m / l)], 1≤i≤l;

[0103] The data slices are combined with the local encoding vector l i and the global encoding vector g i Based on GF(2 8) are multiplied to obtain local verification data and global verification data.

[0104] Because the XOR result of all local checksum data is equal to the sum of all data slices and V m+1 Based on this, in an embodiment of the present application, a standard RS code (n=m+g+1, k=m) is constructed as the first erasure code and l standard RS codes (n=m / l+1, k=m / l) are constructed as the second erasure code for fast encoding, where n represents the length of the code, k represents the length of the user data, and nk represents the number of check blocks.

[0105] Step S220, split the data to be encoded into m sub-data to be encoded, write the m sub-data to be encoded into m data slices respectively, and divide the multiple data slices into multiple data slice groups.

[0106] The data to be encoded is split into m sub-data to be encoded of the same size, and the m sub-data to be encoded are written into m data slices respectively, and the m data slices are divided into l data slice groups.

[0107] Step S230: Use the first erasure code to globally encode the m data shards to obtain g global verification data, and write the g-1 global verification data into the g-1 global verification shards respectively.

[0108] Among them, each global verification data is a linear combination of the data stored in m data shards.

[0109] Step S240: write one global check data other than g-1 global check data as intermediate data into the lth local check slice.

[0110] Step S250, for the i-th data shard group, where 0<i<1, locally encode the i-th data shard group using the i-th second erasure code to obtain the i-th local verification data, and write the i-th local verification data into the i-th local verification data shard.

[0111] Step S260, for the lth data shard group, obtain the lth local check data according to the data written into the 1st to lth local check shards, and write the lth local check data into the lth local check shard.

[0112] At this time, what is written into the 1st to l-1th local check slices is the local check data encoded by the second erasure code, and what is written into the lth local check slice is the intermediate data encoded by the first erasure code. An XOR operation is performed on the l-1th local check data and the intermediate data to obtain the lth local check data, which is written into the lth local check slice.

[0113] At this point, the encoding obtains m data shards, l local check shards, and g global check shards, where a global check shard and a local check shard share a memory address. Further, the m data shards, l local check shards, and g global check shards are written into the storage node to facilitate the management and reading of each shard.

[0114] The following example illustrates the implementation of each step of the method of this embodiment, assuming that the encoding parameters are: (m=4, l=2, g=3), m represents the number of data shards, l=2 represents the number of data shard groups, and g is the number of shards of the global check shard.

[0115] In step S220, four data slices m1, m2, m3, and m4 are generated according to the data to be encoded, and the four data slices are divided into two data slice groups, where the first data slice group includes data slices m1 and m2, and the second data slice group includes data slices m3 and m4.

[0116] In step S230, the first erasure code is used to globally encode the four data shards to generate three global verification data, two of which are written into two global verification shards respectively to obtain global verification shards g2 and g3.

[0117] In step S240, the remaining 1 global check data is written into the second local check slice l2 as intermediate data.

[0118] In step S250, for the first data shard group, the first second erasure code is used to locally encode the first data shard group to obtain local verification data corresponding to the first data shard group, and write it into the first local verification shard l1.

[0119] Through the aforementioned steps, the local check slice l1 stores the local check data obtained by local encoding according to the first data slice group, and the local check slice l2 stores the intermediate data obtained by global encoding according to all data slices.

[0120] Then in step S260, the local check data corresponding to the data stored in m3 and m4 is calculated based on the intermediate data stored in the local check slice l2 and the local check data stored in the local check slice l1, without using the data stored in m3 and m4 to calculate the local check data. The local check data calculated here is the correct data written to the local check slice l2, which is stored in the second local check slice l2. It can be seen that in the method of this embodiment, a global check data and local check data share memory, which can save memory space and improve encoding speed.

[0121] At this point, the encoding has obtained the following shards: data shards (m1, m2, m3, m4), global check shards (g1, g2, g3) and local check shards (l1, l2), among which the global check data stored in the global check shards g1, g2 and g3 are all linear combinations of the data stored in the data shards m1, m2, m3 and m4, the local check data stored in the local check shard l1 is the linear combination of the data stored in the data shards m1 and m2, and the local check data stored in the local check shard l2 is the linear combination of the data stored in the data shards m3 and m4.

[0122] Figure 3 A schematic diagram of an encoding method provided by another embodiment of the present application is shown. Figure 3 As shown, shard 31, shard 32, shard 33, and shard 34 represent four data shards respectively; shard 35 represents the first local check shard, and the stored local check data is obtained by encoding the data stored in shard 31 and shard 32 using the first local erasure code, that is, the local check data stored in shard 35 is a linear combination of the data stored in shard 31 and shard 32; shard 36 represents the second local check shard, and the stored local check data is obtained by encoding the data stored in shard 33 and shard 32 using the second local erasure code. That is, the data stored in shard 36 is a linear combination of the data stored in shard 33 and shard 34; shard 37, shard 38 and shard 39 represent three global check shards respectively, and the global check data stored in the three local check shards are obtained by encoding the data stored in shard 31, shard 32, shard 33 and shard 34 through a global erasure code, and the global check data is a linear combination of the data stored in shard 31, shard 32, shard 33 and shard 34.

[0123] According to the encoding method provided in this embodiment, multiple data slices are encoded using global erasure codes to obtain multiple global check slices, and some data slice groups are encoded using local erasure codes to obtain some local check data. The global check data and the part of the local check data are calculated to obtain another part of the local check data. This can achieve fast encoding and help improve the probability of decoding. In addition, by allowing one of the global check data and one of the local check data to share memory, memory space can be saved.

[0124] Figure 4 A flowchart of a decoding method provided by another embodiment of the present application is shown, and the method is implemented based on a plurality of pre-constructed first erasure codes and a plurality of second erasure codes, such as Figure 4 As shown, the method comprises the following steps:

[0125] Step S410, obtain the first shard sequence number list to be loaded and the second shard sequence number list to be repaired, load the first shard according to the first shard sequence number list; use the first erasure code to globally repair the data shards and / or global check shards contained in the second shard according to the data shards and / or global check shards contained in the first shard, and obtain repaired data shards and / or global check shards; update the first shard sequence number list and the second shard sequence number list.

[0126] In the method of this embodiment, two sequence number lists are maintained, including a first slice sequence number list to be loaded and a second slice sequence number list to be repaired. The first slice sequence number list to be loaded contains the sequence numbers of undamaged slices, and the second slice sequence number list to be repaired contains the sequence numbers of damaged slices.

[0127] The first erasure code is also a global erasure code. The construction method of the first erasure code is specifically described in the aforementioned embodiment and will not be repeated here. According to the first shard sequence number list, undamaged data shards and / or global check shards are loaded, and global erasure codes are used to globally repair damaged data shards and / or global check shards based on the loaded undamaged data shards and / or global check shards to obtain repaired data shards and / or global check shards, and update the two sequence number lists according to the shard repair results, specifically deleting the sequence number of the repaired data shard and / or global check shard from the second shard sequence number list, and using the repaired data shard and / or global check shard as the first shard to update its sequence number to the first shard sequence number list, so that the repaired shard can be used in subsequent repair processes.

[0128] Specifically, when the total number of data shards and / or global check shards included in the first shard meets the repair condition, the data shards and / or global check shards included in the second shard are globally repaired according to the data shards and / or global check shards included in the first shard. When the total number of undamaged designated shards (including data shards and / or global check shards) exceeds a preset value, the global repair can be completed, and the damaged data shards and / or global check shards are globally repaired according to the loaded data shards and / or global check shards.

[0129] Step S420, traverse all data shard groups, if any data shard group has a local check shard included in the second shard, use the second erasure code of the data shard group to partially repair the local check shard according to the first shard, and obtain a repaired local check shard; update the first shard sequence number list and the second shard sequence number list.

[0130] The second erasure code is also a local erasure code. The construction method of the second erasure code is specifically described in the above embodiment and will not be repeated here.

[0131] Through the aforementioned global repair process, the damaged data shards can be repaired. Traverse all data shard groups. If any data shard group has a damaged local check shard, use the local erasure code corresponding to the data shard group to locally repair the local check shard based on the undamaged data shard (including the data shard repaired by the global repair), that is, use the local erasure code to locally encode the data shard to obtain the repaired local check shard, and update the two sequence number lists according to the shard repair results. Specifically, delete the sequence number of the repaired local check shard from the second shard sequence number list and update it to the first shard sequence number list.

[0132] Through the decoding method provided in the embodiment of the present application, by first performing global repair and then performing local repair, the probability of successful data repair can be improved and the performance of data repair can be improved.

[0133] Figure 5 A flowchart of a decoding method provided by another embodiment of the present application is shown. When performing data repair processing, the prior art reads the data of all undamaged fragments into the memory to repair the damaged fragments. However, in fact, only the data of some undamaged fragments are needed for successful repair. Therefore, it is unnecessary to load the data of all undamaged fragments. Instead, it will cause a waste of resources and slow data repair. Based on this, the method of the embodiment of the present application provides a specific implementation method for fragment selection during the repair process.

[0134] like Figure 5 As shown, the method comprises the following steps:

[0135] Step S510, pre-construct a decision tree, obtain a list of second shard sequence numbers to be repaired, traverse the decision tree according to the second shard sequence number list, obtain a list of first shard sequence numbers to be loaded, and load the first shard according to the first shard sequence number list.

[0136] In the embodiment of the present application, a shard selection algorithm is provided for determining a shard selection strategy, and a shard selection strategy that can be successfully decoded is obtained by constructing a decision tree, and a decision tree is constructed in advance according to all feasible repair solutions. The damaged shard is obtained, and the decision tree is traversed to find a shard selection strategy that can be successfully decoded, and a list of first shard sequence numbers to be loaded is obtained, and then the shard data is read from the corresponding storage node and loaded into the memory.

[0137] Furthermore, if there is at least one first shard that has not been loaded successfully, the second shard sequence number list is updated according to the sequence number of at least one first shard that has not been loaded successfully, the decision tree is traversed according to the updated second shard sequence number list to obtain the updated first shard sequence number list, and the first shard is loaded according to the updated first shard sequence number list; this step is repeated until all the first shards corresponding to the updated first shard sequence number list are loaded successfully. If the first shard to be loaded is not loaded successfully, indicating that the first shard is damaged, its sequence number is added to the second shard sequence number list to be repaired, and according to the updated second shard sequence number list, the decision tree is retraversed to obtain the updated first shard sequence number list, and then the first shard corresponding to the updated first shard sequence number list is loaded; the above steps are repeated until all the first shards corresponding to the updated first shard sequence number list are loaded successfully. In the above manner, the strategy that can repair the damaged shard is first determined, and then the shard data is loaded into the memory, which can reduce the shard data loaded into the memory and improve the efficiency of data shard repair.

[0138] Figure 6 A schematic diagram of a decision tree in another embodiment of the present application is shown. Figure 6 As shown, the decision tree is a multi-branch tree. When traversing to a leaf node, judgment is made until the branch traversal is completed. Finally, the final shard selection strategy is determined based on the traversal result. For example, the output shard strategy is [0, 1, 2, 3], which means that data repair can be completed through shard 0, shard 1, shard 2, and shard 3.

[0139] Step S520: If it is determined according to the first shard sequence number list and the second shard sequence number list that any data shard group has only one data shard or local check shard included in the second shard, then the one data shard or local check shard included in the second shard is partially repaired to obtain a repaired data shard or local check shard; the first shard sequence number list and the second shard sequence number list are updated.

[0140] If one and only one data slice or local check slice of a data slice group is damaged, then the damaged data slice or local check slice is partially repaired according to the undamaged data slices and / or local check slices in the data slice group to obtain the repaired data slice or local check slice. If only one data slice of a data slice group is damaged, then the damaged data slice is repaired according to other data slices and local check slices. If only one local check slice of a data slice group is damaged, then the damaged local check slice is repaired according to all data slices. The local repair is performed before the global repair is performed to improve the efficiency of data repair. At the same time, the sequence number of the repaired data slice or local check slice is deleted from the second slice sequence number list and added to the first slice sequence number list, so that the slices recorded in the first slice sequence number list (including the repaired data slices and / or global check slices) can be used in other repair processes later.

[0141] Step S530: If it is determined according to the first fragment sequence number list that the 1st to the lth local check fragments are all included in the first fragment, the intermediate data is calculated according to the local check data in the 1st to the lth local check fragments, and the intermediate data is written into the lth local check fragment.

[0142] Wherein, m is the number of data shards, and l is the number of data shard groups. According to the first shard sequence number list at this time, if it is determined that the 1st to the lth local check shards are not damaged, that is, all local check shards are not damaged, then the intermediate data is calculated based on the local check data in the 1st to the lth local check shards, and the intermediate data is written into the lth local check shard.

[0143] Specifically, the local check data of the 1st to the lth local check slices are loaded from the corresponding storage node, and the local check data of the 1st to the lth local check slices are XORed to obtain intermediate data, and the intermediate data is stored in the lth local check slice, wherein the intermediate data obtained after the XOR operation on the local check data of the 1st to the lth local check slices must be linearly related to the m sub-data to be encoded, so the intermediate data can be used as a global check data.

[0144] Step S540, using the first erasure code, globally repair the data shards and / or global check shards contained in the second shard according to the lth local check shard, the data shards contained in the first shard, and / or the global check shards, to obtain repaired data shards and / or global check shards, and update the first shard sequence number list and the second shard sequence number list.

[0145] Using global erasure codes, a global repair process is performed based on the intermediate data contained in the lth local check shard, the data shard contained in the first shard (i.e., the shard corresponding to the first shard sequence number list), and / or the data contained in the global check shard. And after the global repair is completed, the sequence numbers of the repaired data shards and / or global check shards are deleted from the second shard sequence number list, and at the same time, added to the first shard sequence number list, so that the shards recorded in the first shard sequence number list (including the repaired data shards and / or global check shards) can be used in other repair processes later.

[0146] It should be noted that, when the 1st to the lth local check fragments are all included in the first fragment, the global repair is completed with the data stored in at least m-1 fragments.

[0147] For example, local check slice l1 and local check slice l2 are both undamaged, the local check data in local check slice l1 is obtained by encoding the data in data slices m1 and m2, and the local check data in local check slice l2 is obtained by encoding the data in data slices m3 and m4, so the intermediate data obtained by XORing the local check data of local check slice l1 and local check slice l2 is related to the data in data slices m1, m2, m3 and m4, so it is used as a global check data; then, according to the data of any three undamaged slices (including data slices and / or global check slices) that have been loaded and the intermediate data, the global repair can be completed. For the LRC encoding method of (m=4, l=2, g=3), if the local check slices are all undamaged, the global repair can be completed by loading the data of at least three undamaged slices (including data slices and / or global check slices), and the method of the embodiment of the present application greatly improves the probability of successful data decoding.

[0148] Step S550, calculate the local check data of the lth data shard group, and write the local check data into the lth local check shard.

[0149] Through the above steps, all data shards can be repaired. For the lth data shard group, intermediate data was previously stored in its local check shard. After all data shards are repaired, the corresponding second erasure code is used to recalculate the local check data of the lth data shard group, and the calculated local check data is written into the lth local check shard.

[0150] Step S560, traverse the 1st to l-1th data shard groups. If any data shard group has a local check shard contained in the second shard, use the second erasure code of the data shard group to partially repair the local check shard according to the first shard to obtain a repaired local check shard, and update the first shard sequence number list and the second shard sequence number list.

[0151] Traverse the 1st to l-1th data shard groups. If any data shard group still has a damaged local check shard, use the local erasure code corresponding to the data shard group to partially repair the local check shard according to the data shard to obtain a repaired local check shard. At the same time, delete the sequence number of the repaired local check shard from the second shard sequence number list and add it to the first shard sequence number list.

[0152] In the embodiment of the present invention, the above step S530 is an optional step, and step S530 is performed based on the specific situation that all local check slices are not damaged before the global repair (i.e., step S540) (including the situation that they have been repaired in step S520). It should be understood that the above specific situation is a special situation in the implementation process of the embodiment of the present invention. If this specific situation occurs, there is no need to perform step S560, that is, there is no need to perform local repair on the local check slices.

[0153] The following describes the implementation of the decoding method of the embodiment of the present application with reference to the accompanying drawings. Figure 7 A schematic diagram of a decoding method in another embodiment of the present application is shown, wherein the first dotted line enclosed area includes slice 31 (representing data slice m1), slice 32 (representing data slice m2) and slice 35 (representing local check slice l1), the second dotted line enclosed area includes slice 33 (representing data slice m3), slice 34 (representing data slice m4) and slice 36 (representing local check slice l2), the dotted line enclosed area includes slice 32, slice 33, slice 36 and slice 38 (representing global check slice g2) and slice 39 (representing global check slice g3), and the decoding process is as follows:

[0154] First, obtain the second fragment sequence number list to be repaired, the second fragment sequence number list includes: fragment 31, fragment 34, fragment 35 and fragment 37; determine the first fragment sequence number list, the first fragment sequence number list includes: fragment 32, fragment 33, fragment 36, fragment 38, fragment 39;

[0155] Secondly, perform partial repair based on slices 33 and 36 to repair the damaged slice 34;

[0156] Then, a global repair is performed based on slice 32, slice 33, slice 36, slice 38, slice 39 and the repaired slice 34, thereby repairing slice 31 and slice 37;

[0157] Finally, slice 35 is partially repaired using slice 31 and slice 32.

[0158] In another embodiment of the present application, in this embodiment, a shard selection strategy that can be successfully decoded is obtained based on the currently loaded shards and the currently known damaged shards. The input of the shard selection strategy algorithm is: availiableShards[]int and brokensShards[]int, which respectively represent the list of shard sequence numbers that have been loaded and the list of shard sequence numbers that are known to be damaged; the output of the algorithm is: nextLoadShards[]int, which represents the list of shard sequence numbers that can be decoded with the optimal bandwidth.

[0159] The specific process of the algorithm is as follows:

[0160] Step 1: Check whether availiableShards and brokenShards are reasonable.

[0161] Step 2: If the number of damaged shards in brokensShards is 1, and there are no loaded shards in availiableShards, the returned shard number list includes: the shard numbers of all other shards in the local group corresponding to the damaged shard except the damaged shard. In this case, it is equivalent to performing a local repair and the algorithm ends.

[0162] Step 3: Construct all possible solutions of the optimal repair plan into a multi-branch decision tree.

[0163] Step 4: According to the current availiableShards and brokenShards, traverse the decision tree according to the pre-order traversal of the multi-branch tree. When traversing to the leaf node, determine whether the current lexicographic order already contains all availiableShards and does not contain brokenShards. If the conditions are met, return directly. Otherwise, continue traversing.

[0164] Step 5: Return to nextLoadShards or return an error that cannot be optimally repaired.

[0165] In the implementation of object storage systems, the impact of repair bandwidth on disk repair is considered, and the general parameters are maintained at a relatively small and reasonable value.

[0166] The following is a specific example to illustrate the specific decoding process based on the fragment selection algorithm. In this example, 5 fragments are damaged.

[0167] Step 1: Through the (m=4, l=2, g=3)-LRC decoding algorithm, traverse all possible combinations of damaged 5 slices, and write all successfully decoded solutions into the configuration file;

[0168] Step 2: Load all the solutions that can be successfully decoded from the configuration file and construct a multi-branch decision tree. In this case, there are 97 solutions that can be successfully decoded;

[0169] Step 3: Assuming that the fragment {2, 3, 5} is damaged, the codec service selects a strategy through the fragment selection algorithm and obtains the fragment selection scheme for optimal decoding: {0, 1, 6, 7};

[0170] Step 4: The codec service attempts to concurrently read data from the storage nodes corresponding to shards {0, 1, 6, 7}. The data of shards {0, 1, 7} is successfully read, but shard 6 cannot be connected and the read fails.

[0171] Step 5: The codec service sets availiableShards to {0, 1, 7} and brokenShards to {2, 3, 5, 6}, calls the shard selection algorithm again to select a strategy, and obtains a new shard loading solution {8}.

[0172] Step 6: The codec service reads data from the storage node corresponding to shard 8 and loads it successfully.

[0173] Step 7: The codec service performs global decoding based on the successfully loaded segments {0, 1, 7, 8} to repair all damaged segments.

[0174] According to the decoding method provided in this embodiment, data can be encoded and decoded in units of stripes. When there are damaged slices, a pre-constructed decision tree can be used to quickly decide whether it is possible to repair and how to load other slices for optimal repair, thereby improving the efficiency of decoding; secondly, when all local check slices are not damaged, an intermediate data is obtained by calculating the data stored in all local check slices as global check data, and global repair is performed using the global check data, which can reduce the number of slices required to be loaded for repair, thereby greatly improving the probability of data repair; in addition, by first performing local repair when a data group has only one data slice or local check slice, the efficiency of data recovery can be improved.

[0175] Figure 8The structural diagram of the codec system in another embodiment of the present application is shown, in which there are two associated services, one is the LRC codec service, and the other is the storage node service. The LRC codec service has the following characteristics: high availability, that is, multiple LRC codec services provide codec calculations at the same time; stateless, that is, there is no association between codec services, and the failure of individual codec services will not affect the normal operation of other codec services; memory controllable, that is, according to the configuration of the deployment node or container, according to the control of the number of concurrent codec strips, precise control of memory is achieved; node-level failures are recoverable, that is, each stripe of the code does not have more than 2 slices of data stored in the same node. The storage node service has the following characteristics: management and allocation of fixed blocks, that is, each slice of the LRC code is stored on a designated storage node, and the storage node is responsible for the selection and organization of the storage location; there is no dependency between nodes.

[0176] Fig. 9 FIG. 4 shows a schematic diagram of the structure of an encoding device provided in another embodiment of the present application. Fig. 9 As shown, the device comprises:

[0177] A division module 91, adapted to generate a plurality of data slices according to the data to be encoded, and divide the plurality of data slices into a plurality of data slice groups;

[0178] A global encoding module 92, adapted to globally encode the plurality of data shards using a first erasure code to generate a global check shard;

[0179] The local encoding module 93 is suitable for locally encoding at least a part of the multiple data shard groups using multiple second erasure codes to generate a part of local check shards; and generating another part of local check shards based on the part of local check shards.

[0180] In an optional manner, the device also includes a storage module, which is suitable for storing multiple data shards, local check shards and global check shards in different storage nodes respectively.

[0181] In an optional manner, the division module 91 is further adapted to: split the data to be encoded into m sub-data to be encoded, and write the m sub-data to be encoded into m data slices respectively.

[0182] In an optional manner, the global encoding module 92 is further adapted to: globally encode the m data shards using the first erasure code to obtain g global check data, and write the g-1 global check data into the g-1 global check shards respectively;

[0183] The device also includes: a data writing module, adapted to write one global check data other than g-1 global check data as intermediate data into the lth local check shard, where l is the number of data shard groups.

[0184] In an optional manner, the local encoding module 93 is further adapted to: for the i-th data shard group, 0<i<l, locally encode the i-th data shard group using the i-th second erasure code to obtain the i-th local verification data, and write the i-th local verification data into the i-th local verification shard.

[0185] In an optional manner, the local encoding module 93 is further adapted to:

[0186] For the lth data shard group, the lth local verification data is obtained according to the data written into the 1st to lth local verification shards, and the lth local verification data is written into the lth local verification data shard.

[0187] Fig.10 A schematic diagram of the structure of a decoding device provided by another embodiment of the present application is shown. Fig. 9 As shown, the device comprises:

[0188] The global repair module 101 is adapted to obtain a first shard sequence number list to be loaded and a second shard sequence number list to be repaired, and load the first shard according to the first shard sequence number list; use the first erasure code to perform global repair on the data shards and / or global check shards included in the second shard according to the data shards and / or global check shards included in the first shard, to obtain repaired data shards and / or global check shards; and update the first shard sequence number list and the second shard sequence number list;

[0189] The local repair module 102 is suitable for traversing all data shard groups. If any data shard group has a local check shard contained in the second shard, the local check shard is locally repaired according to the first shard using the second erasure code of the data shard group to obtain a repaired local check shard; and the first shard sequence number list and the second shard sequence number list are updated.

[0190] In an optional manner, the global repair module 101 is further adapted to:

[0191] When the total number of data shards and / or global check shards contained in the first shard meets the repair condition, the data shards and / or global check shards contained in the second shard are globally repaired according to the data shards and / or global check shards contained in the first shard.

[0192] In an optional manner, the local repair module 102 is further adapted to: before globally repairing the data shards and / or global check shards included in the second shard using the first erasure code according to the data shards and / or global check shards included in the first shard, if it is determined according to the first shard sequence number list and the second shard sequence number list that any data shard group has only one data shard or local check shard included in the second shard, then locally repairing the one data shard or local check shard included in the second shard to obtain a repaired data shard or local check shard; and updating the first shard sequence number list and the second shard sequence number list.

[0193] In an optional manner, the global repair module 101 is further adapted to: if it is determined according to the first shard sequence number list that the 1st to the lth local check shards are all included in the first shard, then intermediate data is calculated based on the local check data in the 1st to the lth local check shards, and the intermediate data is written into the lth local check shard; using the first erasure code, globally repairing the data shards and / or the global check shards included in the second shard according to the lth local check shard, the data shards and / or the global check shards included in the first shard; wherein m is the number of data shards, and l is the number of data shard groups.

[0194] In an optional manner, the device further includes: a local calculation module, adapted to calculate local verification data of the lth data shard group, and write the local verification data into the lth local verification shard.

[0195] In an optional manner, the device further comprises: a construction module, adapted to pre-construct a decision tree;

[0196] The global repair module 101 is further adapted to: obtain a list of second shard sequence numbers to be repaired; and traverse the decision tree according to the list of second shard sequence numbers to obtain a list of first shard sequence numbers to be loaded.

[0197] In an optional manner, the global repair module 101 is further adapted to: if there is at least one first fragment that has not been loaded successfully, updating the second fragment sequence number list according to the sequence number of at least one first fragment that has not been loaded successfully, traversing the decision tree according to the updated second fragment sequence number list to obtain an updated first fragment sequence number list, and loading the first fragment according to the updated first fragment sequence number list; repeating this step until all the first fragments corresponding to the updated first fragment sequence number list are loaded successfully.

[0198] An embodiment of the present application provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction, and the computer executable instruction can execute the encoding method and / or decoding method in any of the above method embodiments.

[0199] Fig.11 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0200] like Fig.11 As shown, the computing device may include: a processor, a communications interface, a memory, and a communication bus.

[0201] Wherein: the processor, the communication interface, and the memory communicate with each other via a communication bus. The communication interface is used to communicate with other devices such as a client or other server network elements. The processor is used to execute a program, and specifically can execute the relevant steps in the above-mentioned encoding method and / or decoding method embodiment for a computing device.

[0202] Specifically, the program may include program codes including computer operation instructions.

[0203] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0204] The memory is used to store programs. The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0205] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0206] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0207] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0208] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0209] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0210] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0211] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A coding method, the method being implemented based on a pre-constructed first erasure code and a plurality of second erasure codes, comprising: Generate multiple data slices according to the data to be encoded, and divide the multiple data slices into multiple data slice groups; Using a first erasure code, globally encode the multiple data shards to generate a global check shard; Locally encoding at least a portion of the data shard groups in the plurality of data shard groups by using a plurality of second erasure codes to generate a portion of local check shards; generating another portion of local check shards according to the portion of local check shards and the intermediate data; Among them, the intermediate data is a global verification data obtained through global encoding, the global verification data stored in the global verification shard is associated with the data stored in all data shards, and the local verification data stored in the local verification shard is associated with the data stored in each data shard in the data shard group.

2. The method according to claim 1, further comprising: Multiple data shards, local check shards, and global check shards are stored in different storage nodes respectively.

3. The method according to claim 1 or 2, wherein generating a plurality of data slices according to the data to be encoded further comprises: The data to be encoded is split into m sub-data to be encoded, and the m sub-data to be encoded are written into m data slices respectively.

4. According to the method of claim 3, the step of using the first erasure code to globally encode the plurality of data shards to generate global check shards further comprises: Use the first erasure code to globally encode the m data shards to obtain g global verification data, and write the g-1 global verification data into the g-1 global verification shards respectively; The method further includes: writing one global check data other than the g-1 global check data as intermediate data into the first l There are local check shards, where l The number of data shard groups.

5. According to the method of claim 4, the using of the plurality of second erasure codes to locally encode at least a portion of the plurality of data shard groups to generate a portion of the local check shards further comprises: For the i-th data shard group, 0<i< l , use the i-th second erasure code to locally encode the i-th data shard group, obtain the i-th local verification data, and write the i-th local verification data into the i-th local verification shard.

6. The method according to claim 5, wherein generating another part of the local check slices according to the part of the local check slices and the intermediate data further comprises: For the l Data shard groups, based on the 1st to the l The data written by the local verification shard is obtained l Partial verification data, l The local verification data is written into l A local checksum shard.

7. A decoding method, the method being implemented based on a pre-constructed first erasure code and a plurality of second erasure codes, comprising: Obtain a first shard sequence number list to be loaded and a second shard sequence number list to be repaired, and load the first shard according to the first shard sequence number list; Using the first erasure code, globally repair the data shards and / or global check shards included in the second shard according to the data shards and / or global check shards included in the first shard, to obtain repaired data shards and / or global check shards; Update the first fragment sequence number list and the second fragment sequence number list; Traverse all data shard groups, if any data shard group has a local check shard included in the second shard, use the second erasure code of the data shard group to partially repair the local check shard according to the first shard, and obtain a repaired local check shard; update the first shard sequence number list and the second shard sequence number list; The global check slice and the local check slice are generated according to the encoding method described in any one of claims 1-6.

8. According to the method of claim 7, the method of using the first erasure code to globally repair the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard comprises: When the total number of data shards and / or global check shards contained in the first shard meets the repair condition, the data shards and / or global check shards contained in the second shard are globally repaired according to the data shards and / or global check shards contained in the first shard.

9. The method according to claim 7, before using the first erasure code to globally repair the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard, the method further comprises: If it is determined based on the first shard sequence number list and the second shard sequence number list that any data shard group has one and only one data shard or local check shard included in the second shard, then the one data shard or local check shard included in the second shard is partially repaired to obtain a repaired data shard or local check shard; and the first shard sequence number list and the second shard sequence number list are updated.

10. According to any one of claims 7 to 9, before using the first erasure code to globally repair the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard, the method further comprises: If the first to the first fragment sequence number list is used to determine l The local check fragments are all included in the first fragment, then according to the 1st to the l The local check data in the local check slice is calculated to obtain the intermediate data, and the intermediate data is written into the l In a local checksum shard; The method of using the first erasure code to globally repair the data shards and / or the global check shards included in the second shard according to the data shards and / or the global check shards included in the first shard is specifically as follows: Using the first erasure code, according to the first l The local check shard, the data shards included in the first shard and / or the global check shard perform global repair on the data shards and / or the global check shards included in the second shard; Where m is the number of data shards, l The number of data shard groups.

11. The method according to claim 10, after performing global repair, the method further comprises: Calculate the l The local checksum data of the data shard group is written into the l A local checksum shard.

12. The method according to claim 7, further comprising: Pre-construct decision trees; The obtaining of the first fragment sequence number list to be loaded and the second fragment sequence number list to be repaired further comprises: Get the list of second shard sequence numbers to be repaired; The decision tree is traversed according to the second fragment sequence number list to obtain the first fragment sequence number list to be loaded.

13. The method according to claim 12, after loading the first fragment according to the first fragment sequence number list, the method further comprises: If there is at least one first shard that has not been loaded successfully, update the second shard sequence number list according to the sequence number of at least one first shard that has not been loaded successfully, traverse the decision tree according to the updated second shard sequence number list, obtain an updated first shard sequence number list, and load the first shard according to the updated first shard sequence number list; repeat this step until all the first shards corresponding to the updated first shard sequence number list are loaded successfully.

14. An encoding device, comprising: A partitioning module, adapted to generate a plurality of data slices according to the data to be encoded, and to partition the plurality of data slices into a plurality of data slice groups; A global encoding module, adapted to globally encode the plurality of data shards using a first erasure code to generate a global check shard; A local encoding module, adapted to locally encode at least a portion of the data shard groups in the plurality of data shard groups using a plurality of second erasure codes to generate a portion of local check shards; and generate another portion of local check shards according to the portion of local check shards and intermediate data; Among them, the intermediate data is a global verification data obtained through global encoding, the global verification data stored in the global verification shard is associated with the data stored in all data shards, and the local verification data stored in the local verification shard is associated with the data stored in each data shard in the data shard group.

15. A decoding device, comprising: A global repair module, adapted to obtain a first fragment sequence number list to be loaded and a second fragment sequence number list to be repaired, and load the first fragment according to the first fragment sequence number list; Using the first erasure code, globally repair the data shards and / or global check shards included in the second shard according to the data shards and / or global check shards included in the first shard, to obtain repaired data shards and / or global check shards; Update the first fragment sequence number list and the second fragment sequence number list; The local repair module is adapted to traverse all data shard groups, and if any data shard group has a local check shard included in the second shard, locally repair the local check shard according to the first shard using the second erasure code of the data shard group to obtain a repaired local check shard; and update the first shard sequence number list and the second shard sequence number list; The global check slice and the local check slice are generated according to the encoding method described in any one of claims 1-6.

16. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the encoding method according to any one of claims 1-6 and / or an operation corresponding to the decoding method according to any one of claims 7-13.

17. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, wherein the executable instruction enables a processor to perform operations corresponding to the encoding method as described in any one of claims 1 to 6 and / or operations corresponding to the decoding method as described in any one of claims 7 to 13.

18. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the encoding method according to any one of claims 1 to 6 and / or operations corresponding to the decoding method according to any one of claims 7 to 13.

Images