An encoding / decoding method and system

Abstract

The embodiment of the specification provides an encoding method, comprising: determining a shard data group containing m shard data, the m shard data being stored by one-to-one corresponding m user data storage disks; the m shard data corresponding to one-to-one m information bit sub-matrices split from a user data sub-matrix, the user data sub-matrix belonging to a check matrix used for error correction encoding and corresponding to information bits of the check matrix, and the check matrix further containing a check bit sub-matrix corresponding to check bits; obtaining m sub-feature vectors corresponding to the m shard data respectively, and summing to obtain a feature vector corresponding to the shard data group; a sub-feature vector corresponding to any shard data being generated based on any shard data and an information bit sub-matrix corresponding to the shard data; and generating check data corresponding to the shard data group based on the feature vector corresponding to the shard data group and an inverse matrix of the check bit sub-matrix, the check data and the m shard data jointly constituting an error correction code code word corresponding to the shard data group.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to an encoding / decoding method and system. Background Technology

[0002] In related technologies, to ensure high availability and data stability while reducing costs, erasure coding storage schemes are commonly used to protect data, replacing traditional multi-backup storage schemes. This involves splitting the original data into m original data blocks, and simultaneously adding k check data blocks through erasure coding. These m+k data blocks are then stored in different locations. Because there is a mathematical relationship between the original data blocks and the check data blocks, these m+k data blocks can tolerate the loss of at most k data blocks while still allowing the original data to be recovered.

[0003] Figure 1 This diagram illustrates a system architecture for a storage cluster employing an erasure coding storage scheme. Each server in the storage cluster is independently equipped with an erasure coding encoder / decoder (implemented in software or hardware) to independently perform erasure coding encoding and decoding. Additionally, each storage disk has its own error correction code encoder / decoder to generate error correction codewords, thereby improving its fault tolerance for data retrieval. In this scenario, the erasure coding storage scheme and the error correction code storage scheme operate independently, providing multi-disk data protection capabilities and single-disk data protection capabilities respectively, to jointly achieve a collaborative data protection scenario. Specifically, for the original data requiring protection, the erasure coding encoder / decoder on the server first performs erasure coding to generate corresponding erasure codewords. Then, these erasure codewords are split into multiple data slices (forming an erasure coding protection group) and stored separately on different storage disks on different servers. For any storage disk intended to store any data slice, it also generates the corresponding error correction codeword for that data slice through its built-in error correction code encoder / decoder, thus ensuring that the original data is simultaneously protected by erasure coding between storage disks and by error correction coding within each storage disk.

[0004] However, current erasure coding storage solutions have at least the following technical problems: 1. They require separate configuration of erasure coding encoders / decoders on the server, setting up dedicated memory space for encoding or decoding erasure codes, and independently constructing and optimizing the code structure of erasure codes, which increases the cost of hardware and software, server computing costs, and code structure design costs; 2. Due to the code length limitation, the current erasure code has a lower code rate than the error correction code, requiring more additional redundancy and consuming more storage space. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide an encoding / decoding method and system. The technical solutions provided by one or more embodiments of this specification are as follows:

[0006] According to a first aspect of one or more embodiments of this specification, an encoding method is proposed, comprising:

[0007] A data fragment group containing m data fragments is determined, wherein each of the m data fragments is stored by a corresponding m user data storage disk; wherein each of the m data fragments corresponds one-to-one with an m information bit submatrix split from a user data submatrix, the user data submatrix belongs to a check matrix for error correction coding and corresponds to the information bit of the check matrix, and the check matrix also contains a check bit submatrix corresponding to the check bit.

[0008] Obtain the m sub-feature vectors corresponding to the m data segments respectively, and sum them to obtain the feature vector corresponding to the data segment group; wherein, the sub-feature vector corresponding to any data segment is generated based on any data segment and its corresponding information position submatrix;

[0009] Based on the feature vector corresponding to the data segment and the inverse matrix of the parity bit submatrix, the parity data corresponding to the data segment is generated, and the parity data and the m data segments together constitute the error correction codeword corresponding to the data segment, where m>1 and m is an integer.

[0010] According to a second aspect of one or more embodiments of this specification, a decoding method is provided, comprising:

[0011] Each of the m user data storage disks reads the m fragments of data contained in the fragmented data group in a one-to-one correspondence; wherein, the m fragments of data correspond one-to-one with the m information bit submatrices split from the user data submatrices, the user data submatrices belong to the check matrix used for error correction coding and correspond to the information bits of the check matrix, and the check matrix also contains check bit submatrices corresponding to the check bits.

[0012] Obtain the verification data corresponding to the data shard group. The verification data is generated based on the feature vector corresponding to the data shard group and the inverse matrix of the verification position submatrix. The feature vector corresponding to the data shard group is obtained by summing the m sub-feature vectors corresponding to the m data shards respectively. The sub-feature vector corresponding to any data shard is generated based on any data shard and its corresponding information position submatrix.

[0013] Using the check data corresponding to the m data segments and the data segments as the error correction codewords corresponding to the data segments, an error correction result vector is generated based on the check matrix and the error correction codewords corresponding to the data segments, and the decoding result of the error correction codewords corresponding to the data segments is determined based on the error correction result vector, where m > 1 and m is an integer.

[0014] According to a third aspect of one or more embodiments of this specification, an encoding system is provided, comprising: a user data storage disk and a verification data storage disk deployed on at least one server, wherein the user data storage disk is used to store user data and the verification data storage disk is used to store verification data;

[0015] In this embodiment, an error correction code encoder is deployed in any server, any user data storage disk, or any verification data storage disk, and the error correction code encoder is used to perform the method as described in any one of the first aspects above.

[0016] According to a fourth aspect of one or more embodiments of this specification, a decoding system is provided, comprising: a user data storage disk and a verification data storage disk deployed on at least one server; the user data storage disk is used to store user data, and the verification data storage disk is used to store verification data.

[0017] In this configuration, an error correction code decoder is deployed in any server, any user data storage disk, or any verification data storage disk, and the error correction code decoder is used to perform the method as described in any one of the second aspects above.

[0018] According to a fifth aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0019] A processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of either the first or second aspect described above by executing the executable instructions.

[0020] According to a sixth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores executable instructions thereon; wherein, when executed by a processor, the instructions implement the steps of any of the methods in the first or second aspect described above.

[0021] As can be seen from the above embodiments, this specification implements a multi-disk data protection and data storage scheme based on error correction codes, to replace the multi-disk disaster recovery function of the original erasure code storage scheme. Since the multi-disk data protection function is no longer implemented by erasure codes, the embodiments in this specification effectively eliminate the requirement for the server host to independently complete erasure code encoding / decoding. This eliminates the need to set up erasure code encoders / decoders and code construction, as well as a series of erasure code-related functions and implementations, thus reducing the costs associated with independently setting up erasure codes, such as the required functionalities, computing resources, and code construction design. Furthermore, by replacing the erasure code data protection mechanism with the error correction code data protection mechanism, since the bitrate of error correction codes is close to the Shannon limit and higher than that of erasure codes, less redundancy is required to achieve the same data recovery capability, saving more resources. The embodiments in this specification provide ample storage space. Furthermore, the traditional error correction coding process is broken down into multiple segmented steps involving the calculation of multiple data fragments with corresponding information bit sub-matrices and parity bit sub-matrices. This process generates a large number of intermediate calculation results, such as sub-vector features. Since each storage disk also needs to perform single-disk error correction coding on its stored data to form single-disk data protection capabilities, it is possible for the multi-disk error correction coding involved in the embodiments of this specification to reuse the same encoder / decoder and intermediate calculation results as the native single-disk error correction coding of each storage disk. This greatly reduces the hardware and software costs and computational costs of implementing joint data protection scenarios.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0024] Figure 1 This specification provides a system architecture diagram of a storage cluster employing an erasure coding storage scheme, based on an exemplary embodiment.

[0025] Figure 2 This is a schematic diagram of the system architecture of an encoding method provided in this specification according to an exemplary embodiment.

[0026] Figure 3 This is a flowchart of an encoding method provided in this specification according to an exemplary embodiment.

[0027] Figure 4 This is a schematic diagram of the core process of an encoding method provided in this specification according to an exemplary embodiment.

[0028] Figure 5This specification provides a system architecture diagram of a storage cluster employing an error-correcting code storage scheme, based on an exemplary embodiment.

[0029] Figure 6 This is a schematic diagram of an application scenario provided by this specification according to an exemplary embodiment.

[0030] Figure 7 This is a timing diagram of the execution of computing tasks by each storage disk in an application scenario provided in this specification according to an exemplary embodiment.

[0031] Figure 8 This is a schematic diagram illustrating another application scenario provided by this specification according to an exemplary embodiment.

[0032] Figure 9 This is a timing diagram showing the execution of computing tasks by each storage disk in another application scenario provided by this specification according to an exemplary embodiment.

[0033] Figure 10 This is a flowchart of a decoding method provided in this specification according to an exemplary embodiment.

[0034] Figure 11 This is a schematic diagram of an encoding system provided in this specification according to an exemplary embodiment.

[0035] Figure 12 This is a schematic diagram of a decoding system provided in this specification according to an exemplary embodiment.

[0036] Figure 13 This is a schematic diagram of the structure of a device provided in this specification according to an exemplary embodiment.

[0037] Figure 14 This is a block diagram of an encoding device provided in this specification according to an exemplary embodiment.

[0038] Figure 15 This is a block diagram of a decoding apparatus provided in this specification according to an exemplary embodiment. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0040] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0041] Figure 2 This is a schematic diagram of the system architecture of an encoding method provided in this specification according to an exemplary embodiment. Figure 2 As shown, the storage cluster in this system architecture includes several user data storage disks and parity data storage disks, and the encoding method involved in the embodiments of this specification is actually applied to... Figure 1 The error correction encoder in the system can be built into any storage disk or equipped by any server or third party. The encoder needs to obtain m sub-feature vectors corresponding to m (m is a positive integer not less than 2) data fragments within the same data fragment group. These m data fragments are stored one-to-one in m user data storage disks within the storage cluster. The encoder can obtain the m data fragments by reading them from the m data storage disks in the storage cluster or from other data sources containing the m data fragments, and then calculate the m sub-feature vectors for each fragment. Alternatively, the encoder can directly read the m sub-feature vectors from the m user data storage disks, where each sub-feature vector is generated by the user data storage disk containing the corresponding data fragment based on its built-in error correction encoder, thus distributing the sub-feature vector calculation task across the m user data storage disks. The error correction code encoder can generate the verification data corresponding to the above-mentioned data fragments through error correction encoding. The verification data and the m data fragments together constitute a complete error correction codeword, thereby enabling the data fragments to obtain the data protection capability of the error correction code. The verification data can be stored on the verification data storage disk in the storage cluster for subsequent error correction decoding / data recovery.

[0042] Figure 3 This is a flowchart illustrating an encoding method provided in this specification according to an exemplary embodiment. The method is applied to, for example... Figure 1 The error correction code encoder shown; as Figure 2 As shown, the method includes the following steps:

[0043] S302: Determine a data fragment group containing m fragments of data, wherein the m fragments of data are stored by m user data storage disks that correspond one-to-one; wherein the m fragments of data correspond one-to-one with the m information bit submatrices split from the user data submatrices, the user data submatrices belong to the check matrix used for error correction coding and correspond to the information bits of the check matrix, and the check matrix also contains check bit submatrices corresponding to the check bits.

[0044] like Figure 1 As shown, m user data storage disks (i.e., storage disks used to store user data) each store m data shards, and these m data shards belong to the same data shard group. The basic objective of this embodiment is to generate check data corresponding to a data shard group distributed across multiple storage disks, and to combine this check data with the data shards contained in the data shard group to form the error correction codeword corresponding to the data shard group. This allows the original m data shards and their corresponding check data to be recovered from the data shard group even if there are controllable errors or missing codes (e.g., a data shard is lost) in the data shard group, thanks to the properties of the error correction codeword. This provides data protection for the m data shards and their corresponding check data in the data shard group. Therefore, this embodiment can replace the erasure code storage scheme in related technologies with an error correction code storage scheme, thereby providing multi-disk disaster recovery data protection (if a storage disk loses data or a large number of errors, the data can still be recovered).

[0045] In the embodiments of this specification, the parity check matrix is ​​used to define the linear constraint between the information bits and parity bits contained in a certain error-correcting codeword, and this constraint is also a key factor in realizing error correction capability. Without considering erroneous or missing bits, for an error-correcting codeword generated based on the generator matrix corresponding to a certain parity check matrix, its product with the parity check matrix is ​​a zero vector; if the product is not a zero vector, it indicates that the error-correcting codeword contains errors. Based on the actual product result and the fault tolerance capability, the error-correcting codeword can be adjusted to recover the correct error-correcting codeword. The parity check matrix can be divided into a user data submatrix and a parity bit submatrix. The user data submatrix corresponds to the information bits in the parity check matrix. Specifically, it refers to the submatrix composed of matrix elements used to multiply the information bits of the erasure codeword when the parity check matrix is ​​multiplied by the erasure codeword. The parity bit submatrix corresponds to the parity bits in the parity check matrix. Specifically, it refers to the submatrix composed of matrix elements used to multiply the parity bits of the erasure codeword when the parity check matrix is ​​multiplied by the parity bits of the erasure codeword.

[0046] Figure 4 This is a schematic diagram of the core flow of an encoding method provided in this specification according to an exemplary embodiment. For example... Figure 4 As shown, let m = 4, meaning the data fragment U contains four data fragments u1, u2, u3, and u4. Simultaneously, the user data submatrix (corresponding to Hu1-Hu4 in the diagram) in the parity check matrix Hu used for error correction coding is also split into four corresponding information bit submatrices Hu1, Hu2, Hu3, and Hu4, and the remaining parity bit submatrix Hp. Here, u1 corresponds to Hu1, u2 to Hu2, u3 to Hu3, and u4 to Hu4, while the subsequently generated parity data p corresponds to Hp. It should be noted that the splitting method shown in the diagram for the data fragment group or user data submatrix is ​​uniform, meaning the lengths (i.e., the number of columns) of the four data fragments u1, u2, u3, and u4 are equal, and the lengths of the four information bit submatrices Hu1, Hu2, Hu3, and Hu4 are also equal. Of course, different splitting methods can be used, such as non-uniform splitting, but to ensure the correctness of subsequent matrix multiplication, it is also necessary to ensure that the length of each resulting data fragment is the same as the length of its corresponding information bit submatrix.

[0047] S304: Obtain the m sub-feature vectors corresponding to the m data segments respectively, and sum them to obtain the feature vector corresponding to the data segment group; wherein, the sub-feature vector corresponding to any data segment is generated based on any data segment and its corresponding information position sub-matrix.

[0048] In the embodiments of this specification, the error correction code encoder needs to obtain m sub-feature vectors corresponding to each of the m data segments in the segmented data group, and use the matrix obtained by accumulating the m sub-feature vectors as the feature vector corresponding to the segmented data group. Since the m sub-feature vectors (in form matrices) have the same matrix size, they can be summed by adding the elements in the same row and column.

[0049] like Figure 4As shown, the sub-feature vectors corresponding to the four data segments u1, u2, u3, and u4 are Su1, Su2, Su3, and Su4, respectively. Each sub-feature vector is generated based on that data segment and its corresponding information position sub-matrix. Specifically, it is obtained by matrix multiplication of the data segment and its corresponding information position sub-matrix, i.e., Su1 = u1 * Hu1, Su2 = u2 * Hu2, Su3 = u3 * Hu3, and Su4 = u4 * Hu4. During the matrix multiplication, the information position sub-matrix actually involved in the operation first needs to be transposed and then multiplied by the corresponding data segment. For example, u1 is a 1*3 matrix, and Hu1 is a 2*3 matrix; therefore, the transpose of Hu1 is a 3*2 matrix, and the product of u1 and the transpose of Hu1 is a 1*2 matrix, Su1. Then, the feature vector corresponding to the data segment group is Su = Su1 + Su2 + Su3 + Su4. It is important to note that since we stipulate that all fragmented data uses binary, meaning each information bit is either 0 or 1, u1, u2, u3, and u4 can be considered as binary strings, or matrices where any element is either 0 or 1. Similarly, Hu1, Hu2, Hu3, Hu4, and Hp are also matrices where any element is either 0 or 1. Therefore, when performing matrix summation, we use the XOR addition rule to ensure that the resulting Su is also a matrix where any element is either 0 or 1. In addition, all additions in matrix multiplications involved in the embodiments of this specification are also modified to XOR addition, thereby ensuring that all matrices are matrices where any element is either 0 or 1.

[0050] In one embodiment, the error-correcting code encoder generates and obtains the m sub-feature vectors by multiplying the acquired m data segments with the corresponding m information bit sub-matrices. In another embodiment, the error-correcting code encoder can also directly obtain the m sub-feature vectors from other user data storage disks, while the matrix multiplication task between the specific data segments and the information bit sub-matrices is distributed to other user data storage disks. Specifically, the sub-feature vector corresponding to any data segment is generated and provided by the user data storage disk storing that data segment. Figure 1For example, m data segments (segment data 1-m in the diagram) are stored by m user data storage disks (user data storage disks 1-m in the diagram), and the m sub-vector features (sub-vector features 1-m in the diagram) corresponding to each of the m data segments are provided by the m user data storage disks storing the m data segments. For example, user data storage disk 1 uses its built-in in-disk error correction encoder to perform matrix multiplication to generate sub-vector feature 1 corresponding to its stored segment data 1, and provides sub-vector feature 1 to the error correction encoder that performs feature sub-vector summation. For example, user data storage disk 2 uses its built-in in-disk error correction encoder to perform matrix multiplication to generate sub-vector feature 2 corresponding to its stored segment data 2, and provides sub-vector feature 2 to the error correction encoder that performs feature sub-vector summation, and so on. In this way, the matrix multiplication task is evenly distributed across multiple user storage disks, and each user storage disk can perform matrix multiplication in parallel. The error correction encoder, which performs error correction coding, does not need to perform matrix multiplication of subvector features and corresponding information bit submatrices, but only needs to perform the summation operation of feature subvectors, which greatly enhances the resource utilization of each user data storage disk and the overall execution efficiency of the scheme.

[0051] In the embodiments of this specification, each user data storage disk maintains a shared memory space for caching its own generated sub-feature vectors. The m user data storage disks obtain each other's generated sub-feature vectors through their respective maintained shared memory spaces. After calculating the sub-vector features corresponding to the fragmented data stored by itself, the user data storage disk can store them in its own maintained shared memory space. Since different storage disks (including user data storage disks and verification data storage disks) can quickly obtain shared information by accessing each other's maintained shared memory spaces, any third party (such as an erasure coding encoder maintained in a server) can also conveniently obtain sub-vector features by accessing these shared memory spaces. Therefore, the embodiments of this specification can delegate the information interaction tasks between different storage disks and third parties to the shared memory space built into the storage disk, thereby reducing the overhead of latency, bandwidth, and CPU resources caused by direct CPU involvement in memory copying, while improving data transmission efficiency and realizing direct access to the sub-vector features generated by the user data storage disk. Figure 5 This specification provides a system architecture diagram of a storage cluster employing an error-correcting code storage scheme according to an exemplary embodiment. Figure 5As shown, storage disks 1-5 are user data storage disks and / or verification data storage disks, which are stored in multiple servers or corresponding extended JBODs (Just a Bunch of Disks). Each storage disk has a built-in error correction code encoder, shared memory space, and storage medium. After the error correction code encoder generates the corresponding sub-feature vector, it can be cached in the shared memory space for other storage disks or third parties to access.

[0052] Of course, in addition to enabling the shared acquisition of sub-vector features, the shared memory space can also store any other data for sharing. For example, before the m user data storage disks have stored the corresponding m shard data, one user data storage disk can first acquire the m shard data contained in the shard data group. Then, the user data storage disk caches the m shard data into the shared memory space it maintains. Other user data storage disks can then access the shared memory space to obtain the shard data that they should store, thereby realizing the allocation and storage of each shard data in the shard data group through the shared memory space.

[0053] S306: Generate the check data corresponding to the data segment based on the feature vector corresponding to the data segment and the inverse matrix of the check bit submatrix, and the check data and the m data segments together constitute the error correction codeword corresponding to the data segment, where m>1 and m is an integer.

[0054] In the embodiments of this specification, after summing the feature vectors corresponding to the data segments, the error-correcting encoder responsible for error correction coding will further generate the check data corresponding to the data segments based on the inverse matrix of the feature vector and the check bit submatrix. For example... Figure 4 As shown, firstly, we need to obtain the inverse matrix INV(Hp) corresponding to the parity submatrix Hp. This can be obtained through timely matrix inversion operations or by directly loading a pre-calculated INV(Hp). Here, INV() represents the matrix inversion operation; if the input matrix has a size of i*j, the corresponding output matrix has a size of j*i, and the product between the input and output matrices is an i*i identity matrix. Therefore, the final parity data p = Su*INV(Hp). As mentioned earlier, Su and Su1-Su4 have the same size. Assuming the size of Su is 1*2 and the size of Hp is 3*2, then the size of INV(Hp) is 2*3. Thus, the size of the parity data p obtained by multiplying Su and INV(Hp) should be 1*3. Finally, the parity data p can be concatenated with u1-u4 to form a complete error-correcting codeword, just as... Figure 2As shown in the error correction codeword, the error correction codeword constructed in the embodiments of this specification contains m data fragments in the data fragment group and corresponding check data.

[0055] Furthermore, it can be verified mathematically that if... Figure 4 The final concatenated error-correcting codeword X is multiplied by the parity check matrix (the transpose of the parity check matrix is ​​omitted here). This multiplication can be equivalently decomposed into X*H=[u1,u2,u3,u4,p]*[Hu1,Hu2,Hu3,Hu4,Hp]=u1*Hu1+u2*Hu2+u3*Hu3+u4*Hu4+p*Hp=Su1+Su2+Su3+Su4+(Su*INV(Hp))*Hp=Su+Su*E=Su+Su=0, where “E” represents the identity matrix, and “0” in “Su+Su=0” represents the zero vector, which has the same size as Su. The reason for “Su+Su=0” is that the embodiment in this specification uses XOR addition instead of the traditional numerical addition in matrix multiplication. Therefore, the XOR value between the same values ​​is 0. Therefore, it can be observed that the product of the error-correcting codeword and the parity-check matrix is ​​a zero vector when there is no error. However, if the error-correcting result vector obtained by the product is not a zero vector, further error correction is required to recover the correct error-correcting codeword.

[0056] This specification proposes an encoding method to implement a multi-disk data protection and data storage scheme based on error correction codes, which can replace the multi-disk disaster recovery function of the original erasure code storage scheme. Since the multi-disk data protection function is no longer implemented by erasure codes, this specification's embodiments effectively eliminate the requirement for the server host to independently complete erasure code encoding / decoding, thus eliminating the need for settings such as... Figure 1The document describes a series of erasure coding-related functions and implementations, including erasure coding encoders / decoders and code construction, which reduce the costs associated with independently setting up erasure coding, such as the number of functional units, computing resources, and code construction design. Furthermore, it replaces the erasure coding data protection mechanism with the error correction coding data protection mechanism. Since the error correction code's bitrate is close to the Shannon limit and higher than the erasure coding's bitrate, less redundancy is required to achieve the same data recovery capability, saving more storage space. Additionally, the embodiments in this specification decompose the traditional error correction coding process into multiple segmented steps involving the calculation of multiple data fragments with corresponding information bit submatrices and parity bit submatrices. This process generates a large number of intermediate calculation results, such as subvector features. Since each storage disk also needs to perform single-disk error correction coding on its stored data to form single-disk data protection capability, it makes it possible for the multi-disk error correction coding involved in the embodiments of this specification to reuse the same encoders / decoders and intermediate calculation results as the native single-disk error correction coding of each storage disk. This significantly reduces the hardware, software, and computational costs of implementing a joint data protection scenario.

[0057] Optionally, the m data fragments are obtained by splitting the same user data into m parts. The user data involved in the embodiments of this specification refers to continuous data on the content the information points to. Similarly, user data can also be represented in binary, and different user data have the same specifications, meaning that different user data have the same number of binary bits. As mentioned above, a data fragment group contains m data fragments, and in the embodiments of this specification, these m data fragments originate from the same user data and are obtained by splitting that user data into m parts.

[0058] When storing small amounts of data, related technologies struggle to balance multi-disk disaster recovery and data error correction without incurring additional losses in capacity and efficiency. For example, in a joint protection scenario where erasure coding and error correction coding storage schemes operate independently, the data to be protected must meet the specifications required to generate a complete erasure coding codeword. When the data cannot meet these specifications, it is typically first padded to supplement its specifications to generate a complete erasure coding codeword. Then, the erasure coding codeword is split and stored across multiple storage disks. Finally, each storage disk generates its own error correction codeword based on its stored data, thus establishing data error correction capability. This process indirectly increases storage costs and reduces computational efficiency because it introduces invalid data for padding. Conversely, if the erasure code codeword specification is downgraded instead of data padding, the resulting slice data will not meet the specifications for generating erasure codewords on a single disk, thus failing to achieve data error correction. Alternatively, if data padding is not performed and erasure codeword generation is abandoned, and the data to be protected is simply split according to the error correction codeword specification and stored on multiple storage disks, although error correction code protection can be achieved on a single disk, the lack of multi-disk disaster recovery capability means that if a large amount of slice data stored on a storage disk is lost or corrupted, the slice data cannot be recovered.

[0059] The embodiments in this specification achieve multi-disk disaster recovery based on the same user data by splitting the same user data into m parts and storing them on m user data storage disks. At the same time, since these m data fragments generate verification data to form error correction codes after forming a data fragment group, they simultaneously take into account data error correction and data recovery. This eliminates the need to regenerate error correction codes within a single disk, realizing multi-disk disaster recovery and data error correction for the same user data. It also reduces the required user data specification from one erasure code word in related technologies to one error correction code word (since erasure code storage schemes require multi-disk storage, the erasure code word specification is at least twice that of the error correction code word). This makes it more suitable for application scenarios that store only a small amount of data.

[0060] The embodiments of this specification will be described below with reference to a specific example. Figure 6 This is a schematic diagram of an application scenario provided by this specification according to an exemplary embodiment. Figure 6Storage disks 1-4 shown are user data storage disks, and storage disk 5 is a verification data storage disk, with m=4. In this application scenario, user data A and user data B need to be stored. Assuming the specifications of the user data are the same as the specifications of the error correction code words formed by a single disk, then in the joint protection scenario where the erasure coding storage scheme and the error correction code storage scheme operate independently, at least four user data sets are needed to simultaneously address multi-disk disaster recovery and data error correction for these four user data sets. Therefore, related technologies require first supplementing user data A and user data B with two invalid data sets that have the same user data specifications to form an erasure coding protection group. However, in the embodiment of this specification, no data supplementation is required. Instead, user data A and user data B are directly split into fragments A1, A2, A3, and A4 of user data A, and fragments B1, B2, B3, and B4 of user data B. Simultaneously, A1, A2, A3, and A4 are combined into a single data fragment g, labeled A1_g, A2_g, A3_g, and A4_g; and B1, B2, B3, and B4 are combined into another data fragment y, labeled B1_y, B2_y, B3_y, and B4_y. A1_g and B1_y are stored on storage disk 1, A2_g and B2_y on storage disk 2, A3_g and B3_y on storage disk 3, and A4_g and B4_y on storage disk 4. Finally, data fragment g and data fragment y are encoded using the methods described in this specification to generate corresponding checksums Pg and Py, which are then stored on storage disk 5. In this way, A1_g, A2_g, A3_g, A4_g and Pg form an error correction codeword, and B1_y, B2_y, B3_y, B4_y and Py form another error correction codeword, thus realizing multi-disk disaster recovery and data error correction at the same time, and this process does not add invalid data.

[0061] Figure 6 There is a column representing the calculation parameters, namely Hu1, Hu2, Hu3, Hu4 and Hp. The data of any calculation parameter is horizontally aligned and is a calculation correlation factor with that calculation parameter during the encoding process. For example, Hu1 is horizontally aligned with A1_g and B1_y, which means that Hu1 will participate in the calculation of sub-vector features together with A1_g and B1_y respectively during the encoding process. Or Hp is horizontally aligned with Pg and Py, which means that Hp is required to participate in the calculation when generating Pg and Py during the encoding process.

[0062] As mentioned above, the computational task of generating sub-feature vectors or verification data involved in the embodiments of this specification can be distributed across m user data storage disks. The following will illustrate this further. Figure 7 This example illustrates the task execution status of each storage disk after the computational tasks for generating sub-vector features and verification data are distributed. Figure 7This specification provides a timing diagram of the execution of computing tasks by various storage disks in an application scenario based on an exemplary embodiment. Specifically... Figure 6 Timing diagrams of computing tasks executed by each storage disk in the application scenario. (Example) Figure 7 As shown, assuming the built-in error correction encoders in each storage disk have the same efficiency (i.e., each matrix multiplication operation requires a time unit of T), then storage disk 1 needs to calculate the two sub-vector features corresponding to A1_g and B1_y respectively. Similarly, storage disks 2, 3, and 4 will also calculate two sub-vector features respectively. Therefore, calculating all 8 sub-vector features only requires 2T of time. Finally, since the sub-vector features can be transferred between different storage disks through shared memory space, the process of summing the sub-feature vectors corresponding to A1_g, A2_g, A3_g, and A4_g to obtain the feature vector Sg corresponding to the fragmented data group g, and using matrix multiplication to obtain the final verification data Pg, can be performed by any of the storage disks 1-5. Similarly, the process of summing B1_y, B2_y, B3_y, and B4_y to obtain the feature vector Sy corresponding to the fragmented data group y, and then using matrix multiplication to obtain the final check data Pg, can be performed by another storage disk among storage disks 1-5. Since only two check data are ultimately generated, the matrix multiplication task required to generate the check data only involves two operations. Since there are a total of five storage disks (i.e., five error correction encoders performing matrix multiplication), the entire process of generating check data for all fragmented data groups in the above application scenario only requires one T operation. Furthermore, the generated check data can also be transferred to storage disk 5 through data sharing (stored in shared memory space), and written to storage disk 5 in the third T operation. In summary, to achieve... Figure 6 The encoding process in the application scenario shown takes a total of 3TB, which makes full use of the built-in error correction encoders on each storage disk, resulting in high resource utilization and execution efficiency.

[0063] Optionally, the m data segments are each derived from m pieces of user data, and each data segment is one of the data segments obtained by splitting its corresponding user data. In the embodiments of this specification, the m data segments contained in the above-mentioned data segment group are each derived from the corresponding m pieces of user data. At the same time, each data segment in the data segment group is actually one of the data segments obtained by splitting its corresponding user data.

[0064] Based on the fact that the m data segments are respectively derived from m user data, each user data in the m user data is split into m parts; the method further includes: generating m verification data corresponding to the m data segments respectively, wherein each data segment contains m data segments respectively derived from the m user data, and the data segments contained in different data segments are different from each other.

[0065] In the embodiments of this specification, each piece of data in the fragmented data group originates from one of the fragmented data obtained by splitting its corresponding user data into m parts. Furthermore, this embodiment involves m different fragmented data groups, and verifiable data corresponding to these m different fragmented data groups is generated according to the aforementioned encoding method. Since each fragmented data group contains m fragmented data sources from the m user data entries, and the fragmented data contained in different fragmented data groups are distinct from each other, coupled with the fact that the user data is split into m parts, this means that the m fragmented data groups contain a total of m*m fragmented data sources, all originating from the m user data entries and being mutually exclusive. Clearly, under the m-part splitting principle, these m user data entries also contain a total of m*m fragmented data entries. Therefore, the m fragmented data groups completely cover the m fragmented data entries, jointly forming a multi-layered, orthogonal, and joint data protection for the m fragmented data entries.

[0066] Current erasure coding storage solutions suffer from the following technical problems: Data recovery / decoding requires reading multiple slices of data from the erasure coding-corresponding shard data group from multiple storage disks. However, since the data read from each storage disk is at least a complete error correction code word, the cost of data recovery / decoding is too high in scenarios where a small amount of local data is lost or corrupted. In contrast, the embodiments described in this specification, while also reading multiple slices of data from the error correction code-corresponding shard data group from multiple storage disks during data recovery / decoding, do not construct the corresponding error correction code independently on their respective storage disks. Instead, each slice of data contributes to the information bits of the error correction code word along with other slices. Therefore, the data read from each storage disk in this embodiment is not a complete error correction code word, but only a portion of the error correction code word's information bits. This significantly reduces the cost of data recovery / decoding in scenarios where a small amount of local data is lost or corrupted.

[0067] The embodiments of this specification will be described below with reference to a specific example. Figure 8 This is a schematic diagram illustrating another application scenario provided by this specification according to an exemplary embodiment. Figure 8 For example, Figure 8Storage disks 1-4 shown are user data storage disks, and storage disk 5 is a parity data storage disk. Let m = 4. In this application scenario, there are four user data sets A, B, C, and D that need to be stored. Assume the specifications of the user data are the same as the specifications of the error correction codewords formed on a single disk. As shown in the figure, user data A is split into fragments A1_g, A2_y, A3_o, and A4_b and stored simultaneously on storage disk 1; user data B is split into fragments B1_b, B2_g, B3_y, and B4_o and stored simultaneously on storage disk 2; user data C is split into fragments C1_o, C2_b, C3_g, and C4_y and stored simultaneously on storage disk 3; and user data D is split into fragments D1_y, D2_o, D3_b, and D4_g and stored simultaneously on storage disk 4. Meanwhile, this embodiment of the specification sets up a total of 4 data fragment groups, wherein data fragment group g contains A1_g, B2_g, C3_g and D4_g, data fragment group y contains A2_y, B3_y, C4_y and D1_y, data fragment group o contains A3_o, B4_o, C1_o and D2_o, and data fragment group b contains A4_b, B1_b, C2_b and D3_b. Finally, data fragment groups g, y, o and b are encoded using the encoding method described in this specification to generate corresponding check data Pg, Py, Po and Pb, respectively, and stored on storage disk 5. It should be noted that in this embodiment of the specification, the length of any check data is equal to the length of any data fragment. Thus, A1_g, B2_g, C3_g, D4_g and Pg constitute an error-correcting codeword, and similarly A2_y, B3_y, C4_y, D1_y and Py constitute another error-correcting codeword, and so on.

[0068] In the above application scenario, assuming that the data in part A1_g is lost / erroneous, if the relevant technology is applied, since it forms an erasure code protection group by combining user data A, B, C, and D as a whole, it is necessary to read the entire user data A, B, C, and D, as well as the corresponding erasure code verification data, from storage disks 1-5 respectively. Then, the corresponding erasure code codeword can be recovered through erasure decoding, and the data in part A1_g can be recovered. Obviously, this process requires reading the complete information bits of the error correction codeword from each storage disk, which is costly. In the embodiments described in this specification, it is only necessary to first determine the data fragment group g to which A1_g belongs, and then read A1_g, B2_g, C3_g, D4_g and Pg from storage disks 1-5 respectively. The correct error correction codeword corresponding to the data fragment group g can be recovered through error correction decoding, thereby realizing the data recovery of part A1_g. Therefore, the embodiments described in this specification do not require reading the complete information bit part of the error correction codeword from each storage disk, but only need to read a part of the information bit part of the error correction codeword, thereby reducing the cost of data recovery / decoding in scenarios where a small amount of local data is lost or erroneous.

[0069] Based on the above embodiments, the m verification data are stored on a verification data storage disk, which is different from the user data storage disk. This verification data storage disk also stores special verification data obtained by error correction encoding of the m verification data based on the verification matrix. The m verification data and the special verification data together constitute the error correction codewords corresponding to the m verification data. In the embodiments of this specification, since the m verification data corresponding to the m fragmented data groups are ultimately stored on the verification data storage disk, the verification data storage disk can also perform error correction encoding on these m verification data based on its single-disk or third-party error correction encoding capabilities to generate corresponding special verification data, thereby constituting error correction codewords to provide single-disk error correction protection for the m verification data. Here, the error correction encoding for the m verification data is the same encoding method as the aforementioned encoding method based on fragmented data groups, sharing the same process, specifications, and calculation parameters, such as sharing the same verification matrix. The specific process can be referred to Figure 4 .like Figure 8 As shown, storage disk 5 stores four data fragments corresponding to parity data Pg, Py, Po, and Pb. It can also store special parity data PP generated by error correction encoding of Pg, Py, Po, and Pb. Pg, Py, Po, Pb, and PP together constitute an error correction codeword, thereby providing error correction protection for a single disk.

[0070] Based on the m data segments originating from m user data entries, each of the m user data entries is further divided into m parts. Each of the m user data entries is stored on a corresponding set of m user data storage disks. Each user data storage disk also stores check data generated from the feature vector corresponding to its stored user data and the inverse matrix of the check sub-matrix. This check data, together with the stored user data, constitutes the error correction codeword for that user data. The feature vector corresponding to its stored user data is obtained by summing the m sub-feature vectors corresponding to the m data segments from which the user data is divided. Obtaining the sub-feature vector corresponding to any data segment in the data segment group includes: determining the user data to which the data segment belongs, and obtaining the sub-feature vector generated for that data segment to generate the feature vector corresponding to that user data.

[0071] In the embodiments of this specification, m user data storage disks correspond to m user data. That is, each user data storage disk not only stores a certain fragment of data belonging to the fragmented data group, but also stores the entire user data from which the fragmented data originates. This means that each user data storage disk can also perform error correction coding on the corresponding stored user data based on its single disk or third-party error correction coding capabilities to generate verification data corresponding to that user data, thereby forming error correction codewords to provide single disk error correction protection for the entire user data. Here, the error correction coding for user data is the same coding method as the aforementioned coding method based on fragmented data groups, sharing the same process, specifications, and calculation parameters, such as sharing the same verification matrix and reused sub-vector features. The specific process can be referred to Figure 4 Specifically, since the feature vectors corresponding to the m data shards and the m user data are both obtained by filtering and summing from a sub-vector feature pool containing the same m*m sub-vector features, when performing error correction coding on data shards or user data, sub-vector features can be reused without repeatedly calculating the sub-vector features, thus greatly reducing computational costs and improving execution efficiency. For example, when performing error correction coding based on data shards, it is necessary to determine the data shards contained in the data shards and sum the sub-feature vectors corresponding to each data shard to obtain the feature vector corresponding to the data shards. However, when obtaining the sub-vector features corresponding to each data shard, we can first determine the user data from which a certain data shard originates, and then directly obtain the sub-feature vector generated for that data shard to generate the feature vector corresponding to that user data, thus avoiding repeated calculation of the sub-feature vector. Figure 8As shown, storage disk 1 stores A1_g, A2_y, A3_o, and A4_b, which are obtained by splitting user data A. It can also store parity data PA generated by error correction encoding of A1_g, A2_y, A3_o, and A4_b. A1_g, A2_y, A3_o, and A4_b, together with PA, constitute an error correction codeword, thus providing single-disk error correction protection. Other user data storage disks work similarly. For example, storage disk 2 also stores parity data PB generated by error correction encoding of B1_b, B2_g, B3_y, and B4_o; storage disk 3 also stores parity data PC generated by error correction encoding of C1_o, C2_b, C3_g, and C4_y; and storage disk 4 also stores parity data PD generated by error correction encoding of D1_y, D2_o, D3_b, and D4_g, thus providing single-disk error correction protection for each user's data. Meanwhile, taking the fragmented data group g as an example, the sub-vector features corresponding to the fragmented data A1_g contained therein are not only used to generate the feature vector corresponding to the fragmented data group g, but also used to generate the feature vector corresponding to the user data A.

[0072] Figure 8 One row represents the calculation parameters, namely Hu1, Hu2, Hu3, Hu4, and Hp. The data aligned vertically with any calculation parameter are calculation-related factors for each other during the encoding process. For example, Hu1 is vertically aligned with A1_g, B1_b, C1_o, D1_y, and Pg, which means that during the encoding process, Hu1 will participate in the calculation of sub-vector features together with A1_g, B1_b, C1_o, D1_y, and Pg respectively. Or, Hp is vertically aligned with PA, PB, PC, PD, and PP, which means that Hp is required to participate in the calculation of PA, PB, PC, PD, and PP during the encoding process.

[0073] As mentioned above, the computational task of generating sub-feature vectors or verification data involved in the embodiments of this specification can be distributed across m user data storage disks. The following will illustrate this further. Figure 9 This example illustrates the task execution status of each storage disk after the computational tasks for generating sub-vector features and verification data are distributed. Figure 9 This is a timing diagram illustrating the execution of computing tasks by various storage disks in another application scenario provided by this specification according to an exemplary embodiment. Specifically, Figure 8 Timing diagrams of computing tasks executed by each storage disk in the application scenario. (Example) Figure 7As shown, assume that the built-in error correction encoders in each storage disk have the same efficiency, meaning that the time unit for each matrix multiplication operation is T. Then, storage disk 1 needs to calculate the four sub-vector features corresponding to A1_g, A2_y, A3_o, and A4_b respectively. Similarly, storage disks 2, 3, and 4 will each calculate four sub-vector features. Therefore, calculating all 16 sub-vector features only takes 4T of time. After this, each storage disk can sum its own generated sub-feature vectors to obtain the feature vector corresponding to its stored user data, and then use matrix multiplication to obtain the corresponding check data for its stored user data. This step takes one T. Specifically, storage disk 1 generates check data PA corresponding to user data A based on the inverse matrix of the feature vector SA and Hp corresponding to user data A obtained by summing. Storage disk 2 generates check data PB corresponding to user data B based on the inverse matrix of the feature vector SB and Hp corresponding to user data B obtained by summing. Storage disk 3 generates check data PC corresponding to user data C based on the inverse matrix of the feature vector SC and Hp corresponding to user data C obtained by summing. Storage disk 4 generates check data PD corresponding to user data D based on the inverse matrix of the feature vector SD and Hp corresponding to user data D obtained by summing.

[0074] Next, since the sub-vector features can be transferred between different storage disks via shared memory space, any storage disk can sum the sub-feature vectors corresponding to A1_g, B2_g, C3_g, and D4_g respectively to obtain the feature vector Sg corresponding to the data shard g, and use matrix multiplication to obtain the final check data Pg; similarly, any storage disk can also obtain the feature vectors Sy, So, and Sb corresponding to the data shards y, o, and b respectively, and use matrix multiplication to obtain the final check data Py, Po, and Pb respectively. Since only 4 check data are ultimately generated, that is, the matrix multiplication task required to generate the check data only includes 4 times, and there are a total of 5 storage disks (that is, there are 5 error correction encoders performing matrix multiplication), the global cost only needs to be one T to complete the process of generating check data corresponding to all data shards in the above application scenario. In addition, the generated check data can also be transferred to storage disk 5 through data sharing (stored in shared memory space), and written to storage disk 5 in the 6th T. After 6T of time has elapsed, the error correction encoders built into storage disks 1-4 are idle. They can start from the 7th T to perform error correction encoding on the other 4 groups of fragmented data and the other 4 user data from their sources, thus entering the next workflow. The execution sequence of the tasks in the next workflow is completely consistent, involving storage disks 1-4 first spending 6T, then storage disk 5 spending 5T alone, and so on in a loop.

[0075] Finally, the error correction encoder in storage disk 5 will also execute the encoding method involved in this application based on the obtained check data Pg, Py, Po, and Pb as a user data or fragment data group. Specifically, it needs to multiply Pg, Py, Po, and Pb with the corresponding information bit sub-matrix to obtain their respective sub-vector features, sum them to obtain the feature vector SP corresponding to Pg, Py, Po, and Pb, and perform matrix multiplication again to obtain the special check data PP corresponding to Pg, Py, Po, and Pb, thereby constructing the error correction capability in the disk. This part takes a total of 5T, and this time can be completely masked by the 6T required for storage disks 1-4 to execute the next workflow in parallel at this time. Therefore, from the perspective of long-term continuous data storage workflow, the time for the system to complete one workflow is 6T.

[0076] In summary, in order to achieve Figure 8 The complete encoding process in the application scenario shown requires a total of 11TB of time. Starting from the 7th TB, storage disks 1-4 can execute the next workflow. Therefore, from the perspective of a continuous workflow, to achieve... Figure 8 The encoding process in the application scenario shown takes an average of 6T, which makes full use of the built-in error correction encoders of each storage disk, resulting in high resource utilization and execution efficiency.

[0077] Based on the premise that the m data fragments each originate from m user data entries, and these m user data entries belong to the same user data group, with each user data group containing m user data entries, and the user data entries contained in different user data groups being either distinct or partially identical. In the embodiments of this specification, the set of m user data entries undergoing joint data protection is referred to as a user data group, and the workflow for storing m user data entries in the aforementioned scheme is also a workflow for a single user data group. Clearly, in a continuous workflow, the workflow for a single user data group can be repeatedly executed, which involves multiple different user data groups, each containing m user data entries.

[0078] In the embodiments of this specification, different user data groups can be configured to contain completely different or partially identical user data. For example, when the data required for the next workflow cannot fill a user data group, such as containing only two user data groups, the traditional approach, as described above, is to supplement with invalid data to form a complete user data group and execute the workflow for that user data group. However, this approach obviously increases storage costs and reduces computational efficiency because it requires additional storage of invalid data on the user data storage disk. In the embodiments of this specification, by setting different user data groups to contain partially identical user data, that is, by using the user data already stored on the user data storage disk in the previous workflow as the virtually supplemented user data for the user data group in the current workflow, a complete user data group can still be formed and the corresponding workflow can be executed. Compared to traditional solutions, this embodiment does not actually add invalid data to the user data storage disk. Instead, it reuses previously stored user data to virtually supplement the data. This process only requires marking the location information of the virtually supplemented user data in the current user data group, without storing any new data, thus avoiding increased storage costs. Furthermore, when calculating the check data corresponding to the m shard data groups anchored to the m user data in the current user data group, it is necessary to utilize the sub-vector features corresponding to the shard data obtained from each user data split. Since this embodiment reuses user data already stored in the user data storage disk, the sub-vector features corresponding to the shard data have obviously already been calculated. Therefore, this embodiment can directly reuse these sub-vector features without recalculating them, thus introducing no additional computational costs and having a minimal impact on computational efficiency. Therefore, by setting different user data groups to contain the same portion of user data, it is applicable to application scenarios where only a small amount of data needs to be stored, without introducing additional storage costs or reducing computational efficiency.

[0079] Based on the m data shards each originating from m user data entries, the m user data storage disks and / or k checksum data storage disks, distinct from the user data storage disks, are deployed on the same or different servers, where k ≥ 1 and k is an integer. For example... Figure 5As shown, storage disks 1-5 are user data storage disks and / or verification data storage disks. These storage disks can be deployed on the same server or on different servers. For example, storage disk 1 and storage disk 2 are stored on the same server A, while storage disk 3 and storage disk 4 are deployed on servers X and Z, respectively. In the erasure coding storage scheme, the erasure code encoder and decoder used for erasure coding / decoding are deployed on the server. The configuration of the erasure code encoder / decoder equipped on each server is relatively fixed, making it difficult to expand its capabilities. This means that after any server expands its storage disks, the erasure coding / decoding task on that server will inevitably increase, resulting in significant lag. Therefore, in the system architecture where erasure coding functionality is bound to the server, different servers have strong homogeneous needs, and the number of storage disks deployed on each server is relatively fixed and average, making it difficult to achieve flexible expansion of computing resources (the configuration of the erasure code encoder / decoder) and storage resources (the number of storage disks) at the server level. In the embodiments described in this specification, since erasure coding protection is replaced by error correction coding protection, the strong binding relationship between computing resources and servers is eliminated. Therefore, flexible expansion of storage resources can be achieved at the server level. That is, different servers can deploy different numbers of storage disks. At the same time, since the main body that performs error correction coding is also built into the storage disk, expanding the storage disk will also give the server more powerful computing resources as a whole, thus enabling flexible expansion of computing resources at the server level.

[0080] Optionally, it also includes: providing the verification data corresponding to the fragmented data group to any verification data storage disk that is different from the user data storage disk for storage; or, providing the k fragmented verification data divided from the verification data corresponding to the fragmented data group to k corresponding verification data storage disks for storage, where k≥1 and k is an integer.

[0081] In the embodiments of this specification, the verification data corresponding to the fragmented data group can be provided to the same verification data storage disk for storage, or the verification data can be split into k fragmented verification data and provided to k corresponding verification data storage disks for storage. After the verification data is stored in the verification data storage disk, it will form a multi-disk disaster recovery capability for the fragmented data group together with other user data storage disks. When the fragmented data belonging to the fragmented data group stored in one or more user data storage disks is lost or corrupted, or when the verification fragmented data / verification data corresponding to the fragmented data group stored in one or more verification data storage disks is lost or corrupted, the fragmented data contained in the fragmented data group and the corresponding verification fragmented data / verification data contained in the fragmented data group can be obtained by reading these storage disks containing the fragmented data or the corresponding verification fragmented data / verification data, thereby recovering through error correction decoding. Of course, the final recovery situation needs to consider that the lost or corrupted data does not exceed the limit of the corresponding error correction capability. It should be noted that when splitting the verification data, in order to make it convenient for the split verification data to still reuse the verification matrix for single disk error correction encoding, the specifications of any split verification data can be set to be the same as the specifications of any fragment data. This also means that the length of the verification submatrix is ​​always k times the length of the user data submatrix, and the length of the verification data is always k times the length of the fragment data.

[0082] Figure 10 This is a flowchart illustrating a decoding method provided in this specification according to an exemplary embodiment. The method is applied to an error-correcting code decoder; such as... Figure 10 As shown, the method includes the following steps:

[0083] S1002: Read the m fragments of data contained in the fragmented data group from the m user data storage disks one by one; wherein, the m fragments of data correspond one-to-one with the m information bit submatrices split from the user data submatrix, the user data submatrix belongs to the check matrix used for error correction coding and corresponds to the information bit of the check matrix, and the check matrix also contains a check bit submatrix corresponding to the check bit.

[0084] The error correction code decoder involved in the embodiments of this specification can be an error correction code decoder built into any user data storage disk or any verification data storage disk, or it can be an error correction code decoder equipped by any server or third party.

[0085] S1004: Obtain the verification data corresponding to the data shards. The verification data is generated based on the inverse matrix of the feature vector corresponding to the data shards and the verification position submatrix. The feature vector corresponding to the data shards is obtained by summing the m sub-feature vectors corresponding to the m data shards respectively. The sub-feature vector corresponding to any data shard is generated based on any data shard and its corresponding information position submatrix.

[0086] As mentioned earlier, the verification data may be stored in a single verification data storage disk, or it may be split into k parts and stored in k separate verification data storage disks, where k ≥ 1 and k is an integer. Therefore, the error correction code decoder can either directly read the acquired verification data from a single verification data storage disk, or read the k verification fragments corresponding to the data group from each of the k user data storage disks, and then concatenate the k verification fragments to obtain the verification data.

[0087] It is important to note that if some or all of the data is lost when acquiring m data fragments or verification data, the lost parts can be reset to preset information bits. For example, all bits of the lost parts can be set to 0 or 1, or set to random data. This ensures that even if data loss exists, a complete error correction code can eventually be constructed, and the correct error correction code can be recovered using error correction decoding. Furthermore, the lost or erroneous data can be recovered.

[0088] S1006: Using the check data corresponding to the m data segments and the data segments as the error correction codewords corresponding to the data segments, an error correction result vector is generated based on the check matrix and the error correction codewords corresponding to the data segments, and the decoding result of the error correction codewords corresponding to the data segments is determined based on the error correction result vector, where m > 1 and m is an integer.

[0089] Specifically, determining the decoding result of the error correction codeword corresponding to the data segment based on the error correction result vector includes: when the error correction result vector is a zero vector, determining the decoding result as the data obtained by concatenating the m data segments; when the error correction result vector is not a zero vector, iteratively adjusting the error correction codeword corresponding to the data segment until the adjusted error correction codeword and the error correction result vector generated by the parity check matrix are both zero vectors, and determining the decoding result as the data corresponding to the information bits in the adjusted error correction codeword. In the embodiments of this specification, generating the error correction result vector based on the parity check matrix and the error correction codeword corresponding to the data segment specifically means performing matrix multiplication between the error correction codeword corresponding to the data segment and the parity check matrix to generate the error correction result vector. When the error correction result vector is a zero vector, it indicates that the currently constructed error correction codeword has no errors, therefore the m data segments can be directly used as the decoding result of the error correction codeword corresponding to the data segment. If the error correction result vector is not zero, it indicates that the currently constructed error correction codeword has errors and needs to be further decoded using an error correction decoding algorithm. The method for cyclically adjusting and repeatedly verifying the error correction codeword corresponding to the fragmented data group involved in the embodiments of this specification may include bit reversal algorithm or belief propagation algorithm, which have been discussed in detail in related technologies.

[0090] Figure 10 The embodiments shown are for illustrative purposes only. Figure 2 The decoding method embodiment corresponding to the encoding method embodiment has technical details regarding its encoding portion in... Figure 2 The embodiments are described in detail, and will not be repeated here.

[0091] This specification presents a decoding method that implements a multi-disk data protection and data storage scheme based on erasure coding, to replace the multi-disk disaster recovery function of the original erasure coding storage scheme. Since the multi-disk data protection function is no longer implemented by erasure coding, this specification effectively eliminates the requirement for the server host to independently complete erasure coding encoding / decoding, thus eliminating the need for settings such as... Figure 1The document describes a series of erasure coding-related functions and implementations, including erasure coding encoders / decoders and code construction, which reduce the costs associated with independently setting up erasure coding, such as the number of functional units, computing resources, and code construction design. Furthermore, it replaces the erasure coding data protection mechanism with the error correction coding data protection mechanism. Since the error correction code's bitrate is close to the Shannon limit and higher than the erasure coding's bitrate, less redundancy is required to achieve the same data recovery capability, saving more storage space. Additionally, the embodiments in this specification decompose the traditional error correction coding process into multiple segmented steps involving the calculation of multiple data fragments with corresponding information bit submatrices and parity bit submatrices. This process generates a large number of intermediate calculation results, such as subvector features. Since each storage disk also needs to perform single-disk error correction coding on its stored data to form single-disk data protection capability, it makes it possible for the multi-disk error correction coding involved in the embodiments of this specification to reuse the same encoders / decoders and intermediate calculation results as the native single-disk error correction coding of each storage disk. This significantly reduces the hardware, software, and computational costs of implementing a joint data protection scenario.

[0092] Optionally, the m data shards are derived from m corresponding user data entries, each shard being one of the shards obtained by splitting its corresponding user data. Each of the m user data entries is split into m parts. The m user data entries are stored on corresponding user data storage disks. Each user data storage disk also stores verification data generated from the feature vector corresponding to its stored user data and the inverse matrix of the verification sub-matrix. This verification data, together with its stored user data, constitutes the error correction codeword for that user data. The feature vector corresponding to the user data is obtained by summing the m sub-feature vectors corresponding to the m fragments of the user data. The step of reading the m fragments of the data group from the m user data storage disks one by one includes: reading the m user data and their corresponding m verification data from the m user data storage disks respectively; performing error correction decoding based on the verification matrix, the m user data and their corresponding m verification data to obtain the m corrected data corresponding to the m user data respectively; and obtaining the m fragments of the data group belonging to the fragments from the m corrected data.

[0093] by Figure 8The embodiments of this specification will be introduced using an example. Suppose that it is now necessary to decode the error correction codewords corresponding to the fragmented data group g. Then, it is necessary to first obtain the complete user data A, B, C, D, and the corresponding check data PA, PB, PC, PD from storage disks 1-4 respectively. Then, based on A and PA, B and PB, C and PC, and D and PD respectively, construct the four error correction codewords corresponding to the user data A, B, C, D respectively, and simultaneously perform error correction decoding on these four error correction codewords to generate four corrected data corresponding to the user data A, B, C, D respectively. For example, when there are no errors in the four error correction codewords, the four corrected data are the user data A, B, C, D themselves. Based on the corrected data, it is further split according to the splitting principle to obtain 4 fragments belonging to the fragmented data group g. For example, when the 4 corrected data are user data A, B, C, and D themselves, user data A is split to obtain A1_g, user data B is split to obtain B2_g, user data C is split to obtain C3_g, and user data D is split to obtain D4_g, so as to finally obtain 4 fragments belonging to the fragmented data group g.

[0094] This specification's embodiments combine the error correction capabilities of multiple disks for sharded data groups with the error correction capabilities of a single disk for user data. First, the error correction capabilities of each user data storage disk are used to perform single-disk error correction decoding on the sharded data belonging to the sharded data group. Then, error correction decoding based on the sharded data group is performed, thus forming a joint error correction and decoding scheme, significantly improving the system-level data error correction and data recovery capabilities. Of course, in addition to utilizing the error correction capabilities of the user data storage disks beforehand, the error correction capabilities of the verification data storage disk can also be used in the same way to perform single-disk error correction decoding on the verification data obtained from the verification data storage disk, thereby further improving the system-level data error correction capabilities. Figure 8 As shown, before obtaining Pg, the error correction code words are constructed using Pg, Py, Po, Pb and PP stored on storage disk 5 and the error correction code words are decoded. The corrected Pg is then obtained from the corrected error correction code words.

[0095] Figure 11 This is a schematic diagram of an encoding system provided in this specification according to an exemplary embodiment. Figure 11 As shown, the system includes: a user data storage disk and a verification data storage disk deployed on at least one server, wherein the user data storage disk is used to store user data, and the verification data storage disk is used to store verification data; wherein, any server, any user data storage disk, or any verification data storage disk is equipped with an error correction code encoder, the error correction code encoder being used to execute the method described in any of the encoding method embodiments involved in this specification. Figure 11As shown, there are m user data storage disks and k verification data storage disks. These storage disks can be deployed on the same server or on multiple different servers, where m > 1 and m is an integer, and k ≥ 1 and k is an integer.

[0096] Figure 12 This is a schematic diagram of a decoding system provided in this specification according to an exemplary embodiment. Figure 12 As shown, the system includes: a user data storage disk and a verification data storage disk deployed on at least one server; the user data storage disk is used to store user data, and the verification data storage disk is used to store verification data; wherein, any server, any user data storage disk, or any verification data storage disk is equipped with an error correction code decoder, and the error correction code decoder is used to execute the method described in any of the decoding method embodiments involved in this specification. Figure 12 As shown, there are m user data storage disks and k verification data storage disks. These storage disks can be deployed on the same server or on multiple different servers, where m > 1 and m is an integer, and k ≥ 1 and k is an integer.

[0097] Figure 13 This is a schematic diagram of the structure of a device provided in this specification according to an exemplary embodiment. Please refer to... Figure 13 At the hardware level, the device includes a processor 1302, an internal bus 1304, a network interface 1306, memory 1308, and non-volatile memory 1310, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 1302 can read the corresponding computer program from the non-volatile memory 1310 into memory 1308 and then run it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0098] like Figure 14 As shown, Figure 14 This is a block diagram of an encoding device provided in this specification according to an exemplary embodiment, the device being applied to, for example... Figure 1 The error correction code encoder shown; the device includes:

[0099] The fragmented data determination unit 1401 is used to determine a fragmented data group containing m fragmented data, wherein the m fragmented data are stored by m user data storage disks that correspond one-to-one; wherein the m fragmented data correspond one-to-one with the m information bit submatrices split from the user data submatrix, the user data submatrix belongs to the check matrix used for error correction coding and corresponds to the information bit of the check matrix, and the check matrix also contains a check bit submatrix corresponding to the check bit.

[0100] The sub-vector feature acquisition unit 1402 is used to acquire m sub-feature vectors corresponding to the m data segments respectively, and sum them to obtain the feature vector corresponding to the data segment group; wherein, the sub-feature vector corresponding to any data segment is generated based on the data segment and its corresponding information position sub-matrix.

[0101] The verification data generation unit 1403 is used to generate verification data corresponding to the segmented data group based on the feature vector corresponding to the segmented data group and the inverse matrix of the verification position submatrix. The verification data and the m segmented data together constitute the error correction codeword corresponding to the segmented data group, where m>1 and m is an integer.

[0102] Optionally, the sub-feature vector corresponding to any of the data shards is generated and provided by the user data storage disk that stores the data shards.

[0103] Optionally, each user data storage disk maintains a shared memory space for caching its own generated sub-feature vectors, and the m user data storage disks obtain each other's generated sub-feature vectors through their respective maintained shared memory spaces.

[0104] Optionally, the m data segments are obtained by splitting the same user data into m parts.

[0105] Optionally, the m data segments are derived from m user data entries, with each data segment being one of the data segments obtained by splitting its corresponding user data.

[0106] Optionally, each of the m user data entries is split into m parts; the device further includes:

[0107] m verification data generation units 1404 are used to generate m verification data corresponding to m data fragments, wherein each data fragment contains m data fragments from the m user data, and the data fragments contained in different data fragments are different from each other.

[0108] Optionally, the m verification data are stored on a verification data storage disk that is different from the user data storage disk. This verification data storage disk also stores special verification data obtained by error correction encoding of the m verification data based on the verification matrix, and the m verification data and the special verification data together constitute the error correction codewords corresponding to the m verification data.

[0109] Optionally, each of the m user data entries is split into m parts; the m user data entries are stored by m corresponding user data storage disks, and each user data storage disk also stores the feature vector corresponding to its own stored user data and the inverse matrix of the check sub-matrix, which together with the user data it stores constitute the error correction codeword for that user data. The feature vector corresponding to its own stored user data is obtained by summing the m sub-feature vectors corresponding to the m fragments of the user data.

[0110] The sub-vector feature acquisition unit 1402 is specifically used to: determine the user data to which any of the data segments belong, and acquire the sub-feature vector generated for any of the data segments in order to generate the feature vector corresponding to the user data.

[0111] Optionally, the m user data entries belong to the same user data group, and different user data groups each contain m user data entries. The user data entries contained in different user data groups are either different from each other or partially the same.

[0112] Optionally, the m user data storage disks and / or the k verification data storage disks, which are different from the user data storage disks, are deployed in the same or different servers, where k ≥ 1 and k is an integer.

[0113] Optional, also includes:

[0114] The verification data storage unit 1405 is used to provide the verification data corresponding to the fragmented data group to any verification data storage disk that is different from the user data storage disk for storage; or, to provide the k fragmented verification data divided from the verification data corresponding to the fragmented data group to k corresponding verification data storage disks for storage, where k≥1 and k is an integer.

[0115] like Figure 15 As shown, Figure 15 This is a block diagram of a decoding apparatus provided in this specification according to an exemplary embodiment. The apparatus is applied to an error correction code decoder; the apparatus includes:

[0116] The fragmented data acquisition unit 1501 is used to read the m fragmented data contained in the fragmented data group from the m user data storage disks one by one; wherein the m fragmented data corresponds one-to-one with the m information bit submatrices split from the user data submatrix, the user data submatrix belongs to the check matrix used for error correction coding and corresponds to the information bit of the check matrix, and the check matrix also contains check bit submatrices corresponding to the check bits.

[0117] The verification data acquisition unit 1502 is used to acquire the verification data corresponding to the segmented data group. The verification data is generated based on the feature vector corresponding to the segmented data group and the inverse matrix of the verification position submatrix. The feature vector corresponding to the segmented data group is obtained by summing the m sub-feature vectors corresponding to the m segmented data respectively. The sub-feature vector corresponding to any segmented data is generated based on any segmented data and its corresponding information position submatrix.

[0118] The decoding unit 1503 is used to take the check data corresponding to the m data segments and the data segments as the error correction code words corresponding to the data segments, generate an error correction result vector based on the check matrix and the error code words corresponding to the data segments, and determine the decoding result of the error correction code words corresponding to the data segments based on the error correction result vector, where m > 1 and m is an integer.

[0119] Optionally, the decoding unit 1503 is specifically used for:

[0120] When the error correction result vector is a zero vector, the decoding result is determined to be the data obtained by concatenating the m data segments;

[0121] If the error correction result vector is not zero, the error correction codewords corresponding to the data fragments are cyclically adjusted until the adjusted error correction codewords and the error correction result vector generated by the parity check matrix are zero. The decoding result is then determined as the data corresponding to the information bits in the adjusted error correction codewords.

[0122] Optionally, the m data segments are derived from m corresponding user data entries, each segment being one of the segments obtained by splitting its corresponding user data. Each of the m user data entries is split into m parts. The m user data entries are stored on corresponding user data storage disks. Each user data storage disk also stores verification data generated by the feature vector corresponding to its stored user data and the inverse matrix of the verification sub-matrix. This verification data, together with its stored user data, constitutes the error correction codeword for that user data. The feature vector corresponding to its stored user data is obtained by summing the m sub-feature vectors corresponding to the m segments of the user data. The segment data acquisition unit 1501 is specifically used for:

[0123] The m user data records and their corresponding m verification data are read from the m user data storage disks respectively; error correction decoding is performed based on the verification matrix, the m user data records and their corresponding m verification data records to obtain m corrected data records corresponding to the m user data records respectively; m fragment data records belonging to the fragment data group are obtained from the m corrected data records.

[0124] Accordingly, this specification also provides an apparatus comprising a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the encoding or decoding methods provided in all the above method embodiments.

[0125] Accordingly, this specification also provides a computer-readable storage medium having executable instructions stored thereon; wherein, when executed by a processor, the instructions implement the steps of the encoding or decoding methods provided in all the above-described method embodiments.

[0126] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0127] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0128] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0130] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0133] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0134] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0135] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An encoding method, comprising: A data fragment group containing m data fragments is determined, wherein each of the m data fragments is stored by a corresponding m user data storage disk; wherein each of the m data fragments corresponds one-to-one with an m information bit submatrix split from a user data submatrix, the user data submatrix belongs to a check matrix for error correction coding and corresponds to the information bit of the check matrix, and the check matrix also contains a check bit submatrix corresponding to the check bit. Obtain the m sub-feature vectors corresponding to the m data segments respectively, and sum them to obtain the feature vector corresponding to the data segment group; wherein, the sub-feature vector corresponding to any data segment is generated based on any data segment and its corresponding information position submatrix; Based on the feature vector corresponding to the data segment and the inverse matrix of the parity bit submatrix, the parity data corresponding to the data segment is generated, and the parity data and the m data segments together constitute the error correction codeword corresponding to the data segment, where m>1 and m is an integer.

2. The method according to claim 1, wherein the sub-feature vector corresponding to any fragment of data is generated and provided by the user data storage disk storing the any fragment of data.

3. The method according to claim 2, wherein each user data storage disk maintains a shared memory space for caching its own generated sub-feature vectors, and the m user data storage disks obtain each other's generated sub-feature vectors through the shared memory space maintained by each of them.

4. The method according to claim 1, wherein the m data fragments are obtained by splitting the same user data into m parts; or, The m data segments are derived from m user data entries, and each data segment is one of the data segments obtained by splitting its corresponding user data.

5. The method according to claim 4, wherein each piece of user data in the m pieces of user data is split into m parts; the method further includes: Generate m data segments corresponding to m verification data segments, where each data segment contains m data segments derived from the m user data segments, and the data segments contained in different data segments are different from each other.

6. The method according to claim 5, wherein the m verification data are stored on a verification data storage disk that is different from the user data storage disk, the verification data storage disk further stores special verification data obtained by error correction encoding of the m verification data based on the verification matrix, and the m verification data and the special verification data together constitute the error correction codewords corresponding to the m verification data.

7. The method according to claim 4, wherein each piece of user data in the m pieces of user data is split into m parts; the m pieces of user data are stored by m user data storage disks that correspond one-to-one, and each user data storage disk also stores check data generated by the feature vector corresponding to the user data it stores and the inverse matrix of the check position submatrix, the check data and the user data it stores together constitute the error correction codeword for the user data, and the feature vector corresponding to the user data it stores is obtained by summing the m sub-feature vectors corresponding to the m pieces of data split from the user data; Obtaining the sub-feature vector corresponding to any fragment of data in the fragmented data group includes: Determine the user data to which any of the data segments belong, and obtain the sub-feature vector generated for that data segment in order to generate the feature vector corresponding to that user data.

8. The method according to claim 4, wherein the m user data belong to the same user data group, different user data groups each contain m user data, and the user data contained in different user data groups are different from each other or partially the same.

9. The method according to claim 1, further comprising: The verification data corresponding to the fragmented data group is provided to any verification data storage disk that is different from the user data storage disk for storage; or, The k fragmented verification data corresponding to the fragmented data group are provided to the k corresponding verification data storage disks for storage, where k≥1 and k is an integer.

10. A decoding method, comprising: Each of the m user data storage disks reads the m fragments of data contained in the fragmented data group in a one-to-one correspondence; wherein, the m fragments of data correspond one-to-one with the m information bit submatrices split from the user data submatrices, the user data submatrices belong to the check matrix used for error correction coding and correspond to the information bits of the check matrix, and the check matrix also contains check bit submatrices corresponding to the check bits. Obtain the verification data corresponding to the data shard group. The verification data is generated based on the feature vector corresponding to the data shard group and the inverse matrix of the verification position submatrix. The feature vector corresponding to the data shard group is obtained by summing the m sub-feature vectors corresponding to the m data shards respectively. The sub-feature vector corresponding to any data shard is generated based on any data shard and its corresponding information position submatrix. Using the check data corresponding to the m data segments and the data segments as the error correction codewords corresponding to the data segments, an error correction result vector is generated based on the check matrix and the error correction codewords corresponding to the data segments, and the decoding result of the error correction codewords corresponding to the data segments is determined based on the error correction result vector, where m > 1 and m is an integer.

11. An encoding system, comprising: A user data storage disk and a verification data storage disk are deployed on at least one server, wherein the user data storage disk is used to store user data and the verification data storage disk is used to store verification data; In this embodiment, an error correction code encoder is deployed in any server, any user data storage disk, or any verification data storage disk, and the error correction code encoder is used to perform the method as described in any one of claims 1-9.

12. A decoding system, comprising: User data storage disks and verification data storage disks are deployed on at least one server; The user data storage disk is used to store user data, and the verification data storage disk is used to store verification data. In this configuration, an error correction code decoder is deployed in any server, any user data storage disk, or any verification data storage disk, and the error correction code decoder is used to perform the method as described in claim 10.

13. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-10 by executing the executable instructions.

14. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-10.

Images