Encoding and decoding method and system of long chain error correction code and readable storage medium

Abstract

The application provides a long-chain error correction code encoding and decoding method, a system and a readable storage medium, and the method comprises the following steps: S1, minimizing single error cross-cabinet recovery cost through quantization, selecting a suitable data grouping strategy and data placement strategy, and determining encoding parameters; S2, sequentially encoding each dimension of three-dimensional encoding; S3, recovering data by using a decoding algorithm, preferentially using low-dimensional encoding for check recovery, and calling high-dimensional encoding for check recovery of each dimension when recovery fails. The application makes a quantitative analysis on the single error cross-cabinet recovery cost, places the cabinet level strip based on the analysis result, so that the strip crosses fewer cabinets; meanwhile, in the decoding stage of the long-chain error correction code, low-dimensional encoding is preferentially used for check recovery, high-dimensional encoding is called when recovery fails, the network topology is fully considered, the cross-cabinet recovery cost is reduced as much as possible, and then the recovery performance is improved.

Description

Technical Field

[0001] This invention relates to the field of cloud storage technology, and in particular to a long-chain erasure coding encoding and decoding method, system, and readable storage medium. Background Technology

[0002] In the field of cloud storage technology, erasure coding (EC) has been widely deployed in major cloud storage systems to provide high reliability. With the rapid growth of data volume, industry and academia are exploring the use of long-chain erasure coding, which has long stripes and near-optimal storage performance. For example, VAST is considering a stripe length of 154 (150+4).

[0003] Based on the generation process, long-chain erasure coding can be divided into two categories.

[0004] (1) Universal codes are extended from traditional short-chain erasure codes. They can be generated by directly configuring more data blocks and check blocks in the stripe, or by reasonably merging multiple short chains. RS-based and LRC-based are the two main types of universal codes because they are very flexible in parameter selection. RS-based TL codes place (n, k) RS codes in z racks, and use topological locality to place nk nodes in a single rack to minimize cross-rack recovery bandwidth. Data transmission is divided into cross-rack and intra-rack parts, where n is the length of a single erasure code stripe and k is the number of data blocks in a single erasure code stripe. LRC codes, on the other hand, use check locality to generate one local check block for every r data blocks, where r is the number of data block groups in a single two-dimensional erasure code stripe. One node is placed in a single rack to minimize the number of nodes involved in the recovery process. Reconstruction is divided into two processes: local recovery and global recovery.

[0005] (2) Dedicated codes are specifically designed for long chains. ECWide was the first systematic study to propose combined locality to minimize cross-rack network transmission traffic due to single errors. Placing (n, k, r) LRC codes in z racks is essentially a compromise between TL codes and LRC codes, with storage costs and recovery costs falling between the two schemes.

[0006] Existing methods, due to limitations in coding layout and data placement, involve a large number of racks in the construction of long-chain erasure coding, significantly increasing cross-rack overhead. Furthermore, existing methods only focus on single-error recovery, lacking consideration for concurrent errors, which often triggers global recovery across the entire stripe, severely degrading overall recovery performance. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, and readable storage medium for encoding and decoding long-chain erasure codes, which solves the problem of performance degradation in the recovery of erasure codes in existing long-chain scenarios.

[0008] To achieve the above objectives, the present invention provides an encoding and decoding method for long-chain erasure coding, comprising the following steps:

[0009] Includes the following steps:

[0010] S1. Minimize the cost of single-error cross-rack recovery through quantification, select appropriate data grouping and data placement strategies, and determine encoding parameters;

[0011] S2. Encode each dimension of the three-dimensional code sequentially;

[0012] S3. Use decoding algorithms to recover the data, prioritizing the use of low-dimensional encoding for verification and recovery. If recovery fails, then call high-dimensional encoding to verify and recover each dimension.

[0013] Optionally, S1 specifically includes:

[0014] S11. Based on the given long chain erasure code stripe length n and the number of data blocks k, determine the total number of check blocks m:

[0015] m=nk

[0016] S12, Quantize the number of blocks c in a single rack for a single erasure code stripe:

[0017] c≤mm l

[0018] Where m is the number of parity blocks in a single erasure code stripe, m l The number of check blocks in the first dimension of a single erasure code bar;

[0019] S13, Quantification of single-error cross-rack recovery cost λ:

[0020]

[0021] S14. Minimize the single-error cross-rack recovery cost λ, select appropriate data grouping strategy and data placement strategy, and determine encoding parameters.

[0022] Optionally, the encoding parameters include the number of racks spanned by a single erasure code stripe, the number of blocks placed in a single rack, and the number of check blocks for each dimension.

[0023] Optionally, S2 specifically includes:

[0024] For the first layer of coding generated between adjacent racks, XOR coding is selected to enable fast in-rack recovery for single errors; for the second layer of coding between racks, HH coding is selected to achieve rack-level fault tolerance with low storage cost; for the third layer of coding generated from all data blocks, RS coding is selected to provide global fault tolerance.

[0025] Optionally, S3 specifically includes:

[0026] S31. Recovery using first-dimensional checksums: For all first-level checksum chains, perform local repair based on the set of surviving first-dimensional checksum blocks.

[0027] S32. Recovery using second-dimensional verification: Perform verification grouping on each surviving second-dimensional verification block set, determine the number of faulty data blocks, and if the number of surviving second-dimensional verification blocks is not exceeded, perform joint recovery based on the surviving first-dimensional verification block set and the surviving second-dimensional verification block set.

[0028] S33. Recovery using third-dimensional checksum: After recovery using first-dimensional checksum and recovery using second-dimensional checksum fail, perform global recovery based on the set of surviving first-dimensional checksum blocks, the set of surviving second-dimensional checksum blocks, and the set of surviving third-dimensional checksum blocks.

[0029] Optionally, in S31, when a single block fails, local recovery is performed based on the set of surviving first-dimensional check blocks; when multiple blocks fail, all surviving first-dimensional check block sets are traversed firstly to restore the check chain of the single-block failure, and the remaining failed data blocks wait for the recovery of higher-dimensional encoding.

[0030] Optionally, before executing S31, S3 further includes:

[0031] After reaching the batch reconstruction cycle, relevant coding parameters and fault conditions are collected for the stripes that need to be restored for a given recovery round.

[0032] Optionally, for each data block successfully recovered, the corresponding data block is added to the lost dataset.

[0033] Based on the same inventive concept, this application also proposes a long-chain erasure coding encoding and decoding system, comprising:

[0034] The quantization module is configured to minimize the cost of single-error cross-rack recovery through quantization, select appropriate data grouping and data placement strategies, and determine encoding parameters.

[0035] The encoding module is configured to encode each dimension of the three-dimensional encoding sequentially.

[0036] The decoding module is configured to recover data using a decoding algorithm, prioritizing the use of low-dimensional encoding for verification and recovery, and only calling high-dimensional encoding to verify and recover each dimension if recovery fails.

[0037] Based on the same inventive concept, this application also proposes a readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the encoding and decoding method of long chain erasure coding as described above.

[0038] In the long-chain erasure coding encoding and decoding method, system, and readable storage medium provided by this invention, the cost of single-error cross-rack recovery is quantitatively analyzed, and rack-level striping is placed based on the analysis results, so that the stripes span fewer racks. At the same time, in the decoding stage of long-chain erasure coding, low-dimensional coding is used first for verification and recovery, and high-dimensional coding is called only if recovery fails. The network topology is fully considered to minimize the cross-rack recovery cost and thus improve recovery performance. Attached Figure Description

[0039] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention. Wherein:

[0040] Figure 1 A flowchart illustrating a long-chain erasure coding encoding and decoding method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of a long chain erasure coding encoding and decoding device provided in an embodiment of the present invention.

[0042] In the attached image:

[0043] 100 - Quantization module; 200 - Encoding module; 300 - Decoding module. Detailed Implementation

[0044] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clearly illustrate the purpose of the embodiments of this invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only used to complement the content disclosed in the specification, for those skilled in the art to understand and read, and are not intended to limit the implementation conditions of this invention. Any modifications to the structure, changes in proportions, or adjustments to the size, if they are the same as or similar to the effects and objectives achieved by this invention, should still fall within the scope of the technical content disclosed in this invention.

[0045] As used herein, the singular forms “a,” “an,” and “the” include plural objects unless otherwise expressly indicated. As used herein, the term “or” is generally used to include “and / or” unless otherwise expressly indicated. As used herein, the term “a number” is generally used to include “at least one” unless otherwise expressly indicated. As used herein, the term “at least two” is generally used to include “two or more” unless otherwise expressly indicated. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature.

[0046] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0047] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a long-chain erasure coding encoding and decoding method according to an embodiment of the present invention. This embodiment provides a long-chain erasure coding encoding and decoding method, including the following steps:

[0048] S1. Minimize the cost of single-error cross-rack recovery through quantification, select appropriate data grouping and data placement strategies, and determine encoding parameters;

[0049] S2. Encode each dimension of the three-dimensional code sequentially;

[0050] S3. Use decoding algorithms to recover the data, prioritizing the use of low-dimensional encoding for verification and recovery. If recovery fails, then call high-dimensional encoding to verify and recover each dimension.

[0051] This invention quantifies the cost of single-error cross-rack recovery and, based on the analysis results, places rack-level stripes to reduce the number of racks the stripes span. Simultaneously, during the decoding stage of long-chain erasure coding, low-dimensional coding is prioritized for verification and recovery, and high-dimensional coding is only called if recovery fails. This fully considers the network topology, minimizes the cost of cross-rack recovery, and thus improves recovery performance.

[0052] First, perform step S1 to minimize the cost of single-error cross-rack recovery through quantization, select appropriate data grouping and data placement strategies, and determine encoding parameters.

[0053] S1 specifically includes:

[0054] S11. Based on the given long chain erasure code stripe length n and the number of data blocks k, determine the total number of check blocks m:

[0055] m=nk

[0056] S12. Quantize the number of blocks c in a single erasure coding stripe placed in a single rack. Long-chain erasure coding uses the first dimension m. l Individual checks provide fast recovery from single errors, using higher-dimensional mm. l Each check provides rack-level fault tolerance. To provide rack-level fault tolerance, the number of blocks *c* placed in a single rack cannot exceed the number of high-dimensional checks.

[0057] c≤mm l

[0058] Where m is the number of parity blocks in a single erasure code stripe, m l The number of check blocks in the first dimension of a single erasure code bar;

[0059] S13. Quantify the cross-rack recovery cost λ for a single error. The number of racks spanned by a single group is determined by the number of groups m. l The single-fault cross-rack recovery cost λ is determined by the number of blocks c placed in a single rack, and is quantified as follows:

[0060]

[0061] S14. To minimize the cross-rack recovery cost for a single error, select appropriate data grouping and data placement strategies, and determine the encoding parameters. Since the total number of checks, m, is fixed, to minimize the cross-rack recovery cost λ, the number of checks at the first layer, m... l The number of blocks placed in a single rack, c, should be as equal as possible. Therefore, the rack placement strategy and rack grouping strategy can be determined based on the above principles, thus completing the parameter selection work for long chain erasure coding.

[0062] In this embodiment, the encoding parameters include the number of racks spanned by a single erasure code stripe, the number of blocks placed in a single rack, and the number of check blocks for each dimension.

[0063] Then, step S2 is executed, sequentially encoding each dimension of the three-dimensional encoding. Specifically, S2 includes:

[0064] For the first layer of coding generated between adjacent racks, XOR coding is selected to enable fast in-rack recovery for single errors; for the second layer of coding between racks, HH coding is selected to enable rack-level fault tolerance with low storage cost; for the third layer of coding generated from all data blocks, RS coding is selected to provide global fault tolerance.

[0065] Finally, step S3 is executed, using a decoding algorithm to recover the data. Low-dimensional encoding is used first for verification and recovery; if recovery fails, high-dimensional encoding is then used to verify and recover each dimension. Specifically, S3 includes:

[0066] S31. Recovery using first-dimensional checksums: For all first-level checksum chains, perform local repair based on the set of surviving first-dimensional checksum blocks.

[0067] S32. Recovery using second-dimensional verification: Perform verification grouping on each surviving second-dimensional verification block set, determine the number of faulty data blocks, and if the number of surviving second-dimensional verification blocks is not exceeded, perform joint recovery based on the surviving first-dimensional verification block set and the surviving second-dimensional verification block set.

[0068] S33. Recovery using third-dimensional checksum: After recovery using first-dimensional checksum and recovery using second-dimensional checksum fail, perform global recovery based on the set of surviving first-dimensional checksum blocks, the set of surviving second-dimensional checksum blocks, and the set of surviving third-dimensional checksum blocks.

[0069] Before executing S31, S3 further includes:

[0070] After reaching the batch reconstruction cycle, relevant coding parameters and fault conditions are collected for the stripes that need to be restored for a given recovery round.

[0071] In this embodiment, S3 is the decoding algorithm, which needs to make full use of the advantages of layered coding. The input of the decoding algorithm includes the location vector F[t] of the data block failure (t is the number of failure data blocks) and the set of surviving data blocks D. S The set of surviving first-dimensional check blocks L, the set of surviving second-dimensional check blocks Q, and the set of surviving third-dimensional check blocks G are given. The output of the decoding algorithm is the lost dataset D. L .

[0072] First, execute S31, using first-dimensional parity checking for recovery. First, perform L-partial repair (XOR-based single-block repair) on all first-level parity chains. When a single block fails, the first-level L-parity chain can perform partial recovery; when multiple blocks fail, first traverse all first-level L-parity chains, recovering the parity chains with single-block failures, while the remaining failed blocks await recovery via higher-dimensional encoding. During single-block recovery, the input to the L-parity chain decoding is D. tempThe output is L_Parity_Decoding(D temp For each data block successfully recovered, the corresponding data block is added to the lost dataset D. L In the middle, the number of faulty blocks t is updated to t-1.

[0073] D temp =D s ∈XORGroup j ∪L j

[0074] D L +=L_Parity_Decoding(D temp )

[0075] Then execute S32, using the second-dimensional checksum for recovery. For each Q-check group, check t; if it does not exceed the number of Q-check blocks, perform L and Q joint recovery (a combination of XOR-based and Hitchhiker (HH) based repair). In a single Q-check chain, if the number of faulty blocks is less than the total number of L and Q checks, recovery is successful; different Q-check chains can perform synchronous recovery. During LQ check chain decoding, the input for joint decoding is D. temp The output is LQ_Parity_Joint_Decoding(D temp Add the recovered data blocks to the lost dataset D. L In the process, the number of faulty blocks t is also updated accordingly.

[0076] D temp =D s ∈HHGroup j ∪Q j ∪L

[0077] D L +=LQ_Parity_Joint_Decoding(D temp )

[0078] Finally, S33 is executed, using third-dimensional verification for recovery. If the above recovery methods fail, global recovery based on L, Q, and G (a combination of XOR-based, HH-based, and RS-based recovery) will be triggered. If decoding is successful, the recovered lost dataset D is output. L If decoding still fails, it is declared that the number of faults exceeds the fault tolerance capacity, and data is lost.

[0079] The above decoding algorithm fully leverages the advantages of layered encoding. Its core idea is bottom-up, prioritizing low-dimensional encoding for recovery, and only resorting to higher-dimensional encoding if recovery fails. This significantly reduces the cost of cross-rack recovery in multi-error scenarios. Simultaneously, it fully considers network topology to minimize cross-rack recovery costs, thereby improving recovery performance.

[0080] Please refer to Figure 2 Based on the same inventive concept, embodiments of the present invention also provide a long-chain erasure coding encoding and decoding system, comprising:

[0081] The quantization module 100 is configured to minimize the cost of single-error cross-rack recovery through quantization, select appropriate data grouping and data placement strategies, and determine encoding parameters.

[0082] Encoding module 200 is configured to encode each dimension of the three-dimensional encoding sequentially.

[0083] The decoding module 300 is configured to recover data using a decoding algorithm, prioritizing the use of low-dimensional encoding for verification and recovery, and only calling high-dimensional encoding to verify and recover each dimension if recovery fails.

[0084] Based on the same inventive concept, this application also proposes a readable storage medium storing a computer program thereon, which, when executed by a processor, can implement the encoding and decoding method of long chain erasure coding as described above in the above feature description.

[0085] The readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device, such as, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer programs described herein can be downloaded from the readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. Each computing / processing device's network adapter card or network interface receives the computer program from the network and forwards it for storage in a readable storage medium within the respective computing / processing device. The computer program used to perform the operations of this invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from a computer program. These electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present invention.

[0086] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by a computer program. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer programs can also be stored in a readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the readable storage medium storing the computer program comprises an article of manufacture including instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0087] A computer program may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the computer program executing on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0088] In summary, the embodiments of the present invention provide a long-chain erasure coding encoding and decoding method, system, and readable storage medium. By quantitatively analyzing the cost of single-error cross-rack recovery, rack-level striping is placed based on the analysis results, so that the stripes span fewer racks. At the same time, in the decoding stage of long-chain erasure coding, low-dimensional coding is used first for verification and recovery, and high-dimensional coding is called only if recovery fails. The network topology is fully considered to minimize the cross-rack recovery cost and thus improve recovery performance.

[0089] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for encoding and decoding long-chain erasure codes, characterized in that, Includes the following steps: S1. Minimize the cost of single-error cross-rack recovery through quantification, select appropriate data grouping and data placement strategies, and determine encoding parameters; S2. Encode each dimension of the three-dimensional code sequentially; S3. Use decoding algorithms to recover the data, prioritizing the use of low-dimensional encoding for verification and recovery. If recovery fails, then call high-dimensional encoding to verify and recover each dimension. Specifically, S1 includes: S11. Based on the given long chain erasure code stripe length n and the number of data blocks k, determine the total number of check blocks m: S12, Quantize the number of blocks c in a single rack for a single erasure code stripe: Where m is the number of parity blocks in a single erasure code stripe. The number of check blocks in the first dimension of a single erasure code bar; S13, Quantification of single-error cross-rack recovery cost λ: S14. Minimize the single-error cross-rack recovery cost λ, select appropriate data grouping strategy and data placement strategy, and determine the encoding parameters; S3 specifically includes: S31. Recovery using first-dimensional checksums: For all first-level checksum chains, perform local repair based on the set of surviving first-dimensional checksum blocks. S32. Recovery using second-dimensional verification: Perform verification grouping on each surviving second-dimensional verification block set, determine the number of faulty data blocks, and if the number of surviving second-dimensional verification blocks is not exceeded, perform joint recovery based on the surviving first-dimensional verification block set and the surviving second-dimensional verification block set. S33. Recovery using third-dimensional checksum: After recovery using first-dimensional checksum and recovery using second-dimensional checksum fail, perform global recovery based on the set of surviving first-dimensional checksum blocks, the set of surviving second-dimensional checksum blocks, and the set of surviving third-dimensional checksum blocks.

2. The encoding and decoding method for long-chain erasure coding according to claim 1, characterized in that, The encoding parameters include the number of racks spanned by a single erasure code stripe, the number of blocks placed in a single rack, and the number of check blocks for each dimension.

3. The encoding and decoding method for long-chain erasure coding according to claim 1, characterized in that, S2 specifically includes: For the first layer of coding generated between adjacent racks, XOR coding is selected to enable fast in-rack recovery for single errors; for the second layer of coding between racks, HH coding is selected to achieve rack-level fault tolerance with low storage cost; for the third layer of coding generated from all data blocks, RS coding is selected to provide global fault tolerance.

4. The encoding and decoding method for long-chain erasure coding according to claim 1, characterized in that, In S31, when a single block fails, local recovery is performed based on the set of surviving first-dimensional check blocks; when multiple blocks fail, all surviving first-dimensional check block sets are traversed firstly to restore the check chain of the single-block failure, and the remaining faulty data blocks wait for the recovery of higher-dimensional encoding.

5. The encoding and decoding method for long-chain erasure coding according to claim 1, characterized in that, Before executing S31, S3 further includes: After reaching the batch reconstruction cycle, relevant coding parameters and fault conditions are collected for the stripes that need to be restored for a given recovery round.

6. The encoding and decoding method for long-chain erasure coding according to claim 1, characterized in that, For each data block that is successfully recovered, the corresponding data block is added to the lost dataset.

7. A long-chain erasure coding encoding and decoding system, characterized in that, include: The quantization module is configured to minimize the cost of single-error cross-rack recovery through quantization, select appropriate data grouping and data placement strategies, and determine encoding parameters. The encoding module is configured to encode each dimension of the three-dimensional encoding sequentially. The decoding module is configured to use a decoding algorithm to recover the data, prioritizing the use of low-dimensional encoding for verification and recovery, and only calling high-dimensional encoding to verify and recover each dimension if recovery fails. Specifically, the quantization module is configured as follows: Given the long chain erasure code stripe length n and the number of data blocks k, determine the total number of check blocks m: The number of blocks c that a single erasure code stripe can be placed in a single rack: Where m is the number of parity blocks in a single erasure code stripe. The number of check blocks in the first dimension of a single erasure code bar; Quantifying the cost of cross-rack recovery for a single error: λ To minimize the single-error cross-rack recovery cost λ, select appropriate data grouping and data placement strategies, and determine encoding parameters; The decoding module is specifically configured as follows: Recovery using first-dimensional checksums: For all first-level checksum chains, perform local repair based on the set of surviving first-dimensional checksum blocks; Recovery using second-dimensional checksums: Each surviving set of second-dimensional checksums is grouped for checksums, the number of faulty data blocks is determined, and if the number of surviving second-dimensional checksums is not exceeded, joint recovery is performed based on the surviving sets of first-dimensional checksums and second-dimensional checksums. Recovery using third-dimensional checksum: After recovery using first-dimensional checksum and recovery using second-dimensional checksum fail, global recovery is performed based on the set of surviving first-dimensional checksum blocks, the set of surviving second-dimensional checksum blocks, and the set of surviving third-dimensional checksum blocks.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the encoding and decoding method of the long chain erasure code according to any one of claims 1-6.

Images