Data storage fault-tolerant redundancy method, system, computer device and storage medium

Abstract

The present invention provides a data storage fault-tolerant redundancy method, system, computer device and storage medium, which belongs to the field of computer data storage technology. In the method, multiple data slices in a data stripe are evenly divided into multiple subgroups in sequence; when a data slice in a target subgroup fails, the remaining data slices in the target subgroup are read to repair the failed data slices; the data slices of all subgroups are encoded to obtain multiple cross-group check slices, and when more than one data slice in the target subgroup fails, the failed data slices are repaired according to the multiple cross-group check slices and the data slices corresponding to the remaining subgroups. In the case where multiple blocks of data are damaged simultaneously in the same stripe, the present invention can avoid falling back to global RS recovery to maintain the advantage of local repair.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer data storage, and in particular relates to a data storage fault-tolerant redundancy method, system, computer equipment and storage medium. Background Art

[0002] With the rapid development of information technology, the amount of data that needs to be stored continues to grow. Large-scale distributed storage clusters have gained widespread application, and the use of inexpensive storage devices has also gradually become widespread. However, as storage scale increases and the number of inexpensive disks increases, the probability of disk failure in distributed storage clusters also increases, resulting in severe challenges to the reliability of storage systems.

[0003] To ensure storage system reliability, most storage systems use fault-tolerant technologies, such as LRC codes (locally repairable codes). LRC codes, based on Reed-Solomon codes (RS codes), group the coded blocks within a stripe into groups, each called a local group. Blocks within each local group are then encoded to generate an additional local parity block. If a single block is lost, the lost data can be recovered by simply decoding it with data from other blocks within the local group.

[0004] However, the classic single-layer LRC code targets "single-shard" failures. If multiple blocks of data are damaged simultaneously in the same stripe, it is necessary to fall back to global RS recovery, and the advantages of local repair cannot be maintained. Summary of the Invention

[0005] The present invention provides a data storage fault-tolerant redundancy method, system, computer device and storage medium to solve the problem in the prior art that when multiple blocks of data are damaged simultaneously in the same strip, global RS recovery must be performed and the advantages of local repair cannot be maintained.

[0006] In a first aspect, the present invention provides a data storage fault-tolerant redundancy method, comprising:

[0007] Divide multiple data shards within a data stripe into multiple subgroups evenly and sequentially;

[0008] When a data shard in the target subgroup fails, the remaining data shards in the target subgroup are read to repair the failed data shard.

[0009] The data shards of all subgroups are encoded to obtain multiple cross-group verification shards. When more than one data shard fails in the target subgroup, the failed data shards are repaired based on the multiple cross-group verification shards and the data shards corresponding to the remaining subgroups.

[0010] Optionally, when only one data shard in the target subgroup fails, reading the remaining data shards in the target subgroup to repair the failed data shard includes:

[0011] Perform finite field addition operations on all data slices in each subgroup to obtain local check slices for each subgroup;

[0012] The remaining data shards in the target subgroup and the local check shards of the target subgroup are read to repair the failed data shards in the target subgroup through finite field addition operations.

[0013] Optionally, encoding the data slices of all subgroups to obtain multiple cross-group check slices, and repairing the failed data slices based on the multiple cross-group check slices and the data slices corresponding to the remaining subgroups when more than one data slice in the target subgroup fails, includes:

[0014] Rearrange N data shards into N / K columns, with each column containing K data shards. Apply RS(K+P, K) encoding to each column's data shard vector to obtain P cross-group parity shards. N is the total number of data shards in a data stripe.

[0015] When more than or equal to 2 and less than or equal to P data shards fail simultaneously in the same subgroup, the failed data shards are repaired using P cross-group check shards and corresponding column data shards of other subgroups.

[0016] Optionally, the first aspect further includes:

[0017] If more than P data shards fail in the target subgroup, the data file is divided into multiple data blocks; each data block contains N data shards;

[0018] Apply RS (N+M+P+G to N data shards, M local check shards, and P cross-group check shards in the target data block. , N+M+P) encoding is performed to obtain G global check slices, so as to use the G global check slices to repair the failed data slices; where M is the total number of subgroups in a data stripe.

[0019] In a second aspect, the present invention provides a data storage fault-tolerant redundant system, comprising:

[0020] A first partitioning module is used to evenly divide multiple data fragments in a data stripe into multiple subgroups in sequence;

[0021] A first repair module is configured to read the remaining data shards in the target subgroup to repair the failed data shard when one data shard in the target subgroup fails;

[0022] The second repair module is used to encode the data fragments of all subgroups to obtain multiple cross-group verification fragments. When more than one data fragment in the target subgroup fails, the failed data fragments are repaired based on the multiple cross-group verification fragments and the data fragments corresponding to the remaining subgroups.

[0023] Optionally, the first repair module includes:

[0024] A finite field addition operation unit, configured to perform a finite field addition operation on all data slices in each subgroup to obtain a local check slice for each subgroup;

[0025] The first repair unit is configured to read the remaining data slices in the target subgroup and the local check slices of the target subgroup, so as to repair the failed data slices in the target subgroup through a finite field addition operation.

[0026] Optionally, the second repair module includes:

[0027] An encoding unit, configured to rearrange N data slices into N / K columns, with each column containing K data slices, and apply RS(K+P, K) encoding to each column of data slice vectors to obtain P cross-group parity slices, where N is the total number of data slices in a data stripe;

[0028] The second repair unit is used to repair the failed data shards by using P cross-group check shards and corresponding column data shards of other subgroups when there are greater than or equal to 2 and less than or equal to P data shards that fail at the same time in the same subgroup.

[0029] Optionally, the second aspect further includes:

[0030] The second partitioning module is configured to partition the data file into multiple data blocks when more than P data shards fail in the target subgroup; each data block contains N data shards;

[0031] The third repair module is used to apply RS (N+M+P+G) to N data fragments, M local check fragments and P cross-group check fragments in the target data block. , N+M+P) encoding is performed to obtain G global check slices, so as to use the G global check slices to repair the failed data slices; where M is the total number of subgroups in a data stripe.

[0032] In a third aspect, the present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the data storage fault-tolerant redundancy method described in the first aspect are implemented.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the data storage fault-tolerant redundancy method described in the first aspect are implemented.

[0034] The present invention provides a data storage fault-tolerant redundancy method, system, computer device, and storage medium. In the method, multiple data slices within a data stripe are evenly divided into multiple subgroups in sequence; when one data slice within a target subgroup fails, the remaining data slices within the target subgroup are read to repair the failed data slice; when more than one data slice within the target subgroup fails, the data slices of all subgroups are encoded to obtain multiple cross-group check slices, and the failed data slices are repaired based on the multiple cross-group check slices and the data slices corresponding to the remaining subgroups. In the case where multiple blocks of data are damaged simultaneously within the same stripe, the present invention does not need to fall back to global RS recovery, thereby maintaining the advantages of local repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A schematic diagram of a flow chart of a data storage fault-tolerant redundancy method provided by an embodiment of the present invention;

[0037] Figure 2 A structural diagram of a data storage fault-tolerant redundancy method provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a local repair process provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a global repair process provided by an embodiment of the present invention;

[0040] Figure 5 A structural diagram of a data storage fault-tolerant redundant system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0042] The implementation environment adopted by this invention is based on a CPU instruction set server as the encoding end. The computing power of the server cluster can support the multi-level encoding and decoding operations of the layered LRC erasure coding method, especially when processing large-scale data.

[0043] Regarding storage media, the present invention deployed 10 nodes, each equipped with four 8TB JBOD (Just a Bunch of Disks) disks, for a total storage capacity of 320TB. The JBOD configuration reduces data access latency by directly connecting the disks, while providing sufficient storage space to support the needs of a distributed storage system.

[0044] Example 1

[0045] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a data storage fault-tolerant redundancy method, comprising:

[0046] Step 101: Divide multiple data fragments in a data stripe into multiple subgroups evenly in sequence.

[0047] Read the data files uploaded by the user (can be files of any size, such as large video files, database backups, etc. "Large files" here refer to files larger than the size of a single data block, usually GB or even TB level), and split them into data blocks of equal length, recorded as Block0 to Block n-1 , n is the total number of data blocks. Each data block contains N data shards (that is, N data shards logically form a data block), and N data shards form a data stripe, recorded as D0 to D N-1 In this embodiment, the default setting is N=64 (i.e. D0 to D 63 ). Divide N data shards into M subgroups evenly in sequence, each subgroup contains N / M data shards. In this embodiment, the default setting is M=4 (i.e., M1, M2, M3, and M4).

[0048] Step 102: When a data slice in the target subgroup fails, the remaining data slices in the target subgroup are read to repair the failed data slice.

[0049] like Figure 3 As shown in FIG, when a single shard (single data shard) fails in a subgroup, a finite field addition operation is performed on all data shards in each subgroup to obtain a local check shard for each subgroup, that is, one subgroup has one local check shard.

[0050] The remaining data shards in the target subgroup and the local check shards of the target subgroup are read to repair the failed data shards in the target subgroup through finite field addition operations.

[0051] Data reconstruction is completed by reading only the remaining N / M-1 data shards within the subgroup and the local parity shard of the subgroup. The resulting data is the damaged original data shard, meaning that "reconstruction" refers to restoring the contents of the lost data shard.

[0052] After the N data slices in a data stripe are evenly divided into M subgroups in sequence, a finite field addition operation is performed on the N / M data slices in each subgroup to obtain M local check slices (i.e., redundant data), which are denoted as P0 to P M-1 This finite field addition operation is simple and efficient, with low computational overhead, and is suitable for fast repair of single-shard failures.

[0053] Assume shard D 17 Damaged, call the LRC-Tiered decoder to read the remaining 15 fragments in the same subgroup (D 16 ,D 18 -D 31 ) and local check fragment P1, and recover D through XOR operation 17 , and writes it back to the storage system. The repair bandwidth is only 16 data slices, which is much lower than the 64 slices of traditional RS encoding.

[0054] Step 103: Encode the data slices of all subgroups to obtain multiple cross-group verification slices. If more than one data slice fails in the target subgroup, repair the failed data slices based on the multiple cross-group verification slices and the data slices corresponding to the remaining subgroups.

[0055] For example, N data slices are rearranged into N / K columns, each column contains K data slices, and RS(K+P, K) encoding is applied to each column data slice vector to obtain P cross-group check slices, denoted as Q0 to Q P-1 ; Where N is the total number of data shards in a data stripe.

[0056] When there are more than or equal to 2 and less than or equal to P data shards that fail at the same time in the same subgroup, P cross-group check shards (Q0 to Q P-1 ) and the corresponding column data shards of other subgroups to repair the failed data shards and avoid global repair.

[0057] If two shards in the same subgroup fail at the same time (e.g. D 17 and D 23), using cross-group parity shards Q0 and Q1, combined with corresponding column data shards from other subgroups, to perform RS decoding and recovery column by column. This approach avoids global decoding, reducing repair latency and bandwidth overhead.

[0058] The parameters of RS(K+P, K) encoding can be adjusted, such as RS(8,4), but RS(4,2) is a compromise that can tolerate the failure of two fragments while maintaining low computational complexity.

[0059] Cross-group coding enhances local repair capabilities for multiple data shard failures, avoiding direct triggering of global decoding. Cauchy matrix RS coding is applied after column reorganization, and a table lookup (gflog / gfilog) converts multiplications into additions, reducing computational complexity.

[0060] The data storage fault-tolerant redundancy method provided in this embodiment further includes:

[0061] When more than P data slices fail in the target subgroup, the data file is divided into multiple data blocks. In this embodiment, there are n data blocks in total; each data block contains N data slices.

[0062] Apply RS (N+M+P+G to N data shards, M local check shards, and P cross-group check shards in the target data block. , N+M+P) encoding, G global check slices are obtained, and the G global check slices are used to repair the failed data slices; where M is the total number of subgroups in a data stripe; each data block includes N data slices and M+P check slices (P0 to P M-1 and Q0 to Q P-1 ), a total of N+M+P shards.

[0063] like Figure 4 As shown, in extreme cases (for example, more than two shards in the same subgroup are damaged and the cross-group parity shard is also damaged), local repair is not enough to recover the data, triggering global RS decoding:

[0064] The remaining 68 shards (including data and parity shards) are read, and the fault tolerance of RS (73,70) is leveraged to recover all damaged shards. This approach is suitable for scenarios where cross-group parity shards are also damaged. Although bandwidth and computational overhead are high, data integrity is guaranteed.

[0065] Parameterizing the data stripe size N, the number of subgroups M, the number of cross-group parity shards P, and the number of global parity shards G allows operations personnel to dynamically adjust based on cluster size, disk failure statistics, and business service level agreements (SLAs).

[0066] In this embodiment, the data stripe size N defaults to 64 and can be adjusted to 32 or 128, affecting storage efficiency and repair bandwidth. The number of subgroups M defaults to 4 and can be increased or decreased based on fault tolerance requirements. The number of cross-group parity shards P defaults to 2 and can be adjusted to 3 or more to improve multi-shard fault tolerance. The number of global parity shards G defaults to 3 and depends on reliability requirements. Flexible adjustment of these parameters allows for a balance between storage overhead, repair efficiency, and reliability.

[0067] In summary, this embodiment provides a data storage fault-tolerant redundancy method. A single shard failure only requires reading 15 / 64 (≈ 23%) of the data volume; compared to global RS recovery, it saves approximately 77% of bandwidth and I / O. Cross-group parity checks can withstand the concurrent failure of three data shards in the same group without triggering global decoding, improving the fault tolerance by 2-3 times compared to traditional local repair coding. Through parameterized design, the storage overhead can be smoothly adjusted between 14% and 20% to accommodate hot and cold data or varying reliability levels. Low implementation cost: The first layer is based on simple finite field addition operations, and the second layer uses only the small RS(4,2) matrix. Both encoding and decoding can be SIMD-accelerated. Highly compatible with existing LRC engines, it is easy to quickly implement in systems such as Hadoop HDFS, Ceph, and Erasure-Coding RAID. High scalability: This embodiment decouples block and node topology, allowing for flexible placement of parity blocks at the rack / cluster level, further reducing the probability of failure within the same fault domain.

[0068] Example 2

[0069] Based on the same inventive concept as Example 1, this embodiment provides a data storage fault-tolerant redundant system. Since the principle of solving the problem by this system is similar to the data storage fault-tolerant redundant method provided in the aforementioned Example 1, the implementation of this system can refer to the implementation of the data storage fault-tolerant redundant method provided in Example 1.

[0070] like Figure 5 As shown, the data storage fault-tolerant redundant system includes:

[0071] The first partitioning module 10 is used to evenly partition multiple data slices in a data stripe into multiple subgroups in sequence.

[0072] The first repair module 20 is configured to read the remaining data slices in the target subgroup to repair the failed data slice when one data slice in the target subgroup fails.

[0073] The second repair module 30 is used to encode the data slices of all subgroups to obtain multiple cross-group verification slices. When more than one data slice in the target subgroup fails, the failed data slices are repaired based on the multiple cross-group verification slices and the data slices corresponding to the remaining subgroups.

[0074] Exemplarily, the first repair module includes:

[0075] The finite field addition operation unit is used to perform a finite field addition operation on all data slices in each subgroup to obtain a local check slice of each subgroup.

[0076] The first repair unit is configured to read the remaining data slices in the target subgroup and the local check slices of the target subgroup, so as to repair the failed data slices in the target subgroup through finite field addition operations.

[0077] Exemplarily, the second repair module includes:

[0078] The encoding unit is used to rearrange N data slices into N / K columns, each column contains K data slices, and apply RS(K+P, K) encoding to each column of data slice vector to obtain P cross-group parity slices; where N is the total number of data slices in a data stripe.

[0079] The second repair unit is used to repair the failed data shards by using P cross-group check shards and corresponding column data shards of other subgroups when there are greater than or equal to 2 and less than or equal to P data shards that fail at the same time in the same subgroup.

[0080] Illustratively, the data storage fault-tolerant redundant system provided in this embodiment further includes:

[0081] The second partitioning module is used to partition the data file into multiple data blocks when more than P data slices fail in the target subgroup; each data block contains N data slices.

[0082] The third repair module is used to apply RS (N+M+P+G) to N data fragments, M local check fragments and P cross-group check fragments in the target data block. , N+M+P) encoding is performed to obtain G global check slices, so as to use the G global check slices to repair the failed data slices; where M is the total number of subgroups in a data stripe.

[0083] For more specific working processes of the above modules, please refer to the corresponding content disclosed in Example 1, which will not be repeated here.

[0084] Example 3

[0085] This embodiment provides a computer device, including a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the method for testing the response time of an analog input card described in Example 1 are implemented.

[0086] For more specific details about the above method, please refer to the corresponding content disclosed in Example 1, which will not be repeated here.

[0087] Example 4

[0088] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the method for testing the response time of an analog input card described in Example 1 are implemented.

[0089] For more specific details about the above method, please refer to the corresponding content disclosed in Example 1, which will not be repeated here.

[0090] Example 5

[0091] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the method for testing the response time of the analog input card described in Example 1 are implemented.

[0092] For more specific details about the above method, please refer to the corresponding content disclosed in Example 1, which will not be repeated here.

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments will be sufficient. The systems, devices, storage media, and computer program products disclosed in the embodiments correspond to the methods disclosed in the embodiments, so their descriptions are relatively simplified. For relevant details, refer to the method descriptions.

[0094] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention, or portions thereof.

[0095] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0096] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0097] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0098] The present invention has been described in detail above with reference to specific embodiments and exemplary examples. However, these descriptions should not be construed as limiting the present invention. Those skilled in the art will appreciate that various equivalent substitutions, modifications, or improvements may be made to the technical solutions and implementations of the present invention without departing from the spirit and scope of the present invention, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A data storage fault-tolerant redundancy method, characterized in that: include: Divide multiple data shards within a data stripe into multiple subgroups evenly and sequentially; If one data shard fails in the target subgroup, the remaining data shards in the target subgroup are read to repair the failed data shard. The data shards of all subgroups are encoded to obtain multiple cross-group check shards. If more than one data shard fails in the target subgroup, the failed data shards are repaired based on the multiple cross-group check shards and the data shards corresponding to the remaining subgroups. In the case that only one data shard in the target subgroup fails, reading the remaining data shards in the target subgroup to repair the failed data shard, including: performing a finite field addition operation on all data shards in each subgroup to obtain a local check shard of each subgroup; reading the remaining data shards in the target subgroup and the local check shard of the target subgroup to repair the failed data shard in the target subgroup through the finite field addition operation; The encoding of the data slices of all subgroups to obtain multiple cross-group check slices, and when more than one data slice fails in the target subgroup, repairing the failed data slices based on the multiple cross-group check slices and the data slices corresponding to the remaining subgroups, includes: rearranging N data slices into N / K columns, each column containing K data slices, and applying RS(K+P, K) encoding to each column of data slice vectors to obtain P cross-group check slices; wherein N is the total number of data slices in a data stripe; and when greater than or equal to 2 and less than or equal to P data slices fail simultaneously in the same subgroup, repairing the failed data slices using the P cross-group check slices and the corresponding column data slices of other subgroups; It also includes: when more than P data shards fail in the target subgroup, dividing the data file into multiple data blocks; each data block has N data shards; applying RS (N+M+P+G, N+M+P) encoding to the N data shards, M local check shards and P cross-group check shards in the target data block to obtain G global check shards, so as to use the G global check shards to repair the failed data shards; wherein M is the total number of subgroups in a data stripe.

2. A data storage fault-tolerant redundant system, characterized in that: include: A first partitioning module is used to evenly divide multiple data fragments in a data stripe into multiple subgroups in sequence; A first repair module is configured to read the remaining data shards in the target subgroup to repair the failed data shard when one data shard in the target subgroup fails; A second repair module is configured to encode the data shards of all subgroups to obtain multiple cross-group check shards. If more than one data shard in the target subgroup fails, the failed data shards are repaired based on the multiple cross-group check shards and the data shards corresponding to the remaining subgroups. The first repair module includes: a finite field addition operation unit, configured to perform a finite field addition operation on all data shards in each subgroup to obtain a local check shard of each subgroup; a first repair unit, configured to read the remaining data shards in the target subgroup and the local check shard of the target subgroup to repair the failed data shards in the target subgroup through the finite field addition operation; The second repair module includes: an encoding unit for rearranging N data slices into N / K columns, each column containing K data slices, and applying RS(K+P, K) encoding to each column of data slice vectors to obtain P cross-group parity slices; where N is the total number of data slices in a data stripe; a second repair unit for repairing the failed data slices using the P cross-group parity slices and corresponding column data slices of other subgroups when greater than or equal to 2 and less than or equal to P data slices fail simultaneously in the same subgroup; It also includes: a second partitioning module, used to divide the data file into multiple data blocks when more than P data slices fail in the target subgroup; each data block has N data slices; a third repair module, used to apply RS (N+M+P+G, N+M+P) encoding to the N data slices, M local check slices and P cross-group check slices in the target data block to obtain G global check slices, so as to use the G global check slices to repair the failed data slices; wherein M is the total number of subgroups in a data stripe.

3. A computer device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the data storage fault-tolerant redundancy method according to claim 1 are implemented.

4. A computer-readable storage medium, characterized in that Used to store computer programs; when the computer program is executed by the processor, the steps of the data storage fault-tolerant redundancy method according to claim 1 are implemented.

Images