A register-friendly and efficient XOR erasure coding method

Abstract

This invention relates to the field of erasure coding technology in computer distributed storage systems, and discloses a register-friendly, high-efficiency XOR erasure coding method, including the steps of splitting data blocks and parity blocks, calculating parity sub-blocks, and merging parity sub-blocks. Compared with existing coding methods, this invention minimizes the number of times data is stored from registers to memory or cache, meaning all data only needs to be stored from registers to memory once, reducing data storage latency; it also reduces the number of times data is read from memory or cache to registers by nearly half, improving data loading efficiency. This invention improves coding speed by reducing the number of times data is read and stored between registers and memory or cache during the coding process.

Description

Technical Field

[0001] This invention relates to the field of erasure coding technology in computer distributed storage systems, specifically to a register-friendly, high-efficiency XOR erasure coding method. Background Technology

[0002] Erasure coding and multiple replication are two of the most common redundancy techniques for data fault tolerance in storage systems. Replication technology stores multiple copies of the same data on different nodes. Replication technology is easy to implement and provides concurrent access for front-end users, but it incurs high additional storage overhead. Compared to replication technology, erasure coding can provide the same reliability, but with an order of magnitude lower additional storage overhead, making it widely used in large-scale storage systems to save storage costs. However, erasure coding-based fault tolerance techniques require encoding and computation of data blocks to obtain checksum blocks when writing data; and decoding and computation of surviving blocks to obtain faulty blocks during fault repair. Encoding and decoding computations not only reduce the speed of writing data and repairing, but also consume CPU and memory resources. Even the current state-of-the-art erasure coding library ISA-L has an encoding speed lower than that of memory I / O and Remote Direct Memory Access (RDMA). Therefore, improving the encoding and decoding speed of erasure coding is of great significance.

[0003] XOR-based erasure coding only requires XOR calculations during the encoding process, resulting in fast encoding speed. However, existing methods divide the encoding computation into multiple rounds. In each round, a data block and a parity block are read into a register, and then an XOR calculation is performed. But the size of a data block is usually much larger than the register size. Therefore, the intermediate results of each round are stored in memory or cache, and then reloaded into the register when needed, leading to frequent swapping of intermediate results in and out of the register. For example, The calculation is divided into three rounds. The first round calculates P0 = 0, and the second round calculates... The third round of calculation is Because registers cannot hold all the data required to compute P0, intermediate results of P0 need to be stored in memory or cache during the first and second rounds of computation, and read into registers during the second and third rounds. Therefore, improving register efficiency in the encoding computation process can significantly improve encoding speed. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a register-friendly and efficient XOR erasure coding method. The basic idea is to split the data block and the parity block according to the register size, so that the intermediate results of calculating the parity block can be stored in the register. The calculation is performed at the register granularity, avoiding swapping the intermediate results in and out of the register and memory or cache, ensuring the register-friendly nature of the encoding process. A corresponding encoding process is also designed to improve the encoding speed of erasure coding.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A register-friendly and efficient XOR erasure coding method is proposed, which uses a (k,m) XOR erasure coding matrix to encode k data blocks D0, D1, ..., D k-1 Encode the data to obtain m parity blocks P0, P1, ..., P m-1 In an XOR erasure coding system, each element in the encoding matrix is ​​either 0 or 1, and each parity block is the XOR sum of multiple data blocks. The entire encoding process only requires XOR calculations; it includes the following steps:

[0007] Step 1: Splitting data blocks and check blocks:

[0008] Let the check block P g The formula for calculating 0≤g≤m-1 is: Each data block 0≤v≤t-1 is split into n register-sized data sub-blocks. Will check block P g Divide into n register-sized parity sub-blocks P g0 ,P g1 ,…,P g(n-1) Each verification sub-block For each check subblock P gl If 0 ≤ l ≤ n-1, proceed to step two;

[0009] Step 2: Calculation of the verification sub-block:

[0010] Read data sub-blocks The parity subblock P is then transferred to register R1. gl The value is initialized to Each data sub-block is read When 1≤v≤t-1 is entered into register R2, process A is performed once: XOR the values ​​in registers R1 and R2, and store the XOR result in register R1;

[0011] After completing process A for t-1 times, the value in register R1 is stored in memory as the check sub-block P. gl ;

[0012] Complete all check subblocks P gl After calculating 0≤l≤n-1, proceed to step three;

[0013] During the entire encoding calculation process of the check sub-block, the intermediate results of the check sub-block are stored in the register until the calculation of the check sub-block is completed. This avoids the frequent swapping of intermediate results into and out of the register, thus improving register efficiency.

[0014] Step 3: Verify the merging of sub-blocks:

[0015] Sequentially assign n check sub-blocks P g0 ,P g1 ,…,P g(n-1) Sequentially written to contiguous address spaces in cache or memory, and merged into a parity block P. g .

[0016] Compared with the prior art, the beneficial technical effects of the present invention are:

[0017] The register-friendly, high-efficiency XOR erasure coding method of this invention includes splitting data blocks and parity blocks, calculating parity sub-blocks, and merging parity sub-blocks. Compared with existing coding methods, this invention minimizes the number of times data is stored from registers to memory or cache; that is, all data only needs to be stored from registers to memory once, reducing data storage latency. This invention also reduces the number of times data is read from memory or cache to registers by nearly half, improving data loading efficiency. This invention improves coding speed by reducing the number of times data is read and stored between registers and memory or cache during the encoding process. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the erasure coding calculation based on the coding matrix of the present invention;

[0019] Figure 2 This is a schematic diagram illustrating the splitting of the data block and the verification block in this invention;

[0020] Figure 3 This is a schematic diagram illustrating the initialization of the verification sub-block in this invention;

[0021] Figure 4 This is a schematic diagram of the XOR calculation of the present invention;

[0022] Figure 5 This is a schematic diagram of the verification sub-block storage of the present invention. Detailed Implementation

[0023] The register-friendly, high-efficiency XOR erasure coding method of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] Using a (12,6) XOR erasure coding matrix, the 12 data blocks D0, D1, D2, D3, D4, D5, D6, D7, D8, D9, D... 10 D 11 Encoding is performed to obtain six parity blocks P0, P1, P2, P3, P4, and P5. The size of the encoding matrix is ​​6×12, and each element in the encoding matrix is ​​either 0 or 1. Figure 1 This is a schematic diagram of the calculation based on the encoding matrix.

[0025] Existing technology:

[0026] Here is the verification block. If the check block P0 is calculated using the existing encoding method, the entire calculation is divided into seven rounds: P0 = 0, In the first to sixth rounds of computation, the intermediate results of the verification block P0 are stored in memory or cache. In the second to seventh rounds of computation, the intermediate results of P0 are read into registers. The amount of data transferred between the intermediate results of the verification block P0 in registers and memory or cache is 12 blocks.

[0027] Example 1

[0028] The register-friendly, efficient XOR erasure coding method in this embodiment specifically includes the following steps:

[0029] Step 1: Splitting data blocks and check blocks:

[0030] Taking the calculation of check block P0 as an example, the check block These seven data blocks are divided into n register-sized sub-blocks, such as Figure 2 As shown, for example, database D0 is split into data sub-blocks D 00 D 01 ,…,D 0(n-1) Similarly, check block P0 is also split into n check sub-blocks P 00 ,P 01 ,…,P 0(n-1) Step two is performed for each check sub-block, here taking P as an example. 00 For example, the calculation.

[0031] Step 2: Calculation of the verification sub-block:

[0032] First, read data sub-block D. 00 The parity subblock P is then transferred to register R1. 00 The value is initialized to data sub-block D 00 ,like Figure 3 As shown; the remaining six data sub-blocks D 30 D 40D 80 D 90 D 10(0) D 11(0) Read (LOAD) data sequentially into register R2; each time register R2 receives a data sub-block, perform the following process: XOR the values ​​in registers R1 and R2, and store the calculated value in register R1, such as... Figure 4 As shown; after completing six XOR calculations, the value in register R1 is stored in memory as the parity sub-block P. 00 ,like Figure 5 As shown. The entire process verifies sub-block P. 00 The intermediate results are always stored in register R1 until the calculation is complete, requiring only one register-to-memory storage operation.

[0033] Step 3: Verify the merging of sub-blocks:

[0034] After all check sub-blocks have completed their calculations, the n check sub-blocks P 00 ,P 01 ,…,P 0(n-1) It is stored in a contiguous block of memory as parity block P0. Note that when writing parity sub-block P... 00 ,P 01 ,…,P 0(n-1) Afterwards, the metadata of each check sub-block is not recorded, only the metadata of check block P0 is recorded, thus realizing the merging of check sub-blocks.

[0035] The specific implementation process of this invention has been described above. Through analysis and explanation, by splitting data blocks and parity blocks into register-level encoding, and storing each parity sub-block only in memory after calculation, the number of storage and retrieval operations between registers and memory is greatly reduced, thus improving encoding speed. In this embodiment, the number of times data is stored from registers to memory during the encoding process is minimized, i.e., reduced by 83.3%, and the number of times data is read from memory to registers is reduced by 45.5%.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0037] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A register-friendly and efficient XOR erasure coding method, employing a (k,m) XOR erasure coding matrix to encode k data blocks D0, D1, ..., D k-1 Encode the data to obtain m parity blocks P0, P1, ..., P m-1 Each element in the XOR erasure coding matrix is ​​either 0 or 1, and each check block is the XOR sum of multiple data blocks; characterized in that... Includes the following steps: Step 1: Splitting data blocks and check blocks: Let the check block P g The formula for calculating 0≤g≤m-1 is: Each data block Data sub-blocks of size n registers Will check block P g Divide into n register-sized parity sub-blocks P g0 ,P g1 ,…,P g(n-1) Each verification sub-block For each check subblock P gl If 0 ≤ l ≤ n-1, proceed to step two; Step 2: Calculation of the verification sub-block: Read data sub-blocks The parity subblock P is then transferred to register R1. gl The value is initialized to Each data sub-block is read When the value is entered into register R2, process A is performed: the values ​​in registers R1 and R2 are XORed, and the result of the XOR calculation is stored in register R1. After completing process A for t-1 times, the value in register R1 is stored in memory as the check sub-block P. gl ; Complete all check subblocks P gl After calculating 0≤l≤n-1, proceed to step three; Step 3: Verify the merging of sub-blocks: Sequentially assign n check sub-blocks P g0 ,P g1 ,…,P g(n-1) Sequentially written to contiguous address spaces in cache or memory, and merged into a parity block P. g .

Images