A computationally and cache-friendly efficient Reed-Solomon code encoding method

Abstract

The present invention relates to the technical field of erasure codes in computer distributed storage systems, and discloses a computationally and cache-friendly efficient Reed-Solomon code encoding method. The present invention splits a data block and a check block, searches for a common XOR part, and calculates two matched check packets. Compared with existing encoding processes, the present invention takes into account both computational efficiency and cache efficiency, and splits the data block and the check block into data packets and check packets of appropriate sizes for encoding. Compared with existing methods for reducing the amount of XOR calculations, the present invention utilizes limited registers to store the calculation results of the common XOR part of the two check packets in the register until their calculations are completed, thereby reducing the amount of XOR calculations and the amount of data reading without introducing additional data transmission. The present invention improves encoding performance by balancing the computational efficiency and cache efficiency of encoding and reducing the amount of XOR calculations during the encoding process.

Description

Technical Field

[0001] The present invention relates to the technical field of erasure codes in computer distributed storage systems, and in particular to a computationally and cache-friendly efficient Reed-Solomon code encoding method. Background Art

[0002] Erasure coding and replication are the two most common redundancy technologies for data fault tolerance in storage systems. Replication stores multiple copies of the same data on different nodes. While easy to implement and can provide concurrent access for front-end users, it incurs high storage overhead. Compared to replication, erasure coding offers the same reliability but reduces storage overhead by an order of magnitude. It is widely used in large-scale storage systems to save storage costs. However, erasure coding-based fault tolerance requires encoding calculations on data blocks to generate parity blocks during data writes. During fault repair, surviving blocks must be decoded to generate faulty blocks. These encoding and decoding calculations not only slow down data writes and repairs but also consume CPU and memory resources. Even the most advanced erasure coding library, ISA-L, still has an encoding speed slower than memory I / O and Remote Direct Memory Access (RDMA). Therefore, improving the encoding and decoding speed of erasure codes is of great significance.

[0003] Reed-Solomon codes are a commonly used erasure code whose encoding process requires multiplication and addition operations. To avoid complex multiplication operations, Reed-Solomon codes based on Cauchy matrices convert the encoding matrix into a bit matrix (composed of 0s and 1s), thereby converting multiplication operations into XOR operations. Existing erasure code libraries that support Cauchy-Reed-Solomon codes ignore the spatial locality of the cache during the encoding process, resulting in frequent data swapping in and out of the cache. Furthermore, they do not efficiently reuse the common parts of the computation, introducing more data read and storage operations when reusing the common XOR parts, resulting in no significant improvement in encoding efficiency. These two reasons lead to the low encoding performance of existing erasure code libraries. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a computationally and cache-friendly efficient Reed-Solomon code encoding method. The basic idea is to split the data block and the check block into 2KB packets for calculation, taking into account the computational and cache-friendliness of the encoding process. The graphical model is used to find the reusable common XOR part, and the corresponding encoding process is designed to avoid additional data transmission when reusing the common XOR part, thereby improving the encoding speed of the erasure code.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] Use (k, m) Cauchy-Solomon code to encode k data blocks D0, D1, ..., D k-1Encode and get m check blocks P0, P1, ..., P m-1 , the encoding calculation is performed in the finite field GF(2 w In order to reduce the computational complexity, the present invention converts the multiplication operation into an XOR operation, wherein the m×k encoding matrix is converted into an mw×kw bit matrix.

[0007] The computationally and cache-friendly efficient Reed-Solomon code encoding method of the present invention comprises the following steps:

[0008] Step 1: Split data blocks and check blocks:

[0009] The data block D i , 0≤i≤k-1 is split into rw data packets, r is a positive integer, and the check block P i , 0≤i≤m-1 is split into rw check packets. The encoding is divided into r rounds, each round starts from the data block D i , take out w data packets D from 0≤i≤k-1 i0 , D i1 ,...,D i(w-1) , calculate the check block P i , w check packets P with 0≤i≤m-1 i0 , P i1 ,...,P i(w-1) ,Each check packet is obtained by performing XOR calculation on multiple data packets.

[0010] To ensure computational efficiency, encoding should be performed in as few rounds as possible. This means that the data and check packets should be as large as possible. To ensure cache efficiency, all data packets required for each round must be stored in the cache. However, the cache size at each level is fixed, so cache efficiency can only be achieved by setting the data packet size. In this case, the smaller the data packet, the better. To balance computational efficiency and cache efficiency, the data and check packet sizes are set to 2KB.

[0011] Step 2: Find the common XOR part:

[0012] To reduce the amount of XOR computation, the results of the common XOR parts can be reused. First, construct a weighted graph G = (V, E, W), where the point set V is the set of mw verification packets, each corresponding to a point in the graph. When the calculations of two verification packets contain at least one common data packet, there is an edge between them, and the weight of this edge is the number of common data packets contained in the two verification packets. To find the most common XOR parts, find the maximum weight matching M in the graph G. The common XOR parts of the verification packets corresponding to the two vertices of each edge in M can be reused.

[0013] Step 3: Calculate the two matched verification packets:

[0014] set up and Among them, P common yes and For the convenience of description, assume that the register can hold a data packet and a check packet (if the register is not large enough to hold a packet, split all data packets and check packets into register granularity and perform the calculation in multiple rounds). First, use registers R1 and R2 to calculate P common and store the result in register R1; the value P in register R1 common Copy to another register R3; use registers R1 and R2 to calculate the check packet Use registers R3 and R2 to calculate the check packet Finally, the values in registers R1 and R3 are stored in the memory as a verification packet and

[0015] Compared with the prior art, the beneficial technical effects of the present invention are:

[0016] The present invention splits the data block and the check block, searches for the common XOR part, and calculates the two matched check packets. Compared to existing encoding processes, this method balances computational efficiency and cache efficiency, splitting the data block and the check block into data packets and check packets of appropriate sizes for encoding. Compared to existing methods for reducing the amount of XOR calculations, this method utilizes limited registers and stores the calculation results of the common XOR part of the two check packets in the registers until their calculations are completed. This reduces the amount of XOR calculations and the amount of data read without introducing additional data transmission. The present invention improves encoding performance by balancing the computational efficiency and cache efficiency of encoding and reducing the amount of XOR calculations during the encoding process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of Cauchy-Reed Solomon code calculation in an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of maximum weight matching in an embodiment of the present invention;

[0019] Figure 3 Schematic diagram of the calculation of the common XOR part in an embodiment of the present invention;

[0020] Figure 4 Schematic diagram of public XOR partial replication in an embodiment of the present invention;

[0021] Figure 5 Schematic diagram of calculation of the first matched verification packet in an embodiment of the present invention;

[0022] Figure 6 A schematic diagram of calculation of a second matched verification packet in an embodiment of the present invention;

[0023] Figure 7 Schematic diagram of verification package storage in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The computationally and cache-friendly efficient Reed-Solomon code encoding method of the present invention will be further described below with reference to the accompanying drawings through specific embodiments.

[0025] The existing method first matches the data packets, calculates the XOR result of the common data packets, and stores it in the memory or cache. Then, each check block is calculated in turn, and the result of the required common XOR part is read from the memory or cache to the register, resulting in additional data transmission. Included in the calibration package P 00 and P 10 In the calculation formula of , the existing method uses D 00 and D 10 Read the register, calculate its XOR value, and store the calculation result in memory or cache. 00 and P 01 During the calculation process, This results in two additional read operations and one store operation.

[0026] Example 1

[0027] This embodiment uses (4, 2) Cauchy-Solomon code to encode four data blocks D0, D1, D2, and D3 to obtain two check blocks P0 and P1. The encoding calculation is performed over the finite field GF(2 3 ) is performed. The 2×4 coding matrix (Parity Coding Matrix) is converted into a 6×12 bit matrix (Parity Coding Bitmatrix).

[0028] The computationally and cache-friendly efficient Reed-Solomon code encoding method in this embodiment specifically includes the following steps:

[0029] Step 1: Split data blocks and check blocks:

[0030] Because the process of each round of encoding calculation is the same, it is assumed that there is only one round of encoding in this embodiment, that is, r = 1. The four data blocks are split into data packets D of size 2KB 00 , D 01 , D 02 , D 10 , D 11 , D 12, D 20 , D 21 , D 22 , D 30 , D 31 , D 32 , 2 check blocks are also split into 6 check packets P 00 , P 01 , P 02 , P 10 , P 11 , P 12 . Figure 1 This is a calculation diagram after the encoding matrix is converted into a bit matrix.

[0031] Step 2: Find the common XOR part:

[0032] In this embodiment, a weighted graph is constructed based on the bit matrix. Figure 2 As shown, there are 6 vertices P 00 , P 01 , P 02 , P 10 , P 11 , P 12 , the weight of each edge is the number of common data packets of the verification packets corresponding to the two vertices, such as P 00 and P 12 There are 4 public data packages D 00 , D 22 , D 30 , D 32 , the edge weight between them is 4. Find the maximum weight matching M={P 00 P 12 , P 01 P 10 , P 02 P 11},like Figure 2 Indicated by the medium thick line.

[0033] Step 3: Calculate the two matched verification packets:

[0034] To verify the package P 00 and P 12 As an example, the calculation of the check package Verification Package set up and in Use registers R1 and R2 to calculate the check packet First read (LOAD) data packet D 00 to register R1, and then read data packets D in sequence 22 , D 30 , D32 To register R2, perform XOR calculation on the values in register R1 and register R2, and store the result in register R1, as shown in Figure 3 As shown; then copy the value in register R1 to register R3, as shown Figure 4 As shown; use registers R1 and R2 to calculate the check packet Read data packets D in sequence 10 , D 11 , D 31 To R2, the values in register R1 and register R2 are XORed and the result is stored in register R1, as shown in Figure 5 As shown; use registers R3 and R2 to calculate the check packet Read data packets D in sequence 12 , D 21 To register R2, perform XOR calculation on the values in registers R2 and R3, and store the result in register R3, as shown in Figure 6 As shown; Finally, the values in registers R1 and R3 are stored (STORE) in the memory as the verification packet P 00 and P 12 ,like Figure 7 shown.

[0035] Compared with the existing method, the present invention does not need to common Instead of storing it in memory or cache, it is stored in registers R1 and R3, and then the check packet P 00 and P 12 The remaining data packets are XORed into registers R1 and R3, respectively, reusing the common part. It also avoids extra data transmission, reducing three XOR operations and four data packet readings in total.

[0036] The above describes the specific implementation process of the present invention. Analysis shows that using 2KB data packets and check packets, all data packets can be placed in the L2 cache, while ensuring a low L1 cache miss rate and high XOR calculation efficiency. A weighted graph is constructed based on the bit matrix, and the common XOR part is found by searching for the maximum weight match. Without storing intermediate values, the common XOR part is reused using three registers, reducing the amount of data read and the amount of XOR calculation. In this embodiment, the amount of data read and XOR calculation was reduced by 23.3%.

[0037] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0038] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A computationally and cache-friendly efficient Reed-Solomon code encoding method, using (k, m) Cauchy Reed-Solomon code to encode k data blocks D0, D1, ..., D k-1 Encode and get m check blocks P0, P1, ..., P m-1 , encoded in the finite field GF(2 w ), converting the multiplication operation into an XOR operation, wherein the m×k encoding matrix is converted into an mw×kw bit matrix, characterized in that, The following steps are involved: Step 1: Split data blocks and check blocks: The data block D i , 0≤i≤k-1 is split into rw packets, r∈N + , check block P i , 0≤i≤m-1 is split into rw check packets, and the encoding is divided into r rounds, each round starts from the data block D i Take out w data packets D i0 , D i1 ,…,D i(w-1) , calculate the check block P i w check packets P i0 , P i1 ,...,P i(w-1) ,Each check packet is obtained by performing XOR calculation on multiple data packets; Step 2: Find the common XOR part: Construct a weighted graph G = (V, E, W), where V is the point set, E is the edge set, W is the edge weight set, and the point set V is a collection of mw verification packages, each verification package corresponds to a point in the weighted graph G; when two verification packages P A , P B When the calculation of the two check packets P contains at least one public data packet, A , P B There is an edge e between the corresponding points A , side e A The weight of the two check packets P A , P B The number of public data packets included; find the maximum weight match M in the weighted graph G, and reuse the public XOR part of the verification packets corresponding to the two vertices of each edge in M; Step 3: Calculate the two matched check packets in the maximum weight matching Set verification package Sum check package Among them, P common To calculate the checksum and The XOR operation result of the public data packet at that time, that is, the verification packet and The public XOR part of To calculate the checksum Time Divide P common The XOR operation result of the data packets outside To calculate the checksum Time Divide P common The XOR operation result of the data packet outside; use registers R1 and R2 to calculate the two check packets The public XOR part P common and store the result in register R1; XOR the public part P in R1 common Copy to register R3; calculate the check packet using registers R1 and R2 Use registers R3 and R2 to calculate the check packet Finally, the values in registers R1 and R3 are stored in the memory as a verification packet and in It is an exclusive OR operation.

2. The computationally and cache-friendly efficient Reed-Solomon code encoding method according to claim 1, characterized in that In step 1, the data block and the check block are split, and the size of the data packet and the check packet is set to 2KB.

3. The computationally and cache-friendly efficient Reed-Solomon code encoding method according to claim 1, characterized in that: In step 3, if the register size is not large enough to hold a check packet or data packet, all data packets and check packets are split into register granularity and step 3 is executed multiple times for calculation.

Images