An erasure code data processing system, method, computer device and medium

Abstract

The present invention relates to the field of computer technologies, and in particular, to an erasure code data processing system, method, computer device, and medium. The system includes: a PCIe device in which to-be-operated data is stored; a host, a data unit and a parameter unit controlled by the host. The host controls the data unit to read the to-be-operated data from the PCIe device based on an erasure processing request, and calculates a corresponding to-be-multiplied parameter for each to-be-operated data according to the read operation of the PCIe device and sends it to the parameter unit; a plurality of cores, each of which respectively obtains the to-be-operated data and the to-be-multiplied parameter corresponding to the to-be-operated data from the data unit and the parameter unit, and performs an operation on the obtained to-be-operated parameter and to-be-multiplied parameter based on a preset rule and outputs erasure encoded data or erasure decoded data. The solution of the present invention uses a pipeline method to achieve high-speed parallel processing, significantly improving the processing efficiency of erasure code data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to an erasure code data processing system, method, computer device, and medium. Background Art

[0002] With the rapid development of communication technology and network technology, digital information has shown an exponential explosion growth, and data storage technology has thus faced huge challenges. The reliability problem of data in the storage system and the energy consumption problem of the storage system have attracted more and more attention. Nowadays, in the face of such a huge data scale, the reliability of data in the storage system is inversely proportional to the number of components in the storage system, that is, the more components in the storage system, the lower the reliability of data in the storage system. According to relevant investigations, in an Internet data center composed of 600 disks, about 30 disks are damaged every month. In large-scale storage systems, the decrease in data reliability caused by disk failures is a quite serious problem, and relevant fault-tolerant technologies have been studied in this regard.

[0003] Erasure coding (EC) is a data protection method that divides data into segments, expands and encodes redundant data, and stores it in different locations, such as disks, storage nodes, or other geographical locations. The original data is divided into k data blocks, and m encoded blocks are generated according to the encoding matrix, and n (n = k + m) blocks are distributed to different servers. Only k blocks are needed to restore the original data. Currently, erasure code data processing chips (DPUs) have emerged. For different encoding algorithms, different chip architectures need to be designed, with poor flexibility and low processing efficiency, so there is an urgent need for improvement. Summary of the Invention

[0004] In view of this, it is necessary to provide an erasure code data processing system, method, computer device, and medium for the above technical problems.

[0005] According to a first aspect of the present invention, an erasure code data processing system is provided, and the system includes:

[0006] A PCIe device, in which the data to be operated is stored;

[0007] A host and a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be operated from the PCIe device based on an erasure processing request, and the host calculates a corresponding parameter to be multiplied for each data to be operated according to the read operation of the PCIe device and sends it to the parameter unit;

[0008] Multiple cores, each core respectively obtains data to be operated and a multiplication parameter corresponding to the data to be operated from the data unit and the parameter unit, and performs an operation on the obtained data to be operated and multiplication parameter based on a preset rule and outputs erasure encoded data or erasure decoded data.

[0009] In some embodiments, the number of cores is equal to the number of redundant blocks set by the erasure coding, and each core includes a data module, a parameter module, a Galois multiplication module, a first exclusive OR module, and a cache module.

[0010] In some embodiments, the erasure processing request is an erasure coding request, each core corresponds to a redundant block, and each core is configured to generate data by the following steps:

[0011] Set the data in the cache module to zero;

[0012] The data module sequentially obtains the data to be operated required to generate a redundant block from the data unit;

[0013] The parameter module sequentially obtains the multiplication parameter corresponding to each data to be operated from the parameter unit;

[0014] The Galois multiplication module sequentially performs a multiplication operation on each pair of data to be operated and multiplication parameter to generate a product result;

[0015] The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result;

[0016] Output the result of the last exclusive OR operation performed by the first exclusive OR module to obtain the erasure encoded data corresponding to the redundant block.

[0017] In some embodiments, the erasure processing request is an erasure decoding request, and the host is configured to:

[0018] Compare the amount of data with errors in the erasure decoding request with the number of redundant blocks set by the erasure coding;

[0019] In response to the amount of data with errors exceeding the number of redundant blocks set by the erasure coding, confirm that decoding data cannot be generated;

[0020] In response to the amount of data with errors not exceeding the number of redundant blocks set by the erasure coding, confirm that erasure decoding data can be generated by multiple cores.

[0021] In some embodiments, in response to the number of errors being equal to the number of redundant blocks set by the erasure coding, each core corresponds to an error data to be recovered, and each core is configured to generate data by the following steps:

[0022] Set the data in the cache module to zero;

[0023] The data module sequentially obtains the data to be operated on required to generate a data to be restored from the data unit;

[0024] The parameter module sequentially obtains the multiplication parameters to be multiplied corresponding to each data to be operated on from the parameter unit;

[0025] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and multiplication parameters to generate a product result;

[0026] The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result;

[0027] Output the result of the last exclusive OR operation performed by the first exclusive OR module to obtain the erasure decoding data corresponding to the data to be restored.

[0028] In some embodiments, in response to the number of errors occurring being less than the number of redundant blocks set by the erasure coding, the host groups multiple cores so that the number of core groups is equal to the number of errors occurring, and each core group corresponds to an error data to be restored.

[0029] In some embodiments, the system further includes a control strobe module and a second exclusive OR module, and each core group is configured to generate data by the following steps:

[0030] Set the data in the cache module to zero;

[0031] The data module sequentially obtains the data to be operated on required to generate the data to be restored from the data unit;

[0032] The data modules of each core in the same core group sequentially obtain the data to be operated on required to generate a data to be restored from the data unit;

[0033] The parameter modules of each core in the same core group sequentially obtain the multiplication parameters to be multiplied corresponding to each parameter to be operated on from the parameter unit;

[0034] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and multiplication parameters to generate a product result;

[0035] The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result;

[0036] Output the result of the last XOR operation performed by the first XOR module;

[0037] In response to the number of cores in a certain core group exceeding one, the control gating module will block the paths between the first XOR module and the second XOR module of each core in the same core group, so that the second XOR module performs an XOR operation on the outputs of the first XOR modules of each core in the same core group to obtain the erasure decoding data corresponding to the data to be recovered;

[0038] In response to the number of cores in a certain core group not exceeding one, the control gating module will output the result of the last XOR operation performed by the first XOR module of the core to obtain the erasure decoding data corresponding to the data to be recovered.

[0039] According to a second aspect of the present invention, there is provided an erasure code data processing method, the method comprising:

[0040] Use the PCIe device to store the data to be operated;

[0041] Set a host and a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be operated from the PCIe device based on an erasure processing request, and the host calculates a corresponding multiplication parameter for each data to be operated according to the read operation of the PCIe device and sends it to the parameter unit;

[0042] Set multiple cores, and each core respectively obtains the data to be operated and the corresponding multiplication parameter for the data to be operated from the data unit and the parameter unit, and performs an operation on the obtained data to be operated and multiplication parameter based on a preset rule and outputs erasure encoded data or erasure decoding data.

[0043] According to a third aspect of the present invention, there is also provided a computer device, the computer device comprising:

[0044] At least one processor; and

[0045] A memory, the memory stores a computer program that can run on the processor, and when the processor executes the program, it executes the foregoing erasure code data processing method.

[0046] According to a fourth aspect of the present invention, there is also provided a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it executes the foregoing erasure code data processing method.

[0047] The above erasure code data processing system uses a host to control data units, parameter units, and multiple cores to perform data interaction to achieve erasure encoding and decryption. By using a pipeline method, it can achieve high-speed parallel data processing on the premise of limited hardware configuration, significantly improving the processing efficiency of erasure code data.

[0048] In addition, the present invention also provides an erasure code data processing method, a computer device, and a computer-readable storage medium, which can also achieve the above technical effects and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other embodiments can be obtained based on these drawings.

[0050] Figure 1 It is a schematic diagram of the traditional erasure code encoding principle;

[0051] Figure 2 It is a schematic diagram of the structure of an erasure code data processing system provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the hardware architecture of the kernel Kernel during erasure code encoding provided by another embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the hardware architecture of the kernel Kernel during erasure code decoding provided by another embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of the flow of an erasure code data processing method provided by another embodiment of the present invention;

[0055] Figure 6 It is the internal structure diagram of a computer device in another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the following further elaborates on the embodiments of the present invention in detail with reference to specific embodiments and the accompanying drawings.

[0057] It should be noted that all the expressions using "first" and "second" in the embodiments of the present invention are used to distinguish two non-identical entities or non-identical parameters with the same name. It can be seen that "first" and "second" are only for the convenience of expression and should not be construed as a limitation on the embodiments of the present invention. This will not be elaborated one by one in the subsequent embodiments.

[0058] To facilitate the understanding of the solution of the present invention, the principles of erasure code encoding and decryption will be described below. Please refer to Figure 1 As shown, the current parameter configuration of the erasure code includes: k: data blocks. k represents the number of blocks into which the original data is divided and the minimum number of blocks required to recover the original data. The smaller the value of k, the greater the cost of data reconstruction in case of a failure; the larger the value of k, multiple data copies are required, increasing the load on the network and IO. m: encoded blocks. m affects the reliability of data storage and the storage cost. The larger the value, the greater the tolerance for failures, the redundancy of the data will also increase, and the storage cost will also increase. n: the number of generated blocks (n = k + m); effective storage ratio: k / n; Traditional erasure code encoding generally uses Vandermonde or Cauchy matrices, and its encoding is as Figure 1 shown: Figure 1 In, the data blocks to be encoded are k = 5, the encoding requirement is m = 3, the final generated code blocks are the D + C part, with a total of k + m = 8, and the effective storage ratio is: k / n = 5 / 8. The erasure system implemented in this way can encode K Ds to obtain m Cs, and the encoding method is as Figure 1 shown in. The erasure system can perform decoding and recovery for any m errors in the system after m encodings are implemented.

[0059] Erasure Code belongs to a forward error correction technology in coding theory and was first applied in the communication field to solve problems such as loss and attenuation in data transmission. Since the erasure code technology has achieved good results in preventing data loss, it has been introduced into the storage field. Erasure codes can effectively reduce the storage overhead while ensuring the same reliability, so the erasure code technology is widely used in major storage systems and data centers such as Microsoft's Azure, Facebook's F4, etc. There are many types of erasure codes, and the more common one in actual storage systems is the RS code (Reed-Solomon Code) applied in a distributed environment. The RS code is related to two parameters k and r. Given two positive integers k and r, the RS code encodes k data blocks into r additional parity blocks. The encoding method of the r parity blocks based on the Vandermonde matrix or Cauchy matrix is called the RS erasure code using the Vandermonde matrix or Cauchy matrix encoding.

[0060] For example, the specific encoding process of the RS erasure code based on the Vandermonde matrix is as follows:

[0061]

[0062] Another example, the specific encoding process of the RS erasure code based on the Cauchy matrix is as follows:

[0063]

[0064] The upper k*k matrix corresponds to k original data blocks, and the lower r*k matrix corresponds to the encoding matrix. By multiplying with the original data D1 to Dk, the newly added P1 to Pr are the r parity data obtained by encoding. When any at most r data are in error or lost during transmission and need to be corrected, the inverse matrix of the corresponding matrix of the remaining data is multiplied by the data, and the original data blocks D1 to Dk will be obtained (the derivation process is not elaborated here).

[0065] Taking the loss of data D1 to Dr and decoding as an example, the specific process of the erasure code to recover data is as follows:

[0066]

[0067] It can be seen that the core concept of the erasure code is to construct an invertible encoding matrix to generate parity data, and its inverse matrix can be calculated to recover the original data. The common RS erasure code uses the Cauchy matrix or Vandermonde matrix introduced above. The advantage is that the obtained matrix is definitely invertible, and any submatrix of it is also invertible, and the expansion of the matrix size is simple.

[0068] In one embodiment, please refer to Figure 2 As shown, the present invention provides an erasure code data processing system. Specifically, the system includes:

[0069] A PCIe device, in which the data to be operated is stored;

[0070] A host and a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be operated from the PCIe device, and the host calculates the corresponding multiplication parameters for each data to be operated according to the read operation of the PCIe device and sends them to the parameter unit;

[0071] Multiple cores, each core respectively obtains the data to be operated and the corresponding multiplication parameters for the data to be operated from the data unit and the parameter unit, and performs operations on the obtained data to be operated parameters and multiplication parameters based on a preset rule and outputs erasure encoded data or erasure decoded data.

[0072] The above erasure code data processing system uses the host to control the data unit, the parameter unit and multiple cores for data interaction to realize erasure encoding and decryption, and uses the pipeline method to realize high-speed parallel data processing on the premise of limited hardware configuration, significantly improving the processing efficiency of erasure code data.

[0073] In some embodiments, the number of cores is equal to the number of redundant blocks set for erasure encoding, and each core includes a data module, a parameter module, a Galois multiplication module, a first exclusive OR module, and a cache module.

[0074] In some embodiments, refer to Figure 3 As shown, the erasure processing request is an erasure coding request. Each core corresponds to a redundant block, and each core is configured to generate data by the following steps:

[0075] Set the data in the cache module to zero;

[0076] The data module sequentially obtains the data to be operated on required to generate a redundant block from the data units;

[0077] The parameter module sequentially obtains the multiplication parameters to be multiplied corresponding to each data to be operated on from the parameter units;

[0078] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be multiplied and the multiplication parameters to be multiplied to generate a product result;

[0079] The first XOR module performs an XOR operation on each product result and the data in the cache module to generate an XOR result, and the cache module caches the current XOR result;

[0080] Output the result of the last XOR operation performed by the first XOR module to obtain the erasure coding data corresponding to the redundant block.

[0081] In some embodiments, the erasure processing request is an erasure decoding request, and the host is configured to:

[0082] Compare the amount of data with errors in the erasure decoding request with the number of redundant blocks set by the erasure coding;

[0083] In response to the amount of data with errors exceeding the number of redundant blocks set by the erasure coding, confirm that the decoding data cannot be generated;

[0084] In response to the amount of data with errors not exceeding the number of redundant blocks set by the erasure coding, confirm that the erasure decoding data can be generated by multiple cores.

[0085] In some embodiments, in response to the number of errors being equal to the number of redundant blocks set by the erasure coding, each core corresponds to an error data to be recovered, and each core is configured to generate data by the following steps:

[0086] Set the data in the cache module to zero;

[0087] The data module sequentially obtains the data to be operated on required to generate an error data to be recovered from the data units;

[0088] The parameter module sequentially obtains the multiplication parameters corresponding to each data to be operated on from the parameter units;

[0089] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and the parameters to be multiplied to generate a product result;

[0090] The first XOR module performs an XOR operation on each product result and the data in the cache module to generate an XOR result, and the cache module caches the current XOR result;

[0091] Output the result of the last XOR operation performed by the first XOR module to obtain the erasure decoding data corresponding to the data to be recovered.

[0092] In some embodiments, in response to the number of errors occurring being less than the number of redundant blocks set by the erasure coding, the host groups multiple cores so that the number of core groups is equal to the number of errors occurring, and each core group corresponds to an error data to be recovered.

[0093] In some embodiments, please refer to Figure 4 As shown, the system further includes a control gating module and a second XOR module. Each core group is configured to generate data using the following steps:

[0094] Set the data in the cache module to zero;

[0095] The data module sequentially obtains the data to be operated on required to generate the data to be recovered from the data unit;

[0096] The data module of each core in the same core group sequentially obtains the data to be operated on required to generate a data to be recovered from the data unit;

[0097] The parameter module of each core in the same core group sequentially obtains the parameters to be multiplied corresponding to each parameter to be operated on from the parameter unit;

[0098] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and the parameters to be multiplied to generate a product result;

[0099] The first XOR module performs an XOR operation on each product result and the data in the cache module to generate an XOR result, and the cache module caches the current XOR result;

[0100] Output the result of the last XOR operation performed by the first XOR module;

[0101] In response to the number of cores in a certain core group exceeding one, the control gating module connects the paths between the first XOR module of each core in the same core group and the second XOR module, so that the second XOR module performs an XOR operation on the outputs of the first XOR module of each core in the same core group to obtain the erasure decoding data corresponding to the data to be recovered;

[0102] In response to the number of cores in a certain core group not exceeding one, the control strobe module outputs the result of the last exclusive OR operation performed by the first exclusive OR module of the core to obtain the erasure decoding data corresponding to the data to be restored.

[0103] In yet another embodiment, to facilitate understanding of the technical solution of the present invention and ensure the security of user data, erasure is a core function in a storage server. To implement the erasure function, a data processing chip (DPU) in a general-purpose storage server often needs to design an erasure code data processing system to perform related functional operations. This embodiment proposes a hardware implementation of a general erasure code data processing system and a data sum operation method under this hardware, enabling it to utilize the pipeline method to achieve the advantage of high-speed parallel processing under the premise of limited hardware configuration. This method is applicable to common erasure function requirements such as RS and RAID. The specific implementation is as follows:

[0104] For an erasure code data processing system, common algorithms, whether RS or RAID algorithms, can summarize their encoding and decoding in the implementation stage as the following method:

[0105]

[0106] As shown in Formula 1, whether it is an erasure algorithm based on RS or RAID, its encoding and decoding can be expressed as the sum of products of data and its corresponding parameters. Therefore, when the erasure code data processing system is implemented in hardware, it can be represented as Figure 2 the architecture shown, where the host is the main control part of the operation, controlling two parts, namely the data unit and the parameter unit. Among them, under the control of the host, the data unit fetches the data to be operated (encoded or decoded data) from the PCIe, and at the same time, the host calculates the parameters to be multiplied corresponding to the data at this time based on different operation requirements and environments, and then sends them to the kernel. All relevant operations are completed within the kernel and finally output.

[0107] The specific operation implementation is achieved through the kernel. There are various ways to implement the operation in the kernel. Considering the loss corresponding to the hardware implementation, the speed obtained, and the improvement in throughput comprehensively, it can be but not limited to fully parallel, fully serial, or a combined operation implementation method. Preferably, in this embodiment, the parallelism of the kernel implementation is set based on the number of parity codes. Taking the RS erasure as an example for explanation, that is, when there are r parity codes (equivalent to r in Formula 1), then r parallel operation structures are set for it. Taking r as 3 for explanation, the hardware structure implemented is as Figure 3 shown:

[0108] When r = 3, we recommend implementing three parallel pipelines for related encoding and decoding operations. The specific implementation method is as follows Figure 3 shown. Working from top to bottom, when encoding, data is first sent to the data modules of three kernels simultaneously through the data unit outside the kernel. The host calculates three sets of parameters, and each set of parameter units sends them to the parameter module of one kernel. Then, Galois multiplication is performed within each of the three kernels, and the results obtained are XORed with the temporary data in the temporary storage module. At the initial stage of data operation, all the data in the temporary storage module is reset to 0. After that, a temporary value is generated in each operation cycle, stored in the temporary storage module, and then taken out in the next cycle and XORed with the data obtained from the multiplication operation in the next cycle. After multiple iterations in this way, a final value is generated and output. Finally, the output kernel outputs three check code values with r = 3.

[0109] The above implementation method is based on the fact that r check code values can be calculated and obtained in parallel. The decoding uses the same module. Because in the case of setting r check codes, the maximum number of data that can be recovered from errors simultaneously is equal to r. Therefore, the three parallel kernels here can work simultaneously during decoding. The working method is the same as that of encoding, as described above. As we know, although the maximum recovery amount we set is r, as introduced above, the most common error scenario is a single error. Therefore, in most cases, the r kernels described above cannot be fully utilized during decoding, that is, there is a certain waste of the set hardware. Therefore, in the scenario where the number of single or arbitrary errors does not reach r errors, there will be a certain waste of hardware, that is, a certain speed loss. For this reason, under this general module, we add a data connection relationship to achieve internal parallel operation again in this case. The following will explain the processing process when the amount of error data during erasure decoding does not reach r:

[0110] As Figure 4 can be seen, the improvement method is to add an XOR and a control gating module before the final output. Among them, the control gating module is controlled by the host based on different operations, and the XOR module performs XOR operations on data of the same size. Different function implementations are mainly distinguished based on the control gating module:

[0111] (1) When performing full-r amount encoding and decoding, the gating module outputs the result of the last XOR operation of the XOR module of each kernel as the decryption result.

[0112] (2) When the decoding environment has an error amount less than r, it is output to the exclusive-OR module based on e beats for exclusive-OR operation respectively, and then the result of the last exclusive-OR operation of all exclusive-OR modules within the same kernel group is exclusive-ORed again, and the result is output as the decryption result of the data to be restored corresponding to this kernel group. Here, e is the actual number of error code blocks to be decoded. The specific implementation method is as follows:

[0113] (a) Determine the number of parallel kernels. The determination method follows formula 2 below:

[0114]

[0115] In formula 2 above, e is the actual number of error code blocks to be decoded, and r is the above-mentioned r. For e error code blocks, the finally obtained e en values are the respective numbers of the parallel operation kernels divided. Taking r = 3 and e = 2 as an example, based on formula 2, en1 = 2 and en2 = 1 can be calculated.

[0116] (b) Divide the parallel operation kernels based on en in step (a). Continuing with the example of r = 3 and e = 2 above, after calculation, en1 = 2 and en2 = 1. Then for Figure 4 kernel1 and 2 are divided into one group, and the remaining kernel3 is in one group. That is, kernel 2 processes the restoration of the error data block 2 alone, while kernel1 and kernel 2 calculate the restoration of error code block 1 in parallel. The division method is to divide the k data blocks that need to be read during decoding. Based on perform a division operation, and divide each dn i data block reading, and then the operation results are combined through exclusive-OR. In the above case, taking k = 32 as an example, it means that every 16 data blocks are read and operated by kernel1 and 2 respectively, and the final operation time of the result is 16 + 1 = 17 cycles.

[0117] For the erasure code data processing system of this embodiment, in any case with less than r errors, the speed can be increased by at most cycles, and the slowest is the same as the original implementation method. Thus, the parallelization inside the decoding iterative operation is achieved, and the prepared decoding kernel hardware is utilized to the maximum extent. The erasure code data processing system of this embodiment not only has versatility, but also can utilize the designed operation hardware to the maximum extent. On the premise of achieving the balance between speed and area, the operation speed is optimized to the maximum extent.

[0118] In yet another embodiment, please refer to Figure 5 as shown, this embodiment provides an erasure code data processing method 100, and the method includes:

[0119] Step 101: Store the data to be operated on in the PCIe device;

[0120] Step 102: Set up the host and the data unit and parameter unit controlled by the host. The host controls the data unit to read the data to be operated on from the PCIe device, and the host calculates the corresponding multiplication parameters for each data to be operated on according to the read operation of the PCIe device and sends them to the parameter unit;

[0121] Step 103: Set up multiple cores. Each core respectively obtains the data to be operated on and the corresponding multiplication parameters for the data to be operated on from the data unit and the parameter unit, and performs operations on the obtained data to be operated on and multiplication parameters based on a preset rule and outputs erasure coding data or erasure decoding data.

[0122] The above erasure code data processing method uses the host to control the data unit, parameter unit and multiple cores to perform data interaction to achieve erasure coding and decryption. Using the pipeline method, under the premise of limited hardware configuration, high-speed parallel data processing is realized, significantly improving the processing efficiency of erasure code data.

[0123] In some embodiments, the number of cores is equal to the number of redundant blocks set for erasure coding. Each core includes a data module, a parameter module, a Galois multiplication module, a first exclusive OR module, and a cache module.

[0124] In some embodiments, the erasure processing request is an erasure coding request. Each core corresponds to a redundant block, and each core is configured to generate data by the following steps:

[0125] Set the data in the cache module to zero;

[0126] The data module sequentially obtains the data to be operated on required to generate a redundant block from the data unit;

[0127] The parameter module sequentially obtains the multiplication parameters corresponding to each data to be operated on from the parameter unit;

[0128] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and multiplication parameters to generate product results;

[0129] The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result;

[0130] Output the result of the last exclusive OR operation performed by the first exclusive OR module to obtain the erasure coding data corresponding to the redundant block.

[0131] In some embodiments, the erasure processing request is an erasure decoding request, and the host is configured to:

[0132] Compare the amount of data with errors in the erasure decoding request with the number of redundant blocks set by the erasure coding;

[0133] In response to the amount of data with errors exceeding the number of redundant blocks set by the erasure coding, confirm that the decoded data cannot be generated;

[0134] In response to the amount of data with errors not exceeding the number of redundant blocks set by the erasure coding, confirm that the erasure decoded data can be generated through multiple cores.

[0135] In some embodiments, in response to the number of errors being equal to the number of redundant blocks set by the erasure coding, each core corresponds to an error data to be recovered, and each core is configured to generate data by the following steps:

[0136] Set the data in the cache module to zero;

[0137] The data module sequentially obtains the data to be operated on required to generate a data to be recovered from the data unit;

[0138] The parameter module sequentially obtains the multiplication parameters to be multiplied corresponding to each data to be operated on from the parameter unit;

[0139] The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and multiplication parameters to generate a product result;

[0140] The first XOR module performs an XOR operation on each product result and the data in the cache module to generate an XOR result, and the cache module caches the current XOR result;

[0141] Output the result of the last XOR operation performed by the first XOR module to obtain the erasure decoded data corresponding to the data to be recovered.

[0142] In some embodiments, in response to the number of errors being less than the number of redundant blocks set by the erasure coding, the host groups multiple cores so that the number of core groups is equal to the number of errors, and each core group corresponds to an error data to be recovered.

[0143] In some embodiments, the method further includes: setting a control gating module and a second XOR module, and each core group is configured to generate data by the following steps:

[0144] Set the data in the cache module to zero;

[0145] The data module sequentially obtains the data to be operated on required to generate the data to be recovered from the data unit;

[0146] Data modules of each core in the same core group sequentially obtain the data to be computed required to generate a data to be restored from the data unit;

[0147] Parameter modules of each core in the same core group sequentially obtain the parameters to be multiplied corresponding to each parameter to be computed from the parameter unit;

[0148] The Galois multiplication module sequentially performs multiplication operations on each pair of the data to be computed and the parameters to be multiplied to generate product results;

[0149] The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result;

[0150] Output the result of the last exclusive OR operation performed by the first exclusive OR module;

[0151] In response to the number of cores in a certain core group exceeding one, the control gating module connects the paths between the first exclusive OR modules of each core in the same core group and the second exclusive OR module, so that the second exclusive OR module performs an exclusive OR operation on the outputs of the first exclusive OR modules of each core in the same core group to obtain the erasure decoding data corresponding to the data to be restored;

[0152] In response to the number of cores in a certain core group not exceeding one, the control gating module outputs the result of the last exclusive OR operation performed by the first exclusive OR module of the core to obtain the erasure decoding data corresponding to the data to be restored.

[0153] It should be noted that the specific limitations on the erasure code data processing method can be referred to the limitations on the erasure code data processing system in the above text, which will not be elaborated here. Each module in the above erasure code data processing system can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0154] According to another aspect of the present invention, there is provided a computer device, which can be a server. The internal structure diagram thereof can be referred to Figure 6As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the erasure code data processing method described above. Specifically, the method includes the following steps:

[0155] Use the PCIe device to store the data to be operated on;

[0156] Set a host and a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be operated on from the PCIe device based on an erasure processing request, and the host calculates a corresponding multiplication parameter for each data to be operated on according to the read operation of the PCIe device and sends it to the parameter unit;

[0157] Set multiple cores. Each core respectively obtains the data to be operated on and the corresponding multiplication parameter for the data to be operated on from the data unit and the parameter unit, and performs operations on the obtained data to be operated on and multiplication parameter based on a preset rule and outputs erasure encoded data or erasure decoded data.

[0158] According to another aspect of the present invention, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the erasure code data processing method described above. Specifically, it includes performing the following steps:

[0159] Use the PCIe device to store the data to be operated on;

[0160] Set a host and a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be operated on from the PCIe device based on an erasure processing request, and the host calculates a corresponding multiplication parameter for each data to be operated on according to the read operation of the PCIe device and sends it to the parameter unit;

[0161] Set multiple cores. Each core respectively obtains the data to be operated on and the corresponding multiplication parameter for the data to be operated on from the data unit and the parameter unit, and performs operations on the obtained data to be operated on and multiplication parameter based on a preset rule and outputs erasure encoded data or erasure decoded data.

[0162] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0163] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0164] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of the patent of this application shall be subject to the appended claims.

Claims

1. An erasure code data processing system, characterized in that The system includes: A PCIe device, in which the data to be computed is stored; A host, a data unit and a parameter unit controlled by the host. The host controls the data unit to read the data to be computed from the PCIe device based on an erasure processing request, and the host calculates a corresponding multiplication parameter for each data to be computed according to the read operation of the PCIe device and sends it to the parameter unit; Multiple cores, each core respectively obtains the data to be computed and the corresponding multiplication parameter for the data to be computed from the data unit and the parameter unit, and performs operations on the obtained data to be computed and multiplication parameter based on a preset rule and outputs erasure encoded data or erasure decoded data; wherein, the number of cores is equal to the number of redundant blocks set by the erasure encoding, and each core includes a data module, a parameter module, a Galois multiplication module, a first exclusive OR module, and a cache module; The host is configured to: if the amount of data with errors does not exceed the number of redundant blocks set by the erasure encoding, confirm that erasure decoded data can be generated by multiple cores; wherein, the host groups the multiple cores so that the number of core groups is equal to the number of errors that occur, and each core group corresponds to an error data to be recovered. The system further includes a control gating module and a second exclusive OR module, and each core group is configured to generate data by the following steps: Set the data in the cache module to zero; The data module sequentially obtains the data to be computed required to generate the data to be recovered from the data unit; The data module of each core in the same core group sequentially obtains the data to be computed required to generate a data to be recovered from the data unit; The parameter module of each core in the same core group sequentially obtains the multiplication parameter corresponding to each data to be computed from the parameter unit; The Galois multiplication module sequentially performs a multiplication operation on each pair of data to be computed and multiplication parameter to generate a product result; The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result; Output the result of the last exclusive OR operation performed by the first exclusive OR module; If the number of cores in a certain core group exceeds one, the control gating module connects the paths between the first exclusive OR modules of each core in the same core group to the second exclusive OR module, so that the second exclusive OR module performs an exclusive OR operation on the outputs of the first exclusive OR modules of each core in the same core group to obtain the erasure decoded data corresponding to the data to be recovered; If the number of cores in a certain core group does not exceed one, the control gating module outputs the result of the last exclusive OR operation performed by the first exclusive OR module of the core to obtain the erasure decoded data corresponding to the data to be recovered.

2. The erasure code data processing system according to claim 1, wherein The erasure processing request is an erasure encoding request, each core corresponds to a redundant block, and each core is configured to generate data by the following steps: Set the data in the cache module to zero; The data module sequentially obtains the data to be computed required to generate a redundant block from the data unit; The parameter module sequentially obtains from the parameter unit to generate a multiplication parameter corresponding to each data to be calculated; The Galois multiplication module sequentially performs multiplication operations on each pair of data to be calculated and the multiplication parameter to generate a product result; The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result; Output the result of the last exclusive OR operation performed by the first exclusive OR module to obtain the erasure coding data corresponding to the redundant block.

3. The erasure code data processing system according to claim 1, wherein, The erasure processing request is an erasure decoding request, and the host is configured as: Compare the amount of data with errors in the erasure decoding request with the number of redundant blocks set by the erasure coding; In response to the amount of data with errors exceeding the number of redundant blocks set by the erasure coding, confirm that the decoding data cannot be generated.

4. The erasure code data processing system according to claim 3, wherein In response to the number of errors being equal to the number of redundant blocks set by the erasure coding, each core corresponds to an error data to be recovered, and each core is configured to generate data using the following steps: Set the data in the cache module to zero; The data module sequentially obtains from the data unit to generate the data to be calculated required for recovering an error data; The parameter module sequentially obtains from the parameter unit the multiplication parameter corresponding to each data to be calculated; The Galois multiplication module sequentially performs multiplication operations on each pair of data to be calculated and the multiplication parameter to generate a product result; The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result; Output the result of the last exclusive OR operation performed by the first exclusive OR module to obtain the erasure decoding data corresponding to the error data to be recovered.

5. A method for processing erasure code data, characterized in that, The method includes: Use the PCIe device to store the data to be calculated; Set the host and the data unit and parameter unit controlled by the host. The host controls the data unit to read the data to be calculated from the PCIe device based on the erasure processing request, and the host calculates the corresponding multiplication parameter for each data to be calculated according to the read operation of the PCIe device and sends it to the parameter unit; Set multiple cores. Each core respectively obtains the data to be calculated and the multiplication parameter corresponding to the data to be calculated from the data unit and the parameter unit, and performs operations on the obtained data to be calculated parameters and multiplication parameters based on a preset rule and outputs erasure coding data or erasure decoding data; wherein, the number of cores is equal to the number of redundant blocks set by the erasure coding, and each core includes a data module, a parameter module, a Galois multiplication module, a first exclusive OR module, and a cache module; The host is configured as: in response to the amount of data with errors not exceeding the number of redundant blocks set by the erasure coding, confirm that the erasure decoding data can be generated by multiple cores; wherein, the host groups the multiple cores so that the number of core groups is equal to the number of errors, each core group corresponds to an error data to be recovered, a control gating module and a second exclusive OR module are set, and each core group is configured to generate data using the following steps: Set the data in the cache module to zero; The data module sequentially obtains the data to be operated on required to generate the data to be restored from the data unit; The data modules of each core in the same core group sequentially obtain the data to be operated on required to generate a piece of data to be restored from the data unit; The parameter modules of each core in the same core group sequentially obtain the parameters to be multiplied corresponding to each parameter to be operated on from the parameter unit; The Galois multiplication module sequentially performs multiplication operations on each pair of data to be operated on and parameters to be multiplied to generate product results; The first exclusive OR module performs an exclusive OR operation on each product result and the data in the cache module to generate an exclusive OR result, and the cache module caches the current exclusive OR result; Output the result of the last exclusive OR operation performed by the first exclusive OR module; In response to the number of cores in a certain core group exceeding one, the control gating module connects the paths between the first exclusive OR modules of each core in the same core group and the second exclusive OR module, so that the second exclusive OR module performs an exclusive OR operation on the outputs of the first exclusive OR modules of each core in the same core group to obtain the erasure decoding data corresponding to the data to be restored; In response to the number of cores in a certain core group not exceeding one, the control gating module outputs the result of the last exclusive OR operation performed by the first exclusive OR module of the core to obtain the erasure decoding data corresponding to the data to be restored.

6. A computer device, characterized in that, Comprising: At least one processor; And A memory that stores a computer program that can run in the processor, and when the processor executes the program, it executes the method according to claim 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it executes the method according to claim 5.

Images