Erasure processing method, system and computer storage medium

Abstract

The present invention provides an erasure correction processing method, system and computer storage medium, the method comprising asynchronously obtaining information required for an erasure correction task, synchronizing the required information, the required information including data, configuration and matrix; performing multiple multiplication operations on a byte of the matrix and the data based on a data bit width preset in the configuration; performing XOR processing on the result of each multiplication operation to obtain an erasure correction check block and / or data block. The present invention proposes a hardware circuit implementation method for the processing unit part in erasure correction, the hardware comprising an input and output control unit and a calculation unit. A cache is provided in the processing unit for the intermediate results of the calculation, and all intermediate results are cached inside the processing unit before the final result is obtained. A hardware acceleration solution is used for erasure correction encoding and decoding, which reduces the CPU overhead and provides a faster throughput than the CPU solution.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to an erasure processing method, system and computer storage medium. Background Art

[0002] Faced with the need to store massive amounts of data, distributed storage, with its advantages of low cost and good scalability, has gradually replaced the dominant position of centralized storage and has garnered increasing attention in both theoretical research and practical applications. Distributed storage systems often use inexpensive disks as storage nodes, resulting in limited reliability for each storage node. Furthermore, a distributed storage system typically comprises numerous nodes, and node failures are common due to software and hardware failures, human error, and other factors. To improve the data reliability of distributed storage systems and ensure that data collection nodes can reconstruct the original files with a high probability, it is necessary to store a certain amount of redundancy in addition to the original data. This ensures that even if some nodes fail, the system can still operate normally and the data collection nodes can still decode and recover the original files. Furthermore, to maintain system reliability, failed nodes must be repaired promptly, making the design of a robust node repair mechanism crucial. Erasure codes are a type of forward error correction technique in coding theory. They were first applied in the communications field to address data loss and loss issues. Due to their proven effectiveness in preventing data loss, erasure codes have been introduced into the storage field. Erasure coding can effectively reduce storage overhead while ensuring the same reliability. Therefore, erasure coding technology is widely used in major storage systems and data centers such as Microsoft's Azure and Facebook's F4.

[0003] There are many types of erasure codes. The most common one in real-world storage systems is the Reed-Solomon Code (RS Code), used in distributed environments. RS codes are associated with two parameters, k and r. Given two positive integers, k and r, the RS code encodes k data blocks into r additional parity blocks. The encoding of these r parity blocks based on a Vandermonde matrix or a Cauchy matrix is ​​called an RS erasure code encoded with a Vandermonde matrix or a Cauchy matrix. The specific encoding process is shown below for RS erasure codes encoded with a Vandermonde matrix or a Cauchy matrix, respectively:

[0004]

[0005]

[0006] In the above formula, the k*k matrix corresponds to the k original data blocks, and the r*k matrix in the lower part corresponds to the encoding matrix. By multiplying the original data D1 to Dk, the newly added P1 to Pr are the r check data obtained by encoding. When any data less than r is erroneous or lost during transmission and error correction is required, the inverse matrix of the matrix corresponding to the remaining data is multiplied with the data (the multiplication is a multiplication operation in the GF domain, which will not be described in detail), and the original data blocks D1 to Dk will be obtained. Taking the case of data loss from D1 to Dr and decoding as an example, the process is shown in the following formula:

[0007]

[0008] The core concept of erasure codes is to construct a reversible encoding matrix to generate checksum data, whose inverse can be calculated to restore the original data. Common RS erasure codes use the Cauchy or Vandermonde matrices described above. The advantages of these are that the resulting matrix is ​​guaranteed to be reversible, and any of its submatrices are also reversible, and the matrix size can be easily expanded.

[0009] The data units of D1 to Dk, P1 to Pr in the above introduction can be 8 bits, 16 bits or other values ​​(here called sign bits, called symbols), which depends on the bit width of the multiplication operation, that is, the finite field definition of the Galois field operation. For a task, if the block size is Z bits. For multiplication operations, if 8-bit Galois field operations are used, it is necessary to split it into (Z / 8) independent data and perform operations on the same matrix. And output the operation results. The encoding output of the first byte can be obtained as shown in the following formula,

[0010] By performing the same calculations on the first byte 4096 times with different data and the same matrix, we can find an erasure code with a block size of 4Kbyte. This shows that a distinct characteristic of erasure is that the data blocks, check blocks, and symbols are uncorrelated.

[0011] In summary, building a high-performance erasure correction system requires a highly concurrent hardware system. Currently, the industry's most common approach is to run open source software on general-purpose CPUs, such as Intel's ISA and Jerasure 2.0 open source software. This approach either increases data throughput by operating at higher CPU frequencies or accelerates data using hardware acceleration instructions provided by CPUs, such as Intel's SIMD. Currently, for encoding and decoding RS erasure codes in distributed storage, the industry generally uses open source software solutions such as Intel-ISA and Jerasure 2.0. This involves running RS-related software on the CPU for encoding and decoding. A significant drawback of this approach is the significant CPU resource consumption. Summary of the Invention

[0012] The present invention provides an erasure correction processing method, system and computer storage medium, which are used to solve the problem that the existing erasure correction process occupies CPU computing resources.

[0013] To achieve the above object, the present invention adopts the following technical solutions:

[0014] A first aspect of the present invention provides an erasure processing method, the method comprising the following steps:

[0015] Asynchronously obtain the information required for the erasure task and synchronize the required information, including data, configuration, and matrix;

[0016] Based on the data width preset in the configuration, perform several multiplication operations on a byte of the matrix and the data;

[0017] The result of each multiplication operation is XORed to obtain an erasure-corrected check block and / or data block.

[0018] Furthermore, the data includes the number of data blocks, the number of check blocks, missing data blocks and check data blocks; the matrix includes an encoding matrix and a decoding matrix; and the configuration includes a data block size and a data bit width.

[0019] Furthermore, performing XOR processing on the result of each multiplication operation is specifically as follows:

[0020] Cache the result of the first multiplication operation;

[0021] Perform XOR processing on the result of the middle multiplication operation and the result in the cache, and cache the processed result;

[0022] XOR the result of the last multiplication operation with the last cached result.

[0023] A second aspect of the present invention provides an erasure processing system, the system comprising:

[0024] The input control unit is used to asynchronously obtain the information required for the erasure task and synchronize the required information, including data, configuration, and matrix;

[0025] The calculation unit performs a number of multiplication operations on a byte of the matrix and the data based on the data bit width preset in the configuration;

[0026] The output control unit performs XOR processing on the result of each multiplication operation to obtain an erasure-corrected check block and / or data block.

[0027] Furthermore, the input control unit includes:

[0028] Data queue, used to cache input data blocks and check block data;

[0029] Input configuration queue, used to cache input configuration information;

[0030] Matrix queue, used to cache input matrix data;

[0031] The input processing subunit aligns and synchronizes the input data and the matrix based on the configuration information.

[0032] Furthermore, the calculation unit includes a plurality of multiplication subunits, and the number of the multiplication subunits is determined by a preset data bit width.

[0033] Furthermore, the output control unit includes:

[0034] Output configuration queue, used to cache output configuration information, including the size and number of data blocks;

[0035] The output control subunit performs XOR calculation on the operation results of the multiplication subunit;

[0036] Local cache, used to cache intermediate results calculated by the multiplication subunit.

[0037] Furthermore, the output control subunit performs XOR calculation including:

[0038] Cache the result of the first multiplication operation;

[0039] Perform XOR processing on the result of the middle multiplication operation and the result in the cache, and cache the processed result;

[0040] XOR the result of the last multiplication operation with the last cached result.

[0041] Furthermore, a configuration signal is set between the input configuration queue and the output configuration queue for back pressure.

[0042] A fourth aspect of the present invention provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed on the system, the system executes the steps of the method.

[0043] The erasure processing system of the second aspect of the present invention can implement the method of the first aspect and the various implementations of the first aspect, and achieve the same effect.

[0044] The effects provided in the summary of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0045] 1. This invention proposes a hardware circuit implementation for the processing unit in erasure correction. This hardware comprises input and output control units and a computation unit. The processing unit provides a cache for intermediate computational results; all intermediate results are cached within the processing unit until the final result is obtained. Hardware-accelerated erasure correction encoding and decoding is implemented, reducing CPU overhead while providing faster throughput than CPU-based solutions.

[0046] 2. The input data is back-pressured through a 1-bit configurable back-pressure signal to achieve complete decoupling of input and output. The input, output, and computing units have a fully pipelined hardware structure. The computing unit has a configurable multi-stage pipeline design, and the number of pipeline stages is related to the back-pressure configuration.

[0047] 3. The three types of information input by the input control unit are completely asynchronous. The configuration, data, and matrix are synchronized in the control part of the input control unit to ensure that the three parts of the input can be independently and efficiently sent to their respective queues.

[0048] 4. The data bit width (16 bytes is used as an example in the embodiment) can be directly expanded, the size of the local buffer can also be adjusted (increased or reduced), and the back pressure control signal is configurable, with a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 This is a schematic diagram of the existing distributed storage node erasure application;

[0051] Figure 2 is a schematic flow chart of an embodiment of the method of the present invention;

[0052] Figure 3 It is a structural diagram of an embodiment of the system of the present invention;

[0053] Figure 4 This is a schematic diagram of a typical erasure scenario implemented by the present invention;

[0054] Figure 5 This is a data flow diagram of an erasure task of the present invention. DETAILED DESCRIPTION

[0055] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits descriptions of well-known components and processing technologies and processes to avoid unnecessary limitations on the present invention.

[0056] In a distributed storage system, a processing node can handle many erasure strategies. A storage node can have different strategy configurations such as 4+2, 6+2, 3+1, etc. In the case of multiple processes or multiple threads concurrently, the tasks that need to be handled by a hardware system that handles erasure are also different. Figure 1 As shown, a distributed storage node has multiple storage pools. For example, the erasure strategies for the three storage pools in the diagram are 4+2, 6+2, and 2+1. The host distributed storage node sends five consecutive task requests to the hardware processing unit. The hardware processing unit processes the erasure tasks in sequence and feeds the encoded / decoded data back to the host.

[0057] like Figure 2 As shown, the method embodiment of the present invention provides an erasure processing method, which includes the following steps:

[0058] S1, asynchronously obtains the information required for the erasure task and synchronizes the required information, including data, configuration, and matrix;

[0059] S2, based on the data bit width preset in the configuration, performs several multiplication operations on a byte of the matrix and the data;

[0060] S3, perform XOR processing on the results of each multiplication operation to obtain the erasure correction check block and / or data block.

[0061] In step S1, the available environment is defined. For k data blocks and r check blocks, encoding requires configuring the number of data blocks k and check blocks r, encoding and decoding matrices, and block size information. Decoding requires configuring the number of data blocks k and check blocks r, encoding or decoding matrices, block size, missing data blocks, existing check blocks, and the sum of missing data blocks and check blocks is lbn. As long as lbn is less than or equal to r, erasure decoding can recover all missing data blocks and check blocks.

[0062] In step S3, the XOR processing of the result of each multiplication operation is specifically as follows:

[0063] Cache the result of the first multiplication operation;

[0064] Perform XOR processing on the result of the middle multiplication operation and the result in the cache, and cache the processed result;

[0065] XOR the result of the last multiplication operation with the last cached result.

[0066] like Figure 3 As shown, an embodiment of the present invention provides an erasure processing system, comprising an input control unit, a computation unit, and an output control unit disposed at the top level of a PE. The erasure correction hardware proposed in this embodiment supports an unlimited maximum number of data nodes, while the maximum number of supported check nodes, r, only needs to be expanded by the number of processing units.

[0067] External data scheduling in this system embodiment is performed through slicing. The size of a data slice is statically configured or dynamically calculated. This slice size is determined by combining the system's DMA scheduling method and the hardware output cache capacity. The slice size is unrelated to the data block size and can be larger, smaller, or equal to the data block size. The maximum slice size is the size of the local buffer unit in this system.

[0068] Before describing the functional units of the system, Figure 4 Taking the typical application scenario of erasure correction as an example, we will explain which part of the calculation is completed by the processing unit in the entire erasure correction encoding and decoding process. Taking 4+2 encoding and decoding as an example, the system is responsible for solving only one data block or one check block output in the figure based on the four input data blocks or check blocks. The system consumes Figure 4 The encoding matrix or partial matrix information in decoding matrix 1, 2, 3.

[0069] Figure 3 The input control unit is used to asynchronously obtain the information required for the erasure correction task and synchronize the required information, which includes data, configuration and matrix; the calculation unit performs several multiplication operations on a byte of the matrix and the data based on the data bit width preset in the configuration; the output control unit performs XOR processing on the result of each multiplication operation to obtain the correction check block and / or data block.

[0070] The input control unit includes a data queue, an input configuration queue, a matrix queue and an input processing subunit.

[0071] The data queue is used to cache the input data blocks and check block data; when the queue is about to be full, no more data can be sent to the data queue from the outside.

[0072] The input configuration queue is used to cache input configuration information; for example, the size of the data block, k information, etc. When the queue is about to be full, no more configuration information can be sent to the data queue from the outside.

[0073] The matrix queue is used to cache input matrix data. When the queue is full, no more matrices can be sent to the data queue from the outside.

[0074] The input processing subunit aligns and synchronizes the input data and the matrix based on the configuration information, and then sends them to the next level hardware module.

[0075] The calculation unit includes several multiplication subunits, the number of which is determined by a preset data bit width, and the figure includes 16 identical multiplication subunits.

[0076] The output control unit includes an output configuration queue, an output control subunit and a local cache.

[0077] The output configuration queue is used to cache output configuration information, such as the size of the data block, k information, etc. When the queue is about to be full, the external configuration information can no longer be sent to the data queue.

[0078] The output control subunit is mainly responsible for processing the data output from the multiplication unit and selectively performing an XOR calculation on the data and the data in the local buffer. Specifically, when the data block is the first block (data block or check block), the data is sent to the local buffer; when the data block is the last block (data block or check block), the data output by the multiplication unit and the data read out of the local buffer are XORed, and the calculation result is sent out of this processing unit; in other cases (i.e., the data is not the first or last), the data output by the multiplication unit and the data read out of the local buffer are XORed, and the calculation result is written back to the local buffer.

[0079] The local cache is used to cache the intermediate results calculated by the multiplication subunit.

[0080] like Figure 5 As shown, taking a 4Kbyte matrix as an example, in steps ①, ②, ③, and ④, each multiplication unit performs a calculation on the xth byte (x = 1, 2, 3, 4) of the matrix and the input data. The difference is that the data is from different data blocks. In step ①, the data is simply multiplied and the result is directly stored in the buffer. In steps ② and ③, the data in the local buffer is retrieved, XORed with the multiplication result, and then stored in the local buffer. In step ④, the data in the local buffer is retrieved, XORed with the multiplication result, and the final calculation result is sent to the processing unit, completing the calculation of the check block.

[0081] A configuration signal is set between the input configuration queue and the output configuration queue for back pressure to achieve input and output decoupling. Back pressure is performed only through a 1-bit configurable signal, which greatly reduces the control complexity of the system and allows the system to truly achieve a bubble-free pipeline design. When the output data is not back pressured, the throughput of the input data can reach 100%.

[0082] The present invention also provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed on the system, the system executes the steps of the method.

[0083] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A method for erasure correction, characterized in that: The method is used for distributed storage and comprises the following steps: Asynchronously obtain the information required for the erasure task and synchronize the required information, including data, configuration, and matrix; Based on the data width preset in the configuration, perform several multiplication operations on a byte of the matrix and the data; Perform XOR processing on the result of each multiplication operation to obtain an erasure-corrected check block and / or data block; The data includes the number of data blocks, the number of check blocks, missing data blocks and check data blocks; the matrix includes an encoding matrix and a decoding matrix; the configuration includes a data block size and a data bit width; The XOR processing of the result of each multiplication operation is specifically as follows: Cache the result of the first multiplication operation; Perform XOR processing on the result of the middle multiplication operation and the result in the cache, and cache the processed result; XOR the result of the last multiplication operation with the last cached result; The method further comprises: Data queue, used to cache input data blocks and check block data; Input configuration queue, used to cache input configuration information; Matrix queue, used to cache input matrix data; Input processing, aligning and synchronizing the input data and the matrix based on the configuration information; Output configuration queue, used to cache output configuration information, including the size and number of data blocks; Output control, performing XOR calculation on the operation results of the multiplication sub-units; the number of the multiplication sub-units is determined by the preset data bit width; Local cache, used to cache the intermediate results calculated by the multiplication subunit; A configuration signal is set between the input configuration queue and the output configuration queue for back pressure.

2. An erasure processing system, characterized in that: The system is used for distributed storage, and the system includes: The input control unit is used to asynchronously obtain the information required for the erasure task and synchronize the required information, including data, configuration, and matrix; The calculation unit performs a number of multiplication operations on a byte of the matrix and the data based on the data bit width preset in the configuration; An output control unit performs XOR processing on the result of each multiplication operation to obtain an erasure-corrected check block and / or data block; The input control unit comprises: Data queue, used to cache input data blocks and check block data; Input configuration queue, used to cache input configuration information; Matrix queue, used to cache input matrix data; An input processing subunit aligns and synchronizes the input data and the matrix based on the configuration information; The calculation unit includes a plurality of multiplication subunits, and the number of the multiplication subunits is determined by a preset data bit width; The output control unit includes: Output configuration queue, used to cache output configuration information, including the size and number of data blocks; The output control subunit performs XOR calculation on the operation results of the multiplication subunit; Local cache, used to cache the intermediate results calculated by the multiplication subunit; The output control subunit performs XOR calculation including: Cache the result of the first multiplication operation; Perform XOR processing on the result of the middle multiplication operation and the result in the cache, and cache the processed result; XOR the result of the last multiplication operation with the last cached result; A configuration signal is set between the input configuration queue and the output configuration queue for back pressure.

3. A computer storage medium storing computer instructions, wherein: When the computer instructions are executed on the system of claim 2 , the computer instructions cause the system to perform the steps of the method of claim 1 .

Images