An erasure correction task processing system, method, electronic device and storage medium

Abstract

The application discloses a kind of erasure correction task processing systems, the erasure correction task processing system includes: data slice scheduling unit, for determining the slice parameter corresponding to erasure correction task, and slice parameter is sent to input data address calculation unit and output data address calculation unit;Input data address calculation unit is used to calculate the first address information of each input slice data block according to slice parameter;Output data address calculation unit is used to calculate the second address information of each output slice data block according to slice parameter;DMA unit is used to control the input and output of erasure correction processing module according to the first address information and the second address information, so as to complete erasure correction task.The application can reduce the occupation of computing resources for erasure correction task processing.The application also discloses an erasure correction task processing method, a storage medium and an electronic device, with the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an erasure task processing system, method, electronic device and storage medium. Background Technology

[0002] Faced with the demands of storing massive amounts of data, distributed storage, with its advantages of low cost and high scalability, has gradually replaced unified storage as the dominant system, attracting increasing attention in both theoretical research and practical applications. Distributed storage systems often use inexpensive disks as storage nodes, and the reliability of each node is often not very high. Furthermore, a distributed storage system typically contains many nodes, and node failures are common due to hardware / software malfunctions, human error, and other reasons. To improve the data reliability of distributed storage systems and ensure that data collection nodes can reconstruct the original files with a high probability, a certain amount of redundancy needs to be stored in addition to the original data. This allows the system to continue operating normally even if some nodes fail, and data collection nodes can still decode and recover the original files. Simultaneously, to maintain system reliability, failed nodes need to be repaired promptly; therefore, designing a robust node repair mechanism is crucial.

[0003] Currently, in distributed storage, the industry mostly uses open-source software solutions such as Intel-ISA and Jerasure 2.0 for erasure coding and decoding. This involves running Reed-Solomon Code (RS) related software on the CPU for erasure coding and decoding. However, the above-mentioned conventional methods for processing erasure tasks require a significant amount of computing resources.

[0004] Therefore, how to reduce the computational resource consumption of erasure tasks is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide an erasure task processing system, method, electronic device, and storage medium that can reduce the CPU computing resources occupied by erasure task processing.

[0006] To address the aforementioned technical problems, this application provides an erasure task processing system, the status early warning system comprising:

[0007] A data slicing scheduling unit is used to determine the slicing parameters corresponding to the erasure task and send the slicing parameters to the input data address calculation unit and the output data address calculation unit; wherein, the slicing parameters include: preset slicing size, data block size of the erasure task, number of input data blocks and number of output data blocks;

[0008] The input data address calculation unit is used to calculate the first address information of each input slice data block according to the slice parameters, and send the first address information to the DMA unit; wherein, the first address information includes the starting address and data size of the input slice data block;

[0009] The output data address calculation unit is used to calculate the second address information of each output slice data block according to the slice parameters, and send the second address information to the DMA unit; wherein, the second address information includes the starting address and data size of the output slice data block;

[0010] The DMA unit is used to control the input and output of the erasure processing module according to the first address information and the second address information, so as to complete the erasure task.

[0011] Optionally, the data slice scheduling unit includes:

[0012] The slice parameter determination subunit is used to set the preset slice size according to the scheduling mode of the DMA unit and the buffering capacity of the hardware output;

[0013] Alternatively, a parameter adaptive adjustment subunit can be used to adaptively adjust the preset slice size based on the current cache idle status.

[0014] Optionally, the erasure processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit;

[0015] The task scheduling unit is used to receive and parse the erasure task to obtain the parsing information, and send the parsing information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.

[0016] Optionally, the data scheduling unit is configured to input each input slice data block into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block.

[0017] Optionally, the matrix scheduling unit is configured to determine an encoding matrix and / or a decoding matrix based on the number of input data blocks and the number of output data blocks, and input the determined encoding matrix and / or decoding matrix into the processing unit.

[0018] Optionally, the configuration scheduling unit is used to generate configuration information based on the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.

[0019] Optionally, the output data scheduling unit is used to determine whether the erasure task has been completed based on the configuration information; and is also used to return response information to the task scheduling unit after the erasure task has been completed.

[0020] This application also provides a method for processing erasure tasks, the method comprising:

[0021] Determine the slice parameters corresponding to the erasure task; wherein, the slice parameters include: preset slice size, data block size of the erasure task, number of input data blocks, and number of output data blocks;

[0022] The first address information of each input slice data block is calculated based on the slice parameters; wherein, the first address information includes the starting address and data size of the input slice data block;

[0023] The second address information for each output slice data block is calculated based on the slice parameters; wherein the second address information includes the starting address and data size of the output slice data block;

[0024] The input and output of the erasure processing module are controlled based on the first address information and the second address information in order to complete the erasure task.

[0025] This application also provides a storage medium storing a computer program thereon, which, when executed, implements the steps of the above-described erasure task processing method.

[0026] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the above-described erasure task processing method.

[0027] This application provides an erasure task processing system, comprising: a data slice scheduling unit, configured to determine slice parameters corresponding to the erasure task and send the slice parameters to an input data address calculation unit and an output data address calculation unit; wherein, the slice parameters include: a preset slice size, the data block size of the erasure task, the number of input data blocks, and the number of output data blocks; the input data address calculation unit, configured to calculate first address information for each input slice data block according to the slice parameters and send the first address information to a DMA unit; wherein, the first address information includes the starting address and data size of the input slice data block; the output data address calculation unit, configured to calculate second address information for each output slice data block according to the slice parameters and send the second address information to the DMA unit; wherein, the second address information includes the starting address and data size of the output slice data block; the DMA unit, configured to control the input and output of the erasure processing module according to the first address information and the second address information, so as to complete the erasure task.

[0028] This application utilizes a data slicing scheduling unit to determine the slicing parameters of the erasure task. The input data address calculation unit determines the starting address and data size of each input slice data block based on the slicing parameters, enabling the DMA module to control the input of the erasure processing module according to the starting address and data size of each input slice data block. Similarly, the output data address calculation unit determines the starting address and data size of each output slice data block based on the slicing parameters, enabling the DMA module to control the output of the erasure processing module according to the starting address and data size of each output slice data block. This scheme performs data slicing processing on the data blocks of the erasure task, achieving the erasure of larger data blocks through dynamic rolling scheduling of small data blocks. This process is transparent to upper-layer application software and, through configurable slice sizes and hardware slicing methods, can be applied to hardware systems with different on-chip caches. This application can reduce the CPU computing resource consumption of erasure task processing. This application also provides an erasure task processing method, a storage medium, and an electronic device, all possessing the aforementioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0029] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of erasure application in a distributed storage node provided in an embodiment of this application;

[0031] Figure 2 This is a hardware block diagram of RS erasure encoding / decoding provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating the operation of an RS erasure encoding / decoding hardware provided in an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of two consecutive task scheduling provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] The erasure task processing system provided in this application includes:

[0036] A data slicing scheduling unit is used to determine the slicing parameters corresponding to the erasure task and send the slicing parameters to the input data address calculation unit and the output data address calculation unit; wherein, the slicing parameters include: preset slicing size, data block size of the erasure task, number of input data blocks and number of output data blocks;

[0037] The input data address calculation unit is used to calculate the first address information of each input slice data block according to the slice parameters, and send the first address information to the DMA unit; wherein, the first address information includes the starting address and data size of the input slice data block;

[0038] The output data address calculation unit is used to calculate the second address information of each output slice data block according to the slice parameters, and send the second address information to the DMA unit; wherein, the second address information includes the starting address and data size of the output slice data block;

[0039] The DMA unit is used to control the input and output of the erasure processing module according to the first address information and the second address information, so as to complete the erasure task.

[0040] After receiving the erasure task, the above embodiment determines the data block required for the erasure task and divides it into multiple input slice data blocks for erasure processing. This embodiment uses an input data address calculation unit to calculate the starting address and data size of each input slice data block. This embodiment also uses an output data address calculation unit to calculate the starting address and data size of each output slice data block. This embodiment sends the starting addresses and data sizes of the input slice data blocks, as well as the starting addresses and data sizes of the output slice data blocks, to the DMA (Direct Memory Access) unit. The DMA unit can control the erasure processing module to input data according to the starting addresses and data sizes of the input slice data blocks, and can also control the erasure processing module to output data according to the starting addresses and data sizes of the output slice data blocks.

[0041] This embodiment utilizes a data slicing scheduling unit to determine the slicing parameters for the erasure task. The input data address calculation unit determines the starting address and data size of each input slice data block based on the slicing parameters, enabling the DMA module to control the input of the erasure processing module according to the starting address and data size of each input slice data block. Similarly, the output data address calculation unit determines the starting address and data size of each output slice data block based on the slicing parameters, enabling the DMA module to control the output of the erasure processing module according to the starting address and data size of each output slice data block. This scheme performs data slicing processing on the data blocks of the erasure task, achieving the erasure of larger data blocks through dynamic rolling scheduling of small data blocks. This process is transparent to upper-layer application software and, through configurable slice sizes and hardware slicing methods, can be applied to hardware systems with different on-chip caches.

[0042] Furthermore, the data slice scheduling unit includes:

[0043] The slice parameter determination subunit is used to set the preset slice size according to the scheduling mode of the DMA unit and the buffering capacity of the hardware output;

[0044] Alternatively, a parameter adaptive adjustment subunit can be used to adaptively adjust the preset slice size based on the current cache idle status.

[0045] The data slicing scheduling unit can statically configure the preset slice size using the slicing parameter determination subunit, or dynamically configure it using the parameter adaptive adjustment subunit. The aforementioned hardware output caching capacity includes buffer size and throughput, and the preset slice size refers to the size of the input slice data block and the output slice data block. This embodiment introduces an adaptive method for data slicing, which can divide the cache into several sizes based on a fixed cache size and schedule different slice sizes according to the real-time cache idle status, further improving cache utilization and system throughput.

[0046] Furthermore, the erasure processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit.

[0047] The task scheduling unit is used to receive and parse the erasure task to obtain the parsing information, and send the parsing information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.

[0048] A data scheduling unit is used to input each of the input data blocks into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input data block.

[0049] A matrix scheduling unit is used to determine an encoding matrix and / or a decoding matrix based on the number of input data blocks and the number of output data blocks, and input the determined encoding matrix and / or decoding matrix into the processing unit.

[0050] A configuration scheduling unit is configured to generate configuration information based on the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.

[0051] The output data scheduling unit is used to determine whether the erasure task has been completed based on the configuration information; it is also used to return response information to the task scheduling unit after the erasure task has been completed.

[0052] The process described in the above embodiments is illustrated below through examples in practical applications.

[0053] Erasure coding is a forward error correction technique in coding theory, originally applied in the field of communications to address data loss and degradation during transmission. Due to its effectiveness in preventing data loss, erasure coding has been introduced into the storage field. It can effectively reduce storage overhead while maintaining the same reliability, and therefore is widely used in various storage systems and data centers, such as Microsoft Azure and Facebook's F4.

[0054] There are many types of erasure codes, but Reed-Solomon (RS) codes, commonly used in distributed environments, are a common feature in practical storage systems. RS codes are related to two parameters, k and r. Given two positive integers k and r, an RS code encodes k data blocks into r additional parity blocks. The method of encoding these r parity blocks based on a Vandermonde matrix or a Cauchy matrix is ​​called an RS erasure code using either a Vandermonde matrix or a Cauchy matrix.

[0055] The RS erasure code based on the Vandermonde matrix is ​​shown below:

[0056]

[0057] The RS erasure code based on the Cauchy matrix is ​​shown below:

[0058]

[0059] The upper k*k matrix corresponds to k original data blocks, and the lower r*k matrix corresponds to the encoding matrix. This is achieved by comparing the original data D1 to D... k Multiplying them together gives the newly added P1 to P. r These are the r checksums obtained from the encoding. If any fewer than r of these data are corrupted or lost during transmission, and error correction is required, the inverse of the matrix corresponding to the remaining data is multiplied by the data (multiplication is performed in the GF field), which yields the original data blocks D1 to D2. k GF (Galois Field) is the Galois field.

[0060] Taking the decoding of lost data from D1 to Dr as an example, the RS erasure coding data recovery method is as follows:

[0061]

[0062] The core concept of erasure coding is to construct an invertible coding matrix to generate check data, the inverse of which can be calculated to recover the original data. Common RS erasure codes use the Cauchy matrix or VanderMonte matrices introduced above. The advantage of this is that the resulting matrix is ​​always invertible, and any of its submatrices are also invertible, and the size of the matrix is ​​simple to expand.

[0063] The above introduction refers to D1 to D k P1 to P r The data unit can be 8 bits, 16 bits, or other values ​​(here called the sign bit, or symbol), depending on the bit width of the multiplication operation, i.e., the finite field definition of Galois field operations. For a task, if the block size is Z bits, for multiplication operations, if using 8-bit Galois field operations, it needs to be split into (Z / 8) independent data and the same matrix for operation. The result is then output. As shown below, the encoded output of the first byte can be obtained. Similarly, after 4096 calculations with different data and the same matrix, an erasure coding with a block size of 4Kbyte can be obtained. Therefore, the characteristic of erasure is that the different data blocks, check blocks, and different symbols are independent.

[0064] The erasure coding method for data with a block size of 4Kbyte is as follows:

[0065] Calculate for the first byte;

[0066] Based on the above analysis, it can be seen that for an erasure hardware system to complete erasure encoding or decoding functions, the following three elements are required:

[0067] Erasure task allocation: number of data blocks, number of check blocks, task type (encoding or decoding), status of data blocks (existing, error, or missing), status of check blocks (existing, error, or missing).

[0068] Matrix: For encoding, the original encoding matrix is ​​required; for decoding, the decoding matrix is ​​required, or the original encoding matrix is ​​provided and the hardware calculates the inverse matrix through Gaussian elimination to complete the decoding.

[0069] Input data: For encoding, data for all data blocks is required; for decoding, data containing data blocks and data containing check blocks are required.

[0070] In addition, the performance of erasure propagation, i.e., data throughput, is also a factor to consider. As described above, data throughput can be improved by maximizing parallelism across various symbols. In summary, a high-performance erasure propagation system requires a highly concurrent hardware system. Currently, the industry commonly uses open-source software running on general-purpose CPUs, such as Intel's ISA and Jerasure 2.0. This either increases data throughput through higher CPU operating frequencies or accelerates the process using hardware acceleration instructions provided by the CPU, such as Intel's SIMD (Single Instruction Multiple Data) instruction set.

[0071] Currently, in distributed storage, the industry mostly uses open-source software solutions such as Intel-ISA and Jerasure 2.0 for encoding and decoding based on RS erasure coding, which means running RS-related software on the CPU for erasure encoding and decoding. A significant problem with this solution is its high CPU computational resource consumption. This application proposes a hardware-accelerated solution for erasure encoding and decoding, which can reduce CPU overhead and provide a faster throughput than CPU-based solutions.

[0072] Please see Figure 1 , Figure 1 This is a schematic diagram of erasure application in a distributed storage node provided in an embodiment of this application. The distributed storage system includes distributed storage nodes, erasure hardware processing units, and storage pools 1 to 3. After receiving a task request, the erasure hardware processing unit can respond to the task. Figure 1 In this diagram, D represents a data block and C represents a checksum block. In a distributed storage system, a single processing node can handle multiple erasure strategies; for example, a storage node can have different strategy configurations such as 4+2, 6+2, and 3+1. In multi-process or multi-threaded concurrent scenarios, the tasks that a hardware system handling erasure will need to process will also differ. As shown below, a distributed storage node has multiple storage pools. For example, the three storage pools in the diagram have erasure strategies of 4+2, 6+2, or 2+1. The host storage node initiates five consecutive tasks to the hardware processing unit, which needs to process the erasure tasks sequentially and then feed back the encoded / decoded data to the host.

[0073] This embodiment proposes a hardware implementation of data slicing technology for data scheduling in erasure hardware systems. This hardware can automatically calculate the size of the data block for each data scheduling based on the user-defined target task's data block size, achieving high data throughput with a relatively small on-chip cache. The hardware can automatically calculate the size of the input data block and the source address of the input data for each scheduling, and can also automatically calculate the size of the output data block and the destination address of the output data for each scheduling.

[0074] The environment in this embodiment is as follows: For k data blocks and r parity blocks, encoding requires configuring the number of data blocks k and parity blocks r, the encoding matrix, the decoding matrix, and the block size information. Decoding requires configuring the number of data blocks k and parity blocks r, the encoding matrix or decoding matrix, the block size, missing data blocks, and existing parity block information. The sum of missing data blocks and parity blocks is lbn (Lost Block Number). As long as lbn is less than or equal to r, erasure decoding can recover all missing data blocks and parity blocks. The size of the data slice is implemented through static configuration. This slice size is determined by combining the system's DMA scheduling method and the hardware output buffer capacity. The slice size is not related to the data block size and can be greater than, less than, or equal to the data block size. In this embodiment, data scheduling, matrix scheduling, and other operations can be completed by external auxiliary hardware, such as DMA.

[0075] Please see Figure 2 , Figure 2 The following is a hardware block diagram of RS erasure encoding / decoding provided in the embodiments of this application, such as Figure 2 As shown, the RS erasure hardware proposed in this embodiment takes an example with an unlimited maximum number of data nodes and a maximum number of verification nodes r=6.

[0076] The module is divided into ① Task scheduling unit, ② Input data scheduling unit, ③ Matrix scheduling unit, ④ Configuration scheduling unit, ⑤ Galois multiplication unit (6 units), ⑥ Output data scheduling unit (6 units), ⑦ Data slice scheduling unit, ⑧ Input data address calculation unit, and ⑨ Output data address calculation unit. The Galois multiplication unit is the processing unit. The specific functions of each module are described below:

[0077] ① The task scheduling unit is responsible for receiving and parsing erasure task requests, and based on the parsed information, sending the data information that needs to be scheduled to the input data scheduling unit; sending the matrix information required for encoding and decoding to the matrix scheduling unit; and sending the specific configuration information of the task to the configuration scheduling unit. After the erasure task is completed, it is responsible for sending a response indicating that the task is completed.

[0078] ② The input data scheduling unit sends the input data of the erasure task to the PE (Process Element) processing unit that needs to be calculated, according to the input data scheduled as needed. The scheduling unit also needs to ensure that no data loss occurs, that is, to have necessary data flow control with the target PE.

[0079] ③ The matrix scheduling unit is responsible for fetching the required erasure encoding or decoding matrices from external parts according to the erasure task, or for calculating the required decoding matrix itself based on the original encoding matrix. Simultaneously, it sends the encoding or decoding matrix to the target PE unit that needs to calculate it. This scheduling unit also needs to ensure that no matrix is ​​lost, i.e., to maintain necessary data flow control with the target PE.

[0080] ④ The configuration scheduling unit sends the configuration information to the target PE and the output data scheduling unit according to the erasure task. The scheduling unit also needs to ensure that the configuration information is not lost, that is, to have necessary data flow control with the target PE and the output data scheduling unit.

[0081] ⑤ The Galois multiplication unit performs corresponding matrix calculations based on the input configuration, data, and matrix.

[0082] ⑥ The output data scheduling unit outputs the generated data according to the input configuration, and at the same time counts the output data of the target PE. When the number of output data is consistent with the number of data expected for the task, the processing of the task ends and a task end notification is sent to the task scheduling unit.

[0083] ⑦ The data slicing scheduling unit is responsible for distributing specific data switching tasks to the input data scheduling unit and the output data scheduling unit according to the type of task.

[0084] ⑧ The input data address calculation unit is responsible for calculating the starting address and data block size of each scheduled slice data block based on the starting address, slice size, and data block size of each input data block, and for scheduling input data from the outside.

[0085] ⑨ The output data address calculation unit is responsible for calculating the starting address and data block size of each scheduled slice data block based on the starting address, slice size, and data block size of each output data block, and then scheduling the generated output data to be output externally.

[0086] Please see Figure 3 , Figure 3 The flowchart of an RS erasure codec hardware provided in this application embodiment is as follows:

[0087] The task scheduling unit receives a deletion task request.

[0088] The task scheduling unit pushes the number K of input data blocks required for the erasure task, the number lbn of output data blocks, and the size of each data block into the input data scheduling unit. The input data scheduling unit, based on the number K of input data blocks, the number lbn of output data blocks, and the size of each data block, pushes lbn target PEs (i.e., Galois multiplication units) into the target PEs in a predetermined data order, ensuring that the data received by each target PE is identical.

[0089] The task scheduling unit pushes the number of input data blocks K and the number of output data blocks lbn required for the erasure task into the matrix scheduling unit. The matrix scheduling unit, based on the number of input data blocks K and the number of output data blocks lbn, sends a matrix of size lbn*K to lbn target PEs (each PE receives K matrix elements).

[0090] The task scheduling unit pushes the number K of input data blocks required for the erasure task, the number lbn of output data blocks, and the size of each data block into the configuration scheduling unit. The configuration scheduling unit, based on the number lbn of output data blocks, sends configuration information to lbn target PEs (each PE receives the same configuration information). The configuration scheduling unit, based on the number lbn of output data blocks, sends configuration information to lbn output data scheduling units (each unit receives the same configuration information).

[0091] The data slice scheduling unit notifies the two address calculation units (i.e., the input data address calculation unit and the output data address calculation unit) based on the configured slice size, the total data block size of the task, the number of output data blocks K, and the number of output data blocks lbm.

[0092] The input data address calculation unit calculates the starting address and data block size of each scheduled slice data block based on the starting address, slice size, and data block size of each input data block, and schedules the input data from the outside. The output data address calculation unit calculates the starting address and data block size of each scheduled slice data block based on the starting address, slice size, and data block size of each output data block, and schedules the generated output data to be output to the outside.

[0093] Each target output data scheduling unit calculates the size of each slice of data according to the block size and slice size specified in the configuration information, and then outputs the data. Once the entire task's data block size has been calculated, the data is output and status information is fed back to the task scheduling unit. The task scheduling unit sends a task response upon detecting the completion of the erasure task.

[0094] Please see Figure 4 , Figure 4 This is a schematic diagram of two consecutive task scheduling provided in an embodiment of this application. Figure 4This paper describes the working process of the hardware architecture using the hardware structure and working data flow method proposed in this application, taking the continuous processing of two erasure tasks as an example. As shown in the diagram, two tasks can be continuously fed into the hardware processing unit, achieving full-load data throughput. The six PE units output data according to the number of blocks of encoded and decoded output data for each task. It can be seen that the system generates erasure output data using a rolling scheduling method with small data slices. Figure 4 As shown, the vertical axis represents hardware resources, and the horizontal axis represents time. Task request input: Task Request 1 is 7+3 encoded. Input data: The scheduling unit outputs data blocks 1-6. The matrix scheduling unit outputs 7 slices (4KByte each) of encoded matrix 1 each time. The configuration scheduling unit outputs the configuration for Task 1. Output scheduling units 1-6 output 3 slices of parity blocks 1-3 each time. Task response output: The response of Task 1 is 7+3 encoded. Task request input: Task Request 1 is 7+3 decoded. Input data: The scheduling unit outputs parity blocks 1-6. The matrix scheduling unit outputs 7 slices (4KByte each) of decoded matrix 1 each time. The configuration scheduling unit outputs the configuration for Task 2. Output scheduling units 1-6 output 4 slices of data blocks 1-4 each time. Task response output: The response of Task 2 is 7+3 encoded.

[0095] This embodiment addresses RS erasure coding in distributed storage. It performs data slicing on large block-size tasks in hardware and dynamically rolls small data blocks to achieve erasure of large data blocks. This process is transparent to upper-layer application software and, through configurable slice sizes and hardware slicing methods, can be applied to hardware systems with different on-chip caches. Without this invention, when facing erasure tasks with large data blocks (e.g., 2Mbytes), one approach is to split the large data block erasure task into smaller ones at the software layer. This increases scheduling costs because a large number of task requests and responses are transmitted in the hardware system, increasing software overhead. Another approach is to add a very large buffer to each processing unit in hardware. For example, if the task is 2Mbytes in size, each processing unit needs a 2Mbyte buffer, which significantly increases on-chip cache resources and places high demands on the buffer size of the system input units.

[0096] This application also provides an erasure task processing method, which may include the following steps:

[0097] Determine the slice parameters corresponding to the erasure task; wherein, the slice parameters include: preset slice size, data block size of the erasure task, number of input data blocks, and number of output data blocks;

[0098] The first address information of each input slice data block is calculated based on the slice parameters; wherein, the first address information includes the starting address and data size of the input slice data block;

[0099] The second address information for each output slice data block is calculated based on the slice parameters; wherein the second address information includes the starting address and data size of the output slice data block;

[0100] The input and output of the erasure processing module are controlled based on the first address information and the second address information in order to complete the erasure task.

[0101] This embodiment determines the slicing parameters for the erasure task. Based on these parameters, the starting address and data size of each input slice data block can be determined, allowing the DMA module to control the input of the erasure processing module according to the starting address and data size of each input slice data block. Similarly, this embodiment determines the starting address and data size of each output slice data block based on the slicing parameters, allowing the DMA module to control the output of the erasure processing module according to the starting address and data size of each output slice data block. This scheme performs data slicing processing on the data blocks of the erasure task, achieving the erasure of larger data blocks through dynamic rolling scheduling of small data blocks. This process is transparent to upper-layer application software and, through configurable slice sizes and hardware slicing methods, can be applied to hardware systems with different on-chip caches.

[0102] Furthermore, determining the slice parameters corresponding to the erasure task includes:

[0103] The preset slice size is set according to the scheduling method of the DMA unit and the buffering capacity of the hardware output;

[0104] Alternatively, the preset slice size can be adaptively adjusted based on the current cache availability.

[0105] Furthermore, the erasure processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, an output data scheduling unit, a processing unit, and an output data scheduling unit;

[0106] The task scheduling unit is used to receive and parse the erasure task to obtain the parsing information, and send the parsing information to the data scheduling unit, the matrix scheduling unit, the output data scheduling unit and the data slice scheduling unit.

[0107] Furthermore, the data scheduling unit is used to input each input slice data block into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block.

[0108] Furthermore, the matrix scheduling unit is used to determine the encoding matrix and / or decoding matrix based on the number of input data blocks and the number of output data blocks, and input the determined encoding matrix and / or decoding matrix into the processing unit.

[0109] Furthermore, the configuration scheduling unit is used to generate configuration information based on the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.

[0110] Furthermore, the output data scheduling unit is used to determine whether the erasure task has been completed based on the configuration information; it is also used to return response information to the task scheduling unit after the erasure task has been completed.

[0111] Since the embodiments of the method section correspond to the embodiments of the system section, please refer to the description of the embodiments of the system section for the embodiments of the method section, and they will not be repeated here.

[0112] This application also provides a storage medium on which a computer program is stored, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0113] This application also provides an electronic device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the electronic device may also include various network interfaces, power supplies, and other components.

[0114] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0115] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A system for processing erasure tasks, characterized in that, include: A data slicing scheduling unit is used to determine the slicing parameters corresponding to the erasure task and send the slicing parameters to the input data address calculation unit and the output data address calculation unit; wherein, the slicing parameters include: preset slicing size, data block size of the erasure task, number of input data blocks and number of output data blocks; The input data address calculation unit is used to calculate the first address information of each input slice data block according to the slice parameters, and send the first address information to the DMA unit; wherein, the first address information includes the starting address and data size of the input slice data block; The output data address calculation unit is used to calculate the second address information of each output slice data block according to the slice parameters, and send the second address information to the DMA unit; wherein, the second address information includes the starting address and data size of the output slice data block; The DMA unit is used to control the input and output of the erasure processing module according to the first address information and the second address information, so as to complete the erasure task; The data slice scheduling unit includes: The slice parameter determination subunit is used to set the preset slice size according to the scheduling mode of the DMA unit and the cache capability of the hardware output; the cache capability includes buffer size and throughput. Alternatively, a parameter adaptive adjustment subunit is used to adaptively adjust the preset slice size based on the current cache idle status; The erasure processing module includes a task scheduling unit, an input data scheduling unit, a matrix scheduling unit, a processing unit, a configuration scheduling unit, and an output data scheduling unit; The task scheduling unit receives and parses the erasure task to obtain parsing information, and sends the parsing information to the input data scheduling unit, matrix scheduling unit, configuration scheduling unit, and data slice scheduling unit. The input data scheduling unit sends the input data of the erasure task to the PE processing unit that needs to be calculated according to the input data to be scheduled. The input data scheduling unit can ensure that no data loss occurs.

2. The erasure task processing system according to claim 1, characterized in that, The input data scheduling unit is used to input each input slice data block into the processing unit according to the number of input data blocks, the number of output data blocks, and the size of each input slice data block.

3. The erasure task processing system according to claim 1, characterized in that, The matrix scheduling unit is used to determine the encoding matrix and / or decoding matrix according to the number of input data blocks and the number of output data blocks, and input the determined encoding matrix and / or decoding matrix into the processing unit.

4. The erasure task processing system according to claim 1, characterized in that, The configuration scheduling unit is used to generate configuration information based on the number of input data blocks, the number of output data blocks, and the size of each input slice data block, and input the configuration information into the processing unit and the output data scheduling unit.

5. The erasure task processing system according to claim 4, characterized in that, The output data scheduling unit is used to determine whether the erasure task is completed based on the configuration information; it is also used to return response information to the task scheduling unit after the erasure task is completed.

6. A method for processing erasure tasks, characterized in that, include: Determine the slice parameters corresponding to the erasure task; wherein, the slice parameters include: preset slice size, data block size of the erasure task, number of input data blocks, and number of output data blocks; The first address information of each input slice data block is calculated based on the slice parameters; wherein, the first address information includes the starting address and data size of the input slice data block; The second address information for each output slice data block is calculated based on the slice parameters; wherein the second address information includes the starting address and data size of the output slice data block; The input and output of the erasure processing module are controlled according to the first address information and the second address information in order to complete the erasure task; the erasure processing module is the erasure processing module in the erasure task processing system of any one of claims 1 to 5; Determining the slice parameters corresponding to the erasure task includes: setting the preset slice size according to the scheduling mode of the DMA unit and the cache capacity output by the hardware; the cache capacity includes buffer size and throughput; or, a parameter adaptive adjustment subunit is used to adaptively adjust the preset slice size according to the current cache idle status.

7. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the erasure task processing method as described in claim 6.

8. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the erasure task processing method as described in claim 6 above.

Images