Data processing method and device and electronic equipment

Abstract

The invention provides a data processing method and device and electronic equipment, and relates to the technical field of data processing.The method comprises the steps that data are read through each data reading channel, and the read data are stored in a first cache configured for the data reading channel; according to a pre-configured transposition calculation principle, carrying out segmented transposition on the data written into the first cache, and writing a plurality of data segments after segmented transposition into a plurality of second caches; and each processing unit reads each data segment of the data in parallel from the corresponding second cache to complete erasure code calculation. According to the data processing method and device and the electronic equipment provided by the invention, based on the second cache, even if the data read by each data reading channel is delayed, namely the situation of asynchronous reading occurs, parallel computing processing can be realized based on the data stored in the transfer blocks in the second cache, so that the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method, device and electronic equipment. Background Art

[0002] Erasure Coding (EC) is a data protection method that encodes the original k data blocks to generate m check blocks. These k data blocks and m check blocks are stored together. When no more than n blocks (n represents the data block and check block, and n ≤ m) of data are damaged, the original n blocks of data can be restored through reconstruction. In this mode, the disk space utilization is k / (k+m), and the reliability of the data block is determined by the size of m. The larger the m, the higher the reliability.

[0003] When EC calculations involve the continuous encoding of multiple data blocks, the time complexity is high, seriously affecting the efficiency of parallel processing. Existing improvement solutions often sacrifice bandwidth to improve time efficiency, reducing disk utilization and making it difficult to achieve a balanced improvement in processing efficiency. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a data processing method, device and electronic device to alleviate the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, which is applied to a computing module, wherein the computing module is configured with multiple data reading channels, each data reading channel is configured with a first cache, a second cache and a processing unit, and the method includes: reading data through each of the data reading channels, and storing the read data into the first cache configured by the data reading channel; according to a pre-configured transposition calculation principle, segmented transposing the data written into the first cache through each of the data reading channels, and writing the multiple data segments after segmented transposition into multiple second caches; each processing unit reads the data segments of the data in parallel from the corresponding second cache to complete the erasure code calculation.

[0006] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation method of the first aspect, wherein the above-mentioned data includes multiple data blocks, and each of the data blocks is segmented and transposed into multiple data segments; the step of writing the multiple data segments after segmented transposition into multiple second caches includes: dividing each data block into the same number of data segments according to the number of the multiple data reading channels; reading the multiple data segments included in the data block from the first cache according to the transposition calculation principle, transposing them in sequence and writing them into multiple second caches; wherein the transposition calculation principle is used to characterize the order in which the multiple data segments are written from the first cache to each second cache.

[0007] In combination with the first possible implementation of the first aspect, an embodiment of the present invention provides a second possible implementation of the first aspect, wherein each of the above-mentioned second caches allocates corresponding cache space to the multiple data reading channels, and the method further includes: writing the read data segment to the cache space allocated by the second cache for each of the data reading channels.

[0008] In combination with the first aspect, an embodiment of the present invention provides a third possible implementation of the first aspect, wherein the above-mentioned data includes multiple data blocks; the step of reading data through each of the data reading channels and storing the read data into the first cache configured by the data reading channel includes: reading the data blocks of the data in sequence through each of the data reading channels, and storing the read data blocks into the first cache configured by the data reading channel; in response to the completion of reading each of the data blocks, transposing the data block in segments and writing them into multiple second caches.

[0009] In combination with the second possible implementation of the first aspect, an embodiment of the present invention provides a fourth possible implementation of the first aspect, wherein the above-mentioned data reading channel includes a first reading channel and a second reading channel; the data read by the first reading channel is first data, and the data read by the second reading channel is second data; each second cache is allocated cache space corresponding to the first reading channel and the second reading channel; the step of storing the read data into the first cache configured by the data reading channel includes: storing the first data read by the first reading channel into the first cache of the first reading channel, and storing the second data read by the second reading channel into the first cache of the second reading channel.

[0010] In combination with the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, wherein the step of segmentally transposing the data written into the first cache through each of the data read channels, and writing the multiple data segments after the segmented transposition into the multiple second caches includes:

[0011] Read a data segment of each data block of the first data from the first cache of the first read channel, and write the data segments in sequence into the cache space allocated for the first read channel in the multiple second caches; at the same time, read a data segment of each data block of the second data from the first cache of the second read channel; and write the data segments into the cache space allocated for the second read channel in the multiple second caches.

[0012] In combination with the first aspect, an embodiment of the present invention provides a sixth possible implementation of the first aspect, wherein each of the above-mentioned processing units reads the data segments of the data in parallel from the corresponding second cache to complete the erasure code calculation, including: in response to the completion of the writing of the data into the second cache, each of the processing units reads the data segments of the data in parallel from the corresponding second cache; encodes the data segments read from each second cache based on the calculation matrix corresponding to the data; and merges the encoding results of each of the processing units to obtain the erasure code of the data.

[0013] In the second aspect, an embodiment of the present invention also provides a data processing device, including a computing module, wherein the computing module is configured with multiple data reading channels, and each data reading channel is configured with a first cache, a second cache and a processing unit. The device also includes: a reading module, used to read data through each of the data reading channels, and store the read data into the first cache configured by the data reading channel; a processing module, used to segmentally transpose the data written into the first cache through each of the data reading channels according to a pre-configured transposition calculation principle, and write the multiple data segments after segmented transposition to multiple second caches; a parallel module, used to read the data segments of the data in parallel from the corresponding second cache through each processing unit to complete the erasure code calculation.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are executed.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] A data processing method, device and electronic device provided by an embodiment of the present invention can read data through each data reading channel and store the read data into a first cache configured for the data reading channel; according to a pre-configured transposition calculation principle, the data written into the first cache through each data reading channel is segmented and transposed, and multiple data segments after segmented transposition are written into multiple second caches; each processing unit reads each data segment of the data in parallel from the corresponding second cache to complete the erasure code calculation. The above-mentioned method of writing data segments to multiple second caches can allow the data read by each data reading channel to be delayed, that is, even if the reading is asynchronous, parallel computing and processing can be achieved based on the segmented transposed data in the second cache, thereby improving data processing efficiency.

[0018] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram for generating verification data and a schematic diagram for recovering data;

[0022] Figure 2 A flowchart of a data processing method provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a data processing process provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a data transposition timeline provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of cache space division of a second cache provided by an embodiment of the present invention;

[0026] Figure 6 A calculation process for transposed blocks provided by an embodiment of the present invention;

[0027] Figure 7 A schematic structural diagram of a data processing device provided by an embodiment of the present invention;

[0028] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0030] Generally, the calculation of erasure codes is mainly divided into two types: check block generation and data recovery. Figure 1 The check data generation diagram and data recovery diagram shown in the figure, wherein the check block generation is the process of multiplying the vector composed of k data blocks and the k×m generator matrix to obtain k+m data blocks and check blocks, that is, Figure 1 In the process shown on the left (a), when a data block is damaged, as long as the damaged data block does not exceed m blocks, the damaged data can be recovered by matrix multiplication of the intact data block and the recovery matrix, that is, Figure 1 The process shown on the right side (b).

[0031] To improve computational efficiency, multi-channel computing circuits are often used to complete matrix calculations in parallel to save time. However, each channel may have different delays due to media issues, and the input of data blocks will not be completely synchronized. The calculation completion time will be greatly affected by channel delays.

[0032] Based on this, the embodiments of the present invention provide a data processing method, device, and electronic device that can achieve a more efficient parallel computing process.

[0033] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present invention is first introduced in detail.

[0034] In a possible implementation, an embodiment of the present invention provides a data processing method, which is applied to a computing module. The computing module is configured with multiple data reading channels, and each data reading channel is configured with a first cache, a second cache, and a processing unit. Specifically, Figure 2 A flow chart of a data processing method is shown in FIG. Figure 2 As shown, the following steps are included:

[0035] Step S202, reading data through each data reading channel, and storing the read data into a first cache configured for the data reading channel;

[0036] Step S204 , according to a pre-configured transposition calculation principle, the data written into the first cache through each data reading channel is segmented and transposed, and the plurality of data segments after the segmented transposition are written into the plurality of second caches;

[0037] In step S206 , each processing unit reads each data segment of the data from the corresponding second cache in parallel to complete erasure code calculation.

[0038] In actual use, the above-mentioned calculation module is used to read data from the data source, and then perform certain calculations on the read data and write it to the cache for subsequent use. In order to achieve more efficient parallel computing, in an embodiment of the present invention, two caches are set for each data reading channel, namely the above-mentioned first cache and second cache, wherein the set second cache can continuously read and write data from the first cache for transposition and block processing, and the second cache is set for each first cache, that is, the first cache is configured to correspond to multiple second caches to achieve parallel processing of data.

[0039] Moreover, based on the above-mentioned second cache, even if the data read by each data reading channel is delayed, that is, the reading is asynchronous, parallel computing can be achieved based on the segmented transposed data in the second cache, thereby improving data processing efficiency.

[0040] Furthermore, the transposition calculation principle in the embodiments of the present invention is used to characterize the order in which multiple data segments are written from the first cache to each second cache. Specifically, based on this transposition calculation principle, the multiple second caches corresponding to the data read channels, the cache space allocated by each second cache for each data read channel, the order in which the data is placed in the cache space, and the time sequence in which the data is written from the first cache to the second cache can be recorded.

[0041] Furthermore, the above data includes multiple data blocks. Through the above step S204, each data block can be segmented and transposed into multiple data segments.

[0042] Specifically, in the above step S204, the step of writing the multiple data segments after segment transposition into multiple second caches includes: dividing each data block into the same number of data segments according to the number of multiple data reading channels; reading the multiple data segments included in the data block from the first cache according to the above transposition calculation principle, transposing them in sequence and writing them into multiple second caches.

[0043] Furthermore, in an embodiment of the present invention, each second cache allocates corresponding cache space to multiple data reading channels. Therefore, when writing each data segment into the second cache, the read data segment can be written into the cache space allocated by the second cache for each data reading channel.

[0044] In actual use, considering that the data read by the above-mentioned data reading channel includes multiple data blocks; therefore, when performing segmented transposition on the data written into the first cache through each data reading channel, the process can be performed after all the data blocks are written into the first cache, or segmented transposition can be performed on the data block after one data block is cached, that is, data blocks of the data are read in sequence through each data reading channel, and the read data blocks are stored in the first cache configured by the data reading channel; in response to the completion of reading each data block, the data block is segmentedly transposed and written into multiple second caches.

[0045] The specific execution situation can be set according to actual usage, and the embodiment of the present invention does not limit this.

[0046] Furthermore, in an embodiment of the present invention, the data reading channel generally includes a first reading channel and a second reading channel; wherein the data read by the first reading channel is the first data, and the data read by the second reading channel is the second data; and the first reading channel and the second reading channel are provided with two caches, namely the first cache and the second cache. For ease of understanding, Figure 3 A schematic diagram of a data processing process is shown, wherein: Figure 3 In the figure, the block indicated by the dotted line is a computing unit. Taking two data reading channels as an example, namely, RD_CH0 and RD_CH1, these two data reading channels read data from the data source Source Space, and, Figure 3 The figure also shows a first buffer PRE_BUF and a second buffer CAL_BUF, wherein the first buffer PRE_BUF0 and the second buffer CAL_BUF0 are the first buffer and the second buffer of the first read channel RD_CH0, and the first buffer PRE_BUF1 and the second buffer CAL_BUF1 are the first buffer and the second buffer of the second read channel RD_CH1.

[0047] further, Figure 3 In FIG, a write buffer WRITE BUFFER and a write channel WR_CH are also shown. The write channel WR_CH eventually writes the data into the result space result space.

[0048] Furthermore, each second cache is allocated with cache space corresponding to the first read channel and the second read channel; based on Figure 3In the data processing process diagram shown, the first buffer PRE_BUF0 and the first buffer PRE_BUF1 point to the second buffer CAL_BUF0 and the second buffer CAL_BUF1 respectively, that is, each second buffer is allocated with the buffer space of the first buffer PRE_BUF0 and the first buffer PRE_BUF1.

[0049] And, based on Figure 3 The data processing process diagram shown in the above step S202, the process of storing the read data into the first cache configured by the data reading channel includes storing the first data read by the first reading channel into the first cache of the first reading channel, and storing the second data read by the second reading channel into the first cache of the second reading channel.

[0050] Specifically, for ease of understanding, based on Figure 3 The data processing process shown is described by taking the example of reading the data group km(6,2) from the data source, the number of data reading channels being 2, and the size of each data block being 4 KB.

[0051] When reading data from a data channel, the first read channel RD_CH0 needs to read data blocks a0-a5 and store them in the first buffer PRE_BUF0. During the actual reading process, once the first read channel RD_CH0 has read a complete database block, such as a0, subsequent operations can be performed. The second read channel RD_CH1 needs to read data blocks b0-b5 and store them in the first buffer PRE_BUF1.

[0052] Then, in the above step S204, the data written into the first cache by each data reading channel is segmented and transposed, and then written into the second cache of each data reading channel, so that the parallel processing units of multiple data reading channels (i.e., Figure 3 The PE (processing element) in the process can use the same generator matrix GM to complete the calculation of a group of data blocks from the same data reading channel in parallel.

[0053] Furthermore, considering that each cache usually has only one read data port and one write data port, in order to avoid data conflicts, in an embodiment of the present invention, the data transposition timeline from the first cache to the second cache is characterized by the above-mentioned transposition calculation principle. Moreover, based on the above-mentioned transposition calculation principle, in an embodiment of the present invention, when writing data to the second cache, data blocks in different data reading channels are written to the second cache in sequence in the time sequence of writing. At the same time, in the second cache, the cache space is divided according to the data reading channel, and the data is written into the areas corresponding to different data reading channels respectively.

[0054] further, Figure 4A schematic diagram showing a data transposition timeline, and Figure 5 A schematic diagram of the cache space division of the second cache is shown. Similarly, Figure 4 and Figure 5 All Figure 3 The data processing process is described as an example.

[0055] in, Figure 4 In , TIME represents the timeline, that is, the order in which the second cache reads data over time, and, Figure 4 In the example of data block a0, a0L and a0H are the data segments of data block a0, which usually represent the low-order segment or high-order segment of a0, or are called the first half and the second half. That is, data block a0 is divided into two data segments, a0L and a0H, according to the number of second caches or the number of data reading channels. At the same time, Figure 5 In the embodiment, for each second cache, a corresponding cache control is divided for each data reading channel, ie, CH0 and CH1, which respectively store the data segment of each data block.

[0056] Further, based on Figure 4 and Figure 5 , Figure 6 A calculation process of segmented transposition is also shown. Specifically, based on Figure 6 In an embodiment of the present invention, the specific process of transposing and dividing the data in the first cache and writing the data into the second cache includes the following steps:

[0057] (1) Read the first data segment of each data block of the first data from the first buffer PRE_BUF0 of the first read channel RD_CH0, and write the first data segment into the buffer space allocated for the first read channel in the second buffer of the first read channel; at the same time, read the second data segment of each data block of the second data from the first buffer PRE_BUF1 of the second read channel RD_CH1; and write the second data segment into the buffer space allocated for the second read channel in the second buffer of the second read channel.

[0058] Specifically, taking a0 as an example, the first half data a0L (2048B) of a0 is read from the first cache PRE_BUF0 of the first read channel RD_CH0 and stored in the corresponding position of the ch0 segment of CAL_BUF0; at the same time, the second half data b0H (2048B) of b0 is read from the first cache PRE_BUF1 of the second read channel RD_CH1 and stored in the corresponding position of the ch1 segment of CAL_BUF1; that is, the read data segment is written to the cache space allocated by the second cache for each data read channel.

[0059] (2) Response writing is completed;

[0060] (3) Read the second data segment of each data block of the first data from the first buffer PRE_BUF0 of the first read channel RD_CH0, and write the second data segment into the buffer space allocated for the first read channel by the second buffer of the second read channel; at the same time, read the first data segment of each data block of the second data from the first buffer PRE_BUF1 of the second read channel RD_CH1, and write the first data segment into the buffer space allocated for the second read channel by the second buffer of the first read channel.

[0061] Taking a0 as an example, in (3), the second half of a0 data a0H (2048B) is read from the first buffer PRE_BUF0 of the first read channel RD_CH0 and stored in the corresponding position of the ch0 segment of CAL_BUF1; at the same time, the first half of b0 data b0L (2048B) is read from the first buffer PRE_BUF1 of the second read channel RD_CH1 and stored in the corresponding position of the ch1 segment of CAL_BUF1.

[0062] At this point, the a0 data block is completely written into the second cache.

[0063] By analogy, data is continuously read from the first buffers PRE_BUF0 and PRE_BUF1 and alternately written into the second buffers CAL_BUF0 and CAL_BUF1 until all data blocks complete the above-mentioned segmented transposition calculation process.

[0064] Furthermore, in the embodiment of the present invention, when the second cache reads data from the first cache, it reads data in an alternating manner, and in the second cache, the cache space of each data reading channel is set together, that is, arranged in sequence according to the reading order.

[0065] Furthermore, in an embodiment of the present invention, Figure 3 PE0 and PE1 in the figure are processing units configured for each data reading channel. In addition, in an embodiment of the present invention, further, when performing erasure code calculation, each processing unit can respond to the completion of data writing into the second cache, and each processing unit can read each data segment of the data from the corresponding second cache in parallel; encode the data segments read from each second cache based on the calculation matrix corresponding to the data; and merge the encoding results of each processing unit to obtain the erasure code of the data.

[0066] Specifically, the same as above Figure 3 For example, after the above-mentioned transposition calculation process is completed, the two groups of PEs, i.e., PE0 and PE1, can respectively read the data stored in the corresponding second cache to perform EC matrix calculation. Moreover, based on the above-mentioned transposition calculation process, for the same group of EC blocks, the calculation type is consistent. Therefore, the two groups of PEs only need to share a set of parameters. For example, the same generation matrix GM can be used to complete the calculation of the same group of data blocks from the same data reading channel in parallel.

[0067] Furthermore, in the embodiment of the present invention, based on the segmented transposition calculation process of the second cache, even if the data input delays of the various data reading channels are different, the calculation can be completed with high parallel efficiency.

[0068] In addition, even if only one data read channel has data transmission, the PEs of all data read channels can be fully utilized to complete the calculation, and the efficiency of multi-channel parallelism can still be achieved.

[0069] Therefore, the data processing method provided in the embodiment of the present invention has the following technical effects:

[0070] (1) For the data of the same set of data blocks, the type of processing required for the data is consistent. Therefore, the same control logic and parameters of the generation matrix or recovery matrix can be shared during calculation, which can effectively reduce chip resources.

[0071] (2) The PEs of each data reading channel jointly provide calculations for a set of data. Even when multiple data reading channels are not synchronized, all computing resources can be involved, shortening the delay of a single calculation.

[0072] Furthermore, based on the above embodiment, an embodiment of the present invention further provides a data processing device, including a computing module, the computing module is configured with multiple data reading channels, and each data reading channel is configured with a first cache, a second cache and a processing unit, such as Figure 7 The schematic structural diagram of a data processing device shown in FIG. 1 shows a calculation module 70, and the device further includes:

[0073] a reading module 72, configured to read data through each of the data reading channels and store the read data into a first cache configured for the data reading channel;

[0074] a processing module 74 configured to perform segmented transposition on the data written into the first cache through each of the data reading channels according to a pre-configured transposition calculation principle, and write the plurality of segmented transposed data segments into a plurality of second caches;

[0075] The parallel module 76 is configured to read the data segments of the data from the corresponding second cache in parallel through each of the processing units to complete erasure code calculation.

[0076] The data processing device provided in the embodiment of the present invention has the same technical features as the data processing method provided in the above embodiment, and therefore can also solve the same technical problems and achieve the same technical effects.

[0077] Furthermore, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0078] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0079] Furthermore, an embodiment of the present invention also provides a structural diagram of an electronic device, such as Figure 8 As shown, it is a structural diagram of the electronic device, wherein the electronic device includes a processor 81 and a memory 80, the memory 80 stores computer executable instructions that can be executed by the processor 81, and the processor 81 executes the computer executable instructions to implement the above method.

[0080] exist Figure 8 In the illustrated embodiment, the electronic device further includes a bus 82 and a communication interface 83 , wherein the processor 81 , the communication interface 83 and the memory 80 are connected via the bus 82 .

[0081] Among them, the memory 80 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 83 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 82 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 82 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0082] Processor 81 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be performed by hardware integrated logic circuits within processor 81 or by software instructions. The processor 81 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory, and the processor 81 reads the information in the memory and completes the above method in combination with its hardware.

[0083] The computer program products of the data processing methods, devices, and electronic devices provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0085] In addition, in the description of the embodiments of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0086] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0087] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0088] Finally, it should be noted that the above embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: Applied to a computing module, the computing module is configured with a plurality of data reading channels, each data reading channel is configured with a first cache, a second cache and a processing unit, the method comprises: Reading data through each of the data reading channels, and storing the read data into a first cache configured for the data reading channel; According to a pre-configured transposition calculation principle, the data written into the first cache through each of the data reading channels is transposed in segments, and a plurality of data segments after the segmented transposition are written into a plurality of the second caches; Each of the processing units reads each data segment of the data in parallel from the corresponding second cache to complete erasure code calculation.

2. The method according to claim 1, characterized in that The data includes a plurality of data blocks, and each of the data blocks is segmented and transposed into a plurality of data segments; The step of writing the plurality of data segments after segment transposition into the plurality of second caches comprises: Dividing each data block into the same number of data segments according to the number of the multiple data reading channels; According to the transposition calculation principle, multiple data segments included in the data block are read from the first cache, and are sequentially transposed and written into multiple second caches; The transposition calculation principle is used to characterize the order in which the multiple data segments are written from the first cache to each of the second caches.

3. The method according to claim 2, characterized in that Each of the second caches allocates a corresponding cache space to the multiple data reading channels, and the method further includes: The read data segment is written into the cache space allocated by the second cache for each of the data reading channels.

4. The method according to claim 1, characterized in that: The data includes a plurality of data blocks; The step of reading data through each of the data reading channels and storing the read data into a first cache configured for the data reading channel comprises: Reading data blocks of the data in sequence through each of the data reading channels, and storing the read data blocks into a first cache configured for the data reading channel; In response to completion of reading each of the data blocks, the data block is transposed in segments and written into a plurality of the second caches.

5. The method according to claim 3, characterized in that: The data reading channel includes a first reading channel and a second reading channel; the data read by the first reading channel is first data, and the data read by the second reading channel is second data; Each second cache is allocated with cache space corresponding to the first reading channel and the second reading channel; The step of storing the read data into a first cache configured in the data reading channel comprises: The first data read by the first reading channel is stored in a first buffer of the first reading channel, and the second data read by the second reading channel is stored in a first buffer of the second reading channel.

6. The method according to claim 5, characterized in that The step of transposing the data written into the first cache through each of the data reading channels in segments, and writing the plurality of data segments after the segment transposition into the plurality of second caches comprises: reading a data segment of each data block of the first data from the first cache of the first read channel, and sequentially writing the data segment into the cache space allocated for the first read channel in the plurality of second caches; and at the same time, A data segment of each data block of the second data is read from the first cache of the second read channel; and the data segment is written into a cache space allocated for the second read channel in the plurality of second caches.

7. The method according to claim 1, characterized in that Each of the processing units reads the data segments of the data in parallel from the corresponding second cache to complete erasure code calculation, including: In response to completion of writing the data into the second cache, each of the processing units reads each data segment of the data from the corresponding second cache in parallel; encoding the data segments read from each second cache based on the calculation matrix corresponding to the data; The encoding results of the processing units are combined to obtain the erasure code of the data.

8. A data processing device, characterized in that: The device comprises a computing module, wherein the computing module is configured with a plurality of data reading channels, and each data reading channel is configured with a first cache, a second cache and a processing unit. The device further comprises: A reading module, used for reading data through each of the data reading channels, and storing the read data into a first cache configured for the data reading channel; A processing module, configured to perform segmented transposition of the data written into the first cache through each of the data reading channels according to a pre-configured transposition calculation principle, and write a plurality of segmented transposed data segments into a plurality of the second caches; The parallel module is used to read the data segments of the data in parallel from the corresponding second cache through each processing unit to complete the erasure code calculation.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are executed.

Images