Data processing method, electronic device, storage medium and product

Abstract

The present application discloses a data processing method, electronic device, storage medium and product, which relate to the field of data processing, including: responding to a first operation instruction, dividing a file to be processed to obtain multiple data blocks corresponding to the file to be processed; performing a first processing operation on a first data block among the multiple data blocks, and simultaneously performing a second processing operation on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation according to the execution order of the multiple processing operations, the multiple processing operations include at least two of reading processing, compression processing, encryption processing and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing and writing processing. Parallel data processing with coordinated software and hardware can be realized, and the requirements of efficient data transmission, efficient and transparent compression and encryption, and efficient and transparent decompression and decompression can be met.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, electronic equipment, storage medium and product. Background Art

[0002] With the advent of the big data era, the demand for data security is increasing, and the amount of data is also growing rapidly. In order to protect the security of data, data encryption technology has become indispensable. In order to improve data storage efficiency, data compression and decompression technology in data centers has also received more and more attention in the industry.

[0003] At present, relevant technologies can use software methods to encrypt and decrypt data and compress and decompress data, but software methods for encryption and decryption and software compression and decompression require a large amount of software computing resources, resulting in reduced efficiency of other programs. In addition, some hardware-implemented encryption and decryption and software compression and decompression methods in relevant technologies are relatively cumbersome to operate, cannot achieve transparent operation, and have a poor user experience. Summary of the Invention

[0004] The present application provides a data processing method, an electronic device, a storage medium, and a product to at least solve the problems in the related art of low efficiency of data encryption and decryption, compression and decompression methods, and the inability to achieve transparent compression, transparent encryption, transparent decryption, and transparent decompression of files.

[0005] The present application provides a data processing method, comprising: in response to a first operation instruction, dividing a file to be processed to obtain multiple data blocks corresponding to the file to be processed; performing a first processing operation on a first data block among the multiple data blocks, and simultaneously performing a second processing operation on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation in the execution order of the multiple processing operations, and the multiple processing operations include at least two of reading processing, compression processing, encryption processing and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing and writing processing.

[0006] The present application also provides a data processing device, including: a first processing unit, for dividing a file to be processed in response to a first operation instruction to obtain multiple data blocks corresponding to the file to be processed; a second processing unit, for performing a first processing operation on a first data block among the multiple data blocks, and at the same time performing a second processing operation on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation in the execution order of the multiple processing operations, and the multiple processing operations include at least two of reading processing, compression processing, encryption processing and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing and writing processing.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data processing methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0009] The present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0010] Through the present application, in response to a first operation instruction, a file to be processed is divided to obtain multiple data blocks corresponding to the file to be processed; a first processing operation is performed on a first data block among the multiple data blocks, and a second processing operation is performed on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation according to the execution order of the multiple processing operations, and the multiple processing operations include at least two of reading processing, compression processing, encryption processing, and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing, and writing processing. Parallel data processing with coordinated software and hardware can be achieved, and the requirements of efficient data transmission, efficient and transparent compression and encryption, and efficient and transparent decryption and decompression can be met simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1A flowchart of a data processing method provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;

[0014] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of the topology of a PCIe bus protocol provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of a traditional DMA data transmission solution provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of a point-to-point DMA data transmission solution provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of implementing point-to-point data stream transmission between a disk and an FPGA provided in an embodiment of the present application;

[0019] Figure 8 A schematic diagram of the single-step execution steps of a data encryption and decryption algorithm and a query algorithm provided in an embodiment of the present application when offloading to FPGA hardware;

[0020] Figure 9 A design scheme for a compression and encryption pipeline software stack provided in an embodiment of the present application;

[0021] Figure 10 A serial system call method provided in an embodiment of the present application;

[0022] Figure 11 A four-stage pipeline software stack design for data decryption and decompression provided in an embodiment of the present application;

[0023] Figure 12 A schematic diagram of the structure of a compression encryption, decryption and decompression heterogeneous computing data transmission solution provided in an embodiment of the present application;

[0024] Figure 13 A schematic diagram of a specific implementation process of transparent data compression and encryption provided in an embodiment of the present application;

[0025] Figure 14 A schematic diagram of a specific implementation process of transparent decryption and decompression of data provided in an embodiment of the present application;

[0026] Figure 15A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] With the advent of the big data era, the demand for data security is increasing, and the amount of data is also growing rapidly. In order to protect the security of data, data encryption technology has become indispensable. In order to improve data storage efficiency, data compression and decompression technology in data centers has also received more and more attention in the industry.

[0030] Regarding data security and data storage technology, traditional technology can use software methods to encrypt and decrypt data and perform software compression and decompression. However, this traditional software encryption and decryption, software compression and decompression method has many disadvantages in the computer system of the data center. Both software encryption and decryption and software compression and decompression fall into the category of software acceleration technology, and ultimately require the integration of the Central Processing Unit (CPU) to execute. This will consume a large amount of CPU computing resources in the server and will affect other software programs in the system. In addition, the efficiency of software acceleration technology is generally low, and the acceleration effect is poor.

[0031] Traditional data compression, decompression, and encryption / decryption are typically implemented in software, but this approach suffers from the following issues: 1) Slow processing speed: Software-implemented data compression, decompression, and encryption rely on CPU processing power. For large-scale data processing, CPU performance bottlenecks can lead to slow processing speeds. 2) High CPU resource consumption: Software implementations require significant CPU and memory resources, especially in multitasking environments, where resource competition can lead to performance degradation.

[0032] To address these issues, hardware acceleration technology has been introduced into data compression, decompression, and encryption / decryption systems. Existing hardware solutions lack efficient data transmission, transparent operation, or pipelined software stack design. They also lack efficient parallel scheduling of hardware and software computing IP, hindering effective hardware and software collaboration.

[0033] While these solutions address some of the data processing challenges, they cannot simultaneously meet the requirements of high-speed data transmission, high security, and high storage efficiency. Therefore, how to achieve both efficient data transmission and transparent compression and decompression, as well as transparent encryption and decryption computations at the hardware level, remains a pressing challenge.

[0034] In order to solve the problems of slow data processing speed, high resource usage, and lack of transparent operation in related solutions, an embodiment of the present application provides a data processing method. By introducing a transparent file system and designing two four-level pipeline software stacks inside it, efficient data transmission, efficient data transparent compression and encryption, and efficient data transparent decryption and decompression are achieved. It aims to solve a series of problems existing in related technologies, such as low data storage efficiency, high CPU resource consumption, low security, low transmission efficiency, poor flexibility and scalability, and provide an efficient, secure and flexible data transmission, compression and decompression, encryption and decryption processing solution.

[0035] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure.

[0037] like Figure 1 As shown, the method can be executed by a host, and the method includes the following steps:

[0038] Step 101: In response to a first operation instruction, a file to be processed is divided to obtain a plurality of data blocks corresponding to the file to be processed.

[0039] In some embodiments, the files to be processed may be, for example, video files, document files, etc. The host of this solution may include an operating system and a file system. This embodiment uses a Linux operating system as an example for description. The system may be used to perform multiple processing operations on the files to be processed, wherein the multiple processing operations include at least two of reading, compression, encryption, and writing, or the multiple processing operations include at least two of reading, decryption, decompression, and writing.

[0040] In some embodiments, the method further includes: determining that the first operation instruction is a data write operation instruction when the multiple processing operations include at least two of reading, compressing, encrypting, and writing; and determining that the first operation instruction is a data read operation instruction when the multiple processing operations include at least two of reading, decrypting, decompressing, and writing. For example, when a file needs to be stored (written) or transmitted, the file to be processed can be read, compressed, encrypted, and written; when a file to be processed needs to be read or transmitted, the file to be processed can be read, decrypted, decompressed, and written, i.e., when the file to be processed needs to be transmitted.

[0041] In some embodiments, the host may include a user layer and a kernel layer, which are two different layers in the host-side operating system. The user can trigger a first operation instruction in the user layer of the host. Optionally, after the user initiates a data write operation, the data write operation enters the kernel layer through a system call. The kernel layer can divide the file to be processed in response to the data write operation and obtain multiple data blocks corresponding to the file to be processed. Similarly, after the user initiates a data read operation, the data read operation enters the kernel layer through a system call. The kernel layer can divide the file to be processed in response to the data read operation and obtain multiple data blocks corresponding to the file to be processed.

[0042] Optionally, when the first operation instruction is a data write operation instruction, when the user triggers the first operation instruction, he can input the file to be processed (file to be written) that needs to be transferred or stored. After that, when the data write operation enters the kernel layer through a system call, the file to be processed can be transferred to the kernel layer. The kernel layer can first temporarily store the file to be processed in the memory of the operating system, and then divide the file to be processed into blocks; when the first operation instruction is a data read operation instruction, after the user triggers the first operation instruction, he can divide the file to be processed into blocks at the location where the file to be processed (file to be read) is stored.

[0043] In some embodiments, when a file to be processed is divided into blocks to obtain multiple data blocks corresponding to the file to be processed, a task block corresponding to each data block may be generated. Each task block may include information related to the data block corresponding to the task block. That is, when the data blocks are generated, information related to each data block, such as the size and location of the data block, may be determined. Optionally, the specific method for dividing the file to be processed into blocks is not limited in this disclosure. For example, the file to be processed may be divided according to the amount of data.

[0044] Heterogeneous computing: Heterogeneous computing refers to the use of computing resources from multiple different architectures, processors, or accelerators, combined to achieve higher performance and greater efficiency. This approach can improve overall system performance by assigning tasks to the devices best suited to execute them. Heterogeneous computing typically involves different types of processors, such as CPUs, graphics processing units (GPUs), and field-programmable gate arrays (FPGAs), to accelerate and optimize a variety of applications.

[0045] As a programmable hardware platform, FPGAs offer high flexibility and parallel processing capabilities, making them ideal for implementing complex, heterogeneous accelerated computing and data processing tasks. However, while some solutions currently implement FPGA encryption and decryption, as well as compression and decompression, these solutions generally focus solely on hardware implementation, failing to achieve transparent compression and encryption of files, nor transparent decryption and decompression of files at the software level. Current solutions also lack efficient data transmission at the software level.

[0046] Encryption and Decryption: Encryption and decryption refer to the process of protecting or concealing data through the use of cryptographic techniques. Encryption converts raw data into data processed using a specific algorithm, making it difficult to understand or interpret by unauthorized access. Decryption restores encrypted data to its original form. Encryption and decryption are commonly used to ensure the confidentiality, integrity, and authentication of data, preventing unauthorized access and modification. This technology is widely used in information security fields, such as data transmission, storage, and communication.

[0047] Compression: Processing data through specific algorithms to reduce the storage space occupied by the data without losing or minimizing the loss of data information.

[0048] Decompression is the reverse process of compression, which restores compressed data to its original form. Decompression algorithms decode and restore compressed data based on the compression algorithm's rules and encoding methods.

[0049] Step 102: Perform a first processing operation on a first data block among the multiple data blocks, and simultaneously perform a second processing operation on a second data block among the multiple data blocks.

[0050] In some embodiments, as Figure 9 As shown, the user layer and the kernel layer are two different layers in the host-side operating system. After the kernel layer divides the file to be processed into blocks, it can enter the pipeline software stack to perform processing operations on the data blocks. Optionally, the kernel layer can schedule the execution unit of the device layer to perform processing operations on the data blocks through a scheduling function.

[0051] In the solution of the present disclosure, the device layer may include a second storage device for storing data and an execution unit for performing processing on the data block, wherein the second storage device may be, for example, a disk, such as an NVME-SSD disk, a mechanical hard disk, a USB flash drive, etc., which is not limited by the present disclosure; the execution unit may be, for example, an accelerator board device, such as an FPGA accelerator board. FPGA is a programmable hardware platform with high flexibility and parallel processing capabilities, and is very suitable for implementing complex data processing tasks.

[0052] Optionally, the execution unit may include a data encryption intellectual property core (Intellectual Property Core, IP), a data compression IP, DEV_DMA-IP, and a device memory, in which case the execution unit may perform read processing, encryption processing, compression processing, and write processing on the data block; or, the execution unit may include a data decryption IP, a data decompression IP, DEV_DMA-IP, and a device memory, in which case the execution unit may perform read processing, decryption processing, decompression processing, and write processing on the data block, where the intellectual property core is an integrated circuit design module with specific functions.

[0053] Optionally, the device layer may include one or more execution units, such as Figure 4 As shown, it may include a first execution unit (FPGA-1) and a second execution unit (FPGA-2), wherein the first execution unit (compression encryption accelerator board / FPGA-1 device) may include a data encryption IP, a data compression IP, a DEV_DMA-IP, and a device memory, and the second execution unit may include a data decryption IP, a data decompression IP, a DEV_DMA-IP, and a device memory; or the device layer may include the first execution unit, or include the second execution unit, wherein the device memory may be used to cache data blocks that need to be processed.

[0054] Optionally, data encryption IP, data compression IP, data decryption IP, and data decompression IP are logic circuits that perform data acceleration calculations (offloading encryption and compression algorithms to hardware), effectively implementing encryption and compression algorithms in hardware. Alternatively, encryption and decryption refer to the process of protecting or hiding data using cryptographic techniques. Encryption converts raw data into data processed using a specific algorithm, making it difficult to understand or interpret in the event of unauthorized access. Decryption restores encrypted data to its original form. Encryption and decryption are typically used to ensure data confidentiality, integrity, and authentication, preventing unauthorized access and modification.

[0055] In some embodiments, in order to solve the data transmission delay problem in heterogeneous computing systems and achieve efficient data transmission, the present application adopts the point-to-point communication method in the high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) protocol. Specifically, all devices are mounted on the system PCIe bus. The solution disclosed in the present invention allows direct data transmission between different devices connected to the PCIe bus without passing through the host's CPU and host memory, thereby improving the efficiency of data transmission, reducing the CPU load, and significantly improving system performance in computationally intensive tasks. Specifically, in the solution disclosed in the present invention, point-to-point data transmission can be performed between the second storage device and the host through the PCIe bus (that is, a DMA transmission process of a point-to-point communication method can be performed), and point-to-point data transmission can be performed between the second storage device and the execution unit through the PCIe bus (that is, a DMA transmission process of a point-to-point communication method can be performed). For example, Figure 6 As shown, FPGA-1 and the disk can perform point-to-point data transmission via the bus (Peer-to-Peer is a point-to-point communication method in the PCIe protocol, P2P), and FPGA-2 and the disk can perform point-to-point data transmission via the bus. Figure 7 As shown, compared to non-point-to-point transmission solutions, the point-to-point transmission method disclosed in this disclosure eliminates the need to transfer data from disk to host-side memory and then transfer the data from host-side memory to accelerator board memory. Using a point-to-point data transmission method, data can be transferred directly from disk to accelerator board memory, thereby saving a large amount of HOST host-side system memory space. This mechanism also greatly improves data transmission efficiency and significantly enhances system performance in computationally intensive tasks (such as data compression and decompression, encryption and decryption).

[0056] In some embodiments, in the solution of the present disclosure, a transparent file system is added on the basis of the original virtual file system and the underlying file system of the kernel layer, and a pipeline software stack is designed in the transparent file system, such as Figure 9As shown, the pipeline software stack can process multiple data blocks in a pipeline manner. Specifically, the first data block is read before the second data block is read, and the first processing operation is an operation performed before the second processing operation according to the execution order of the multiple processing operations. In other words, the first data block can be read first, and after the first data block is read, the second data block can be read to realize the pipeline reading of data blocks. In the process of reading the second data block, the first processing can be performed on the first data block for the first time. When the second data block is read and the first processing is performed on the second data block, the second first processing can be performed on the first data block at the same time, that is, the first processing can be performed on both the first data block and the second data block at the same time, which can realize parallel data processing of the pipeline and improve processing efficiency.

[0057] Optionally, the transparent file system of this solution can include two pipelined software stacks, each serving as a read / write interface, which in turn calls various hardware computing IPs of two FPGA devices to implement transparent data compression and encryption, as well as transparent data decryption and decompression. Transparent operation means that it is imperceptible to the user. For example, when a user initiates a read or write operation, by adding a transparent file system to the kernel layer, the transparent file system intercepts the system call function for the read or write operation, triggering the block mechanism and corresponding pipeline scheduling, and then further triggering the FPGA to perform the corresponding data compression and encryption, decryption and decompression functions. Therefore, the user only needs to perform regular read and write operations, and data compression and decryption and decompression functions can be achieved without any additional operations.

[0058] Through the present application, in response to a first operation instruction, a file to be processed is divided to obtain multiple data blocks corresponding to the file to be processed; a first processing operation is performed on a first data block among the multiple data blocks, and a second processing operation is performed on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation according to the execution order of the multiple processing operations, and the multiple processing operations include at least two of reading processing, compression processing, encryption processing, and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing, and writing processing. Parallel data processing with coordinated software and hardware can be achieved, and the requirements of efficient data transmission, efficient and transparent compression and encryption, and efficient and transparent decryption and decompression can be met simultaneously.

[0059] Figure 2 The flowchart of another data processing method proposed in the present disclosure is further shown. Figure 1 In the embodiment shown, the data processing method of the present disclosure may further include the following steps.

[0060] Step 201 : determining a plurality of task blocks corresponding to a plurality of data blocks, wherein each task block includes information related to the data block corresponding to the task block.

[0061] In some embodiments, task blocks corresponding to multiple data blocks can be determined based on multiple data blocks, and each task block includes information related to the data block corresponding to the task block, such as task block information describing the size of the data block to be processed, file information where the data file is located (file size, file location, file offset), data block identifier (id), task block identifier (id), and other information.

[0062] Step 202: Determine the identifiers of multiple task blocks.

[0063] In some embodiments, the identification of multiple task blocks can be determined. Optionally, the identification of the task block can be determined based on the identification of the data block. The multiple data blocks can be sorted according to the position of the data block in the file, and the identification of the task block can be determined based on the sorting result. Alternatively, the identification of the task block can be determined based on the amount of data contained in the data block, or the identification of the task block can be determined based on the priority of the data contained in the data block, and so on.

[0064] Step 203: Determine the reading order of the multiple data blocks according to the identifiers of the multiple task blocks.

[0065] In some embodiments, the reading order of multiple blocks can be determined based on the identifiers of multiple task blocks. For example, multiple data blocks can be read in the order of ID values from small to large, or multiple data blocks can be read in the order of ID values from large to small, etc.

[0066] Optionally, the reading order of the multiple data blocks is the order in which the multiple data blocks are read and processed. The reading order of the multiple data blocks may be the order in which the multiple data blocks are input into the pipeline software stack, or the order in which the multiple data blocks are started to be processed.

[0067] In summary, the above embodiments of the present application can determine multiple task blocks corresponding to multiple data blocks, and can determine the order of processing multiple data blocks based on the task block identifiers, can realize pipeline processing of data blocks as needed, and can improve the flexibility and efficiency of data processing.

[0068] Figure 3 The flowchart of another data processing method proposed in the present disclosure is further shown. Figure 1 and Figure 2 In the illustrated embodiment, step 102 is further explained and may include the following steps.

[0069] Step 301: Input the first task block into the pipeline software stack in the transparent file system.

[0070] In some embodiments, the first task block may be input into the pipeline software stack in the transparent file system first, that is, the first data block corresponding to the first task block may be first processed.

[0071] In step 302 , a first-stage pipeline in the pipeline software stack performs a read process on a first data block in response to a first task block.

[0072] In some embodiments, the first-level pipeline in the pipeline software stack responds to the first task block and performs read processing on the first data block, including: determining that the first operation instruction is a data write operation instruction, and the execution order of multiple processing operations is read processing, compression processing, encryption processing, and write processing; the first-level pipeline uses the first scheduling function to schedule the execution unit, and obtains the first data block stored in the first storage device through the write operation of the execution unit; and writes the first data block into the first device memory of the execution unit.

[0073] In other words, when the first operation instruction is a data write operation instruction, read processing, compression processing, encryption processing and write processing can be performed on multiple data blocks. At this time, the first scheduling function can instruct the execution unit to obtain the first data block from the first storage device, and the first scheduling function can be used to schedule the execution unit to initiate a write operation on the host side. For example, the FPGA can initiate a DEV_DMA write operation on the host side to obtain the first data block stored in the first storage device, where DEV_DMA represents the DMA of the FPGA, not the DEV_DMA of the disk. The first storage device can be the storage space on the host side. After obtaining the first data block, the execution unit can temporarily store the first data block in the device memory of the execution unit to facilitate subsequent compression processing, encryption processing and write processing. Optionally, the first scheduling function can include a first task block, and the execution unit can obtain the first data block from the first storage device according to the first task block.

[0074] In some embodiments, the first-level pipeline in the pipeline software stack responds to the first task block and performs read processing on the first data block, including: determining that the first operation instruction is a data read operation instruction, and the execution order of multiple processing operations is read processing, decryption processing, decompression processing, and write processing; the first-level pipeline uses the fifth scheduling function to schedule the execution unit to obtain the first data block stored in the second storage device through the bus; and write the first data block into the first device memory of the execution unit.

[0075] In other words, when the first operation instruction is a data read operation instruction, read processing, decryption processing, decompression processing, and write processing can be performed on multiple data blocks. At this time, the first scheduling function can instruct the execution unit to obtain the first data block from the second storage device. The first scheduling function can be used to schedule the execution unit to obtain the first data block stored in the second storage device point-to-point via the PCIe bus. After obtaining the first data block, the execution unit can temporarily store the first data block in the device memory of the execution unit to facilitate subsequent decryption processing, decompression processing, and write processing. Optionally, the first scheduling function can include a first task block, and the execution unit can obtain the first data block from the second storage device according to the first task block.

[0076] Step 303: When the first-stage pipeline completes the read processing of the first data block, the second task block is input into the pipeline software stack.

[0077] In some embodiments, the first-level pipeline can perform a first processing on the first data block. When the first-level pipeline completes the reading processing of the first data block, the second task block can be input into the pipeline software stack, that is, the first processing of the second data block can be started, and multiple data blocks can be processed in a pipeline manner. That is, after the first scheduling function completes the reading processing of the first data block, the first scheduling function can be used to continue to read the second data block, and so on. After that, the third data block can be read and processed until the reading processing of multiple data blocks is completed.

[0078] In step 304 , the second stage pipeline in the pipeline software stack performs a second processing operation on the second data block in response to the second task block, while the first stage pipeline performs a first processing operation on the first data block.

[0079] In some embodiments, the second-level pipeline in the pipeline software stack responds to the second task block and performs a second processing operation on the second data block, that is, the second-level pipeline can perform the first processing on the second data block. Optionally, the first scheduling function can be used to perform read processing on the second data block, that is, the way of reading and processing multiple data blocks is the same. The scheme of the present disclosure can use the scheduling function at the kernel layer to schedule the hardware of the device layer to perform the first processing, which can realize the coordinated processing of the software layer and the hardware layer, reduce the consumption of software resources, and realize efficient data transmission and transparent operation.

[0080] In some embodiments, the second-level pipeline in the pipeline software stack responds to the second task block and performs a second processing operation on the second data block, while the first-level pipeline performs a first processing operation on the first data block, including: at a first time, the first-level pipeline uses the second scheduling function to schedule the execution unit to perform compression processing on the first data block, and at the same time, the second-level pipeline uses the first scheduling function to schedule the execution unit to obtain the second data block stored in the first storage device through the write operation of the execution unit, and stores the second data block to the first device memory; at a second time, the first-level pipeline uses the third scheduling function to schedule the execution unit to perform encryption processing on the first data block, and at the same time, the second-level pipeline uses the second scheduling function to schedule the execution unit to perform compression processing on the second data block; at a third time, the first-level pipeline uses the fourth scheduling function to schedule the execution unit to perform write processing on the first data block, and at the same time, the second-level pipeline uses the third scheduling function to schedule the execution unit to perform encryption processing on the second data block.

[0081] In some embodiments, the first-level pipeline uses the fourth scheduling function to schedule the execution unit to perform write processing on the first data block, including: the first-level pipeline uses the fourth scheduling function to schedule the execution unit to send the first data block after compression and encryption processing to the second storage device through the bus.

[0082] In other words, when the first operation instruction is a data write operation, after the first data block is read and the second task block is input into the pipeline software stack, the first scheduling function and the second scheduling function can be used at the same time to synchronously process the second data block and the first data block. At this time, the second scheduling function schedules the FPGA's compression IP to compress the first data block, and at the same time, the first scheduling function schedules the FPGA to use the FPGA's DEV-DMA write operation to obtain the second data block from the first storage device; thereafter, the second scheduling function and the third scheduling function can be used at the same time to synchronously process the second data block and the first data block. At this time, the second scheduling function schedules the FPGA's compression IP to compress the second data block, and at the same time, the third scheduling function schedules the FPGA's compression IP to compress the second data block. The scheduling function schedules the encryption IP of the FPGA to encrypt the first data block; thereafter, at a third time, the third scheduling function and the fourth scheduling function can be used simultaneously to synchronously process the second data block and the first data block. At this time, the third scheduling function schedules the encryption IP of the FPGA to encrypt the second data block, and at the same time, the fourth scheduling function schedules the FPGA to send the first data block after compression and encryption through the bus to the second storage device. The second storage device can be a disk for storing user-written data; thereafter, at a fourth time, the processing of the first data block has been completed, and at the fourth time, the fourth scheduling function can be used to schedule the FPGA to send the second data block after compression and encryption through the bus to the second storage device.

[0083] When there are multiple data blocks to be processed, the pipeline processing method of the first data block and the second data block can be referred to above, for example, Figure 9 As shown, first, task block 0 is input into the pipeline software stack at the first time, and the DEV-DMA-IP of FPGA-1 can be scheduled at the first time to obtain data block 0 corresponding to task block 0 from the first storage device, and store data block 0 in the device memory; then, task block 1 can be input into the pipeline software stack at the second time, and the data compression IP of FPGA-1 can be scheduled at the second time to take out data block 0 from the device memory and compress data block 0, and at the same time, DEV-DMA-IP obtains data block 1 corresponding to task block 1, and stores data block 1 in the device memory, wherein after data block 0 is compressed, the compressed data block can be stored in the device memory; at the third time, task block 2 can be input into the pipeline software stack, and the DEV-DMA-IP of FPGA-1 can be scheduled to obtain data block 2 corresponding to task block 2, and store data block 2 in the device memory, and at the same time, the data encryption IP of FPGA-1 takes out the compressed data block 0 from the device memory and encrypts data block 0, and at the same time At the same time, the data compression IP of FPGA-1 takes out data block 1 from the device memory and performs compression on data block 1. Similarly, the encrypted data block 0 is saved in the device memory, and the compressed data block 1 is saved in the device memory. At the fourth time, task block 3 can be input into the pipeline software stack, and the DEV-DMA-IP of FPGA-1 can be scheduled to obtain data block 3 corresponding to task block 3, and store data block 3 in the device memory. At the same time, FPGA-1 sends the compressed and encrypted data block 0 in the device memory to the disk through the PCIe bus for storage. At the same time, the data encryption IP of FPGA-1 takes out the compressed data block 1 from the device memory and performs encryption on data block 1. At the same time, the data compression IP of FPGA-1 takes out data block 2 from the device memory and performs compression on data block 2. Similarly, the encrypted data block 1 is saved in the device memory, and the compressed data block 2 is saved in the device memory. At this point, the processing of data block 0 is completed, and the above operations are repeated for the remaining data blocks until all data blocks are processed.

[0084] In some embodiments, the second-level pipeline in the pipeline software stack responds to the second task block, performs a second processing operation on the second data block, and at the same time, the first-level pipeline performs a first processing operation on the first data block, including: at a first time, the first-level pipeline uses the sixth scheduling function to schedule the execution unit to perform decryption processing on the first data block, and at the same time, the second-level pipeline uses the fifth scheduling function to schedule the execution unit to obtain the second data block stored in the second storage device through the bus, and stores the second data block to the first device memory; at a second time, the first-level pipeline uses the seventh scheduling function to schedule the execution unit to perform decompression processing on the first data block, and at the same time, the second-level pipeline uses the sixth scheduling function to schedule the execution unit to perform decompression processing on the second data block; at a third time, the first-level pipeline uses the eighth scheduling function to schedule the execution unit to perform write processing on the first data block, and at the same time, the second-level pipeline uses the seventh scheduling function to schedule the execution unit to perform decompression processing on the second data block.

[0085] In some embodiments, the first-level pipeline uses the eighth scheduling function to schedule the execution unit to perform write processing on the first data block, including: the first-level pipeline uses the eighth scheduling function to schedule the execution unit, and sends the first data block that has been decrypted and decompressed to the first cache space for caching through the read operation of the execution unit.

[0086] In other words, when the first operation instruction is a data read operation, after the first data block is read and the second task block is input into the pipeline software stack, the fifth scheduling function and the sixth scheduling function can be used at the same time to synchronously process the second data block and the first data block at the first time. At this time, the sixth scheduling function schedules the FPGA's data decryption IP to decrypt the first data block, and at the same time, the fifth scheduling function schedules the FPGA to obtain the second data block from the first storage device through the bus; thereafter, the sixth scheduling function and the seventh scheduling function can be used at the same time to synchronously process the second data block and the first data block at the second time. At this time, the sixth scheduling function schedules the FPGA's data decryption IP to decrypt the second data block, and at the same time, the seventh scheduling function schedules the FPGA's data decompression IP to decrypt the first data block. The block is decompressed; thereafter, at a third time, the seventh scheduling function and the eighth scheduling function can be used simultaneously to synchronously process the second data block and the first data block. At this time, the seventh scheduling function schedules the data decompression IP of the FPGA to decompress the second data block, and at the same time, the eighth scheduling function schedules the FPGA to cache the first data block after decryption and decompression to the first cache space through the DEV-DMA read operation of the FPGA. The first cache space can be the kernel layer user cache of the decrypted and compressed data block; thereafter, at a fourth time, the processing of the first data block has been completed, and at the fourth time, the eighth scheduling function can be used to schedule the FPGA to cache the second data block after decryption and decompression to the first cache space through the DEV-DMA read operation of the FPGA.

[0087] When there are multiple data blocks to be processed, the pipeline processing method of the first data block and the second data block can be referred to above, for example, Figure 11As shown, first, task block 0 is input into the pipeline software stack at the first time. At the first time, FPGA-2 can be scheduled to obtain data block 0 corresponding to task block 0 from the second storage device through the bus, and data block 0 is stored in the device memory; then, task block 1 can be input into the pipeline software stack at the second time. At the second time, the data decryption IP of FPGA-2 can be scheduled to take out data block 0 from the device memory and decrypt data block 0. At the same time, FPGA-2 obtains data block 1 corresponding to task block 1 from the second storage device through the bus, and stores data block 1 in the device memory. After decrypting data block 0, the decrypted data block can be stored in the device memory; at the third time, task block 2 can be input into the pipeline software stack. At the same time, FPGA-2 can be scheduled to obtain data block 2 through the bus, and store data block 2 in the device memory. At the same time, the data decompression IP of FPGA-2 takes out the decrypted data block 0 from the device memory, encrypts data block 0, and at the same time, FPGA -2's data decryption IP takes out data block 1 from the device memory and decrypts data block 1. Similarly, the decompressed data block 0 is stored in the device memory, and the decrypted data block 1 is also stored in the device memory. At the fourth time, task block 3 can be input into the pipeline software stack, and FPGA-2 can be scheduled to obtain data block 3 through the bus and store data block 3 in the device memory. At the same time, FPGA-2's DEV-DMA-IP sends the decrypted and decompressed data block 0 in the device memory to the first cache space for caching. At the same time, FPGA-2's data decompression IP takes out the decrypted data block 1 from the device memory and decompresses data block 1. At the same time, FPGA-2's data decryption IP takes out data block 2 from the device memory and decrypts data block 2. Similarly, the decompressed data block 1 is stored in the device memory, and the decrypted data block 2 is stored in the device memory. At this point, data block 0 is processed, and the above operation is repeated for the remaining data blocks until all data blocks are processed.

[0088] In some embodiments, the method further includes: acquiring the first data block that has been decrypted and decompressed from the first cache space; and displaying the first data block that has been decrypted and decompressed.

[0089] Optionally, the decrypted and decompressed data blocks can be sent to the first cache space for caching. When the same data blocks are subsequently read repeatedly, they can be directly retrieved from the cache space without having to decrypt and decompress them again. That is, the purpose of caching is to reduce repeated decryption and decompression operations on the same data file. After performing a decryption and decompression operation on a data block, the decompressed data result is stored in the cache. The next time the same operation is performed, the data can be directly retrieved from the cache without executing the pipeline decryption and decompression process. The user layer of the host can obtain the cached data blocks from the first cache space, which can improve acquisition efficiency and complete the user's data read operation.

[0090] In some embodiments, at least one of the first, second, third, fourth, fifth, sixth, seventh, and eighth scheduling functions includes the first task block or the second task block. In other words, when a scheduling function is used to schedule the FPGA to process a data block, the scheduling function may include information about the data block to be processed by the execution unit, so that the execution unit can determine the data block to be processed based on the information related to the data block.

[0091] In summary, this application embeds a transparent file system in the kernel layer, and designs a task partitioning mechanism and two four-level pipeline software stacks for reading and writing, and successively calls point-to-point data transmission, DEV_DMA data reading and writing transmission, data transparent compression and encryption, and data transparent decompression. This achieves efficient collaborative and parallel calls to software and hardware, greatly improving the computing throughput of the heterogeneous acceleration system. And by using memory address mapping technology, the device memory is inserted, managed, and allocated in the kernel layer of the Linux operating system. In this way, files on the disk can be transferred point-to-point to the accelerator device through the kernel layer interface function. This data transmission method significantly reduces the delay of data transmission and improves the overall performance of the system.

[0092] based on Figure 1 、 Figure 2 and Figure 3 The following describes a specific example further illustrating the above method. This example provides a method for implementing a heterogeneous accelerated computing software scheduling system for efficient data transmission, data storage, and data encryption and decryption. The following aspects are presented: 1. The PCIe point-to-point transmission principle. 2. Schematic diagram of the data processing process of this design. 3. The design of a four-stage pipeline software stack for point-to-point data file transmission, data compression and encryption, and data decryption and decompression. 4. A block diagram of the actual solution's components.

[0093] 1. Point-to-point data transmission principle

[0094] In traditional heterogeneous computing systems, multiple DMA transfers are required to transfer data from disk storage devices to accelerator board devices. First, the disk data is transferred to the host memory through the disk's DMA, and then the data is transferred directly from the host side to the accelerator board through the device's DMA. This method has a large transmission delay.

[0095] In order to solve the data transmission delay problem in heterogeneous computing systems and achieve efficient data transmission, the present invention adopts a peer-to-peer data transmission method. Peer-to-Peer is a peer-to-peer communication method in the PCIe protocol.

[0096] The PCIe protocol supports multiple data transfer modes, including point-to-point and direct memory access (DMA). When a device has the ability to act as a bus master, it can independently initiate data transfers, sending data directly to another PCIe device without relying heavily on the central processing unit (CPU) and other system resources. This direct data transfer between devices is called a point-to-point transaction. Because point-to-point transactions occur only between devices within the system, they significantly reduce data transfer latency and improve overall system performance.

[0097] Figure 4 This article provides a topology diagram of the PCIe bus protocol. Point-to-point DMA (point-to-point for short) technology allows direct data transfer between different devices connected to the PCIe bus, eliminating the need to transfer data from disk to host-side memory and then from host-side memory to accelerator board memory. Using point-to-point data transfer, data can be transferred directly from disk to accelerator board memory, saving a significant amount of host-side system memory space. This mechanism also greatly improves data transfer efficiency and significantly enhances system performance in compute-intensive tasks (such as data compression and decompression, encryption and decryption).

[0098] like Figure 5 As shown in FIG, a schematic diagram of a traditional DMA data transmission scheme is shown. Figure 6 As shown in the figure, this is a schematic diagram of a point-to-point DMA data transmission solution provided by this solution. In heterogeneous computing scenarios, such as data compression and decompression, encryption and decryption, heterogeneous computing of data is itself very time-consuming. The impact is small for small data volumes, but for larger data, if the data transmission path is particularly long and multiple copies are performed, the data delay will be very large, which will ultimately affect the performance of the entire system. Figure 6 This is a schematic diagram of point-to-point data transmission in a data heterogeneous computing system. Figure 7 Schematic diagram of point-to-point data streaming between disk and FPGA.

[0099] 2. Schematic diagram of the data processing process of this design scheme

[0100] Figure 8 This is a schematic diagram of the single-step execution steps when the data encryption and decryption algorithm and the query algorithm are offloaded to FPGA hardware. It is divided into two parts, left and right, and each part has four steps to complete. Figure 8 The device FPGA-1 on the left performs data compression and encryption tasks. The device FPGA-1 on the left transmits the executed data compression and encryption results to the disk through point-to-point data transmission (disk DMA write operation). The device FPGA-2 on the right performs data decryption and decompression tasks. First, the data is transferred from the disk to the FPGA-2 side through point-to-point data transmission (disk DMA read operation), and then the data decryption and decompression steps are performed.

[0101] (1) Data compression and encryption (FPGA-1 on the left executes steps ①②③④)

[0102] First, a write operation is performed at the user level. FPGA-1 initiates a DEV_DMA write operation to the host (DEV_DMA represents the FPGA's DMA, which is different from the disk's DMA). This writes the data to FPGA-1's memory. FPGA-1 then performs compression and encryption operations on the data. Finally, FPGA-1 transmits the compressed and encrypted data to the disk via point-to-point data transmission (a disk DMA write operation). Therefore, the user's write operation triggers the data compression and encryption operations.

[0103] (2) Data decryption and decompression (FPGA-2 on the right executes steps ①②③④)

[0104] First, a read operation is performed at the user level. Then, through point-to-point data transfer, the data is transferred from the disk to the memory on the FPGA-2 side (disk DMA read operation). Then, decryption and decompression operations are performed on the FPGA-2 side. Finally, a DEV_DMA read operation (DEV_DMA represents the FPGA's DMA function, which is different from the disk's DMA function) is initiated on the host side through the FPGA-2. The result is returned to the user level, completing the read operation. Therefore, the user's read operation triggers the decryption and decompression operations of the data.

[0105] 3. Design of a four-stage pipeline software stack for data compression and encryption and data decryption and decompression

[0106] (1) Data compression and encryption pipeline software stack

[0107] like Figure 9 As shown in the figure, the design scheme of the compression and encryption pipeline software stack provided by this solution is as follows: Figure 9This is explained in three layers: the user layer, the kernel layer, and the device layer. The user layer and kernel layer represent two distinct layers within the host operating system. The device layer includes disks and FPGA accelerator boards, all of which are mounted on the system's PCIe bus. The compression and encryption accelerator board in the device layer is an FPGA-1 device, which contains data encryption IP, data compression IP, and device memory. The data encryption and compression IPs are logic circuits for accelerating data calculations (offloading the encryption and compression algorithms to hardware), effectively implementing the encryption and compression algorithms in hardware.

[0108] The pipeline software stack is designed to improve the parallel scheduling capability of the system software to adapt to the data processing throughput of the FPGA hardware parallel computing. If a serial system call method (such as Figure 10 ), which consumes considerable time and causes significant latency, reducing overall system performance. For example, this solution employs a four-stage pipeline to perform DEV_DMA write data transfer, data compression, data encryption, and point-to-point data transfer. Each pipeline level executes these four steps, and each step represents a related scheduling function or algorithm. For example, this solution has designed corresponding functions to schedule DEV_DMA write data transfer, data compression, data encryption, and point-to-point data transfer.

[0109] If serial scheduling is used, it would take 16 time slots to complete four tasks, but if a four-stage pipeline is used, it would only take 7 time slots, greatly reducing latency.

[0110] When a user initiates a write operation, the write operation enters the kernel layer through a system call. After the write operating system call, the kernel layer divides the data file to be written into task blocks and then evenly divides the data file into several task blocks (task block 0, task block 1, ..., task block n-1). The task block information describes the size of the data block to be processed, the file information (file size, file location, file offset), the data block ID, and other information. After the first stage of the four-stage pipeline receives task block 0, it begins to perform DEV_DMA write data transfer, data compression, data encryption, and point-to-point data transfer in sequence. Due to the pipeline's characteristics, the DEV_DMA write data transfer in the second stage coincides with the data compression in the first stage. The second stage then begins DEV_DMA write data transfer, and so on. The task block IDs for each stage of the four-stage pipeline are 4i, 4i+1, 4i+2, and 4i+3, respectively. i (starting at 0 and increasing by 1) represents the number of task dispatch cycles. Each time a task is dispatched to the fourth stage, the value of i increases by 1. After each stage of the pipeline completes, the results are transferred from the device memory on FPGA-1 to the disk.

[0111] (2) Data decryption and decompression pipeline software stack

[0112] like Figure 11 Figure 2 shows the design of a four-stage pipeline software stack for data decryption and decompression. This design is also divided into three layers: the user layer, the kernel layer, and the device layer. The user layer and the kernel layer represent two distinct layers within the host operating system. The device layer includes disks and FPGA accelerator boards, all of which are mounted on the system's PCIe bus. The decryption and decompression accelerator board in the device layer is an FPGA-2 device, which contains data decryption IP, data decompression IP, and device memory. The data decryption and decompression IPs are logic circuits for accelerating data computation (offloading the decryption and decompression algorithms to hardware), effectively implementing the decryption and decompression algorithms in hardware.

[0113] When a user initiates a read operation, it enters the kernel layer through a system call. After the kernel reads the operating system call, it divides the data file to be read into task blocks and then evenly distributes them into several task blocks (task block 0, task block 1, ..., task block n-1). The task block information describes the size of the data block to be processed, the file information (file size, file location, file offset), the data block ID, and the task block ID. After the first stage of the four-stage pipeline receives task block 0, it begins executing point-to-point data transfer, data decryption, data decompression, and DEV_DMA write data transfer. Due to the pipeline's characteristics, the pipeline read process is similar to the write performed by FPGA-1 above. Decryption and decompression are the inverse operations of compression and encryption. The task block IDs for each stage in the four-stage pipeline are 4i, 4i+1, 4i+2, and 4i+3, respectively. i (starting at 0 and increasing by 1) represents the number of task dispatch cycles. The value of i increases by 1 each time a task is dispatched to the fourth stage. After each stage of the pipeline is completed, the result will be moved from the device memory on the FPGA-2 side to the host-side cache (the purpose of the cache is to reduce the repeated decryption and decompression operations on the same data file. After performing a decryption and decompression operation on a data block, the decompressed data result is stored in the cache. The next time the same operation is performed, the data can be directly obtained from the cache without executing the pipeline decryption and decompression process). Finally, the host will read the final result from the cache, and the read operation ends.

[0114] 4. Structural diagram of the actual solution

[0115] like Figure 12 Figure 1 shows the structure of a heterogeneous computing data transmission solution using compression, encryption, decryption, and decompression. In this solution, both the disk and the FPGA accelerator board are connected to the server host via the PCIe bus. The FPGA accelerator board contains several key modules, such as DEV_DMA-IP, encryption IP, decryption IP, compression IP, decompression IP, memory controller, and device memory. Among them:

[0116] DEV_DMA-IP: responsible for DMA data transfer on the PCIe bus.

[0117] Encryption IP: responsible for data encryption operations.

[0118] Decryption IP: Responsible for data decryption operations.

[0119] Compression IP: responsible for implementing the compression operation of data files.

[0120] Decompression IP: responsible for decompressing data files

[0121] The server host runs the Linux operating system, which is divided into the kernel and application layers. At the application layer, users can initiate read and write operations on data files, including data compression and encryption, as well as decryption and decompression. At the kernel layer, memory management mechanisms manage and allocate memory resources. Operations such as point-to-point data transmission, data encryption and decryption, and data compression and decompression are then initiated, invoking the software stack workflow. The kernel layer's FPGA driver is responsible for managing various devices.

[0122] In this solution, the kernel layer focuses on implementing a data partitioning mechanism, a four-stage pipeline software stack, and an FPGA driver. These components work together to ensure efficient data transfer between the disk and the FPGA, and complete the required compression, encryption, decryption, and decompression operations.

[0123] This solution specifically emphasizes the addition of a transparent file system to the kernel's existing virtual file system and underlying file system. Two four-stage pipelines are designed within the transparent file system, serving as read and write interfaces. These pipelines then call various hardware computing IPs from two FPGA devices to implement transparent data compression and encryption, as well as transparent data decryption and decompression. Transparent operations are user-imperceptible. For example, when a user initiates a read or write operation, the addition of a transparent file system to the kernel layer intercepts the system call function for the read or write operation, triggering the block partitioning mechanism and corresponding pipeline scheduling. This in turn triggers the FPGA to perform the corresponding data compression and encryption, decryption and decompression, and other functions. Therefore, users only need to perform regular read and write operations, achieving data compression and decryption, and decryption and decompression without any additional steps. This is transparent operation.

[0124] like Figure 13 The following figure shows the specific implementation process of heterogeneous computing - transparent compression and encryption of data, including:

[0125] 1) The user initiates a data write operation.

[0126] 2) The operating system kernel receives data and writes a system call.

[0127] 3) The kernel-layer block mechanism divides the data file to be written into n equal task blocks. Each task block contains information such as the size of the data block to be processed, file information where the data is located (file size, file location, file offset), data block ID, and task block ID.

[0128] 4) Use the for loop mechanism to allocate task block information to the four-stage pipeline software stack.

[0129] 5) The four-stage pipeline software stack begins execution, which can improve the collaborative data processing capabilities of software and hardware.

[0130] 5-1) Start the DEV_DMA write operation to transfer data to FPGA-1.

[0131] 5-2) FPGA-1 performs data compression operations;

[0132] 5-3) FPGA-1 performs data encryption operations;

[0133] 5-4) Start the disk DMA write operation and perform point-to-point data transfer to transfer the data in the FPGA-1 memory to the disk.

[0134] 6) The transparent compression and encryption operation of the data is completed, and the entire write operation process ends here.

[0135] like Figure 14 The figure below shows the specific implementation process of heterogeneous computing - transparent decryption and decompression of data, including:

[0136] 1) The user initiates a data read operation.

[0137] 2) The operating system kernel receives data from the read system call.

[0138] 3) The kernel-layer block mechanism divides the data file to be read into n equal task blocks. Each task block contains information such as the size of the data block to be processed, file information (file size, file location, file offset), data block ID, and task block ID.

[0139] 4) Use the for loop mechanism to allocate task block information to the four-stage pipeline software stack.

[0140] 5) The four-stage pipeline software stack begins execution, which can improve the collaborative data processing capabilities of software and hardware.

[0141] 5-1) Start the disk DMA read operation, perform point-to-point data transfer, and transfer the data from the disk to the FPGA-2 memory;

[0142] 5-2) FPGA-2 performs data compression operations;

[0143] 5-3) FPGA-2 performs data encryption operations;

[0144] 5-4) Start the DEV_DMA read operation to transfer the data to the host buffer.

[0145] 6) The user layer reads the decompressed result from the cache, and the transparent decompression and decompression of the data is completed, and the entire read operation process ends here.

[0146] In summary, this example proposes a solution for simultaneously achieving efficient data transmission, transparent data compression and encryption, and transparent data decryption and decompression in a heterogeneous computing environment. This solution utilizes memory address mapping technology to insert, manage, and allocate device memory at the Linux kernel layer. This allows point-to-point transfer of files from disk to the accelerator device via kernel-layer interface functions. This data transmission method significantly reduces data transmission latency and improves overall system performance. By embedding a transparent file system at the kernel layer and designing a task partitioning mechanism and a four-stage pipeline software stack for reads and writes within it, point-to-point data transmission, DEV_DMA data read and write transfers, transparent data compression and encryption, and transparent data decompression are sequentially called. This enables efficient, coordinated, and parallel calls of software and hardware, significantly improving the computing throughput of the heterogeneous acceleration system.

[0147] The scheme of this application:

[0148] 1. Through memory address mapping technology, the device memory is mapped to the host memory address space, and data is efficiently transmitted point-to-point, reducing the number of data copies and data transmission delays.

[0149] 2. Designed DEV_DMA write data transmission, data encryption, data compression, and point-to-point data transmission, implemented a four-level pipeline software stack for write operations, and thus achieved efficient software and hardware system scheduling.

[0150] 3. Implement the task partitioning mechanism in heterogeneous accelerated computing solutions.

[0151] 4. Designed point-to-point data transmission, data decryption, data decompression, and DEV_DMA read data transmission, implemented a four-level pipeline software stack for read operations, and thus achieved efficient software and hardware system scheduling.

[0152] 5. The data compression and encryption operations and the data decryption and decompression operations in this solution are transparent operations. Users only need to perform data quick writing operations, and the system will automatically implement the data compression and encryption operations and decryption and decompression operations without the user having to perform additional operations.

[0153] 6.Point-to-point data transmission, data compression and encryption operations, and data decryption and decompression operation function API interfaces are all implemented in kernel state, reducing system calls and process switching between kernel state and user state.

[0154] 7. Achieve compatibility with storage disk types. The disks used in this application do not distinguish between disk types. The disks can be NVMe-SSD disks, mechanical hard disks, USB flash drives, etc.

[0155] In summary, the proposed method for implementing an efficient data transmission, data storage, data encryption and decryption heterogeneous accelerated computing software scheduling system not only overcomes the shortcomings of existing technologies, but also significantly improves the performance and security of data processing, effectively saves storage space, and greatly reduces the storage resource occupancy rate of data centers, and has broad application prospects.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0157] The embodiment of the present application further provides a data processing device 1500, Figure 15 A structural diagram of a data processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 15 Shown, including:

[0158] The first processing unit 1510 is configured to divide the file to be processed into multiple data blocks corresponding to the file to be processed in response to the first operation instruction;

[0159] The second processing unit 1520 is used to perform a first processing operation on a first data block among multiple data blocks, and simultaneously perform a second processing operation on a second data block among the multiple data blocks, wherein the first data block is read before the second data block is read, and the first processing operation is an operation performed before the second processing operation according to the execution order of the multiple processing operations, and the multiple processing operations include at least two of reading processing, compression processing, encryption processing and writing processing, or the multiple processing operations include at least two of reading processing, decryption processing, decompression processing and writing processing.

[0160] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data processing device also includes a third processing unit, which is used to: determine multiple task blocks corresponding to the multiple data blocks, each task block including information related to the data block corresponding to the task block; determine the identifiers of the multiple task blocks; and determine the reading order of the multiple data blocks based on the identifiers of the multiple task blocks.

[0161] Furthermore, in a possible implementation of the embodiment of the present disclosure, the third processing unit is used to: when the multiple processing operations include at least two of reading processing, compression processing, encryption processing and writing processing, determine that the first operation instruction is a data write operation instruction; when the multiple processing operations include at least two of reading processing, decryption processing, decompression processing and writing processing, determine that the first operation instruction is a data read operation instruction.

[0162] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: input the first task block into the pipeline software stack in the transparent file system; the first-level pipeline in the pipeline software stack performs read processing on the first data block in response to the first task block; when the first-level pipeline completes the read processing on the first data block, the second task block is input into the pipeline software stack; the second-level pipeline in the pipeline software stack performs the second processing operation on the second data block in response to the second task block, and at the same time, the first-level pipeline performs the first processing operation on the first data block.

[0163] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: determine that the first operation instruction is a data write operation instruction, and the execution order of multiple processing operations is read processing, compression processing, encryption processing, and write processing; the first-level pipeline uses the first scheduling function to schedule the execution unit, and obtains the first data block stored in the first storage device through the write operation of the execution unit; and writes the first data block into the first device memory of the execution unit.

[0164] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: at a first time, the first-level pipeline uses the second scheduling function to schedule the execution unit to perform compression processing on the first data block, and at the same time, the second-level pipeline uses the first scheduling function to schedule the execution unit to obtain the second data block stored in the first storage device through the write operation of the execution unit, and store the second data block to the first device memory; at a second time, the first-level pipeline uses the third scheduling function to schedule the execution unit to perform encryption processing on the first data block, and at the same time, the second-level pipeline uses the second scheduling function to schedule the execution unit to perform compression processing on the second data block; at a third time, the first-level pipeline uses the fourth scheduling function to schedule the execution unit to perform write processing on the first data block, and at the same time, the second-level pipeline uses the third scheduling function to schedule the execution unit to perform encryption processing on the second data block.

[0165] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is configured to: the first-stage pipeline uses the fourth scheduling function to schedule the execution unit to send the first data block after compression and encryption processing to the second storage device through the bus.

[0166] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: determine that the first operation instruction is a data read operation instruction, and the execution order of multiple processing operations is read processing, decryption processing, decompression processing, and write processing; the first-level pipeline uses the fifth scheduling function to schedule the execution unit to obtain the first data block stored in the second storage device through the bus; and write the first data block into the first device memory of the execution unit.

[0167] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: at a first time, the first-level pipeline uses the sixth scheduling function to schedule the execution unit to perform decryption processing on the first data block, and at the same time, the second-level pipeline uses the fifth scheduling function to schedule the execution unit to obtain the second data block stored in the second storage device through the bus, and store the second data block to the first device memory; at a second time, the first-level pipeline uses the seventh scheduling function to schedule the execution unit to perform decompression processing on the first data block, and at the same time, the second-level pipeline uses the sixth scheduling function to schedule the execution unit to perform decompression processing on the second data block; at a third time, the first-level pipeline uses the eighth scheduling function to schedule the execution unit to perform write processing on the first data block, and at the same time, the second-level pipeline uses the seventh scheduling function to schedule the execution unit to perform decompression processing on the second data block.

[0168] Furthermore, in a possible implementation of the embodiment of the present disclosure, the second processing unit 1520 is used to: the first-level pipeline uses the eighth scheduling function to schedule the execution unit, and sends the first data block that has been decrypted and decompressed to the first cache space for caching through the read operation of the execution unit.

[0169] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data processing device also includes a fourth processing unit, which is used to: obtain the first data block that has been decrypted and decompressed from the first cache space; and display the first data block that has been decrypted and decompressed.

[0170] Furthermore, in a possible implementation of the embodiment of the present disclosure, at least one of the first scheduling function, the second scheduling function, the third scheduling function, the fourth scheduling function, the fifth scheduling function, the sixth scheduling function, the seventh scheduling function, and the eighth scheduling function includes a first task block or a second task block.

[0171] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, and no further details will be given here.

[0172] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above data processing method embodiments.

[0173] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when run.

[0174] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0175] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above data processing method embodiments are implemented.

[0176] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data processing method embodiments are implemented.

[0177] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The above is a detailed introduction to a data processing method, electronic device, storage medium and product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: In response to the first operation instruction, the file to be processed is divided to obtain a plurality of data blocks corresponding to the file to be processed; performing a first processing operation on a first data block among the plurality of data blocks, and simultaneously performing a second processing operation on a second data block among the plurality of data blocks, wherein the first data block is read before the second data block is read, the first processing operation is an operation performed before the second processing operation according to an execution order of the plurality of processing operations, and the plurality of processing operations include at least two of a reading process, a compression process, an encryption process, and a writing process, or the plurality of processing operations include at least two of a reading process, a decryption process, a decompression process, and a writing process; The performing of a first processing operation on a first data block among the plurality of data blocks and performing a second processing operation on a second data block among the plurality of data blocks comprises: Inputting a first task block corresponding to the first data block into a pipeline software stack in a transparent file system; The first stage pipeline in the pipeline software stack performs the read process on the first data block in response to the first task block; When the first-stage pipeline completes the reading process on the first data block, inputting the second task block corresponding to the second data block into the pipeline software stack; The second stage pipeline in the pipeline software stack performs the second processing operation on the second data block in response to the second task block, while the first stage pipeline performs the first processing operation on the first data block.

2. The method according to claim 1, characterized in that The method further comprises: Determining a plurality of task blocks corresponding to the plurality of data blocks, each task block including information related to the data block corresponding to the task block; determining identifiers of the plurality of task blocks; A reading order of the multiple data blocks is determined according to the identifiers of the multiple task blocks.

3. The method according to claim 1, characterized in that The method further comprises: In a case where the plurality of processing operations include at least two of the reading process, the compression process, the encryption process, and the writing process, determining that the first operation instruction is a data writing operation instruction; In a case where the plurality of processing operations include at least two of the read process, the decryption process, the decompression process, and the write process, the first operation instruction is determined to be a data read operation instruction.

4. The method according to claim 1, wherein The first stage pipeline in the pipeline software stack performs the read processing on the first data block in response to the first task block, including: Determining that the first operation instruction is a data write operation instruction, and the execution order of the multiple processing operations is the reading process, the compression process, the encryption process, and the writing process; The first stage pipeline schedules the execution unit using the first scheduling function, and obtains the first data block stored in the first storage device through the write operation of the execution unit; The first data block is written into a first device memory of the execution unit.

5. The method according to claim 4, characterized in that The second stage pipeline in the pipeline software stack performs the second processing operation on the second data block in response to the second task block, while the first stage pipeline performs the first processing operation on the first data block, including: At a first time, the first-stage pipeline uses a second scheduling function to schedule the execution unit to perform the compression process on the first data block, and at the same time, the second-stage pipeline uses the first scheduling function to schedule the execution unit to obtain the second data block stored in the first storage device through a write operation of the execution unit, and store the second data block in the first device memory; At a second time, the first-stage pipeline uses a third scheduling function to schedule the execution unit to perform the encryption process on the first data block, and simultaneously the second-stage pipeline uses the second scheduling function to schedule the execution unit to perform the compression process on the second data block; At a third time, the first-stage pipeline uses the fourth scheduling function to schedule the execution unit to perform the write process on the first data block, while the second-stage pipeline uses the third scheduling function to schedule the execution unit to perform the encryption process on the second data block.

6. The method according to claim 5, characterized in that The first-stage pipeline uses a fourth scheduling function to schedule the execution unit to perform the write process on the first data block, including: The first-stage pipeline uses a fourth scheduling function to schedule the execution unit to send the first data block after the compression processing and the encryption processing to the second storage device through a bus.

7. The method according to claim 1, characterized in that The first stage pipeline in the pipeline software stack performs the read processing on the first data block in response to the first task block, including: Determining that the first operation instruction is a data read operation instruction, and the execution order of the multiple processing operations is the reading process, the decryption process, the decompression process, and the writing process; The first-stage pipeline uses a fifth scheduling function, and the scheduling execution unit obtains the first data block stored in the second storage device through a bus; The first data block is written into a first device memory of the execution unit.

8. The method according to claim 7, characterized in that The second stage pipeline in the pipeline software stack performs the second processing operation on the second data block in response to the second task block, while the first stage pipeline performs the first processing operation on the first data block, including: At a first time, the first-stage pipeline uses the sixth scheduling function to schedule the execution unit to perform the decryption process on the first data block, and at the same time, the second-stage pipeline uses the fifth scheduling function to schedule the execution unit to obtain the second data block stored in the second storage device through the bus and store the second data block in the memory of the first device; At a second time, the first-stage pipeline uses the seventh scheduling function to schedule the execution unit to perform the decompression process on the first data block, and the second-stage pipeline uses the sixth scheduling function to schedule the execution unit to perform the decryption process on the second data block; At a third time, the first-stage pipeline uses the eighth scheduling function to schedule the execution unit to perform the write processing on the first data block, while the second-stage pipeline uses the seventh scheduling function to schedule the execution unit to perform the decompression processing on the second data block.

9. The method according to claim 8, characterized in that The first-stage pipeline uses the eighth scheduling function to schedule the execution unit to perform the write process on the first data block, including: The first-stage pipeline schedules the execution unit using the eighth scheduling function, and sends the first data block that has undergone the decryption and decompression processing to the first cache space for caching through the read operation of the execution unit.

10. The method according to claim 9, characterized in that The method further comprises: Acquire the first data block that has undergone the decryption and decompression processing from the first cache space; The first data block that has undergone the decryption and decompression processes is displayed.

11. The method according to any one of claims 4 to 10, characterized in that At least one of the first scheduling function, the second scheduling function, the third scheduling function, the fourth scheduling function, the fifth scheduling function, the sixth scheduling function, the seventh scheduling function, and the eighth scheduling function includes the first task block or the second task block.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 11.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 11 are implemented.

Images