Data transmission method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of data transmission, and discloses a data transmission method and device, computer equipment and a storage medium, and the method comprises the steps: determining to-be-processed data in first storage equipment according to a data transmission request; a first control instruction is sent to a first storage device, so that the first storage device transmits to-be-processed data to a data processing component through a bus, the data processing component is used for carrying out preset processing on the to-be-processed data to obtain target data, and the preset processing comprises compression processing and / or encryption processing or compression processing and / or encryption processing. The preset processing comprises decryption processing and / or decompression processing; and under the condition that a preset signal sent by the data processing component is received, a second control instruction is sent to the second storage device, so that the second storage device obtains the target data from the data processing component through the bus. The problem that in the data transmission process, it is difficult to quickly compress, decompress, encrypt and decrypt data under the condition that low transmission delay is ensured is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data transmission method, device, computer equipment and storage medium. Background Art

[0002] Currently, during the data transmission process, the schemes for compressing, decompressing, encrypting, and decrypting data include: processing data only using software and processing data only using hardware. However, processing data only using software depends on the processing power of the CPU (Central Processing Unit). For large-scale data processing, a large amount of CPU and memory resources are required, resulting in slow data processing speed. During the data transmission process, processing data only using hardware requires four data transmissions, which will generate a huge transmission delay and cannot efficiently transmit data.

[0003] Therefore, the related technology has the problem that it is difficult to quickly compress, decompress, encrypt and decrypt data while ensuring low transmission delay during data transmission. Summary of the invention

[0004] In view of this, the present invention provides a data transmission method, apparatus, computer equipment and storage medium to solve the problem that it is difficult to quickly compress, decompress, encrypt and decrypt data while ensuring low transmission delay during data transmission.

[0005] In a first aspect, the present invention provides a data transmission method, which is applied to a host, and the method comprises: In case of receiving a data transmission request, determining the data to be processed in the first storage device according to the data transmission request; Sending a first control instruction to the first storage device so that the first storage device transmits the data to be processed to the data processing component through the bus, wherein the data processing component is used to perform a preset process on the data to be processed to obtain target data, wherein the preset process includes compression process and / or encryption process, or the preset process includes decryption process and / or decompression process; When receiving a preset signal sent by the data processing component, a second control instruction is sent to the second storage device so that the second storage device obtains target data from the data processing component through the bus, wherein the preset signal is used to indicate completion of preset processing on the data to be processed.

[0006] In the data transmission method provided in this embodiment, the host instructs the first storage device to directly transmit the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. The data processing component is used to perform preset operations on the data to be processed, which improves the efficiency of data processing and ensures the security and compliance of data transmission and storage. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0007] In a second aspect, the present invention provides a data transmission method, which is applied to a data processing component, and the method includes: In the case of receiving the data to be processed, performing preset processing on the data to be processed to obtain target data, wherein the data to be processed is transmitted to the data processing component through the bus after the first storage device receives the first control instruction, the first control instruction is sent to the first storage device after the host determines the data to be processed in the first storage device, the data to be processed is determined according to the data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; After the preset processing is performed on the data to be processed, a preset signal is sent to the host so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate that the preset processing of the data to be processed is completed, and the second control instruction is used to instruct the second storage device to obtain target data from the data processing component through the bus.

[0008] The data transmission method provided in this embodiment uses the data processing component to perform preset operations on the data to be processed, thereby improving the efficiency of data processing and ensuring the security and compliance during data transmission and storage. The first storage device directly transmits the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0009] In a third aspect, the present invention provides a data transmission system, the system comprising: a host, a data processing component, a first storage device and a second storage device; The host is connected to the first storage device via a first bus, the host is connected to the data processing component via a second bus, and the host is connected to the second storage device via a third bus; The host is used to execute the data transmission method of the first aspect or any corresponding embodiment thereof; The data processing component is used to execute the data transmission method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0010] In the data transmission system provided in this embodiment, the host instructs the first storage device to directly transmit the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. The data processing component is used to perform preset operations on the data to be processed, which improves the efficiency of data processing and ensures the security and compliance of data transmission and storage. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0011] In a fourth aspect, the present invention provides a data transmission device, which is deployed on a host and includes: A data determination module, configured to determine the data to be processed in the first storage device according to the data transmission request when a data transmission request is received; A first instruction sending module is used to send a first control instruction to the first storage device, so that the first storage device transmits the to-be-processed data to the data processing component through the bus, wherein the data processing component is used to perform a preset processing on the to-be-processed data to obtain target data, wherein the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; The second instruction sending module is used to send a second control instruction to the second storage device when receiving a preset signal sent by the data processing component, so that the second storage device obtains target data from the data processing component through a bus, wherein the preset signal is used to indicate the completion of preset processing on the data to be processed.

[0012] In a fifth aspect, the present invention provides a data transmission device, which is deployed in a data processing component, and includes: A data processing module, configured to perform preset processing on the data to be processed to obtain target data when receiving the data to be processed, wherein the data to be processed is transmitted to the data processing component through a bus after the first storage device receives the first control instruction, the first control instruction is sent to the first storage device by the host after the host determines the data to be processed in the first storage device, the data to be processed is determined according to the data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; A signal sending module is used to send a preset signal to the host after performing preset processing on the data to be processed, so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate the completion of the preset processing of the data to be processed, and the second control instruction is used to instruct the second storage device to obtain target data from the data processing component through the bus.

[0013] In a sixth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions being stored in the memory, and the processor executing the computer instructions to thereby execute the data transmission method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the data transmission method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0014] In the seventh aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data transmission method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the data transmission method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0015] In an eighth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the data transmission method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the data transmission method of the above-mentioned second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a schematic diagram of end-to-end data transmission according to an embodiment of the present invention; Figure 2 is a flow chart of a data transmission method applied to a host according to an embodiment of the present invention; Figure 3 is a schematic diagram of end-to-end data transmission between disks according to an embodiment of the present invention; Figure 4 is a schematic diagram of memory address mapping according to an embodiment of the present invention; Figure 5 is a structural diagram of a heterogeneous computing data transmission solution according to an embodiment of the present invention; Figure 6is a flow chart of a data transmission method applied to a data processing component according to an embodiment of the present invention; Figure 7 is a schematic diagram of the structure of a data transmission system according to an embodiment of the present invention; Figure 8 is a flow chart of a method for compressing and encrypting data transmission according to an embodiment of the present invention; Fig. 9 is a structural block diagram of a data transmission device deployed on a host according to an embodiment of the present invention; Fig.10 is a structural block diagram of a data transmission device deployed in a data processing component according to an embodiment of the present invention; Fig.11 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] In the process of data storage and transmission, in order to effectively utilize storage space and transmission bandwidth, data compression technology is used to compress data. At the same time, in order to protect the security of data, encryption technology is used to encrypt data. Accordingly, it is necessary to use data decompression technology to decompress the compressed data, and use decryption technology to decrypt the encrypted data. At present, in the process of data storage and transmission, pure software performs data processing such as compression, decompression, encryption, and decryption on data. The processing speed of pure software for data processing is slow. The data compression and encryption implemented by software depend on the processing power of the CPU. For large-scale data processing, the performance bottleneck of the CPU will lead to slow processing speed. In addition, data processing requires a large amount of CPU and memory resources, especially in a multi-tasking environment, resource competition may lead to performance degradation. Moreover, data encryption implemented by software is vulnerable to attack or tampering.

[0020] Introducing hardware acceleration technology into data compression and encryption, generating data processing hardware solutions, including: pure hardware encryption and decryption, which is fast and highly secure for data encryption and decryption, but can only encrypt and decrypt data, and does not solve the problem of data compression storage, resulting in low storage efficiency; pure hardware compression and decompression, which is fast and highly secure for data compression and decompression, but can only compress and decompress data, and does not solve the problem of data security. In addition, both of the above hardware solutions do not solve the problem of efficient data transmission, and data latency also brings challenges to the performance of the entire system. For example, in the FPGA (Field Programmable Gate Array) heterogeneous computing compression, encryption and decryption scenario, the traditional DMA (Direct Memory Access) data transmission first moves data from disk 1 to the host memory, and then moves data from the host memory to the memory on the FPGA device side for compression and encryption calculations. After the calculations are completed, the compressed and encrypted data is transferred to the host memory, and finally the data is moved from the host memory to disk 2 through DMA. In order to compress and encrypt the data file from disk 1 and then move it to disk 2, four DMA data transfers are required. This frequent data copying will cause a huge transmission delay, greatly reducing the overall system performance of heterogeneous encryption and decryption calculations.

[0021] None of the above data processing technical solutions can simultaneously meet the requirements of high-speed data transmission, high security and high storage rate.

[0022] In conjunction with the application scenarios on which the execution of the data transmission method depends, the application scenarios are described here.

[0023] Heterogeneous computing refers to the use of computing resources of multiple different architectures, different processors or different accelerators, combining them together to achieve higher performance and more efficient computing. Heterogeneous computing can improve the overall performance of the system by assigning tasks to the devices that are most suitable for executing them. Heterogeneous computing usually involves integrating different types of processors such as CPU, GPU (Graphics Processing Unit), FPGA, etc. to achieve acceleration and optimization of various applications.

[0024] Encryption and decryption refer to the process of protecting or hiding data by using cryptographic techniques. Encryption is the process of converting raw data into data processed by a specific algorithm, making it difficult to understand or interpret when unauthorized access occurs. Decryption is the process of restoring encrypted data to its original form. Encryption and decryption are often used to ensure the confidentiality, integrity, and authentication of data to prevent unauthorized access and modification.

[0025] Compression is a data processing technology that reduces the storage space or transmission bandwidth of data by reducing the redundant parts of data. Decompression is the reverse process of compression, which restores the compressed data to the original data. The decompression algorithm must match the compression algorithm to ensure that the data can be correctly restored.

[0026] In traditional heterogeneous computing systems, multiple DMA transfers are required for the disk storage device to transfer data to the accelerator board device. First, the disk data is transferred to the host memory through the disk's DMA, and then the data is directly transferred from the host side to the accelerator board through the device's DMA. This method has a large transmission delay. In order to solve the problem of data transmission delay in heterogeneous computing systems and achieve efficient data transmission, this application adopts the point-to-point communication method in the PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) protocol. The PCIe protocol supports multiple data transmission modes, including point-to-point communication methods and DMA. When a device has the ability to act as a bus master, it can independently initiate data transmission and send data directly to another PCIe device without relying heavily on the host's central processing unit and other system resources. Since data transmission through point-to-point communication is only carried out between devices within the system, it can significantly reduce data transmission delays and improve the overall system performance. Figure 1 As shown, disk A, the device-side memory of the FPGA, and disk B can transmit data to each other in a point-to-point communication manner through the PCIe bus without relying on the host-side CPU and host-side memory.

[0027] Based on the above content, an embodiment of the present invention provides a data transmission method, which uses memory address mapping technology to insert, manage and allocate device memory in the kernel layer of the host operating system. In this way, files in the disk can be transmitted to the data processing component in a point-to-point communication manner through the kernel layer interface function. In addition, by designing a transparent file system at the kernel layer, transparent compression and decompression and transparent encryption and decryption operations on ordinary files are realized. During use, the user does not need to care about the specific compression and decompression and encryption and decryption processes, and only needs to perform conventional read or write operations to automatically complete the compression and decompression, encryption and decryption processing of the file. This design makes user operation easier and also ensures the security and efficiency of data. This data transmission method significantly reduces the delay of data transmission, improves the efficiency and security of data processing, and improves the overall performance of the system, and has a wide range of application prospects.

[0028] According to an embodiment of the present invention, an embodiment of a management daemon switching method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a host, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] In this embodiment, a data transmission method is provided, which can be used by a host. Figure 2 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201: upon receiving a data transmission request, determining data to be processed in a first storage device according to the data transmission request.

[0030] Specifically, the host may be a server, a mobile terminal, or a host of another computer. In addition to being an instruction to transmit data, a data transmission request may also be a user-initiated read and write operation instruction, a modification instruction, etc. The operations corresponding to these instructions include operations to compress, decompress, encrypt, and decrypt data. The first storage device, for example: a hardware device that can store data, such as a disk and a flash memory, wherein the disk type is not distinguished, and the disk may be a solid-state drive, a mechanical hard disk, a USB flash drive, etc.

[0031] The data to be processed is determined in the first storage device according to the data transmission request. For example, if the first storage device is disk A and the data transmission request is to transfer file a in disk A to disk B, the data to be processed is the data of file a.

[0032] Step S202, sending a first control instruction to the first storage device so that the first storage device transmits the data to be processed to the data processing component through the bus, wherein the data processing component is used to perform preset processing on the data to be processed to obtain target data, and the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing.

[0033] Specifically, the host sends a first control instruction to the first storage device, and starts the DMA transmission process of the first storage device based on the point-to-point communication method through the first control instruction. In the DMA transmission process based on the point-to-point communication method, the first storage device directly transfers the data to be processed to the device memory of the data processing component through the bus, without relying on the CPU and host-side memory of the host. The bus is, for example, a PCIe bus, which can be a PCIe bus between the first storage device and the host, or a PCIe bus between the host and the data processing component.

[0034] Data processing components include, for example, FPGA accelerator boards with encryption, decryption, compression, and decompression function modules, where FPGA is a programmable hardware platform with high flexibility and parallel processing capabilities, and is very suitable for implementing complex data processing tasks; encryption, decryption, compression, and decompression function modules include, for example, compression and decompression IP (Intellectual Property Core), encryption and decryption IP, etc., where the intellectual property core is an integrated circuit design module with specific functions. After receiving the data to be processed, the data processing component will perform preset processing on the data to be processed to obtain the target data. The preset processing may include only one of compression processing, encryption processing, decryption processing, and decompression processing, or may include either compression processing or decompression processing, or both encryption processing and decryption processing. For example, the preset processing includes only compression processing; the preset processing includes only encryption processing; the preset processing includes compression processing and encryption processing; the preset processing includes decryption processing and decompression processing... The operations included in the preset processing are set according to the data processing requirements.

[0035] The DMA transfer process based on point-to-point communication adopts point-to-point communication and DMA technology, allowing data to be directly transferred between different devices connected to the PCIe bus without passing through the host CPU and host memory, improving the efficiency of data transmission, reducing the CPU load, and significantly improving system performance in computationally intensive tasks. Figure 3 As shown in the figure, the physical address of the device memory of the FPGA accelerator board is mapped to the memory address space in the host, and then converted into virtual memory in the kernel state for use by the system program. Based on this, a DMA transmission process based on point-to-point communication is designed between disk A and the data processing component, and a DMA transmission process based on point-to-point communication is also designed between the data processing component and disk B. DMA transmission based on point-to-point communication is implemented between disk A and FPGA, and DMA transmission based on point-to-point communication is implemented between FPGA and disk B. In addition, disk A, data processing component and disk B all contain DMA engine (Direct Memory Access Engine). DMA engine (Direct Memory Access Engine) is a hardware component in a computer system, which allows data to be directly transferred between external devices (such as disks, network cards, etc.) and system memory without continuous intervention of the CPU.

[0036] Step S203, upon receiving a preset signal sent by the data processing component, sending a second control instruction to the second storage device so that the second storage device obtains target data from the data processing component through the bus, wherein the preset signal is used to indicate completion of preset processing of the data to be processed.

[0037] Specifically, the data processing component performs preset processing on the data to be processed. After the processing is completed, the target data is obtained and stored in the device memory of the data processing component. The data processing component will send a preset signal to the host and feedback to the host that the preset processing of the data to be processed is completed. The preset signal is, for example, an interrupt signal or a signal agreed upon by other data processing components and the host. The preset signal is used to indicate the completion of the preset processing of the data to be processed. The second storage device is, for example, a hardware device that can store data, such as a disk and a flash memory. The disk type is not distinguished. The disk can be a solid-state drive, a mechanical hard disk, a USB flash drive, etc.

[0038] After receiving the preset signal, the host determines that the data processing component has completed the preset processing of the data to be processed. The host sends a second control instruction to the second storage device, and starts the DMA transmission process of the second storage device based on the point-to-point communication method through the second control instruction. In the DMA transmission process based on the point-to-point communication method, the second storage device directly obtains the target data from the device memory of the data processing component through the bus, without relying on the CPU and host-side memory of the host. The bus is, for example, a PCIe bus, which can be specifically a PCIe bus between the second storage device and the host, and a PCIe bus between the host and the data processing component.

[0039] In the data transmission method provided in this embodiment, the host instructs the first storage device to directly transmit the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. The data processing component is used to perform preset operations on the data to be processed, which improves the efficiency of data processing and ensures the security and compliance of data transmission and storage. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0040] In some optional implementations, before determining the data to be processed in the first storage device according to the data transmission request, the method further includes: Using a preset driver program to obtain a physical address of a device memory sent by a data processing component, wherein the device memory is included in the data processing component; The physical address is mapped to the virtual address space of the host by using the memory address mapping technology to obtain the kernel virtual address, wherein the kernel virtual address is used to locate the storage space in the device memory.

[0041] Specifically, this embodiment designs a data transmission process based on a point-to-point communication method, such as Figure 4As shown, firstly, the PCIe-IP, compression IP, encryption and decryption IP and memory controller are connected inside the FPGA accelerator board through the AXI (Advanced eXtensible Interface) bridge, and then the memory controller is connected to the device memory. Among them, PCIE-IP is responsible for data transmission on the PCIe bus; compression and decompression IP realizes the compression and decompression functions of files; encryption and decryption IP is responsible for data encryption and decryption operations; AXI bridge connects PCIE-IP, compression and decompression IP, encryption and decryption IP and memory controller to ensure efficient communication between modules; memory controller manages and controls device memory for temporary storage of encrypted and decrypted data. When the FPGA accelerator board device is installed on the host, the system is started. At this time, the FPGA hardware device will map the physical address of the device memory to the host side, and the driver will call the relevant functions in the kernel to obtain the physical address of the device memory, and then add the device memory address space to the host side memory address space through the memory address mapping technology of the Linux kernel, and then convert it into kernel state virtual memory for system program use. Next, the DMA engine in Disk A directly transfers the data from Disk A to the device memory through point-to-point communication. After that, compression and encryption calculations are started on the FPGA. After the calculations are completed, the data is directly transferred to Disk B through point-to-point communication. The entire compression and encryption process only requires two DMA transmission processes based on point-to-point communication.

[0042] The host has an operating system. This embodiment is described by taking the operating system as Linux as an example. The preset driver is, for example, a driver for a point-to-point communication mode. The preset driver is loaded into the kernel of the operating system. First, the driver for the point-to-point communication mode is loaded into the Linux kernel so that subsequent operations can use the point-to-point communication mode for efficient data transmission.

[0043] After the data processing unit is installed on the host, the data processing unit obtains the physical address (PA) of the device memory in the data processing unit. If the host operating system is started, the data processing unit will map the physical address of the device memory to the host.

[0044] Memory address mapping technology, for example: Linux kernel memory mapping technology, Linux kernel hot-swap technology, etc. The host uses a preset driver to obtain the physical address of the device memory sent by the data processing component, and then uses memory address mapping technology to map the physical address to the virtual address space of the host. Through the memory mapping operation, the kernel virtual address (Virtual Address, VA) corresponding to the physical address on the device side is obtained. The kernel virtual address can be further converted into kernel-state virtual memory for use by system programs. The host's system program can locate the storage space in the device memory based on the kernel virtual address.

[0045] In this embodiment, the physical address of the device memory in the data processing component is mapped to the virtual address space of the host to obtain the kernel virtual address space. The kernel virtual address is used to achieve end-to-end efficient transmission between the storage device and the data processing component, reduce the number of data copies, and reduce data transmission delays.

[0046] In some optional implementations, sending the first control instruction to the first storage device includes: In the transparent file system, determining a first location address of the data to be processed in the first storage device and a first bus address corresponding to the device memory; Generate a first descriptor including a first location address and a first bus address, wherein the first descriptor is included in a first control instruction, the source address in the first descriptor is the first location address, and the destination address in the first descriptor is the first bus address; The first descriptor is written into a first register corresponding to the first storage device, so that the first storage device sends the data to be processed from the source address of the first descriptor to the destination address of the first descriptor.

[0047] Specifically, the host has an operating system, and this embodiment is explained by taking the Linux operating system as an example. The operating system is divided into a kernel layer and an application layer. At the application layer, users can initiate read and write operations on files, including compression, decompression, encryption and decryption. At the kernel layer, memory resources are managed and allocated through a memory management mechanism, and file operations are managed through a transparent file system. The kernel layer also includes disk drivers and point-to-point communication drivers. The transparent file system is not only responsible for reading and writing files, but also triggers data transmission based on point-to-point communication, and calls compression, decompression, and encryption and decryption operation processes. In addition, the kernel layer driver is responsible for managing various devices. Figure 5As shown in the figure, disk A, disk B and FPGA accelerator board are connected to the host through the PCIe bus. The data processing component contains multiple key modules, such as PCIE-IP, compression and decompression IP, encryption and decryption IP, AXI bridge, memory controller and device memory. These components work together to ensure efficient data transmission between the disk and the data processing component and complete the required compression, decompression and encryption and decryption operations.

[0048] The user initiates a read / write operation at the application layer, and the host receives the data transmission request, triggering the compression and encryption process. Enter the kernel layer transparent file system, and the read operation enters the kernel layer transparent file system. In the transparent file system, the kernel_read function is called, and the kernel_read function is used to determine the first location address (faddr1) of the data to be processed in the first storage device and the bus address (Bus Address, BA) corresponding to the device memory as the first bus address (BA1).

[0049] A DMA descriptor including a first location address and a first bus address is constructed as a first descriptor, and a DMA source address in the first descriptor is set to faddr1 and a destination address to BA1. The first descriptor is included in a first control instruction.

[0050] The first descriptor is written into a DMA-related register corresponding to the first storage device, namely, the first register, and a DMA transmission process based on a point-to-point communication method of the first storage device is started according to the data in the first register. In the DMA transmission process based on the point-to-point communication method, the first storage device directly sends the data to be processed from the source address of the first descriptor to the destination address of the first descriptor through the bus, and writes it into the device memory of the data processing component.

[0051] In this embodiment, a first descriptor is generated in the transparent file system, and the first descriptor is written into the first register corresponding to the first storage device, so that the first storage device directly sends the data to be processed to the data processing component from the source address of the first descriptor to the destination address of the first descriptor. Transparent operation is achieved, and data transmission and data processing are all implemented in the kernel state, reducing system calls and process switching between the kernel state and the user state, and the user is not aware of it.

[0052] In some optional implementations, determining a first location address of the data to be processed in the first storage device and a first bus address corresponding to the device memory includes: Obtaining a first sub-kernel virtual address, a first file descriptor of a file where the data to be processed is located, and a first file offset, wherein the first sub-kernel virtual address is a kernel virtual address corresponding to a space in a device memory for storing the data to be processed; Determine, according to the first file descriptor and the first file offset, first meta information of the file where the to-be-processed data is located in the first storage device, and obtain a first position address in the first meta information; The first sub-kernel virtual address is converted into a first page, and the first page is converted into a first bus address.

[0053] Specifically, a blank space is determined in the device memory for storing the data to be processed, and the first sub-kernel virtual address (VA1) corresponding to the space for storing the data to be processed is determined in the virtual address (VA). The function input information of the user layer is determined, and the first file descriptor (fp1), the first file offset (off1) and the file size of the file where the data to be processed are located are obtained.

[0054] According to the first file descriptor (fp1) and the first file offset (off1), first meta information of the file where the data to be processed is located in the first storage device is determined, and a first position address (faddr1) of the file where the data to be processed is located in the first storage device is obtained from the first meta information.

[0055] The first sub-core virtual address (VA1) is converted into a page as a first page, and the first page is then converted into a first bus address.

[0056] In this embodiment, the first location address of the file where the data to be processed is located in the first storage device is obtained in the first metadata, and the first sub-kernel virtual address is converted into the first bus address. The first location address and the first bus address are accurately obtained to ensure that the subsequent data transmission address is correct, and the correct data is transmitted to the destination address.

[0057] In some optional implementations, sending the second control instruction to the second storage device includes: Creating a target file in a second storage device using a transparent file system; In the transparent file system, determining a second location address of the target file in the second storage device and a second bus address corresponding to the device memory; Generate a second descriptor including a second location address and a second bus address, wherein the second descriptor is included in a second control instruction, the source address in the second descriptor is the second bus address, and the destination address in the second descriptor is the second location address; The second descriptor is written into a second register corresponding to the second storage device, so that the second storage device sends the target data from the source address of the second descriptor to the destination address of the second descriptor.

[0058] Specifically, after receiving the preset signal sent by the data processing component, the transparent file system of the host determines that the data processing component completes the preset processing of the data to be processed. The transparent file system creates an empty file in the second storage device as the target file.

[0059] The transparent file system determines a second location address (faddr2) of the target file in the second storage device and a second bus address (BA2) corresponding to the device memory.

[0060] A DMA descriptor including a second location address and a second bus address is constructed as a second descriptor, and a DMA source address in the second descriptor is set to BA2 and a destination address to faddr2. The second descriptor is included in a second control instruction.

[0061] The second descriptor is written into a DMA-related register corresponding to the second storage device, namely, a second register, and a DMA transmission process based on a point-to-point communication method of the second storage device is started according to the data in the second register. In the DMA transmission process based on the point-to-point communication method, the second storage device sends the target data from the source address of the second descriptor to the destination address of the second descriptor, and writes it into the target file of the second storage device.

[0062] In some optional implementations, determining the second location address of the target file in the second storage device and the second bus address corresponding to the device memory includes: Obtaining a second sub-kernel virtual address, a second file descriptor of the target file, and a second file offset, wherein the second sub-kernel virtual address is a kernel virtual address corresponding to a space in the device memory for storing target data; Determine second meta information of the target file in the second storage device according to the second file descriptor and the second file offset, and obtain a second location address in the second meta information; The second sub-kernel virtual address is converted into a second page, and the second page is converted into a second bus address.

[0063] Specifically, a space for storing target data in the device memory is determined, and a second sub-kernel virtual address (VA2) corresponding to the space for storing target data is determined in the virtual address (VA). VA2 is obtained through a memory allocation function of a point-to-point communication method in the Linux kernel.

[0064] Through the special open function in the kernel, the second file descriptor (fp2) and the second file offset (off2) of the target file are parsed in the driver according to the information of the target file, and the file size of the target file can also be obtained.

[0065] According to the second file descriptor and the second file offset, second meta information of the target file in the second storage device is determined, and a second location address (faddr2) of the target file in the second storage device is obtained from the second meta information.

[0066] The second sub-core virtual address (VA2) is converted into a page as a second page, and the second page is then converted into a second bus address.

[0067] According to an embodiment of the present invention, an embodiment of a data transmission method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a data processing component, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0068] In this embodiment, a data transmission method is provided, which can be used in a data processing component. Figure 6 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps: Step S601, when receiving the data to be processed, perform preset processing on the data to be processed to obtain target data, wherein the data to be processed is transmitted to the data processing component through the bus after the first storage device receives the first control instruction, the first control instruction is sent to the first storage device after the host determines the data to be processed in the first storage device, the data to be processed is determined according to the data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing.

[0069] Specifically, the data processing components include: an FPGA accelerator board with encryption, decryption, compression, and decompression functional modules, wherein FPGA is a programmable hardware platform with high flexibility and parallel processing capabilities, and is very suitable for implementing complex data processing tasks; encryption, decryption, compression, and decompression functional modules such as compression and decompression IP (Intellectual Property Core), encryption and decryption IP, etc., and the intellectual property core is an integrated circuit design module with specific functions.

[0070] The host determines the data to be processed in the first storage device according to the data transmission request. The host sends a first control instruction to the first storage device, and starts the DMA transmission process of the first storage device based on the point-to-point communication method through the first control instruction. In the DMA transmission process based on the point-to-point communication method, the first storage device directly transfers the data to be processed to the device memory of the data processing component through the bus, without relying on the CPU and host-side memory of the host. The bus is, for example, a PCIe bus, which can be specifically a PCIe bus between the first storage device and the host, or a PCIe bus between the host and the data processing component.

[0071] After receiving the data to be processed, the data processing component will perform preset processing on the data to be processed to obtain the target data. The preset processing may include only one of compression processing, encryption processing, decryption processing, and decompression processing, or it may include either compression processing or decompression processing, or both encryption processing and decryption processing. For example: the preset processing includes only compression processing; the preset processing includes only encryption processing; the preset processing includes compression processing and encryption processing; the preset processing includes decryption processing and decompression processing... The operations included in the preset processing are set according to the data processing requirements.

[0072] Step S602, after performing preset processing on the data to be processed, sending a preset signal to the host so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate that the preset processing of the data to be processed is completed, and the second control instruction is used to instruct the second storage device to obtain target data from the data processing component through the bus.

[0073] Specifically, the data processing component performs preset processing on the data to be processed, obtains target data after the processing is completed, and stores the target data in the device memory of the data processing component. The data processing component sends a preset signal to the host to feedback to the host that the preset processing of the data to be processed is completed. The preset signal is, for example, an interrupt signal or a signal agreed upon by the other data processing component and the host. The preset signal is used to indicate that the preset processing of the data to be processed is completed.

[0074] After receiving the preset signal, the host determines that the data processing component has completed the preset processing of the data to be processed. The host sends a second control instruction to the second storage device, and starts the DMA transmission process of the second storage device based on the point-to-point communication method through the second control instruction. In the DMA transmission process based on the point-to-point communication method, the second storage device directly obtains the target data from the device memory of the data processing component through the bus, without relying on the CPU and host-side memory of the host. The bus is, for example, a PCIe bus, which can be specifically a PCIe bus between the second storage device and the host, and a PCIe bus between the host and the data processing component.

[0075] The data transmission method provided in this embodiment uses the data processing component to perform preset operations on the data to be processed, thereby improving the efficiency of data processing and ensuring the security and compliance during data transmission and storage. The first storage device directly transmits the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0076] In some optional implementations, before performing preset processing on the data to be processed, the method further includes: Obtain the physical address of the device memory in the data processing unit; The physical address is sent to the host, so that the host maps the physical address to the virtual address space of the host according to the preset driver and memory address mapping technology to obtain the kernel virtual address, wherein the kernel virtual address is used to determine the data stored in the device memory.

[0077] Specifically, after the data processing component is installed on the host, the data processing component obtains the physical address of the device memory in the data processing component.

[0078] If the host operating system is started, the data processing component will map the physical address of the device memory to the host. The host uses the preset driver to obtain the physical address of the device memory sent by the data processing component, and then uses the memory address mapping technology to map the physical address to the virtual address space of the host. Through the memory mapping operation, the kernel virtual address corresponding to the physical address on the device side is obtained. The system program of the host can locate the storage space in the device memory according to the kernel virtual address.

[0079] In this embodiment, the physical address of the device memory in the data processing component is sent to the host, and the host maps the physical address to the virtual address space of the host to obtain the kernel virtual address space. The kernel virtual address is used to achieve end-to-end efficient transmission between the storage device and the data processing component, reduce the number of data copies, and reduce data transmission delays.

[0080] In some optional implementations, performing preset processing on the data to be processed to obtain target data includes: In the case where the preset processing includes compression processing and encryption processing, compressing the data to be processed to obtain first intermediate data; encrypting the first intermediate data to obtain target data; or, In the case where the preset processing includes decryption processing and decompression processing, the data to be processed is decrypted to obtain second intermediate data; and the second intermediate data is decompressed to obtain target data.

[0081] Specifically, the data processing component includes: a first component and a second component. The first component is used to compress and decompress the data to be processed, such as a compressed IP. The second component is used to encrypt and decrypt the data to be processed, such as an encrypted and decrypted IP.

[0082] In the case where the preset processing includes compression processing and encryption processing, the first component is used to compress the data to be processed to obtain the first intermediate data, and the second component is used to encrypt the first intermediate data to obtain the target data.

[0083] In the case where the preset processing includes decryption processing and decompression processing, the second component is used to decrypt the data to be processed to obtain the second intermediate data, and the first component is used to decompress the second intermediate data to obtain the target data.

[0084] In this embodiment, the data processing component is used to perform preset operations on the data to be processed, which not only improves the data processing speed, but also has a high data compression ratio, in order to effectively utilize storage space and transmission bandwidth, and encrypts the data to ensure data security.

[0085] In this embodiment, a data transmission system is provided. Figure 7 As shown, the system includes: a host, a data processing component, a first storage device and a second storage device; The host is connected to the first storage device via a first bus, the host is connected to the data processing component via a second bus, and the host is connected to the second storage device via a third bus; The host is used to execute steps S201 to S203, or the data transmission method of any corresponding implementation manner.

[0086] The data processing component is used to execute step S601 to step S602, or the data transmission method of any corresponding implementation manner.

[0087] Specifically, the specific contents of steps S201 to S203, or the data transmission method of any corresponding implementation manner, refer to the above embodiments, and the specific contents of steps S601 to S602, or the data transmission method of any corresponding implementation manner, refer to the above embodiments, which will not be repeated here.

[0088] In the data transmission system provided in this embodiment, the host instructs the first storage device to directly transmit the data to be processed to the data processing component through the bus, and the second storage device directly obtains the target data from the data processing component through the bus without relying on the host's CPU and host-side memory, thereby reducing the number of data copies, reducing the delay in data transmission, and improving the overall performance of the system. The data processing component is used to perform preset operations on the data to be processed, which improves the efficiency of data processing and ensures the security and compliance of data transmission and storage. It solves the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission.

[0089] In some optional implementations, the data processing component includes: a first component, a second component, a connection component, a memory controller, and a device memory; the connection component is connected to the first component, the second component, and the memory controller; Wherein, in the case where the preset processing includes compression processing and encryption processing, the memory controller is used to transmit the data to be processed in the device memory to the first component through the connection component; the first component is used to compress the data to be processed to obtain first intermediate data; the second component is used to encrypt the first intermediate data to obtain target data, and transmit the target data to the memory controller through the connection component; the memory controller is used to save the target data to the device memory; or, When the preset processing includes decryption processing and decompression processing, the memory controller is used to transmit the data to be processed in the device memory to the second component through the connection component; the second component is used to decrypt the data to be processed to obtain second intermediate data; the first component is used to decompress the second intermediate data to obtain target data, and transmit the target data to the memory controller through the connection component; the memory controller is used to save the target data to the device memory.

[0090] Specifically, the first component is used to compress and decompress the data to be processed, and the first component is, for example, a compression IP. The second component is used to encrypt and decrypt the data to be processed, and the second component is, for example, an encryption and decryption IP. The connection component is, for example, an AXI bridge. Figure 4 As shown, the connection component is connected to the first component, the second component, and the memory controller.

[0091] In the case where the preset processing includes compression processing and encryption processing, the memory controller transmits the to-be-processed data in the device memory to the first component through the connection component, and uses the first component to compress the to-be-processed data to obtain first intermediate data. The second component encrypts the first intermediate data to obtain target data. The second component transmits the target data to the memory controller through the connection component; the memory controller saves the target data to the device memory; In the case where the preset processing includes decryption processing and decompression processing, the memory controller transmits the data to be processed in the device memory to the second component through the connection component, and uses the second component to decrypt the data to be processed to obtain second intermediate data. The second intermediate data is decompressed by the first component to obtain target data. The first component transmits the target data to the memory controller through the connection component; the memory controller saves the target data to the device memory.

[0092] In some optional implementations, when the preset processing includes compression processing and encryption processing, the specific process of the data processing component processing the data to be processed includes steps A1 to A4.

[0093] In step A1, the data processing component obtains the hash sequence of the key using the streamingXXHash32 function according to the preset key, and uses shiftWeight to calculate the shift weight value corresponding to the hash sequence. ShiftWeight is a technology or algorithm for calculating the hash sequence.

[0094] In step A2, the data processing component uses the WarpCompressor function to compress the data to be processed to obtain compressed text, and uses the compress_WT function to shift-encrypt the compressed text according to the shift weight value to obtain shift ciphertext.

[0095] Step A3: The data processing component uses the getEncoderBytes function to perform Base64 encryption on the shifted ciphertext, so that the ciphertext is an ASCII code sequence, and obtains Base64 ciphertext.

[0096] Step A4: The data processing component uses the getUniformConfuseStringList function to generate a confusion sequence according to the hash sequence, so that the confusion sequence is an ASCII code sequence, and uses the getCipherString function to insert the confusion sequence into the Base64 ciphertext to obtain the target data.

[0097] In this embodiment, the data processing component uses a key to encrypt the data to be processed, so that the formal ciphertext obtained each time is different and the length is also different, which greatly increases the difficulty of ciphertext comparison and cracking. Effective data compression of the data to be processed can save a lot of storage space.

[0098] In some optional implementations, a method for compressed and encrypted data transmission is disclosed, which can solve the problem of difficulty in quickly compressing, decompressing, encrypting and decrypting data while ensuring low transmission delay during data transmission. Figure 8As shown, the method includes: Load the point-to-point communication driver into the kernel. The point-to-point communication driver obtains the physical address PA of the device memory mapped by the FPGA. Using the Linux kernel hot-plug technology, the physical address (PA) of the device memory is inserted into the physical address space on the host side, and then the kernel virtual address (VA) corresponding to the physical address on the device side is obtained. The user initiates a read operation (triggering the compression-encryption process). Enter the transparent file system. In the transparent file system, call the kernel_read function, the function parameters are as follows (virtual address VA, file descriptor fp, file offset off, file size), convert the virtual address VA to a page page, and convert the page page to a bus address BA; obtain the original information of the file on the disk according to the file descriptor and file offset off, and then obtain the file's location faddr on the disk. Construct a DMA descriptor, use faddr as the DMA source address, BA as the destination address, and write them into the DMA-related registers. Start the DMMA transmission of disk A based on the point-to-point communication method, and transfer the data from disk A to the DDR on the FPGA side. The FPGA compression IP performs compression calculations on the file data in the DDR on the device side. The FPGA encryption and decryption IP accelerates the calculation of the compressed data. After the calculation is completed, the data is saved to another address of the device DDR, and then an interrupt is generated. After receiving the interrupt, the transparent file system opens an empty file to obtain a file fp2, and initiates a kernel write operation (the process is similar to kernel read). The parameters of this function are as follows (virtual address VA2, file descriptor fp2, file offset offf2, file size). According to the file descriptor fp2 and the file offset of2, the metadata of the file on the disk is obtained, and then the location of the file on the disk faddr2 is obtained. The virtual address VA2 is converted to a page page, and the page page is converted to the bus address BA2. Construct a DMA descriptor, use BA2 as the DMA source address, faddr2 as the destination address, and write it to the DMA-related registers. Finally, the DMMA transmission based on the point-to-point communication method of disk B is started to transfer the encrypted data from the DDR on the FPGA side to disk B.

[0099] In this embodiment, memory address mapping technology is used to insert, manage and allocate device memory at the kernel layer of the Linux operating system. In this way, files in the disk can be transferred to the accelerator device in a point-to-point communication mode through the kernel layer interface function. This data transmission method significantly reduces the delay of data transmission and improves the overall performance of the system.

[0100] In this embodiment, a data transmission device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0101] This embodiment provides a data transmission device, which is deployed on a host, such as Fig. 9 As shown, including: A data determination module 901 is used to determine the data to be processed in the first storage device according to the data transmission request when a data transmission request is received; A first instruction sending module 902 is used to send a first control instruction to the first storage device, so that the first storage device transmits the to-be-processed data to the data processing component through the bus, wherein the data processing component is used to perform a preset processing on the to-be-processed data to obtain target data, wherein the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; The second instruction sending module 903 is used to send a second control instruction to the second storage device when receiving a preset signal sent by the data processing component, so that the second storage device obtains target data from the data processing component through the bus, wherein the preset signal is used to indicate the completion of preset processing on the data to be processed.

[0102] In some optional embodiments, the device further comprises: A first acquisition module, used for acquiring a physical address of a device memory sent by a data processing component by using a preset driver, wherein the device memory is included in the data processing component; The address mapping module is used to map the physical address to the virtual address space of the host by using the memory address mapping technology to obtain the kernel virtual address, wherein the kernel virtual address is used to locate the storage space in the device memory.

[0103] In some optional implementations, the first instruction sending module 902 includes: A first determining unit, configured to determine, in the transparent file system, a first location address of the data to be processed in the first storage device and a first bus address corresponding to the device memory; A first generating unit, configured to generate a first descriptor including a first location address and a first bus address, wherein the first descriptor is included in a first control instruction, a source address in the first descriptor is the first location address, and a destination address in the first descriptor is the first bus address; The first writing unit is used to write the first descriptor into a first register corresponding to the first storage device, so that the first storage device sends the to-be-processed data from the source address of the first descriptor to the destination address of the first descriptor.

[0104] In some optional implementations, the first determining unit includes: A first acquisition submodule is used to acquire a first sub-kernel virtual address, a first file descriptor of a file where the data to be processed is located, and a first file offset, wherein the first sub-kernel virtual address is a kernel virtual address corresponding to a space in a device memory for storing the data to be processed; A first determination submodule, configured to determine first metadata of a file where the to-be-processed data is located in a first storage device according to the first file descriptor and the first file offset, and obtain a first position address in the first metadata; The first conversion submodule is used to convert the first sub-kernel virtual address into a first page, and convert the first page into a first bus address.

[0105] In some optional implementations, the second instruction sending module 903 includes: A creating unit, used for creating a target file in a second storage device using a transparent file system; A second determining unit, configured to determine, in the transparent file system, a second location address of the target file in the second storage device and a second bus address corresponding to the device memory; A second generating unit, configured to generate a second descriptor including a second location address and a second bus address, wherein the second descriptor is included in a second control instruction, the source address in the second descriptor is the second bus address, and the destination address in the second descriptor is the second location address; The second writing unit is used to write the second descriptor into a second register corresponding to the second storage device, so that the second storage device sends the target data from the source address of the second descriptor to the destination address of the second descriptor.

[0106] In some optional implementations, the second determining unit includes: A second acquisition submodule is used to acquire a second sub-kernel virtual address, a second file descriptor of the target file, and a second file offset, wherein the second sub-kernel virtual address is a kernel virtual address corresponding to a space in the device memory for storing target data; A second determination submodule, configured to determine second meta information of the target file in the second storage device according to the second file descriptor and the second file offset, and obtain a second location address in the second meta information; The second conversion submodule is used to convert the second sub-kernel virtual address into a second page, and convert the second page into a second bus address.

[0107] This embodiment provides a data transmission device, which is deployed in a data processing component, such as Fig.10 As shown, including: The data processing module 1001 is used to perform preset processing on the data to be processed to obtain target data when receiving the data to be processed, wherein the data to be processed is transmitted to the data processing component through the bus after the first storage device receives the first control instruction, the first control instruction is sent to the first storage device after the host determines the data to be processed in the first storage device, the data to be processed is determined according to the data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; The signal sending module 1002 is used to send a preset signal to the host after performing preset processing on the data to be processed, so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate the completion of the preset processing on the data to be processed, and the second control instruction is used to instruct the second storage device to obtain target data from the data processing component through the bus.

[0108] In some optional embodiments, the device further comprises: A second acquisition module is used to acquire the physical address of the device memory in the data processing component; The sending module is used to send the physical address to the host, so that the host maps the physical address to the virtual address space of the host according to the preset driver and memory address mapping technology to obtain the kernel virtual address, wherein the kernel virtual address is used to determine the data stored in the device memory.

[0109] In some optional implementations, the data processing module 1001 includes: A first processing unit, configured to compress the data to be processed to obtain first intermediate data when the preset processing includes compression processing and encryption processing; A second processing unit is used to encrypt the first intermediate data to obtain target data; A third processing unit, configured to perform decryption processing on the data to be processed to obtain second intermediate data when the preset processing includes decryption processing and decompression processing; The fourth processing unit is used to decompress the second intermediate data to obtain target data.

[0110] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0111] The data transmission device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0112] The embodiment of the present invention also provides a computer device having the above Fig. 9 or Fig.10 The data transmission device shown.

[0113] See also Fig.11 , Fig.11 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.11 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.

[0114] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0115] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0116] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0118] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0119] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0120] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0121] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined in this application.

Claims

1. A data transmission method, characterized in that: The method is applied to a host, and the method comprises: In case of receiving a data transmission request, determining the data to be processed in the first storage device according to the data transmission request; Sending a first control instruction to the first storage device, so that the first storage device transmits the data to be processed to a data processing component through a bus, wherein the data processing component is used to perform a preset process on the data to be processed to obtain target data, wherein the preset process includes compression processing and / or encryption processing, or the preset process includes decryption processing and / or decompression processing; Upon receiving a preset signal sent by the data processing component, a second control instruction is sent to a second storage device so that the second storage device obtains the target data from the data processing component through the bus, wherein the preset signal is used to indicate the completion of preset processing of the data to be processed.

2. The method according to claim 1, characterized in that Before determining the data to be processed in the first storage device according to the data transmission request, the method further includes: Using a preset driver program to obtain a physical address of a device memory sent by the data processing component, wherein the device memory is included in the data processing component; The physical address is mapped into the virtual address space of the host by using the memory address mapping technology to obtain a kernel virtual address, wherein the kernel virtual address is used to locate the storage space in the device memory.

3. The method according to claim 2, characterized in that The sending a first control instruction to the first storage device includes: In the transparent file system, determining a first location address of the to-be-processed data in the first storage device and a first bus address corresponding to the device memory; generating a first descriptor including the first location address and the first bus address, wherein the first descriptor is included in the first control instruction, the source address in the first descriptor is the first location address, and the destination address in the first descriptor is the first bus address; The first descriptor is written into a first register corresponding to the first storage device, so that the first storage device sends the to-be-processed data from a source address of the first descriptor to a destination address of the first descriptor.

4. The method according to claim 3, characterized in that The determining of a first location address of the to-be-processed data in the first storage device and a first bus address corresponding to the device memory includes: Obtaining a first sub-kernel virtual address, a first file descriptor of a file where the data to be processed is located, and a first file offset, wherein the first sub-kernel virtual address is a kernel virtual address corresponding to a space in the device memory for storing the data to be processed; Determine, according to the first file descriptor and the first file offset, first metadata of the file where the to-be-processed data is located in the first storage device, and obtain the first location address from the first metadata; The first sub-kernel virtual address is converted into a first page, and the first page is converted into the first bus address.

5. The method according to claim 2, characterized in that: The sending the second control instruction to the second storage device includes: Creating a target file in the second storage device using a transparent file system; In the transparent file system, determining a second location address of the target file in the second storage device and a second bus address corresponding to the device memory; Generate a second descriptor including the second location address and the second bus address, wherein the second descriptor is included in the second control instruction, the source address in the second descriptor is the second bus address, and the destination address in the second descriptor is the second location address; The second descriptor is written into a second register corresponding to the second storage device, so that the second storage device sends the target data from the source address of the second descriptor to the destination address of the second descriptor.

6. The method according to claim 5, characterized in that The determining of a second location address of the target file in the second storage device and a second bus address corresponding to the device memory includes: Obtaining a second sub-kernel virtual address, a second file descriptor of the target file, and a second file offset, wherein the second sub-kernel virtual address is a kernel virtual address corresponding to a space in the device memory for storing the target data; Determine second meta information of the target file in the second storage device according to the second file descriptor and the second file offset, and obtain the second location address from the second meta information; The second sub-kernel virtual address is converted into a second page, and the second page is converted into the second bus address.

7. A data transmission method, characterized in that: The method is applied to a data processing component, and the method comprises: In the case of receiving data to be processed, performing preset processing on the data to be processed to obtain target data, wherein the data to be processed is transmitted to the data processing component through a bus after the first storage device receives a first control instruction, the first control instruction is sent to the first storage device after the host determines the data to be processed in the first storage device, the data to be processed is determined according to a data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; After performing preset processing on the data to be processed, a preset signal is sent to the host so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate that the preset processing of the data to be processed is completed, and the second control instruction is used to instruct the second storage device to obtain the target data from the data processing component through the bus.

8. The method according to claim 7, characterized in that Before performing the preset processing on the data to be processed, the method further includes: Obtaining a physical address of a device memory in the data processing component; The physical address is sent to the host so that the host maps the physical address to the virtual address space of the host according to a preset driver and memory address mapping technology to obtain a kernel virtual address, wherein the kernel virtual address is used to determine the data stored in the device memory.

9. The method according to claim 7, characterized in that: The step of performing preset processing on the data to be processed to obtain target data includes: In the case where the preset processing includes compression processing and encryption processing, compressing the data to be processed to obtain first intermediate data; encrypting the first intermediate data to obtain the target data; or, In the case where the preset processing includes decryption processing and decompression processing, the data to be processed is decrypted to obtain second intermediate data; and the second intermediate data is decompressed to obtain the target data.

10. A data transmission system, characterized in that: The system comprises: a host, a data processing component, a first storage device and a second storage device; The host is connected to the first storage device via a first bus, the host is connected to the data processing component via a second bus, and the host is connected to the second storage device via a third bus; The host is used to execute the data transmission method according to any one of claims 1 to 6; The data processing component is used to execute the data transmission method according to any one of claims 7 to 9.

11. The system according to claim 10, characterized in that The data processing component includes: a first component, a second component, a connection component, a memory controller and a device memory; the connection component is connected to the first component, the second component and the memory controller; Wherein, in the case where the preset processing includes compression processing and encryption processing, the memory controller is used to transmit the data to be processed in the device memory to the first component through the connection component; the first component is used to compress the data to be processed to obtain first intermediate data; the second component is used to encrypt the first intermediate data to obtain target data, and transmit the target data to the memory controller through the connection component; the memory controller is used to save the target data to the device memory; or, In the case where the preset processing includes decryption processing and decompression processing, the memory controller is used to transmit the data to be processed in the device memory to the second component through the connection component; the second component is used to decrypt the data to be processed to obtain second intermediate data; the first component is used to decompress the second intermediate data to obtain the target data, and transmit the target data to the memory controller through the connection component; the memory controller is used to save the target data to the device memory.

12. A data transmission device, characterized in that: The device is deployed on a host, and includes: a data determination module, configured to determine the data to be processed in the first storage device according to the data transmission request when a data transmission request is received; a first instruction sending module, configured to send a first control instruction to the first storage device, so that the first storage device transmits the data to be processed to a data processing component through a bus, wherein the data processing component is configured to perform a preset process on the data to be processed to obtain target data, wherein the preset process includes compression process and / or encryption process, or the preset process includes decryption process and / or decompression process; A second instruction sending module is used to send a second control instruction to a second storage device upon receiving a preset signal sent by the data processing component, so that the second storage device obtains the target data from the data processing component through the bus, wherein the preset signal is used to indicate the completion of preset processing of the data to be processed.

13. A data transmission device, characterized in that: The device is deployed in a data processing component, and the device includes: a data processing module, configured to, upon receiving data to be processed, perform preset processing on the data to be processed to obtain target data, wherein the data to be processed is transmitted to the data processing component through a bus after the first storage device receives a first control instruction, the first control instruction is sent to the first storage device by the host after the host determines the data to be processed in the first storage device, the data to be processed is determined according to a data transmission request, the preset processing includes compression processing and / or encryption processing, or the preset processing includes decryption processing and / or decompression processing; A signal sending module is used to send a preset signal to the host after performing preset processing on the data to be processed, so that the host sends a second control instruction to the second storage device, wherein the preset signal is used to indicate that the preset processing of the data to be processed is completed, and the second control instruction is used to instruct the second storage device to obtain the target data from the data processing component through the bus.

14. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data transmission method according to any one of claims 1 to 9 by executing the computer instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data transmission method according to any one of claims 1 to 9.

Images