A data processing method, device and medium based on PCIe P2P

Abstract

The embodiment of the present application provides a data processing method, device and medium based on PCIe P2P, the method comprising: according to the correspondence between the FPGA board and the GPU device, transmitting the data to the GPU device through the PCIe bus between the FPGA board and its corresponding GPU device; calculating the data through the GPU device to obtain the corresponding data processing result; transmitting the data processing result to the back end through the PCIe bus between the GPU device and the network card; determining the device type communicating with the NVMe SSD, and storing the data or the data processing result in the NVMe SSD through the PCIe bus between the NVMe SSD and the device type, so as to solve the problem that the traditional PCIe data transmission method usually requires data to move back and forth between the device and the host memory, resulting in a certain delay and system load.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a data processing method, device and medium based on PCIe P2P. Background Art

[0002] In some scenarios that require high data throughput and low latency, such as scientific simulation and big data analysis, real-time signal processing, and online training of artificial intelligence models, the performance requirements for high-speed data collection, transmission, reception, and computational processing are very high. Traditional computing architectures and data transmission methods have been unable to meet the requirements for data processing speed and efficiency, and the high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) as a high-bandwidth, low-latency bus standard is widely used to connect various high-speed devices on computer motherboards to increase data processing speed. However, traditional PCIe data transmission methods usually require data to move back and forth between the device and host memory. This process involves CPU scheduling, resulting in certain delays and system loads. Summary of the invention

[0003] One or more embodiments of the present application provide a PCIe P2P-based data processing method, which is applied to a PCIe Switch server, wherein the PCIE Switch server includes an FPGA board, a GPU device, an NVMe SSD, and a network card. The method includes:

[0004] Collecting data through the FPGA board, and transmitting the data to the GPU device through a PCIe bus between the FPGA board and its corresponding GPU device according to the corresponding relationship between the FPGA board and the GPU device;

[0005] Calculate the data by using the GPU device to obtain corresponding data processing results;

[0006] Transmitting the data processing result to a backend via a PCIe bus between the GPU device and the network card;

[0007] According to the type of data that the NVMe SSD needs to store, the type of device that communicates with the NVMe SSD is determined, and the data or the data processing result is stored in the NVMe SSD through a PCIe bus between the NVMe SSD and the device type.

[0008] Furthermore, before the GPU device calculates the data to obtain the corresponding data processing result, the method further includes:

[0009] Based on the FPGA board, a first buffer in an idle state is screened out from a receiving circular buffer pool, the data is written into the first buffer, and a state flag bit corresponding to the first buffer is converted from the idle state to a current write state;

[0010] When the capacity limit of the first buffer is reached, the state flag corresponding to the next buffer of the first buffer is marked as an idle state, and the unfinished written data is written into the next buffer until the writing of the data is completed.

[0011] Furthermore, the data is calculated by the GPU device to obtain corresponding data processing results, which specifically include:

[0012] Based on the GPU device, determining whether the number of buffers in the receiving circular buffer pool that are not in an idle state is greater than 0;

[0013] If yes, filter out the designated buffer whose status flag is the current read status from the receiving circular buffer pool, read the data from the first designated buffer in the designated buffers, calculate the data, and obtain the corresponding data processing result;

[0014] The status flag corresponding to the first designated buffer is converted from the current read state to the idle state to release the first designated buffer, and the status flag corresponding to the next designated buffer of the first designated buffer is marked as the current read state to realize the cyclic calculation of the next designated buffer.

[0015] Furthermore, the processing tasks performed by the FPGA board, the GPU device, the NVMe SSD and the network card are all controlled by their corresponding CPU cores.

[0016] Furthermore, after obtaining the corresponding data processing result, the method further includes:

[0017] Based on the GPU device, a second buffer in an idle state is screened out from a result circular buffer pool, and the data processing result is written into the second buffer.

[0018] Further, the data processing result is transmitted to the back end through the PCIe bus between the GPU device and the network card, specifically including:

[0019] Based on the PCIe bus between the GPU device and the network card, a memory mapping table between the memory address space of the network card and the video memory address space of the GPU device is established through the RDMA technology, so that the network card obtains the data processing result from the result circular buffer pool through the memory mapping table, and transmits the data processing result to the back end.

[0020] Further, according to the type of data required to be stored by the NVMe SSD, determining the type of device communicating with the NVMe SSD specifically includes:

[0021] Determine the data type that the NVMe SSD needs to store; wherein the data type includes original data and data processing results;

[0022] When the data type is the original data, the device type communicating with the NVMe SSD is the FPGA board;

[0023] When the data type is the data processing result, the device type communicating with the NVMe SSD is the GPU device.

[0024] Furthermore, the receiving circular buffer pool and the result circular buffer pool are composed of a plurality of buffers, and the plurality of buffers are distributed in a ring shape.

[0025] An embodiment of the present application provides an electronic device, including:

[0026] processor; and,

[0027] a memory arranged to store computer executable instructions which, when executed, cause the processor to:

[0028] Collecting data through the FPGA board, and transmitting the data to the GPU device through a PCIe bus between the FPGA board and its corresponding GPU device according to the corresponding relationship between the FPGA board and the GPU device;

[0029] Calculate the data by using the GPU device to obtain corresponding data processing results;

[0030] Transmitting the data processing result to a backend via a PCIe bus between the GPU device and the network card;

[0031] According to the type of data that the NVMe SSD needs to store, the type of device that communicates with the NVMe SSD is determined, and the data or the data processing result is stored in the NVMe SSD through a PCIe bus between the NVMe SSD and the device type.

[0032] The present application provides a storage medium for storing computer executable instructions, which, when executed, implement:

[0033] Collecting data through the FPGA board, and transmitting the data to the GPU device through a PCIe bus between the FPGA board and its corresponding GPU device according to the corresponding relationship between the FPGA board and the GPU device;

[0034] Calculate the data by using the GPU device to obtain corresponding data processing results;

[0035] Transmitting the data processing result to a backend via a PCIe bus between the GPU device and the network card;

[0036] According to the type of data that the NVMe SSD needs to store, the type of device that communicates with the NVMe SSD is determined, and the data or the data processing result is stored in the NVMe SSD through a PCIe bus between the NVMe SSD and the device type.

[0037] By adopting the embodiment of the present invention, high-speed devices such as an FPGA board, a GPU device, a network card, and an NVMe SSD are integrated, and data is directly exchanged and transmitted through a PCIe bus. Data can be transmitted between the FPGA board and the GPU device, between the FPGA board and the NVMe SSD, between the GPU device and the NVMe SSD, and between the GPU device and the network card through PCIe P2P, so that the CPU no longer participates in data transfer and is only used for scheduling, reducing data transfer between system memories, improving data transmission performance, and avoiding the packet loss problem caused by Ethernet transmission.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A first scenario schematic diagram of a PCIe P2P-based data processing method provided for one or more embodiments of the present application;

[0041] Figure 2 A PCIe Switch server connection diagram provided for one or more embodiments of the present application;

[0042] Figure 3 A schematic diagram of a data transmission direction provided for one or more embodiments of the present application;

[0043] Figure 4 A schematic diagram of a dynamic buffer management method provided for one or more embodiments of the present application;

[0044] Figure 5 A schematic diagram of the data flow of a GPU video memory provided for one or more embodiments of the present application;

[0045] Figure 6 A control schematic diagram of a CPU and a PCIe Switch server provided for one or more embodiments of the present application;

[0046] Figure 7 A schematic diagram of the structure of a PCIe P2P-based data processing device provided for one or more embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0048] Method Embodiment

[0049] According to an embodiment of the present application, a data processing method based on PCIe P2P is provided. Figure 1is a flow chart of a data processing method based on PCIe P2P according to an embodiment of the present invention. Figure 1 As shown, the method is applied to a PCIE Switch server, which includes an FPGA board, a GPU device, an NVMe SSD, and a network card.

[0050] S101: Collect data through the FPGA board card, and according to the corresponding relationship between the FPGA board card and the GPU device, transmit the data to the GPU device through the PCIe bus between the FPGA board card and its corresponding GPU device.

[0051] The PCIe Switch server integrates multiple functional modules including data acquisition module, data processing module, processing module and transmission module. The data acquisition module is composed of FPGA board, the data processing module is composed of GPU device, the data storage module is composed of NVMe SSD, and the transmission module is composed of network card. The data acquisition module is responsible for real-time acquisition and preprocessing of high-speed data, the data processing module is responsible for high-performance parallel calculation of data, the storage module is responsible for storing raw data or processing result data in the hard disk, and the transmission module is responsible for transmitting the result to the network card and then transmitting it to the backend through the network card.

[0052] like Figure 2 The PCIe Switch server connection diagram shown in the figure uses the PCIe bus to implement P2P data transmission between the FPGA board and the GPU device, and the data can be directly read and written into the memory of the other device through the DMA controller. Figure 3 The data transmission direction diagram shown in the figure shows that the data acquisition module is composed of 2 FPGA boards, the data processing module is composed of 2 GPU A100s, the data storage module is composed of 2 NVMe SSDs, and the transmission module is composed of a dual-interface network card (MellanoxMT2892 ConnectX-6). Among them, there is a one-to-one correspondence between the FPGA board and the GPU device, that is, one FPGA board corresponds to one GPU device, and the data collected by each FPGA board can be transmitted to the corresponding GPU device through the PCIe bus between the FPGA board and its corresponding GPU device according to the correspondence between the FPGA board and the GPU device.

[0053] In order for the FPGA board and GPU device to access each other's memory, appropriate memory mapping and address management must be set up to ensure that the two devices can access data correctly and handle memory access rights and consistency issues. On the GPU side, CUDA is used to allocate video memory, and on the FPGA side, base address registers are used to map the FPGA memory space. The specific implementation method is as follows:

[0054] On the FPGA side, PCIe BAR (Base Address Register) is used to map the internal memory space of FPGA. The size and type of PCIe BAR are specified in the FPGA design. A DMA controller is integrated in the FPGA to control the direct data transfer from FPGA to GPU.

[0055] On the GPU side, use the CUDA programming tool to allocate GPU memory. CUDA provides the cudaMalloc() function to allocate memory, and obtains the attributes of the device pointer and the physical address information through the cudaPointerGetAttributes() function. Use CUDA's GPUDirect function to set DMA control and directly access the FPGA's memory.

[0056] In the embodiment of the present application, a circular buffer pool is set on the GPU side to improve data throughput, so that data transmission and data processing can be carried out in parallel, reducing waiting time. The buffer pool is pre-allocated, and the circular buffer pool is composed of multiple rows of buffers, each row of buffers is a continuous video memory area. The GPU device and the FPGA board use the circular buffer pool in a certain order, and set a status flag for each buffer, such as the current write status, the current read status, the idle status, etc., to coordinate the synchronous access of the GPU device and the FPGA board to the buffer.

[0057] Specifically, when the FPGA board writes data into the circular buffer pool, it first needs to select the first buffer in the idle state from the receiving circular buffer pool, write the data into the first buffer, and convert the state flag corresponding to the first buffer from the idle state to the current write state. When the first buffer reaches the capacity limit, that is, it is full, the state flag corresponding to the next buffer of the first buffer is marked as idle, and the unfinished data is written into the next buffer until the data is written.

[0058] S102: Calculate the data through the GPU device to obtain corresponding data processing results.

[0059] The GPU device is responsible for parallel processing and calculation of data. On the GPU side, it is determined whether the number of buffers in the receiving circular buffer pool that are not in an idle state is greater than 0. If it is 0, it means that there is no unprocessed data in the circular buffer pool at this time. If it is not 0, it is necessary to filter out the specified buffers whose status flags are the current read state from the receiving circular buffer pool, and start the GPU Kernel to read data from the first specified buffer in the specified buffers, calculate the data, and obtain the corresponding data processing results. After completing the data processing of the first specified buffer, the status flag corresponding to the first specified buffer needs to be converted from the current read state to the idle state, so that the memory of the first specified buffer can be released. When new data arrives at the receiving circular buffer pool, the new data can be stored in the space buffer through the FPGA board side. At the same time, the status flag corresponding to the next specified buffer of the first specified buffer needs to be marked as the current read state, so as to realize the cyclic calculation of the next specified buffer. Figure 4 As shown, the receiving circular buffer pool and the result circular buffer pool are composed of multiple buffers, and the multiple buffers are distributed in a ring shape. The buffers with different status flags can realize the circular reading and writing of data, thereby realizing the effective utilization of memory resources.

[0060] It should be noted that after obtaining the data processing results, the storage and transmission of the data processing results also need to adopt the dynamic buffer management method as described above, that is, a result circular buffer pool is used by the GPU side device to store the data processing results. In this way, after calculating the data processing results of a part of the data, a second buffer in an idle state will be screened out from the result circular buffer pool, and the data processing results will be written into the second buffer.

[0061] S103: Transmitting the data processing result to the backend through the PCIe bus between the GPU device and the network card.

[0062] like Figure 2 and Figure 3 As shown in the figure, the PCIe bus can be used to directly transmit data between the GPU device and the network card. The two GPUs correspond to the dual interfaces of a network card one by one, and the data processing results of each GPU device are directly transmitted to the network card through GPUDirect. Through RDMA technology, the network card can directly access the GPU video memory, establish a memory mapping table between the memory address space of the network card and the video memory address space of the GPU device, so that the network card and the GPU can correctly access each other's memory area, obtain the data processing results from the result circular buffer pool through the memory mapping table and transmit them to the backend, reduce the data handling involved by the CPU, and improve the transmission efficiency.

[0063] It should be noted that if Figure 5A schematic diagram of the data flow of a GPU video memory is shown. The dynamic buffer management method proposed in the embodiment of the present application can be applied not only to the receiving circular buffer pool between the FPGA and the GPU, but also to the result circular buffer pool on the network card and the GPU side. The network card side can directly obtain the corresponding data processing result by reading the result circular buffer pool in the video memory, and then send the data processing result to the back end according to the network protocol.

[0064] S104: Determine the type of device that communicates with the NVMe SSD according to the type of data that the NVMe SSD needs to store, and store the data or the data processing result in the NVMe SSD through the PCIe bus between the NVMe SSD and the device type.

[0065] NVMe SSD is responsible for storing raw data to the hard disk, determining the device type that communicates with the NVMe SSD based on the type of data that the NVMe SSD needs to store, and storing the data or data processing results to the NVMe SSD through the PCIe bus between the NVMe SSD and the device type. The data type includes raw data and data processing results, and the device type refers to GPU devices and FPGA boards.

[0066] like Figure 2 and Figure 3 As shown, the NVMe SSD and FPGA board are interconnected through the PCIe bus, and there is a one-to-one correspondence between the two FPGAs and the two NVMe SSDs. If the original data needs to be stored, the device type that communicates with the NVMe SSD is determined to be an FPGA board. At this time, the original data collected by each FPGA board needs to be directly transferred to the corresponding NVMe SSD through PCIe P2P. If the data processing results need to be stored, the device type that communicates with the NVMe SSD is determined to be a GPU device. At this time, the data processing results need to be directly stored in the SSD hard disk through PCIe.

[0067] Compared with the traditional system architecture in which the transmission and reception of high-speed data require a large number of CPU cores to complete, in the embodiment of the present application, the CPU does not participate in data transportation and only makes system calls, which also greatly reduces the energy consumption of the CPU. Figure 6 A control diagram of a CPU and a PCIe Switch server is provided for an embodiment of the present application, wherein the tasks of each PCIe Switch server, namely, the processing tasks performed by the FPGA board, GPU device, NVMe SSD and network card are all controlled by a corresponding CPU core.

[0068] Device Embodiment

[0069] like Figure 7 As shown, an embodiment of the present application provides an electronic device, including:

[0070] processor 72; and,

[0071] The memory 70 is arranged to store computer executable instructions, which, when executed, enable the processor 72 to implement any of the steps of PCIe P2P-based data processing as described above.

[0072] Medium Embodiment

[0073] An embodiment of the present application provides a storage medium for storing computer-executable instructions, wherein the computer-executable instructions implement the steps of any of the above-mentioned PCIe P2P-based data processing methods when executed.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on PCIe P2P, characterized in that: Applied to a PCIe Switch server, the PCIe Switch server includes an FPGA board, a GPU device, an NVMe SSD, and a network card, and the method includes: The FPGA board collects data, and according to the corresponding relationship between the FPGA board and the GPU device, transmits the data to the GPU device through the PCIe bus between the FPGA board and its corresponding GPU device, and the data can be directly read and written into the other device memory through the DMA controller; Based on the FPGA board, a first buffer in an idle state is screened out from a receiving circular buffer pool, the data is written into the first buffer, and a state flag bit corresponding to the first buffer is converted from the idle state to a current write state; When the capacity limit of the first buffer is reached, the state flag bit corresponding to the next buffer of the first buffer is marked as an idle state, and the unfinished written data is written into the next buffer until the writing of the data is completed; The data is calculated by the GPU device to obtain corresponding data processing results; specifically: Based on the GPU device, determining whether the number of buffers in the receiving circular buffer pool that are not in an idle state is greater than 0; If yes, filter out the designated buffer whose status flag is the current read status from the receiving circular buffer pool, read the data from the first designated buffer in the designated buffers, calculate the data, and obtain the corresponding data processing result; Convert the state flag bit corresponding to the first designated buffer from the current read state to the idle state to realize the release of the first designated buffer, and mark the state flag bit corresponding to the next designated buffer of the first designated buffer as the current read state to realize the cyclic calculation of the next designated buffer; Transmitting the data processing result to a backend via a PCIe bus between the GPU device and the network card; Determine, according to the type of data to be stored by the NVMe SSD, a device type for communicating with the NVMe SSD, and store the data or the data processing result in the NVMe SSD through a PCIe bus between the NVMe SSD and the device type; The processing tasks performed by the FPGA board, the GPU device, the NVMe SSD and the network card are all controlled by their corresponding CPU cores.

2. The PCIe P2P-based data processing method according to claim 1, characterized in that: After obtaining the corresponding data processing result, the method further includes: Based on the GPU device, a second buffer in an idle state is screened out from a result circular buffer pool, and the data processing result is written into the second buffer.

3. A PCIe P2P-based data processing method according to claim 2, characterized in that: Transmitting the data processing result to the backend through the PCIe bus between the GPU device and the network card, specifically including: Based on the PCIe bus between the GPU device and the network card, a memory mapping table between the memory address space of the network card and the video memory address space of the GPU device is established through the RDMA technology, so that the network card obtains the data processing result from the result circular buffer pool through the memory mapping table, and transmits the data processing result to the back end.

4. The PCIe P2P-based data processing method according to claim 1, characterized in that: According to the type of data that the NVMe SSD needs to store, determining the type of device that communicates with the NVMe SSD specifically includes: Determine the data type that the NVMe SSD needs to store; wherein the data type includes original data and data processing results; When the data type is the original data, the device type communicating with the NVMe SSD is the FPGA board; When the data type is the data processing result, the device type communicating with the NVMe SSD is the GPU device.

5. The PCIe P2P-based data processing method according to claim 2, characterized in that: The receiving circular buffer pool and the result circular buffer pool are composed of a plurality of buffers, and the plurality of buffers are distributed in a ring shape.

6. An electronic device, characterized in that: include: processor; as well as, A memory arranged to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the steps of PCIe P2P-based data processing according to any one of claims 1 to 5.

7. A storage medium, characterized in that: Used to store computer executable instructions, which, when executed, implement the steps of the PCIe P2P-based data processing method according to any one of claims 1 to 5.

Images