RDMA communication-oriented Linux kernel driver design method
By designing a Linux kernel driver for RDMA, using GPUDirect RDMA technology to realize direct communication between GPU and FPGA, the CPU-mediated communication delay problem in traditional methods is solved, and efficient data exchange of heterogeneous computing platform is achieved, reaching 32GB/ The transmission speed of s.
Patent Information
- Application Number
- CN202411165297.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
AI Technical Summary
In heterogeneous computing platforms, traditional driver design methods cause communication between GPU and FPGA to be required through CPU memory, which increases transmission delay and limits the improvement of computing power.
Design a Linux kernel driver for RDMA, using GPUDirect RDMA technology, realizes direct data exchange between GPU and FPGA through PCIe bus, bypasses the CPU, adopts DMA transmission and mapping mechanism of virtual address to physical address, and generates character device handles. , and initiates the transmission through the DMA descriptor register.
It realizes direct communication between the GPU and the FPGA, significantly reduces transmission delay, accelerates data exchange speed, and the theoretical transmission speed reaches 32GB/s, meeting the computing speed requirements of heterogeneous platforms.
Smart Images

Figure CN119065846A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Linux kernel driver design, and provides a Linux kernel driver design method for RDMA (Remote Direct Memory Access) communication, so that a GPU and a third-party device can realize direct communication based on a PCIe (Peripheral Component Interconnect express) bus without the participation of a CPU (Central Processing Unit). Background Art
[0002] With the continuous development of computer technology, computing power has become a key productivity in modern society. Especially in the fields of artificial intelligence, Internet of Things, virtual reality, etc., the demand for efficient computing power is more urgent. In this context, heterogeneous computing technology that combines multiple different types of processors or computing devices to perform computing tasks has gradually emerged. Compared with traditional single-type computing, heterogeneous computing can give full play to the capabilities of different computing resources according to the characteristics of computing tasks, and improve server computing performance, energy efficiency and real-time performance. However, the communication bandwidth and communication delay between different computing units limit the improvement of the computing power of heterogeneous platforms.
[0003] Computing tasks have two characteristics: simple but large amount of calculation, complex but small amount of calculation. The heterogeneous parallel computing platform composed of the lower computer FPGA (Field Programmable Gate Array) and the upper computer CPU is responsible for calculating simple and large amount of tasks and complex and small amount of tasks respectively, which can make full use of computing resources. Since computing fields such as deep learning usually require a lot of computing resources for training, the parallel performance of GPU (Graphics Processing Unit) makes it a rational choice for training large neural networks. Therefore, it is possible to consider replacing CPU with GPU and FPGA to form a heterogeneous parallel computing platform. If the traditional driver design method is used to achieve communication between the two, the data must pass through the CPU memory, which will increase the transmission delay. A technology is urgently needed to solve the above problems.
[0004] However, GPUDirect RDMA, a technology developed by NVIDIA, can bypass the remote host when the GPU communicates with third-party devices and directly transfer data between other devices (such as network cards and storage devices), thereby significantly reducing transmission delays and speeding up data exchange. Therefore, the present invention develops a corresponding Linux kernel driver based on RDMA technology. Summary of the invention
[0005] In order to meet the communication requirements of GPU and FPGA in heterogeneous computing platforms, the present invention provides a Linux kernel driver design method for molecular dynamics calculation. Different from traditional drivers, it can realize direct data exchange between GPU and FPGA under PCIe bus, bypassing CPU, and meeting the computing speed requirements of heterogeneous platforms.
[0006] The present invention adopts the following technical solution:
[0007] Step S1, similar to general device drivers, the RDMA driver sets the Device ID and Vendor ID corresponding to the FPGA, mounts it on the FPGA, and generates a character device handle that can be operated by the program end. After the driver is loaded into the lower computer FPGA, there are two key steps. The first is to map the PCI BAR (Base Address Register), that is, the I / O space or memory space of the device, to the specified area in the kernel address space, so as to facilitate the kernel to access and control the device; the second is to map the physical address required for DMA operation with the virtual address used by the kernel. These processes all occur when the driver is loaded into the FPGA, and these mappings will be released once the driver is unloaded.
[0008] Step S2, on the host computer GPU side, allocate a memory space to store the data to be transmitted, and then pass the parameters of this memory space including the virtual address and length to the driver. After receiving it, the driver fixes this memory and converts its virtual address to a physical address. The program side uses the character device handle generated in step S1 to open it in the form of a file, and then executes the subsequent steps. The driver uses the API provided by the GPUDirect RDMA technology to fix the memory allocated by the GPU and convert its virtual address into a physical page list.
[0009] Step S3, DMA transmission requires the physical addresses of the host and slave computers. The driver maps the physical page list converted in step S2 to a physical address, which is the physical address of the GPU. The FPGA physical address can be directly read through the BRAM memory address in the custom VIVADO project and directly filled into the driver through the host computer program end.
[0010] Step S41, the host computer uses the character device handle of step S1 to actively initiate DMA transmission, and the kernel driver executes DMA transmission after receiving the correct instruction. First, during DMA transmission, the source address, destination address and transmission data length need to be passed into the DMA descriptor register, and then the driver needs to fill the parameters such as the GPU physical address of step S2, the FPGA physical address and the data length determined by the host computer into the DMA descriptor register.
[0011] Step S42, the driver initializes key parameters such as the control register descriptor and the interrupt register descriptor, and then the driver maps the physical address that the DMA needs to operate to a virtual address that can be used by the kernel through the key step 2 in step S1. After preparing the above conditions, the driver initiates the transmission by writing the control register. After the data transmission is completed, the device will generate an interrupt signal and reset the corresponding bit of the interrupt register to inform the driver that the transmission has been completed. The driver will read the interrupt register, reset it, and complete the DMA transmission by writing the control register.
[0012] After the transmission is completed, the communication speed needs to be evaluated. This design framework adopts the PCIe 4.0 architecture, with a theoretical data transmission speed of 32GB / s. Its communication speed is measured by the following formula (1):
[0013]
[0014] Where BW is the communication speed, size is the transmission data size, and time is the transmission time. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a schematic diagram of the structure of the PCIe bus of the present invention;
[0016] Figure 2 is an operation flow chart of an embodiment of the present invention;
[0017] Figure 3 is a flow chart of the DMA operation mechanism in an embodiment of the present invention;
[0018] Figure 4 is a graph showing the relationship between the communication bandwidth and the amount of transmitted data obtained by actual measurement in an embodiment of the present invention; DETAILED DESCRIPTION
[0019] The following will be further described in detail with reference to the accompanying drawings and specific embodiments.
[0020] The present invention provides a Linux kernel driver design method for RDMA communication. While ensuring correct communication, shorter communication delay and the most direct and simple communication method between heterogeneous components are achieved. To ensure the normal operation of the driver, both hardware and software preparations need to be performed. In terms of hardware, components such as CPU, GPU and FPGA need to be configured as follows: Figure 1 The GPU and FPGA are mounted on the same PCIe Switch of the Root Complex, so that they can communicate directly without the CPU memory. In terms of software, the VIVADO software needs to be used to burn a project containing certain internal logic on the corresponding FPGA. Figure 2 The implementation steps are described in detail by taking the communication process of GPU writing data into FPGA's BRAM (Block Random Access Memory) as an example.
[0021] Step S1, similar to general device drivers, the RDMA driver sets the Device ID and Vendor ID corresponding to the FPGA, mounts it on the FPGA, and generates a character device handle that can be operated by the program end. After the driver is loaded into the lower computer FPGA, there are two key steps. The first is to map the PCI BAR (Base Address Register), that is, the I / O space or memory space of the device, to the specified area in the kernel address space, so that the kernel can access and control the device; the second is to map the physical address required for DMA operation with the virtual address used by the kernel. These processes all occur when the driver is loaded into the FPGA, and these mappings will be released once the driver is unloaded.
[0022] Step S2, on the host computer GPU side, allocate a memory space to store the data to be transmitted, and then pass the parameters of this memory space including the virtual address and length to the driver. After receiving it, the driver fixes this memory and converts its virtual address to a physical address. The program side uses the character device handle generated in step S1 to open it in the form of a file, and then executes the subsequent steps. The driver uses the API provided by the GPUDirect RDMA technology to fix the memory allocated by the GPU and convert its virtual address into a physical page list.
[0023] Step S3, DMA transmission requires the physical addresses of the host and slave computers. The driver maps the physical page list converted in step S2 to a physical address, which is the physical address of the GPU. The FPGA physical address can be directly read through the BRAM memory address in the custom VIVADO project and directly filled into the driver through the host computer program end.
[0024] Step S41: The host computer uses the character device handle of step S1 to actively initiate DMA transfer. After the kernel driver receives the correct instruction, it executes DMA transfer. Figure 3 As shown, first, during DMA transmission, the source address, destination address and transmission data length need to be passed into the DMA descriptor register, and then the driver needs to fill in the parameters such as the GPU physical address of step S2, the FPGA physical address and the data length determined by the host computer into the DMA descriptor register.
[0025] Step S42, the driver initializes key parameters such as the control register descriptor and the interrupt register descriptor. The driver internally maps the physical address that the DMA needs to operate to a virtual address that can be used by the kernel through the key step 2 in step S1. After preparing the above conditions, the driver initiates the transmission by writing the control register. After the data transmission is completed, the device generates an interrupt signal and sets the corresponding bit of the interrupt register to inform the driver that the transmission has been completed. The driver will read the interrupt register and reset it, and complete the DMA transmission by writing the control register.
[0026] After the transmission is completed, the communication speed needs to be evaluated. This design framework adopts the PCIe4.0 architecture, and the theoretical data transmission speed is 32GB / s. The communication speed of this design is measured by the following formula (2):
[0027]
[0028] Where BW is the communication speed, size is the length of the transmitted data, and time is the transmission time. Figure 4 As shown in FIG. 1 , the transmission bandwidth increases with the increase in the number of transmitted bytes. When it reaches about 25 GB / s, the transmission bandwidth tends to be stable, that is, the maximum actual transmission bandwidth is reached.
Claims
1. A Linux kernel driver design method for RDMA communication, characterized in that: The following steps are involved: Step S1, mount the FPGA and GPU on the same PCIe Switch and load the driver; Step S2, the host computer allocates a section of memory to the GPU and then passes its parameters to the driver and performs mapping; Step S3, preparing for DMA transmission; Step S4, the host computer starts DMA transmission, and after a series of operations, performs read and write communication with the slave computer FPGA; Finally, after completing the above steps, the communication speed is evaluated.
2. A Linux kernel driver design method for RDMA communication as claimed in claim 1, characterized in that: In step S1, the FPGA and GPU are mounted on the same PCIe Switch, and the driver is loaded to perform mapping.
3. A Linux kernel driver design method for RDMA communication as claimed in claim 1, characterized in that: In step S2, the host computer allocates a section of memory to the GPU for storing transmission data, and transmits its virtual address to the driver, mapping the virtual address to a physical address.
4. A Linux kernel driver design method for RDMA communication as claimed in claim 1, characterized in that: In step S3, the physical addresses of the upper and lower computer hardware required for DMA transmission are prepared.
5. A Linux kernel driver design method for RDMA communication as claimed in claim 1, characterized in that: In step S4, the host computer starts a DMA transfer instruction, and the driver receives the instruction and executes the DMA transfer, thereby realizing the reading and writing operations of the GPU to the FPGA.
6. A Linux kernel driver design method for RDMA communication as claimed in claim 1, characterized in that: After the DMA transfer is completed, the communication speed of the communication process is evaluated in real time.
Citation Information
Cited By
Method and device for quickly accessing memory between PCIe devices and medium
CN119557258A
