Transmission method for realizing RDMA completely based on FPGA

Abstract

The invention discloses a transmission method for realizing RDMA (Remote Direct Memory Access) completely based on an FPGA (Field Programmable Gate Array), which comprises the following steps of: mounting a control plane service of an RDMA main node on the FPGA, carrying management and control of the RDMA main node by a MicroBlaze IP (Intellectual Property) core in the FPGA, and replacing a CPU (Central Processing Unit) by using the MicroBlaze IP core as a controller of the RDMA main node; and meanwhile, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU. By means of the design architecture, the RDMA main node and the CPU can be completely decoupled, the CPU part is omitted, device type selection and performance indexes of the CPU do not need to be considered any more, meanwhile, a high-speed data transmission link is optimized, the data transmission efficiency is improved, and possible middle fault points are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of high-speed data communication, and in particular to a transmission method for implementing RDMA completely based on FPGA. Background Art

[0002] With the rapid development of high-speed data communication, the requirements for transmission links are becoming higher and higher. Among them, RDMA, namely remote direct memory access, is a high-performance network communication technology with the advantages of high bandwidth, low latency, no CPU consumption, and zero copy. Figure 3 The figure shows a schematic diagram of the transmission structure of the mainstream RDMA in the industry. During the data transmission process, the protocol conversion must first be performed via the FPGA. After that, the data after the protocol conversion is transmitted to the CPU through the PCIE interface and stored in the DDR4 of the RDMA master node CPU. As the controller of RDMA, the CPU will receive the RDMA transmission instruction WQE and initiate RDMA transmission to transfer data from the RDMA master node to the RDMA slave node, where WQE refers to the completion queue element. In this transmission system, the FPGA only assumes the function of the network card to realize the protocol conversion; the data transmission is essentially from the DDR4 of the RDMA master node CPU to the DDR4 of the RDMA slave node CPU, and the entire RDMA control is the responsibility of the CPU.

[0003] However, the measures currently taken contain at least the following three problems: 1) High cost: This is because the transmission process requires many processes such as protocol conversion, PCIE transmission, and DDR writing. The data transmission link is too long and the overhead is too high to meet the growing data demand. At the same time, the solution must involve the CPU and has rigid requirements for the CPU's PCIE resources. 2) Inefficiency: This is because there are many processes involved in the transmission process, which leads to long data transmission links, data transmission delays, and intermediate failure points; 3) Poor adaptability: For the design of slave nodes, DDR can only be mounted under the CPU, and there is transmission loss between the CPU and FPGA, so the device selection and performance indicators of the CPU need to be considered. Summary of the invention

[0004] In view of the above three problems, the purpose of the present invention is to propose a transmission method for implementing RDMA completely based on FPGA. By mounting the control plane service of the RDMA master node on the FPGA, the management and control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA, and the MicroBlaze IP core is used instead of the CPU as the controller of the RDMA master node, the problems of long data transmission link, high overhead and low efficiency in the traditional RDMA transmission method are overcome; at the same time, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU, so that the method has more flexible scenario applicability.

[0005] This is achieved through the following technical solutions: A transmission method for implementing RDMA completely based on FPGA, wherein the control plane service of the RDMA master node is mounted on the FPGA, the control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA, and the MicroBlaze IP core is used to replace the CPU as the controller of the RDMA master node; at the same time, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU, and the data transmission process of the method includes the following steps: S1, the RDMA master node obtains front-end data through network transmission, and stores the front-end data in the DDR4 of the FPGA; at the same time, the MicroBlaze in the FPGA The IP core acts as the controller of the RDMA master node. When the controller of the RDMA master node receives the transmission instruction WQE sent by the RDMA master node, the RDMA-core is used to parse the WQE instruction and obtain the parsing result; S2, according to the parsing result of the WQE instruction, the configuration elements in the WQE instruction are encapsulated into a data packet and sent to the RDMA slave node. At the same time, the controllers of the RDMA master node and the RDMA slave node interact. The RDMA master node tells the RDMA slave node the DDR4 starting address, data length and data transmission mode of the data to be sent by sending a data packet; after the RDMA slave node receives the data packet sent by the RDMA master node, it parses the data packet and obtains the DDR4 starting address, data length and data transmission mode of the data stored in the RDMA master node according to the parsing result. At the same time, according to the parsing result of the WQE instruction, the RDMA-core initiates RDMA transmission, starts data transmission, and transfers the data from the master node memory to the slave node memory.

[0006] The present invention uses MicroBlaze IP core instead of CPU as the controller of RDMA master node, omits CPU part, optimizes high-speed data transmission link and control mode, reduces transmission path and intermediate nodes, and realizes high-speed lossless transmission of data in DDR4 of FPGA of master node and slave node.

[0007] Preferably, the control plane service of the RDMA master node is mounted on the FPGA to achieve complete decoupling of the RDMA master node and the CPU, wherein complete decoupling is to optimize the transmission link and control mode by reducing the dependency between the RDMA master node and the CPU. By completely decoupling the RDMA master node from the CPU, the transmission loss between the CPU and the FPGA is eliminated, and there is no need to consider the device selection and performance indicators of the CPU.

[0008] Preferably, when the RDMA master node communicates with the RDMA slave node, the default data transmission mode is that data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA via the 100GE network; and when the data channel between the CPU and the FPGA of the RDMA slave node is opened, a DMA operation is initiated, and data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node CPU via the 100GE network. Through the design of the RDMA slave node, the DDR4 can be mounted under the FPGA or the CPU, breaking through the design concept of the traditional RDMA transmission method, so that the RDMA transmission method of the present invention can be applied to more application scenarios.

[0009] Preferably, a DMA device is built into RDMA, and the DMA device is used to perform DMA operations, transfer data from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA, and use the DMA controller to manage and control the data transmission process; when data is transmitted, the DMA device sends a DMA operation request signal to the CPU through the DMA controller, obtains bus control, and performs DMA operations through the bus control to complete data transmission; wherein the DMA controller includes an address bus, a data bus and a control register. Through DMA operations, not only can data be transmitted from the DDR4 of the master node FPGA to the DDR4 of the slave node FPGA via the 100GE network, but data can also be transmitted from the DDR4 of the master node FPGA to the DDR4 of the slave node CPU via the 100GE network, thereby achieving efficient data transmission and improving the overall performance and efficiency of the RDMA transmission method.

[0010] Preferably, in step S1, the WQE instruction is used to describe the work request in the RDMA communication, including remote read, remote write, data initiation and data reception, and also includes the memory address where the data is located, the data length and the access key. During the RDMA transmission process, the WQE instruction is used to tell the RDMA controller the operation to be performed and the related parameter information, ensuring the efficient transmission of data.

[0011] Preferably, in step S2, the configuration element at least includes: a transmission mode, a memory address, and a length of data to be operated. By sending the configuration element to the RDMA slave node, the RDMA slave node obtains the storage address, transmission mode, and data length of the data, thereby ensuring the reliability and accuracy of data transmission.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The technical solution of the present invention mounts the control plane service of the RDMA master node on the FPGA, and the control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA. The MicroBlaze IP core is used instead of the CPU as the controller of the RDMA master node, so that the link of high-speed data transmission can be optimized, the efficiency of data transmission is improved, and possible intermediate failure points are reduced; at the same time, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU, so that the RDMA transmission method of the present invention can be applied to more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of a structure for implementing RDMA transmission completely based on FPGA; Figure 2 A schematic diagram of a derivative design structure that implements RDMA transmission completely based on FPGA; Figure 3 A schematic diagram of the structure of a traditional RDMA transmission. DETAILED DESCRIPTION

[0014] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0015] like Figure 1As shown, it is a structural diagram of RDMA transmission completely based on FPGA. In the RDMA master node, the control plane service of the RDMA master node is mounted on the FPGA, and the control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA. The MicroBlaze IP core is used to replace the CPU as the controller of the RDMA master node; wherein, the MicroBlaze IP core in the FPGA of the RDMA master node is mainly used to initiate RDMA transmission in the method of the present invention, and interact with the CPU of the RDMA slave node, thereby eliminating the transmission loss between the CPU and the FPGA, and no longer needing to consider the device selection and performance indicators of the CPU, and eliminating the entire link of the intermediate node becomes more concise. The efficiency of the MicroBlaze IP core in initiating RDMA transmission is also significantly higher than that of the PCIe interface, which can significantly reduce the transmission delay caused by the control plane interaction. For the design of the RDMA slave node, DDR4 can be mounted under the FPGA or the CPU according to different application scenarios, so that the RDMA transmission method of the present invention can be applied to more application scenarios.

[0016] The method specifically comprises the following steps: S1, RDMA master node obtains front-end data through network transmission and stores the front-end data in FPGA's DDR4; at the same time, the MicroBlaze IP core in FPGA is used as the controller of RDMA master node. When the controller of RDMA master node receives the transmission instruction WQE sent by RDMA master node, RDMA-core is used to parse WQE instruction and obtain the parsing result. RDMA-core is an open source project that provides user space libraries and tools for RDMA communication. Its goal is to simplify and unify the development and use of RDMA, so that developers can use RDMA technology more easily.

[0017] In this embodiment, in step S1, the MicroBlaze IP core is a soft-core processor that can be embedded in an FPGA, has high processing performance, and can realize efficient transmission of data from the RDMA master node to the RDMA slave node. In the present invention, the control plane service of the RDMA master node is mounted on the FPGA, which realizes the complete decoupling of the RDMA master node and the CPU, and uses the MicroBlaze IP core to replace the CPU as the controller of the RDMA master node, wherein the complete decoupling is to optimize the transmission link and control mode by reducing the dependency between the RDMA master node and the CPU. By completely decoupling the RDMA master node from the CPU, the transmission loss between the CPU and the FPGA is eliminated, and there is no need to consider the device selection and performance indicators of the CPU, and the use of the MicroBlaze IP core as the controller of the RDMA master node, the data transmission efficiency is significantly improved.

[0018] In this embodiment, in step S1, the WQE instruction is used to describe the work request in the RDMA communication, including remote read, remote write, data initiation and data reception, and also includes the memory address where the data is located, the data length and the access key. During the RDMA transmission process, the WQE instruction is used to tell the RDMA controller the operation to be performed and the related parameter information, ensuring the efficient transmission of data.

[0019] S2. According to the parsing result of the WQE instruction, the configuration elements in the WQE instruction are encapsulated into a data packet and sent to the RDMA slave node. At the same time, the controllers of the RDMA master node and the RDMA slave node interact. The RDMA master node tells the RDMA slave node the DDR4 start address, data length and data transmission mode of the data to be sent by sending a data packet. After receiving the data packet sent by the RDMA master node, the RDMA slave node parses the data packet and obtains the DDR4 start address, data length and data transmission mode of the data stored in the RDMA master node according to the parsing result. At the same time, according to the parsing result of the WQE instruction, the RDMA-core initiates RDMA transmission, starts data transmission, and transfers the data from the master node memory to the slave node memory. Among them, the configuration elements at least include: transmission mode, memory address and data length to be operated. By sending the configuration elements to the RDMA slave node, the RDMA slave node obtains the storage address, transmission mode and data length of the data, ensuring the reliability and accuracy of data transmission.

[0020] In this embodiment, in step S2, when the RDMA master node communicates with the RDMA slave node, the default data transmission mode is that data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA via the 100GE network; and when the data channel between the CPU and the FPGA of the RDMA slave node is opened, a DMA operation is initiated, and data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node CPU via the 100GE network; wherein, a DMA device is built into the RDMA, and the DMA device is used to perform DMA operations, transfer data from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA, and use a DMA controller to manage and control the data transmission process; when data transmission is performed, the DMA device sends a DMA operation request signal to the CPU through the DMA controller, obtains bus control, and performs DMA operations through the bus control to complete data transmission; wherein the DMA controller includes an address bus, a data bus and a control register. Through the design of the RDMA slave node, DDR4 can be mounted under the FPGA or the CPU, breaking through the design concept of the traditional RDMA transmission method, so that the RDMA transmission method of the present invention can be applicable to more application scenarios; and through DMA operation, not only can data be transmitted from the DDR4 of the master node FPGA to the DDR4 of the slave node FPGA via the 100GE network, but also data can be transmitted from the DDR4 of the master node FPGA to the DDR4 of the slave node CPU via the 100GE network, thereby realizing efficient data transmission and improving the overall performance and efficiency of the RDMA transmission method.

[0021] Based on the RDMA transmission method of the present invention, derivative designs can be made on the basis of innovation. Since the CPU limitation is eliminated, the power consumption and cost of RDMA can be further optimized. Based on the same thermal design capability, a single board can carry more RDMA networks. In the future, multiple master devices can be deployed on the RDMA master node, and RDMA data transmission can be initiated to multiple RDMA slave nodes at the same time.

[0022] like Figure 2As shown in the figure, it is a schematic diagram of a derivative design structure that implements RDMA transmission completely based on FPGA. Multiple FPGA devices are deployed in the RDMA master node, and the control plane business of the RDMA master node is mounted on each FPGA. The control of the RDMA master node is carried by the MicroBlaze IP core in each FPGA. Each MicroBlaze IP core is used instead of the CPU as multiple controllers of the RDMA master node, saving multiple CPUs; and for the design of each RDMA slave node, DDR4 can be mounted under the FPGA or the CPU. When performing RDMA transmission, the RDMA master node can initiate data transmission to multiple RDMA slave nodes at the same time, and the data transmission from each RDMA master node to the RDMA slave node is carried out according to the step sequence t1, t2, t3, t4 and t5 in the schematic diagram. The derivative design of the RDMA transmission method gathers massive data at the RDMA master node for shunting processing, which can realize the pipeline design of high-speed links and produce the maximum performance benefits with the minimum investment.

[0023] In summary, the present invention mounts the control plane service of the RDMA master node on the FPGA, and the control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA. The MicroBlaze IP core is used instead of the CPU as the controller of the RDMA master node, so that the link of high-speed data transmission can be optimized, the efficiency of data transmission is improved, and possible intermediate failure points are reduced; at the same time, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU, so that the RDMA transmission method of the present invention can be applied to more application scenarios, which is significantly progressive.

[0024] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A transmission method for implementing RDMA completely based on FPGA, characterized in that: The control plane service of the RDMA master node is mounted on the FPGA. The control of the RDMA master node is carried by the MicroBlaze IP core in the FPGA. The MicroBlaze IP core is used instead of the CPU as the controller of the RDMA master node. At the same time, the DDR4 of the RDMA slave node is mounted under the FPGA or the CPU. The data transmission process of the method includes the following steps: S1, the RDMA master node obtains the front-end data through network transmission and stores the front-end data in the DDR4 of the FPGA; at the same time, the MicroBlaze IP core in the FPGA is used as the controller of the RDMA master node. When the controller of the RDMA master node receives the transmission instruction WQE sent by the RDMA master node, the RDMA-core is used to parse the WQE instruction and obtain the parsing result; S2. According to the parsing result of the WQE instruction, the configuration elements in the WQE instruction are encapsulated into a data packet and sent to the RDMA slave node. At the same time, the controllers of the RDMA master node and the RDMA slave node interact. The RDMA master node tells the RDMA slave node the DDR4 start address, data length and data transmission mode of the data to be sent by sending a data packet. After receiving the data packet sent by the RDMA master node, the RDMA slave node parses the data packet and obtains the DDR4 start address, data length and data transmission mode of the data stored in the RDMA master node according to the parsing result. At the same time, according to the parsing result of the WQE instruction, the RDMA-core initiates RDMA transmission and starts data transmission, transferring the data from the master node memory to the slave node memory.

2. According to the transmission method of implementing RDMA completely based on FPGA in claim 1, it is characterized in that: The control plane service of the RDMA master node is mounted on the FPGA to achieve complete decoupling of the RDMA master node and the CPU. The complete decoupling is to optimize the transmission link and control mode by reducing the dependency between the RDMA master node and the CPU.

3. According to the transmission method of implementing RDMA completely based on FPGA in claim 1, it is characterized in that: When the RDMA master node communicates with the RDMA slave node, the default data transmission mode is that data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA via the 100GE network; And when the data channel between the CPU and FPGA of the RDMA slave node is opened, a DMA operation is initiated, and the data is transmitted from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node CPU via the 100GE network.

4. According to claim 3, a transmission method for implementing RDMA completely based on FPGA is characterized in that: RDMA has a built-in DMA device, which is used to perform DMA operations, transfer data from the DDR4 of the RDMA master node FPGA to the DDR4 of the RDMA slave node FPGA, and use the DMA controller to manage and control the data transmission process; when data is transmitted, the DMA device sends a DMA operation request signal to the CPU through the DMA controller, obtains bus control, and performs DMA operations through the bus control to complete data transmission; the DMA controller includes an address bus, a data bus and a control register.

5. The transmission method for implementing RDMA completely based on FPGA according to claim 1, characterized in that: In step S1, the WQE instruction is used to describe the work request in the RDMA communication, including remote reading, remote writing, initiating data and receiving data, and also includes the memory address where the data is located, the data length and the access key.

6. The transmission method for implementing RDMA completely based on FPGA according to claim 1, characterized in that: In step S2, the configuration elements include at least: a transmission mode, a memory address and a length of data to be operated.

Images