Control plane data kernel bypass system for RDMA network cards

Abstract

The present invention discloses a control layer data kernel bypass system for RDMA network card, comprising: a kernel state software module, a user state software module and an FPGA hardware module; the kernel state software module is used for direct interaction with the FPGA hardware module when creating and destroying resources, and is also responsible for the registration and cancellation of the control layer data memory area shared by the user state software module and the FPGA hardware module, and performs various address mappings, etc.; the user state software module is used to interact with the application through the IB_Verbs interface, and is responsible for the reading, writing and processing of the control layer data during the RDMA network card transmission process; the FPGA hardware module is used to complete the actual transmission of the RDMA network card by directly reading and writing the control layer data of the user state software module. The benefit of the present invention lies in that the control layer data kernel bypass system for RDMA network card provided avoids the switching between kernel state and user state during the RDMA network card transmission process, reduces the delay of network card transmission, and improves the network performance of the application.

Description

Technical Field

[0001] The invention belongs to the field of network technology, and in particular relates to a control layer data kernel bypass system for an RDMA network card. Background Art

[0002] RDMA (Remote Direct Memory Access) networks have become the mainstream of data center networks in recent years due to their excellent characteristics of low latency, low CPU overhead and high bandwidth. RDMA networks include Infiniband networks and RoCE networks. RDMA supports data transfer directly from the memory of one computer to the memory of another computer without the intervention of the operating systems of both parties. RDMA technology eliminates the overhead of copying and context switching of data packets in user space and kernel space, thereby freeing up memory bandwidth and CPU cycles to improve application system performance. Most RDMA networks use dedicated smart network cards to offload network services to hardware, further reducing the CPU load.

[0003] RDMA provides a set of software transmission interfaces, through which applications can send work queue elements (WorkQueue Element, referred to as WQE) and polling completion queue elements (Completion Queue Element, referred to as CQE). WQE describes the address, length and key of the data to be sent / received; CQE describes the status, sequence number and operation code of the sent / received message. Taking the send operation (Send) request as an example, the sender driver writes the address, length and key of the data to be sent into the WQE, the sender network card parses the WQE, completes the sending of the message and then generates a CQE, and the sender driver parses the CQE and reports the completion status.

[0004] The kernel bypass claimed by some RDMA network cards mostly means that the data path can bypass the kernel state, but the control path still needs to be trapped in the kernel and completed through interrupts. This method has good feasibility through the Verbs interface. However, due to the frequent kernel-user state switching and process context-interrupt context switching, the software overhead is large and the latency is high, which is not suitable for RDMA networks with high bandwidth and latency requirements. Therefore, in order to reduce kernel-user state switching and process context-interrupt context switching, save software overhead and reduce latency, it is necessary to further study the control path kernel bypass system. Summary of the invention

[0005] The present invention provides a control layer data kernel bypass system for an RDMA network card to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0006] A control layer data kernel bypass system for an RDMA network card, comprising: a kernel state software module, a user state software module and an FPGA hardware module;

[0007] The kernel state software module is set in the kernel state driver of the host RDMA network card, and is used to directly interact with the FPGA hardware module when creating and destroying resources. The kernel state software module is also used to process the registration and deregistration of the control layer data memory area shared by the user state software module and the FPGA hardware module, and to perform mapping of the kernel state virtual address space to the user state virtual address space and mapping of the kernel state virtual address space to the physical address space;

[0008] The user state software module is set in the host RDMA network card user state driver, used to interact with the application through the IB_Verbs interface, and is responsible for the reading, writing and processing of control layer data during the RDMA network card transmission process;

[0009] The FPGA hardware module is arranged in the RDMA network card, and is used to complete the actual transmission of the RDMA network card by directly reading and writing the control layer data of the user state software module.

[0010] Furthermore, the kernel state software module includes:

[0011] An address mapping unit is used to convert virtual addresses in kernel state into virtual-physical addresses, convert virtual addresses in kernel state into kernel state-user state addresses, and map hardware device addresses into virtual addresses in kernel state and user state;

[0012] The control layer data management unit is used to create and destroy the corresponding CQE and WQE memory spaces, and then call the address mapping unit to obtain the addresses required by the hardware and user-mode drivers, and pass them to the hardware and user-mode drivers.

[0013] Furthermore, the CQE and WQE memory space is a circular data buffer, adopting a first-in-first-out FIFO structure, shared by the user-state driver module and the FPGA hardware module, and each read and write will be offset by a CQE / WQE size based on the previous time, so that the user-state software module and the FPGA hardware module can accurately read and write data in the buffer. The length of the CQE buffer is determined by the length of a single CQE and the bufferable quantity, and the length of the WQE buffer is determined by the length of a single WQE and the maximum sending depth of the driver.

[0014] Furthermore, the user-mode software module includes:

[0015] The task completion unit is used to read the CQE generated by the FPGA hardware from the CQE buffer, parse it, pass the task completion information to the application, and also inform the FPGA hardware that the CQE at the corresponding position has been obtained by the application;

[0016] The task request unit is used to convert the send / receive instructions issued by the user into a WQE that can be parsed by the hardware, and fill it into the corresponding position of the WQE buffer, and then write the DoorBell to the FPGA hardware register through MMIO so that the FPGA hardware can read the WQE.

[0017] Furthermore, the FPGA hardware module includes:

[0018] A CQE processing unit, configured to generate a CQE according to the completion status after the packet sending and receiving task is completed, and fill it into a corresponding position of the CQE buffer for reading by the user-mode software module;

[0019] The WQE processing unit is used to read the WQE instruction sent by the user-mode software module from the WQE buffer after receiving the DoorBell, parse it, and complete the task of sending and receiving packets according to its content.

[0020] Further, the control layer data includes DooBell, WQE and CQE, which are accessed by the user state software module and the FPGA hardware module;

[0021] After DoorBell is processed by the address mapping unit, it is directly read and written by the user-mode software module through MMIO, so that the user-mode software module notifies the FPGA hardware module to read WQE after writing the WQE buffer;

[0022] WQE includes information such as the address, length, and key of the data to be sent / received, which is written into the corresponding position of the WQE buffer by the user-mode software module and notified to the FPGA hardware module through DoorBell. After receiving the notification, the FPGA hardware module reads the content of the corresponding position of the WQE buffer, parses it, and completes the corresponding packet sending and receiving action;

[0023] CQE includes information such as the status, serial number, operation code and valid flag of the sent / received message, which is written by the FPGA hardware module to the corresponding position of the CQE buffer. The user-mode software module that continuously polls the CQE valid flag at this position in the CQE buffer obtains CQE, reports the completion status of sending and receiving packets to the application according to the CQE content, and clears the flag in the CQE at this position so that the FPGA hardware module can smoothly write CQE in the next round.

[0024] Furthermore, the control layer data kernel bypass system for the RDMA network card is used for an RDMA network card device based on an Infiniband network or a RoCE network.

[0025] The present invention is also beneficial in that the control layer data kernel bypass system for the RDMA network card provided can reduce kernel state-user state switching and process context-interrupt context switching when using the RDMA network, thereby enabling the control layer data to achieve kernel bypass, which not only helps to give full play to the high-speed transmission performance of the network card and greatly reduces the network transmission delay, but also can reduce the CPU occupancy rate to a certain extent and relieve the processor pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1 A schematic diagram of a control layer data kernel bypass system for an RDMA network card according to the present invention;

[0028] Figure 2 It is a schematic diagram of the internal structure of the control layer data kernel bypass system for the RDMA network card of the present invention;

[0029] Figure 3 A flow chart of the steps of executing a control layer data kernel bypass task is provided for an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0030] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] like Figure 1 The control layer data kernel bypass system 100 for an RDMA network card of the present application is shown, which is applicable to an RDMA network card device based on an Infiniband network or a RoCE network. Specifically, the control layer data kernel bypass system 100 for an RDMA network card includes a kernel state software module 10, a user state software module 20 and an FPGA hardware module 30.

[0032] The kernel state software module 10 is set in the kernel state driver of the host RDMA network card, and is used to directly interact with the FPGA hardware module 30 when creating and destroying resources. The kernel state software module 10 is also used to process the registration and deregistration of the control layer data memory area shared by the user state software module 20 and the FPGA hardware module 30, and to map the kernel state virtual address space to the user state virtual address space and the kernel state virtual address space to the physical address space.

[0033] The user state software module 20 is set in the host RDMA network card user state driver, used to interact with the application through the IB_Verbs interface, and is responsible for reading, writing and processing the control layer data during the RDMA network card transmission process.

[0034] The FPGA hardware module 30 is set in the RDMA network card, and is used to complete the actual transmission of the RDMA network card by directly reading and writing the control layer data of the user-mode software module 20.

[0035] like Figure 2 As shown, in an implementation of the present application, the kernel-mode software module 10 includes: an address mapping unit 11 and a control layer data management unit 12 .

[0036] The address mapping unit 11 is used to convert the virtual address of the kernel state from virtual to physical address, and to map a continuous virtual address space into a continuous physical address space, so as to facilitate the hardware to directly read and write the control layer data. The address mapping unit 11 is also used to convert the virtual address of the kernel state from kernel state to user state address, and to map a continuous virtual address space of the kernel state into a continuous virtual address space of the user state, so as to facilitate the user state driver to directly read and write the control layer data. The address mapping unit 11 is also used to map the device address of the hardware into the virtual address of the kernel state and the user state, so as to facilitate the driver to read and write the register of the hardware. The control layer data management unit 12 is used to create and destroy the corresponding CQE and WQE memory space, and then call the address mapping unit 11 to obtain the address required by the hardware and user state driver, and pass it to the hardware and user state driver.

[0037] In the implementation of the present application, the CQE and WQE memory space maintained by the control layer data management unit 12 is a circular data buffer. It adopts a first-in-first-out FIFO structure and is shared by the user-mode driver module and the FPGA hardware module 30. Each read and write will offset the size of a CQE / WQE based on the previous one to ensure that the user-mode software module 20 and the FPGA hardware module 30 can accurately read and write data in the buffer. The length of the CQE buffer is determined by the length of a single CQE and the number of buffers that can be buffered, and the length of the WQE buffer is determined by the length of a single WQE and the maximum send depth of the driver.

[0038] In the implementation manner of the present application, the user mode software module 20 includes: a task completion unit 21 and a task request unit 22 .

[0039] The task completion unit 21 is used to read the CQE generated by the FPGA hardware from the CQE buffer, parse it, pass the task completion information to the application, and also inform the FPGA hardware that the CQE at the corresponding position has been acquired by the application.

[0040] The task request unit 22 is used to convert the send / receive instruction issued by the user into a WQE that can be parsed by the hardware, and fill it into the corresponding position of the WQE buffer, and then write DoorBell into the FPGA hardware register through MMIO so that the FPGA hardware can read the WQE.

[0041] In the implementation manner of the present application, the FPGA hardware module 30 includes: a CQE processing unit 31 and a WQE processing unit 32 .

[0042] The CQE processing unit 31 is used to generate a CQE according to the completion status after the packet sending and receiving task is completed, and fill it into the corresponding position of the CQE buffer for the user mode software module 20 to read.

[0043] After receiving the DoorBell, the WQE processing unit 32 is used to read the WQE instruction sent by the user-mode software module 20 from the WQE buffer, parse it, and then complete the task of sending and receiving packets according to its content.

[0044] In the implementation manner of the present application, the control layer data includes DooBell, WQE and CQE, which are accessed jointly by the user-mode software module 20 and the FPGA hardware module 30 .

[0045] DoorBell is essentially an FPGA hardware register, which is directly read and written by the user-mode software module 20 through MMIO after being processed by the address mapping unit 11, and is used for the user-mode software module 20 to notify the FPGA hardware module 30 to read the WQE after writing the WQE buffer.

[0046] WQE includes information such as the address, length, and key of the data to be sent / received, which is written by the user-mode software module 20 to the corresponding position of the WQE buffer and notified to the FPGA hardware module 30 through DoorBell. After receiving the notification, the FPGA hardware module 30 reads the content of the corresponding position of the WQE buffer, parses it, and completes the corresponding packet sending and receiving actions.

[0047] CQE includes information such as the status, serial number, operation code and valid flag of the sent / received message, which is written by the FPGA hardware module 30 to the corresponding position of the CQE buffer. The user-mode software module 20 that continuously polls the CQE valid flag at this position in the CQE buffer obtains the CQE, reports the completion status of the packet sending and receiving to the application according to the CQE content, and clears the flag in the CQE at this position so that the FPGA hardware module 30 can smoothly write CQE in the next round.

[0048] like Figure 3 As shown, in the implementation manner of the present application, the specific method for executing the control layer data kernel bypass task includes:

[0049] S11: The application program calls a send / receive instruction, and then calls the callback function of the user-mode software module 20 for processing the send / receive request through the IB_Verbs interface.

[0050] S12: The user-mode software module 20 parses the instruction. That is, the callback function in the user-mode software module 20 generates a WQE that is convenient for the FPGA hardware module 30 to parse according to the instruction issued by the application.

[0051] S13: The user-mode software module 20 sends WQE and notifies the hardware, that is, the user-mode software module 20 fills the generated WQE into the corresponding WQE buffer and notifies the FPGA hardware module 30 via DoorBell.

[0052] S14: The FPGA hardware module 30 receives the notification and parses the WQE. That is, after receiving the DoorBell from the user-mode software module 20, the FPGA hardware module 30 reads the WQE from the corresponding WQE buffer, parses it, and obtains the task information.

[0053] S15: The FPGA hardware module 30 completes the task of sending and receiving packets. That is, after parsing the WQE, the FPGA hardware module 30 starts to process the corresponding sending or receiving task according to its task information and completes it.

[0054] S16: The FPGA hardware module 30 generates a CQE and writes it into the memory. That is, the FPGA hardware module 30 generates a CQE according to the completion status of the task, sets its valid flag to valid, and writes it into the CQE buffer.

[0055] S17: The user-mode software module 20 polls the CQE. That is, after the FPGA hardware module 30 writes the CQE, the user-mode software module 20 continuously polls the CQE valid flag at the position in the CQE buffer to obtain the CQE.

[0056] S18: The user-mode software module 20 clears the CQE valid flag, so that the FPGA hardware module 30 can smoothly write CQE in the next round.

[0057] S19: The user-mode software module 20 reports the completion status. That is, the user-mode software module 20 obtains the completed task status by parsing the CQE generated by the FPGA hardware module 30, and reports it to the application through the callback function of the IB_Verbs interface.

[0058] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A control layer data kernel bypass system for an RDMA network card, characterized in that: Contains: kernel-mode software module, user-mode software module and FPGA hardware module; The kernel state software module is set in the kernel state driver of the host RDMA network card, and is used to directly interact with the FPGA hardware module when creating and destroying resources. The kernel state software module is also used to process the registration and deregistration of the control layer data memory area shared by the user state software module and the FPGA hardware module, and to perform mapping of the kernel state virtual address space to the user state virtual address space and mapping of the kernel state virtual address space to the physical address space; The user state software module is set in the host RDMA network card user state driver, used to interact with the application through the IB_Verbs interface, and is responsible for the reading, writing and processing of control layer data during the RDMA network card transmission process; The FPGA hardware module is arranged in the RDMA network card, and is used to complete the actual transmission of the RDMA network card by directly reading and writing the control layer data of the user state software module; The user state software module includes: The task completion unit is used to read the CQE generated by the FPGA hardware from the CQE buffer, parse it, pass the task completion information to the application, and also inform the FPGA hardware that the CQE at the corresponding position has been obtained by the application; The task request unit is used to convert the send / receive instructions issued by the user into a WQE that can be parsed by the hardware, and fill it into the corresponding position of the WQE buffer, and then write the DoorBell to the FPGA hardware register through MMIO so that the FPGA hardware can read the WQE; The FPGA hardware module includes: A CQE processing unit, configured to generate a CQE according to the completion status after the packet sending and receiving task is completed, and fill it into a corresponding position of the CQE buffer for reading by the user-mode software module; The WQE processing unit is used to read the WQE instruction sent by the user-mode software module from the WQE buffer after receiving the DoorBell, parse it, and complete the task of sending and receiving packets according to its content.

2. The control layer data kernel bypass system for RDMA network card according to claim 1, characterized in that: The kernel state software module includes: An address mapping unit is used to convert virtual addresses in kernel state into virtual-physical addresses, convert virtual addresses in kernel state into kernel state-user state addresses, and map hardware device addresses into virtual addresses in kernel state and user state; The control layer data management unit is used to create and destroy the corresponding CQE and WQE memory spaces, and then call the address mapping unit to obtain the addresses required by the hardware and user-mode drivers, and pass them to the hardware and user-mode drivers.

3. The control layer data kernel bypass system for RDMA network card according to claim 2, characterized in that: The CQE and WQE memory space is a circular data buffer with a first-in-first-out FIFO structure, shared by the user-state driver module and the FPGA hardware module. Each read and write will be offset by the size of a CQE / WQE based on the previous one, so that the user-state software module and the FPGA hardware module can accurately read and write data in the buffer. The length of the CQE buffer is determined by the length of a single CQE and the bufferable quantity, and the length of the WQE buffer is determined by the length of a single WQE and the maximum sending depth of the driver.

4. The control layer data kernel bypass system for RDMA network card according to claim 3, characterized in that: The control layer data includes DooBell, WQE and CQE, which are accessed by the user state software module and the FPGA hardware module; After DoorBell is processed by the address mapping unit, it is directly read and written by the user-mode software module through MMIO, so that the user-mode software module notifies the FPGA hardware module to read WQE after writing the WQE buffer; WQE includes the address, length and key of the data to be sent / received, which is written into the corresponding position of the WQE buffer by the user-mode software module and notified to the FPGA hardware module through DoorBell. After receiving the notification, the FPGA hardware module reads the content of the corresponding position of the WQE buffer, parses it and completes the corresponding packet sending and receiving action; CQE includes the status, serial number, operation code and valid flag of the sent / received message, which are written by the FPGA hardware module to the corresponding position of the CQE buffer. The user-mode software module that continuously polls the CQE valid flag at this position in the CQE buffer obtains CQE, reports the completion status of sending and receiving packets to the application according to the CQE content, and clears the flag in the CQE at this position so that the FPGA hardware module can smoothly write CQE in the next round.

5. The control layer data kernel bypass system for RDMA network card according to claim 1, characterized in that: The control layer data kernel bypass system for the RDMA network card is used for an RDMA network card device based on an Infiniband network or a RoCE network.

Images