RDMA communication method and device based on user mode Ethernet data transmission

Abstract

The invention provides an RDMA (Remote Direct Memory Access) communication method based on user mode Ethernet data transmission, which comprises the following steps: in a user mode rDMA-core driver, defining an RDMA verbs operation structural body, establishing a mapping relationship between an RDMA verbs operation function and a DPDK API (Application Program Interface), mapping an RDMA operation to a DPDK data plane, and calling the DPDK API to realize the RDMA verbs operation function; when the network card queue is operated to send data, the RDMA sends a request to call the DPDK API, constructs a RoCEv2 protocol data packet, and sends the data packet to the network card; when the network card queue is operated to receive data, after a DPDK polling mode driver receives a data packet, the received data packet is analyzed according to the RoCEv2 protocol, effective data is written into a pre-registered memory, and a user mode is triggered to complete a queue event. The invention further provides an RDMA communication device based on the user mode Ethernet data transmission.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data transmission, and particularly relates to an RDMA communication method and device based on user-mode Ethernet data transmission. BACKGROUND

[0002] RDMA (Remote Direct Memory Access) is a bypass-kernel remote memory direct access technology, which is widely used in data-intensive and computing-intensive scenarios, and is an important solution in the fields of high-performance computing, machine learning, data center, mass storage, etc. RDMA has the characteristics of zero-copy and protocol stack offloading. RDMA sinks the implementation of the protocol stack to the RDMA network card (RNIC), and directly accesses the data in the remote memory without passing through the kernel. Compared with the traditional TCP / IP implementation, RDMA not only saves the CPU resources required for protocol processing and data copying, but also improves the network throughput and reduces the network communication delay.

[0003] Although RDMA technology can bring satisfactory network performance improvement, for existing Ethernet-based applications, in order to enjoy the performance dividend brought by RDMA, it is necessary to develop RDMA based on the original Ethernet. The current solution is Soft-RoCE, which is a pure software-based RDMA technology. It enables RoCE (RDMA over Converged Ethernet) to run on any type of network card. Soft-RoCE provides a solution for high-performance data transmission using existing Ethernet infrastructure without relying on dedicated hardware.

[0004] Soft-RoCE is to do the packet and parsing work that should be offloaded to hardware again in the software layer. It is implemented based on the Linux kernel TCP / IP protocol stack, and the network card itself does not perceive the transmitted and received data packets as RoCE messages. The driver program encapsulates user data into IB transport layer messages according to the message format in the IB specification, and then fills the messages into the Socket Buffer as data for the next packet transmission and reception processing by the Ethernet network card. Soft-RoCE still brings performance improvement due to the reduction of system calls, zero-copy on the sending side, and single-copy on the receiving side, but the performance improvement is not as obvious as that of hardware RDMA.

[0005] See Figure 1The Soft-RoCE in the linux system kernel state implements the RDMA communication scheme, the Soft-RoCE uses the Linux kernel state Ethernet protocol stack, the RoCEv2 protocol processing module and the Linux kernel state network card driver, RDMA data comes to the Linux kernel through the network card driver, and then is parsed by the Ethernet protocol stack, and then is parsed by the RoCEv2 protocol stack, and then the data is copied to the user state application memory, and the data is copied multiple times, and the kernel state network user state data is copied, and the data transmission performance is affected.

[0006] DPDK (Data Plane Development Kit) is an open-source software development kit for building high-performance data plane applications. It provides a set of optimized user space libraries and HYPERLINK "https: / / so.csdn.net / so / search?q=%E9%A9%B1%E5%8A%A8%E7%A8%8B%E5%BA%8F&spm=1001.2101.3001.7020" \t "https: / / blog.csdn.net / lincolnjunior_lj / article / details / _blank" drivers that enable network packet processing to achieve extremely high throughput and low latency on general-purpose servers. The design goal of DPDK is to bypass the operating system kernel's network protocol stack and directly perform packet reception and processing in user space, thereby improving the performance of network applications.

[0007] Therefore, in order to improve the processing speed of Soft-RoCE and improve the RDMA performance, a technical scheme combining DPDK technology and Soft-RoCE technology needs to be developed. SUMMARY

[0008] The present application provides an RDMA communication method and device based on user state Ethernet data transmission, which combines DPDK technology and Soft-RoCE technology, uses user state network card driver, and directly transmits and receives data in user state.

[0009] Other purposes and advantages of the present application can be further understood from the technical features disclosed in the present application.

[0010] To achieve one or part or all of the above purposes or other purposes, a user-mode Ethernet data transmission-based RDMA communication method is provided in a technical solution of the present application, characterized in that, in a user-mode rdma-core driver, an RDMA verbs operation structure body is defined, a mapping relationship between RDMA verbs operation functions and DPDK APIs is established, RDMA operations are mapped to the DPDK data plane, and the RDMA verbs operation functions are called by using the DPDK APIs; when data is sent, the RDMA sending request calls the DPDK API, a RoCEv2 protocol data packet is constructed, and the data packet is sent to a network card; when data is received, after the DPDK polling mode driver receives the data packet, the received data packet is parsed according to the RoCEv2 protocol, and valid data is written into pre-registered memory, triggering the user mode to complete a queue event.

[0011] The RDMA verbs operation functions called by using the DPDK APIs include: queue pair creation, memory region registration, completion queue creation, and RDMA SEND / RDMA WRITE / RDMA READ operation issuing and completion event polling.

[0012] The queue pair creation includes: binding the sending queue and the receiving queue in the RDMA verbs operation function queue pair to the TX queue and the RX queue of the DPDK, realizing lock-free communication of the TX and RX queues of the DPDK by using the rte_ring; after the binding is completed, two worker threads are created, which are respectively: a sending worker thread responsible for processing tasks on the sending queue, and a receiving worker thread responsible for processing tasks on the receiving queue.

[0013] When the sending worker thread processes the sending task request, the sending worker thread polls the sending queue, if there is a task in the sending queue, the task is taken out, a data packet is constructed according to the RoCEv2 protocol, and the data packet is sent to the network card by using the DPDK driver sending interface function, and the completed sending task is mounted to the completion queue.

[0014] When the DPDK driver sends a message, a retransmission timeout timer is started, a timeout flag is set in advance, and a retransmission flag is set, if there is a retransmission flag in the request task, the retransmission operation is performed after the timeout.

[0015] The TX queue depth of the DPDK network card is used to limit the sending flow of the queue pair, when the TX queue depth of the DPDK network card is full, the sending request is suspended until the TX queue of the DPDK network card is not full, and then the sending request is processed.

[0016] When the receiving worker thread processes the receiving task request, the DPDK polls the receiving queue, when a data packet arrives, the DPDK driver receives the interface function to receive the data packet, and parses the data packet according to the RoCEv2 protocol, copies the data to the address specified in the request according to the address and size in the receiving request, and according to the parsing result, the receiving task that has completed processing is mounted on the completion queue.

[0017] When processing the receiving task request, the packet header information of the data packet is parsed according to the RoCEv2 protocol, and the parsed data packet is subjected to CRC data integrity check; according to the parsed header information, a queue pair is found, and the corresponding completion queue is found in the queue pair, and the task receiving request that has completed processing is mounted in the completion queue.

[0018] The DPDK driver receives the message using the interface function, monitors the legality of the message according to the RoCEv2 protocol, analyzes the content of the message, triggers the state machine to switch, and replies to the corresponding message according to the state.

[0019] The registration of the memory region includes calling the DPDK memory management interface to allocate continuous physical memory, ensuring that the network card DMA is normally addressed, locking the physical memory, and generating a control authority key for accessing the physical memory region.

[0020] The completion queue event processing mode uses the DPDK PMD polling mode, when the task on the sending queue and the receiving queue is completed, the task that has been processed is hung on the completion queue in order, and the completion of the task is waited for by the upper layer application program.

[0021] Another technical solution provided by the application is an RDMA communication device based on user-mode Ethernet data transmission, which is used to realize the RDMA communication method based on user-mode Ethernet data transmission described above, and includes: creating a DPDK driver in the user mode, the DPDK driver directly accessing the network card hardware; a user-mode rdma-core driver, the user-mode rdma-core driver defining an RDMA verbs operation structure body; establishing a mapping relationship between the RDMA verbs operation function and the DPDK API, mapping the RDMA operation to the DPDK data plane, and calling the DPDK API to realize the RDMA verbs operation function; a user-mode RoCE protocol stack, the data packet received by the DPDK driver is subjected to packet processing and unpacking processing according to the RoCEv2 protocol.

[0022] Compared with the prior art, the beneficial effects of the present application mainly include that the present application combines the DPDK technology with the Soft-RoCE technology, uses a user state network card driver, directly transmits and receives data in the user state, and the received data is parsed according to the RoCEv2 protocol of the user state. The data copying and the system call between the user state and the kernel state are greatly reduced, and the performance influence caused by the network card interrupt and other factors is also reduced.

[0023] In order to make the above and other objects, features and advantages of the present application more apparent, the following preferred embodiments will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 The scheme architecture diagram of the Soft-RoCE in the prior art for implementing the software RDMA in the Linux system kernel state.

[0026] Figure 2 The scheme architecture diagram of the DPDK-RoCE of the present application for implementing the software RDMA in the Linux system kernel state.

[0027] Figure 3 The schematic diagram of the user state rdma-core driver verbs processing function interface combined with the DPDK network API function of the present application. DETAILED DESCRIPTION

[0028] The foregoing and other technical contents, features and effects of the present application will be clearly presented in the following detailed description of a preferred embodiment with reference to the drawings. The directional terms mentioned in the following embodiments, such as up, down, left, right, front or back, etc., are only the directions of the drawings. Therefore, the directional terms used are used for explanation, not for limitation of the present application.

[0029] Embodiment one Embodiment one provides a RDMA communication method based on user-mode Ethernet data transmission. In a user-mode rdma-core driver, an RDMA verbs operation structure body is defined, a mapping relationship between RDMA verbs operation functions and DPDK APIs is established, an RDMA operation is mapped to the DPDK data plane, and the RDMA verbs operation functions are called by invoking the DPDK API; when a network card queue is operated for data sending, the RDMA sending request calls the DPDK API, constructs a RoCEv2 protocol data packet, and sends the data packet to the network card; when the network card queue is operated for data receiving, after the DPDK polling mode driver receives the data packet, the received data packet is parsed according to the RoCEv2 protocol, and the valid data is written into the pre-registered memory, so as to trigger the user mode to complete the queue event.

[0030] The technical solutions provided by embodiment one are described in detail below with reference to the accompanying drawings.

[0031] Referring to Figure 2 , the DPDK-RoCE includes a user-mode DPDK driver, a user-mode protocol stack, and a user-mode rdma-core driver. The RDMA communication method based on user-mode Ethernet data transmission provided by embodiment one includes the following steps. Step S1: An Ethernet network card device suitable for the DPDK network driver is selected, the network card is configured, and the network card is bound to the DPDK-compatible driver program, so that the DPDK driver can directly access the network card hardware, and the DPDK driver environment is initialized, including setting up a memory pool, a ring buffer, and other data for storing and transmitting data packets. The DPDK mainly includes the following key components and features: a user space driver program to realize direct control of network devices and transmission and reception of data packets, support for multi-core concurrent processing to realize high throughput and low latency data packet processing. Large page memory, memory pool, and ring buffer, etc. to reduce the overhead of memory allocation and release and improve data access efficiency, bypass the operating system kernel network protocol stack, and directly access the network card and network device to reduce data packet processing delay and overhead.

[0032] Step S2: In the user-mode rdma-core driver, an RDMA verbs operation structure body is defined, that is, a struct verbs_context_ops dpdk_ctx_ops processing function in the Figure 3 The operation structure includes functions such as creating and destroying a protection domain, a queue pair, a completion queue, and memory registration. The custom function operation structure includes a plurality of Verbs processing function interfaces, and these Verbs processing functions are set for implementing RDMA functions. The Verbs processing functions are related in the prior art, and are not specifically described herein. For example: Figure 3query_device_ex in the above is an interface for querying device extension information; alloc_pd (Allocate Protection Domain) is an allocation protection domain function interface, which is a core mechanism for hardware resource management and authority management.

[0033] Step S3: Establishing a mapping relationship between RDMA verbs operation functions and DPDK APIs, mapping RDMA operations to the DPDK data plane, and calling DPDK APIs to implement RDMA verbs operation functions.

[0034] RDMA verbs operation functions implemented by calling DPDK APIs include: queue pair (QP) creation, memory region (MR) registration, completion queue (CQ) creation, RDMA SEND / RDMA WRITE / RDMA READ operation issuing and completion event polling, etc.

[0035] Specifically, as shown in Figure 3 , in the operation structure body struct verbs_context_ops dpdk_ctx_ops processing function, the left side is the verbs processing function interface, and the right side is the RDMA verbs processing interface to be implemented by calling the DPDK network API function. For example Figure 3 query_device_ex = dpdk_query_device in the above, at which time the RDMA verbs operation function can be implemented by calling a specific DPDK API.

[0036] For the creation of a queue pair, including: binding the sending queue (SQ) and the receiving queue (RQ) in the RDMA verbs operation function queue pair to the TX queue (a queue for data sending in the DPDK driver) and the RX queue (a queue for data receiving in the DPDK driver) of the DPDK, realizing lock-free communication of the TX and RX queues of the DPDK through rte_ring; after the binding is completed, creating two worker threads, including: a sending worker thread responsible for processing tasks on the sending queue; and a receiving worker thread responsible for processing tasks on the receiving queue.

[0037] For the registration of a memory region, including: calling a DPDK memory management interface to allocate continuous physical memory, ensuring normal addressing of the network card DMA, locking the physical memory, and generating a control authority key for accessing the physical memory region.

[0038] Step S4: Implementing data receiving and sending operations based on DPDK APIs.

[0039] Step S4-1: When sending data in RDMA mode based on the DPDK API, the dpdk_post_send function (shown in Figure 3 The data is sent by operating the sending of data using the dpdk_post_send function (shown in

[0040] When sending a task, the DPDK driver sending interface function (rte_eth_tx_burst function) is used to send a message (a message is different from a data packet, and a message is a complete data unit), a retransmission timeout timer is started, a timeout flag is set in advance, and a retransmission flag is set. When a task request with a retransmission flag is retransmitted in the request task, the retransmission operation is triggered after the timeout.

[0041] Step S4-2: When receiving data in RDMA mode based on the DPDK API, the dpdk_post_recv function (shown in Figure 3 The receiving request is submitted to the queue pair. Then, the receiving worker thread polls the DPDK RX queue, and when a data packet arrives, the DPDK driver receiving interface function (rte_eth_rx_burst function) receives the data packet from the network card and saves it in the memory pool. The received data packet is parsed according to the RoCEv2 protocol. Specifically, when processing a receiving request task, the DPDK driver polls the DPDK RX queue, and when a data packet arrives, it is received by the DPDK driver receiving interface function (rte_eth_rx_burst function). The received data is parsed according to the RoCEv2 protocol, and the data is copied to the address specified in the request according to the address and size in the receiving request. The completed receiving request is mounted on the completion queue CQ and reported to the application program, which processes the data accordingly.

[0042] When processing the received task request, the packet header information of the data packet is parsed according to the RoCEv2 protocol, and the parsed data packet is subjected to CRC data integrity check; according to the parsed packet header information, the queue pair QP is found, and the corresponding completion queue CQ is found in the queue pair QP, and the task receiving request of the completion processing is mounted in the completion queue CQ.

[0043] The completion queue CQ is used for processing and reporting the results of sending and receiving, and when the corresponding sending queue SQ and receiving queue RQ have task completion, the task is mounted on the completion queue CQ in the order of priority, waiting for the upper application program to obtain the completion of the task.

[0044] When receiving data, the DPDK driver receives the interface function (rte_eth_rx_burst function) to receive the message, and according to the RoCEv2 protocol, the message legality monitoring and message content analysis are performed in the user state protocol stack, the state machine is switched, and the corresponding message is replied according to the state.

[0045] In order to realize the RDMA QP queue flow control, the TX queue depth of the DPDK network card is controlled to realize the limitation of the queue pair QP sending flow, and when the TX queue depth of the DPDK network card is full, the sending request is suspended until the TX queue of the DPDK network card is not full, and then the sending request is processed.

[0046] Step S5: When the sending request and the receiving request are executed, a completion result is generated according to the ID of the request, and is mounted on the completion queue CQ, and the sending and receiving conditions are reported to the upper layer application through the DPDK polling completion queue function, that is, dpdk_poll_cq in Figure 3 .

[0047] Embodiment Two Embodiment Two provides an RDMA communication device based on user state Ethernet data transmission, which is used to realize the RDMA communication method based on user state Ethernet data transmission in Embodiment One, see Figure 3 , which includes the following parts: DPDK driver created in user state, DPDK driver directly accessing network card hardware; user state rdma-core driver, user state rdma-core driver defining RDMA verbs operation structure; establishing mapping relationship between RDMA verbs operation function and DPDK API, mapping RDMA operation to the DPDK data plane, and calling DPDK API to realize RDMA verbs operation function; user state RoCE protocol stack, packet processing and unpacking processing are performed on the data packet received by the DPDK driver according to the RoCEv2 protocol.

[0048] The RDMA communication method and device based on user-mode Ethernet data transmission are described in detail above, and the structure and working principle of the present application are described by using specific examples in this paper. The above example is only used to help understand the method and core idea of the present application. It should be pointed out that for ordinary skilled persons in the art, without departing from the principle of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. An RDMA communication method based on user-mode Ethernet data transmission, characterized in that: In the user-mode rdma-core driver, define the RDMA verbs operation structure, establish a mapping relationship between the RDMA verbs operation function and the DPDK API, map the RDMA operation to the DPDK data plane, and call the DPDK API to implement the RDMA verbs operation function; When sending data, the RDMA send request calls the DPDK API, constructs the RoCEv2 protocol data packet, and sends the data packet to the network card; When receiving data, the DPDK polling mode driver receives the data packet, parses the received data packet according to the RoCEv2 protocol, writes the valid data to the pre-registered memory, and triggers the user state completion queue event.

2. The RDMA communication method based on user-mode Ethernet data transmission according to claim 1, characterized in that: The RDMA verbs operation functions implemented by calling the DPDK API include: queue creation, memory area registration, completion queue creation, RDMA SEND / RDMA WRITE / RDMA READ operation issuance and completion event polling.

3. The RDMA communication method based on user-mode Ethernet data transmission according to claim 2, characterized in that: The creation of a queue pair includes binding the send queue and receive queue in the RDMA verbs operation function queue pair to the DPDK TX queue and RX queue, and implementing lock-free communication between the DPDK TX and RX queues through rte_ring; After the binding is completed, two working threads are created: The sending worker thread is responsible for processing the tasks on the sending queue; the receiving worker thread is responsible for processing the tasks on the receiving queue.

4. The RDMA communication method based on user-mode Ethernet data transmission according to claim 3, characterized in that: When the sending worker thread processes the sending task request, the sending worker thread polls the sending queue. If there is a task in the sending queue, the task is taken out, a data packet is constructed according to the RoCEv2 protocol, and the data packet is sent to the network card through the DPDK driver sending interface function, and the completed sending task is mounted on the completion queue.

5. The RDMA communication method based on user-mode Ethernet data transmission according to claim 3, characterized in that: When using the DPDK driver to send a message, start the retransmission timeout timer, set the timeout mark in advance, and then set the retransmission mark. If a task with a retransmission mark in the request task requests a retransmission operation, the retransmission operation will be triggered after the timeout.

6. The RDMA communication method based on user-mode Ethernet data transmission according to claim 3, characterized in that: The queue limits the sending traffic through the TX queue depth of the DPDK network card. When the TX queue depth of the DPDK network card is full, the processing of sending requests is suspended until the TX queue of the DPDK network card is not full, and then the sending requests are continued.

7. The RDMA communication method based on user-mode Ethernet data transmission according to claim 3, characterized in that: When the receiving worker thread processes the receiving task request, DPDK polls the receiving queue. When a data packet arrives, DPDK drives the receiving interface function to receive the data packet, and parses the data packet according to the RoCEv2 protocol. According to the address and size in the receiving request, the data is copied to the address specified by the request. Based on the parsing result, the receiving task that has been completed is mounted on the completion queue.

8. The RDMA communication method based on user-mode Ethernet data transmission according to claim 7, characterized in that: When processing a received task request, the packet header information is parsed according to the RoCEv2 protocol, and the CRC data integrity check is performed on the parsed packet; According to the parsed packet header information, a queue pair is found, and a corresponding completion queue is found in the queue pair, and the task receiving request that has been processed is mounted to the completion queue.

9. The RDMA communication method based on user-mode Ethernet data transmission according to claim 8, characterized in that: Use the DPDK driver receiving interface function to receive messages, monitor the legitimacy of the messages and analyze the message content according to the RoCEv2 protocol, trigger the state machine switching, and reply to the corresponding message based on the status.

10. The RDMA communication method based on user-mode Ethernet data transmission according to claim 2, characterized in that: The registration of the memory area includes calling the DPDK memory management interface to allocate continuous physical memory, ensuring normal addressing of the network card DMA, locking the physical memory, and generating a control permission key for accessing the physical memory area.

11. The RDMA communication method based on user-mode Ethernet data transmission according to claim 2, characterized in that: The completion queue event processing mode adopts the DPDK PMD polling mode. When tasks are completed on the send queue and receive queue, the processed tasks are placed on the completion queue in order, waiting for the upper-level application to obtain the completion status of the tasks.

12. An RDMA communication device based on user-mode Ethernet data transmission, characterized in that: The RDMA communication device is used to implement the RDMA communication method based on user-mode Ethernet data transmission according to any one of claims 1 to 10, comprising: creating a DPDK driver in user mode, wherein the DPDK driver directly accesses the network card hardware; A user-mode rdma-core driver, wherein the user-mode rdma-core driver defines an RDMA verbs operation structure; Establish a mapping relationship between the RDMA verbs operation function and the DPDK API, map the RDMA operation to the DPDK data plane, and call the DPDK API to implement the RDMA verbs operation function; The user-mode RoCE protocol stack performs packet processing and unpacking on the data packets received by the DPDK driver according to the RoCEv2 protocol.

Images