Data transmission method and device based on RDMA, storage medium and program product

Abstract

The embodiment of the invention provides a data transmission method and device based on RDMA, a storage medium and a program product. In the embodiment of the invention, a local working queue for RDMA communication with the same host and a remote working queue for cross-host RDMA communication are created at the same time in the RDMA connection establishment process, a same host identification function is added, and under the condition that communication with the same host is identified, the local working queue is utilized to perform RDMA communication with the cross host. According to the method, the data copying operation is directly executed in the registered memories of the two applications to replace the traditional RDMA transmission operation, so that the transmission delay between different RDMA applications on the same host can be reduced under the conditions of compatibility with an RDMA programming interface and transparency or no perception of the applications, the consumption of PCIe bandwidth is reduced, and the transmission efficiency is improved. And meanwhile, the fairness of the local flow and the network flow during resource competition is ensured.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data transmission method, device, storage medium, and program product based on RDMA. Background Art

[0002] Remote Direct Memory Access (RDMA) is a technology that bypasses the kernel of a remote host's operating system to access data in its memory. This technology allows data to be transferred directly between user mode and network card hardware, effectively avoiding the overhead of context switching between user mode and kernel mode, as well as the kernel's network protocol processing overhead. This improves communication performance while reducing CPU (central processing unit) overhead. Therefore, RDMA has been widely used in scenarios such as cloud storage, high-performance computing (HPC), and distributed systems.

[0003] With the rapid adoption of cloud-native containerization, different RDMA applications can be deployed in different containers on the same host. However, communication between RDMA applications in different containers on the same host has exposed the shortcomings of RDMA. RDMA network cards are typically connected to the CPU via the high-speed Peripheral Component Interconnect Express (PCIe) bus. Therefore, when RDMA applications in different containers on the same host communicate, data must also undergo two PCIe operations, which not only consumes PCIe bandwidth but also increases transmission latency. Summary of the Invention

[0004] Multiple aspects of the present application provide an RDMA-based data transmission method, device, storage medium, and program product, which are used to reduce the transmission delay between different RDMA applications on the same host and save bandwidth resources while being compatible with the RDMA programming interface.

[0005] An embodiment of the present application provides an RDMA-based data transmission method, which is applied to a physical machine on which a first application is deployed. The method includes: when the first application initiates an RDMA communication request to a second application, creating a first RDMA work queue for the first application and registering it in memory, the first RDMA work queue including a local work queue and a remote work queue; based on a control plane connection between the first application and the second application, obtaining description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue; based on the description information of the second RDMA work queue, identifying whether the communication between the first application and the second application is same-host RDMA communication, the second RDMA work queue also including a local work queue and a remote work queue; and if it is identified as same-host RDMA communication, performing a data copy operation in the memory registered by the two applications based on the local work queue of at least one application in the two applications, to replace the RDMA data transmission operation between the two applications.

[0006] An embodiment of the present application also provides a physical machine, which includes an RDMA network card, and a first application and an RDMA management program are deployed on the physical machine, wherein the RDMA management program includes: a user-state interface management module and a kernel-state driver management module; the driver management module cooperates with the interface management module to: when a first application initiates an RDMA communication request to a second application, create a first RDMA work queue for the first application and register it in memory; based on the control plane connection between the first application and the second application, obtain description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue; the first and second RDMA work queues both include a local work queue and a remote work queue; based on the description information of the second RDMA work queue, identify whether the communication between the first application and the second application is the same-host RDMA communication; and, in the case of identifying the same-host RDMA communication, perform a data copy operation in the memory registered by the two applications based on the local work queue of at least one application of the two applications to replace the RDMA data transmission operation between the two applications.

[0007] An embodiment of the present application also provides a physical machine, comprising: a memory and a processor; the memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in the above method.

[0008] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method.

[0009] An embodiment of the present application further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, enables the processor to implement the steps in the above-described method.

[0010] In an embodiment of the present application, the interface management module in the user state and the driver management module in the kernel state in the RDMA management program deployed on the physical machine are improved. With the cooperation of the driver management module and the interface management module, a local work queue and a remote work queue are created simultaneously during the RDMA connection process, and a same-host identification function is added. When it is identified that the communication is with the same host, the local work queue is used to directly perform data copy operations in the registered memory of the two applications to replace the traditional RDMA transmission operation. For the application, it can directly initiate an RDMA communication request like traditional RDMA communication. When it is compatible with the RDMA programming interface and transparent or unaware to the application, the transmission delay between different RDMA applications on the same host can be reduced, the consumption of PCIe bandwidth can be reduced, and the fairness of local traffic and network traffic in resource competition can be ensured. Further, when it is identified that the communication is across hosts, the remote work queue is used to perform traditional RDMA transmission operations without affecting the cross-host RDMA communication process. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0012] Figure 1 A schematic diagram of the interconnection structure and data transmission process of physical machines provided in an exemplary embodiment of the present application;

[0013] Figure 2a-2b A schematic diagram of the internal architecture of a physical machine provided by another exemplary embodiment of the present application;

[0014] Figure 3a A schematic structural diagram of an interface management module provided in another exemplary embodiment of the present application;

[0015] Figure 3b A flowchart of a method for creating a CQ queue provided by another exemplary embodiment of the present application;

[0016] Figure 3c A flowchart of a method for creating a QP queue provided by another exemplary embodiment of the present application;

[0017] Figure 3d A flowchart of a method for same-host identification provided by another exemplary embodiment of the present application;

[0018] Figure 3e A flowchart of a memory registration method provided as another exemplary embodiment of the present application;

[0019] Figure 4 A schematic structural diagram of a drive management module provided in yet another exemplary embodiment of the present application;

[0020] Figure 5 A flowchart of a method performed by a transmitting end when the type of a data transmission task is a bilateral operation is provided as another exemplary embodiment of the present application;

[0021] Figure 6 A flowchart of a method performed by a receiving end when the type of a data transmission task is a bilateral operation is provided as another exemplary embodiment of the present application;

[0022] Figure 7 A flowchart of a method for an RDMA application to check a CQ queue is provided as another exemplary embodiment of the present application;

[0023] Figure 8 A flowchart of an RDMA-based data transmission method provided by an exemplary embodiment of the present application;

[0024] Figure 9 A schematic structural diagram of an RDMA-based data transmission device provided by an exemplary embodiment of the present application;

[0025] Figure 10 A schematic diagram of the structure of a physical machine provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] RDMA is a high-speed direct access technology for peer memory that is optimized for network card devices and software architecture. In the embodiments of this application, the implementation of RDMA technology at the protocol level is not limited, and may include but is not limited to the following:

[0029] InfiniBand protocol: This RDMA technology, based on the InfiniBand architecture, was proposed by the IBTA (InfiniBand Trade Association) and is abbreviated as IB. Building an RDMA network based on IB technology requires dedicated IB network cards and switches. While InfiniBand networks offer superior performance, network cards and switches are relatively expensive.

[0030] iWARP (Internet Wide Area RDMA Protocol), short for Internet Wide Area RDMA Protocol, is an RDMA technology based on the Transmission Control Protocol / Internet Protocol (TCP / IP) and defined by the Internet Engineering Task Force (IETF). iWARP supports RDMA over standard Ethernet infrastructure, eliminating the need for switches to support lossless Ethernet transmission. However, servers must use iWARP-capable network adapters.

[0031] RoCE (RoCE) protocol, also proposed by IBTA, stands for RDMA over Converged Ethernet. It supports RDMA technology over standard Ethernet infrastructure, but requires switches that support lossless Ethernet transmission and servers that use RoCE network adapters. Its performance is comparable to IB.

[0032] RDMA technology using different protocols will have different hardware implementations, specifically different physical layers and link layers. However, no matter which protocol or hardware implementation RDMA uses, the same set of application programming interfaces (APIs) can be used for applications. The APIs provided by RDMA for applications can also be called software transport interfaces (software transport interfaces) between applications and network card devices, that is, Verbs interfaces. In other words, RDMA provides a series of Verbs interfaces in the user space of the application. The application operates the RDMA hardware through these Verbs interfaces, allowing RDMA to bypass the kernel and directly access the user space of the application. The application in the embodiment of the present application refers to an application that can directly read and write remote memory through RDMA technology, that is, an application implemented based on the API provided by RDMA, which can be called an RDMA application or simply an application. From a functional point of view, the application in the embodiment of the present application can be a video processing application, instant messaging application, e-commerce application, and other applications implemented based on the API provided by RDMA.

[0033] Traditional verbs interfaces include two types: memory verbs and messaging verbs. Based on these two verbs interfaces, RDMA provides two basic operations for applications: unilateral operations and bilateral operations.

[0034] Memory verbs: These include RDMA read, write, and atomic operations. These operations are unilateral, meaning that only the local end needs to specify the source and destination addresses of the data to be transmitted. The application on the other end does not need to be aware of this communication. Data reading or writing is completed between the peer network card device and the application's buffer using Direct Memory Access (DMA) technology, without the other end's awareness or participation.

[0035] RDMA Read: The local application uses RDMA technology to read some memory data from the peer. Before performing an RDMA read operation, the peer needs to provide the local application with appropriate permissions so that the local application has permission to access the peer's memory. Once the permissions are set, the RDMA read operation can be performed without any notification to the peer. The local application needs to specify a virtual address on the peer, and then the local network card device uses RDMA technology to read data from the peer's memory directly according to the specified virtual address, just like reading data from local memory, without the peer's application being aware of it.

[0036] RDMA Write: Similar to RDMA Read, this operation writes data to the peer's memory. Before executing an RDMA write operation, the peer needs to provide the local end with appropriate permissions so that the local end has access to the peer's memory. Once the permissions are set, the RDMA write operation can be executed without notifying the peer end. The local application needs to specify a virtual address on the peer end. The local network card device then uses RDMA technology, just like writing data to local memory, to write data directly to the peer's memory according to the specified virtual address without the peer application being aware of it.

[0037] RDMA Atomic: includes atomic load, atomic add, atomic compare, and atomic swap, which are extensions of RDMA atomic operations. This embodiment of the application does not focus on this.

[0038] Messaging verbs: These include RDMA send and receive operations. These operations are bilateral, meaning they require the peer application to be aware of and participate in data transmission and reception.

[0039] RDMA Send: is the process of sending data in the local memory to the peer memory using RDMA technology. Specifically, the local application needs to put a work queue element (WQE) representing a send task into the send queue (SQ) of the queue pair (QP). Similarly, the peer application needs to send a WQE representing a receive task to the gateway device (i.e., RDMA hardware) to the receive queue (RQ) of the peer QP, so that the RDMA hardware knows where to put the received data in the memory. Then, the local RDMA hardware obtains the send task from the SQ, obtains the data from the local memory according to the send task, and transmits it to the peer RDMA hardware via the network. The peer RDMA hardware receives the data, verifies it, and then sends an acknowledgment (ACK) to the sender. It then places the data in the location specified in the WQE (Warning Queue Element). It then generates a Completion Queue Element (CQE) for the receiving task and places it in the Completion Queue (CQ). The peer application then receives the task completion message. After receiving the ACK, the local application also generates a CQE for the sending task and places it in the CQ. The local application then receives the task completion message. This entire process requires the awareness and participation of both the local and peer applications.

[0040] RDMA Receive: This is the process of receiving data from the peer's memory into the local memory using RDMA technology. Specifically, the peer's application needs to send a WQE (Warning Query Entity) to the gateway device (i.e., RDMA hardware) representing a send task to the peer's QP's SQ. The local application needs to place the WQE representing a receive task into the QP's RQ. This allows the RDMA hardware to know where to place the received data in memory. The peer's RDMA hardware then retrieves the send task from the SQ, retrieves the data from the peer's memory based on the send task, and transmits it over the network to the local RDMA hardware. The local RDMA hardware receives the data, verifies it, and sends an acknowledgment (ACK) to the peer. It then places the data in the location specified in the WQE. It then generates a CQE for the receive task and places it in the CQ. The local application then receives the task completion information. After receiving the ACK, the peer also generates a CQE for the send task and places it in the CQ. The peer application then receives the task completion information. The entire process requires the awareness and participation of both the local and peer applications.

[0041] Optionally, the local application and the opposite application may be instant messaging applications, or the local application may be a client of a database storage system and the opposite application may be a server of the database storage system, and so on.

[0042] In actual applications, the local application and the peer application may be located on different physical machines or on the same physical machine. In particular, with the development of virtualization technology, a virtualization manager can be run on a physical machine. The virtualization manager is responsible for creating and maintaining virtualization instances. The virtualization instance provides an independent operating environment for carrying various applications. Among them, the virtualization manager can be a virtual machine monitor (VMM), which can also be called a hypervisor. The virtualization instance can be a cloud host, an elastic computing service (ECS), a virtual machine (VM) or a container, etc. Applications in different virtualization instances can communicate using RDMA technology. For example, in a cloud-native containerization scenario, multiple containers are run on the same physical machine, RDMA applications are deployed in different containers, and RDMA applications in different containers communicate with each other using RDMA technology. In an embodiment of the present application, the focus is on improving the RDMA communication process between different applications deployed on the same physical machine, and an improved RDMA communication method is provided. The following describes the technical problems faced by RDMA communication between different applications on the same physical machine, the improvement ideas proposed to address these technical problems, and the improved RDMA communication process.

[0043] like Figure 1 As shown, the hardware resources of the physical machine include at least a CPU, memory and an RDMA network card, and the RDMA network card is interconnected with the CPU and memory via a PCIe bus. In this embodiment, the implementation form of the physical machine is not limited. The physical machine can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device, or a server-side device such as a conventional server, a cloud server or a server array. Similarly, in the embodiment of the present application, the implementation method of the RDMA network card is not limited. It can be any network card device that supports the RDMA protocol, for example, a cloud infrastructure unit (Cloud Infrastructure Processing Unit, CIPU), a smart network card (Smart NIC), a network interface card (Network Interface Card, NIC) or an infrastructure processor (Infrastructure Processing Unit, IPU) and other chips or modules that are integrated with a network card module that supports the RDMA protocol. In addition, the interconnection bus between RDMA and the CPU and memory takes PCIe as an example, but is not limited to this.

[0044] For different applications located on the same physical machine, for example, the first application and the second application, the two applications have their own memory space in the memory, respectively storing their own data. For example, if data A is stored in the first memory space of the first application and the first application needs to send data A to the second application, in the traditional RDMA communication process, data A is transferred from the first memory space to the RDMA network card via the PCIe bus ( Figure 1 Then, it is transmitted from the RDMA network card to the second memory space of the second application via the PCIe bus ( Figure 1 In line ②), during the entire data transmission process, two PCIe operations are required, consuming twice the PCIe bandwidth. For the data transmission process on the same physical machine, its transmission performance is not high and the transmission delay is increased.

[0045] Secondly, RDMA communication on the same physical machine includes both local traffic and network traffic. Local traffic refers to the traffic generated by communication between two RDMA applications on the same physical machine, while network traffic refers to the traffic generated by communication between two RDMA applications on different physical machines. Given that RDMA network cards handle local and network traffic differently, and because local traffic is not affected by congestion control algorithms and does not experience packet loss, resource allocation allocates more cache resources to network traffic and less to local traffic. Given the inherent imbalance in resource allocation, when local and network traffic compete for resources on the same RDMA network card, although local and network traffic are equivalent, network traffic stores more information. Therefore, during scheduling, network traffic, which consumes more resources, is prioritized, triggering resource preemption and squeezing out local traffic, resulting in unfairness.

[0046] In response to the above technical problems, in the embodiments of the present application, with compatibility with the RDMA programming interface and transparency or imperception to the application as the prerequisite, improvements are made to the RDMA communication between different applications on the same physical machine, so as to reduce the PCIe bandwidth resources consumed by RDMA communication between different RDMA applications on the same physical machine, reduce data transmission latency, and ensure fairness between local traffic and network traffic when competing for resources.

[0047] In order to be compatible with the RDMA programming interface and to be transparent or unaware to the application, in an embodiment of the present application, a local work queue and a remote work queue are simultaneously created during the RDMA connection process, and a same-host identification function is added. When it is identified that it is same-host RDMA communication, the local work queue is used to directly perform data copy operations in the registered memory of the two applications to replace the traditional RDMA transmission operation. For the application, it can directly initiate an RDMA communication request just like traditional RDMA communication. When it is compatible with the RDMA programming interface and is transparent or unaware to the application, the transmission delay between different RDMA applications on the same host can be reduced, the consumption of PCIe bandwidth can be reduced, and the fairness of local traffic and network traffic in resource competition can be ensured. Furthermore, when it is identified that it is cross-host RDMA communication, the remote work queue is used to perform traditional RDMA transmission operations without affecting the cross-host RDMA communication process.

[0048] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0049] The embodiment of the present application provides a distributed system, which can be a distributed storage system, such as various distributed database systems, or a distributed application system, such as a microservice system, etc., without limitation. The distributed system of this embodiment includes multiple physical machines, each of which runs a virtualization manager, which creates and manages virtualization instances, and in which RDMA applications are deployed. The RDMA applications are implemented based on traditional RDMA programming interfaces, and different RDMA applications can communicate with each other using RDMA technology. Figure 2a and Figure 2b The software and hardware architecture inside the physical machine shown illustrates the RDMA communication process between different RDMA applications provided in the embodiment of the present application.

[0050] Figure 2a This is a schematic diagram of the structure of a physical machine provided by an exemplary embodiment of the present application. Figure 2a As shown in Figure 1, the internal hardware and software architecture of the physical machine includes: application layer, kernel layer and hardware layer. The hardware layer includes various hardware resources of the physical machine. The hardware resources of the physical machine include at least CPU, memory and RDMA network card. In addition, the hardware resources of the physical machine also include communication components, display, power components, audio components and various external devices. Figure 2a In the embodiment, the RDMA network card 11 is shown, but it is not limited thereto. In this embodiment, a kernel layer runs on the hardware resources of the physical machine, and the kernel layer includes at least an operating system. Above the operating system is an application layer, which includes various virtualization instances running on the physical machine, and each virtualization instance has an application deployed therein. Figure 2aIn the example, a virtualization instance is illustrated using containers A and B as examples, with a first application deployed in container A and a second application deployed in container B. In contrast to applications, an operating system is a set of interrelated system software programs that manage and control various operations on a physical machine, utilize and run hardware and software resources, and provide public services to organize user interactions. In the embodiments of this application, the operating system acts as a bridge between applications and hardware resources. The operating system includes drivers for various hardware resources, which are used to drive these resources.

[0051] In an embodiment of the present application, an RDMA management program is also deployed on the physical machine, and the RDMA management program includes an interface management module 13 located in user state and a driver management module 12 located in kernel state. User state and kernel state are two different privilege levels in a computer system. The permissions of programs in user state are lower and restricted, generally referring to the running state of program code in the application layer; while programs in kernel state have higher permissions and can directly access and operate the core functions and hardware resources of the operating system, generally referring to the running state of program code in the kernel layer. User state is executed in user space, and kernel state is executed in kernel space. The two can be switched through system calls.

[0052] In the above distributed system, including the communication between different applications on the same host, also includes the communication between cross-host applications. In this embodiment, combined with Figure 2a and Figure 2b The internal architecture of the physical machine shown takes the communication process between a first application and a second application as an example to illustrate the communication between different applications on the same host and the communication between applications across hosts.

[0053] In this embodiment, the driver management module 12 and the interface management module 13 can cooperate to complete the RDMA communication between the first application and the second application. As a user-state library of RDMA technology, the interface management module can provide RDMA programming interfaces for RDMA applications (e.g., the first application and the second application) for initiating RDMA communication. These RDMA programming interfaces include but are not limited to: ibv_modify_qp (for modifying QP), ibv_reg_mr (for memory registration), ibv_create_qp (for creating QP queues), ibv_post_send (for issuing work requests), ibv_post_recv (for issuing work requests for receive operations), and ibv_poll_cq (for obtaining completion queue elements). These RDMA programming interfaces can be divided into two categories: one is the programming interfaces required for RDMA connection establishment, such as ibv_create_cq, ibv_create_qp, ibv_modiy_qp, and ibv_reg_mr. These programming interfaces, also known as control-plane programming interfaces, are used for preparatory work before data transfer, such as link establishment, memory registration, resource configuration and management between applications; the other is the programming interfaces for RDMA data transfer, such as ibv_post_recv, ibv_post_send, and ibv_poll_cq. These programming interfaces, also known as data-plane programming interfaces, are used for RDMA data transfer operations, such as sending and receiving data and monitoring the completion status of operations. Some of these interfaces are prefixed with "ibv" (for user mode). These interfaces are part of the InfiniBand architecture, which provides a set of user-space (user-mode) programming interfaces for developing high-efficiency, low-latency network communication applications.

[0054] For the first application, RDMA communication can be initiated to the second RDMA application through the RDMA programming interface provided by the interface management module 13. A complete RDMA communication process includes a control plane process and a data plane process. Specifically, the first application can initiate a call to the programming interface for RDMA connection establishment to request RDMA connection establishment with the second application; after the RDMA connection operation is completed, the first application can initiate a call to the programming interface for RDMA data transmission to perform RDMA data transmission with the second application. In an embodiment of the present application, the RDMA connection process and the RDMA data transmission process are improved, but the improvement is not perceived by the first application and the second application. Both the operation of initiating RDMA connection establishment and the operation of initiating RDMA data transmission are the same as the traditional RDMA communication process.

[0055] In this embodiment, the interface management module 13 can respond to the first application's call to each RDMA programming interface, and cooperate with the driver management module 12 to complete the RDMA communication process between the first application and the second application based on the call. Specifically, the interface management module 13 can respond to the first application's call to the programming interface for RDMA connection establishment, and then call the driver management module 12 to establish an RDMA connection between the first application and the second application. The RDMA connection establishment process of this embodiment is different from the traditional RDMA connection establishment process. In this embodiment, the driver management module 12 provides a system call interface for the interface management module 13, such as an application binary interface (ABI). The interface management module 13 can issue a call request to the driver management module 12 through the system call interface provided by the driver management module 12. The driver management module 12 is responsible for processing user-mode system call requests through the system call interface (such as ABI).

[0056] Specifically, the driver management module 12 is used to respond to the call of the interface management module 13 when the first application initiates an RDMA communication request to the second application, create a first RDMA work queue for the first application, and perform memory registration. The memory registration generates memory index information, where the memory index information includes LKey (Local Key, local memory index information) and RKey (Remote Key, remote memory index information), which are used to protect access rights to the local and remote memory areas, respectively. Through memory registration, the memory region (Memory Region, MR) of the first application can be obtained. MR refers to a memory area requested by the driver management module 12 for the first application in the memory of the physical machine to which it belongs, used to store sent and received data.

[0057] The driver management module 12 is further configured to respond to a call from the interface management module 13 and, based on the control plane connection between the first and second applications, obtain description information of a second RDMA work queue corresponding to the second application as context information for the first RDMA work queue. The first and second RDMA work queues each include a local work queue and a remote work queue, wherein the local work queue is used for intra-host RDMA communication, and the remote work queue is used for inter-host RDMA communication. Optionally, the local work queue includes a local CQ queue (Completion Queue) and a local QP queue (QueuePair), and the remote work queue includes a remote CQ queue and a remote QP queue. It is explained here that a control plane link can be established between the first application and the second application. Compared with the RDMA connection, the control plane connection between the first application and the second application can be regarded as an out-of-band communication method. The description information of the first RDMA work queue and the description information of the second RDMA work queue can be exchanged between the first application and the second application through the control plane link; the driver management module 12 on the physical machine where the first application is located can obtain the description information of the second RDMA work queue as the context information of the first RDMA work queue in this process; similarly, the driver management module 12 on the physical machine where the second application is located can obtain the description information of the first RDMA work queue as the context information of the second RDMA work queue in this process.

[0058] In this embodiment, the implementation method of the control plane connection between the first application and the second application is not limited. For example, it can be a TCP connection or a UDP connection, and there is no limitation on this. Furthermore, the control plane connection between the first application and the second application can be established based on the socket interface (Socket API) or the communication management interface (Communication Manager API, CM API). This application does not limit this. In this embodiment, establishing a connection generally refers to establishing a connection between the remote QP queue in the first RDMA work queue and the remote QP queue in the second RDMA work queue. Based on this connection, the description information in the first RDMA work queue and the second RDMA work queue can be exchanged, and the description information includes but is not limited to the queue pair number QPN (Queue Pair Number) and the global identity GID (Global Identifier).

[0059] After exchanging description information based on the control plane connection at both ends, the first application and the second application can conduct RDMA communication based on their respective registered memory areas. At this time, the interface management module 13 can respond to the first application's call to the programming interface for RDMA data transmission and perform different processing depending on whether it is intra-host RDMA communication. If it is intra-host RDMA communication, the interface management module 13 triggers the driver management module 12 to enter kernel mode, and the driver management module 12 converts the RDMA data transmission process into a local memory copy operation. If it is inter-host RDMA communication, the interface management module 13 directly performs traditional RDMA operations in user mode, that is, sends the data through the RDMA network card.

[0060] In this embodiment, the driver management module 12 is further configured to identify, based on the description information of the second RDMA work queue, whether the first application and the second application perform intra-host RDMA communication; and, in the case where intra-host RDMA communication is identified, respond to a call from the interface management module 13 and, based on the local work queue of at least one of the two applications, perform a data copy operation in the memory registered by the two applications, to replace the RDMA data transmission operation between the two applications.

[0061] Furthermore, the driver management module 12 is also configured to provide the interface management module 13 with an identification result of whether the communication is intra-host RDMA communication. The interface management module 13 is further configured to: if the identification result is intra-host RDMA communication, respond to the first application's call to the programming interface for RDMA data transmission, trigger the driver management module 12 to enter kernel mode, and have the driver management module 12 convert the RDMA data transmission process into a local memory copy operation; and if the identification result is inter-host RDMA communication, perform an RDMA data transmission operation between the memories registered by the two applications via the RDMA network card based on the remote work queue of at least one of the two applications. It should be noted that, if the identification result is inter-host RDMA communication, the RDMA-based communication process between the two applications is the same as the traditional RDMA communication process.

[0062] In the embodiment of the present application, when the identification result is RDMA communication with the host, the implementation method of the interface management module 13 triggering the driver management module 12 to enter the kernel state is not limited. In an optional embodiment, the interface management module 13 may issue a call request to the driver management module 12 through a system call interface provided by the driver management module 12, so as to notify the driver management module 12 to perform a data copy operation in the kernel state based on the local work queue of at least one of the two applications in the memory registered by the two applications, thereby replacing the RDMA data transmission operation between the two applications. In another optional embodiment, in the process of creating the work queue, the driver management module 12 creates a command queue and shares the command queue with the interface management queue. Specifically, when the interface management module 13 identifies that the communication is intra-host RDMA communication, in response to the first application's call to the programming interface for RDMA data transmission, it writes a local memory copy command to the command queue. The driver management module 12 periodically polls the command queue. When the local memory copy command written by the interface management module 13 appears in the command queue, it can be determined that the communication between the first application and the second application is intra-host RDMA communication. Therefore, based on the local work queue of at least one of the two applications, a data copy operation is performed in the memory registered by the two applications, thereby replacing the RDMA data transmission operation between the two applications.

[0063] In an alternative embodiment, the command queue can be a local SQ queue in the local work queue, and the local memory copy command is the SQE. In another alternative embodiment, the command queue can also be an independent queue. A separate command queue can be created at the same time as the first RDMA work queue is created. It should be noted that in this embodiment, the first application is used as the sender, so the command queue is created at the same time as the first RDMA queue is created. In actual situations, if the second application acts as the sender, the command queue should be created at the same time as the second RDMA work queue.

[0064] In an optional embodiment, when same-host RDMA communication is identified, the first application and the second application are applications deployed in different virtualized instances on the same physical machine. That is, although the two applications are deployed on the same physical machine, they are deployed in different independent operating environments (i.e., virtualized instances) through virtualization technology, such as virtual machines or containers. In this case, the second application and the first application are deployed on the same physical machine, such as Figure 2aAs shown, the RDMA management program includes an interface management module 13 corresponding to the first application and an interface management module 13 corresponding to the second application. The interface management module 13 of the second application is functionally identical to the interface management module 13 of the first application, performing both interface management functions when acting as a sender and when acting as a receiver. In actual applications, if the corresponding application acts as a sender, it performs the interface management functions when acting as a sender; if the corresponding application acts as a receiver, it performs the interface management functions when acting as a receiver. In this embodiment, since the second application acts as a receiver, the interface management module 13 corresponding to the second application is responsible for responding to the second application's request to create a second RDMA work queue and memory registration request, and initiating calls to the driver management module 12. Correspondingly, the driver management module 12 is also used to: when the first application initiates an RDMA communication request to the second application, respond to the call of the interface management module 13 of the second application, create a second RDMA work queue for the second application and register it in memory, and save the index information of the registered memory; and based on the control plane connection between the first application and the second application, obtain the description information of the first RDMA work queue as the context information of the second RDMA work queue.

[0065] In the embodiment of the present application, the internal implementation structure of the interface management module 13 is not limited. Any internal implementation structure that can complete the functions implemented by the interface management module 13 described above is applicable to the embodiment of the present application. Similarly, the embodiment of the present application does not limit the internal implementation structure of the driver management module 12. Any internal implementation structure that can complete the functions implemented by the driver management module 12 described above is applicable to the embodiment of the present application. In the following embodiments, Figure 2b As shown, an internal implementation structure of the interface management module 13 and the driver management module 12 is provided.

[0066] In an optional embodiment, the interface management module 13 includes: a resource management unit 131 for providing management plane functions of RDMA communication and a data transceiver unit 132 for providing data plane functions of RDMA communication; accordingly, the driver management module 12 includes: a transmission service unit 122 for providing communication services based on local shared memory copies for same-host RDMA communication and a same-host identification unit 121 for identifying whether it is same-host RDMA communication.

[0067] The resource management unit 131 is configured to respond to a completion queue (CQ) creation request, a queue pair (QP) creation request, and a memory registration request initiated by the first application, and initiate calls to the transmission service unit 122 respectively. Accordingly, the transmission service unit 122 is configured to respond to the call of the resource management unit 131, create a local CQ queue and a remote CQ queue, a local QP queue, and a remote QP queue for the first application, register memory for the first application, and save registered memory index information. The local work queue includes a local CQ queue and a local QP queue, the remote work queue includes a remote CQ queue and a remote QP queue, the local QP queue includes a local send queue (SQ) and a local receive queue (RQ), and the remote QP queue includes a remote SQ queue and a remote RQ queue.

[0068] In the embodiment of the present application, the interface management module 13 operates in user mode, while the driver management module 12 operates in kernel mode. When a user-mode application needs to perform privileged operations (such as interacting directly with hardware), it must "fall into" kernel mode via system calls or other means to complete the operation. For example, a call from the resource management unit 131 to the transmission service unit 122 is an interaction between user mode and kernel mode. The resource management unit 131 can access the kernel-mode transmission service unit 122 through a system call to prepare for data transmission.

[0069] Further optionally, as Figure 3a As shown, the resource management unit 131 also includes: qp management submodule, mr management submodule and cq management submodule; the data transceiver unit 132 includes a query cq (poll_cq) submodule, a receiving (post_recv) submodule and a sending (post_send) submodule. Figure 4As shown, the transmission service unit 122 includes: a resource mapping submodule and a memory copy submodule. The resource mapping submodule corresponds to each submodule in the resource management unit 131 and is used to cooperate with each submodule in the resource management unit 131 to complete the RDMA connection process. During the RDMA connection process, the same-host identification unit 121 in the driver management module 12 is used to identify whether the communication between the first application and the second application is the same-host RDMA communication. Specifically, the same-host identification unit 121 determines the IP address of the second application based on the GID in the description information of the second RDMA work queue. In a single-tenant scenario, whether the communication between the first application and the second application is the same-host RDMA communication can be identified based on the routing table of the physical machine and the IP address of the second application. Specifically, if the IP address of the second application is included in the routing table, it is determined that the communication between the first application and the second application is the same-host RDMA communication; conversely, if the IP address of the second application is not included in the routing table, it is determined that the communication between the first application and the second application is cross-host RDMA communication. In a multi-tenant scenario, the physical machine includes a virtual switch (vSwitch). The virtual switch is used to virtualize the functions of the physical switch on the physical machine and divide a physical switch into multiple logical switches to meet the needs of multiple tenants. The virtual switch maintains a forwarding table, also known as a flow table. The flow table contains multiple entries, each of which includes at least a header field and an action. The header field stores information such as the switch ingress port, source MAC address, destination MAC address, source IP address, destination IP address, source port number, and destination port number. Based on the flow table of the virtual switch on the physical machine and the IP address of the second application, it is determined whether the first application and the second application are communicating with the same host through RDMA. Specifically, the IP address of the second application can be matched in the flow table to obtain information such as the source MAC address, destination MAC address, source IP address, destination IP address, source port number, and destination port number of the second application. Based on this information, it can be determined whether the first application and the second application are communicating with the same host through RDMA. The memory copy submodule corresponds to each submodule in the data transceiver unit 132 and is used to cooperate with each submodule in the data transceiver unit 132 to complete data transmission operations.

[0070] The resource mapping submodule corresponds to each submodule in the resource management unit 131 and is used to cooperate with each submodule in the resource management unit 131 to complete the RDMA connection process:

[0071] During the RDMA connection establishment process, as shown in Figure 3b, when the first application calls the create_cq programming interface, the cq management submodule of the resource management unit 131 accesses the resource mapping submodule of the transport service unit 122 through a system call. The resource mapping submodule then performs the following steps: creating a local CQ queue to store CQEs generated by intra-host RDMA communication; creating a remote CQ queue to store CQEs generated by cross-host RDMA communication; and establishing a mapping between the local CQ queue and the remote CQ queue to support cross-host RDMA communication. The order of creating the local and remote CQ queues is not restricted.

[0072] Furthermore, as shown in Figure 3c, when the first application calls the create_qp programming interface, the qp management submodule of the resource management unit 131 accesses the resource mapping submodule of the transport service unit 122 through a system call to perform the following steps: creating local QP queues (including local SQ queues and local RQ queues) for storing send and receive task elements in intra-host RDMA communications; creating remote QP queues for storing send and receive task elements in cross-host RDMA communications; and establishing a mapping between local QP queues and remote QP queues to support cross-host RDMA communications. Optionally, a separate command queue can also be created. The order in which local and remote QP queues are created is not limited.

[0073] The remote work queue has a queue pair number (QPN) and a global identifier (GID). The description of the RDMA work queue to which the remote work queue belongs includes the remote work queue's QPN and GID. The GID is a network layer address used to uniquely identify each node (e.g., a physical machine) in the network. It serves the same purpose as the network layer address in the Transmission Control Protocol / Internet Protocol (TCP / IP), i.e., the IP address (Internet Protocol Address). The QPN is the unique number of a node's remote QP queue. By combining the GID and QPN, a QP queue can be uniquely identified in the network.

[0074] To achieve unawareness of RDMA applications, the local work queue can be either QPN and GID-free, or it can share the QPN and GID of the remote work queue within its RDMA work queue. From the application's perspective, the local and remote work queues are abstracted as the same RDMA work queue, which has a unified QPN and GID for applications. This ensures compatibility with traditional RDMA programming interfaces and obscures application awareness.

[0075] If the peer qp has been created, such as Figure 3d As shown, when the first application calls the modify_qp programming interface, the qp management submodule of the resource management unit 131 accesses the same-host identification unit 121 of the transmission service unit 122 through a system call to perform the following steps: Based on the GID in the description information of the QP queue of the second application, it determines whether the communication is same-host RDMA communication; if it is same-host RDMA communication, the identification result of same-host RDMA communication is saved to the resource management unit 131; optionally, if it is cross-host RDMA communication, the identification result of cross-host RDMA communication is saved to the resource management unit 131, so that the interface management module 13 can use the remote work queue to perform traditional RDMA transfer operations if the communication is cross-host communication. Of course, it is also possible to save only the identification result of same-host RDMA communication and not the identification result of cross-host RDMA communication. In this way, if there is no identification result of same-host RDMA communication, the default is cross-host RDMA communication.

[0076] like Figure 3e As shown, when the first application calls the reg_mr programming interface to register memory, the mr management submodule of the resource management unit 131 will enter the resource mapping submodule in the transmission service unit 122 through a system call, and perform the following steps through the resource mapping submodule: perform the memory registration operation corresponding to the same-host RDMA communication, map the registered memory area to its own kernel address space, and save the memory index information (LKey and RKey); perform cross-host RDMA communication to register the same memory area so that the local QP queue and the remote QP queue can use the same memory area for data transmission and reception. Optionally, in the case of same-host RDMA communication, the second application will also call reg_mr to register memory. The registration process is similar to the above steps, except that the mr management submodule of the resource management unit 131 of the second application will enter the resource mapping submodule in the transmission service unit 122 through a system call, and the resource mapping submodule will complete the memory registration operation for the second application.

[0077] It should be noted that during the RDMA connection establishment process, create_cq is executed before create_qp, and there is no restriction on the order between reg_mr and create_cq and create_qp.

[0078] The process in which the transmission service unit 122 and the data transceiver unit 132 cooperate with each other to complete the RDMA data transmission operation with the host is as follows:

[0079] When the data transceiver unit 132 identifies intra-host RDMA communication, it responds to the data transmission task initiated by the first application, generates a send queue entry (SQE), which is used to describe information about the transmission task, writes the SQE into the local SQ queue in the local work queue of the first application, and triggers the transmission service unit 122. Optionally, the data transceiver unit 132 triggers the transmission service unit 122 by: after writing the SQE into the local SQ queue in the local work queue of the first application, initiating a system call to the transmission service unit 122; or, after writing the SQE into the local work queue of the first application, writing a local memory copy command into a command queue, so that the transmission service unit 122 polls the command queue to find that the command queue contains the local memory copy command; wherein the command queue is shared by the data transceiver unit 132 and the transmission service unit 122. Accordingly, the transmission service unit 122, under the triggering of the data transceiver unit 132, utilizes the local work queue of the first application or the local work queues of the transceiver and the second application (the first application and the second application) to perform a data copy operation in the memory registered by the two applications according to the operation type of the data sending task, so as to replace the RDMA data transmission operation between the two applications; wherein, in this embodiment, the first application is the sending end and the second application is the receiving end.

[0080] In an optional embodiment, the transmission service unit 122 is specifically used to: obtain an SQE from the local SQ queue of the first application, and obtain the operation type of the data sending task from the SQE; if the type of the data sending task is a bilateral operation, use the local work queues at both ends of the sending and receiving to perform a data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications; if the type of the data sending task is a unilateral operation, use the local work queue of the first application to perform a data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications.

[0081] Further optionally, when the data sending task is a bilateral operation, the data transceiver unit 132 of the second application is further configured to: generate a receive queue element (RQE) in response to the data receiving task initiated by the second application, the RQE describing the receiving task information, and write the RQE into the local RQ queue in the local work queue of the second application. Accordingly, the transmission service unit 122 is specifically configured to: determine the description information of the RDMA work queue of the second application based on the context information of the RDMA work queue of the first application; read the SQE from the local SQ queue of the first application, and read the RQE from the local RQ of the second application based on the description information of the RDMA work queue of the second application; copy the data in the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE to the memory pointed to by the local memory index information (LKey of the second application) contained in the RQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queues in the local work queues of the first application and the second application, respectively. In bilateral operations, both the sending and receiving ends generate CQEs and write them into their respective local CQ queues.

[0082] Optionally, in the case where the type of the data sending task is a unilateral operation, the transmission service unit 122 is specifically used to: if the operation type of the data sending task is a unilateral write operation, read the SQE from the local SQ queue of the first application, and copy the data in the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE to the memory pointed to by the remote memory index information (RKey of the second application) contained in the SQE; if the operation type of the data sending task is a unilateral read operation, read the SQE from the local SQ queue of the first application, and copy the data in the memory pointed to by the remote memory index information (RKey of the second application) contained in the SQE to the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queue in the local work queue of the first application respectively. Among them, in unilateral operation, the sending end can generate CQE on demand, while the receiving end does not generate it. In a special case, if the operation type of the data sending task is a write operation with immediate value (Write with Immediate), the receiving end will generate CQE.

[0083] In the embodiment of the present application, the RDMA application can query whether there are CQEs in the local CQ queue and the remote CQ queue by calling the data transceiver unit 132, and return the query result to the RDMA application. The local CQ queue is used for intra-host RDMA communication, and the CQE generated after the copy is completed is placed in the local CQ queue. The remote CQ queue is used for inter-host RDMA communication, and the CQE generated after the copy is completed is placed in the remote CQ queue.

[0084] Further, combined Figure 4 The memory copy submodule in the transmission service unit 122 is shown as Figure 3a The corresponding submodules in the data transceiver unit 132 are used to cooperate with the submodules in the data transceiver unit 132 to complete the data transmission operation process as follows:

[0085] like Figure 5 As shown, when the first application calls the post_send programming interface, the post_send submodule in the data transceiver unit 132 determines that it is RDMA communication with the same host based on the same-host identification result, responds to the data sending task initiated by the first application, generates an SQE, writes the SQE into the local SQ queue in the local work queue of the first application, and triggers the memory copy submodule of the transmission service unit 122. Correspondingly, the memory copy submodule of the transmission service unit 122, under the triggering of the post_send submodule in the data transceiver unit 132, will perform the following steps: obtain the SQE from the local SQ queue of the first application, and obtain the operation type of the data sending task from the SQE; when the type of the data sending task is a bilateral operation, determine the description information of the RDMA work queue of the second application according to the context information of the RDMA work queue of the first application; read the SQE from the local SQ queue of the first application, and read the RQE from the local RQ of the second application according to the description information of the RDMA work queue of the second application; copy the data in the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE to the memory pointed to by the local memory index information (LKey of the second application) contained in the RQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queues in the local work queues of the first application and the second application respectively. In bilateral operations, both the sending and receiving ends generate CQEs and write them into their respective local CQ queues.

[0086] In the case where the type of the data sending task is a unilateral operation, if the operation type of the data sending task is a unilateral write operation, the SQE is read from the local SQ queue of the first application, and the data in the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE is copied to the memory pointed to by the remote memory index information (RKey of the second application) contained in the SQE; if the operation type of the data sending task is a unilateral read operation, the SQE is read from the local SQ queue of the first application, and the data in the memory pointed to by the remote memory index information (RKey of the second application) contained in the SQE is copied to the memory pointed to by the local memory index information (LKey of the first application) contained in the SQE; after completing the data copy, a CQE is generated, and the CQE is written into the local CQ queue in the local work queue of the first application. Among them, in unilateral operation, the sending end can generate CQE on demand, while the receiving end does not generate it. In a special case, if the operation type of the data sending task is a write operation with immediate value (Write with Immediate), the receiving end will generate CQE.

[0087] like Figure 6 As shown, when the first application calls the post_send programming interface to communicate with the second application, when the second application calls the post_recv programming interface, the post_recv submodule in the data transceiver unit 132 of the second application is also used to: respond to the data receiving task initiated by the second application, generate an RQE, write the RQE into the local RQ queue in the local work queue of the second application, and prepare to receive the data sent by the first application; after receiving the data sent by the first application, copy the data to the memory pointed to by the local memory index information (LKey of the second application) contained in the RQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queue in the local work queue of the second application.

[0088] In the embodiment of the present application, whether it is the first application or the second application, the poll_cq submodule in the receipt transceiver unit 132 queries whether there is a CQE in the local CQ queue and the remote CQ queue, and returns the query result to the corresponding application. Figure 7As shown, when any application calls the poll_cq interface, the poll_cq submodule in the data transceiver unit 132 will perform the following steps: check whether there is a CQE in the local CQ queue, and then check whether there is a CQE in the corresponding remote CQ queue, and return them to the application. Among them, the local CQ queue is used for RDMA communication with the same host. The CQE generated after the copy is completed is placed in the local CQ queue. The remote CQ queue is used for RDMA communication across hosts. The CQE generated after the copy is completed is placed in the remote CQ queue. It is explained here that the poll_cq submodule checks the local CQ queue and the remote CQ queue without going through the kernel, and can poll the local CQ queue and the remote CQ queue regularly, or can check the local CQ queue and the remote CQ queue in response to the call of the poll_cq interface. In addition, there is no limit on the order of checking the local CQ queue and the remote CQ queue.

[0089] Optionally, when performing a data copy operation in the memories registered by two applications, the transmission service unit 122 (specifically the memory copy submodule in the transmission service unit 122) is specifically used to: execute the memory copy command of the processor CPU, and copy data from the memory registered by one application to the memory registered by another application. In this method, CPU instructions are used to perform memory copy, which usually involves load and store instructions, which will occupy the CPU core.

[0090] Alternatively, if the CPU has a Direct Memory Access (DMA) acceleration engine, the CPU's DMA acceleration engine can be used to directly copy data from a memory registered by one application to a memory registered by another application without occupying the CPU core.

[0091] Figure 8 A flow chart of a data transmission method based on RDMA provided by an exemplary embodiment of the present application is applied to a physical machine on which a first application is deployed. For the description of the physical machine, please refer to the aforementioned system embodiment and will not be repeated here. Figure 8 As shown, the method includes:

[0092] S801: When a first application initiates an RDMA communication request to a second application, a first RDMA work queue is created for the first application and registered in memory, where the first RDMA work queue includes a local work queue and a remote work queue;

[0093] S802: Based on the control plane connection between the first application and the second application, obtain description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue;

[0094] S803: Identify, based on description information of the second RDMA work queue, whether the first application and the second application perform intra-host RDMA communication, where the second RDMA work queue also includes a local work queue and a remote work queue.

[0095] S804: When it is identified as intra-host RDMA communication, a data copy operation is performed in the memory registered by the two applications based on the local work queue of at least one of the two applications, to replace the RDMA data transmission operation between the two applications.

[0096] In an optional embodiment, when it is identified as same-host RDMA communication, the first application and the second application are applications deployed in different virtualization instances on the physical machine; the method further includes: when the first application initiates an RDMA communication request to the second application, creating a second RDMA work queue for the second application and registering it in memory; and, based on the control plane connection, obtaining description information of the first RDMA work queue as context information of the second RDMA work queue.

[0097] In an optional embodiment, the method further includes: when identified as cross-host RDMA communication, based on the remote work queue of at least one of the two applications, performing RDMA data transmission operations between the memories registered by the two applications through the RDMA network card on the physical machine.

[0098] In an optional embodiment, when a first application initiates an RDMA communication request to a second application, a first RDMA work queue is created for the first application and memory registration is performed, including: responding to a completion queue CQ creation request initiated by the first application, creating a local CQ queue and a remote CQ queue for the first application; responding to a queue pair QP creation request initiated by the first application, creating a local QP queue and a remote QP queue for the first application; responding to a memory registration request initiated by the first application, registering memory for the first application, and saving index information of the registered memory; wherein, the local work queue includes a local CQ queue and a local QP queue, and the remote work queue includes a remote CQ queue and a remote QP queue.

[0099] In an optional embodiment, the remote work queue has a queue pair number QPN and a global identity GID, and the description information of the RDMA work queue to which the remote work queue belongs includes the QPN and GID of the remote work queue; the local work queue does not have a QPN and GID, or the local work queue shares the QPN and GID of the remote work queue in the RDMA work queue to which it belongs.

[0100] In an optional embodiment, based on the description information of the second RDMA work queue, identifying whether the first application and the second application are communicating with the same host via RDMA, includes: determining the IP address of the second application based on the GID in the description information of the second RDMA work queue; in a single-tenant scenario, identifying whether the first application and the second application are communicating with the same host via RDMA based on the routing table of the physical machine and the IP address of the second application; in a multi-tenant scenario, identifying whether the first application and the second application are communicating with the same host via RDMA based on the flow table of the virtual switch on the physical machine and the IP address of the second application.

[0101] On the basis of the above embodiment, optionally, when it is identified as intra-host RDMA communication, based on the local work queue of at least one of the two applications, a data copy operation is performed in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications, including: when it is identified as intra-host RDMA communication, in user state, responding to the data sending task initiated by the sending end, generating an SQE, and writing the SQE into the local SQ queue in the local work queue of the sending end; entering kernel state from user state, and in kernel state, according to the operation type of the data sending task, using the local work queue of the sending end or the local work queue of the sending and receiving ends, performing a data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications; wherein the sending end is any application of the two applications, and the other application is the receiving end.

[0102] In an optional embodiment, the method of entering kernel state from user state includes: responding to a system call initiated by the user state after writing the SQE into the local SQ queue in the local work queue of the sending end, entering kernel state from user state; or polling a command queue shared by the user state and the kernel state, and entering kernel state from user state when the command queue polled contains a local memory copy command; wherein the local memory copy command is written into the command queue after the user state writes the SQE into the local SQ queue in the local work queue of the sending end.

[0103] In an optional embodiment, in kernel state, according to the operation type of the data sending task, a data copy operation is performed in the memory registered by the two applications using the local work queue of the sending end or both the sending and receiving ends, so as to replace the RDMA data transmission operation between the two applications, including: in kernel state, obtaining the SQE from the local SQ queue of the sending end, and obtaining the operation type of the data sending task from the SQE; if the type of the data sending task is a bilateral operation, using the local work queue of the sending and receiving ends, performing the data copy operation in the memory registered by the two applications, so as to replace the RDMA data transmission operation between the two applications; if the type of the data sending task is a unilateral operation, using the local work queue of the sending end, performing the data copy operation in the memory registered by the two applications, so as to replace the RDMA data transmission operation between the two applications.

[0104] In an optional embodiment, when the type of the data sending task is a bilateral operation, it also includes: in user mode, responding to the data receiving task initiated by the receiving end, generating an RQE, and writing the RQE into the local RQ queue in the local work queue of the receiving end; using the local work queues at both ends of the sending and receiving end, performing a data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications, including: determining the description information of the RDMA work queue of the receiving end based on the context information of the RDMA work queue of the sending end; reading the SQE from the local SQ queue of the sending end, and reading the RQE from the local RQ of the receiving end based on the description information of the RDMA work queue of the receiving end; copying the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the RQE; after completing the data copy, generating a CQE, and writing the CQE into the local CQ queues in the local work queues of the sending end and the receiving end respectively.

[0105] In an optional embodiment, a data copy operation is performed in the memory registered by the two applications using the local work queue of the sending end to replace the RDMA data transmission operation between the two applications, including: if the operation type of the data sending task is a one-sided write operation, the SQE is read from the local SQ queue of the sending end, and the data in the memory pointed to by the local memory index information contained in the SQE is copied to the memory pointed to by the remote memory index information contained in the SQE; if the operation type of the data sending task is a one-sided read operation, the SQE is read from the local SQ queue of the sending end, and the data in the memory pointed to by the remote memory index information contained in the SQE is copied to the memory pointed to by the local memory index information contained in the SQE; after the data copy is completed, a CQE is generated, and the CQEs are written into the local CQ queues in the local work queue of the sending end respectively.

[0106] In an optional embodiment, performing a data copy operation in memories registered by two applications includes: executing a memory copy command of a processor CPU to copy data from a memory registered by one application to a memory registered by another application; or using a DMA acceleration engine of a CPU to copy data from a memory registered by one application to a memory registered by another application.

[0107] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 801 to 803 can be device A; for another example, the execution entity of steps 801 and 802 can be device A, and the execution entity of step 803 can be device B; and so on.

[0108] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 801, 802, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0109] Figure 9 The following is a schematic diagram of a data transmission device based on RDMA provided by an exemplary embodiment of the present application. Figure 9 As shown, the apparatus 900 includes: a creation module 91, an acquisition module 92, an identification module 93 and a copy module 94, wherein:

[0110] A creation module 91 is configured to create a first RDMA work queue for the first application and register it in memory when the first application initiates an RDMA communication request to the second application, wherein the first RDMA work queue includes a local work queue and a remote work queue;

[0111] an acquisition module 92 configured to acquire, based on a control plane connection between the first application and the second application, description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue;

[0112] an identification module 93 for identifying whether the first application and the second application perform same-host RDMA communication based on description information of the second RDMA work queue, where the second RDMA work queue also includes a local work queue and a remote work queue;

[0113] The copy module 94 is configured to perform a data copy operation in the memory registered by the two applications based on the local work queue of at least one of the two applications when identifying intra-host RDMA communication, to replace the RDMA data transmission operation between the two applications.

[0114] Further optionally, in the case of identification of same-host RDMA communication, the first application and the second application are applications deployed in different virtualization instances on the physical machine, and the creation module 91 is further used to: when the first application initiates an RDMA communication request to the second application, create a second RDMA work queue for the second application and register it in memory; the acquisition module 92 is used to: based on the control plane connection, obtain the description information of the first RDMA work queue as the context information of the second RDMA work queue.

[0115] In an optional embodiment, the copy module 94 is further used to: when identified as cross-host RDMA communication, based on the remote work queue of at least one of the two applications, perform RDMA data transmission operations between the memories registered by the two applications through the RDMA network card on the physical machine.

[0116] In an optional embodiment, when the first application initiates an RDMA communication request to the second application, the creation module 91 creates a first RDMA work queue for the first application and registers the memory, and is specifically used to: respond to the completion queue CQ creation request initiated by the first application, and create a local CQ queue and a remote CQ queue for the first application; respond to the queue pair QP creation request initiated by the first application, and create a local QP queue and a remote QP queue for the first application; respond to the memory registration request initiated by the first application, register the memory for the first application, and save the index information of the registered memory; wherein, the local work queue includes the local CQ queue and the local QP queue, and the remote work queue includes the remote CQ queue and the remote QP queue.

[0117] Optionally, the remote work queue has a queue pair number QPN and a global identity GID, and the description information of the RDMA work queue to which the remote work queue belongs includes the QPN and GID of the remote work queue; the local work queue does not have a QPN and GID, or the local work queue shares the QPN and GID of the remote work queue in the RDMA work queue to which it belongs.

[0118] In an optional embodiment, when the identification module 93 identifies whether the first application and the second application are communicating with the same host via RDMA based on the description information of the second RDMA work queue, the identification module 93 is specifically used to: determine the IP address of the second application based on the GID in the description information of the second RDMA work queue; in a single-tenant scenario, identify whether the first application and the second application are communicating with the same host via RDMA based on the routing table of the physical machine and the IP address of the second application; in a multi-tenant scenario, identify whether the first application and the second application are communicating with the same host via RDMA based on the flow table of the virtual switch on the first physical machine and the IP address of the second application.

[0119] In an optional embodiment, the apparatus of this embodiment further comprises: a data transceiver module in user state, configured to, in user state, respond to a data transmission task initiated by a sender, generate an SQE, and write the SQE into a local SQ queue in a local work queue of the sender, when identifying intra-host RDMA communication. A copy module 94, when identifying intra-host RDMA communication, is configured to, based on the local work queue of at least one of the two applications, perform a data copy operation in the memory registered by the two applications, replacing the RDMA data transfer operation between the two applications, by entering kernel state from user state and, in kernel state, performing a data copy operation in the memory registered by the two applications, using the local work queue of the sender or both the sender and the receiver, based on the operation type of the data transmission task, replacing the RDMA data transfer operation between the two applications; wherein the sender is either of the two applications, and the other application is the receiver.

[0120] In an optional embodiment, the copy module 94 is specifically used to enter the kernel state from the user state in a manner that: responds to a system call initiated after the user state data transceiver module writes the SQE into the local SQ queue in the local work queue of the sending end, and enters the kernel state from the user state; or, polls the command queue shared by the user state and the kernel state, and enters the kernel state from the user state when the command queue contains a local memory copy command; wherein the local memory copy command is written into the command queue after the user state data transceiver module writes the SQE into the local SQ queue in the local work queue of the sending end.

[0121] In an optional embodiment, the copy module 94 is specifically used in the kernel state to: obtain the SQE from the local SQ queue of the sender end and obtain the operation type of the data sending task from the SQE; if the type of the data sending task is a bilateral operation, use the local work queues at the sender end or the receiver end to perform the data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications; if the type of the data sending task is a unilateral operation, use the local work queue of the sender end to perform the data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications.

[0122] In an optional embodiment, when the type of the data sending task is a bilateral operation, the copy module 94 is further configured to: in user mode, respond to the data receiving task initiated by the receiving end, generate an RQE, and write the RQE into a local RQ queue in a local work queue of the receiving end; when using the local work queues at both ends of the sending and receiving end to perform a data copy operation in memories registered by two applications to replace the RDMA data transmission operation between the two applications, specifically configured to: determine description information of the RDMA work queue of the receiving end based on context information of the RDMA work queue of the sending end; read the SQE from the local SQ queue of the sending end, and read the RQE from the local RQ of the receiving end based on the description information of the RDMA work queue of the receiving end; copy the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the RQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queues in the local work queues of the sending end and the receiving end, respectively.

[0123] In an optional embodiment, when the copy module 94 uses the local work queue of the sending end to perform a data copy operation in the memory registered by two applications to replace the RDMA data transmission operation between the two applications, it is specifically used to: if the operation type of the data sending task is a one-sided write operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the remote memory index information contained in the SQE; if the operation type of the data sending task is a one-sided read operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the remote memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the SQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queue in the local work queue of the sending end respectively.

[0124] In an optional embodiment, when the copy module 94 performs a data copy operation in the memories registered by two applications, it is specifically used to: execute the memory copy command of the processor CPU to copy data from the memory registered by one application to the memory registered by another application; or use the DMA acceleration engine of the CPU to copy data from the memory registered by one application to the memory registered by another application.

[0125] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0126] Figure 10 This is a schematic diagram of a physical machine structure provided by an exemplary embodiment of the present application. Figure 5 As shown, the device includes: one or more memories 104 and one or more processors 105 .

[0127] Memory 104 is used to store computer programs and can be configured to store various other data to support operations on the physical machine. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0128] The memory 104 may be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0129] The processor 105 is coupled to the memory 104 and is configured to execute a computer program in the memory 104, and is configured to: create a first RDMA work queue for the first application and register it in memory when a first application initiates an RDMA communication request to a second application, wherein the first RDMA work queue includes a local work queue and a remote work queue; obtain, based on a control plane connection between the first application and the second application, description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue; identify, based on the description information of the second RDMA work queue, whether same-host RDMA communication is between the first application and the second application, wherein the second RDMA work queue also includes a local work queue and a remote work queue; and, if same-host RDMA communication is identified, perform a data copy operation in the memory registered by the two applications based on the local work queue of at least one of the two applications, to replace the RDMA data transfer operation between the two applications.

[0130] In an optional embodiment, when the processor 105 identifies same-host RDMA communication and the first application and the second application are applications deployed in different virtualized instances on the physical machine, the processor 105 is further configured to: create a second RDMA work queue for the second application and register it in memory when the first application initiates an RDMA communication request to the second application; and, based on the control plane connection, obtain description information of the first RDMA work queue as context information of the second RDMA work queue.

[0131] In an optional embodiment, the processor 105 is further configured to: when cross-host RDMA communication is identified, perform RDMA data transmission operations between the memories registered by the two applications through the RDMA network card on the physical machine based on the remote work queue of at least one of the two applications.

[0132] In an optional embodiment, when the first application initiates an RDMA communication request to the second application, the processor 105 creates a first RDMA work queue for the first application and performs memory registration, and is specifically used to: respond to the completion queue CQ creation request initiated by the first application, and create a local CQ queue and a remote CQ queue for the first application; respond to the queue pair QP creation request initiated by the first application, and create a local QP queue and a remote QP queue for the first application; respond to the memory registration request initiated by the first application, register memory for the first application, and save index information of the registered memory; wherein, the local work queue includes the local CQ queue and the local QP queue, and the remote work queue includes the remote CQ queue and the remote QP queue.

[0133] Optionally, the remote work queue has a queue pair number QPN and a global identity GID, and the description information of the RDMA work queue to which the remote work queue belongs includes the QPN and GID of the remote work queue; the local work queue does not have a QPN and GID, or the local work queue shares the QPN and GID of the remote work queue in the RDMA work queue to which it belongs.

[0134] In an optional embodiment, when the processor 105 identifies whether the first application and the second application are communicating with the same host via RDMA based on the description information of the second RDMA work queue, the processor 105 is specifically configured to: determine the IP address of the second application based on the GID in the description information of the second RDMA work queue; in a single-tenant scenario, identify whether the first application and the second application are communicating with the same host via RDMA based on the routing table of the physical machine and the IP address of the second application; and in a multi-tenant scenario, identify whether the first application and the second application are communicating with the same host via RDMA based on the flow table of the virtual switch on the first physical machine and the IP address of the second application.

[0135] In an optional embodiment, when the processor 105 is identified as intra-host RDMA communication, based on the local work queue of at least one of the two applications, a data copy operation is performed in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications. The processor 105 is specifically used to: when it is identified as intra-host RDMA communication, in user state, respond to the data sending task initiated by the sender, generate an SQE, and write the SQE into the local SQ queue in the local work queue of the sender; enter the kernel state from the user state, and in the kernel state, according to the operation type of the data sending task, use the local work queue of the sender or the local work queues of the sender and the receiver to perform the data copy operation in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications; wherein the sender is any application of the two applications, and the other application is the receiver.

[0136] In an optional embodiment, when the processor 105 enters the kernel state from the user state, it is specifically used to: respond to a system call initiated by the user state after writing the SQE into the local SQ queue in the local work queue of the sending end, and enter the kernel state from the user state; or poll the command queue shared by the user state and the kernel state, and enter the kernel state from the user state when the command queue contains a local memory copy command; wherein the local memory copy command is written into the command queue after the user state writes the SQE into the local SQ queue in the local work queue of the sending end.

[0137] In an optional embodiment, the processor 105, in kernel state, uses the local work queues of the sending end or both the sending and receiving ends to perform a data copy operation in the memory registered by the two applications, instead of the RDMA data transmission operation between the two applications, according to the operation type of the data sending task. Specifically, it is used to: in kernel state, obtain the SQE from the local SQ queue of the sending end, and obtain the operation type of the data sending task from the SQE; if the type of the data sending task is a bilateral operation, use the local work queues of both the sending and receiving ends to perform a data copy operation in the memory registered by the two applications, instead of the RDMA data transmission operation between the two applications; if the type of the data sending task is a unilateral operation, use the local work queue of the sending end to perform a data copy operation in the memory registered by the two applications, instead of the RDMA data transmission operation between the two applications.

[0138] In an optional embodiment, when the type of the data sending task is a bilateral operation, the processor 105 is further configured to: in user mode, respond to the data receiving task initiated by the receiving end, generate an RQE, and write the RQE into a local RQ queue in a local work queue of the receiving end; when using the local work queues at both ends of the sending and receiving end to perform a data copy operation in memories registered by two applications to replace the RDMA data transmission operation between the two applications, specifically configured to: determine description information of the RDMA work queue of the receiving end based on context information of the RDMA work queue of the sending end; read the SQE from the local SQ queue of the sending end, and read the RQE from the local RQ of the receiving end based on the description information of the RDMA work queue of the receiving end; copy the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the RQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queues in the local work queues of the sending end and the receiving end, respectively.

[0139] In an optional embodiment, when the processor 105 uses the local work queue of the sending end to perform a data copy operation in the memory registered by two applications to replace the RDMA data transmission operation between the two applications, it is specifically used to: if the operation type of the data sending task is a one-sided write operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the remote memory index information contained in the SQE; if the operation type of the data sending task is a one-sided read operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the remote memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the SQE; after completing the data copy, generate a CQE, and write the CQE into the local CQ queue in the local work queue of the sending end respectively.

[0140] In an optional embodiment, when the processor 105 performs a data copy operation in memories registered by two applications, it is specifically used to: execute a memory copy command of the processor CPU to copy data from the memory registered by one application to the memory registered by another application; or use the DMA acceleration engine of the CPU to copy data from the memory registered by one application to the memory registered by another application.

[0141] Further, if Figure 10 As shown, the physical machine also includes: a communication component 106, a display 107, a power component 108, an audio component 109 and other components. Figure 10 Only some components are shown schematically, which does not mean that the physical machine only includes Figure 10 In addition, Figure 10 The components in the dotted box are optional components, not mandatory components, and the specific components depend on the product form of the working node. The working node of this embodiment can be implemented as a terminal device such as a desktop computer, laptop computer, smart phone or IOT device, or a server device such as a conventional server, cloud server or server array. If the working node of this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, smart phone, etc., it can include Figure 10 If the working node of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 10 Components within the dotted box.

[0142] The detailed implementation and beneficial effects of each step in the method of this embodiment have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0143] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement each step in the above method embodiment.

[0144] Accordingly, an embodiment of the present application further provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the processor is enabled to implement the steps in the above method embodiment.

[0145] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0146] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0147] The above-mentioned display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0148] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0149] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0150] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) that contain computer-usable program code.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0154] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0155] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0156] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0157] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0158] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data transmission method based on RDMA, characterized in that: Applied to a physical machine, on which a first application is deployed, the method includes: When the first application initiates an RDMA communication request to the second application, a first RDMA work queue is created for the first application and registered in memory, wherein the first RDMA work queue includes a local work queue and a remote work queue; Based on a control plane connection between the first application and the second application, obtaining description information of a second RDMA work queue corresponding to the second application as context information of the first RDMA work queue; Identifying, based on description information of the second RDMA work queue, whether the first application and the second application perform same-host RDMA communication, where the second RDMA work queue also includes a local work queue and a remote work queue; When it is identified as intra-host RDMA communication, a data copy operation is performed in the memory registered by the two applications based on the local work queue of at least one of the two applications, to replace the RDMA data transmission operation between the two applications.

2. The method according to claim 1, characterized in that In the case of identifying same-host RDMA communication, the first application and the second application are applications deployed in different virtualized instances on the physical machine; the method further includes: When the first application initiates an RDMA communication request to the second application, creating a second RDMA work queue for the second application and registering it in memory; and Based on the control plane connection, description information of the first RDMA work queue is obtained as context information of the second RDMA work queue.

3. The method according to claim 1, characterized in that Also includes: In the case of cross-host RDMA communication, an RDMA data transmission operation is performed between the memories registered by the two applications through the RDMA network card on the physical machine based on the remote work queue of at least one of the two applications.

4. The method according to claim 1, wherein When the first application initiates an RDMA communication request to the second application, a first RDMA work queue is created for the first application and memory registration is performed, including: In response to the completion queue CQ creation request initiated by the first application, a local CQ queue and a remote CQ queue are created for the first application; In response to the queue pair QP creation request initiated by the first application, a local QP queue and a remote QP queue are created for the first application; Responding to the memory registration request initiated by the first application, registering the memory for the first application, and saving index information of the registered memory; The local work queue includes the local CQ queue and the local QP queue, and the remote work queue includes the remote CQ queue and the remote QP queue.

5. The method according to claim 4, characterized in that The remote work queue has a queue pair number QPN and a global identity GID, and the description information of the RDMA work queue to which the remote work queue belongs includes the QPN and GID of the remote work queue; The local work queue does not have a QPN and a GID, or the local work queue shares the QPN and the GID of the remote work queue in the RDMA work queue to which the local work queue belongs.

6. The method according to claim 5, characterized in that Identifying, based on the description information of the second RDMA work queue, whether the first application and the second application perform same-host RDMA communication, includes: Determining the IP address of the second application according to the GID in the description information of the second RDMA work queue; In a single-tenant scenario, identifying whether the first application and the second application are communicating with the same host using RDMA based on the routing table of the physical machine and the IP address of the second application; In a multi-tenant scenario, whether the first application and the second application communicate with each other using the same host RDMA is identified based on the flow table of the virtual switch on the physical machine and the IP address of the second application.

7. The method according to any one of claims 1 to 6, characterized in that When intra-host RDMA communication is identified, a data copy operation is performed in the memory registered by the two applications based on the local work queue of at least one of the two applications, to replace the RDMA data transfer operation between the two applications, including: In the case of identifying intra-host RDMA communication, in user mode, responding to the data sending task initiated by the sender, generating an SQE, and writing the SQE into a local SQ queue in the local work queue of the sender; Entering kernel state from user state, in kernel state, based on the operation type of the data sending task, using the local work queues of the sending end or the sending and receiving ends, a data copy operation is performed in the memory registered by the two applications to replace the RDMA data transmission operation between the two applications; wherein the sending end is any application in the two applications, and the other application is the receiving end.

8. The method according to claim 7, characterized in that Ways to enter kernel mode from user mode include: In response to a system call initiated by the user state after writing the SQE into the local SQ queue in the local work queue of the sending end, entering the kernel state from the user state; or Poll the command queue shared by user state and kernel state. When the command queue contains a local memory copy command, enter kernel state from user state. The local memory copy command is written into the command queue after the SQE is written into the local SQ queue in the local work queue of the sending end in the user state.

9. The method according to claim 7, characterized in that In kernel mode, based on the operation type of the data transmission task, a data copy operation is performed in the memory registered by the two applications using the local work queues of the sending end or the sending and receiving ends, thereby replacing the RDMA data transmission operation between the two applications. This includes: In kernel mode, obtaining the SQE from the local SQ queue of the sending end, and obtaining the operation type of the data sending task from the SQE; If the data transmission task is a bilateral operation, the local work queues at both ends of the transmission and reception are used to perform a data copy operation in the memory registered by the two applications, thereby replacing the RDMA data transmission operation between the two applications. If the type of the data sending task is a unilateral operation, a data copy operation is performed in the memory registered by the two applications using the local work queue of the sending end, to replace the RDMA data transmission operation between the two applications.

10. The method according to claim 9, characterized in that In the case where the type of the data sending task is a bilateral operation, the method further includes: in a user state, responding to the data receiving task initiated by the receiving end, generating an RQE, and writing the RQE into a local RQ queue in a local work queue of the receiving end; Using the local work queues at both ends, data copy operations are performed in the memory registered by the two applications, replacing the RDMA data transfer operations between the two applications, including: Determine the description information of the RDMA work queue at the receiving end based on the context information of the RDMA work queue at the sending end; Read the SQE from the local SQ queue of the sending end, and read the RQE from the local RQ of the receiving end according to the description information of the RDMA work queue of the receiving end; The data in the memory pointed to by the local memory index information in the SQE is copied to the memory pointed to by the local memory index information in the RQE; After the data copy is completed, a CQE is generated and written into the local CQ queues in the local work queues of the sending end and the receiving end respectively.

11. The method according to claim 9, characterized in that Using the local work queue on the sending end, the data copy operation is performed in the memory registered by the two applications to replace the RDMA data transfer operation between the two applications, including: If the operation type of the data sending task is a unilateral write operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the local memory index information contained in the SQE to the memory pointed to by the remote memory index information contained in the SQE; If the operation type of the data sending task is a unilateral read operation, read the SQE from the local SQ queue of the sending end, and copy the data in the memory pointed to by the remote memory index information contained in the SQE to the memory pointed to by the local memory index information contained in the SQE; After the data copy is completed, a CQE is generated and the CQE is written into a local CQ queue in the local work queue of the sender.

12. The method according to claim 9, characterized in that Perform data copy operations in the memory registered by two applications, including: Execute the memory copy command of the processor CPU to copy data from the memory registered by one application to the memory registered by another application; or Use the CPU's DMA acceleration engine to copy data from the memory registered by one application to the memory registered by another application.

13. A physical machine, characterized in that: The physical machine includes an RDMA network card, and a first application and an RDMA management program are deployed on the physical machine, wherein the RDMA management program includes: an interface management module in user state and a driver management module in kernel state; The driver management module cooperates with the interface management module to: When a first application initiates an RDMA communication request to a second application, a first RDMA work queue is created for the first application and registered in memory; based on a control plane connection between the first application and the second application, description information of a second RDMA work queue corresponding to the second application is obtained as context information of the first RDMA work queue; the first and second RDMA work queues both include a local work queue and a remote work queue; Identifying, based on the description information of the second RDMA work queue, whether the first application and the second application perform same-host RDMA communication; and When it is identified as intra-host RDMA communication, a data copy operation is performed in the memory registered by the two applications based on the local work queue of at least one of the two applications, to replace the RDMA data transmission operation between the two applications.

14. The physical machine according to claim 13, wherein: In the case of identifying same-host RDMA communication, the first application and the second application are applications deployed in different virtualized instances on the physical machine; The driver management module is further configured to: create a second RDMA work queue for the second application and register it in memory when the first application initiates an RDMA communication request to the second application; and obtain description information of the first RDMA work queue based on the control plane connection as context information of the second RDMA work queue.

15. The physical machine according to claim 13, wherein: The driver management module is further configured to provide the interface management module with an identification result of whether the communication is with the host RDMA; The interface management module is further configured to: when the identification result is cross-host RDMA communication, based on the remote work queue of at least one of the two applications, perform an RDMA data transmission operation between the memories registered by the two applications through the RDMA network card.

16. The physical machine according to claim 13, characterized in that: The interface management module includes: a resource management unit, and the drive management module includes a transmission service unit; The resource management unit is configured to respond to a completion queue CQ creation request, a queue pair QP creation request, and a memory registration request initiated by the first application, and initiate calls to the transmission service unit respectively; The transmission service unit is configured to respond to the call of the resource management unit, create a local CQ queue and a remote CQ queue, a local QP queue and a remote QP queue for the first application, register a memory for the first application, and save index information of the registered memory; The local work queue includes the local CQ queue and the local QP queue, and the remote work queue includes the remote CQ queue and the remote QP queue.

17. The physical machine according to any one of claims 13 to 16, wherein: The interface management module further includes: a data transceiver unit, and the drive management module includes: a transmission service unit; The data transceiver unit is configured to, when identifying RDMA communication with the host, respond to a data sending task initiated by the sender, generate an SQE, write the sending task description information into a local SQ queue in a local work queue of the sender, and trigger the transmission service unit; The transmission service unit is configured to: under the triggering of the data transceiver unit, utilize the local work queues of the sending end or both the sending and receiving ends to perform a data copy operation in the memory registered by the two applications according to the operation type of the data sending task, thereby replacing the RDMA data transmission operation between the two applications; wherein the sending end is any one of the two applications, and the other application is the receiving end.

18. A physical machine, characterized in that: include: memory and processor; The memory is used to store a computer program, and the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 12.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 1 to 12.

20. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, causes the processor to implement the steps of the method according to any one of claims 1 to 12.

Images