Communication method, apparatus, device and system

Abstract

The application discloses a communication method, device, equipment and system, and belongs to the technical field of communication. The method comprises the following steps: determining a first virtual QP in response to a first RDMA request; determining a first physical QP corresponding to the first virtual QP; processing the first RDMA request through the first physical QP; and disconnecting the correspondence between the first virtual QP and the first physical QP. The first virtual QP comprises a first QPC and does not comprise a queue. The first physical QP comprises a second QPC and a queue. In the method, the virtual QP is used to establish a QP connection, and the QP actually performing the RDMA operation is the physical QP corresponding to the virtual QP. Since the virtual QP does not comprise a queue, the memory occupied by the virtual QP is small. Therefore, when the communication is performed through the method, a large number of virtual QPs with small memory occupation are created, a limited number of physical QPs are created, and the communication demand of a large-scale connection scene can be met through time-sharing multiplexing of the physical QPs.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a communication method, apparatus, device and system. Background Technology

[0002] Remote direct memory access (RDMA) is a high-efficiency network communication technology. RDMA allows one computer system to directly access the memory of another computer system remotely over a network, without the intervention of the other computer system's central processing unit (CPU). Therefore, RDMA offers advantages such as reducing the CPU load on computer systems, increasing data transfer speed and bandwidth, and reducing network latency.

[0003] Because RDMA communication requires establishing queue pair (QP) connections, a large number of QPs need to be created for establishing QP connections in large-scale RDMA connection scenarios. A large number of QPs generates a relatively large QP context, which consumes excessive computer system memory and thus impacts system performance. Therefore, how to reduce the number of QPs used in large-scale connection scenarios is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a communication method, apparatus, device, and system that can reduce the number of physical QPs required in large-scale connection scenarios.

[0005] The technical solution provided in this application is as follows.

[0006] In a first aspect, this application provides a communication method, the method comprising: in response to a first RDMA request, determining a first virtual QP; determining a first physical QP corresponding to the first virtual QP; processing the first RDMA request through the first physical QP; and disconnecting the correspondence between the first virtual QP and the first physical QP. The first virtual QP includes a first queue pair context (QPC), but does not include a queue. The first physical QP includes a second QPC and a queue.

[0007] In the method provided in this application, a virtual QP can be used to establish a QP connection, while the QP that actually handles the RDMA request is the physical QP corresponding to the virtual QP. Since the virtual QP does not include a queue, it occupies very little memory on the communication node. Thus, when performing RDMA communication using this application's scheme, creating a large number of virtual QPs will not consume excessive memory on the communication node. The physical QP that handles the RDMA request for the virtual QP can disconnect from the virtual QP after processing the request and can subsequently be used to handle RDMA requests for other virtual QPs. That is, the same physical QP can handle RDMA requests for different virtual QPs at different times; in other words, the same physical QP can be time-multiplexed by different virtual QPs to handle the RDMA requests of those different virtual QPs. Therefore, the number of physical QPs required by this application's scheme is limited, thus avoiding excessive memory consumption on the communication node by too many physical QPs, and consequently avoiding the impact of excessive memory consumption on the communication node's performance in large-scale RDMA connection scenarios.

[0008] In one possible design, the method further includes updating the state of the first physical QP to idle. Updating the state of the first physical QP to idle indicates that the first physical QP can be invoked by a virtual QP to handle RDMA requests.

[0009] In another possible design, the method further includes: in response to a second RDMA request, determining a second virtual QP; determining a first physical QP as the physical QP corresponding to the second virtual QP; processing the second RDMA request through the first physical QP; and disconnecting the correspondence between the second virtual QP and the first physical QP. The second virtual QP includes a third QPC, but does not include a queue.

[0010] Through the two possible implementation methods described above, the first physical QP can be called by the first virtual QP and the second virtual QP in a time-sharing manner.

[0011] In another possible design approach, determining the first physical QP corresponding to the first virtual QP includes: querying a first correspondence relationship to determine whether there exists a physical QP that corresponds to the first virtual QP; if it is determined that there exists a physical QP that corresponds to the first virtual QP, then determining the physical QP that corresponds to the first virtual QP as the first physical QP. The first correspondence relationship includes the correspondence between virtual QPs and physical QPs in the physical QP pool.

[0012] With this possible design, for the first virtual QP and the first physical QP recorded in the first correspondence, it means that the first virtual QP can achieve RDMA-based data transmission through the first physical QP. In this way, before the communication node removes the correspondence between the first virtual QP and the first physical QP from the first correspondence, if an application / process needs to transmit data multiple times through a QP connection established based on the first virtual QP, it can always use the first physical QP to perform RDMA data transmission / reception operations. This avoids the problem that the communication node needs to specify a different physical QP as the corresponding physical QP for each data transmission in a short period of time to achieve RDMA data transmission. This reduces the number of times the communication node updates the physical QPC of the corresponding physical QP based on the virtual QPC (i.e., the first QPC) of the first virtual QP, thus reducing the bandwidth and packet per second (PPS) consumption of the high-speed serial computer expansion bus (PCIe) in the communication node.

[0013] In another possible design, the above method further includes: if it is determined that there is no physical QP corresponding to the first virtual QP, then any idle physical QP in the physical QP pool is identified as the first physical QP.

[0014] This possible design allows specifying that idle QPs in the physical QP pool be allocated to the first virtual QP for use.

[0015] In another possible design approach, the above method further includes adding a correspondence between the first virtual QP and the first physical QP to the first correspondence.

[0016] In another possible design approach, the above method also includes updating the state of the first physical QP in the physical QP pool to non-idle.

[0017] With this possible design, since an idle physical QP in the physical QP pool is one that has not yet been associated with any virtual QP, and a non-idle physical QP in the physical QP pool is one that has been associated with any virtual QP, when a communication node establishes an association between a first virtual QP and a first physical QP in the physical QP pool and updates the state of the first physical QP to non-idle, when the communication node needs to allocate an idle physical QP from the physical QP pool to another virtual QP, the communication node will not reassign the first physical QP already allocated to the first virtual QP to that other virtual QP. In other words, by updating the state of the physical QPs in the physical QP pool, the communication node can avoid allocating the same physical QP to multiple virtual QPs, thus preventing conflicts.

[0018] In another possible design, the first correspondence includes the correspondence between the virtual QP and the send queue (SQ) contained in the physical QP in the physical QP pool, and / or, the first correspondence includes the correspondence between the virtual QP and the receive queue (RQ) contained in the physical QP in the physical QP pool.

[0019] In another possible design approach, when the above method is applied to the sending end, the first correspondence includes the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool.

[0020] With this possible design, when the method described in this application is applied to the sending end, i.e., the communication node is implemented as the sending end, the virtual QP only uses the SQ of the physical QP in the physical QP pool when processing RDMA requests, and there is no processing flow association with the RQ of that physical QP. Therefore, the first correspondence can only include the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool. In this way, the RQ of the physical QP in the physical QP pool can be used by calls corresponding to other virtual QPs, thus improving the utilization rate of the physical QP in the physical QP pool.

[0021] In another possible design approach, updating the state of the first physical QP in the physical QP pool to non-idle includes: updating the SQ state of the first physical QP in the physical QP pool to non-idle.

[0022] With this possible design, when the method described in this application is applied to the sending end, and the first correspondence only includes the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool, the communication node only needs to update the state of the SQ of the first physical QP in the physical QP pool.

[0023] In another possible design approach, when the above method is applied to the receiving end, the first correspondence includes the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool.

[0024] With this possible design, when the method described in this application is applied to the receiving end, i.e., the communication node is implemented as the receiving end, the virtual QP only uses the RQ of the physical QP in the physical QP pool when processing RDMA requests, and there is no processing flow association with the SQ of that physical QP. Therefore, the first correspondence can only include the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool. In this way, the SQ of the physical QP in the physical QP pool can be used by calls corresponding to other virtual QPs, thus improving the utilization rate of the physical QP in the physical QP pool.

[0025] In another possible design approach, updating the state of the first physical QP in the physical QP pool to non-idle includes: updating the RQ state of the first physical QP in the physical QP pool to non-idle.

[0026] With this possible design, when the method described in this application is applied to the receiving end, and the first correspondence only includes the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool, the communication node only needs to update the state of the RQ of the first physical QP in the physical QP pool.

[0027] In another possible design, the above method is applied to the transmitting end, where the first RDMA request is a request to send data to be transmitted to the receiving end based on RDMA. Determining the first virtual QP includes: identifying the virtual QP used to establish the first QP connection in the transmitting end as the first virtual QP. The first QP connection is the QP connection established between the first QP in the transmitting end and the receiving end.

[0028] In another possible design approach, the first QP is either a virtual QP or a physical QP.

[0029] Through this possible design, on the one hand, if the first QP in the receiving end is a physical QP, that is, the first QP in the receiving end is the QP described in the current RDMA standard, then the receiving end is a current standard RDMA communication node. In other words, when the method provided in this application is applied to the sending end, the receiving end can be a standard RDMA communication node, meaning the receiving end does not need special modifications to adapt to the communication method provided in this application. Thus, the sending end applying the method described in this application can be used in the current standard RDMA ecosystem, meaning the method described in this application is friendly to the current RDMA ecosystem. On the other hand, if the first QP in the receiving end is the virtual QP described in this application, that is, both the sending end and the receiving end apply the communication method provided in this application. In this case, both the sending end and the receiving end can reduce the number of physical QPs that need to be created in large-scale connection scenarios, thereby reducing the memory footprint of physical QPs on the communication node. Since the QP used to establish the QP connection in this application is a virtual QP, and the virtual QPC of the virtual QP contains standard connection information, even if the communication method provided in this application is applied at both the sending end and the receiving end, there is no need to modify the message format transmitted between the sending end and the receiving end. That is, the communication messages in the method provided in this application can all use the standard RDMA RC protocol fields. Therefore, the method described in this application is easy to implement and is eco-friendly.

[0030] In another possible design approach, before determining the first virtual QP, the above method further includes: creating the first virtual QP; and establishing a first QP connection based on the first virtual QP.

[0031] The two possible designs described above can be used to determine the first virtual QP for establishing a QP connection for the first RDMA request.

[0032] In another possible design, the above-mentioned processing of the first RDMA request through the first physical QP includes: generating a first work queue element (WQE), the first WQE including a first QPC; and processing the first WQE through the SQ of the first physical QP to complete the transmission of the data to be transmitted.

[0033] With this possible design, the WQE generated by the sending end includes the QPC of the first virtual QP (i.e., the first QPC). In this way, when the network card of the sending end processes the WQE, it can assemble packets based on the QPC of the first virtual QP and transmit data through the assembled packets.

[0034] In another possible design, the above-mentioned processing of the first RDMA request through the first physical QP includes: updating the second QPC based on the first QPC; and processing the first RDMA request through the updated second QPC.

[0035] With this possible design, when processing the first RDMA request, the sending end first updates the QPC (i.e., the second QPC) of the first physical QP based on the QPC of the first virtual QP (i.e., the first QPC). In this way, the sending end's network card can process the generation of a standard WQE based on the RDMA request based on the updated second QPC cached in the network card. For example, it can assemble an RDMA packet based on the updated second QPC and the data indicated for transmission by the standard WQE. In this implementation, the WQE generated by the sending end based on the RDMA request does not need to carry the first QPC. Therefore, when multiple messages are sent through the first QP connection containing the first virtual QP, the PCIe bandwidth and PPS required for the sending end to transmit the WQE corresponding to each message from memory to the network card can be saved. That is, this implementation can save the sending end's PCIe bandwidth and PPS.

[0036] In another possible design, disconnecting the correspondence between the first virtual QP and the first physical QP includes: disconnecting the correspondence between the first virtual QP and the first physical QP after determining that the data to be transmitted has been sent through the first QP connection, or after determining that the data to be transmitted has been sent through the first QP connection and that there is no need to continue communicating with the peer (i.e., the receiving end) of the first QP connection, or after determining that the data to be transmitted has been sent through the first QP connection and that there is no need to continue communicating with the peer of the first QP connection within a preset time period.

[0037] This possible design approach enables the disconnection of the correspondence between the first virtual QP and the first physical QP when certain conditions are met.

[0038] In another possible design, the above method is applied to the receiving end. Determining the first virtual QP includes: receiving a first RDMA request sent by the transmitting end, the first RDMA request indicating the reception of data carried in the first RDMA request; and determining the virtual QP represented by the destination QP number (QPN) carried in the first RDMA request as the first virtual QP.

[0039] In another possible design, before determining the first virtual QP, the method further includes: in response to receiving connection information sent by the sender, creating a first virtual QP, the connection information including connection information in the QPC of the second QP in the sender; and establishing a second QP connection between the first virtual QP and the second QP.

[0040] The above two possible designs provide a scheme for determining the first virtual QP when the communication node is the receiving end.

[0041] In another possible design approach, the second QP is either a virtual QP or a physical QP.

[0042] With this possible design, the technical effect of the second QP being a physical QP can be referred to the above-mentioned technical effect of the first QP being a physical QP, and the technical effect of the second QP being a virtual QP can be referred to the above-mentioned technical effect of the first QP being a virtual QP, which will not be repeated here.

[0043] In another possible design, the above-mentioned processing of RDMA requests through the first physical QP includes: in the case of determining the first virtual QP and determining any idle physical QP in the physical QP pool as the first physical QP, updating the second QPC based on the first QPC; and processing the first RDMA request through the updated second QPC.

[0044] In this possible design, after updating the QPC (i.e., the second QPC) of the first physical QP based on the QPC of the first virtual QP, the first physical QP can process the RDMA request according to the standard procedure.

[0045] In another possible design, disconnecting the correspondence between the first virtual QP and the first physical QP includes: disconnecting the correspondence between the first virtual QP and the first physical QP if no message is received through the second QP connection for a preset time period, or upon receiving a disconnection indication from the sending end. Here, the second QP connection is a QP connection based on the first virtual QP, and the disconnection indication is used to indicate the disconnection of the correspondence between the first virtual QP and the first physical QP.

[0046] This possible design approach enables the disconnection of the correspondence between the first virtual QP and the first physical QP when certain conditions are met.

[0047] In another possible design, the first QPC is a virtual QPC, which contains connection information but not queue information. The second QPC is a physical QPC, which contains both connection and queue information.

[0048] Secondly, this application provides a communication device. This communication device is used to execute any of the methods provided in the first aspect. This application can divide the communication device into functional modules according to any of the methods provided in the first aspect. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module. For example, this application can divide the communication device into a determining unit and a processing unit, etc., according to their functions. Descriptions of the possible technical solutions and beneficial effects of the various functional modules described above can be found in the solutions provided in the first aspect and any possible design method within the first aspect, and will not be repeated here.

[0049] Thirdly, this application provides a communication device. The communication device includes: a memory, a network interface, and one or more processors. The one or more processors receive or transmit data through the network interface, and are configured to read program instructions stored in the memory to execute the methods provided by the first aspect and any possible design of the first aspect.

[0050] Fourthly, this application discloses a communication device. The communication device includes a processor, a network interface card (NIC), and a NIC driver. The NIC driver manages the hardware of the NIC, and the processor coordinates the NIC to execute the methods provided by the first aspect and any possible design scheme within the first aspect.

[0051] Fifthly, this application provides a communication system comprising: a transmitting end and a receiving end. The transmitting end is configured to perform operations performed by the transmitting end in the methods provided by the first aspect and any possible design scheme of the first aspect, and / or the receiving end is configured to perform operations performed by the receiving end in the methods provided by the first aspect and any possible design scheme of the first aspect.

[0052] In a sixth aspect, this application provides a computer-readable storage medium that is a non-volatile computer-readable storage medium, the computer-readable storage medium including computer program instructions that, when executed by a processor, a computing device including a processor, or a computer system, perform the methods provided by the first aspect and any possible design of the first aspect.

[0053] In a seventh aspect, this application provides a computer program product comprising instructions that, when executed by a processor, a computing device including a processor, or a computer system, cause the processor, the computing device including a processor, or the computer system to perform the methods provided by the first aspect and any possible design of the first aspect.

[0054] Eighthly, this application provides a chip that includes a processor for running program instructions or code. The chip or a device including the chip can be used to perform methods provided by the first aspect and any possible design embodiment within the first aspect. Exemplarily, the chip further includes an input interface, an output interface, and a memory. The chip's input interface, output interface, processor, and memory are connected via internal interconnection paths. The memory in the chip stores program instructions or code executed by the processor, and the input and output interfaces are used for communication and connection between the chip and other chips or devices.

[0055] It is understood that any of the communication devices, communication equipment, communication systems, computer-readable storage media, computer program products or chips provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0056] In this application, the names of the aforementioned communication devices, communication equipment, communication systems, etc., do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those in this application, they all fall within the protection scope of this application. Attached Figure Description

[0057] Figure 1 A schematic diagram of data transmission in an RDMA RC mode is shown;

[0058] Figure 2 A schematic diagram of data transmission in DCT mode is shown;

[0059] Figure 3 This is a schematic diagram of an implementation environment for the method provided in the embodiments of this application;

[0060] Figure 4 This is a flowchart illustrating a communication method provided in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram illustrating the process by which a communication node determines the first physical QP corresponding to the first virtual QP, as provided in an embodiment of this application.

[0062] Figure 6 This is a schematic diagram of a communication method provided in an embodiment of this application;

[0063] Figure 7 This is another schematic diagram of the communication method provided in the embodiments of this application;

[0064] Figure 8 This is another schematic diagram of the communication method provided in the embodiments of this application;

[0065] Figure 9 This is a schematic diagram of the application architecture for applying the communication method provided in the embodiments of this application in an asymmetric manner;

[0066] Figure 10 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0067] Figure 11 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0069] To facilitate understanding, the technologies and background involved in the embodiments of this application will be explained below.

[0070] Currently, there are two main implementation methods for RDMA technology: Infiniband (IB) and RDMA over Converged Ethernet (RoCE). IB defines a complete set of specifications from the link layer to the transport layer, but it is not compatible with existing Ethernet. Therefore, deploying IB requires not only dedicated network interface cards (NICs) but also dedicated switches. RoCE can be seen as a "low-cost solution" for IB. RoCE encapsulates IB messages into Ethernet packets for transmission and reception. Because RoCE version 2 (RoCE v2) can use Ethernet switching devices, it is currently widely used in enterprises. However, in the same scenario, RoCE suffers some performance loss compared to IB.

[0071] As the mainstream standard for RDMA communication technology, IB provides four transmission modes: reliable connection (RC) mode, unreliable connection (UC) mode, reliable datagram (RD) mode, and unreliable datagram (UD) mode.

[0072] RC mode and UC mode are connection-oriented RDMA services that enable data transmission by establishing and maintaining a persistent connection. The difference between RC mode and UC mode is that RC mode requires an acknowledgment mechanism (such as replying with an acknowledgment character (ACK) message) during data transmission to ensure reliable data transmission, while UC mode does not use an acknowledgment mechanism during data transmission.

[0073] RD mode is a datagram service that provides reliability guarantees in RDMA communication. Specifically, it uses mechanisms such as checksums and retransmissions to ensure that sent data packets are received correctly, thus ensuring data integrity. Each data packet in RD mode is transmitted independently, typically without requiring a connection, making it suitable for scenarios with high requirements for real-time performance and independence.

[0074] UD mode refers to a data transmission method that does not provide reliability guarantees. Unlike RD mode, UD mode does not include acknowledgment mechanisms, error detection mechanisms, and retransmission mechanisms. Therefore, data loss, duplication, or out-of-order transmission may occur during data transmission.

[0075] In the connection-oriented RC mode described above, the RC mode uses QPs as the static carrier of the connection. Therefore, each connection requires one QP from both the sending and receiving ends. Thus, a message can be reliably transmitted from the SQ of one QP to the RQ of another QP within a connection (or QP connection), and messages are delivered in order. The SQ stores the sending task, which is represented by a WQE generated by the application / process based on the data sending requirements. The RQ stores the receiving task, which is represented by a WQE generated by the application or process based on the data receiving requirements.

[0076] It should be understood that QP includes queues and QPC.

[0077] The queues include the SQ, RQ, and completion queue (CQ) mentioned above. The primary purpose of the CQ is to notify the application / process of the execution result of the WQE. The CQ can report the execution status of the WQE to the application / process through completion queue entries (CQEs), such as whether a task (e.g., sending / receiving a task) was successfully completed, and the reasons for any failures.

[0078] A QPC typically consists of two parts: connection information and queue information. The connection information describes the QP and related information about the QP connections built upon it, while the queue information describes information about the queues. In one example, the size of a QPC is typically 512 bytes (B) or 1 kilobyte (KB).

[0079] For example, the connection information of an established QP includes, but is not limited to, the Internet Protocol (IP) address, Media Access Control (MAC) address, address handle (AH), QP number (QPN), expected message sequence number (eMSN), and expected packet sequence number (ePSN) of the peer QP node; and the AH, QPN, send packet sequence number (sPSN), and send message sequence number (sMSN) of the local QP, as well as the retransmission time and maximum transmission unit (MTU). Here, sMSN generally refers to the starting sequence number of the sent message, typically starting from 0 and incrementing. sPSN generally refers to the starting sequence number of the sent packet, and its initial value is not limited. The peer QP's QPN (or destination QPN), eMSN, and ePSN are all provided by the peer QP. The eMSN is the peer QP's sMSN, and the ePSN is the peer QP's sPSN. In RDMA, AH represents the set of information used to locate the node where the peer QP resides, mainly including the global identifier (GID) and port number. Furthermore, a message is the logical unit of communication between applications or users, while a packet is the basic unit of network transmission; typically, one message requires at least one packet to carry its transmission.

[0080] For example, the queue information of a QP includes the buffer address of its contained RQ and / or SQ, queue depth (or "queue length"), etc.

[0081] refer to Figure 1 , Figure 1 An exemplary schematic diagram of data transmission in RDMA RC mode is shown. For example... Figure 1As shown, for nodes 101 and 102 that support RDMA RC mode, application process 1 in node 101 can reliably transmit data with application process 1 in node 102 through the QP connection between QP11-1 and QP12-1 in node 102. For example, it can transmit message 1 with a size of 1 gigabyte (GB) and message 2 with a size of 1KB.

[0082] Specifically, when application process 1 in node 101 determines that it needs to send data (such as data including message1 and message2) to application process 1 in node 102, application process 1 in node 101 creates QP11-1 and sends the connection information of QP11-1 (including but not limited to the QPN, sMSN, sPSN, and AH of QP11-1) to application process 1 in node 102. In response to the received connection information of QP11-1, application process 1 in node 102 creates QP12-1, updates the eMSN, ePSN, QPN, and AH of the peer QP in the QPC of QP12-1 based on the connection information of QP11-1, and sends its own connection information (including but not limited to the QPN, sMSN, sPSN, and AH of QP12-1) to application process 1 in node 101. Application process 1 in node 101 updates the eMSN, ePSN, QPN, and AH information of the peer QP in the QPC of QP11-1 based on the received connection information from QP12-1. Thus, the QP connection between QP11-1 and QP12-1 is established. Next, application process 1 in node 101 generates a WQE based on the data to be sent (such as the data in message 1) and sends the WQE to the SQ of QP11-1. The network interface card (NIC) of node 101 processes the WQE in the SQ, such as assembling a message based on the QPC of QP11-1 and the data to be sent indicated by the WQE, and sends the assembled message to complete the transmission of message 1. Correspondingly, application process 1 in node 102 generates a WQE based on the data to be received (such as the data in message 1) and sends the WQE to the RQ of QP12-1. When the network interface card (NIC) of node 102 receives the message 1, it verifies the message 1 based on the QPC of QP12-1. This includes determining whether the sPSN carried in the received message matches the ePSN in the QPC of QP12-1, and whether the sMSN carried in the message matches the eMSN in the QPC of QP12-1. Then, if the received message verification is successful, node 102 stores the data of message 1 at the address indicated by WQE in the RQ. After receiving message 1, node 102's QP12-1 also sends an ACK message to node 101's QP11-1 to inform node 101 that message 1 has been successfully received.

[0083] Similarly, application process 1 in node 101 can reliably transmit data with application process 2 in node 102 via the QP connection between QP11-2 and QP22-1 in node 102. Application process 2 in node 101 can reliably transmit data with application process 1 in node 102 via the QP connection between QP21-1 and QP12-2 in node 102, and application process 2 in node 101 can reliably transmit data with application process 2 in node 102 via the QP connection between QP21-2 and QP22-2 in node 102.

[0084] However, when using RDMA RC mode for data transmission, the need to use QP as the static carrier of the connection leads to the following problems in scenarios with large-scale reliable connections (i.e., QP connections): 1. QP consumes too much memory, and the memory consumption of QP increases linearly with the number of connections. For example, if each QP occupies 283KB of memory, the computer system needs 27GB of memory to establish 100 kilo,k QP connections. This is due to the limited on-chip memory of the computer system's network card (such as the network card's static random access memory). 1. System memory (SRAM) cannot hold all QPCs for QP connections, so the computer system's memory is needed to store all QPCs for QP connections. 2. QPC switching overhead is high. This is because when the computer system's network card's on-chip memory stores all QPCs for QP connections, the network card usually uses its own on-chip memory's cache mechanism to quickly access frequently used QPCs (i.e., frequently used QPCs are cached in the network card's on-chip memory). In this case, when the required QPC is not found in the network card's on-chip memory, it needs to be read from memory. This causes the QPC to switch between the network card's on-chip memory and the computer system's memory. QPC switching consumes PCIe bandwidth and PPS, resulting in a decrease in network card throughput.

[0085] In one related technology, dynamic connection transport (DCT) was proposed to address the problem of excessive memory consumption in large-scale reliable connections (i.e., QP connections) in RDMA scenarios. DCT aims to solve the scalability problem of QP connections in RC mode by using dynamic QP connections. The design goal of DCT is to reduce the number of QPs that each node needs to prepare while providing RC-like reliable transmission, thereby reducing the resources consumed by QPs and solving the scalability problem. It should be understood that the scalability problem of QP connections refers to how to obtain more QP connections to achieve more data transmission in scenarios with a limited number of QPs. The basic idea of ​​DCT is QP pooling, and temporary QP connections are built based on the QPs in the QP pool. When a QP connection needs to be created, an idle QP is taken from the QP pool to establish the connection. Therefore, the same QP may be used in different QP connections at different times, so that nodes do not need to create too many QPs. Here, idle QPs refer to QPs in the QP pool that have not been used to establish QP connections, while non-idle QPs are QPs in the QP pool that have been used to establish QP connections.

[0086] refer to Figure 2 , Figure 2 An exemplary diagram illustrating data transmission in DCT mode is shown. Figure 2As shown, when the sending application process needs to send data to receiving end A and also needs to send data to receiving end B, the sending end, acting as a dynamic connection initiator (DCI), can send a DC connection request 1 to receiving end A, acting as a DC target (DCT), through its own host channel adapter (HCA) to establish a connection with an idle QP 1 in its own QP pool. Receiving end A's HCA responds to DC connection request 1 by allocating DC resources (DCR), that is, receiving end A's HCA selects any idle QP (such as QP 2) from its own QP pool to establish a DC connection with the sending end's QP 1. After the DC connection is established, the sending end's HCA sends data to receiving end A through QP 1 and receives an ACK returned by receiving end A's HCA through QP 2. Thus, the HCA at the sending end can determine that the data sent to the receiving end A via QP1 has been received by the receiving end A's QP2. Therefore, the sending end's HCA sends a DC disconnection indication to the receiving end A, instructing the receiving end A to disconnect the DC connection between QP1 and QP2. In response, the receiving end A's HCA disconnects the DC connection and releases the DCR used to establish the DC connection, i.e., QP2. Here, the HCA is a hardware device primarily used in RDMA communication to handle address translation and memory access control during data transmission and reception, enabling remote direct memory access to data. For example, the HCA is a network interface card (NIC) in a computer system.

[0087] Next, the transmitting end can continue to send a DC connection request 2, acting as the DCI, to the receiving end B (the DC target) via the HCA to establish a connection with QP1. In response to DC connection request 2, the receiving end B's HCA allocates a DCR, meaning it selects any idle QP (such as QP3) from its own QP pool to establish a DC connection with the transmitting end's QP1. After the DC connection is established, the transmitting end's HCA sends data to the receiving end B via QP1 and receives an ACK returned by the receiving end B's HCA via QP3. Thus, the transmitting end's HCA can determine that the data sent to the receiving end B via QP1 has been received by the receiving end B's QP3. Therefore, the transmitting end's HCA sends a DC disconnection indication to the receiving end B, instructing the receiving end B to disconnect the DC connection between QP1 and QP3. In response, the receiving end B's HCA disconnects the DC connection and releases the DCR used to establish the DC connection, i.e., QP3.

[0088] from Figure 2It can be seen that QP1 in the QP pool of the sending end establishes DC connections (essentially QP connections) with receiver A and receiver B respectively. That is, after the sending end finishes sending a message to a receiver, it can automatically disconnect the DC connection between the DCI and the DC target, thereby releasing the DCR for subsequent use. Furthermore, the QP connection established in DCT mode has the same connection properties as the QP connection established in RC mode. Therefore, work requests (WR) in DCT mode can be handled like WRs in RC mode.

[0089] However, while DCT can improve the scalability of QP connections through dynamic connection establishment, it still faces the following problems: 1. Closed ecosystem: Because DCT is based on the proprietary RoCE protocol, both ends of the communication in DCT mode require dedicated network cards; 2. Higher transmission latency: Compared to RC mode, in RoCE networking, DCI needs to send an additional connect message (usually carrying the DC target's metadata) (i.e., the DC connection request mentioned above) to request the establishment of a DC connection. Furthermore, when switching connections in DCT, an additional disconnection indication needs to be sent, resulting in a higher round-trip time (RTT) for establishing a QP connection in DCT mode compared to RC mode. Therefore, DCT is not friendly to latency-sensitive scenarios; 3. Lower data transmission efficiency: In DCT mode, DCI needs to send an additional connect message to the DC target, and the message in DCT mode includes an additional header (approximately 20 bytes in size). This header is used to distinguish messages in DCT mode from messages in other modes. Therefore, the additional connect message and the additional header reduce data transmission efficiency.

[0090] In another related technology, Scalable Reliable Datagram (SRD), as a novel network transport protocol, can also achieve high-throughput, low-latency data transmission. SRD can be viewed as an enhancement to the UD service type in the IB protocol. SRD requires dedicated chip implementation. On the host side, SRD is used through the Elastic Fabric Adaptor (EFA) kernel bypass interface call. Unlike the one-to-one QP connection of the RC service, SRD supports one-to-many QP connections, thus effectively alleviating the scalability issues of QP connections.

[0091] However, while SRD based on enhanced UD improves the scalability of QP connections, it still faces the following problems: 1. Closed ecosystem: Because SRD supports reliable transmission through a proprietary protocol provided by the network interface card (NIC), both ends of SRD communication require dedicated NICs. 2. Limited operation types: Since SRD is based on UD, and UD itself only supports two-sided (send / receive, recv) operations, not one-sided (read / write) operations, SRD also only supports two-sided (send / recv) operations, not one-sided (read / write) operations. 3. Limited message length (maximum 8KB): Because SRD is based on UD, and UD itself has a limited message length (maximum 8KB), when using SRD, the sending end needs to break down large blocks of data to obtain data blocks that meet the message length requirements of SRD, which increases the CPU consumption of the sending end. Two-sided operations refer to operations in which the CPUs of both communication ends participate, while one-sided operations refer to operations in which only the CPUs of one of the communication ends participate.

[0092] Based on the problems of the above solutions, this application provides a communication method that can improve the scalability of QP connections while avoiding excessive consumption of computer system memory in large-scale QP connection scenarios.

[0093] refer to Figure 3 , Figure 3 A schematic diagram of an implementation environment for the method provided in the embodiments of this application is shown. Figure 3 As shown, this implementation environment is a communication system including a transmitter and a receiver. Both the transmitter and receiver support standard RDMA RC mode communication. Optionally, the transmitter and receiver can be connected via a wired or wireless network, which can be an IP-based network or a TCP / IP-based network; there is no limitation on the specific network type.

[0094] The aforementioned sender and receiver can be implemented using two computer systems. In one example, the sender and receiver are two nodes in any scenario where data transmission is required.

[0095] Optionally, the two nodes that need to transmit data can be two physical nodes or two virtual nodes; there is no limitation on this. When the two nodes that need to transmit data are two virtual nodes, these two virtual nodes can be implemented by two physical nodes or by one physical node. When two virtual nodes are implemented by one physical node, it means that these two virtual nodes correspond to different processing cores within the same physical node.

[0096] For example, scenarios requiring data transmission include, but are not limited to, artificial intelligence (AI) training systems, storage systems (such as distributed storage systems), high-performance computing systems, and cloud service systems. This application embodiment does not limit the specific service content of cloud services. Taking a storage system as an example, then... Figure 3 The sending end shown can be a management node in a distributed storage system, and the receiving end can be a storage node in the distributed storage system.

[0097] In the embodiments of this application, the method provided in the embodiments of this application can be applied to the above-mentioned sending end, or to the above-mentioned receiving end, or to both the sending end and the receiving end, and there is no limitation thereto.

[0098] In one example, when the method provided in this application embodiment is applied to the sending end described above, the receiving end that receives the data sent by the sending end can be a standard RDMA RC node, or a receiving end node that applies the method described in this application embodiment.

[0099] In another example, when the method provided in this application embodiment is applied to the receiving end described above, the sending end that sends data to the receiving end can be a standard RDMA RC node, or a sending end node that applies the method described in this application embodiment.

[0100] In a node (which can serve as either the sender or receiver, hereinafter collectively referred to as a communication node) using the method described in this application embodiment, the application / process running on the communication node creates a physical QP pool comprising multiple physical QPs, so that the application / process can transmit data with other nodes through the physical QPs in the physical QP pool. This application embodiment does not specifically limit the timing of the application / process creating the physical QP pool in the communication node; the application / process in the communication node only needs to complete the creation of the physical QP pool before the method described in this application embodiment is executed. In one example, the application / process in the communication node can create the physical QP pool during its initialization process after installation. In another example, the application / process in the communication node creates the physical QP pool when it needs to access data on a remote node.

[0101] It should be noted that the embodiments of this application introduce the concepts of physical QP and virtual QP. Physical QP refers to the QP as described in the current RDMA standard, meaning a physical QP includes QPCs and queues. The queues of a physical QP include SQs, RQs, and CQs, and the QPCs of a physical QP contain connection information and queue information. Virtual QP, on the other hand, refers to a QP that only includes QPCs and does not include queues, and the QPCs of a virtual QP only include connection information, not queue information. Therefore, compared to physical QPs, virtual QPs occupy very little storage space. Detailed descriptions of SQs, RQs, CQs, connection information, and queue information can be found above and will not be repeated here.

[0102] For ease of description, in the embodiments of this application, the QPC of the physical QP is referred to as the physical QPC, and the QPC of the virtual QP is referred to as the virtual QPC.

[0103] Optionally, applications / processes running on the communication node can create physical QP pools by calling the verbs interface. The context of a physical QP pool includes the physical QP pool number, a device pointer (or network card pointer) used to create the physical QP, a protected domain (PD) pointer, the number of physical QPs contained in the physical QP pool, and a QP array used to represent the physical QPs contained in the physical QP pool.

[0104] The verbs interface is an application programming interface (API) defined by RDMA. The physical QP pool number is specified by the application / process. The PD pointer is used to uniquely identify a physical QP pool, or in other words, to isolate different physical QP pools. Each physical QP in the QP array includes a queue and a QPC.

[0105] It should be understood that an application / process can create one or more physical QP pools, and each physical QP pool includes multiple physical QPs, without limitation. It should also be understood that any pre-created physical QP pool in a communication node corresponds to one application / process.

[0106] It should be understood that the above content is an exemplary description of the implementation environment of the communication method provided in the embodiments of this application, and does not constitute a limitation on the implementation environment of the communication method. As those skilled in the art know, as business needs change, the implementation environment can be adjusted according to application requirements, and the embodiments of this application do not list them one by one.

[0107] This application also provides a communication device for executing the communication method described in this application.

[0108] In one example, the communication device may be implemented as the transmitter or a functional module of the transmitter as described above, or as the receiver or a functional module of the receiver as described above, without limitation.

[0109] In another example, the communication device can be applied to any communication node that supports RDMA RC communication. The communication node can be implemented as any computing device, including but not limited to physical computers, virtual computers, laptops, tablets, etc. Alternatively, the computing device can be implemented as a server, including but not limited to storage servers, computing servers, etc.

[0110] The implementation process of the communication method provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0111] refer to Figure 4 , Figure 4 A flowchart illustrating a communication method provided in an embodiment of this application is shown. Optionally, this method can be applied to... Figure 3 The communication system shown is executed by communication nodes that act as either the sender or the server in the communication system. For example... Figure 4 As shown, the method includes the following steps.

[0112] Step 101: In response to the first RDMA request, the communication node determines a first virtual QP for handling the first RDMA request. The first virtual QP includes a first QPC, but does not include a queue.

[0113] The first QPC is a virtual QP, meaning that the first QPC contains connection information but not queue information.

[0114] For example, the first RDMA request can be a request from an application / process in the communication node to send data to be transmitted based on RDMA, or a request from an application / process in the communication node to receive data based on RDMA, or an RDMA request received by the communication node from the sending end; there is no limitation on this. When the first RDMA request is an RDMA request received by the communication node from the sending end, the RDMA request includes, but is not limited to, an RDMA receive request, an RDMA read request, or an RDMA write request.

[0115] In this embodiment of the application, when a communication node needs to process a first RDMA request, it needs to first determine a first virtual QP for processing the first RDMA request.

[0116] This application's embodiments introduce a virtual QP in the communication node, which includes only a virtual QPC and does not include a queue. The virtual QP can establish a QP connection with another node's QP, and this QP connection can be used for data transmission. It should be understood that this QP connection has the same nature and purpose as the QP connection in standard RDMA RC mode.

[0117] It should be noted that the QP in the other node that establishes a QP connection with the virtual QP in the communication node can be a physical QP or the virtual QP described in the embodiments of this application, and there is no limitation on this.

[0118] A virtual QP consists only of virtual QPCs, and a virtual QPC only includes the connection information of that virtual QP, excluding queue information. Therefore, a single virtual QP requires very little storage space, and even when the number of virtual QPs is very large, the required storage space remains limited.

[0119] For example, a virtual QPC includes, but is not limited to: a pointer to the corresponding physical QP pool, the virtual QP's ID (virtual QPN, vQPN), sPSN, and sMSN, the peer QP's AH, QPN, ePSN, and eMSN, retransmission time, and MTU. Here, "corresponding physical QP pool" refers to the physical QP pool created by the application / process that created the virtual QP. It is understood that after a virtual QP is created but before establishing a connection with other QPs, the peer QP's AH, QPN, ePSN, and eMSN information in the virtual QP's QPC (i.e., the virtual QPC) is a preset value or empty. When the virtual QP establishes a connection with other QPs, the communication node updates the peer QP's AH, QPN, ePSN, and eMSN information in the virtual QP's QPC based on the other QP's AH, QPN, sPSN, and sMSN.

[0120] In a first possible implementation, when the communication node is implemented as a transmitter, i.e., the method described in this embodiment is applied to the transmitter, the first RDMA request is a request initiated by an application / process running on the transmitter to send data to be transmitted based on RDMA. In this case, the transmitter can respond to the first RDMA request and determine the virtual QP included in the first QP connection as the first virtual QP. The first QP connection is a QP connection established between the transmitter and the first QP in the receiver. Here, the first QP can be any QP in the receiver; for example, the first QP can be any physical QP or any virtual QP in the receiver, without limitation.

[0121] It should be understood that in one possible scenario of the first possible implementation, when there is no first QP connection between the sender and receiver, the sender may, in response to the first RDMA request, create a virtual QP and designate this virtual QP as the first virtual QP for handling the first RDMA request, and establish a first QP connection based on the first virtual QP and the receiver's first QP. In this case, the first QP connection is the QP connection between the first virtual QP and the first QP. In one example, the application / process in the sender that initiates the first RDMA request can create the first virtual QP by calling the verbs interface.

[0122] In another possible scenario within the first possible implementation, if a first QP connection already exists between the sender and receiver, it indicates that the sender previously created a first virtual QP based on its need to send data to the receiver, and established a first QP connection based on the first virtual QP and the receiver's first QP. Therefore, in response to the first RDMA request, the sender can identify the QP located at the sender within the first QP connection as the first virtual QP.

[0123] In a second possible implementation, when the communication node is implemented as a receiver, that is, the method described in the embodiments of this application is applied to the receiver. In this case, the communication node can receive a first RDMA request sent by the sender and determine the virtual QP represented by the destination QPN carried in the first RDMA request as the first virtual QP. The first RDMA request indicates the receipt of the data carried in the first RDMA request.

[0124] It should be understood that in the second possible implementation, the first virtual QP has pre-established a QP connection with a QP at the sending end (denoted as the second QP connection). Optionally, the QP at the sending end that establishes the second QP connection with the first virtual QP can be a physical QP or the virtual QP described in the embodiments of this application, and there is no limitation on this. The process of the first virtual QP establishing the second QP connection with the QP at the sending end can be described in the following description. Figure 7 As previously mentioned, this will not be elaborated upon here.

[0125] Step 102: The communication node determines the first physical QP corresponding to the first virtual QP. The first physical QP includes the second QPC and the queue.

[0126] The second QPC is a physical QPC, meaning it contains connection information and queue information.

[0127] The first physical QP is a physical QP in the physical QP pool created by the application / process that creates the first virtual QP in the communication node.

[0128] In this embodiment, applications / processes in the communication node that need to use RDMA to transmit data can pre-create physical QP pools. Each physical QP pool includes multiple physical QPs, and each physical QP includes a physical QPC and a queue. A detailed description of the physical QP pool can be found above and will not be repeated here.

[0129] In an exemplary embodiment, the process by which a communication node determines the first physical QP corresponding to the first virtual QP can be achieved through... Figure 5 Steps 1021 to 1023 shown are implemented.

[0130] Step 1021: The communication node queries the first correspondence to determine whether there is a physical QP that corresponds to the first virtual QP.

[0131] The first correspondence includes the correspondence between virtual QPs and physical QPs in the aforementioned physical QP pool. In this embodiment, the physical QP corresponding to the virtual QP is actually the physical QP that carries the RDMA communication function of the virtual QP. Alternatively, it can be understood that in a QP connection established based on a virtual QP, the physical QP corresponding to the virtual QP is the QP that actually processes the data to be transmitted.

[0132] In one exemplary embodiment, based on actual application requirements, the first correspondence may only include the correspondence between virtual QPs and SQs contained in physical QPs in the physical QP pool, or the first correspondence may only include the correspondence between virtual QPs and RQs contained in physical QPs in the physical QP pool.

[0133] For example, when the method described in this application embodiment is applied to the sending end, that is, the communication node is implemented as the sending end. In this case, when the virtual QP processes the RDMA request, it only uses the SQ of the physical QP in the physical QP pool, and there is no processing flow association with the RQ of the physical QP. Therefore, the first correspondence can only include the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool. In this way, the RQ of the physical QP in the physical QP pool can be used by the calls corresponding to other virtual QPs (as described below). Figure 7 The aforementioned correspondence will not be elaborated here. This improves the utilization rate of physical QPs in the physical QP pool. Furthermore, since RDMA transmission operations require CPU (and thus application / process) participation, the sending end can maintain the first correspondence in memory, making it easier for application / processes or network card drivers to query and use it.

[0134] For example, when the method described in this application embodiment is applied to the receiving end, i.e., the communication node is implemented as the receiving end, the virtual QP only uses the RQ of the physical QP in the physical QP pool when processing RDMA requests, and there is no processing flow association with the SQ of that physical QP. Therefore, the first correspondence can only include the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool. In this way, the SQ of the physical QP in the physical QP pool can be used by calls corresponding to other virtual QPs (as described below). Figure 6 The aforementioned correspondence will not be elaborated here. This improves the utilization rate of physical QPs in the physical QP pool. Furthermore, since the RDMA receive operation is completed within the network card, the receiving end can maintain the first correspondence in the network card, facilitating network card lookup and use.

[0135] For example, the first correspondence can be implemented as a QP mapping table with a data structure of a map table. This QP mapping table is used to record the correspondence between the vQPN of a virtual QP and the physical QPN (pQPN) of the physical QP corresponding to the virtual QP in the aforementioned physical QP pool. The key of the QP mapping table is vQPN, and the value is pQPN.

[0136] In this way, the communication node can query the first correspondence based on the vQPN of the first virtual QP to determine whether the first correspondence records the pQPN of the physical QP corresponding to the vQPN of the first virtual QP. That is, the communication node can determine whether there is a physical QP corresponding to the first virtual QP in the aforementioned physical QP pool.

[0137] It should be understood that for the first virtual QP and first physical QP recorded in the first correspondence, it means that the first virtual QP can realize RDMA-based data transmission through the first physical QP. In this way, before the communication node deletes the correspondence between the first virtual QP and the first physical QP from the first correspondence, if the application / process needs to transmit data multiple times through the QP connection established based on the first virtual QP, it can always use the first physical QP to perform RDMA data transmission / reception operations. This avoids the problem that the communication node needs to specify a different physical QP as the corresponding physical QP for each data transmission in a short period of time to realize RDMA data transmission. This reduces the number of times the communication node updates the physical QPC of the corresponding physical QP based on the virtual QPC (i.e., the first QPC) of the first virtual QP (detailed description can be found in the relevant descriptions of S15 and S43 below, and will not be repeated here). This reduces the consumption of PCIe bandwidth and PPS in the communication node.

[0138] Step 1022: If the communication node determines that there is a physical QP corresponding to the first virtual QP, it determines the physical QP corresponding to the first virtual QP as the first physical QP.

[0139] If a communication node determines that the first correspondence records a pQPN corresponding to the vQPN of the first virtual QP, it indicates that the physical QP represented by the pQPN is a physical QP that has established a correspondence with the first virtual QP. In other words, the communication node determines that there exists a physical QP corresponding to the first virtual QP. In this case, the communication node determines the physical QP corresponding to the first virtual QP as the first physical QP.

[0140] Step 1023: If the communication node determines that there is no physical QP corresponding to the first virtual QP, it determines any idle physical QP in the physical QP pool as the first physical QP.

[0141] In this context, idle physical QPs in the physical QP pool are those that do not have a corresponding relationship with any virtual QP, while non-idle physical QPs are those that have a corresponding relationship with any virtual QP. Here, the corresponding relationship refers to the one recorded in the first corresponding relationship.

[0142] If a communication node determines that no pQPN corresponding to the vQPN of the first virtual QP is recorded in the first correspondence, then the first virtual QP does not have a corresponding physical QP. In other words, the communication node determines that there is no physical QP corresponding to the first virtual QP. In this case, the communication node can randomly select any idle physical QP from the aforementioned physical QP pool as the first physical QP. That is, the communication node determines any idle physical QP in the physical QP pool as the first physical QP.

[0143] Optionally, the communication node can maintain a state file (such as a state table with a map data structure) recording the status of each physical QP in the physical QP pool, indicating whether each physical QP is idle or not. In this case, the communication node can query the state file to identify which physical QPs are idle and select one as the first physical QP corresponding to the first virtual QP. Based on the established correspondence between the first virtual QP and the first physical QP's QPN, the communication node also adds a vQPN of the first virtual QP and a pQPN of the first physical QP to the first correspondence, thereby establishing the correspondence between the first virtual QP and the first physical QP.

[0144] Optionally, the communication node can also query the status of each physical QP in the physical QP pool in real time to find any physical QP in the pool that is idle, and identify that physical QP as the first physical QP corresponding to the first virtual QP. Based on the established correspondence between the first virtual QP and the first physical QP's QPN, the communication node also adds a vQPN of the first virtual QP and a pQPN of the first physical QP to the first correspondence, thereby establishing the correspondence between the first virtual QP and the first physical QP in the first correspondence.

[0145] Subsequently, the communication node will also update the status of the first physical QP in the physical QP pool to non-idle. In one example, the communication node will also update the status of the first physical QP in the physical QP pool recorded in the status file to non-idle.

[0146] Understandably, since an idle physical QP in the physical QP pool is one that has not yet been associated with any virtual QP, and a non-idle physical QP is one that has been associated with any virtual QP, when a communication node establishes an association between a first virtual QP and a first physical QP in the physical QP pool and updates the state of the first physical QP to non-idle, when the communication node needs to allocate an idle physical QP from the physical QP pool to another virtual QP, the communication node will not reassign the first physical QP already allocated to the first virtual QP to that other virtual QP. In other words, by updating the state of the physical QPs in the physical QP pool, the communication node can avoid allocating the same physical QP to multiple virtual QPs, thus preventing conflicts.

[0147] It should also be understood that if the communication node is the sender, since the virtual QP only uses the SQ of the physical QP in the physical QP pool when processing RDMA requests, and there is no processing flow association with the RQ of that physical QP, when the communication node queries the status of each physical QP in the physical QP pool and determines any physical QP in the physical QP pool with an idle status as the first physical QP corresponding to the first virtual QP, the communication node can actually query the SQ status of each physical QP in the physical QP pool and determine any physical QP in the physical QP pool with an idle SQ status as the first physical QP corresponding to the first virtual QP. Furthermore, when the communication node adds the correspondence between the first virtual QP and the first physical QP to the first correspondence, what is actually added is the correspondence between the SQ of the first virtual QP and the first physical QP. In addition, the specific operation of the communication node updating the status of the first physical QP to non-idle is actually: the communication node updates the SQ status of the first physical QP to non-idle.

[0148] Similarly, if the communication node is the receiving end, since the virtual QP only uses the RQ of the physical QP in the physical QP pool when processing RDMA requests, and there is no processing flow association with the SQ of that physical QP, when the communication node queries the status of each physical QP in the physical QP pool and determines any physical QP in the physical QP pool with an idle status as the first physical QP corresponding to the first virtual QP, the communication node can actually query the status of the RQ of each physical QP in the physical QP pool and determine any physical QP in the physical QP pool with an idle RQ status as the first physical QP corresponding to the first virtual QP. Furthermore, when the communication node adds the correspondence between the first virtual QP and the first physical QP to the first correspondence, what is actually added is the correspondence between the RQ of the first virtual QP and the first physical QP. In addition, the specific operation of the communication node updating the status of the first physical QP to non-idle is actually: the communication node updates the status of the RQ of the first physical QP to non-idle.

[0149] As can be seen from the above description, the SQ and RQ of the same physical QP in the physical QP pool of this application embodiment are decoupled, and the communication node can allocate the SQ and RQ of the same physical QP to different virtual QPs for use. Therefore, the SQ and RQ of the same physical QP in the physical QP pool of this application embodiment can be in the same state (e.g., both are idle or not idle) or different states (e.g., one is idle and the other is not idle). In this way, the utilization rate of physical QPs in the physical QP pool can be improved, thereby further improving the scalability of the method provided by the embodiment of this application.

[0150] Step 103: The communication node processes the first RDMA request through the first physical QP.

[0151] After determining the first physical QP corresponding to the first virtual QP, the communication node processes the first RDMA request through the first physical QP, for example, by processing the RDMA request through the SQ or RQ of the first physical QP. A detailed explanation of how the communication node processes the first RDMA request through the first physical QP can be found below. Figure 6 or Figure 7 The description of the Chinese method will not be repeated here.

[0152] Step 104: The communication node disconnects the correspondence between the first virtual QP and the first physical QP.

[0153] Optionally, when the communication node determines that the first RDMA request has been processed, it immediately disconnects the correspondence between the first virtual QP and the first physical QP.

[0154] Optionally, the communication node may wait for a preset time after determining that the first RDMA request has been processed, and if it determines that there is no data that needs to be transmitted based on the first QP connection (the first QP connection refers to the connection previously established based on the first virtual QP) within the preset time, then disconnect the correspondence between the first virtual QP and the first physical QP. The value of this preset time is not limited in this embodiment.

[0155] For example, the sending end achieves the purpose of disconnecting the correspondence between the first virtual QP and the first physical QP by deleting the correspondence between the first virtual QP and the first physical QP from the first correspondence.

[0156] After the communication node disconnects the correspondence between the first virtual QP and the first physical QP, it is also necessary to update the status of the first physical QP in the physical QP pool to idle, for example, by updating the status of the first physical QP recorded in the status file to idle.

[0157] Optionally, the correspondence between the first virtual QP and the first physical QP is actually the correspondence between RQ (or SQ) in the first virtual QP and the first physical QP. Then the communication node can delete the correspondence between RQ (or SQ) in the first virtual QP and the first physical QP, and update the status of RQ (or SQ) in the first physical QP to idle.

[0158] In this way, the first physical QP or its RQ (or SQ) can subsequently be used to establish correspondences with other virtual QPs and to carry the RDMA communication functions of those other virtual QPs. That is, the first physical QP or its RQ (or SQ) can subsequently be used to process RDMA requests for other QP connections (QP connections established based on other virtual QPs) that are not connected to the aforementioned first QP. In other words, the first physical QP or its RQ (or SQ) can be time-divisionally called by different virtual QPs to handle their respective RDMA requests; that is, the first physical QP or its RQ (or SQ) can be time-division multiplexed.

[0159] For example, in response to a second RDMA request, the communication node determines a second virtual QP for processing the second RDMA request and identifies a first physical QP as the physical QP corresponding to the second virtual QP. Subsequently, the communication node processes the second RDMA request using the first physical QP and then disconnects the mapping between the second virtual QP and the first physical QP.

[0160] The detailed description of the second RDMA request can be found in the description of the first RDMA request. The second virtual QP includes the third QPC but not the queue; the detailed description of the second virtual QP can be found in the description of the first virtual QP, and will not be repeated here. The third QPC is a virtual QP, meaning it includes connection information but not queue information. It should be noted that the second RDMA request and the first RDMA request are RDMA communication requests for different communication nodes.

[0161] Furthermore, for a detailed explanation of how the communication node determines the first physical QP as the physical QP corresponding to the second virtual QP, please refer to the description of determining the first physical QP corresponding to the first virtual QP in step 102. For a detailed explanation of how the communication node processes the second RDMA request through the first physical QP, please refer to the description of the communication node processing the first RDMA request through the first physical QP in step 103. And for a detailed explanation of how the communication node disconnects the correspondence between the second virtual QP and the first physical QP, please refer to the description of disconnecting the correspondence between the first virtual QP and the first physical QP in step 104. These details will not be repeated here.

[0162] Through steps 101 to 104, in scenarios requiring large-scale RDMA connections, the communication node in this embodiment can create a relatively large number of virtual QPs to establish QP connections, while simultaneously creating a relatively small number of physical QPs (i.e., a physical QP pool). Thus, this embodiment achieves large-scale RDMA connection communication by time-division multiplexing the physical QPs in the physical QP pool using virtual QPs. Since the virtual QPs in this embodiment only include virtual QPCs and not queues, and the virtual QPCs only include connection information and not queue information, the memory footprint of the virtual QPs on the communication node is very small. Therefore, the communication node only needs to control the number of physical QPs in the physical QP pool, and the scheme described in this embodiment itself does not require creating too many physical QPs. Therefore, this embodiment can avoid excessive memory consumption by too many physical QPs, thereby avoiding the impact on communication node performance in large-scale RDMA connection scenarios.

[0163] The above method will be further explained below using the aforementioned communication nodes as the sending end and / or receiving end respectively.

[0164] In the first scenario, the aforementioned communication node is implemented as a transmitter. That is, the method described in this embodiment is applied to the transmitter. Furthermore, the receiver communicating with this transmitter is implemented as a node supporting standard RDMA RC mode communication.

[0165] In this embodiment, all applications / processes on the sending end that need to use RDMA to transmit data have pre-created physical QP pools. Detailed instructions on how the sending applications / processes create physical QP pools can be found above and will not be repeated here.

[0166] refer to Figure 6 , Figure 6 A schematic diagram of a communication method provided in an embodiment of this application is shown. Figure 6 As shown, the method includes a phase for establishing a QP connection (denoted as the first phase) and a phase for processing RDAM requests (denoted as the second phase).

[0167] The following is for reference. Figure 6 First, let's explain the implementation process of the first stage mentioned above.

[0168] In an exemplary embodiment, the implementation process of the first stage can be achieved by the following steps S11 to S15.

[0169] Step S11: The sending end creates the first virtual QP.

[0170] When an application / process on the sending end needs to send data to be transmitted based on RDMA, the application / process can create a first virtual QP through the network card driver (running in the sending end's memory) to generate a virtual QPC (i.e., the first QPC) for the first virtual QP. For example, the application / process creates the first virtual QP by calling the verbs interface through the network card driver, thereby generating the connection information of the first virtual QP. This connection information includes, but is not limited to, the AH, QPN, sPSN, sMSN, retransmission time, and MTU of the first virtual QP. The first virtual QP is used to handle RDMA requests indicating the transmission of data to be transmitted, as described above. That is, at this time, the sending end determines the first virtual QP for handling the first RDMA request in response to the first RDMA request.

[0171] Subsequently, based on the first virtual QP created by the sending end, the first QP connection between the first virtual QP and the receiving end can be established.

[0172] Step S12: The sending end sends the first connection information to the receiving end. The first connection information includes the first QPC.

[0173] The first connection information is used to establish the first QP connection. The first connection information includes, but is not limited to, the AH, vQPN, sPSN, sMSN, etc. of the first virtual QP.

[0174] For example, the sending end can send the first connection information to the receiving end through the communication manager (CM) of the RDMA. Here, RDMAM is a manager used to establish and manage communication connections in the RDMA network.

[0175] Step S13: In response to the first connection information, the receiving end sends the second connection information to the sending end. The second connection information includes the connection information in the QPC of the first QP.

[0176] The receiving application or process can create a first QP in response to the received first connection information. The peer AH (or destination AH) in the QPC of the first QP is the AH of the first virtual QP, the peer QPN (or destination QPN) in the QPC of the first QP is the vQPN of the first virtual QP, the ePSN in the QPC of the first QP is the sPSN of the first virtual QP, and the eMSN in the QPC of the first QP is the sMSN of the first virtual QP. Here, since the receiving end in this embodiment is implemented as a node supporting standard RDMA RC mode communication, the first QP created by the receiving end is a physical QP. Of course, the first QP can also be the virtual QP described in the embodiments of this application, such as those described below. Figure 8 The second virtual QP shown here will not be elaborated upon.

[0177] In one example, if the first QP created by the receiving end is a physical QP, that is, the first QP created by the receiving end is the QP described in the current RDMA standard, then the receiving end is a current standard RDMA communication node. In other words, when the method provided in the embodiments of this application is applied at the sending end, the receiving end can be a standard RDMA communication node, meaning the receiving end does not need to be specially modified to adapt to the communication method provided in the embodiments of this application. Thus, the sending end applying the method described in the embodiments of this application can be used in the current standard RDMA ecosystem; that is, the method described in the embodiments of this application is RDMA-friendly.

[0178] In another example, if the first QP created by the receiving end is a virtual QP as described in the embodiments of this application, that is, both the sending end and the receiving end apply the communication method provided in the embodiments of this application. In this case, both the sending end and the receiving end can reduce the number of physical QPs that need to be created in large-scale connection scenarios, thereby reducing the memory occupation of physical QPs on communication nodes. Since the QP used to establish the QP connection in the embodiments of this application is a virtual QP, and the virtual QPC of the virtual QP contains standard connection information, even if both the sending end and the receiving end apply the communication method provided in the embodiments of this application, there is no need to modify the message format transmitted between the sending end and the receiving end. That is, the communication messages in the method provided in the embodiments of this application can all use the standard RDMA RC protocol fields. Therefore, the method described in the embodiments of this application is easy to implement and eco-friendly.

[0179] After the receiving end establishes the first QP, it sends the second connection information to the sending end. The second connection information includes, but is not limited to, the AH, QPN, sPSN, sMSN, etc. of the first QP.

[0180] For example, the receiving end can send a second connection information to the sending end via RDACAM.

[0181] Step S14: The sending end receives the second connection information and updates the first QPC based on the second connection information.

[0182] Specifically, the sending end updates the destination AH, destination QPN, ePSN, and eMSN in the first QPC based on the second connection information. This completes the establishment of the first QP connection between the first virtual QP and the first QP.

[0183] For example, the sending end assigns the peer AH (or destination AH) in the first QPC to the AH of the first QP, assigns the peer QPN (or destination QPN) in the first QPC to the QPN of the first QP, assigns the ePSN in the first QPC to the sPSN of the first QP, and assigns the eMSN in the first QPC to the sMSN of the first QP.

[0184] Step S15: The sending end determines the corresponding first physical QP for the first virtual QP.

[0185] After the sending end creates the first virtual QP by executing step S11, it selects an idle physical QP (or a physical QP with an idle SQ state) from the physical QP pool as the first physical QP corresponding to the first virtual QP. Thus, the sending end can subsequently carry the RDMA communication function of the first virtual QP through the first physical QP.

[0186] In this case, the sending end also needs to add a correspondence between the first virtual QP and the first physical QP (or the SQ in the first physical QP) in the first correspondence, and update the status of the first physical QP (or the SQ in the first physical QP) to non-idle in the status file, so as to prevent the first physical QP (or the SQ in the first physical QP) from being called by other virtual QPs in the sending end other than the first virtual QP.

[0187] Optionally, after determining the first physical QP corresponding to the first virtual QP, the sending end updates the connection information in the physical QPC (i.e., the second QPC) of the first physical QP based on the virtual QPC (i.e., the first QPC) of the first virtual QP. For example, the sending end updates the sPSN in the second QPC to the sPSN in the first QPC, updates the SMSN in the second QPC to the SMSN in the first QPC, updates the ePSN in the second QPC to the ePSN in the first QPC, updates the eMSN in the second QPC to the eMSN in the first QPC, updates the relevant information of the peer QP (including AH, QPN, etc.) in the second QPC to the relevant information of the peer QP in the first QPC, updates the MTU in the second QPC to the MTU in the first QPC, updates the retransmission time in the second QPC to the retransmission time in the first QPC, and so on.

[0188] It should be understood that after the sending end updates the connection information in the physical QPC (i.e., the second QPC) of the first physical QP based on the virtual QPC (i.e., the first QPC) of the first virtual QP, the network card of the subsequent sending end can directly assemble RDMA packets based on the second QPC cached in the network card and the WQE in the SQ. In this way, the sending end does not need to carry the first QPC in the WQE every time it generates a WQE (as described in the first possible implementation in step S21 below), thereby reducing the bandwidth and PPS requirements of PCIe during the transmission of WQE from the sending end's memory to the network card.

[0189] It is understandable that the sending end does not restrict the execution order of steps S15 and S12-S14. For example, the sending end can execute steps S12-S14 first, and then execute step S15. Alternatively, the sending end can execute step S15 first, and then execute steps S12-S14. Yet another example is that the sending end can execute steps S15 and S12-S14 simultaneously.

[0190] The following is a reference. Figure 6 The implementation process of the second stage described above will be explained.

[0191] In an exemplary embodiment, the implementation process of the second stage can be achieved by the following steps S21 to S23.

[0192] Step S21: The sending end generates a first WQE based on the data to be transmitted and sends the first WQE to the SQ of the first physical QP for waiting processing.

[0193] In the first possible implementation, the sending application / process generates a first WQE based on the data to be transmitted, the type of RDMA operation to be performed on the data, and a first QPC. In this case, the first WQE includes the first QPC. That is, compared to the WQE in standard RDMA communication, the WQE in this implementation includes information about the first QPC.

[0194] The first QPC used to generate the first WQE includes, but is not limited to: the AH and QPN of the first QP (i.e., the peer QP or destination QP of the first virtual QP), and the sPSN of the first virtual QP. Optionally, the application / process can maintain the mapping relationship between virtual QPs and virtual QPCs through the network card driver. For example, the mapping relationship between virtual QPs and virtual QPCs can be implemented through a context table with a map data structure. Taking the first virtual QP that has already established a QP connection with the first QP as an example, the key of the context table is the vQPN of the first virtual QP, and the value is the virtual QPC of the first virtual QP (i.e., the first QPC). Here, the first QPC includes, but is not limited to, the AH and sPSN of the first virtual QP. Thus, after the sending application / process creates (or determines) the first virtual QP, it can query the context table through the vQPN of the first virtual QP to obtain the first QPC. Then, it can generate the first WQE based on the first QPC, the data to be transmitted, and the type of RDMA operation to be performed on the data to be transmitted, and send the first WQE to the SQ of the first physical QP for processing.

[0195] In the second possible implementation, the sending application / process generates a first WQE based on the data to be transmitted and the type of RDMA operation to be performed on that data. In this implementation, the sending end can generate the first WQE based on the standard RDMA communication procedure; the detailed process is not elaborated here.

[0196] Step S22: The sending end processes the first WQE through the network card to complete the transmission of the data to be transmitted.

[0197] In the first possible implementation, when the first WQE carries the virtual QPC (i.e., the first QPC) of the first virtual QP, the network card at the sending end assembles an RDMA message based on the first QPC carried in the first WQE and the data indicated for transmission by the first WQE, and then sends the RDMA message. This process can refer to the process of assembling a message based on the data indicated for transmission by the WQE and the QPC of the QP to which the SQ of the WQE belongs in the standard RDMA communication procedure, and will not be described in detail here.

[0198] In the second possible implementation, when the first WQE is generated according to the standard RDMA communication procedure, that is, the first WQE does not carry the virtual QPC (i.e., the first QPC) of the first virtual QP. In this case, the sending end has already updated the connection information in the physical QPC (i.e., the second QPC) of the first physical QP based on the first QPC in step S15. Therefore, the network card of the sending end can assemble the RDMA message based on the connection information of the second QPC with updated connection information cached in the network card and the data indicated by the first WQE. After that, the sending end sends the assembled RDMA message. It can be seen that in this implementation, the first WQE does not need to carry the first QPC. Therefore, when there are multiple messages sent through the first QP connection containing the first virtual QP, the PCIe bandwidth and PPS required for the sending end to transmit the WQE corresponding to each message in the multiple messages in memory to the network card can be saved. That is, this implementation can save the PCIe bandwidth and PPS of the sending end.

[0199] In response, the receiving end receives the RDMA message carrying the data to be transmitted and processes it. The specific process is the same as the receiving end's processing flow in the standard RDMA RC mode, and will not be repeated here.

[0200] Step S23: The sending end disconnects the correspondence between the first virtual QP and the first physical QP.

[0201] Optionally, the sending end disconnects the correspondence between the first virtual QP and the first physical QP when it determines that the data to be transmitted has been transmitted, or when it determines that the data to be transmitted has been transmitted and there is no need to continue communicating with the peer (i.e. the receiving end) connected to the first QP, or when it determines that the data to be transmitted has been transmitted and there is no need to continue communicating with the peer connected to the first QP for a period of time (such as within a preset duration).

[0202] For example, if the sending application / process determines that it has finished sending the data to be transmitted through the first QP connection, and / or determines that the number of messages sent through the first QP connection is the same as the number of ACK confirmation messages received, or if the sending application / process determines that the first QP connection has not transmitted data within a preset time period after it has finished sending the data to be transmitted through the first QP connection, and / or determines that the number of messages sent through the first QP connection is the same as the number of ACK confirmation messages received, and the first QP connection has not transmitted data within a preset time period thereafter, the sending application / process disconnects the correspondence between the first virtual QP and the first physical QP.

[0203] Wherein, if the sending application / process determines that the first QP connection has not transmitted any more data within a preset time period after sending the data to be transmitted through the first QP connection, and / or if the sending application / process determines that the number of messages sent through the first QP connection is the same as the number of ACK confirmation messages received, and the first QP connection has not transmitted any more data within the subsequent preset time period, it indicates that the data to be transmitted by the sending application / process has been transmitted, and the application / process does not need to (or does not need to) continue to communicate with the peer of the first QP connection.

[0204] The aforementioned ACK is used to confirm that the receiving end has received the message sent by the sending end, and the embodiments of this application do not specifically limit the value of the aforementioned preset duration. It can be understood that when the number of messages sent by the sending end through the first QP connection is the same as the number of messages confirmed by the ACK received, it indicates that the number of messages / messages still being transmitted on the communication link (or in-flight messages / messages) is 0, that is, there are no messages / messages being transmitted on the communication link of the first QP connection. In other words, the message / message carrying the data to be transmitted in the sending end has either not yet been transmitted through the first QP connection or has already been transmitted by the first QP connection.

[0205] In one example, taking an ACK as an acknowledgment that a message has been received, the sender can determine whether the number of messages sent and the number of ACKs received through the first QP connection are the same by maintaining two counters with the same initial value. One counter counts the number of messages sent through the first QP connection, and the other counter counts the number of ACKs received through the first QP connection. Thus, when the values ​​of these two counters are equal, it indicates that the number of messages sent and the number of ACKs received by the sender through the first QP connection are the same; that is, the number of messages sent and the number of ACKs received by the sender through the first QP connection are the same.

[0206] In another example, still using an ACK to confirm the completion of a message reception, the sender can determine whether the number of messages sent and the number of ACKs received through the first QP connection are the same by maintaining a counter with an initial value of a preset value (e.g., 0). Specifically, when the sender sends a message through the first QP connection, the counter is incremented by a unit value (e.g., by 1); when the sender receives an ACK through the first QP connection, the counter is decremented by a unit value (e.g., by 1). Thus, when the counter value returns to the preset value, it indicates that the number of messages sent and the number of ACKs received through the first QP connection are the same; that is, the number of messages sent and the number of ACKs received through the first QP connection are the same.

[0207] For example, the sending application / process can disconnect the mapping between the first virtual QP and the first physical QP by deleting the mapping between the first virtual QP and the first physical QP from the first mapping relationship.

[0208] In addition, after the sending end disconnects the correspondence between the first virtual QP and the first physical QP (or the SQ in the first physical QP), the sending end will also update the status of the first physical QP (or the SQ in the first physical QP) to idle, for example, by updating the status of the first physical QP (or the SQ in the first physical QP) in the status file to idle.

[0209] In this way, the first physical QP (or the SQ in the first physical QP) can be used in subsequent calls to other virtual QPs to carry the RDMA communication function of those other virtual QPs.

[0210] In one possible scenario, before the sending end executes step S23 to disconnect the correspondence between the first virtual QP and the first physical QP (or the SQ in the first physical QP), the sending end generates another RDMA request to send the data to be transmitted to the same receiving end based on RDMA. In this case, in response to the RDMA request, the sending end determines that the virtual QP that has established a first QP connection with the first QP in the same receiving end is the first virtual QP. Afterwards, the sending end can query the first correspondence based on the vQPN of the first virtual QP to determine the first physical QP (or the SQ in the first physical QP) that corresponds to the first virtual QP. For a detailed explanation, please refer to the descriptions of steps 1021 to 1023, which will not be repeated here.

[0211] Subsequently, the sending end executes steps S21 to S23, which will not be described again.

[0212] exist Figure 6 In the illustrated embodiment, the communication node applying the method described in this application is the sender of the data to be transmitted. In this case, the first virtual QP only uses the SQ of the first physical QP when processing the RDMA request, and there is no process flow association between the virtual QP and the RQ of the first physical QP. Therefore, the first correspondence maintained by the sender can only include the correspondence between the SQs in the virtual QP and the physical QP. In this way, the RQ in the physical QP can be used by other virtual QPs (as described below). Figure 7 The aforementioned correspondence will not be elaborated here. In this way, the utilization rate of physical QP in the physical QP pool can be improved.

[0213] Since RDMA transmission operations require CPU (and thus the application / process) involvement, the sending end can maintain the initial mapping in memory for easy access by the application / process or network card driver. Similarly, the aforementioned state file, the mapping between virtual QPs and virtual QPCs, etc., can also be maintained in the sending end's memory for easy access by the application / process or network card driver.

[0214] Furthermore, as described above, the first virtual QP in the sending end can be viewed externally as a standard QP from the receiving end's perspective. However, internally, the functionality implemented by the standard QP needs to be achieved by the first virtual QP and its corresponding first physical QP, or vice versa. Thus, the sending end using the embodiment of this application and the receiving end, implemented as a standard RDMA RC node, only need to communicate via standard RDMA RC, and the communication messages can all use standard RDMA RC protocol fields. Therefore, this embodiment supports interfacing with receiving ends that use third-party network cards (such as network cards from any manufacturer that support standard RDMA RC communication), making it ecosystem-friendly.

[0215] In the second scenario, the aforementioned communication node is implemented as a receiving end. That is, the method described in this embodiment is applied to the receiving end. Furthermore, the sending end communicating with this receiving end is implemented as a node supporting standard RDMA RC mode communication.

[0216] In this embodiment, all applications / processes on the receiving end that need to use RDMA to transmit data have pre-created physical QP pools. Detailed instructions on how the receiving end's applications / processes create physical QP pools can be found above and will not be repeated here.

[0217] refer to Figure 7 , Figure 7 This illustration shows another process diagram of the communication method provided in an embodiment of this application. For example... Figure 7 As shown, the method includes a phase for establishing a QP connection (denoted as the third phase) and a phase for processing RDAM requests (denoted as the fourth phase).

[0218] The following is for reference. Figure 7 The implementation process of the third stage described above will be explained.

[0219] In an exemplary embodiment, the implementation process of the third stage can be achieved by the following steps S31 to S34.

[0220] Step S31: The sending end creates a second QP.

[0221] When an application / process in the sending end generates an RDMA request that requires sending data to be transmitted based on RDMA, the application / process creates a second QP (Queued Instance) through the network card driver to handle the RDMA request indicating the transmission of data to be transmitted, and generates a QPC (Queued Instance Configuration) for the second QP. For example, the application / process creates the second QP and generates its QPC by calling the verbs interface through the network card driver. Since the sending end in this embodiment is a node that supports standard RDMA RC mode communication, the second QP is a physical QP. The QPC of the second QP includes connection information and queue information, which can be referred to in detail above and will not be repeated here. Of course, the second QP can also be a virtual QP as described in the embodiments of this application, such as the following... Figure 8 The first virtual QP shown here will not be described in detail.

[0222] In one example, if the second QP created by the sender is a physical QP, that is, the second QP created by the sender is a QP as described in the current RDMA standard, then the sender is a current standard RDMA communication node. In other words, when the method provided in this application embodiment is applied at the receiving end, the sender can be a standard RDMA communication node, meaning the sender does not need to be specially modified to adapt to the communication method provided in this application embodiment. Thus, the receiving end applying the method described in this application embodiment can be used in the current standard RDMA ecosystem; that is, the method described in this application embodiment is RDMA-friendly.

[0223] In another example, if the second QP created by the sending end is a virtual QP as described in the embodiments of this application, that is, both the sending end and the receiving end apply the communication method provided in the embodiments of this application. In this case, both the sending end and the receiving end can reduce the number of physical QPs that need to be created in large-scale connection scenarios, thereby reducing the memory occupation of physical QPs on communication nodes. Since the QP used to establish the QP connection in the embodiments of this application is a virtual QP, and the virtual QPC of the virtual QP contains standard connection information, even if both the sending end and the receiving end apply the communication method provided in the embodiments of this application, there is no need to modify the message format transmitted between the sending end and the receiving end. That is, the communication messages in the method provided in the embodiments of this application can all use the standard RDMA RC protocol fields. Therefore, the method described in the embodiments of this application is easy to implement and eco-friendly.

[0224] Step S32: The sending end sends third connection information to the receiving end. The third connection information includes the connection information in the QPC of the second QP.

[0225] The third connection information is used to establish a QP connection (referred to as the second QP connection). The third connection information includes, but is not limited to, AH, QPN, sPSN, sMSN, etc. contained in the connection information in the QPC of the second QP.

[0226] For example, the sending end can send third connection information to the receiving end via RDACAM.

[0227] Step S33: In response to the third connection information, the receiving end sends the fourth connection information to the sending end. The fourth connection information includes the virtual QPC of the first virtual QP.

[0228] The receiving application, process, or network card driver can respond to the third connection information received from the sending end by creating a first virtual QP. The peer AH (or destination AH) in the virtual QPC (i.e., the first QPC) of the first virtual QP is the AH of the second QP, the peer QPN (or destination QPN) in the first QPC is the QPN of the second QP, the ePSN in the first QPC is the sPSN of the second QP, and the eMSN in the first QPC is the sMSN of the second QP.

[0229] After the receiving end creates the first virtual QP, it sends the fourth connection information to the sending end. The fourth connection information includes the first QPC, and the first virtual QPC includes, but is not limited to, the AH, QPN, sPSN, sMSN, etc. of the first virtual QP.

[0230] For example, the receiving end can send fourth connection information to the sending end via RDACAM.

[0231] Step S34: The sending end receives the fourth connection information and updates the connection information in the QPC of the second QP based on the fourth connection information.

[0232] Specifically, the sending application / process updates the destination AH, destination QPN, ePSN, and eMSN in the second QP's QPC based on the received fourth connection information. This completes the establishment of the second QP connection between the second QP and the first virtual QP.

[0233] For example, the sending end assigns the peer AH (or destination AH) of the connection information in the QPC of the second QP to the AH of the first virtual QP, assigns the peer QPN (or destination QPN) of the connection information in the QPC of the second QP to the QPN of the first virtual QP, assigns the ePSN of the connection information in the QPC of the second QP to the sPSN of the first virtual QP, and assigns the eMSN of the connection information in the QPC of the second QP to the sMSN of the first virtual QP.

[0234] The following is a reference. Figure 7The implementation process of the fourth stage described above will be explained.

[0235] In an exemplary embodiment, the fourth stage can be implemented by the following steps S41 to S44.

[0236] Step S41: The sending end generates a second WQE based on the data to be transmitted, and processes the second WQE through the network card to complete the sending of the RDMA request to the receiving end.

[0237] After the sending application / process creates a second QP and establishes a connection between the second QP and the first virtual QP, it can generate a second WQE based on the data to be transmitted and the type of RDMA operation to be performed on that data, and then send the second WQE into the SQ of the second QP. The sending network card can process the WQE in the SQ of the second QP sequentially, thereby completing the sending of an RDMA request to the receiving end through the second QP connection. This RDMA request indicates the reception of the data it carries. This process can be handled with reference to the send operation procedure in standard RDMA RC mode, and will not be elaborated further.

[0238] Step S42: The receiving end receives the RDMA request sent by the sending end and determines the first virtual QP to process the RDMA request.

[0239] Taking the first RDMA request as an example, after receiving the first RDMA request, the receiving end determines the virtual QP represented by the destination QPN carried in the first RDMA request as the first virtual QP.

[0240] In one example, the basic transmission header (BTH) of the first RDMA request carries the QPN of the destination QP. It should be understood that the target QPN carried in the first RDMA request is the vQPN of the first virtual QP obtained by the sending end from the receiving end when the sending end's second QP and the receiving end's first virtual QP establish a QP connection according to the process of steps S31 to S34.

[0241] Step S43: The receiving end queries the first correspondence to determine the first physical QP corresponding to the first virtual QP in the receiving end, and processes the received RDMA request through the physical QPC of the first physical QP.

[0242] Specifically, the receiving end can query the first correspondence based on the vQPN of the first virtual QP to determine the first physical QP corresponding to the first virtual QP in the receiving end. For details, please refer to the descriptions of steps 1021 to 1023, which will not be repeated here. The implementation of the first correspondence can be implemented as a QP correspondence table as described above, which will not be repeated here.

[0243] In one possible scenario, after the receiving end determines a first virtual QP, queries a first correspondence based on the QPN of the first virtual QP, and determines that the first correspondence does not contain a physical QP corresponding to the first virtual QP, thus identifying any idle physical QP (or any physical QP with an idle RQ state) in the physical QP pool as the first physical QP corresponding to the first virtual QP, the receiving end can update the connection information in the physical QPC (i.e., the second QPC) of the first physical QP based on the virtual QPC (i.e., the first QPC) of the first virtual QP. For example, the receiving end updates the peer AH, destination QPN, ePSN, and eMSN in the second QPC sequentially based on the peer AH, destination QPN, ePSN, and eMSN in the first QPC. For example, the receiving end assigns the peer AH in the connection information of the second QPC to the peer AH in the first QPC, assigns the destination QPN in the connection information of the second QPC to the destination QPN in the first QPC, assigns the ePSN in the connection information of the second QPC to the ePSN in the first QPC, and assigns the eMSN in the connection information of the second QPC to the eMSN in the first QPC. In this way, the goal of lightweightly rotating the virtual QPC of the first virtual QP to the physical QPC of the first physical QP is achieved.

[0244] In one exemplary implementation, the network interface card (NIC) of the receiving end maintains the aforementioned correspondence between virtual QPs and virtual QPCs (after creating a virtual QP in memory, the receiving end loads the virtual QP and its corresponding QPC into the NIC to obtain the correspondence). Therefore, after the receiving end determines the first virtual QP and the first physical QP, it can query the correspondence between the virtual QP and the virtual QPC based on the vQPN of the first virtual QP to find the virtual QPC (i.e., the first QPC) of the first virtual QP, including but not limited to the peer AH, destination QPN, ePSN, and eMSN. Then, based on the peer AH, destination QPN, ePSN, and eMSN in the first QPC, the receiving end sequentially updates the peer AH, destination QPN, ePSN, and eMSN in the physical QPC (i.e., the second QPC) of the first physical QP.

[0245] Subsequently, the receiving end processes the received RDMA request using the updated connection information of the second QPC and according to the standard RDMA RC processing procedure. For example, the receiving end uses the ePSN and eMSN in the updated second QPC to determine whether the data carried in the received RDMA request has been lost to ensure reliability. In addition, the receiving end can also assemble an ACK packet for the RDMA request based on the peer AH, destination QPN, ePSN, and eMSN in the updated connection information of the second QPC, and return the ACK packet to the sending end.

[0246] In this way, the first physical QP can obtain the virtual QPC (i.e., the first QPC) of the first virtual QP that participated in establishing the QP connection. Thus, at the receiving end applying the scheme of this application embodiment, the first physical QP can determine whether the received message is lost based on its own physical QPC updated based on the first QPC, so as to ensure the reliability of the message. It can also generate an ACK to be returned to the sending end based on the updated physical QPC. In this way, the switching between the first QPC and the second QPC can be reduced, so as not to occupy PCIe bandwidth and pps.

[0247] In this implementation, after the receiving end determines any idle physical QP (or any physical QP with an idle RQ state) in the physical QP pool as the first physical QP corresponding to the first virtual QP, it also updates the state of the first physical QP (or the RQ in the first physical QP) to non-idle, for example, by updating the state of the first physical QP (or the RQ in the first physical QP) in the state file to idle. A detailed description of the state file can be found above and will not be repeated here. Furthermore, the receiving end also adds a correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP) to the first correspondence.

[0248] In another possible scenario, when the receiving end queries the first correspondence based on the QPN of the first virtual QP and determines that the first correspondence includes the correspondence between the first virtual QP and the first physical QP, it indicates that the physical QPC (i.e., the second QPC) of the first physical QP has been updated based on the virtual QPC (i.e., the first QPC) of the first virtual QP. In this case, the receiving end can process the received RDMA request through the first physical QP according to the standard RDMA RC processing flow.

[0249] Step S44: The receiving end disconnects the correspondence between the first virtual QP and the first physical QP.

[0250] Optionally, if the receiving end does not receive a message through the second QP connection for a preset time period, or if the receiving end receives a disconnection indication sent by the sending end, the receiving end disconnects the correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP). In this embodiment, the value of the preset time period is not specifically limited. The disconnection indication is used to indicate the disconnection of the correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP).

[0251] For example, the receiving end achieves the purpose of disconnecting the correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP) by deleting the correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP) from the first correspondence.

[0252] In addition, after the receiving end disconnects the correspondence between the first virtual QP and the first physical QP (or the RQ in the first physical QP), the receiving end will also update the status of the first physical QP (or the RQ in the first physical QP) to idle, for example, by updating the status of the first physical QP (or the RQ in the first physical QP) in the status file to idle.

[0253] In this way, the first physical QP (or the RQ in the first physical QP) can be used in subsequent calls to other virtual QPs to carry the RDMA communication function of those other virtual QPs.

[0254] exist Figure 7 In the illustrated embodiment, the communication node applying the method described in this application is a receiving end for receiving data. In this case, the first virtual QP only uses the RQ of the first physical QP when processing RDMA requests, and there is no processing flow association between the virtual QP and the SQ of the first physical QP. Therefore, the first correspondence maintained by the receiving end can be the correspondence between the virtual QP and the RQ in the physical QP. In this way, the SQ in the physical QP can be used by other virtual QPs (as described above). Figure 6 The aforementioned correspondence will not be elaborated here. In this way, the utilization rate of physical QP in the physical QP pool can be improved.

[0255] Since the RDMA receive operation is completed within the network card, the receiving end can maintain the initial mapping in the network card for easy lookup. Similarly, the aforementioned status file, the mapping between virtual QPs and virtual QPCs, etc., can also be maintained in the receiving end's network card for easy lookup.

[0256] Furthermore, as described above, the first virtual QP in the receiving end can be viewed externally as a standard QP from the perspective of the sending end. However, internally, the functions implemented by the standard QP need to be implemented by the first virtual QP and the first physical QP corresponding to it, or the functions implemented by the standard QP need to be implemented by the RQ of the first virtual QP and the first physical QP corresponding to it. In this way, the receiving end using the scheme of this application and the sending end implemented as a standard RDMA RC node only need to communicate through standard RDMA RC, and the communication messages can all use the standard RDMA RC protocol fields. Therefore, the embodiments of this application support the sending end that connects to third-party network cards (such as network cards from any manufacturer that support standard RDMA RC communication), thus making it ecosystem-friendly.

[0257] In the third scenario, for two communication nodes using the scheme described in the embodiments of this application, one communication node is implemented as a transmitter, and the other communication node is implemented as a receiver. That is, the method described in the embodiments of this application is applied to both the transmitter and the receiver.

[0258] refer to Figure 8 , Figure 8 This illustration shows yet another schematic diagram of the communication method provided in an embodiment of this application. (In conjunction with...) Figure 6 and Figure 7 ,like Figure 8 As shown, the implementation process of the sending end can be referred to Figure 6 The implementation process of the sending and receiving ends shown can be referenced. Figure 7 The receiving end is shown. Among them, Figure 8 The first virtual QP in the transmitter shown can correspond to Figure 6 The first virtual QP in the transmitting end shown, therefore, Figure 8 The receiving end shown can create a second virtual QP in response to the connection information of the virtual QPC containing the first virtual QP sent by the sending end. The second virtual QP can correspond to Figure 7 The first virtual QP in the receiver shown. Furthermore, Figure 8 The sending end shown executes Figure 6 The steps performed by the sending end in the method are as follows. Figure 8 The receiver shown executes... Figure 7 The method, executed by the receiving end, establishes a third QP connection between the first and second virtual QPs, and enables data transmission through this third QP connection. A detailed description can be found in [reference needed]. Figure 6 and Figure 7 The relevant descriptions will not be repeated here.

[0259] For a detailed explanation of the technical effects corresponding to the third scenario, please refer to... Figure 6 and Figure 7 The descriptions of the technical effects will not be repeated here.

[0260] In summary, the communication method provided in this application embodiment is applicable to RDMA large-scale connection scenarios. When applying the method provided in this application embodiment in RDMA large-scale connection scenarios, it is not necessary to create too many physical QPs. Instead, physical QPs in the physical QP pool (or SQs or RQs of physical QPs) are used for data transmission in RDMA large-scale connection scenarios through time-division multiplexing. Thus, the communication method provided in this application embodiment can avoid excessive physical QPs occupying too much memory of the communication node, thereby avoiding the impact of excessive physical QPs occupying too much memory on the performance of the communication node.

[0261] Furthermore, the communication method provided in this application embodiment can establish stable QP connections through virtual QPs, thereby storing the connection information of virtual QPs in the network interface card (NIC) of the communication node. Within the communication node, the connection information of virtual QPs can be updated to the physical QPs based on the correspondence between virtual and physical QPs. Thus, the physical QP corresponding to the virtual QP used to establish the QP connection can query the QPC based on the QPN and process the RDMA request based on the queried QPC. Compared to DCT, which uses time-division multiplexing of QPs in the QP pool, communication nodes using DCT carry connection information of the QP used to process RDMA requests in each DCT message, thus increasing network bandwidth consumption. In this application embodiment, however, it is not necessary to carry connection information in the RDMA request; the connection information in the QPC can be queried locally by the communication node based on the QPN. Therefore, this application embodiment achieves time-division multiplexing of physical QPs by replacing transmission with a lookup (searching for connection information in the QPC on the NIC), saving network bandwidth consumption.

[0262] For example, refer to Figure 9 , Figure 9 This diagram illustrates an application architecture schematic of the communication method provided in the embodiments of this application applied in an asymmetric manner. Specifically, applying the communication method provided in the embodiments of this application in an asymmetric manner means applying the method provided in the embodiments of this application only in one of the sending and receiving ends included in the communication system.

[0263] like Figure 9 As shown in (a), a node supporting standard RDMA RC communication and applying the method provided in the embodiments of this application is the sending end, and a node supporting standard RDMA RC communication but not applying the method provided in the embodiments of this application is the receiving end. The sending end establishes a QP connection with the receiving end by using a virtual QP. After the QP connection is established, the sending end performs physical QP scheduling for the virtual QP with the established QP connection, that is, it determines an idle physical QP (or a physical QP with an idle SQ state) for that virtual QP. Then, the sending end implements RDMA communication with the receiving end's physical QP through this physical QP (or the physical QP's SQ), and disconnects the correspondence between the physical QP (or the physical QP's SQ) and the virtual QP used to establish the QP connection after the communication ends, so that the physical QP (or the physical QP's SQ) can be used to provide RDMA communication functionality for other virtual QPs.

[0264] like Figure 9As shown in (b), a node supporting standard RDMA RC communication but not applying the method provided in this application is the sending end, and a node supporting standard RDMA RC communication and applying the method provided in this application is the receiving end. When the receiving end establishes a QP connection with the sending end, the receiving end uses a virtual QP to establish the QP connection. After the QP connection is established, the receiving end performs physical QP scheduling for the virtual QP with the established QP connection, that is, it determines an idle physical QP (or a physical QP with an idle RQ state) for the virtual QP. Then, the receiving end implements RDMA communication with the sending end's physical QP through the physical QP (or the physical QP's RQ), and disconnects the correspondence between the physical QP (or the physical QP's RQ) and the virtual QP used to establish the QP connection after the communication ends, so that the physical QP can be used to provide RDMA communication functionality for other virtual QPs.

[0265] The above mainly describes the solution provided by the embodiments of this application from a methodological perspective.

[0266] To achieve the above functions, such as Figure 10 As shown, Figure 10 A schematic diagram of a communication device according to an embodiment of this application is shown. The communication device 1000 is used to execute the communication method described above, for example, to execute... Figure 4 , Figure 5 , Figure 6 or Figure 7 The method shown. The communication device 1000 may include a determining unit 1001 and a processing unit 1002.

[0267] Determining unit 1001 is configured to determine a first virtual QP in response to a first RDMA request, and to determine a first physical QP corresponding to the first virtual QP. Processing unit 1002 is configured to process the first RDMA request through the first physical QP, and to disconnect the correspondence between the first virtual QP and the first physical QP. The first virtual QP includes a first QPC, does not include a queue, and the first physical QP includes a second QPC and a queue.

[0268] As an example, combined Figure 4 The determining unit 1001 can be used to execute steps 101 to 102, and the processing unit 1002 can be used to execute steps 103 to 104.

[0269] Optionally, the processing unit 1002 is further configured to: update the state of the first physical QP to idle.

[0270] Optionally, the determining unit 1001 is further configured to, in response to the second RDMA request, determine a second virtual QP, and to determine a first physical QP as the physical QP corresponding to the second virtual QP. The processing unit 1002 is further configured to process the second RDMA request through the first physical QP, and to disconnect the correspondence between the second virtual QP and the first physical QP. The second virtual QP includes a third QPC, but does not include a queue.

[0271] Optionally, the determining unit 1001 is specifically configured to: query a first correspondence relationship to determine whether there exists a physical QP that corresponds to the first virtual QP; if it is determined that there exists a physical QP that corresponds to the first virtual QP, then determine the physical QP that corresponds to the first virtual QP as the first physical QP. The first correspondence relationship includes the correspondence between virtual QPs and physical QPs in the physical QP pool.

[0272] As an example, combined Figure 5 The determination unit 1001 can be used to execute steps 1021 to 1022.

[0273] Optionally, the determining unit 1001 is further configured to: if it is determined that there is no physical QP corresponding to the first virtual QP, determine any idle physical QP in the physical QP pool as the first physical QP.

[0274] As an example, combined Figure 5 Unit 1001 can be used to execute step 1023.

[0275] Optionally, the processing unit 1002 is further configured to: update the state of the first physical QP in the physical QP pool to non-idle.

[0276] Optionally, the processing unit 1002 is further configured to: add a correspondence between the first virtual QP and the first physical QP to the first correspondence.

[0277] Optionally, the first correspondence includes the correspondence between the virtual QP and the transmit queue SQ contained in the physical QP in the physical QP pool, and / or, the first correspondence includes the correspondence between the virtual QP and the receive queue RQ contained in the physical QP in the physical QP pool.

[0278] Optionally, when the communication device 1000 is applied to the transmitting end, the first correspondence includes the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool.

[0279] Optionally, when the communication device 1000 is applied to the receiving end, the first correspondence includes the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool.

[0280] Optionally, the communication device 1000 is applied to the transmitting end, and the first RDMA request is a request to send data to be transmitted to the receiving end based on RDMA. The determining unit 1001 is specifically used to: determine the virtual QP used to establish the first QP connection in the transmitting end as the first virtual QP, and the first QP connection is the QP connection established between the transmitting end and the first QP in the receiving end.

[0281] Optionally, the processing unit 1002 is further configured to: create a first virtual QP before determining the first virtual QP, and establish a first QP connection based on the first virtual QP.

[0282] As an example, combined Figure 6 The processing unit 1002 can be used to execute steps S11 to S14.

[0283] Optionally, the processing unit 1002 is specifically used to: generate a first WQE, the first WQE including a first QPC; and process the first WQE through the SQ of the first physical QP to complete the transmission of the data to be transmitted.

[0284] As an example, combined Figure 6 The processing unit 1002 can be used to execute steps S21 to S22.

[0285] Optionally, the processing unit 1002 is specifically used to: update the second QPC based on the first QPC; and process the first RDMA request through the updated second QPC.

[0286] Optionally, the communication device 1000 is applied at the receiving end, and the communication device 1000 further includes: a receiving unit 1003, configured to receive a first RDMA request sent by the transmitting end, the first RDMA request indicating the reception of data carried in the first RDMA request; and a determining unit 1001, specifically configured to determine the virtual QP represented by the destination QPN carried in the first RDMA request as the first virtual QP.

[0287] As an example, combined Figure 7 The receiving unit 1003 determines the unit 1001 and can be used to execute step S42.

[0288] Optionally, the processing unit 1002 is further configured to: before determining the first virtual QP, in response to receiving connection information sent by the sending end, create the first virtual QP, the connection information including connection information in the QPC of the second QP in the sending end; and establish a second QP connection between the first virtual QP and the second QP.

[0289] As an example, combined Figure 7 The processing unit 1002 can be used to execute step S33.

[0290] Optionally, the processing unit 1002 is specifically configured to: when a first virtual QP is determined and any idle physical QP in the physical QP pool is determined as the first physical QP, update the second QPC based on the first QPC; and process the first RDMA request through the updated second QPC.

[0291] Optionally, the first QPC is a virtual QPC, which contains connection information but does not contain queue information. The second QPC is a physical QPC, which contains both connection information and queue information.

[0292] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the communication devices 1000 provided above and the description of their beneficial effects can be found in the corresponding method embodiments described above, which will not be repeated here.

[0293] Those skilled in the art will readily recognize that, based on the units and algorithm steps described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0294] It should be noted that, Figure 10 The module / unit division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. The functions implemented by the integrated modules described above can be implemented either in hardware or as software functional modules.

[0295] Optionally, in Figure 10 When the communication device shown is implemented through software code, the software code used to implement the functions of the communication device can run in kernel mode and / or user mode, without limitation.

[0296] This application provides a communication device for implementing some or all of the functions of the methods provided in this application. For example, the communication device can act as a communication transmitter to implement some or all of the functions of the methods executed by the transmitter in the above method embodiments. As another example, the communication device can act as a communication receiver to implement some or all of the functions of the methods executed by the receiver in the above method embodiments.

[0297] refer to Figure 11 , Figure 11 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Figure 11 As shown, the communication device 1100 includes a processor 1101, a memory 1102, a network interface 1103, and a bus 1104. The processor 1101, memory 1102, and network interface 1103 are connected to each other via the bus 1104.

[0298] Processor 1101 may include a general-purpose processor and / or a dedicated hardware chip. A general-purpose processor may include a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU). A CPU may be a single-core processor or a multi-core processor. A dedicated hardware chip is a high-performance processing hardware module. Dedicated hardware chips include at least one of the following: digital signal processing (DSP), data processing unit (DPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, neural processing unit (NPU), tensor processing unit (TPU), artificial intelligence (AI) chip, or network processor (NP). Processor 1101 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the method provided in the embodiments of this application can be accomplished by the integrated logic circuit of the hardware in the processor 1101 or by instructions in the form of software.

[0299] Memory 1102 is used to store computer programs, including an operating system 1102a and executable code (i.e., program instructions) 1102b. Memory 1102 may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other types of static storage devices capable of storing static information and instructions; or, for example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), or synchronous linked dynamic random access memory (SDRAM). DRAM (SLDRAM) or other types of dynamic storage devices capable of storing information and instructions, such as read-only optical discs or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired executable code in the form of instructions or data structures and accessible by a computer, but not limited thereto. For example, memory 1102 is used to store the first correspondence, the status file of physical QPs, the correspondence between virtual QPs and virtual QPCs, etc. Memory 1102 may exist independently and be connected to processor 1101 via bus 1104. Alternatively, memory 1102 and processor 1101 may be integrated together. Memory 1102 can store executable code. When the executable code stored in memory 1102 is executed by processor 1101, processor 1101 performs some or all of the functions of the method provided in the embodiments of this application. Please refer to the relevant descriptions in the foregoing embodiments for the implementation of the process by processor 1101. The memory 1102 may also include software modules and data required by other running processes, such as the operating system.

[0300] Network interface 1103 uses transceiver modules, such as, but not limited to, transceivers, to enable communication with other devices or communication networks. For example, network interface 1103 can be any one or any combination of the following devices: communication interfaces (such as Ethernet interfaces), wireless network cards, and other devices with network access capabilities. Network interface 1103 includes a receiving unit for receiving data / messages and a sending unit for sending data / messages.

[0301] Bus 1104 can be any type of communication bus used to interconnect internal devices (e.g., memory 1102, processor 1101, network interface 1103) of communication device 1100. For example, a system bus. This embodiment illustrates the interconnection of the aforementioned devices within communication device 1100 via bus 1104. Optionally, the aforementioned devices within communication device 1100 can also communicate with each other using other connection methods besides bus 1104; for example, the aforementioned devices within communication device 1100 can be interconnected via internal logic interfaces.

[0302] It should be noted that the aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices. Furthermore, the descriptions of the processes corresponding to the various figures above each have their own emphasis; for parts of a process not described in detail in one figure, please refer to the relevant descriptions of other processes.

[0303] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product providing the program development platform includes one or more computer instructions, which, when loaded and executed in the kernel mode or user mode of the communication device 1100, implement some or all of the functions of the methods provided in the embodiments of this application.

[0304] Furthermore, computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0305] As an example, combined Figure 10The function implemented by the receiving unit 1003 in the communication device 1000 can be achieved through Figure 11 The network interface 1103 shown is implemented. The functions implemented by the determining unit 1001 and the processing unit 1002 in the communication device 1000 can be achieved through... Figure 11 Processor 1101 in the middle executes Figure 11 The program code is implemented in memory 1102.

[0306] This application also provides a communication device. The communication device includes a processor, a network interface card (NIC), and a NIC driver. The NIC driver manages the NIC's hardware, and the processor coordinates with the NIC to execute the communication method provided in this application.

[0307] This application also provides a communication system, which includes a sending end and a receiving end. The sending end can be used to execute the operations performed by the sending end in the method provided in this application, and / or the receiving end can be used to execute the operations performed by the receiving end in the method provided in this application.

[0308] This application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by a processor, a computing device including the processor, or a computer system, the processor, the computing device including the processor, or the computer system performs the communication method provided in this application.

[0309] This application also provides a computer program product containing instructions that, when executed by a processor, a computing device including a processor, or a computer system, cause the processor, the computing device including a processor, or the computer system to implement the communication method provided in this application.

[0310] A computer system is a system with computational processing capabilities. A computer system generally includes a processor and memory. The processor retrieves and executes instructions stored in memory to enable the computer system to implement the communication methods described above. Optionally, a computer system may also include at least one of an input interface or an output interface. The processor, memory, input interface, and output interface of the computer system are interconnected through internal connection paths.

[0311] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0312] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0313] This application also provides a chip that includes a processor for running program instructions or code. The chip or a device containing the chip can be used to execute the communication method provided in this application. Exemplarily, the chip further includes an input interface, an output interface, and a memory. The chip's input interface, output interface, processor, and memory are connected via internal interconnection paths. The memory in the chip stores program instructions or code executed by the processor, and the input and output interfaces are used for communication between the chip and other chips or devices.

[0314] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "multiple" refers to at least two, unless otherwise expressly defined.

[0315] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0316] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0317] It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0318] It should be understood that the term "comprising" (also referred to as "includes", "including", "comprises" and / or "comprising") as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0319] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0320] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

[0321] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication method, characterized in that, include: In response to a first remote direct memory access (RDMA) request, a first virtual queue pair (QP) is determined, the first virtual QP including a first queue pair context (QPC), and the first virtual QP not including queues. Determine the first physical QP corresponding to the first virtual QP, wherein the first physical QP includes a second QPC and a queue; The first RDMA request is processed through the first physical QP; Disconnect the correspondence between the first virtual QP and the first physical QP.

2. The method according to claim 1, characterized in that, The method further includes: Update the state of the first physical QP to idle.

3. The method according to claim 2, characterized in that, The method further includes: In response to a second RDMA request, a second virtual QP is determined, the second virtual QP including a third QPC, and the second virtual QP excluding queues; The first physical QP is determined as the physical QP corresponding to the second virtual QP; The second RDMA request is processed through the first physical QP; Disconnect the correspondence between the second virtual QP and the first physical QP.

4. The method according to any one of claims 1 to 3, characterized in that, Determining the first physical QP corresponding to the first virtual QP includes: Query the first correspondence relationship to determine whether there is a physical QP that corresponds to the first virtual QP; wherein, the first correspondence relationship includes the correspondence between virtual QP and physical QP in the physical QP pool; If it is determined that there exists a physical QP that corresponds to the first virtual QP, the physical QP that corresponds to the first virtual QP is determined as the first physical QP.

5. The method according to claim 4, characterized in that, The method further includes: If it is determined that there is no physical QP corresponding to the first virtual QP, then any idle physical QP in the physical QP pool is determined as the first physical QP.

6. The method according to claim 5, characterized in that, The method further includes: Update the state of the first physical QP in the physical QP pool to non-idle.

7. The method according to claim 5 or 6, characterized in that, The method further includes: Add a correspondence between the first virtual QP and the first physical QP to the first correspondence.

8. The method according to any one of claims 4 to 7, characterized in that, The first correspondence includes the correspondence between a virtual QP and the transmit queue SQ contained in a physical QP in the physical QP pool, and / or, the first correspondence includes the correspondence between a virtual QP and the receive queue RQ contained in a physical QP in the physical QP pool.

9. The method according to claim 8, characterized in that, When the method is applied to the sending end, the first correspondence includes the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool.

10. The method according to claim 8 or 9, characterized in that, When the method is applied to the receiving end, the first correspondence includes the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool.

11. The method according to any one of claims 1 to 10, characterized in that, The method is applied at the transmitting end, where the first RDMA request is a request to send data to be transmitted to the receiving end based on RDMA, and determining the first virtual QP includes: The virtual QP used to establish the first QP connection in the sending end is identified as the first virtual QP, and the first QP connection is the QP connection established between the sending end and the first QP in the receiving end.

12. The method according to claim 11, characterized in that, Prior to determining the first virtual QP, the method further includes: Create the first virtual QP; The first QP connection is established based on the first virtual QP.

13. The method according to claim 11 or 12, characterized in that, The process of processing the first RDMA request through the first physical QP includes: Generate a first work queue element WQE, wherein the first WQE includes the first QPC; The first WQE is processed by the SQ of the first physical QP to complete the transmission of the data to be transmitted.

14. The method according to claim 11 or 12, characterized in that, The first RDMA updates the second QPC based on the first QPC by processing the first physical QP; The first RDMA request is processed by the updated second QPC.

15. The method according to any one of claims 1 to 10, characterized in that, The method is applied at the receiving end, wherein determining the first virtual QP includes: Receive the first RDMA request sent by the sending end, wherein the first RDMA request indicates that the data carried by the first RDMA request be received; The virtual QP represented by the destination queue pair number QPN carried in the first RDMA request is determined as the first virtual QP.

16. The method according to claim 15, characterized in that, Prior to determining the first virtual QP, the method further includes: In response to receiving connection information sent by the sending end, a first virtual QP is created; the connection information includes connection information in the QPC of the second QP in the sending end. Establish a second QP connection between the first virtual QP and the second QP.

17. The method according to claim 15 or 16, characterized in that, The process of processing the RDMA request through the first physical QP includes: If the first virtual QP is determined and any idle physical QP in the physical QP pool is determined as the first physical QP, the second QPC is updated based on the first QPC; The first RDMA request is processed by the updated second QPC.

18. The method according to any one of claims 1 to 17, characterized in that, The first QPC is a virtual QPC, which contains connection information but does not contain queue information. The second QPC is a physical QPC, which contains both connection information and queue information.

19. A communication device, characterized in that, include: A determining unit is configured to, in response to a first remote direct memory access (RDMA) request, determine a first virtual queue pair (QP) and a first physical QP corresponding to the first virtual QP; wherein the first virtual QP includes a first queue pair context (QPC), the first virtual QP does not include queues, and the first physical QP includes a second QPC and queues. The processing unit is configured to process the first RDMA request through the first physical QP, and to disconnect the correspondence between the first virtual QP and the first physical QP.

20. The apparatus according to claim 19, characterized in that, The processing unit is also used for: Update the state of the first physical QP to idle.

21. The apparatus according to claim 20, characterized in that, The determining unit is further configured to determine a second virtual QP in response to a second RDMA request, and to determine the first physical QP as the physical QP corresponding to the second virtual QP; wherein the second virtual QP includes a third QPC, and the second virtual QP does not include a queue; The processing unit is further configured to process the second RDMA request through the first physical QP, and to disconnect the correspondence between the second virtual QP and the first physical QP.

22. The apparatus according to any one of claims 19 to 21, characterized in that, The determining unit is specifically used for: Query the first correspondence relationship to determine whether there is a physical QP that corresponds to the first virtual QP; wherein, the first correspondence relationship includes the correspondence between virtual QP and physical QP in the physical QP pool; If it is determined that there exists a physical QP that corresponds to the first virtual QP, the physical QP that corresponds to the first virtual QP is determined as the first physical QP.

23. The apparatus according to claim 22, characterized in that, The determining unit is further configured to: If it is determined that there is no physical QP corresponding to the first virtual QP, then any idle physical QP in the physical QP pool is determined as the first physical QP.

24. The apparatus according to claim 23, characterized in that, The processing unit is also used for: Update the state of the first physical QP in the physical QP pool to non-idle.

25. The apparatus according to claim 23 or 24, characterized in that, The processing unit is also used for: Add a correspondence between the first virtual QP and the first physical QP to the first correspondence.

26. The apparatus according to any one of claims 22 to 25, characterized in that, The first correspondence includes the correspondence between a virtual QP and the transmit queue SQ contained in a physical QP in the physical QP pool, and / or, the first correspondence includes the correspondence between a virtual QP and the receive queue RQ contained in a physical QP in the physical QP pool.

27. The apparatus according to claim 26, characterized in that, When the device is applied to the transmitting end, the first correspondence includes the correspondence between the virtual QP and the SQ contained in the physical QP in the physical QP pool.

28. The apparatus according to claim 26 or 27, characterized in that, When the device is applied to the receiving end, the first correspondence includes the correspondence between the virtual QP and the RQ contained in the physical QP in the physical QP pool.

29. The apparatus according to any one of claims 19 to 28, characterized in that, The device is applied to the transmitting end, the first RDMA request is a request to send data to be transmitted to the receiving end based on RDMA, and the determining unit is specifically used for: The virtual QP used to establish the first QP connection in the sending end is identified as the first virtual QP, and the first QP connection is the QP connection established between the sending end and the first QP in the receiving end.

30. The apparatus according to claim 29, characterized in that, The processing unit is also used for: Before determining the first virtual QP, the first virtual QP is created, and the first QP connection is established based on the first virtual QP.

31. The apparatus according to claim 29 or 30, characterized in that, The processing unit is specifically used for: Generate a first work queue element WQE, wherein the first WQE includes the first QPC; The first WQE is processed by the SQ of the first physical QP to complete the transmission of the data to be transmitted.

32. The apparatus according to claim 29 or 30, characterized in that, The processing unit is specifically used for: Update the second QPC based on the first QPC; The first RDMA request is processed by the updated second QPC.

33. The apparatus according to any one of claims 19 to 28, characterized in that, The device is used at the receiving end, and the device further includes: The receiving unit is configured to receive the first RDMA request sent by the sending end, wherein the first RDMA request indicates that the data carried by the first RDMA request be received. The determining unit is specifically used to determine the virtual QP represented by the destination QPN carried in the first RDMA request as the first virtual QP.

34. The apparatus according to claim 33, characterized in that, The processing unit is also used for: Before determining the first virtual QP, in response to receiving connection information sent by the sending end, the first virtual QP is created; the connection information includes the connection information in the QPC of the second QP in the sending end. Establish a second QP connection between the first virtual QP and the second QP.

35. The apparatus according to claim 33 or 34, characterized in that, The processing unit is specifically used for: If the first virtual QP is determined and any idle physical QP in the physical QP pool is determined as the first physical QP, the second QPC is updated based on the first QPC; The first RDMA request is processed by the updated second QPC.

36. The apparatus according to any one of claims 19 to 35, characterized in that, The first QPC is a virtual QPC, which contains connection information but does not contain queue information. The second QPC is a physical QPC, which contains both connection information and queue information.

37. A communication device, characterized in that, include: The device includes a memory, a network interface, and one or more processors, the one or more processors receiving or transmitting data through the network interface, and the one or more processors being configured to read program instructions stored in the memory to perform the method as described in any one of claims 1 to 18.

38. A communication device, characterized in that, include: A processor, a network interface card (NIC), and a NIC driver, wherein the NIC driver is used to manage the hardware of the NIC, and the processor is used to coordinate with the NIC to perform the method as described in any one of claims 1 to 18.

39. A communication system, characterized in that, It includes: a sending end and a receiving end, wherein the sending end is configured to perform the operations performed by the sending end in the method of any one of claims 1 to 18, and / or, the receiving end is configured to perform the operations performed by the receiving end in the method of any one of claims 1 to 18.

40. A computer program product containing instructions, characterized in that, When the instructions are executed by a processor, the processor or a device including the processor performs the method as described in any one of claims 1 to 18.

41. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a processor, cause the processor or a device including the processor to perform the method as described in any one of claims 1 to 18.

Images