A PCIe-based RDMA connection management method

Abstract

The present invention belongs to the field of PCIe chip communication connection, and relates to a PCIe-based RDMA connection management method, which uses two hosts to interconnect through NTB devices to build an NT RDMA network; the hosts respectively build their own net devices and IB devices based on their respective NT EPs; the IB transport layer of the NT RDMA network is extended with an IP header or an IB GRH header as the basic communication information of the UD type QP; the NT RDMA network also needs to extend the IP header, and DETH_NETWORK related information is added by extending DETH; and the connection management of the NT RDMA network is built through the CM of RoCE. The present invention can enable the NT RDMA network card to support the UD type QP and the connection management function without changing the existing IB, MAD, and CM software stacks.

Description

Technical Field

[0001] The invention relates to a PCIe-based RDMA connection management method, belonging to the technical field of PCIe chip communication connection. Background Art

[0002] With the rapid development of cloud computing, big data, and artificial intelligence, RDMA technology has also been widely used, and more and more applications are enjoying the high throughput and low latency capabilities brought by RDMA technology.

[0003] As a new technology, NTB-based RDMA technology can realize RDMA communication only through PCIe, and can provide lower latency and higher bandwidth compared to traditional RoCE / InfiniBand.

[0004] As a tool for building new RDMA connections in RDMA networks, Connection Manager (CM) provides a set of standard CM APIs to applications, which can reduce user development work and system complexity. Figure 1 As shown in the figure, under the IB transport layer, all functions under the IB transport layer are implemented through PCIe), and as a result, there is a demand for NT RDMA connection management.

[0005] The location of the system software stack where CM is located is as follows Figure 1 As shown in the figure: CM, as an application of MAD component, sends and receives connection-related messages through MAD module, and MAD is built on the IB transport layer, and sends and receives through the IB transport layer: When the user needs to create a new QP, call CM API, and the CM components at both ends of the communication will establish a new communication QP through a series of information exchanges. When the user needs to release QP, call CM to notify the other end (only RC QP) to complete the release of QP.

[0006] Currently, RoCE RDMA network cards generally support CM through Ethernet network cards, which requires adding network cards to the system. Therefore, an Ethernet network card is generally extended to the RoCE RDMA nic, so that RDMA + CM + Ethernet network card functions can be completed with one card.

[0007] When the protocol below the IB transport layer, such as Ethernet / IP, becomes PCIe, that is, RDMA over PCIe (NTBRDMA), the IB transport layer is built on the PCIe layer, such as Figure 1 As shown in the figure, since the two ends of the communication are generally directly connected point-to-point, there is no need for routing and forwarding information, and no need for Ethernet / IP and other network layers.

[0008] RoCE RDMA devices generally rely on Ethernet net devices to build GID information, rely on the routing information of the kernel network module to find RDMA devices, create AH information, etc. RoCE CM is based on MAD (using GSI type QP, QPN is 1, and unreliable datagrams), and also requires this information when initiating connection requests and processing requests.

[0009] pass Figure 1 It can be seen that only the IB transport layer is implemented based on NTB RDMA, and there are no fields such as IP / UDP header (RoCEv2) and IB GRH (RoCEv1). This will result in the inability to support UD type QP when using the existing IB software stack without making any changes (the system IB software stack needs to obtain the source information of the data through the IP header or IB GRH field).

[0010] Since UD-type QP cannot be supported, a new QP connection cannot be established directly through the existing RoCE CM processing flow (UD-type QP cannot be used): the sender cannot parse and find the corresponding RDMA network card and port through the host information passed in by the application; and the receiver cannot parse the corresponding RDMA network card and port through the received CM information, so the connection establishment process cannot be completed.

[0011] At the same time, since the IB software stack of Linux kernel NT RDMA lacks information about net device as the NT RDMA network card port, the problem that both ends cannot find the NT RDMA device through GID will occur.

[0012] Based on this, the present invention is proposed. Summary of the invention

[0013] The present invention provides a PCIe-based RDMA connection management method, which can solve the problem that UD type QP is not supported in the NTB RDMA network, and can solve the problem that connection management is not supported in the NTB RDMA network. The specific technical solution is as follows:

[0014] In a first aspect, a PCIe-based RDMA connection management method comprises the following steps:

[0015] The two hosts are interconnected through the NTB device to build an NT RDMA network;

[0016] The hosts build their own net devices and IB devices based on their respective NT EPs;

[0017] The IP header or IB GRH header is extended for the IB transport layer of the NT RDMA network as the basic communication information of the UD type QP; the NT RDMA network also needs to extend the IP header and add DETH_NETWORK related information by extending DETH;

[0018] Build connection management for NT RDMA network through RoCE CM.

[0019] As a further improvement, the NT EP driver on the host side registers two devices with the kernel, namely netdevice and IB device; when registering the IB device with the kernel, it is registered as an RDMA device of RoCE v2 type; when registering, the current NT net device is specified as the net device of this RDMA network card;

[0020] When the requesting QP of UD type wants to send data to the responding QP, the request and AH related information are filled in WR, and then WR is added to SQ. The NT RDMA hardware or simulation software takes out the SQE from SQ, which is a WR, and encapsulates the IB header according to the passed parameters. If the QP type is UD, the DETH header and the DETH_NETWORK structure of DETH are extended on the basis of BTH, and then BTH is filled according to the RQP information in WR, and DETH is filled according to the SQP and RKEY information. The information corresponding to AH is filled in DETH_NETWORK, and then the data description information of WR is continued to be used to fill the payload. After completing the information filling, the entire IB message is sent out through the PCIe link.

[0021] When the responding QP receives the entire IB message through the PCIe link, it checks the message whose Opcode is UD type QP. In addition to checking the correctness of the BTH header and the DETH header information, it also parses the DETH_NETWORK information and copies the parsed result together with the payload to RQE, and uses the information in BTH / DETH / DETH_NETWORK to update the WC in CQE, and then triggers the software to poll CQ to complete the data reception.

[0022] First, CM is an application of MAD. MAD uses GSI type QP in kernel state, and its QPN is 1. The IB device is registered as a RoCEv2 device and supports CM. When a new QP is built using RoCE CM, the process of sending and receiving messages using MAD's QP is as follows:

[0023] When APP1 initiates a connection request, it accesses the kernel CM module through the API provided by RDMA;

[0024] The kernel CM module encapsulates the request into the CM protocol format and then passes the data to the MAD module;

[0025] The MAD module encapsulates the CM protocol encapsulation request into the MAD protocol structure, and then encapsulates it into WR based on the information of this structure and adds it to the SQ;

[0026] The SQ processes this WR and finally sends it to the peer end through the PCIe link;

[0027] After receiving the connection request, the receiver parses the data and then passes the data to the MAD for processing based on the QPN.

[0028] MAD parses the data, and the results of the analysis are further processed by the kernel CM module;

[0029] After receiving the data, the kernel CM module parses and notifies APP2 of a new connection request;

[0030] After receiving the new connection request, APP2 will respond to the new connection request and confirm or reject the connection in the opposite direction of the sender.

[0031] Beneficial effects of the present invention:

[0032] The present invention can enable the NT RDMA network card to support UD type QP and connection management function without changing the existing IB, MAD and CM software stacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a comparison chart between NT RDMA and IB / RoCE protocol layers;

[0034] Figure 2 GID functionality is provided by extending the NT net device on the NT RDMA device (left) or by building a netdevice in the driver software (right);

[0035] Figure 3 Extend the DETH_NETWORK field on DETH for UD type QP in NT RDMA;

[0036] Figure 4 This is the NT RDMA network interconnection diagram;

[0037] Figure 5 To fill the WR information of SQE into the logical diagram of DETH and DETH_NETWORK;

[0038] Figure 6 Parse the elements of each queue from the encapsulated message for the receiving end;

[0039] Figure 7 This is the UD QP data flow diagram for NTB RDMA;

[0040] Figure 8 The process of sending and receiving messages of the QP using MAD when building a new QP using RoCE CM. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] Definitions of Abbreviations and Key Terms

[0043] AH: Address Handle, address handle, used to find address information, encapsulate IP protocol header, and is the transmission information required when UD type QP transmits messages;

[0044] API: Application Programing Interface, application programming interface;

[0045] APP: Application, here refers to the application that calls the libibverbs function library;

[0046] AV: Address Vector, used to store remote address information set;

[0047] BTH: Base Transport Header, the header field of the IB transport protocol;

[0048] Bypass kernel: bypass kernel, a way for applications to access hardware, which can reduce system overhead;

[0049] CM: Communication Manager, a module based on MAD for handling connections;

[0050] CQ: Completion Queue, completion queue;

[0051] CQE: Completion Queue Entry, completion queue element;

[0052] DETH: Datagram Extended Transport Header, an extension of the UD type IB header;

[0053] DETH_NETWORK: network-related information extended on the DETH header;

[0054] Driver: device driver;

[0055] Ethernet Link: Ethernet link layer;

[0056] Ethernet net device: Ethernet network device, the data structure registered by the Ethernet network device driver to the kernel;

[0057] Hardware: Hardware layer, which is managed by the driver and is used to process specific receipt requests;

[0058] IB: short for InfiniBand;

[0059] IB device: This refers specifically to the device data structure registered in the software stack;

[0060] IB Transport: IB transport protocol;

[0061] IB Network Layer: IB network layer;

[0062] IP / UDP: Internet Protocol and User Datagram Protocol, a protocol widely used in network communications, IP protocol at the network layer, UDP protocol at the transport layer;

[0063] IB Link: IB link layer, a link dedicated to the Infiniband network;

[0064] InfiniBand: A communication standard for high-performance computing, which includes protocols such as IB and RoCE;

[0065] GID: Global Identifier, used to uniquely identify a device interface;

[0066] GRH: Global Routing Header, which contains the remote and destination GID information;

[0067] GSI: General Service Interface. The MAD layer uses this service. This service uses UD type QP with QPN 1 and 0x80010000 as QKEY.

[0068] Linux: A widely used open source operating system;

[0069] MAD: Management Datagram, used to process management-related information;

[0070] Net device: A general term for network devices, which are usually registered to the operating system through a driver;

[0071] NIC: Network Interface Card, network interface card;

[0072] NTB: Non-Transparent Bridge, a non-transparent bridge is a way to solve the address conflict and data packet routing problems between systems through address conversion and data packet translation;

[0073] NT EP: Non-Transparent Endpoint, non-transparent bridge equipment;

[0074] NT net device: Non-Transparent EP net device, a network device based on NT EP;

[0075] Opcode: operation code, which is used by both ends of the communication to inform the other end of the instructions that need to be performed;

[0076] Payload: data load, generally refers to the data that the application actually wants to transmit;

[0077] PCIe: Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard, mainly used to expand computer system bus data throughput and improve device communication speed;

[0078] Port: refers to the interface of the registered RDMA network card. An RDMA network card may register multiple interfaces.

[0079] QP: Queue Pair, which is a combination of SQ, RQ and CQ;

[0080] QPN: Query Pair Number. Each QP in a node has a unique number, referred to as QPN.

[0081] RC QP: Reliable Connection QP, reliable connection QP;

[0082] RDMA: Remote Direct Memory Access, remote direct address access;

[0083] RDMA API: The API provided by the RDMA module to the user state, generally refers to the link library provided by libverbs;

[0084] RKEY: remote key, the key of the remote QP. This end needs to include this information of the other end when sending a request. Only after the other end passes the verification, the request of this end is allowed to be executed;

[0085] RoCE: RDMA over Converged Ethernet, RDMA based on Ethernet;

[0086] RoCEv1: RoCE version 1, based on Ethernet, forwarding is only achieved through the Ethernet link layer;

[0087] RoCEv2: RoCE version 2, based on Ethernet, encapsulated in UDP packets, and forwarded through IP;

[0088] RQ: Receive Queue, receiving queue;

[0089] RQE: Receive Queue Entry, receive queue element;

[0090] RQP: The remote QPN. When the local end initiates a request, it needs to know the remote end's QPN.

[0091] SQ: Send Queue, which stores work requests;

[0092] SQE: Send Queue Entry, which is a send queue element that stores each work request;

[0093] SQP: The QPN of the local end. When sending a request, the UD type QP needs to carry the QPN information of the local end;

[0094] UD QP: Unreliable Datagram, unreliable datagram type QP;

[0095] WC: Work Completion, completion event. After a request is completed or a specific request is received from the peer, the device hardware or driver adds a WC;

[0096] WR: Work Request, work request, application request, added to the SQE of SQ;

[0097] User: refers to the user space. Applications generally run in the user space and do not require privileges.

[0098] Kernel: refers to the kernel space. Applications enter the kernel space through system calls and can run privileged instructions. Drivers generally run in the kernel space.

[0099] libverbs: A library for RDMA programming. Example 1

[0100] A PCIe-based RDMA connection management method comprises the following steps:

[0101] The two hosts are interconnected through the NTB device to build an NT RDMA network.

[0102] Since the connection management needs the GID information of RDMA in the process of establishing the connection, the function of extending the netdevice can be realized by extending the NT netdevice function through NT EP or by extending the net device of a software through NT RDMA network (such as Figure 2 shown);

[0103] Then the host builds its own net device (using hardware to extend NT netdevice) and IB device based on its own NT EP, such as Figure 4 shown.

[0104] The IP header or IB GRH header is extended for the IB transport layer of the NT RDMA network as the basic communication information of the UD (GSI) type QP, so that it can use the UD type QP based on the existing RoCE software stack; in addition to implementing the IB transport layer, the NT RDMA network also needs to extend the IP header so that it can continue to use the existing IB software stack, such as Figure 3 As shown, DETH_NETWORK related information is added by extending DETH.

[0105] Since NT RDMA does not have CM function, the present invention uses RoCE's CM to build the connection management of the NT RDMA network: the connection management of the NT RDMA network is built based on the processing flow of RoCE's existing MAD and CM, so that the connection management function of NTB RDMA is realized without modifying the existing MAD and CM software stack.

[0106] The specific software and hardware stack is implemented as follows:

[0107] The NT EP driver on the host side will register two devices with the kernel: a net device and an IB device. When registering the IB device with the kernel, it needs to be registered as a RoCE v2 type RDMA device. When registering, you need to specify the current NT net device (through Figure 2 Either of the two creation methods in (a) as the net device of this RDMA network card.

[0108] When the UD type requester QP wants to send data to the responder QP, it will fill the request and AH related information in WR, and then add WR to SQ, so that the NT RDMA hardware or simulation software takes out the SQE from SQ, that is, a WR, and encapsulates the IB header according to the passed parameters; if the QP type is UD, it is necessary to expand the DETH header and the DETH_NETWORK structure of DETH on the basis of BTH, and then fill BTH according to the RQP information in WR, fill DETH according to the SQP and RKEY information, and fill the information corresponding to AH into DETH_NETWORK, such as Figure 5 As shown, the payload is then filled with the data description information of WR (not shown in the figure). After the information filling is completed, the entire IB message is sent out through the PCIe link.

[0109] When the responding QP receives the entire IB message through the PCIe link, it checks that the Opcode is a UD type QP message. In addition to checking the correctness of the BTH header and the DETH header information, it also parses the DETH_NETWORK information and copies the parsed result together with the payload to the RQE, and uses the information in BTH / DETH / DETH_NETWORK to update the WC in the CQE ( Figure 6 As shown), the software is then triggered to poll CQ to complete the data reception.

[0110] At this point, the sending and receiving of a UD type QP is completed, such as Figure 7 shown. Example 2

[0111] For CM, it depends on MAD. As an application of MAD, MAD uses GSI type QP in kernel state, and its QPN is 1. The rest can be considered to be not much different from the QP process in user state. Since the IB device has been registered as a RoCEv2 device and supports CM, the kernel's IB MAD and CM software stack can be used directly without any changes. When using RoCE CM to build a new QP, the QP sending and receiving process of MAD is as follows: Figure 8 As shown:

[0112] When APP1 initiates a connection request, it accesses the kernel CM module through the API provided by RDMA;

[0113] The kernel CM module encapsulates the request into the CM protocol format and then passes the data to the MAD;

[0114] MAD encapsulates the request for CM protocol encapsulation into the structure of MAD protocol, and then encapsulates it into WR according to the information of this structure and adds it to SQ.

[0115] SQ will follow Figure 7 The WR is processed in the same way as the WR in the process, and finally sent to the peer end through the PCIe link;

[0116] After receiving this connection request, the receiving end will follow Figure 7 The data is parsed in a way, and then the data is handed over to MAD for processing according to QPN;

[0117] MAD analyzes the data, and the analysis results are further processed by the CM module;

[0118] After receiving the data, the CM module parses and notifies APP2 of a new connection request;

[0119] The above is the process of the connection requester sending a request and the connection responder receiving and processing it. Here we only focus on the function of using MAD QP to send and receive data, and do not focus on the specific link establishment process (that is, we do not focus on the protocol message interaction between the connection requester and responder).

[0120] After receiving the connection request, APP2 will respond to the connection request and confirm or reject the connection in the opposite direction of the sender.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A PCIe-based RDMA connection management method, characterized in that: The following steps are involved: The two hosts are interconnected through the NTB device to build an NT RDMA network; The hosts build their own net devices and IB devices based on their respective NT EPs; The IP header or IB GRH header is extended for the IB transport layer of the NT RDMA network as the basic communication information of the UD type QP; the NTRDMA network also needs to extend the IP header and add DETH_NETWORK related information by extending DETH; Build connection management of NT RDMA network through RoCE CM; The NT EP driver on the host side registers two devices with the kernel, namely the net device and the IB device. When registering the IB device with the kernel, it is registered as an RDMA device of the RoCE v2 type. When registering, the current NT netdevice is specified as the net device of this RDMA network card. When the requesting QP of UD type wants to send data to the responding QP, the request and AH related information are filled in WR, and then WR is added to SQ. The NT RDMA hardware or simulation software takes out the SQE from SQ, which is a WR, and encapsulates the IB header according to the passed parameters. If the QP type is UD, the DETH header and the DETH_NETWORK structure of DETH are extended on the basis of BTH, and then BTH is filled according to the RQP information in WR, and DETH is filled according to the SQP and RKEY information. The information corresponding to AH is filled in DETH_NETWORK, and then the data description information of WR is continued to be used to fill the payload. After completing the information filling, the entire IB message is sent out through the PCIe link. When the responding QP receives the entire IB message through the PCIe link, it checks the message whose Opcode is UD type QP. In addition to checking the correctness of the BTH header and the DETH header information, it also parses the DETH_NETWORK information and copies the parsed result together with the payload to RQE, and uses the information in BTH / DETH / DETH_NETWORK to update the WC in CQE, and then triggers the software to poll CQ to complete the data reception.

Images