A RDMA cross-host interconnection communication system based on PCIe NTB
Through the PCIe NTB system and NTRDMA software protocol, RDMA cross-host interconnection communication is achieved using PCIe Switch chip, solving the problems of high latency and high hardware costs in traditional technologies, and achieving low latency and high bandwidth data transmission, suitable for distributed AI Model training and other scenarios.
Patent Information
- Application Number
- CN202411340343.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-25
AI Technical Summary
Traditional remote memory access technologies such as TCP/IP have problems with high latency and heavy CPU load when processing large amounts of data. The existing RDMA technology has high hardware costs or occupies a lot of memory resources, which cannot meet the needs of low latency and high bandwidth, especially In scenarios such as distributed AI model training.
Using PCIe NTB system solution, PCIe Switch chip realizes RDMA cross-host interconnection communication, connects two NTB ports through Crosslink lines, combines NTRDMA software protocol and software simulation implementation to achieve low latency and high bandwidth interconnection communication, saving Demands to traditional network cards and RDMA network cards.
It realizes low latency and high bandwidth data transmission between two hosts without the need for traditional network cards and RDMA network cards, saving hardware costs and is suitable for short-distance interconnected communication systems.
Smart Images

Figure CN118860952B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a PCIe NTB-based RDMA cross-host interconnection communication system, belonging to the technical field of PCIe Switch interconnection. Background Art
[0002] With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, higher requirements are placed on the data processing capabilities and memory access efficiency of computer systems. Traditional remote memory access technologies, such as TCP / IP (Transmission Control Protocol / Internet Protocol), have problems such as high latency and heavy CPU load when processing large amounts of data. In order to solve these problems, RDMA technology came into being. It allows one host or server to directly access the memory of another host or server without the intervention of the operating system and CPU of both parties. RDMA reduces the latency of network transmission and increases the speed of data transmission, making it widely used in a variety of scenarios, such as data centers, high-performance computing centers (HPC), distributed storage, AI training, etc.
[0003] There are generally two network communication solutions: traditional Ethernet based on TCP / IP protocol stack and RDMA communication network.
[0004] In traditional Ethernet, data usually needs to pass through the protocol stack and be copied multiple times between the operating systems of the sender and receiver, resulting in large data delays and high CPU usage.
[0005] The implementation of the RDMA communication network depends on specific hardware and software support. At the hardware level, RDMA technology can be implemented through technologies such as InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP (Internet Wide Area RDMA Protocol). InfiniBand is a network designed specifically for RDMA and requires dedicated equipment. RoCE and iWARP are both Ethernet-based RDMA technologies.
[0006] Problems and shortcomings of existing technologies:
[0007] Traditional Ethernet solutions have large data delays and cannot be used in scenarios that require low latency, such as distributed AI model training.
[0008] InfiniBand is a network designed specifically for RDMA, and its hardware cost is high. RoCE implements RDMA based on Ethernet technology UDP / IP. iWARP implements RDMA based on TCP / IP, which is suitable for large networks but may occupy more memory resources.
[0009] Based on this, the present invention is proposed. Summary of the invention
[0010] The formation process of the present invention:
[0011] The upper-layer software uses the LibIbVerbs library of RDMA, and the underlying hardware uses the PCIe NTB system solution. There is no similar mature system solution on the market, and no related applied patents have been retrieved.
[0012] If this solution is used on two hosts or servers that both use PCIe Switch chips (a type of switching chip, referred to as PCIe Switch), two network cards can be saved. Only one port of the PCIe Switch chip needs to be configured in NTB mode.
[0013] To use it on two hosts or servers that do not use PCIe Switch chips, you only need to insert a PCIe expansion card with a PCIe Switch chip into each of the two hosts and configure one port of the PCIe Switch chip to NTB mode.
[0014] Two hosts or servers can connect two NTB Ports by using a Crosslink cable. The upper-layer application software does not need to be modified. As long as the NTRDMA related software (including NTRDMA user-mode Vendor driver, NTRDMA kernel-mode Vendor driver, NTB device management software, etc.) is integrated, low-latency, high-bandwidth interconnection communication can be achieved.
[0015] The specific technical solutions of the present invention are as follows:
[0016] A PCIe NTB-based RDMA cross-host interconnection communication system includes two PCIe Switch chips respectively connected to hosts, the two PCIe Switch chips are connected via a Crosslink line, the connection port of the PCIe Switch chip is set to NTB mode, and the two hosts achieve interconnection communication through the interconnection of the NTB.
[0017] As a further improvement, the PCIe NTB-based RDMA cross-host interconnection communication system further includes a specific implementation of an NTB-based RDMA software protocol component and a software simulation implementation of an NTB-based RDMA hardware data transmission;
[0018] The specific implementation of the NTB-based RDMA software protocol components, including the simulation implementation of the PD, QP, CQ, MR and MW components in RMDA based on PCIe NTB;
[0019] The software simulation implementation of the NTB-based RDMA hardware data transmission realizes the authorization verification of RMDA Write, RDMA Read, RDMA Send / Receive operation behaviors, local keys and remote keys under two different connection modes: reliable connection and unreliable connection.
[0020] For further improvement, the software simulation implementation of the NTB-based RDMA hardware data transmission requires that the kernel-state Vendor driver simulate the RDMA hardware implementation through software, the hosts are interconnected through PCIe NT Crosslink, the NTB mechanism is used to implement cross-host memory access, and then the DMA inside the PCIe Switch chip is used to move data.
[0021] Further improvements to implement RMDA Write in a reliable connection include the following steps:
[0022] 1.1. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP;
[0023] 1.2. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command;
[0024] 1.3. The RDMA simulator of the host on the requesting end verifies whether the address and remote key are legal. If they are legal, the data content of the source data cache of the host on the requesting end is written to the destination data cache of the host on the receiving end through DMA.
[0025] 1.4. After DMA write is successful, the CPU of the host on the requesting end sends the RDMA message packet to the RDMA receiving buffer of the host on the receiving end, and sends Doorbell to notify the host on the receiving end;
[0026] 1.5. The RDMA simulator of the host at the receiving end reads the message packet in the receiving buffer, parses it and generates an RDMA response message packet. The CPU of the host at the receiving end writes the response message packet directly to the RDMA receiving buffer of the host at the requesting end.
[0027] 1.6. The RDMA simulation of the host on the requesting end parses the RDMA response message packet, generates a CQE, and writes the CQE packet to the CQbuffer;
[0028] 1.7. The APP of the requesting host obtains task completion information by polling CQE.
[0029] Further improvements to implement RMDA Read in a reliable connection mode include the following steps:
[0030] 2.1. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP;
[0031] 2.2. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command;
[0032] 2.3. The RDMA simulator of the host on the requesting end forms an RDMA read request packet according to WQE, and writes the request packet to the RDMA receiving buffer of the host on the receiving end through DMA. After the write is successful, Doorbell notifies the host on the receiving end.
[0033] 2.4. The RDMA simulator of the host on the requesting end parses the RDMA message packet and checks the remote key;
[0034] 2.5. If the address and remote key verification are legal, the data content of the source data cache of the host at the receiving end is written to the destination data cache of the host at the requesting end through DMA;
[0035] 2.6. The RDMA simulator of the receiving host writes the RDMA response message packet to the RDMA receiving buffer of the requesting host through the CPU, and notifies the requesting host through the doorbell.
[0036] 2.7. The RDMA simulator of the host on the requesting end generates a CQE according to the response message packet and writes the CQE packet to the CQ buffer;
[0037] 2.8. The APP of the requesting host obtains task completion information by polling CQE.
[0038] For further improvement, in the case of reliable connection, an RDMA send&receive operation process includes the following steps:
[0039] 3.1. The user-mode Vendor driver of the receiving host converts the APP command information into WQE and writes it to the RQ of QP;
[0040] 3.2. Write the WQE of RQ to the RQ shadow space of the receiving end through the CPU; the host of the receiving end notifies the RDMA simulator that the WQE command needs to be processed. The data content of the RQ Shadow space of the receiving end is the shadow of the local RQ space data content;
[0041] 3.3. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP;
[0042] 3.4. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command;
[0043] 3.5. The RDMA simulator of the host on the requesting end checks the R-KEY and address permissions. If the check passes, the data is moved to the destination data cache of the host on the receiving end through DMA;
[0044] 3.6. The RDMA simulator of the host on the requesting end generates an RDMA message packet, writes it to the RDMA receiving buffer of the host on the receiving end, and notifies the host on the receiving end through Doorbell.
[0045] 3.7. The RDMA simulator of the host at the receiving end generates a completion packet CQE and writes it to the CQ;
[0046] 3.8. The APP of the receiving host obtains completion information by polling CQE;
[0047] 3.9. The RDMA simulator of the host on the receiving end generates an RMDA response message packet, writes it to the receiving buffer of the host on the requesting end through the CPU, and notifies the host on the requesting end through the Doorbell.
[0048] 3.10. The RDMA simulator of the host on the requesting end generates a CQE according to the RDMA response message packet and writes the CQE packet to the CQ ring buffer;
[0049] 3.11. The APP of the requesting host obtains task completion information by polling CQE.
[0050] Beneficial effects of the present invention:
[0051] The present invention does not require a traditional Ethernet card or an RDMA network card, but uses two PCIe Switch chips to implement a short-distance interconnection communication system between two hosts, and achieves the good characteristics of low latency and high bandwidth. By integrating the user-state Vendor driver and kernel-state Vendor driver of NTRDMA, the RDMA application program can directly perform low-latency and high-bandwidth data transmission between the two hosts without any modification. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a hardware connection diagram of the RDMA cross-host interconnection communication system based on PCIe NTB of the present invention;
[0053] Figure 2 A diagram showing the difference between the NTRDMA software stack of the present invention and the traditional RDMA software stack;
[0054] Figure 3 It is the relationship diagram of PD, MR and QP;
[0055] Figure 4 It is the relationship diagram of QP, CQ and WQE;
[0056] Figure 5 This is a schematic diagram of the process of an RMDA Write operation;
[0057] Figure 6 This is a schematic diagram of the process of an RMDA Read operation;
[0058] Figure 7 The figure is a schematic diagram of the process of an RDMA send&receive operation. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0060] Definitions of Abbreviations and Key Terms
[0061] PCIe: Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard, mainly used to expand computer system bus data throughput and improve device communication speed;
[0062] RMDA: Remote Direct Memory Access, remote direct address access;
[0063] NTB: Non-Transparent Bridging, a non-transparent bridge is a way to solve the address conflict and data packet routing problems between systems through address conversion and data packet translation;
[0064] HOST: A host computer is usually defined as a large time-sharing computer system in a centralized computer model. It is also the abbreviation of mainframe;
[0065] RC: Root Complex, RC devices are used to connect CPU / memory subsystems and I / O devices;
[0066] GPU: Graphics Processing Unit, a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.);
[0067] CPU: Central Processing Unit, as the computing and control core of the computer system, is the final execution unit for information processing and program running;
[0068] Crosslink: A feature defined in the PCIe specification that supports the upstream port of a switch to be connected to the upstream port of another switch, and the downstream port to be connected to the downstream port of another switch;
[0069] HCA: Host Channel Adapter, generally the RDMA network card is called HCA;
[0070] PD: Protection Domain, which allows applications to configure which QPs can access which MRs;
[0071] QP: Queue Pair, which is a combination of SQ (Send queue) and RQ (Receive queue);
[0072] RQ: Receive Queue, RDMA receive queue;
[0073] SQ: Send Queue, RDMA send queue;
[0074] CQ: Completion Queue, RDMA completion queue;
[0075] MR: Memory Region, memory area, generally refers to the data space of the user APP;
[0076] CQE: Completion Queue Element, completion queue element, each data structure unit in CQ;
[0077] WQE: Work Queue Element, each data structure unit in RQ / SQ / CQ;
[0078] MW: Memory Window, generally refers to the memory address space range of the receiving host that the local host can access;
[0079] LibIbVerbs: A directory name of the RDMA protocol in user-mode code. This directory can compile the libibverbs.so library and provide APIs to upper-layer applications.
[0080] Port: There are many physical ports on the PCIe interactive chip, which are used to connect to the host or PCIe device;
[0081] NTRDMA: Non-Transparent Remote Direct Memory Access, non-transparent bridge remote memory direct access. Generally refers to accessing the address space of the receiving end through NTB;
[0082] PCIe NT Crosslink: An interconnect technology located on the NT port of PCIe;
[0083] DMA: Direct Memory Access, a function provided by the computer bus architecture that does not require the participation of the CPU. It allows data to be transferred between devices and memory.
[0084] Doorbell: The doorbell mechanism is an efficient asynchronous notification mechanism. Generally, after the CPU writes a large amount of data to a specific place, it initiates an interrupt to notify another CPU;
[0085] R-KEY: Remote Key, remote key;
[0086] IOCTL: input / output control, is a system call dedicated to device input and output operations;
[0087] RDMA write: write operation command of the RDMA protocol;
[0088] RDMA read: RDMA protocol read operation command; generally one host sends a write and the other sends a read;
[0089] RDMA send: send operation command of RDMA protocol;
[0090] RDMA receive: The receive operation command of the RDMA protocol; generally, one host sends a receive and the other host sends a send;
[0091] Vendor User Driver: The vendor Linux user mode driver of the RDMA network card;
[0092] Vendor Kernel Driver: The vendor Linux kernel driver of the RDMA network card;
[0093] NTRDMA User Driver: Linux user-mode driver for emulating RDMA network cards over PCIe NTB interconnects;
[0094] NTRDMA Kernel Driver: Linux kernel driver for emulating RDMA network cards over PCIe NTB interconnects;
[0095] / dev / infiniband / uverb$X: device node of the RDMA network card; user-mode applications can control the RDMA network card by accessing this device node, which is generally provided by the kernel driver.
[0096] Kernel space application: an application running in the kernel space of the Linux system;
[0097] IB: Abbreviation for InfiniBand, wireless bandwidth technology;
[0098] IB uverbs: InfiniBand uverbs, the operation commands of the IB protocol in the InfiniBand driver in the Linux system;
[0099] IB Core: InfiniBand Core, the core directory code in the InfiniBand driver in the Linux system, and the specific implementation of the IB protocol. Example 1
[0100] The hardware connection scheme of the RDMA cross-host interconnection communication system based on PCIe NTB of the present invention is shown in Figure 1 , Host 0 is connected to PCIe Switch 0, Host 1 is connected to PCIe Switch 1, the two PCIe Switch chips are connected via a Crosslink line, and the connection port of the PCIe Switch is set to NTB mode, so that the two hosts (Host 0 and Host 1) can achieve short-distance interconnection communication through NTB interconnection. Compared with the traditional RDMA interconnection solution, it no longer requires a network card, but uses a PCIe Switch chip.
[0101] exist Figure 1 In the above table, EP stands for End Point, which is a PCIe terminal device.
[0102] PP: Virtual PCI-PCI Bridge, refers to the PCI to PCI bridge of PCIe Switch. Generally, data can be transmitted through the connection between PPs inside the Switch. The host is connected to a PP of the Switch. By switching the connectivity of different PPs inside the Switch, the host can access different EPs connected to the Switch. Example 2
[0103] The PCIe NTB-based RDMA inter-host interconnection communication system of the present invention further includes a specific implementation of an NTB-based RDMA software protocol component and a software simulation implementation of an NTB-based RDMA hardware data transmission.
[0104] The specific implementation of the NTB-based RDMA software protocol component includes the simulation implementation of the PD, QP, CQ, MR and MW components in RMDA based on PCIe NTB.
[0105] The software simulation implementation of the NTB-based RDMA hardware data transmission realizes RMDA Write, RDMA Read, RDMA Send / Receive operation behaviors, and authorization verification of local keys and remote keys under two different connection modes: reliable connection and unreliable connection.
[0106] The specific implementation of the RDMA software protocol component based on NTB is equivalent to implementing the NTRDMA software stack based on PCIe. For the difference between the NTRDMA software stack of the present invention and the traditional RDMA software stack, please refer to Figure 2 .like Figure 2 In the RDMA software stack shown on the left, user space applications all call the interface of the Libibverbs library; Libibverbs controls the IB protocol (IB uerbs and IB Core) by accessing the device node of the RDMA network card provided by the kernel driver ( / dev / infiniband / uverb$X), and IB Core calls the Vendor Kernel Driver to control the RDMA hardware. For different RDMA hardware, two drivers need to be implemented (Vendor User Driver and Vendor Kernel Driver). To implement the NTRDMA software stack based on PCIe, it is necessary to ensure that the applications in the RDMA ecosystem are not modified in any way, so that the hardware connection solution of the present invention can be used; therefore, the present invention also needs to implement the corresponding two drivers ( Figure 2 On the right side of the diagram: kernel mode vendor driver (NTRDMA Kernel Driver) and user mode vendor driver (NTRDMA User Driver). NTB device management software is used to manage and adapt different PCIeSwitch chips.
[0107] Therefore, the specific implementation of the NTB-based RDMA software protocol component, that is, the PD, QP, CQ, MR and MW in the RMDA need to be implemented in the user-mode Vendor driver.
[0108] The software simulation implementation of the NTB-based RDMA hardware data transmission requires the kernel-state Vendor driver to simulate the RDMA hardware implementation through software because there is no HCA hardware. The hosts are interconnected through PCIe NT Crosslink, and the NTB mechanism is used to implement cross-host memory access. Then, the DMA inside the PCIe Switch chip is used to move data, thereby bypassing the kernel and CPU. Example 3
[0109] Specific implementation of RDMA software protocol components based on NTB:
[0110] It mainly includes the simulation implementation of PD, QP, CQ, MR and other components in RMDA based on PCIe NTB.
[0111] The relationship diagram of PD, MR and QP, refer to Figure 3 , QP 2 of node A is connected to QP 0 of node C, then node A can access MR 0 in PD 0 of node C through QP, and node C can also access MR 2 and MR 3 in PD 0 where QP 2 of node A is located through QP. However, node A cannot access MR 1 in PD 1 of node C through QP.
[0112] If QP 4 of node B is connected to QP 1 of node C, node B can access MR 1 in PD 1 of node C through QP, and node C can also access MR 3 and MR 4 in PD 0 where QP 4 of node B is located through QP. However, node B cannot access MR 1 in PD 1 of node C through QP. Node C also cannot access MR 5 in PD 1 of node B through QP.
[0113] For the relationship diagram of QP, CQ and WQE, refer to Figure 4 RQ, SQ and CQ are all circular queues, and the processing order is first-in-first-out. Each unit stored in CQ is called CQE, and each unit stored in RQ / SQ is called WQE.
[0114] For example, the SQ in QP 0 has WQE 0 and WEQ 1. The information that needs to be included in WQE 1 is: the starting address of the data to be transmitted by the local host in the memory, the length of the data, and the memory address expected to be written to the receiving host. After the NTRDMA module successfully transmits the data to the receiving host, it needs to generate a CQE (such as CQE 1) and write it to the CQ 0 circular queue buffer. The information that needs to be included in CQE 1 is: which SQ (in this case, the SQ of QP 0) and which WQE (in this case, WQE 1). Example 4
[0115] Software simulation implementation of RDMA hardware data transmission based on NTB:
[0116] It mainly implements RMDA Write, RDMA Read, RDMA Send / Receive and other operations in two different connection modes: reliable connection and unreliable connection, as well as authorization verification of local and remote keys. The main difference between the reliable connection and unreliable connection processes is whether to return a response packet. Example 5
[0117] 1. Implementing RMDA Write in a reliable connection mode mainly includes the following steps, refer to Figure 5 :
[0118] 1.1. The user-mode Vendor driver of host 0 on the requesting side converts the APP command information into WQE and writes it to the SQ of QP.
[0119] 1.2. Host 0 on the requesting side notifies the RDMA simulator that it needs to process the WQE command.
[0120] 1.3. The RDMA simulator of host 0 on the requesting end verifies whether the address and remote key are legal. If they are legal, the data content of the source data cache of host 0 on the requesting end is written to the destination data cache of host 1 on the receiving end through DMA.
[0121] 1.4. After DMA write succeeds, the CPU of the host on the requesting end sends the RDMA message packet (type is data write successful) to the RDMA receive buffer of host 1 on the receiving end, and sends Doorbell to notify host 1 on the receiving end.
[0122] 1.5. The RDMA simulator of host 1 on the receiving end reads the message packet in the receiving buffer, parses it and generates an RDMA response message packet. The CPU of host 1 on the receiving end writes the response message packet directly to the RDMA receiving buffer of host 0 on the requesting end.
[0123] 1.6. The RDMA simulation of host 0 on the requesting side parses the RDMA response message packet, generates a CQE, and writes the CQE packet to the CQbuffer.
[0124] 1.7. The APP of host 0 on the requesting side obtains task completion information by polling CQE. Example 6
[0125] The only difference from Embodiment 5 is that the step of implementing RMDA Write in an unreliable connection mode does not include step 1.5 in Embodiment 5. Example 7
[0126] 2. Implementing RMDA Read in a reliable connection mode mainly includes the following steps, refer to Figure 6 :
[0127] 2.1. The user-mode Vendor driver of host 0 on the requesting side converts the APP command information into WQE and writes it to the SQ of QP.
[0128] 2.2. The requesting host 0 notifies (IOCTL call or Doorbell mechanism) that the RDMA simulator needs to process the WQE command.
[0129] 2.3. The RDMA simulator of host 0 on the requesting side forms an RDMA read request packet (including the receiving end address, R-KEY and the MR physical address and length list of host 1) according to WQE, and writes the request packet to the RDMA receiving buffer of host 1 on the receiving end through DMA. After the write is successful, Doorbell notifies host 1 on the receiving end.
[0130] 2.4. The RDMA simulator of host 0 on the requesting side parses the RDMA message packet and checks the remote key.
[0131] 2.5. If the address and remote key verification are legal, the data content of the source data cache of host 1 on the receiving end is written to the destination data cache of host 0 on the requesting end through DMA.
[0132] 2.6. The RDMA simulator of host 1 on the receiving end writes the RDMA response message packet to the RDMA receive buffer of host 0 on the requesting end through the CPU, and notifies host 0 on the requesting end through doorbell.
[0133] 2.7. The RDMA simulator of host 0 on the requesting side generates a CQE according to the response message packet and writes the CQE packet to the CQ buffer.
[0134] 2.8. The APP of host 0 on the requesting side obtains task completion information by polling CQE. Example 8
[0135] The only difference from Example 7 is that the step of implementing RMDA Read in an unreliable connection mode does not include step 2.6 in Example 7. Example 9
[0136] 3. In the case of reliable connection, an RDMA send&receive operation process includes the following steps, refer to Figure 7 :
[0137] 3.1. The user-mode Vendor driver of host 1 on the receiving end converts the APP command information into WQE and writes it to the RQ of QP.
[0138] 3.2. Write the WQE of RQ to the RQ shadow space of the receiving end through the CPU; the host 1 of the receiving end notifies (IOCTL call or Doorbell mechanism) that the RDMA simulator needs to process the WQE command. The data content of the RQ Shadow space of the receiving end is the shadow of the data content of the local RQ space, and the content is the same.
[0139] 3.3. The user-mode Vendor driver of host 0 on the requesting side converts the APP command information into WQE and writes it to the SQ of QP.
[0140] 3.4. The requesting host 0 notifies (IOCTL call or Doorbell mechanism) that the RDMA simulator needs to process the WQE command.
[0141] 3.5. The RDMA simulator of host 0 on the requesting side checks the R-KEY and address permissions. If the check passes, the data is moved to the destination data cache of host 1 on the receiving side through DMA.
[0142] 3.6. The RDMA simulator of host 0 on the requesting side generates an RDMA message packet, writes it to the RDMA receiving buffer of host 1 on the receiving side, and notifies host 1 on the receiving side via Doorbell.
[0143] 3.7. The RDMA simulator of host 1 on the receiving end generates a completion packet CQE and writes it to CQ.
[0144] 3.8. The APP of host 1 on the receiving end obtains completion information by polling CQE.
[0145] 3.9. The RDMA simulator of host 1 on the receiving end generates an RMDA response message packet, writes it to the receiving buffer of host 0 on the requesting end through the CPU, and notifies host 0 on the requesting end through the Doorbell.
[0146] 3.10. The RDMA simulator of host 0 on the requesting side generates a CQE based on the RDMA response message packet and writes the CQE packet to the CQ ring buffer (multiple CQEs are stored in the CQ).
[0147] 3.11. The APP of host 0 on the requesting side obtains task completion information by polling CQE. Example 10
[0148] The only difference from Example 9 is that, in the case of an unreliable connection, the steps of an RDMA send&receive operation process do not include step 3.9 in Example 9.
[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A PCIe NTB-based RDMA cross-host interconnection communication system, characterized by: It includes two PCIe Switch chips respectively connected to the host, the two PCIe Switch chips are connected via a Crosslink line, the connection port of the PCIeSwitch chip is set to NTB mode, and the two hosts realize interconnection and communication through the interconnection of the NTB; It also includes the specific implementation of the RDMA software protocol component based on NTB and the software simulation implementation of RDMA hardware data transmission based on NTB; The specific implementation of the NTB-based RDMA software protocol components, including the simulation implementation of the PD, QP, CQ, MR and MW components in RMDA based on PCIe NTB; The software simulation implementation of the RDMA hardware data transmission based on NTB realizes the authorization verification of RMDA Write, RDMA Read, RDMA Send / Receive operation behaviors, local keys and remote keys under two different connection modes: reliable connection and unreliable connection; The software simulation implementation of the NTB-based RDMA hardware data transmission requires that the kernel-state Vendor driver simulates the implementation of the RDMA hardware through software, the hosts are interconnected through the PCIe NT Crosslink, the NTB mechanism is used to implement cross-host memory access, and then the DMA inside the PCIe Switch chip is used to move data.
2. According to the PCIe NTB-based RDMA inter-host interconnection communication system of claim 1, it is characterized in that: Implementing RMDA Write in a reliable connection mode includes the following steps: 1.
1. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP; 1.
2. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command; 1.
3. The RDMA simulator of the host on the requesting end verifies whether the address and remote key are legal. If they are legal, the data content of the source data cache of the host on the requesting end is written to the destination data cache of the host on the receiving end through DMA. 1.
4. After DMA write is successful, the CPU of the host on the requesting end sends the RDMA message packet to the RDMA receiving buffer of the host on the receiving end, and sends Doorbell to notify the host on the receiving end; 1.
5. The RDMA simulator of the host at the receiving end reads the message packet in the receiving buffer, parses it and generates an RDMA response message packet. The CPU of the host at the receiving end writes the response message packet directly to the RDMA receiving buffer of the host at the requesting end. 1.
6. The RDMA simulation of the host on the requesting end parses the RDMA response message packet, generates a CQE, and writes the CQE packet to the CQbuffer; 1.
7. The APP of the requesting host obtains task completion information by polling CQE.
3. According to the PCIe NTB-based RDMA inter-host interconnection communication system of claim 1, it is characterized in that: Implementing RMDA Read in a reliable connection mode includes the following steps: 2.
1. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP; 2.
2. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command; 2.
3. The RDMA simulator of the host on the requesting end forms an RDMA read request packet according to WQE, and writes the request packet to the RDMA receiving buffer of the host on the receiving end through DMA; After the write is successful, Doorbell notifies the host at the receiving end; 2.
4. The RDMA simulator of the host on the requesting end parses the RDMA message packet and checks the remote key; 2.
5. If the address and remote key verification are legal, the data content of the source data cache of the host at the receiving end is written to the destination data cache of the host at the requesting end through DMA; 2.
6. The RDMA simulator of the receiving host writes the RDMA response message packet to the RDMA receiving buffer of the requesting host through the CPU, and notifies the requesting host through the doorbell. 2.
7. The RDMA simulator of the host on the requesting end generates a CQE according to the response message packet and writes the CQE packet to the CQ buffer; 2.
8. The APP of the requesting host obtains task completion information by polling CQE.
4. The RDMA inter-host interconnection communication system based on PCIe NTB according to claim 1, characterized in that: In the case of a reliable connection, an RDMA send&receive operation process includes the following steps: 3.
1. The user-mode Vendor driver of the receiving host converts the APP command information into WQE and writes it to the RQ of QP; 3.
2. Write the WQE of RQ to the RQ shadow space of the receiving end through the CPU; the host of the receiving end notifies the RDMA simulator that the WQE command needs to be processed. The data content of the RQ Shadow space of the receiving end is the shadow of the data content of the local RQ space; 3.
3. The user-mode Vendor driver of the requesting host converts the APP command information into WQE and writes it to the SQ of QP; 3.
4. The host on the requesting side notifies the RDMA simulator that it needs to process the WQE command; 3.
5. The RDMA simulator of the host on the requesting end checks the R-KEY and address permissions. If the check passes, the data is moved to the destination data cache of the host on the receiving end through DMA; 3.
6. The RDMA simulator of the host on the requesting end generates an RDMA message packet, writes it to the RDMA receiving buffer of the host on the receiving end, and notifies the host on the receiving end through Doorbell. 3.
7. The RDMA simulator of the host at the receiving end generates a completion packet CQE and writes it to the CQ; 3.
8. The APP of the receiving host obtains completion information by polling CQE; 3.
9. The RDMA simulator of the host on the receiving end generates an RMDA response message packet, writes it to the receiving buffer of the host on the requesting end through the CPU, and notifies the host on the requesting end through the Doorbell. 3.
10. The RDMA simulator of the host on the requesting end generates a CQE according to the RDMA response message packet and writes the CQE packet to the CQ ring buffer; 3.
11. The APP of the requesting host obtains task completion information by polling CQE.
