A GPU cross-host communication interconnection system based on PCIe NTB

Abstract

The present invention belongs to the field of PCIe Switch interconnection, and relates to a GPU cross-host communication interconnection system based on PCIe NTB, which includes two hosts, two GPUs, and two PCIe Switch chips; the two hosts are respectively connected to the PCIe Switch chips, and the two PCIe Switch chips are connected through a Crosslink line, and the connection port of the PCIe Switch chip is set to NTB mode, and the two GPUs realize interconnection communication through NTB. By opening a window in NTB, the address of the remote device is mapped in the local address space, so as to achieve the purpose of accessing the remote device. The present invention uses PCIe Switch to replace high-end network devices, which can significantly reduce hardware costs and reduce the complexity of system deployment. Communication between devices can be directly accessed through memory addresses, without the need to set up complex network connections, IP addresses or special software protocols; low-latency, high-bandwidth GPU cross-host interconnection communication is achieved. GPUs that communicate across hosts through PCIe NTB can achieve the effect of GPUDirect P2P.

Description

Technical Field

[0001] The invention relates to a GPU cross-host communication interconnection system based on PCIe NTB, belonging to the technical field of PCIe Switch interconnection. Background Art

[0002] With the rapid development of cloud computing, big data and artificial intelligence technologies, the global demand for high-performance computing resources has increased significantly, especially the demand for graphics processing units (GPUs). Although traditional central processing units (CPUs) perform well in general computing tasks, their performance and efficiency often fail to meet the needs when faced with complex deep learning algorithms, large-scale data processing, and real-time image / video processing and other highly computing-intensive tasks. In contrast, GPUs, with their highly parallel computing architecture, can process large amounts of data streams and computing tasks simultaneously, significantly accelerating tasks such as training neural networks, image rendering, and high-dimensional data analysis.

[0003] Especially in the field of artificial intelligence, the training and reasoning of deep learning models require processing a large number of matrix multiplications and parallel computing, which is exactly the strength of GPU. Therefore, GPU has gradually become an indispensable computing engine in key technology fields such as big data analysis, autonomous driving, natural language processing, and image recognition. At the same time, the popularity of cloud computing has made it possible to obtain large-scale GPU clusters on demand, and the popularity of GPU communication and interconnection technology has greatly improved the computing power of data centers and promoted the development of emerging technologies.

[0004] The mainstream GPU communication interconnection method is the GPU Direct Transmission (GPUDirect) technology developed by NVIDIA. GPU communication interconnection systems are generally divided into two scenarios: communication interconnection of multiple GPU cards on a single host and cross-host communication interconnection of multiple GPU cards on multiple hosts.

[0005] GPUDirect P2P technology is used in single-machine multi-card scenarios. This technology allows multiple GPUs to communicate with each other directly through high-speed interconnects (such as PCIe or NVLink), bypassing the CPU and system memory. This direct access mechanism reduces the need for the CPU to participate in data transmission, reduces latency, and improves bandwidth utilization.

[0006] GPU Direct Remote Memory Access (GPUDirect RDMA) technology is applied to multi-machine and multi-card scenarios, perfectly solving the communication problem of GPU cards between nodes in a computing cluster. GPUDirect RDMA is designed to support fast cross-host communication between GPUs. It combines GPU accelerated computing and RDMA (Remote Direct Memory Access) technology to achieve the ability to directly transfer and communicate data between GPUs and RDMA network devices. It allows GPUs to directly access data in RDMA network devices without the mediation of host memory or CPU. The implementation of RDMA communication networks depends on specific hardware and software support. At the hardware level, RDMA technology can be implemented through technologies such as InfiniBand (a high-performance network interconnect technology mainly used in data centers, supercomputers and high-performance computing fields), RoCE (RDMA over ConvergedEthernet, a technology that allows remote direct memory access over Ethernet) and iWARP (Internet WideArea RDMA Protocol, a protocol for implementing remote direct memory access on Ethernet). InfiniBand is a network designed specifically for RDMA and requires dedicated equipment. RoCE and iWARP are both Ethernet-based RDMA technologies.

[0007] GPUDirect RDMA technology bypasses the CPU and host memory by allowing the GPU to communicate directly with network devices, significantly improving data transmission efficiency, especially in high-performance computing, deep learning, and big data analysis. However, although GPUDirect RDMA has significant advantages in improving communication performance between GPUs, it also has some problems and disadvantages:

[0008] 1. High hardware dependence and high cost

[0009] GPUDirect RDMA relies on specific high-performance hardware, including network devices that support RDMA (such as InfiniBand). Building a distributed computing cluster that supports GPUDirect RDMA requires a large number of high-end network devices and switches, which have very high deployment and maintenance costs.

[0010] 2. High requirements for network quality

[0011] Although GPUDirect RDMA can bypass the CPU to directly transfer data, it is highly dependent on the performance of the underlying network. If the network bandwidth is insufficient or there is latency, the advantages of GPUDirect RDMA will not be realized and may even lead to inefficient communication.

[0012] 3. Large data delay and high protocol overhead

[0013] Traditional Ethernet-based RDMA technology (such as iWARP) transmits data through the TCP / IP (Transmission Control Protocol / Internet Protocol, one of the core protocols of the Internet) protocol stack, which introduces additional protocol overhead, such as congestion control and flow control mechanisms, resulting in large data delays and limiting its application in high-performance computing environments.

[0014] Based on this, the present invention is proposed. Summary of the invention

[0015] In view of the shortcomings of the existing GPUDirect RDMA technology, the present invention proposes a GPU cross-host communication interconnection system based on PCIe NTB. The technical problems to be solved include:

[0016] GPUDirect RDMA relies on high-performance dedicated network devices such as InfiniBand and RoCE, which have high deployment and maintenance costs. The present invention hopes to reduce hardware dependence and costs through PCIe bus technology.

[0017] GPUDirect RDMA is sensitive to network bandwidth and latency, and network performance degradation will affect data transmission efficiency. The present invention hopes to avoid excessive reliance on network quality by using the PCIe bus.

[0018] GPUDirect RDMA uses a network communication protocol to transfer data, which brings additional protocol overhead and higher latency. The present invention aims to reduce latency and protocol overhead by directly connecting to PCIe.

[0019] Based on the above problems, the specific technical solutions of the present invention are as follows:

[0020] A GPU cross-host communication interconnection system based on PCIe NTB, including two hosts, two GPUs, and two PCIeSwitch chips;

[0021] The two hosts are connected to PCIe Switch chips respectively, and the two PCIe Switch chips are connected via a Crosslink line. The connection port of the PCIe Switch chip is set to NTB mode, and the two GPUs achieve interconnection and communication through NTB.

[0022] As a further improvement, GPU heterogeneous communication software is integrated in the two hosts.

[0023] For further improvement, the hosts are interconnected via PCIe NT Crosslink, the NTB mechanism is used to implement cross-host memory access, and then the DMA inside the PCIe Switch chip is used to move data.

[0024] As a further improvement, by opening a window in NTB, the remote node address space is mapped to the local address space through NTB, so as to open up the access path between the two GPUs.

[0025] For further improvement, the two GPUs are the local GPU and the remote GPU, and the two hosts are the local host and the remote host;

[0026] The remote GPU BAR space is windowed in the local NTB, and the local GPU initiates access to the remote GPU BAR space through DMA, thereby realizing P2P transmission.

[0027] For further improvement, depending on whether IOMMU is enabled on both sides of the host node, the address translation process and data path of the local GPU accessing the remote GPU BAR space are divided into the following four cases:

[0028] Case 1: When the IOMMUs of both hosts are disabled, the local GPU accesses the remote GPU address only through one translation;

[0029] Case 2: When the local host turns on IOMMU and the remote host disables IOMMU, the local GPU accesses the remote GPU address through secondary conversion; the secondary conversion is that all data requests need to be translated by the host's IOMMU hardware, translated into actual physical addresses, and then data packets are forwarded.

[0030] Case 3: When the local host disables IOMMU and the remote host turns on IOMMU, the local GPU accesses the remote GPU address after secondary conversion; the NTB windowing address in case 3 is different from the NTB windowing address in case 2;

[0031] Case 4: When the IOMMUs of both hosts are turned on, the local GPU accesses the remote GPU address through three conversions.

[0032] Beneficial effects of the present invention:

[0033] 1. The present invention uses PCIe NTB to perform GPU cross-host communication, configures a PCIe port into NTB mode, and uses Crosslink (a special PCIe Switch connection method that allows two PCIe Switch upstream ports to be directly connected, or two downstream ports to be directly connected) to connect. By opening a window in NTB, the address of the remote device is mapped in the local address space, thereby achieving the purpose of accessing the remote device. Four address conversion methods are supported according to the IOMMU enable status. When both sides of the IOMMU are disabled, only one address conversion is required; when one side of the IOMMU is turned on and the other side is turned off, a second address conversion is required; when both sides of the IOMMU are turned on, three address conversions are required. The purpose of bypassing the CPU is achieved through DMA access.

[0034] 2. Compared with the prior art, the present invention uses PCIe Switch to replace high-end network equipment, which can significantly reduce hardware costs and reduce the complexity of system deployment. The high bandwidth of the PCIe bus can greatly reduce communication delays. Since it no longer relies on network equipment and network protocols, there is no requirement for network quality, and efficient data transmission can be performed between computing cluster nodes without a network. Compared with RDMA, cross-host access of GPUs based on PCIe NTB is transparent, and the MMIO (Memory-Mapped I / O, memory-mapped I / O, is a technology that allows hardware devices to access and control through memory addresses) access mechanism can be used, so that data transmission is like operating in local memory, without the need for special network configuration and connection. Communication between devices can be directly accessed through memory addresses, without the need to set up complex network connections, IP addresses or special software protocols.

[0035] 3. The present invention is applied to the scenario of GPU cross-host communication. Two or more hosts / servers need to use the PCIeSwitch chip (this chip is usually configured in large server cluster nodes). It only needs to configure one port of the PCIe Switch chip to NTB mode, and then use the Crosslink method to connect the NTB ports of the two hosts / servers. In this way, there is no need to configure a high-end RDMA network card, and the existing resources are used to maximize resources. In addition, the upper-layer application software does not need to be modified (there is no need to modify the hardware driver code, and no software middle layer is required). It only needs to integrate the GPU heterogeneous communication software to realize direct interconnection and communication between dual-host GPU cards, and achieve low-latency, high-bandwidth GPU cross-host interconnection and communication. The GPU that communicates across hosts through the PCIe NTB method can achieve the effect of GPUDirect P2P, as if two GPUs are communicating in the same host, truly realizing software and hardware non-perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A hardware connection diagram of a GPU cross-host communication interconnection system based on PCIe NTB;

[0037] Figure 2 This is a schematic diagram of NTB address conversion;

[0038] Figure 3 This is the address translation process diagram when both IOMMUs are disabled;

[0039] Figure 4 This is the address translation process diagram when Host 0 turns on IOMMU and Host 1 disables IOMMU;

[0040] Figure 5 This is the address translation process diagram when Host 0 disables IOMMU and Host 1 enables IOMMU;

[0041] Figure 6 This is a diagram of the address translation process when the IOMMUs of both hosts are turned on. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] Definitions of Abbreviations and Key Terms

[0044] PCIe: Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard, mainly used to expand computer system bus data throughput and improve device communication speed;

[0045] NTB: Non-Transparent Bridging, a non-transparent bridge is a way to solve the address conflict and data packet routing problems between systems through address conversion and data packet translation;

[0046] HOST: A host computer is usually defined as a large time-sharing computer system in a centralized computer model;

[0047] GPU: Graphics Processing Unit, is a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.);

[0048] FPGA: Field Programmable Gate Array, field programmable gate array;

[0049] CPU: Central Processing Unit, as the computing and control core of the computer system, is the final execution unit for information processing and program running;

[0050] Crosslink: A feature defined in the PCIe specification that supports the upstream port of a switch to be connected to the upstream port of another switch, and the downstream port to be connected to the downstream port of another switch;

[0051] PCIe NT Crosslink: An interconnection technology based on PCIe NT ports;

[0052] RDMA: Remote Direct Memory Access, remote direct address access;

[0053] Port: There are many physical ports on the PCIe interactive chip, which are used to connect to the host or PCIe device;

[0054] DMA: Direct Memory Access, a function provided by the computer bus architecture that does not require the participation of the CPU. It allows data to be transferred between devices and memory.

[0055] IOMMU: Input-Output Memory Management Unit; IOMMU is a hardware unit located between the device and the main memory, responsible for managing the device's direct memory access requests. It prevents devices from accessing memory areas across boundaries through address translation and memory protection mechanisms, ensuring data security and system stability.

[0056] IOVA: Input-Output Virtual Address, input-output virtual address. IOVA is a virtual address space provided by IOMMU and used by devices when performing DMA transfers.

[0057] PCIe Switch: A hardware device used to expand PCIe ports, allowing more devices to be connected to a PCIe bus;

[0058] IP address: Internet Protocol Address, refers to the Internet Protocol address;

[0059] NVLink: A high-speed interconnect technology developed by NVIDIA that is used to connect central processing units (CPUs) and graphics processing units (GPUs), as well as direct communication between multiple GPUs.

[0060] GPUDirect: GPU direct transmission; enables direct communication and data transfer between the GPU and other devices (such as network interface cards and storage devices) without involving the CPU;

[0061] GPUDirect P2P: GPU direct peer-to-peer transmission; it is a technology that allows direct data transmission between multiple GPUs in the same host;

[0062] GPUDirect RDMA: GPU Direct Remote Memory Access; is a technology that allows direct data transfer and communication between GPUs and RDMA network devices;

[0063] 0xF000, 0x5000, 0x1F000, 0xA000, 0x5F00, 0x1F000, 0x85000, these are all hexadecimal characters used to represent space addresses. Example 1

[0064] A hardware connection method for GPU cross-host communication interconnection system based on PCIe NTB Figure 1 , Host 0 is connected to PCIe Switch 0, Host 1 is connected to PCIe Switch 1, the two PCIe Switch chips are connected via a Crosslink cable, and the PCIe Switch port is set to NTB mode, so that the two GPUs (GPU 0 and GPU 1) can communicate with each other through NTB. Compared with the GPUDirect RDMA interconnection solution, the network card is no longer required, but the PCIeSwitch chip is used.

[0065] exist Figure 1 In the above diagram, USP: Upper Stream Port refers to the upstream port of PCIe Switch, which is usually connected to the host. DSP: Down Stream Port refers to the downstream port of PCIe Switch, which is usually connected to terminal devices such as GPU, FPGA, etc. Example 2

[0066] In the present invention, hosts are interconnected through PCIe NT Crosslink, and cross-host memory access is achieved using the NTB mechanism. Then, data is moved using the DMA inside the PCIe Switch chip. This process does not require the participation of the central CPU, which can improve data transmission efficiency. Cross-host communication needs to solve the problem of address domain conversion. If you want to open the access path between two GPUs, the key is to open a window in the NTB, that is, to map the remote node address space to the local address space through the NTB. The NTB plays a role in address conversion. NTB address conversion is as follows: Figure 2 As shown in the figure, Host 0 wants to access the RAM (Random Access Memory) space address 0xF000 of Host 1. The window address of 0xF000 in the NTB on the Host 0 side is 0x5000. After the data packet of Host 0 accessing 0x5000 reaches the NTB, it will automatically penetrate the NTB and reach the Host 1 side, thereby achieving the purpose of accessing the address 0xF000 on the Host 1 side.

[0067] Based on PCIe NTB, GPU cross-host communication opens a window in the local NTB for the remote GPU BAR space, and the local GPU initiates access to the remote GPU BAR space through DMA, thereby realizing P2P (Peer-to-Peer) transmission. Depending on whether the IOMMU is turned on on both sides of the host node, the address conversion process and data path are divided into four cases. Example 3

[0068] In Example 3, when both host IOMMUs are disabled:

[0069] The address translation process with both IOMMUs disabled is as follows Figure 3 As shown in the figure, GPU 0 accesses the address of GPU 1 only through one conversion. GPU 1' is the address window mapping space of GPU 1 in the NTB on the host side of Host 0. The address 0x1F000 of GPU 1 is mapped to the address 0xA000 on the Host 0 side, which falls within the address range of NTB. GPU 0 initiates a DMA access request to GPU 1. The address passed to DMA is the physical address 0xA000. When the data packet reaches the NTB on the Host 0 side, it passes through the NTB and is automatically converted to 0x1F000, and finally accesses the actual physical address of GPU 1.

[0070] In the figure, GPU 1' is the physical address range of GPU 1 on the Host 1 side and the window address range in the NTB on the Host 0 side. GPU 0' is the physical address range of GPU 0 on the Host 0 side and the window address range in the NTB on the Host 1 side. Example 4

[0071] In Example 3, when Host 0 enables IOMMU and Host 1 disables IOMMU:

[0072] In a system with IOMMU turned on, the IO data request initiated by the device is no longer a physical address, but an IOVA address. All data requests need to be translated by the host's IOMMU hardware, translated into the actual physical address, and then the data packet is forwarded. This can ensure data security and avoid address overlap problems.

[0073] For the case where Host 0 has IOMMU enabled and Host 1 has IOMMU disabled, the address needs to go through two levels of translation, such as Figure 4 As shown. The address 0x1F000 of GPU 1 is mapped to the address 0xA000 on the Host 0 side, which falls within the NTB address range. The IOVA address corresponding to the physical address 0xA000 is 0x5F00. GPU 0 initiates a DMA access request to GPU 1, and the address passed to DMA is the IOVA address 0x5F00. In this case, the data packet will first perform the first-level address translation in the IOMMU, converting the IOVA address 0x5F00 to the physical address 0xA000 of GPU 1', and the data packet will be routed to the NTB on the Host 0 side, then pass through the NTB and perform the second-level address translation, converting the physical address 0xA000 of GPU 1' to the actual GPU 1 address 0x1F000. Example 5

[0074] When Host 0 disables IOMMU and Host 1 enables IOMMU:

[0075] In this case, the address also needs to be converted twice, but the NTB window address is different. Host 1 turns on IOMMU, so the IO data request addresses initiated by all devices on Host 1, including NTB, are IOVA addresses, which need to be converted to actual physical addresses by Host 1 IOMMU hardware. When the access data packet from GPU 0 to GPU 1 passes through the NTB on the Host 0 side and reaches Host 1, it should be an IOVA address. Therefore, the IOVA address corresponding to the physical address of GPU 1 on the Host 1 side needs to be mapped to the NTB on the Host 0 side. The address conversion process and data transmission path are as follows: Figure 5, the IOVA address corresponding to GPU 1 physical address 0x1F000 on Host 1 is 0x85000, and 0x85000 is mapped to address 0xA000 on Host 0. When GPU 0 initiates a DMA access request to GPU 1, since Host 0 disables IOMMU, DMA uses physical address 0xA000, and the data packet directly reaches NTB for first-level address translation, which is converted to IOVA address 0x85000 on Host 1. After the data packet reaches Host 1, since Host 1 turns on IOMMU, the data packet is forwarded upward to Host 1 IOMMU hardware, and after the second-level address translation, the IOVA address 0x85000 is converted to the actual physical address 0x1F000. Example 6

[0076] When both host IOMMUs are enabled:

[0077] This situation is a combination of Example 5 and Example 6. The address needs to be converted three times. The data packets on both sides must be converted by the IOMMU hardware. Figure 6 As shown, GPU 0 initiates DMA access to GPU 1, using the IOVA address. The data packet is forwarded upward to the IOMMU hardware for first-level address translation, converting the IOVA address 0x5F00 to the physical address 0xA000 of GPU 1'. The data packet then reaches the NTB on the Host 0 side for second-level address translation, converting the physical address 0xA000 of GPU 1' to the IOVA address 0x85000 of GPU 1. The data packet passes through the NTB to Host 1, and continues to be forwarded upward to the IOMMU hardware (IOMMU 1) on the Host 1 side, converting the IOVA address 0x85000 to the actual physical address 0x1F000 of GPU 1. At this point, GPU 0 completes the access to GPU 1, and cross-host access is achieved.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A GPU inter-host communication interconnection system based on PCIe NTB, characterized by: Includes two hosts, two GPUs, and two PCIe Switch chips; The two hosts are respectively connected to a PCIe Switch chip, and the two PCIe Switch chips are connected via a Crosslink line. The connection port of the PCIe Switch chip is set to NTB mode, and the two GPUs are interconnected and communicated via NTB. Hosts are interconnected through PCIe NT Crosslink, using the NTB mechanism to achieve cross-host memory access, and then using the DMA inside the PCIe Switch chip to move data; By opening a window in NTB, the remote node address space is mapped to the local address space through NTB, so that the access path between the two GPUs is opened; The two GPUs are local GPU and remote GPU respectively, and the two hosts are local host and remote host respectively; The remote GPU BAR space is windowed in the local NTB, and the local GPU initiates access to the remote GPU BAR space through DMA to achieve P2P transmission; Depending on whether IOMMU is enabled on both sides of the host node, the address translation process and data path of the local GPU accessing the remote GPU BAR space are divided into the following four cases: Case 1: When the IOMMUs of both hosts are disabled, the local GPU accesses the remote GPU address only through one translation; Case 2: When the local host turns on IOMMU and the remote host disables IOMMU, the local GPU accesses the remote GPU address through secondary conversion; Case 3: When the local host disables IOMMU and the remote host turns on IOMMU, the local GPU accesses the remote GPU address after secondary conversion; the NTB windowing address in case 3 is different from the NTB windowing address in case 2; Case 4: When the IOMMUs of both hosts are turned on, the local GPU accesses the remote GPU address through three conversions.

Images