GPU BOX, server, interconnection system and high-speed interconnection and data interaction method

Abstract

The embodiment of the invention discloses a GPU BOX, a server, an interconnection system and a high-speed interconnection and data interaction method. The GPU BOX high-speed interconnection method is characterized in that the GPU BOX comprises a GPU and a control mainboard, an internal PCIe switch, an internal storage device, a first DMA controller and a system controller are integrated in the control mainboard, and a second DMA controller is integrated in the GPU; the system controller receives the reading instruction and performs data read-write initialization on the first DMA controller and the second DMA controller based on the reading instruction; the first DMA controller after data read-write initialization reads cache data from the internal storage device, and transmits the cache data to the GPU via the internal PCIe switch through the established GDS connection; wherein the GDS connection between the GPU and the internal storage device is established through the system controller, the second DMA controller, the internal PCIe switch and the first DMA controller. According to the method, high-speed direct data transmission of the GPU BOX can be achieved, CPU data transfer is not needed, and the data transmission efficiency is greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the field of high-performance computing technology, and in particular to a GPU BOX, a server, an interconnection system, a high-speed interconnection, and a data interaction method. Background Art

[0002] In the fields of high-performance computing and artificial intelligence, GPUs (graphics processing units) have become the core of data processing with their powerful parallel computing capabilities. However, in traditional models, data transmission involving the GPU requires a second copy through the CPU: external data first enters system memory, then the CPU copies it to the GPU's video memory. After the GPU completes the calculation, the results are copied from the video memory back to the system memory for further processing by the CPU.

[0003] This double copying method requires data to be transferred back and forth between the CPU and GPU within the GPU Box, increasing transmission latency, significantly consuming CPU resources, and reducing hardware utilization within the box, preventing the CPU from focusing on other important tasks. For example, during deep learning training, frequent copying can significantly slow down training.

[0004] Because each GPU box relies on the CPU to relay data between them, data exchange between boxes is complex and inefficient. In distributed computing, multiple boxes need to collaborate to process large amounts of data, but traditional secondary copying creates a transmission bottleneck, impacting system scalability and parallel computing capabilities. As computing performance demands increase, traditional solutions struggle to meet the requirements of modern high-performance computing systems. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure provide a GPU BOX, server, interconnection system, high-speed interconnection and data interaction method, which can realize direct transmission of data within the GPU BOX, interconnection of multiple boxes within the server and cross-server interaction, without the need for CPU to transfer data, thereby greatly improving data transmission efficiency.

[0006] In a first aspect, the embodiments of the present disclosure provide a GPU BOX high-speed interconnection method, which adopts the following technical solutions: The GPU BOX includes a GPU and a control mainboard, wherein the control mainboard is internally integrated with an internal PCIe switch, an internal storage device, a first DMA controller and a system controller, and the GPU is internally integrated with a second DMA controller; The system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

[0007] In a second aspect, the embodiments of the present disclosure further provide a method for high-speed interconnection within a GPU server, which employs the following technical solutions: Optionally, the server is configured with at least a first GPU BOX, a second GPU BOX, and a node, wherein the node includes a bus PCIe switch, and each GPU BOX includes a GPU, a second DMA controller, and a control mainboard, wherein the control mainboard integrates an internal PCIe switch and a system controller; The system controller of the first GPU BOX receives the initialization configuration instruction, and performs data interaction initialization on the second DMA controller based on the initialization configuration instruction; Determine a source end and a target end from the first GPU BOX and the second GPU BOX based on the data transmission requirements of the node; The second DMA controller in the source end, after data exchange initialization, receives a data read / write instruction, and based on the data read / write instruction, reads the video memory data from the GPU of the source end, and transmits the video memory data to the GPU of the target end via the bus PCIe switch through the established GDP2P connection; A GDP2P connection is established between the GPUs in the first GPU BOX and the second GPU BOX through the system controller, the second DMA controller, the internal PCIe switch, and the CPU control board of the first GPU BOX and the second GPU BOX.

[0008] Optionally, each of the GPU BOXs further includes an internal CXL switch, and the node further includes a bus CXL switch and a shared memory pool; The second DMA controller receives the memory configuration instruction, and maps the video memory of the corresponding GPU to the shared memory pool of the node through the established CXL connection according to the memory configuration instruction, and establishes a correspondence between the video memory address space and the shared memory pool address space; The GPU of the GPU BOX, the internal CXL switch, the bus CXL switch, and the shared memory pool are sequentially connected via CXL; Determine the source GPU BOX and target GPU BOX based on data sharing requirements; The second DMA controller of the source GPU BOX generates a data transfer request based on the correspondence between the video memory address space of the corresponding GPU and the address space of the shared memory pool, and sends the data transfer request and the shared data of the corresponding GPU to the shared memory pool; The shared memory pool writes the shared data into a designated allocation area based on the data transmission request; The second DMA controller of the target GPU BOX reads the shared data from the designated allocation area and writes the shared data into the video memory of the corresponding GPU based on the correspondence between the video memory address space of the corresponding GPU and the shared memory pool address space.

[0009] Optionally, each of the GPU BOXs further includes an internal storage device and a first DMA controller; In the same GPU BOX, the system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

[0010] In a third aspect, the present disclosure also provides a data interaction method for a GPU high-speed interconnection system, which adopts the following technical solution: The system includes at least a first GPU server, a second GPU server, and a network switch. The first GPU server includes at least a first GPU BOX, a second GPU BOX, and a first node. The second GPU server includes at least a third GPU BOX, a fourth GPU BOX, and a second node. Each of the GPU BOXs includes a GPU, a second DMA controller, a network card, a third DMA controller, and a control mainboard, wherein the control mainboard integrates an internal PCIe switch and a system controller; Based on cross-server data transmission requirements, determine a source node and a target node from the first node and the second node, determine a sending end from the GPU box of the source node, and determine a receiving end from the GPU box of the target node; The system controller at the transmitting end receives a data preparation instruction; the system controller at the receiving end receives a data reception instruction; The system controller of the transmitting end performs data preparation initialization on the second DMA controller and the third DMA controller of the transmitting end based on the data preparation instruction; the system controller of the receiving end performs data reception initialization on the second DMA controller and the third DMA controller of the receiving end based on the data reception instruction; The second DMA controller in the transmitting end, after data preparation and initialization, reads the operation data from the corresponding GPU and transmits the operation data to the GPU of the receiving end through the established GDR connection; The GPU of any GPU BOX in the first GPU server establishes a GDR connection with the GPU of any GPU BOX in the second GPU server through the system controller, the second DMA controller, the internal PCIe switch, the third DMA controller, the network card, the network layer interface and the network switch.

[0011] Optionally, each of the GPU BOXs further includes an internal storage device and a first DMA controller; In the same GPU BOX, the system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

[0012] In a fourth aspect, an embodiment of the present disclosure further provides a control system for controlling a GPU, the control system comprising a control motherboard, an internal storage device, a first DMA controller, a PCIe switch, and a system controller; the internal storage device, the first DMA controller, the internal PCIe switch, and the system controller are integrated on the control motherboard; The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The first DMA controller is used to control the reading and writing of data of the internal storage device; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch.

[0013] In a fifth aspect, an embodiment of the present disclosure further provides a GPU BOX, comprising a GPU and a control motherboard, wherein the control motherboard integrates an internal PCIe switch, an internal storage device, a first DMA controller, and a system controller, and the GPU integrates a second DMA controller. The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The first DMA controller is used to control the reading and writing of data of the internal storage device; The second DMA controller is used to control data reading and writing of the GPU; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

[0014] Optionally, a GDP2P connection is established between GPUs in any two GPU BOXs on the same node through the system controller, the second DMA controller, and the internal PCIe switch of the GPU BOX.

[0015] Optionally, a GDP2P connection is established between GPUs in any two GPU BOXs on the same node through the system controller, the second DMA controller, and the internal PCIe switch of the GPU BOX.

[0016] Optionally, the GPU BOX further includes a network card and a third DMA controller; The network card is used to connect the GPU BOX to the network; The third DMA controller is used to control data transmission of the GPU BOX in the network.

[0017] In a sixth aspect, an embodiment of the present disclosure further provides a GPU server, the server being configured with at least a first GPU BOX, a second GPU BOX, and a node, the node including a bus PCIe switch, each GPU BOX including a GPU, a second DMA controller, and a control motherboard, wherein the control motherboard is internally integrated with an internal PCIe switch and a system controller; The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The second DMA controller is used to control data reading and writing of the GPU; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch; A GDP2P connection is established between the GPUs in the first GPU BOX and the second GPU BOX through the system controllers, second DMA controllers, internal PCIe switches, and the CPU control board of the first GPU BOX and the second GPU BOX.

[0018] Optionally, each of the GPU BOXs further includes an internal CXL switch, and the node further includes a bus CXL switch and a shared memory pool; The GPU of the GPU BOX, the internal CXL switch, the bus CXL switch, and the shared memory pool are connected in sequence through CXL.

[0019] Optionally, each of the GPU BOXs further includes an internal storage device and a first DMA controller; A GDS connection between the GPU in the GPU BOX and the internal storage device is established through the system controller of the GPU BOX, the second DMA controller, the internal PCIe switch and the first DMA controller.

[0020] In a seventh aspect, an embodiment of the present disclosure further provides a GPU high-speed interconnection system, the system comprising at least a first GPU server, a second GPU server, and a network switch, wherein the first GPU server comprises at least a first GPU BOX, a second GPU BOX, and a first node; the second GPU server comprises at least a third GPU BOX, a fourth GPU BOX, and a second node; Each of the GPU BOXs includes a GPU, a second DMA controller, a network card, a third DMA controller, and a control mainboard, wherein the control mainboard integrates an internal PCIe switch and a system controller; Each node includes a network layer interface, and the network layer interface is connected to the network card and the network switch respectively; A GPU of any GPU BOX in the first GPU server establishes a GDR connection with a GPU of any GPU BOX in the second GPU server through the system controller, the second DMA controller, the internal PCIe switch, the third DMA controller, the network card, the network layer interface and the network switch.

[0021] Optionally, each of the GPU BOXs further includes an internal storage device and a first DMA controller; A GDS connection between the GPU in the GPU BOX and the internal storage device is established through the system controller of the GPU BOX, the second DMA controller, the internal PCIe switch and the first DMA controller.

[0022] The GPU BOX provided by the disclosed embodiments integrates important components such as the internal PCIe switch and system controller on a control board, making it easier to manage the spatial layout of these components, thereby avoiding the large amount of space occupied by the dispersed layout of each component and reducing the size of the entire GPU BOX. Furthermore, the inter-board connector enables connection between the GPU BOX and the node, thereby establishing communication between the internal components of the control board and the node. Furthermore, since the GPU, internal storage device, and system controller are all physically connected to the internal PCIe switch, the GPU, internal storage device, and system controller can communicate with the node via the internal PCIe switch. Based on the physical connection between the GPU, internal storage device, and internal PCIe switch, a second DMA controller can control data exchange between the GPU and the internal PCIe switch, and a first DMA controller can control data exchange between the internal storage device and the internal PCIe switch. Furthermore, through the second DMA controller and the first DMA controller, a GDS connection is established between the GPU and the internal storage device. The GDS utilizes P2P DMA technology, allowing data to flow directly between the GPU and the internal storage device, greatly improving data transmission efficiency and thereby enhancing the overall performance of the GPU BOX.

[0023] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of the physical structure of a GPU BOX provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of the internal connections of a GPU BOX provided in an embodiment of the present disclosure; Figure 3A flowchart of a GPU BOX high-speed interconnection method provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a GPU server structure provided in an embodiment of the present disclosure; Figure 5 A flowchart of a method for high-speed interconnection within a GPU server provided in an embodiment of the present disclosure; Figure 6 A flowchart of a CXL communication method provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of a GPU high-speed interconnection system structure provided by an embodiment of the present disclosure; Figure 8 A flowchart of a data interaction method for a GPU high-speed interconnection system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0027] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0028] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0030] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0031] First, some of the terms and related technologies involved in this disclosure are explained to facilitate understanding by those skilled in the art.

[0032] (1) Hot plug Hot swapping means plugging and unplugging the host while it is powered on. It allows users to remove or replace the GPU BOX (also known as GPU Box) attached to the host without shutting down the system or cutting off the host power, without affecting the operation of the host server system, thereby improving the system's timely recovery capabilities, scalability, and flexibility.

[0033] (2) GDS GDS (GPUDirect Storage) technology allows the GPU to transfer data directly to internal storage devices without going through CPU memory, thereby improving data transmission efficiency and performance.

[0034] (3) P2P DMA P2P DMA (Peer-to-Peer Direct Memory Access) is a technology that enables direct memory data transfer between different devices in a computer system. It allows devices connected to the same PCIe bus to transfer data directly without going through CPU memory, thereby reducing latency and improving data transfer efficiency.

[0035] (4) GDP2P GDP2P (GPU Redirect P2P) allows a GPU to directly transfer data with other GPUs through the PCIe bus. The GPU redirection function optimizes the data path and improves communication efficiency in a multi-GPU environment.

[0036] (5) CXL CXL (Compute Express Link) is a high-speed interconnect technology based on the PCIe protocol. It is used to connect processors, accelerators, and memory expansion devices. It supports functions such as memory consistency and cache consistency and is suitable for high-performance computing and data center applications.

[0037] (6) GDR GDR (GPUDirect RDMA) allows the GPU to directly transfer data to other devices (such as other GPUs or network devices) through RDMA (Remote Direct Memory Access), reducing the CPU load and improving communication efficiency.

[0038] In traditional data transmission, data usually needs to be transferred through the CPU, that is, the data is first read from the source device to the CPU memory, and then the CPU writes the data to the target device, resulting in complex data interaction and low transmission efficiency.

[0039] Example 1: The present disclosure provides a GPU BOX and a GPU BOX high-speed interconnection method to solve the above technical problems. Figure 1 A schematic diagram of the GPU BOX's physical structure is shown. The GPU BOX includes a GPU and a control motherboard. The control motherboard integrates an internal PCIe switch, internal storage devices, a first DMA controller, and a system controller, while the GPU integrates a second DMA controller. The control motherboard, internal storage devices, first DMA controller, PCIe switch, and system controller form the control system for the GPU, enabling resource allocation and scheduling, data transmission and interaction, operational status monitoring, function configuration, and initialization.

[0040] The control motherboard controls communication between internal components and the node's CPU control board. The control motherboard is equipped with inter-board connectors, which can be BP connectors (board-to-board connectors). These BP connectors connect to the node's switching backplane via a backplane interface. The switching backplane, serving as the core hub for data exchange within the node, is in turn connected to the node's CPU control board. This connection architecture, utilizing the BP connectors and switching backplane, enables effective communication between key components within the GPU BOX and the CPU control board. The BP connectors feature bidirectional signal transmission capabilities. They can transmit signals generated by the control motherboard's internal components to other parts of the node. They can also receive signals from external devices and forward them to the control motherboard's internal components, ensuring smooth information exchange and coordinated operation across the entire system.

[0041] The control motherboard has a slot, such as a PCIE x16 slot, through which the GPU is connected to the control motherboard. A GPU power supply interface, such as an 8-pin power connector, is provided on the node or inside the GPU BOX. This interface is connected to the GPU's own power supply interface to provide power to the GPU.

[0042] By controlling the integration of various components on the motherboard, not only can efficient signal acquisition, processing, and transmission be achieved, but the layout of various components can also be effectively integrated, avoiding the large amount of space occupied by the scattered layout of each component, thereby reducing the volume of the entire GPU BOX. This is especially suitable for application scenarios with high space requirements.

[0043] Reference Figure 2 A schematic diagram of the GPU BOX's internal connections shows the GPU and system controller physically connected to the internal PCIe switch. Internal storage devices are also physically connected to the internal PCIe switch via the internal storage device interface, enabling communication between the system controller, GPU, and internal storage devices through the internal PCIe switch. Furthermore, the GPU, system controller, and internal PCIe switch are all connected to a system bus (e.g., a PCIe bus). A second DMA controller is typically built into and connected to the GPU, controlling data reads and writes to the GPU. A first DMA controller is typically built into and connected to the internal storage device, controlling data reads and writes to the internal storage device. The system controller is used to control the data interaction and resource allocation of components connected to the internal PCIe switch. When the GPU BOX is inserted into the node or the node system is started, the system controller will initialize the second DMA controller and the first DMA controller. During the initialization operation, the memory address space of the internal storage device and the GPU will be mapped to the system bus through their respective DMA controllers. This is because in the initial stage of the system, the status of each component needs to be correctly configured, the DMA controller needs to be set to the appropriate working mode and transmission parameters, etc., and the mapping relationship between the component memory address and the system bus address must be established so that subsequent data can be smoothly transmitted between components.

[0044] A GDS connection is established between the GPU and the internal storage device via the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller. Based on the physical connections among the system controller, the GPU, the internal storage device, and the internal PCIe switch, and through initialization and configuration by the system controller, the second DMA controller can control data exchange between the GPU and the internal PCIe switch. This allows the GPU to quickly communicate with the internal PCIe switch via the second DMA controller. Similarly, the first DMA controller can control data exchange between the internal storage device and the internal PCIe switch. This allows the internal storage device to quickly communicate with the internal PCIe switch via the first DMA controller, thereby establishing a GDS connection between the GPU and the internal storage device. This allows data to flow directly between the GPU and the internal storage device, significantly improving data transfer efficiency. Furthermore, both the second DMA controller and the first DMA controller are connected to the system bus, which provides a channel for data transfer. Through the system bus, the DMA controller coordinates data transfer between the GPU, the internal storage device, and the internal PCIe switch, ensuring accurate and efficient data exchange between each component, thereby achieving a stable GDS connection. GDS uses P2P DMA technology to provide users and applications with the ability to transfer data directly from internal storage devices to GPU memory.

[0045] The system controller can be a SoC (System on Chip), which offers advantages such as small size, high integration, low power consumption, and high performance. This makes it suitable for embedding in the GPU Box, reducing the overall footprint of the GPU Box and improving its integration and reliability. The internal storage device interface uses the E1.S (Enterprise 1.S) interface, and internal storage devices such as SSDs should support the E1.S interface. The E1.S interface offers high bandwidth, low latency, and excellent scalability, meeting the data transmission requirements of high-performance storage devices, thereby improving the overall performance and efficiency of the GPU Box.

[0046] Reference Figure 3 The flow chart of the GPU BOX high-speed interconnection method is shown. The internal high-speed interconnection method of the GPU server provided by the present disclosure can be applied to Figure 1 The GPU BOX shown and Figure 2 The control mainboard shown in FIG. 1 includes but is not limited to the following steps: S1: The system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; S2: After data reading and writing are initialized, the first DMA controller reads the cached data from the internal storage device and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection.

[0047] Taking the scenario of deep learning training GPU reading data from the internal storage device as an example, in S1, when the node has the demand for the GPU to read data from the internal storage device, the node's CPU sends a read instruction, which is specifically an I / O read instruction, instructing to read the cached data from the specified location of the internal storage device and load it into the GPU's video memory. The inter-board connector receives the instruction and forwards it to the system controller.

[0048] The internal PCIe switch enables information transfer between the GPU, the second DMA controller, internal storage devices, the first DMA controller, and the system controller. Furthermore, the system controller initializes data reads and writes for the second and first DMA controllers. Based on the read instructions, it configures the relevant parameters of the second and first DMA controllers, including the source address (i.e., the source address of the cached data), the target address (the destination address of the cached data), the transfer length (the amount of data to be transferred), and the transfer mode (e.g., single transfer or batch transfer).

[0049] In S3, in the GDS connection link between the GPU and the internal storage device, the first DMA controller is responsible for reading cached data from the internal storage device. When receiving a data transfer instruction sent by the system controller, the first DMA controller will configure parameters such as the source address (the internal storage unit address of the internal storage device), the target address (the GPU's video memory address), and the amount of data to be transferred according to the instruction. Based on these parameters, the cached data is directly read from the flash memory chip of the internal storage device and pushed to the system bus. It is then transmitted to the second DMA controller of the GPU through the internal PCIe switch without excessive intervention from the CPU.

[0050] The second DMA controller is responsible for receiving cached data transmitted from the internal PCIe switch and writing the cached data to the specified location of the GPU memory based on pre-configured parameters. At the same time, the second DMA controller will also generate an interrupt signal to notify the GPU that the internal data has been transferred, facilitating subsequent data processing operations.

[0051] GDS primarily utilizes peer-to-peer (P2P) DMA (DMA) technology, which allows direct data transfer between components connected to the same bus (such as a PCIe bus). This technology relies on hardware support, and each component that supports P2P DMA has its own DMA controller. When a source component transfers data to a target component, the source component's DMA controller directly transfers the data from the source component's memory to the target component's memory via the system bus according to preset parameters. This process eliminates the need for the CPU to frequently participate in data movement. The CPU only needs to perform initialization configuration before the transfer begins, such as setting the DMA controller's parameters and generating instructions based on these parameters. This allows the CPU to handle other tasks during the transfer, significantly improving data transfer efficiency and reducing CPU overhead and data transmission latency.

[0052] The GPU BOX architecture features a built-in system controller that connects the GPU and internal storage devices via an internal PCIe switch. Typically, data transfers between external devices require transit through the CPU's memory buffer. For example, when the GPU reads data from internal storage, the CPU first transfers the cached data to its own memory buffer, which is then copied to the GPU's video memory. However, with P2P DMA enabled, the GPU can read cached data directly from the internal storage device, eliminating the need for node CPU transit. In the GPU BOX, tasks such as P2P DMA initialization configuration, transfer control, and monitoring, previously handled by the node CPU, are now handled by the system controller's internal processor core (a CPU or FPGA), completely eliminating the need for the node CPU or memory. The internal PCIe switch provides data routing and switching, ensuring accurate data transfer between internal storage devices and the GPU. In a simple one-to-one connection scenario, P2P DMA transfers are primarily handled by the internal storage device and GPU's respective DMA controllers, enabling efficient transfer of internal cached data. Furthermore, through the coordinated collaboration of these components, high-speed interaction is achieved between components within the GPU BOX.

[0053] Example 2: To improve computing performance, multiple GPU boxes are typically configured on the same node. In this case, frequent data transmission is often required between multiple GPUs on the same node. However, in traditional data transmission methods, data exchange between GPUs typically requires the intervention of the node's CPU. This not only increases the CPU workload and leads to a large utilization of CPU resources, but also lengthens the data transmission path, increases data transmission latency, and reduces data transmission efficiency, thereby affecting the node's overall computing performance and response speed.

[0054] To solve this technical problem, the present disclosure also provides a GPU server and a method for high-speed interconnection within the GPU server. Figure 1 and Figure 2Based on the reference Figure 4 A schematic diagram of the GPU server structure is shown. The GPU server includes at least a first GPU BOX, a second GPU BOX, and a node. Both the GPU BOX and the node are equipped with a PCIe switch (Peripheral Component Interconnect Express Switch, PCIe Switch). For ease of distinction, the PCIe switch within the GPU BOX is called an internal PCIe switch, while the PCIe switch within the node is called a bus PCIe switch. The bus PCIe switch is integrated into the node's CPU control board.

[0055] Each GPU Box's internal PCIe switch is connected to the node's bus PCIe switch, enabling a communication link to be formed between the GPUs within the GPU Box, the internal PCIe switch, and the bus PCIe switch. Furthermore, a GDP2P connection is established between any two GPU Boxes on the same node through the system controller, second DMA controller, internal PCIe switch, and bus PCIe switch. GDP2P also utilizes P2P DMA technology, enabling GPUs within multiple GPU Boxes on a single node to be interconnected via the PCIe switch. Simultaneously, the system controller within the GPU Box replaces the node's CPU to complete DMA controller initialization configuration, enabling P2P DMA communication between GPUs directly using the system bus without going through the node's CPU.

[0056] In traditional solutions, GPUs are typically inserted directly into PCIE slots on nodes. Some nodes are directly connected to the CPU, requiring CPU transfer, while others are first connected to a bus PCIe switch, which then connects to the CPU. Even if these GPU servers can achieve P2P communication without CPU data transfer, this places a significant burden on the bus PCIe switch. This solution, while skipping CPU data transfer, also reduces the burden on the bus PCIe switch. This is because neither the GDS link nor the GDR link described in the subsequent third embodiment passes through the bus PCIe switch, preventing the bus PCIe switch from being blocked due to high load, which in turn affects communication speeds.

[0057] Reference Figure 5 The present disclosure also provides a method for high-speed interconnection of GPU BOX to a single node, which can be applied to Figure 4 The GPU server shown in FIG. 1 includes, but is not limited to, the following steps: T1: The system controller of the first GPU BOX receives the initialization configuration instruction and initializes data exchange with the second DMA controller based on the initialization configuration instruction; T2: Based on the data transmission requirements of the node, the source and target ends are determined from the first and second GPU boxes. T3: The second DMA controller on the source side, after data interaction initialization, receives data read and write instructions, and based on the data read and write instructions, reads the video memory data from the GPU on the source side, and transmits the video memory data to the GPU on the target side via the bus PCIe switch through the established GDP2P connection.

[0058] Taking the data transfer scenario between nodes and GPUs used for deep learning training as an example, in T1, when nodes need to perform collaborative computing tasks between GPUs—for example, in model parallelism, different GPUs are responsible for different parts of the model and need to exchange intermediate computation results to complete the forward and backward propagation of the entire model—the node's CPU (which, in this context, can be considered the execution vehicle for the deep learning training program) generates and sends an initialization configuration instruction. This instruction contains the initial parameters required for the second DMA controller to perform data exchange, including transfer mode parameters, buffer address parameters, transfer direction parameters, transfer length parameters, and interrupt control parameters. Through the inter-board connector, the system controller inside the GPU BOX receives the initialization configuration instruction. Based on this instruction, the system controller initializes the data exchange with the second DMA controller, including register configuration and state machine initialization, to determine the rules and parameters for data transfer and ensure correct and efficient data transmission.

[0059] In T2, the node's CPU analyzes the status and load of each GPU Box based on the current computing tasks and data processing requirements, taking into account the overall allocation of system resources. Based on the analysis results, the source and target are determined from the first and second GPU Boxes. The source is the sender of data, and the target is the receiver of data.

[0060] In T3, the node's CPU generates data read and write instructions based on data transmission requirements. The instructions are given to the system controller in the source-side GPUBOX. The system controller further confirms and adjusts the source-side second DMA controller based on the instructions (because it has been initialized before), and then passes the instructions to the internal PCIe switch. Finally, the internal PCIe switch passes the data read and write instructions to the second DMA controller in the source side that has been initialized after data exchange.

[0061] The data read and write instructions contain information such as the address and data length of the data to be read in the source-side GPU video memory. Based on this information, the second DMA controller initialized by data exchange reads the video memory data from the source-side GPU and transmits it to the second DMA controller initialized by data exchange on the target side through the internal PCIe switch on the source side, the bus PCIe switch, and the internal PCIe switch on the target side. The controller then writes the video memory data to the target-side GPU.

[0062] Each GPU uses the data and model parameters in the video memory to perform forward and backward propagation calculations, and sends the calculated gradient data directly to the video memory of other GPUs through the GDP2P link. At the same time, it also obtains the gradient data sent by other GPUs from the PCIe switch. In this process, there is no need for the CPU to transfer and process data, achieving efficient gradient synchronization.

[0063] Using this method, when one GPU in a node needs to transfer data to another, it triggers its own built-in second DMA controller to run. This second DMA controller manages the data movement, passing through the internal PCIe switch and the bus PCIe switch to transfer the memory data to the target GPU, completing the entire data transfer process. This method, through the GDP2P connection established, enables efficient and direct data transfer between GPUs in two GPU boxes on the same node.

[0064] Furthermore, although the GPU is equipped with video memory, its video memory space is relatively small, which makes it difficult to support the continuous transmission of large-scale data between GPUs, thereby resulting in the inability to effectively share data between GPUs. To address this problem, the present disclosure also provides a structure for realizing memory sharing between GPUs, referring to Figure 4Both the GPU Box and the node are equipped with a Compute Express Link Switch (CXL switch). For ease of distinction, the GPU Box's CXL switch is referred to as the internal CXL switch. The internal CXL switch controls resource sharing between the GPU and the GPU Box's peripheral components. The node's CXL switch is referred to as the bus CXL switch, which is integrated into the node's CPU control board. The GPU, internal CXL switch, and bus CXL switch are connected in sequence via CXL. The node is also equipped with multiple memory devices, such as DDR (Double Data Rate Synchronous Dynamic Random Access Memory). These memory devices form the node's shared memory pool. The bus CXL switch is connected to the node's shared memory pool via CXL. Based on this connection structure, the GPU establishes a CXL connection with the shared memory pool through the internal CXL switch and the bus CXL switch.

[0065] CXL connections utilize CXL interconnect technology, enabling GPUs to bypass the CPU and directly access the node's unified memory cache pool, enabling multi-GPU shared memory. By integrating node memory into a unified CXL protocol-based memory pool and connecting it to the local GPU box, GPUs can quickly access resources when memory is insufficient, manage on-demand allocation, and improve cluster performance and stability. Furthermore, the CXL protocol's low latency and high bandwidth enable high-speed device collaboration and efficient data sharing between GPUs.

[0066] The internal CXL switch is connected to the system controller. When the GPU BOX is inserted into the node or the node system is started, the system controller will initialize the internal CXL switch, including switch configuration, port binding, link establishment and other steps to ensure that the CXL switch can work normally and communicate with other components.

[0067] Further, refer to Figure 6 The flow chart of the CXL communication method is shown. On the same node, GPU boxes can share data through CXL interconnection technology. The method includes but is not limited to the following steps: T4: The second DMA controller receives the memory configuration instruction and, based on the memory configuration instruction, maps the corresponding GPU's video memory to the node's shared memory pool through the established CXL connection, establishing a correspondence between the video memory address space and the shared memory pool address space. T5: Determine the source GPU BOX and target GPU BOX based on data sharing requirements; T6: The second DMA controller of the source GPU BOX generates a data transfer request based on the correspondence between the video memory address space of the corresponding GPU and the address space of the shared memory pool, and sends the data transfer request and the shared data of the corresponding GPU to the shared memory pool; T7: The shared memory pool writes the shared data into the specified allocation area based on the data transfer request; T8: The second DMA controller of the target GPU BOX reads the shared data from the designated allocation area and writes it into the video memory of the corresponding GPU based on the correspondence between the video memory address space of the corresponding GPU and the address space of the shared memory pool.

[0068] Taking the example of data sharing between GPUs used for deep learning training, in T4, upon detecting the need to map GPU memory to a shared memory pool, the node's CPU generates a memory configuration instruction. This instruction contains key information required to map GPU memory to the shared memory pool, such as the shared memory pool's starting address, the size of the mapped memory, and the mapping mode (partial or full mapping). The node then sends the memory configuration instruction via the inter-board connector to the system controller, which performs a preliminary analysis of the instruction, checking its legitimacy and integrity to ensure it can be processed correctly. It then passes the instruction to the internal PCIe switch. Upon receiving the instruction, the internal PCIe switch forwards the memory configuration instruction to the GPU Box's secondary DMA controller based on its internal routing rules.

[0069] After receiving the memory configuration instruction, the second DMA controller of each GPU BOX begins parsing the various parameters in the instruction. Based on the information specified in the instruction, such as the shared memory pool starting address and the mapped video memory size, the second DMA controller begins the video memory mapping operation. The second DMA controller sends a mapping request to the shared memory pool through the CXL connection established between the internal CXL switch and the bus CXL switch. During this process, the internal CXL switch and the bus CXL switch are responsible for routing and forwarding the mapping request to ensure that the request reaches the shared memory pool accurately. After receiving the mapping request, the shared memory pool allocates a segment of address space in the shared memory pool to the corresponding GPU based on the information in the request. This address space will be mapped to the GPU's video memory.

[0070] The second DMA controller creates a local address mapping table, associating each address in the GPU's video memory with a corresponding address in the shared memory pool. It also configures transfer priority policies and cache coherence protocol parameters (based on CXL.cache). The transfer priority policy ensures the proper allocation of bandwidth and resources when multiple data transfer requests are made simultaneously, while the cache coherence protocol parameters ensure data consistency across caches during data transfers and accesses. Once configured, the second DMA controller can locally establish a mapping between the video memory address space and the shared memory pool address space.

[0071] In T6, the node's CPU monitors and analyzes the status and load of each GPU box in real time based on the current computing task and data processing requirements. For example, in a model-parallel computing task, different GPUs are responsible for different parts of the model. When a GPU needs to exchange intermediate calculation results with other GPUs, data sharing becomes necessary.

[0072] Based on the analysis results, the node's CPU determines the source and target GPU boxes from all GPU boxes on the node (for example, the first and second GPU boxes). This determination is based on factors such as the GPU's computing task progress, video memory usage, and data processing capabilities. For example, if GPUA has completed a portion of its computing task and needs to send the results to GPUB for further processing, GPU Box A, where GPUA is located, is the source GPU box, and GPU Box B, where GPUB is located, is the target GPU box.

[0073] In T7, when a source GPU (e.g., GPUA) needs to send data to a target GPU (e.g., GPUB), the source GPU first prepares the shared data to be sent in its local video memory. For example, after completing a specific computational task, the source GPU can mark data such as intermediate or final results as data to be shared. The source GPU's box's second DMA controller generates a data transfer request based on the correspondence between its locally constructed video memory address space and the shared memory pool address space. This request includes information such as the source address (the address of the data in the source GPU's local video memory), the destination address (the address of a pre-allocated area in the shared memory pool allocated to the target GPU), and the data length.

[0074] The second DMA controller sends data transfer requests and shared data to the shared memory pool via the internal CXL switch and the bus CXL switch. During the transfer process, the internal CXL switch and the bus CXL switch route and forward the shared data and data transfer requests according to pre-configured routing information, ensuring that the shared data arrives accurately and securely at the designated location in the shared memory pool. Simultaneously, the second DMA controller triggers a cache flush to ensure data consistency. This means that any stale data in the cache is flushed to the video memory before data is transferred, ensuring that the data being sent is up-to-date.

[0075] After receiving the data transfer request and shared data from the source GPU BOX, the shared memory pool parses the request according to the management mechanism, including checking the legitimacy and integrity of information such as the source address, destination address, and data length. Based on the destination address in the data transfer request, the shared memory pool writes the received shared data into the designated allocation area. During the writing process, the shared memory pool operates according to the cache consistency protocol to ensure data consistency. For example, if multiple devices access the shared memory pool simultaneously, the shared memory pool uses a lock mechanism or other synchronization mechanism to ensure that the data is correctly written. At the same time, the shared memory pool's management system records the location of the data and related metadata so that other GPUs can accurately read the data.

[0076] In T8, the target GPU BOX's second DMA controller maintains a connection to the shared memory pool via an internal CXL switch and a bus CXL switch. It continuously monitors the designated allocation area in the shared memory pool and obtains real-time status information for that area. When the second DMA controller detects the arrival of new data in the designated allocation area, it generates a data read request based on the mapping between the locally constructed video memory address space and the shared memory pool address space. This request includes information such as the source address (the address of the designated allocation area in the shared memory pool) and the destination address (the address in the target GPU's local video memory used to store the data).

[0077] The second DMA controller sends a data read request to the shared memory pool through the internal CXL switch and the bus CXL switch. After receiving the request, the shared memory pool reads the shared data from the specified allocation area according to the source address in the request and returns it to the second DMA controller of the target GPU BOX through the CXL link. After receiving the shared data, the second DMA controller writes it to the local video memory of the target GPU.

[0078] Example 3: In scenarios such as large-scale AI training, complex scientific computing simulations, and real-time data analysis in ultra-large data centers, GPU data interaction between multiple nodes is often involved. However, in traditional methods, GPU data interaction requires CPU transfer and coordination, which places a heavy burden on the CPU, has limited bandwidth and processing speed, and increases data transmission latency, resulting in low efficiency of GPU interaction across nodes.

[0079] In order to solve the above technical problems, the present disclosure also provides a GPU high-speed interconnection system and a data interaction method thereof. Figure 1-3 Based on the reference Figure 7 Schematic diagram of the GPU high-speed interconnection system structure (for the sake of simplicity, the second GPU BOX and the fourth GPU BOX are Figure 7 (Not shown in the figure), the GPU high-speed interconnection system includes at least a first GPU server, a second GPU server, and a network switch. The first GPU server includes at least a first GPU Box, a second GPU Box, and a first node; the second GPU server includes at least a third GPU Box, a fourth GPU Box, and a second node. The first and second GPU Boxes are hot-swappable on the first node, while the third and fourth GPU Boxes are hot-swappable on the second node. Each GPU Box also includes a network card and a third DMA controller. The third DMA controller is typically built into the network card. The network card is used to connect the GPU Box to the network, and the third DMA controller is used to control data transmission between the GPU Boxes on the network.

[0080] Each node includes a network layer interface, which is connected to a network card and a network switch. The network card is physically connected to an internal PCIe switch. Based on this connection structure, a GPU in any GPU Box in the first GPU server establishes a GDR connection with a GPU in any GPU Box in the second GPU server via the system controller, the second DMA controller, the internal PCIe switch, the third DMA controller, the network card, the network layer interfaces of the first and second nodes, and the network switch. GDR allows RDMA-based network cards to bypass the CPU and host memory and directly access GPU video memory for high-speed data transfer. This enables RDMA-supported network devices to directly initiate DMA accesses on the system bus and directly read and write GPU video memory (video memory addresses are registered and mapped), achieving zero-copy data transmission with low latency. Therefore, the network card (NIC) must support the RDMA protocol, such as an RDMA NIC (Remote Direct Memory Access Network Interface Card). The network switch must also support the RDMA protocol, such as an RDMA Switch (Remote Direct Memory Access Switch). The network layer interface must use an Octal Small Form-factor Pluggable (OSFP) interface, which is integrated into the node's CPU control board. Based on the internal connections within the GPU Box, the RDMA NIC in the GPU Box transmits network signals to the network layer interface connected to the node's CPU control board via the BP connector on the control board. Inter-node GPU interconnection is achieved through the RDMA Switch.

[0081] Reference Figure 8 The flow chart of the data interaction method of the GPU high-speed interconnection system is shown. This method can be applied to Figure 7 The GPU high-speed interconnection system shown in FIG. 1 includes, but is not limited to, the following steps: R1: Based on the cross-server data transmission requirements, the source node and target node are determined from the first node and the second node. The sending end is determined from the GPU box of the source node, and the receiving end is determined from the GPU box of the target node. R2: The system controller at the sending end receives the data preparation instruction; R3: The system controller at the receiving end receives the data receiving instruction; R4: The system controller at the transmitting end performs data preparation initialization on the second DMA controller and the third DMA controller at the transmitting end based on the data preparation instruction; R5: The system controller at the receiving end initializes the second DMA controller and the third DMA controller at the receiving end for data reception based on the data reception instruction; R6: After data preparation and initialization, the second DMA controller in the sending end reads the operation data from the corresponding GPU and transmits the operation data to the GPU of the receiving end through the established GDR connection.

[0082] Taking the cross-node data storage scenario of GPUs for deep learning training as an example, in R1, the system identifies specific data transmission requirements, such as the type of data to be transferred, the amount of data, and the timeliness of the transfer, based on cross-node data transmission tasks such as distributed training and data migration. The source and target nodes are determined based on factors such as node load, the physical location of data storage, task scheduling strategies, data storage location, and the distribution of computing tasks.

[0083] Based on information such as GPU usage status and data cache status, the first and second GPU boxes of the source node determine the sending end according to the source of the data. Similarly, the third and fourth GPU boxes of the target node determine the receiving end according to the final destination of the data.

[0084] In R2-R3, the source node's CPU generates and sends a data preparation instruction based on the data transmission requirements. This instruction contains detailed information about the data transmission, such as the starting address of the data to be transmitted on the sending-end GPU, the data length, and the identifier of the receiving end. Using the inter-board connector, the sending-end system controller receives the data preparation instruction sent by the source node's CPU and parses and verifies the instruction to ensure its integrity and legitimacy. Similarly, the destination node's CPU generates a data reception instruction based on the data transmission task and sends it to the receiving-end system controller. This instruction contains relevant information about the received data, such as the storage address and data length of the received data on the receiving-end GPU.

[0085] In R4-R5, the system controller on the sending end parses the data preparation instructions in detail and extracts key information related to data transmission. Based on this information, the system controller sends an initialization command to the second DMA controller to configure its relevant parameters, including the source address (the starting address of the data in the sending GPU), the destination address (the interface address of the internal PCIe switch), and the data length. Similarly, the system controller sends an initialization command to the third DMA controller to configure its parameters. For the third DMA controller, the source address is the interface address of the internal PCIe switch, and the destination address is the interface address of the network card, so that data can be transferred from the internal PCIe switch to the network card. Through this method, the system controller controls the second and third DMA controllers to map their own addresses to the PCIe bus, enabling communication between the GPU, the internal PCIe switch, and the network card via the PCIe bus, achieving direct access to data.

[0086] Similarly, the system controller at the receiving end parses the data reception instruction and obtains relevant information about the received data. Based on this information, the system controller sends an initialization command to the third DMA controller, configuring its parameters. The source address is the interface address of the network card, and the destination address is the interface address of the internal PCIe switch. This is used to transfer data from the network card to the internal PCIe switch. The system controller sends an initialization command to the second DMA controller to configure its relevant parameters. The source address is the interface address of the internal PCIe switch, and the destination address is the address used to store data in the receiving end GPU.

[0087] In R6, after initialization, the second and third DMA controllers on the sending and receiving ends work together to establish a GDR connection between the sending GPU and the receiving GPU through components such as the system bus, network card, network layer interface, and network switch.

[0088] The second DMA controller on the sending end, initialized after data preparation, reads the computational data from the corresponding GPU according to the configured parameters and transmits the read computational data to the sending end's internal PCIe switch. The third DMA controller on the sending end, initialized after data preparation, transfers the computational data from the internal PCIe switch to the sending end's network interface card, and then transmits the data from the network interface card to the network switch through the source node's network layer interface.

[0089] The network switch correctly routes the computational data to the network layer interface of the target node based on the target address. Through the network layer interface of the target node, the third DMA controller at the receiving end, which has been initialized by data reception, performs protocol processing and flow control on the computational data, transferring the computational data from the receiving end's network card to the internal PCIe switch. The second DMA controller, which has also been initialized by data reception, then writes the computational data to the receiving end's GPU. Throughout the data transmission process, due to the establishment of a GDR connection, data can be transferred directly between GPUs. The third DMA controller built into the network card takes over control of the data transmission, reducing CPU interrupts and context switching during the data transmission process and lowering the CPU load. The third DMA controller also assists the network card in processing the computational data, encapsulating the computational data and performing other operations to adapt it to network transmission requirements. The computational data is transmitted to the network switch via the network layer interface of the source node. The network switch is responsible for forwarding the data to the target node, greatly improving the speed and efficiency of data transmission.

[0090] Based on the connection structure between the above components, this embodiment can also realize GPU cross-node access to internal storage devices. The method includes: based on the cross-node internal storage device access requirements, determining the initiator node and the responder node, and determining the initiator from the GPU BOX of the initiator node, and determining the responder from the GPU BOX of the responder node. The CPU of the initiator node sends a storage device access instruction to the system controller of the initiator. After receiving the instruction, the system controller of the initiator parses the instruction and clarifies the detailed information required to access the storage device. Based on the storage device access instruction, the system controller initializes the second DMA controller and the third DMA controller of the initiator to read data, preparing for the subsequent reading of specified data from the GPU of the initiator. At the same time, the CPU of the responder node sends a storage device response instruction to the system controller of the responder. Based on the instruction, the system controller of the responder initializes the first DMA controller and the third DMA controller of the responder to write data, so as to facilitate the subsequent introduction of specified data into the specified internal storage device.

[0091] The second and third DMA controllers, initialized for data reading, and the third DMA controller, initialized for data writing, read the designated data related to storage device access from the corresponding GPU based on the GDS link and transmit it to the internal PCIe switch of the responding end. The first DMA controller, initialized for data writing, writes the designated data to the corresponding internal storage device of the responding end based on the GDS connection, thus completing the cross-node internal storage device access process. Conversely, via this transmission path, the GPU can write its own data to the internal storage device across nodes.

[0092] In summary, this solution comprehensively covers data access scenarios within a GPU Box, between GPU Boxes within the same node, and across nodes. Within a GPU Box, the system controller controls the DMA controller to enable efficient GPU access to internal storage devices, shortening data read times and improving processing efficiency. Within a node, cross-GPU Box interaction is enabled by leveraging the GPU Box and the node's PCIe switch, with DMA controllers enabling more efficient data flow. Across nodes, clear initiator and responder instructions and multi-controller initialization are used, combining PCIe and network switches to transmit data. This enables reliable interaction between GPUs across nodes and between GPUs and remote internal storage devices, reducing interference, enhancing system scalability and flexibility, and meeting the diverse storage resource requirements of complex computations. The system controller not only replaces the node CPU for data transfer control but also monitors the execution of each process and controls interrupts, thereby improving operational efficiency and data processing accuracy.

[0093] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0094] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0095] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0096] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0097] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0098] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0099] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A GPU BOX high-speed interconnection method, characterized in that: The GPU BOX includes a GPU and a control mainboard, wherein the control mainboard is internally integrated with an internal PCIe switch, an internal storage device, a first DMA controller and a system controller, and the GPU is internally integrated with a second DMA controller; The system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

2. A method for high-speed interconnection within a GPU server, characterized in that: The server is configured with at least a first GPU BOX, a second GPU BOX, and a node, wherein the node includes a bus PCIe switch, and each GPU BOX includes a GPU, a second DMA controller, and a control motherboard, wherein the control motherboard integrates an internal PCIe switch and a system controller; The system controller of the first GPU BOX receives the initialization configuration instruction, and performs data interaction initialization on the second DMA controller based on the initialization configuration instruction; Determine a source end and a target end from the first GPU BOX and the second GPU BOX based on the data transmission requirements of the node; The second DMA controller in the source end, after data exchange initialization, receives a data read / write instruction, and based on the data read / write instruction, reads the video memory data from the GPU of the source end, and transmits the video memory data to the GPU of the target end via the bus PCIe switch through the established GDP2P connection; A GDP2P connection is established between the GPUs in the first GPU BOX and the second GPU BOX through the system controller, the second DMA controller, the internal PCIe switch, and the CPU control board of the first GPU BOX and the second GPU BOX.

3. The method for high-speed interconnection within a GPU server according to claim 2, wherein: Each of the GPU BOXs further includes an internal CXL switch, and the node further includes a bus CXL switch and a shared memory pool; The second DMA controller receives the memory configuration instruction, and maps the video memory of the corresponding GPU to the shared memory pool of the node through the established CXL connection according to the memory configuration instruction, and establishes a correspondence between the video memory address space and the shared memory pool address space; The GPU of the GPU BOX, the internal CXL switch, the bus CXL switch, and the shared memory pool are sequentially connected via CXL; Determine the source GPU BOX and target GPU BOX based on data sharing requirements; The second DMA controller of the source GPU BOX generates a data transfer request based on the correspondence between the video memory address space of the corresponding GPU and the address space of the shared memory pool, and sends the data transfer request and the shared data of the corresponding GPU to the shared memory pool; The shared memory pool writes the shared data into a designated allocation area based on the data transmission request; The second DMA controller of the target GPU BOX reads the shared data from the designated allocation area and writes the shared data into the video memory of the corresponding GPU based on the correspondence between the video memory address space of the corresponding GPU and the shared memory pool address space.

4. The method for high-speed interconnection within a GPU server according to claim 2, wherein: Each of the GPU BOXs further includes an internal storage device and a first DMA controller; In the same GPU BOX, the system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

5. A data interaction method for a GPU high-speed interconnection system, characterized in that: The system includes at least a first GPU server, a second GPU server, and a network switch. The first GPU server includes at least a first GPU BOX, a second GPU BOX, and a first node. The second GPU server includes at least a third GPU BOX, a fourth GPU BOX, and a second node. Each of the GPU BOXs includes a GPU, a second DMA controller, a network card, a third DMA controller, and a control mainboard, wherein the control mainboard integrates an internal PCIe switch and a system controller; Based on cross-server data transmission requirements, determine a source node and a target node from the first node and the second node, determine a sending end from the GPU box of the source node, and determine a receiving end from the GPU box of the target node; The system controller at the transmitting end receives a data preparation instruction; the system controller at the receiving end receives a data reception instruction; The system controller of the transmitting end performs data preparation initialization on the second DMA controller and the third DMA controller of the transmitting end based on the data preparation instruction; the system controller of the receiving end performs data reception initialization on the second DMA controller and the third DMA controller of the receiving end based on the data reception instruction; The second DMA controller in the transmitting end, after data preparation and initialization, reads the operation data from the corresponding GPU and transmits the operation data to the GPU of the receiving end through the established GDR connection; The GPU of any GPU BOX in the first GPU server establishes a GDR connection with the GPU of any GPU BOX in the second GPU server through the system controller, the second DMA controller, the internal PCIe switch, the third DMA controller, the network card, the network layer interface and the network switch.

6. The data interaction method of the GPU high-speed interconnection system according to claim 5, characterized in that: Each of the GPUBOXs further includes an internal storage device and a first DMA controller; In the same GPU BOX, the system controller receives a read instruction and initializes data reading and writing for the first DMA controller and the second DMA controller based on the read instruction; The first DMA controller after data reading and writing initialization reads the cached data from the internal storage device, and transmits the cached data to the GPU via the internal PCIe switch through the established GDS connection; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

7. A control system for controlling a GPU, characterized in that: The control system includes a control mainboard, an internal storage device, a first DMA controller, a PCIe switch and a system controller; the internal storage device, the first DMA controller, the internal PCIe switch and the system controller are integrated on the control mainboard; The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The first DMA controller is used to control the reading and writing of data of the internal storage device; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch.

8. A GPU BOX, characterized in that: The GPU BOX includes a GPU and a control mainboard, wherein the control mainboard is internally integrated with an internal PCIe switch, an internal storage device, a first DMA controller and a system controller, and the GPU is internally integrated with a second DMA controller; The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The first DMA controller is used to control the reading and writing of data of the internal storage device; The second DMA controller is used to control data reading and writing of the GPU; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch; A GDS connection is established between the GPU and the internal storage device through the system controller, the second DMA controller, the internal PCIe switch, and the first DMA controller.

9. The GPU BOX according to claim 8, wherein: A GDP2P connection is established between GPUs in any two GPU BOXs on the same node through the system controller, the second DMA controller and the internal PCIe switch of the GPU BOX.

10. The GPU BOX according to claim 8, wherein: The GPU BOX also includes an internal CXL switch; The internal CXL switch is used to control resource sharing between the GPU and peripheral components.

11. The GPU BOX according to claim 8, wherein: The GPU BOX also includes a network card and a third DMA controller; The network card is used to connect the GPU BOX to the network; The third DMA controller is used to control data transmission of the GPU BOX in the network.

12. A GPU server, characterized in that: The server is configured with at least a first GPU BOX, a second GPU BOX, and a node, wherein the node includes a bus PCIe switch, and each GPU BOX includes a GPU, a second DMA controller, and a control motherboard, wherein the control motherboard integrates an internal PCIe switch and a system controller; The control mainboard is used to control the communication between the internal components and the CPU control board of the node; The second DMA controller is used to control data reading and writing of the GPU; The internal PCIe switch is used to control data transmission between connected components; The system controller is used to control data interaction and resource allocation of components connected to the internal PCIe switch; A GDP2P connection is established between the GPUs in the first GPU BOX and the second GPU BOX through the system controllers, second DMA controllers, internal PCIe switches, and the CPU control board of the first GPU BOX and the second GPU BOX.

13. The GPU server according to claim 12, wherein: Each of the GPU BOXs further includes an internal CXL switch, and the node further includes a bus CXL switch and a shared memory pool; The GPU of the GPU BOX, the internal CXL switch, the bus CXL switch, and the shared memory pool are connected in sequence through CXL.

14. The GPU server according to claim 12, wherein: Each of the GPU BOXs further includes an internal storage device and a first DMA controller; A GDS connection between the GPU in the GPU BOX and the internal storage device is established through the system controller of the GPU BOX, the second DMA controller, the internal PCIe switch and the first DMA controller.

15. A GPU high-speed interconnection system, characterized in that: The system includes at least a first GPU server, a second GPU server, and a network switch. The first GPU server includes at least a first GPU BOX, a second GPU BOX, and a first node. The second GPU server includes at least a third GPU BOX, a fourth GPU BOX, and a second node. Each of the GPU BOXs includes a GPU, a second DMA controller, a network card, a third DMA controller, and a control mainboard, wherein the control mainboard integrates an internal PCIe switch and a system controller; Each node includes a network layer interface, and the network layer interface is connected to the network card and the network switch respectively; A GPU of any GPU BOX in the first GPU server establishes a GDR connection with a GPU of any GPU BOX in the second GPU server through the system controller, the second DMA controller, the internal PCIe switch, the third DMA controller, the network card, the network layer interface and the network switch.

16. The GPU high-speed interconnection system according to claim 15, characterized in that: Each of the GPU BOXs further includes an internal storage device and a first DMA controller; A GDS connection between the GPU in the GPU BOX and the internal storage device is established through the system controller of the GPU BOX, the second DMA controller, the internal PCIe switch and the first DMA controller.

Images