A three-stage AGC intelligent computing architecture system based on PCIe protocol

Abstract

This application discloses a three-segment AGC intelligent computing architecture system based on the PCIe protocol, including a data interaction GPU unit, an external PCIe interconnect module, an RDMA switch module, and a CPU unit. Within the data interaction GPU unit, a P2P DMA-enabled channel is established between the GPU module and the storage module. Each data interaction GPU unit on the same node is connected to the external PCIe interconnect module. A GPUDirect P2P connection channel is established between the GPU modules of any two data interaction GPU units on the same node. A GPUDirect RDMA connection channel is established between the GPU modules of any two data interaction GPU units on different nodes. The CPU unit is not co-located with the data interaction GPU unit and has a hot-swappable connection channel with it. This system enables decoupling from the CPU and high-speed data interaction.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a three-stage AGC intelligent computing architecture system based on the PCIe protocol. Background Technology

[0002] In the traditional field of intelligent computing servers, the ACC (AI computer system with the CPU at its core) architecture design, which is generally adopted with the CPU as the core, is commonly used. This design pattern requires that all components in the chassis must be CPU-centric and follow a unified PCIe protocol standard. This design pattern has obvious technical limitations: On the one hand, the number of PCIe lanes provided by the CPU is limited. If all devices are forced to follow a unified standard, it will lead to a shortage of lane allocation. For example, if a CPU only supports 128 PCIe 5.0 lanes, 8 PCIe 5.0 x16 GPUs will occupy all 128 lanes, leaving other devices such as SSDs and network cards with no lanes available. On the other hand, when low-speed peripherals such as USB controllers and low-speed network cards use high-version PCIe interfaces, the actual bandwidth utilization is extremely low, resulting in a waste of physical layer resources.

[0003] From a cost perspective, uniformly adopting the latest PCIe standard (such as PCIe 5.0 / 6.0) will significantly increase the cost of components (such as GPUs, SSDs, and network cards), especially for low-bandwidth devices (such as SATA controllers) that do not require high-speed interfaces. Forced uniformity will result in wasted resources; for example, the cost of a PCIe 5.0 x4 network card (16 GB / s bandwidth) is far higher than that of a 1GbE network card that only requires 1 Gb / s bandwidth. In terms of compatibility, if a unified standard is enforced, existing PCIe 3.0 / 4.0 devices will not be directly compatible, leading to increased hardware iteration costs and hindering smooth upgrades. Higher PCIe protocol versions have higher signal frequencies, placing more stringent requirements on PCB trace length, impedance matching, and electromagnetic interference. Uniform adoption of higher versions will exacerbate problems such as the need for repeaters for long-distance traces, increasing costs, and high-density wiring leading to increased crosstalk risk and higher bit error rates.

[0004] Meanwhile, PCIe version upgrades are accompanied by increased power consumption. For example, PCIe 5.0 devices consume about 30% more power than 4.0 devices, significantly increasing the overall system's heat dissipation pressure after standardization. Data shows that PCIe 4.0 x16 slots consume 25-30W, while PCIe 5.0 x16 slots consume 35-45W. Furthermore, low-speed devices operating in higher PCIe versions experience a significant increase in energy consumption per unit data throughput (J / GB), violating green computing principles. Utility Model Content

[0005] In view of this, the present disclosure provides a three-segment AGC intelligent computing architecture system based on the PCIe protocol, which at least partially solves the problems of performance and resource waste, high hardware cost, poor compatibility, complex topology and high power consumption in the prior art.

[0006] This disclosure provides a three-stage AGC intelligent computing architecture system based on the PCIe protocol, including:

[0007] The data interaction GPU unit includes an internal PCIe interconnect module, and a GPU module, a control module, a storage module, and a network module, all of which are connected to the internal PCIe interconnect module via a first type of PCIe protocol. A P2P DMA-enabled channel is established between the GPU module and the storage module.

[0008] An external PCIe interconnect module is provided, and each of the data interaction GPU units on the same node is connected to the external PCIe interconnect module via the second type of PCIe protocol; a GPUDirect P2P connection channel is established between the GPU modules of each pair of data interaction GPU units on the same node.

[0009] The RDMA switch module is connected to each of the data interaction GPU units on different nodes via the third type of PCIe protocol; a GPUDirect RDMA connection channel is established between the GPU modules of each pair of data interaction GPU units on different nodes.

[0010] The CPU unit is not on the same board as the data interaction GPU unit, and is connected to the data interaction GPU unit via the fourth type PCIe protocol; a hot-swappable connection channel is established between the data interaction GPU unit and the CPU unit.

[0011] Optionally, the internal PCIe interconnect module, the GPU module, the control module, the storage module, and the network module in each data interaction GPU unit are all integrated and packaged.

[0012] Each of the aforementioned data interaction GPU units is configured independently.

[0013] Optionally, the CPU unit includes a plurality of servers, each server being connected to one or more of the data interaction GPU units; each of the data interaction GPU units is an independent hot-swappable unit;

[0014] The network module in the data interaction GPU unit is connected to the corresponding OSFP interface in a single server, and the OSFP interfaces connected to different data interaction GPU units are all different.

[0015] Optionally, the version of the second type of PCIe protocol is lower than the version of the first type of PCIe protocol;

[0016] The version of the third type of PCIe protocol is the same as the version of the first type of PCIe protocol.

[0017] The version of the fourth type of PCIe protocol is no higher than the version of the second type of PCIe protocol.

[0018] Optionally, the first type of PCIe protocol is the PCIe-Gen5 / 6 protocol;

[0019] The second type of PCIe protocol is the PCIe-Gen4 / 5 protocol;

[0020] The fourth type of PCIe protocol is the PCIe-GEN3 / 4 / 5 protocol.

[0021] Optionally, the transmission rate of the channel supporting P2P DMA is V1, where 100Gb / s≤V1≤400Gb / s.

[0022] Optionally, the transmission rate of the GPUDirect RDMA connection channel is V2;

[0023] 2Tb / s≤V2≤8TbGb / s.

[0024] Optionally, the P2P DMA-supporting channel, the GPUDirect P2P connection channel, and the GPUDirectRDMA connection channel do not occupy CPU lanes.

[0025] Optionally, the GPU module integrates a DMA controller.

[0026] The storage module integrates a DMA controller.

[0027] The channel supporting P2P DMA is the transmission channel between the DMA controller inside the GPU module and the DMA controller inside the storage module.

[0028] Optionally, the storage module is an NVMe SSD;

[0029] Both the internal PCIe interconnect module and the external PCIe interconnect module are PCIe switch chips.

[0030] The control module is an integrated system-on-a-chip;

[0031] The network module is an RDMA network card.

[0032] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a three-segment AGC intelligent computing architecture system based on the PCIe protocol, provided in an embodiment of this disclosure.

[0035] Figure 2 for Figure 1 A schematic diagram of the data interaction GPU unit in the diagram. Detailed Implementation

[0036] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0037] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0038] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0039] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0040] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0041] For descriptive purposes, this disclosure may use spatial relative terms such as “below,” “under,” “below,” “down,” “above,” “above,” “higher,” and “side (e.g., in a “sidewall”)” to describe the relationship between one component and another component as shown in the accompanying drawings. In addition to the orientations depicted in the drawings, the spatial relative terms are also intended to encompass different orientations of the device during use, operation, and / or manufacture. For example, if the device in the drawings is flipped, a component described as “below” or “under” another component or feature would subsequently be positioned “above” said other component or feature. Thus, the exemplary term “below” can encompass both “above” and “below” orientations. Furthermore, the device may be otherwise positioned (e.g., rotated 90 degrees or in other orientations), thus interpreting the spatial relative descriptive terms used herein accordingly.

[0042] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0043] Reference Figure 1 and Figure 2 This application discloses a three-segment AGC intelligent computing architecture system based on the PCIe protocol, including a data interaction GPU unit, an external PCIe interconnect module, an RDMA switch module, and a CPU unit. The data interaction GPU unit includes a GPU module, a control module, a storage module, a network module, and an internal PCIe interconnect module. The GPU module, control module, storage module, and network module are all connected to the internal PCIe interconnect module through the first type of PCIe protocol. A P2P DMA channel is established between the GPU module and the storage module. This channel is used for direct access between the storage device and the video memory in the GPU module, and supports direct read and write access to the disk.

[0044] In this embodiment, the storage module is preferably an NVMe SSD, and the NVMe SSD interface is preferably an Edge 1.S interface, which can meet the data transmission requirements of high-performance storage devices, thereby improving the overall performance and efficiency of the data interaction GPU unit; the internal PCIe interconnect module and the external PCIe interconnect module are both preferably PCIe switch chips; the network module is preferably an RDMA network card, which can realize high-speed, low-latency network data transmission; in a distributed computing environment, the RDMA network card can significantly improve the data exchange efficiency between nodes.

[0045] Compared to traditional data access methods where data is first transferred from the disk to system memory and then processed and scheduled by the CPU to move the data to the GPU memory, the data interaction GPU unit disclosed in this embodiment provides a direct access method that can significantly reduce data transmission latency and improve data processing efficiency. This solution bypasses the CPU and enables direct communication between the GPU and storage devices, greatly improving the efficiency and speed of data transmission.

[0046] Specifically, during deep learning training, GPUs need to frequently read large amounts of training data from storage devices. Through the P2P DMA-enabled channel established in this application, GPUs can quickly acquire data, accelerating training speed and shortening model training time. In high-performance computing fields such as scientific computing and simulation, large amounts of data need to be processed. The established P2P DMA-enabled channel can improve data transfer efficiency, enabling GPUs to perform calculations more efficiently and improving computational performance. For data-intensive applications such as video processing and image analysis, the established P2P DMA-enabled channel can accelerate data read and write speeds, improving application processing power and response speed.

[0047] In each data interaction GPU unit, the internal PCIe interconnect module, GPU module, control module, storage module, and network module are all integrated and packaged, preferably within a GPU BOX. This GPU BOX is installed on the server via a server expansion backplane. The installed GPU BOX and the server form an intelligent computing server, effectively shortening the physical distance between components, reducing signal transmission path length, lowering signal latency and interference, improving data transmission stability and speed, and ensuring high-speed and accurate data interaction between modules, thereby enhancing the overall computing performance of the GPU unit. After the modules are integrated and packaged, communication between modules is more efficient, reducing unnecessary energy consumption. At the same time, unified power management is more convenient, intelligently adjusting power consumption according to the GPU unit's operating status, thereby reducing overall energy consumption and improving energy utilization efficiency.

[0048] In this embodiment, each data interaction GPU unit is independently configured. These independently configured GPU units function like standardized components; during server or computing system deployment, they simply need to be installed onto the corresponding interface, eliminating the need for complex wiring and debugging. This plug-and-play feature significantly shortens system deployment time and improves efficiency. When computing demands increase, independent data interaction GPU units can be easily added to enhance the system's computing power. This can be achieved easily, whether increasing the number of GPU units within a single server or expanding within a cluster of multiple servers, without requiring large-scale modifications to the existing system. Because each data interaction GPU unit is independently configured, a failure in one unit does not affect the normal operation of other units. The system can quickly identify and isolate the faulty unit, continuing to utilize other functioning GPU units to complete computing tasks, thereby improving the overall system reliability and availability and reducing downtime due to hardware failures. The independently configured data interaction GPU units also offer greater operability during maintenance. If a unit malfunctions, it can be directly removed from the system for repair or replacement without affecting other units or the normal operation of the entire system. This makes maintenance simpler, more efficient, and reduces maintenance costs and difficulty. Since each GPU unit is independent, its performance can be monitored and managed individually. Through monitoring software, information such as the working status and performance indicators of each unit can be obtained in real time, so as to promptly identify potential problems and make adjustments and optimizations to ensure that the entire system is always in the best operating state.

[0049] In this embodiment, the CPU unit may include several servers, each server connected to one or more data interaction GPU units; each data interaction GPU unit is an independent hot-swappable unit; when at least two data interaction GPU units are installed on the same server, a single-server multi-BOX mode is formed. Through the solution provided in this application, users can flexibly adjust GPU resources according to actual business needs. For example, when performing large-scale deep learning training, more GPU units can be added to the server to enhance parallel computing capabilities; when the workload is small, the number of connected GPU units can be reduced to lower energy consumption and costs.

[0050] The network module in the data interaction GPU unit is connected to the corresponding OSFP interface in a single server. Different data interaction GPU units are connected to different OSFP interfaces, and the network module in each GPU BOX has its own unique OSFP interface on the corresponding server.

[0051] The OSFP interface features high bandwidth and low latency, meeting the high-speed data transmission requirements between GPU units and servers. Different data interaction GPU units connect to different OSFP interfaces, effectively avoiding network conflicts and bandwidth contention. Each GPU unit can independently utilize the bandwidth resources of one OSFP interface, ensuring data transmission stability and reliability, and improving overall system performance. Because each GPU unit connects independently to a different OSFP interface on the server, independent network monitoring and management are possible for each unit. Maintenance personnel can understand the network usage of each GPU unit in real time, promptly identify and resolve potential network problems such as bandwidth bottlenecks and network congestion. When a network failure occurs in a GPU unit, the independence of the interface allows for rapid location of the faulty unit, isolating it from other normal units, reducing the impact of the failure on the entire system, and facilitating troubleshooting and repair, thus improving maintenance efficiency.

[0052] Each data interaction GPU unit on the same node is connected to an external PCIe interconnect module via the Type II PCIe protocol; the version of the Type II PCIe protocol is lower than the version of the Type I PCIe protocol; GPU modules in the data interaction GPU units on the same node establish a GPUDirect P2P connection channel, enabling GPUs to directly use the PCIe bus for P2P DMA communication.

[0053] Specifically, the GPU module integrates a DMA controller connected to the GPU card, used to control data read / write operations on the GPU card and data interaction with the PCIe switch chip. The storage module integrates a DMA controller connected to the storage chip, used to control data read / write operations on the storage chip and data interaction with the PCIe switch chip. The P2PDMA supported channel is the transmission channel between the DMA controllers within the GPU module and the storage module.

[0054] Furthermore, the control module is preferably an integrated system-on-a-chip, i.e., SoC, which is responsible for the initialization and configuration of the DMA controller inside the GPU module and the DMA controller inside the storage module. It can support P2PDMA (i.e., point-to-point direct memory access) without the participation of the server CPU, that is, it can directly read data from the storage module without passing through the server CPU and memory.

[0055] Each data interaction GPU unit on different nodes is connected to the RDMA switch module via the Type 3 PCIe protocol; the version of the Type 3 PCIe protocol is consistent with the version of the Type 1 PCIe protocol; GPU modules in the data interaction GPU units on different nodes establish GPUDirect RDMA (i.e., GDR) connection channels, and cross-node GPU interconnection is realized through the RDMA switch module.

[0056] In this design, the CPU unit (i.e., the server) and the data interaction GPU unit are not on the same board, and they are connected via Type 4 PCIe protocol, with the Type 4 PCIe protocol version no higher than that of Type 2 PCIe protocol. Specifically, the CPU unit manages the GPU module, network module, and storage module through the PCIe bus. This non-shared-board structure eliminates the dependence on PCIe interface switching devices and retimers, among other components.

[0057] A hot-swappable connection channel is established between the data interaction GPU unit and the CPU unit. The data interaction GPU unit is integrated within the GPU BOX and connects to the server via a hot-swappable connection channel. This channel is a dynamically pluggable interface link based on a specific high-speed data transmission protocol. This channel provides high-speed, low-latency data transmission capabilities, enabling real-time data interaction and command transmission between the data interaction GPU unit and the CPU unit. Furthermore, the data interaction GPU unit can be plugged in or unplugged at any time during normal server operation, ensuring no electrical damage to the server system or the data interaction GPU unit during hot-swapping, thus guaranteeing system stability and data integrity. This channel employs a reliable interface design to ensure stable data transmission and prevent signal interference and electrical problems during plugging and unplugging.

[0058] In this embodiment, the CPU unit is only responsible for issuing initialization configuration instructions, transmission control, and monitoring tasks. In this architecture, tasks originally handled by the server CPU are now handled by the SoC chip within the data interaction GPU unit, eliminating the need for the server CPU and memory. The internal PCIe interconnect module primarily functions as a data routing and switching mechanism, ensuring accurate data transfer between the storage module and the GPU module. Therefore, in this simple one-to-one connection scenario, the DMA controllers of the storage module and GPU module are mainly involved in the P2P DMA transfer process. Thus, through the internal settings of the data interaction GPU unit, users and applications can be provided with the ability to directly transfer data from the storage device to the GPU memory. The preferred PCIe protocol is PCIe-GEN3 / 4 / 5.

[0059] In this embodiment, the channels supporting P2P DMA, GPUDirect P2P connection channels, and GPUDirect RDMA connection channels do not occupy CPU lanes, meaning they do not occupy the physical channels for data transmission between the CPU and other devices, i.e., the PCI-Express (PCIe) bus channels. The CPU exchanges data with these devices through PCIe channels. Each PCIe channel has a certain bandwidth for data transmission. The more channels there are, the greater the total data transmission bandwidth. However, the number of PCIe channels that the CPU can provide is limited. By setting up channels that do not occupy CPU lanes as described in this application, CPU intervention is effectively reduced, greatly improving the data transmission speed between local devices, while also effectively reducing data transmission latency.

[0060] To maximize the performance of the GPU card and disk, the three-segment AGC intelligent computing architecture system preferentially uses a higher version of PCIe-GEN5 / 6 (i.e., the first type of PCIe protocol) for data transmission in the functional area inside the data interaction GPU unit. Without occupying the CPU lane, it supports each GPU card to establish a link binding with the high-speed network chip and realize direct read and write access to the disk.

[0061] In this embodiment, the transmission rate of the P2P DMA-supporting channel is V1, where 100Gb / s ≤ V1 ≤ 400Gb / s. P2P DMA-supporting channels allow direct memory data transfer between devices without CPU intervention. The higher transmission rate (100Gb / s - 400Gb / s) enables large amounts of data to move rapidly between different devices (such as GPUs, storage devices, network interface cards, etc.), effectively accelerating data transfer between GPUs, reducing waiting time, and thus fully leveraging the parallel computing capabilities of multiple GPUs. The 100Gb / s-400Gb / s transmission rate provides sufficient bandwidth expansion space for the system, allowing it to adapt to the growth in data volume and computing demands over the next few years or even longer. This means that system upgrades and expansions do not require immediate replacement of the entire data transmission architecture, reducing upgrade costs and complexity.

[0062] The GPUDirect RDMA connection channel has a transmission rate of V2: 2Tb / s ≤ V2 ≤ 8TbGb / s. This high transmission rate allows data to move at lightning speed between different GPUs, and between GPUs and storage devices or other computing nodes, enabling rapid data sharing and synchronized computation results, thus enhancing the overall computing power and scalability of the cluster. For example, in a supercomputer cluster, multiple GPU nodes can work collaboratively through high-speed channels to solve more complex scientific computing and engineering simulation problems.

[0063] The application discloses a three-stage AGC intelligent computing architecture system based on the PCIe protocol, which enables communication between GPU cards in different GPU boxes without occupying CPU lanes. Specifically, it includes two scenarios: scenario one is the interaction between different GPUs in the same node (i.e., the same server), and scenario two is the interaction between different GPUs in different nodes (i.e., the same server).

[0064] For scenario one: single-server multi-GPU BOX mode, the constructed GPUDirect P2P connection channel enables communication between different GPUs on the same node (i.e., the same server) (i.e., GPU-to-GPU communication). The following detailed explanation of the data transmission process uses deep learning training as an example. Deep learning training programs (such as TensorFlow and PyTorch) initiate collaborative computation between multiple GPUs. For example, in model parallelism, different GPUs are responsible for different parts of the model, and they need to exchange intermediate computation results to complete the forward and backward propagation of the entire model. Taking the case where GPU BOX A needs to copy data to GPU BOX B as an example, the specific transmission process includes:

[0065] 1) In response to the request instruction (i.e., the request instruction of GPU BOX A to copy data to GPU BOX B), obtain the request information of the request instruction; the request information includes the source (i.e. GPU BOX A), the target (i.e. GPU BOX B), the transfer length, and the transfer mode.

[0066] 2) In response to the initialization configuration instruction, the control module (SoC chip) in the source end (i.e. GPU BOX A) is called to perform the initialization configuration of the DMA controller in the source GPU module and the DMA controller in the source network module.

[0067] 3) In response to the initialization configuration command, the control module (SoC chip) of the target side (i.e. GPU BOX B) is called to perform the initialization configuration of the DMA controller in the target GPU module and the DMA controller in the target network module.

[0068] 4) Establish a GPUDirect P2P connection channel between the DMA controller in the source GPU module and the DMA controller in the target GPU module after initialization by using an external PCIe interconnect module (external PCIe Switch) between the source and target ends.

[0069] 5) In response to the completion signal of the establishment of the GPUDirect P2P connection channel, the source GPU is triggered to execute the target data sending instruction.

[0070] 6) Based on the transmission length and mode, read the target data from the source GPU memory (i.e., read data from the GPU memory via the GPU's DMA controller in GPU BOX A), and transmit the target data to the GPU module in GPU BOX B via the GPUDirect P2P connection channel (i.e., via the PCIe interconnect module in GPU BOX A, the external PCIe interconnect module, and the PCIe interconnect module in GPU BOX B). The DMA controller in the GPU module in GPU BOX B then writes the target data to the specified location in the GPU module's memory. The external PCIe interconnect module is responsible for correctly routing the data to GPU BOX B.

[0071] Each GPU uses the data and model parameters in its memory to perform forward and backward propagation calculations, sending the calculated gradient data directly to the memory of other GPUs through the corresponding external PCIe interconnect module, and at the same time, obtaining gradient data sent by other GPUs from the external PCIe interconnect module. In this process, no CPU is needed for data transfer and processing, achieving efficient gradient synchronization.

[0072] In a single-server multi-GPU BOX mode, GPU-to-GPU communication preferably uses a medium version of the PCIe-GEN4 / 5 protocol (i.e., the second type of PCIe protocol), which can support horizontal topology networking communication for up to 20 GPU cards.

[0073] For scenario two: the multi-server, multi-GPU BOX mode, the constructed GPUDirect RDMA connection channel enables communication between different GPUs on different nodes (i.e., different servers) (i.e., host-to-host cross-node communication). For example, deep learning training programs (such as TensorFlow and PyTorch) initiate collaborative computation across multiple GPUs on different nodes. In model parallelism, different GPUs are responsible for different parts of the model, and they need to exchange intermediate computation results to complete the forward and backward propagation of the entire model. Taking multi-level, multi-GPU distributed computing for deep learning training as an example, specifically when GPU BOX A needs to copy data to GPU BOX B, the specific transmission process includes:

[0074] 1) In response to the request instruction (i.e., the request instruction of GPU BOX A to copy data to GPU BOX B), obtain the request information of the request instruction; the request information includes the source end, the target end, the transmission length, and the transmission mode.

[0075] 2) In response to the initialization configuration instruction, the control module of the source end (i.e., GPU BOX A) is called to perform the initialization configuration of the DMA controller in the source GPU module and the DMA controller in the source network module.

[0076] 3) In response to the initialization configuration command, the control module of the target side (i.e., GPU BOX B) is invoked to perform the initialization configuration of the DMA controller in the target GPU module and the DMA controller in the target network module;

[0077] 4) Establish an RDMA connection channel between the DMA controller in the source network module and the DMA controller in the target network module after initialization through the RDMA switch (i.e., RDMA switch module) between the source and target ends, i.e., GPUDirect RDMA connection channel.

[0078] 5) In response to the completion signal of the RDMA connection channel establishment, the source GPU is triggered to execute the target data transmission command;

[0079] 6) Read the target data from the GPU memory of the source end according to the transmission length and transmission mode (i.e., read the data from the GPU memory of the GPU through the DMA controller of the GPU in GPU BOX A), transmit the target data to the GPU module of the target end through the GPUDirect RDMA connection channel, and write the target data to the specified location of the GPU memory of the target end GPU module through the DMA controller in the GPU module of the target end.

[0080] Specifically, the data is transmitted from the PCIe interconnect module at the source end to the network module at the source end. The OSFP interface connected to the network module determines the first server corresponding to the source end, and the data is transmitted to the RDMA switch through the first server. The second server corresponding to the RDMA switch, which is located on a different node from the source end, is determined, and the target data is transmitted to the network module at the target end through the second server. Then, the target data is written to the specified location of the video memory of the GPU module at the target end through the PCIe interconnect module at the target end.

[0081] Regarding the host-to-host network interconnection technology for multiple intelligent computing servers, a higher version of the PCIe-GEN5 / 6 protocol (i.e., the third type of PCIe protocol) is preferred to achieve high-speed network switching and interconnection of 2Tb-8Tb nodes of a single intelligent computing server.

[0082] Furthermore, the CPU unit in this application is compatible with products from all domestic domestic IT innovation CPU manufacturers, while ensuring that the overall performance of the corresponding intelligent computing server is not affected.

[0083] This application describes the data transfer process within a single-server, single-GPU BOX (i.e., within the data interaction GPU unit) using a scenario where a deep learning training GPU reads data from a storage module (i.e., an NVMe SSD) as an example. The specific transfer methods include:

[0084] 1) In response to a request instruction, i.e., in response to a data read instruction initiated by a deep learning training program (such as TensorFlow or PyTorch), the system obtains the request information of the request instruction. This request information includes the source (i.e., source address) and target (i.e., target address) within the data interaction GPU unit (i.e., within a single GPU box), the transmission length (i.e., the amount of data to be transmitted), and the transmission mode (e.g., single transmission or batch transmission). In this embodiment, the source (i.e., source address) is the storage module (i.e., NVMe SSD), specifically referring to the storage address of the data in the NVMe SSD, and the target (i.e., target address) is the GPU module, specifically referring to the GPU's video memory address.

[0085] 2) In response to the initialization configuration command issued by the server's CPU, the control module (i.e., the SoC chip) in the source end is invoked to initialize and configure the DMA controller in the source end (i.e., the DMA controller in the NVMe SSD) and the DMA controller in the target end (i.e., the DMA controller in the GPU). In the initial stage of the system, the state of each component needs to be correctly configured, the corresponding DMA controller needs to be set to the appropriate working mode, transmission parameters, etc., and the mapping relationship between the component memory address and the system bus address needs to be established so that subsequent data can be successfully transferred between the components.

[0086] 3) Establish a P2PDMA-enabled channel between the DMA controller in the source end and the DMA controller in the target end after initialization.

[0087] 4) In response to the GDS channel establishment completion signal, the source end is triggered to execute the target data transmission command.

[0088] 5) Read target data from the source end according to the transmission length and transmission mode. That is, read data from the flash memory chip of the NVMe SSD through the DMA controller in the NVMe SSD, and then transmit the target data to the target end through a channel that supports P2P DMA. Specifically, the target data is transmitted to the internal PCIe interconnect module through the NVMe SSD, and then to the GPU module. The DMA controller in the GPU module writes the target data to the specified location of the GPU's video memory.

[0089] Furthermore, in response to the write completion instruction of the target data, an interrupt signal is generated and sent to the source end. The source end stops reading data based on the interrupt signal and can receive other instructions for execution, which facilitates subsequent data processing operations.

[0090] In this application, the interconnection pattern of different modules within a data interaction GPU unit constitutes the first intelligent computing architecture; the interaction pattern between different data interaction GPU units constitutes the second intelligent computing architecture; and the interaction pattern between a data interaction GPU unit and a CPU unit constitutes the third intelligent computing architecture. The second intelligent computing architecture includes interactions between different data interaction GPU units on the same node, as well as interactions between different data interaction GPU units on different nodes. These different intelligent computing architecture segments allow the system to integrate hardware resources of different types and performance levels. This heterogeneous computing approach can fully leverage the advantages of different hardware and improve the overall computing efficiency of the system.

[0091] The first segment of the intelligent computing architecture focuses on the interconnection of modules within the GPU unit for data interaction. This allows the GPU unit itself to be independently optimized for performance and expanded for functionality. For example, for specific computing tasks, such as convolution operations in deep learning, the computing cores and cache modules inside the GPU can be customized and upgraded without affecting other parts of the entire system.

[0092] The second segment of the intelligent computing architecture covers the interaction between different data interaction GPU units on the same node and different nodes. This design allows the system to easily improve its overall computing power by increasing the number of GPU units. Whether it is adding GPU cards within a single server node or expanding the GPU cluster across multiple server nodes, it can be seamlessly expanded based on the existing interaction mode.

[0093] The third segment of the intelligent computing architecture handles the interaction between the GPU and CPU units, allowing the system to flexibly adjust the GPU and CPU ratio according to actual needs. In scenarios requiring a large amount of parallel computing, the use of GPUs can be increased; while in tasks requiring complex logic control and serial processing, the CPU can play a greater role.

[0094] Furthermore, the three-stage AGC intelligent computing architecture system disclosed in this application can effectively improve computer performance. Specifically, the first stage of the intelligent computing architecture optimizes the interconnection of modules within the GPU unit, reducing data transmission latency between modules and increasing data processing speed. The interaction mode between GPU units on the same node and different nodes in the second stage of the intelligent computing architecture can fully leverage the parallel computing advantages of the GPU. Multiple GPU units can process different subsets of data simultaneously, and then merge and summarize the data through an efficient interaction mode, effectively enhancing parallel computing capabilities. The third stage of the intelligent computing architecture enables the GPU unit and CPU unit to work collaboratively. The CPU can be responsible for task scheduling, data preprocessing, etc., while the GPU focuses on large-scale parallel computing. This collaborative computing approach can fully utilize the control capabilities of the CPU and the computing power of the GPU, improving the overall computing performance of the system.

[0095] The disclosed solution supports diverse application scenarios. In practical applications, different scenarios have different requirements for computing resources. The three-stage intelligent computing architecture can flexibly adjust the configuration of each stage according to the characteristics of the specific application. For example, in the field of scientific computing, more emphasis may be placed on the parallel computing capabilities of GPUs, so the number of GPU units in the second stage architecture can be increased; while in the field of real-time data processing, closer collaboration between CPU and GPU may be required, in which case the interaction mode of the third stage architecture can be optimized.

[0096] Furthermore, the solution disclosed in this application can improve the portability of applications; through the relatively independent settings of each segment of the intelligent computing architecture, applications can be more easily ported between different system configurations. For example, a GPU-based deep learning application can be developed and tested on a single GPU system, and then easily scaled to a multi-GPU cluster environment by adjusting the second and third segment architectures.

[0097] In traditional CPU-centric computer systems, all data exchange between peripherals must be processed by the CPU. This places a heavy burden of data transfer and scheduling on the CPU. In the AI ​​era, with the explosive growth of data volume, this model overwhelms the CPU, becoming a bottleneck for system performance improvement. This application discloses a three-stage AGC (AI computer system with the GPU as its core) intelligent computing architecture system based on the PCIe protocol. It pioneers a GPU-centric AGC architecture, reconstructing the computer system by transferring some data exchange tasks originally handled by the CPU to the GPU. For example, during large-scale image data processing, the GPU can directly interact with storage devices, reducing the CPU's workload in data transfer and thus alleviating CPU pressure, allowing the CPU to allocate more resources to core computing tasks. Because GPUs possess powerful parallel computing capabilities and high-bandwidth data processing capabilities, using the GPU as the core for data exchange significantly improves data transfer speed. Compared to the CPU's serial data processing, the GPU can handle multiple data transfer tasks simultaneously, accelerating the flow of data between different peripherals. For example, during deep learning training, GPUs can quickly read training data from storage devices and rapidly feed the processing results back to other devices, greatly shortening the data processing cycle and improving the overall efficiency of the system.

[0098] GPUs are designed specifically for parallel computing, possessing a large number of computing cores and a natural advantage in handling large-scale matrix operations and parallel computing tasks. In the field of AI, many tasks such as deep learning, image recognition, and natural language processing involve large-scale matrix operations and parallel computing. The AGC intelligent computing architecture, with GPUs at its core, can fully leverage the parallel computing capabilities of GPUs, directly assigning AI computing tasks to the GPU for processing. This avoids frequent data transfer and conversion between the CPU and GPU, thereby improving the efficiency and performance of AI computing. For example, when training deep neural networks, GPUs can perform calculations on multiple samples simultaneously, significantly shortening training time.

[0099] Traditional computer system architectures have certain limitations when dealing with AI computing tasks, while the AGC intelligent computing architecture, reconstructed around the GPU, is better suited to the needs of AI computing. It can optimize the configuration of system hardware resources according to the characteristics and requirements of AI algorithms, improving overall system performance. For example, in designing the storage system, high-speed storage devices can be used and directly connected to the GPU, reducing data transmission latency and thus improving the speed and efficiency of AI computing.

[0100] The GPU-centric AGC intelligent computing architecture allows users to flexibly configure hardware resources according to different application scenarios and computing needs. Users can increase or decrease the number of GPUs as needed to meet AI computing tasks of different scales. Simultaneously, they can select different types and capacities of storage devices based on data storage and processing requirements and connect them directly to the GPUs, achieving flexible hardware resource configuration. For example, when conducting small-scale AI experiments, only a small number of GPUs and storage devices can be used; while for large-scale deep learning training, the number of GPUs and storage devices can be increased to improve the system's computing and data processing capabilities.

[0101] With the continuous development of AI technology and the increasing demands of its applications, computer systems need to possess excellent scalability. The AGC intelligent computing architecture, centered on the GPU, boasts strong scalability. It can easily connect more GPUs and other peripherals via the PCIe protocol, achieving seamless system expansion. For example, when the system's computing power needs to be increased, a new GPU card can be directly added and connected to the system via the PCIe interface, without requiring large-scale modifications to the entire system. This scalability enables the system to adapt to ever-changing AI application requirements, extending the system's lifespan.

[0102] This application discloses a three-segment AGC intelligent computing architecture system based on the PCIe protocol. Within the data interaction GPU unit, the GPU module, control module, storage module, and network module are connected via an internal PCIe interconnect module and the Type I PCIe protocol. Furthermore, a P2P DMA-enabled channel is established between the GPU module and the storage module, enabling direct data transfer between the GPU and storage devices without CPU intervention, significantly improving data transfer speed between local devices and reducing data transfer latency. On the same node, GPU modules in pairs of data interaction GPU units establish GPUDirect P2P connection channels, allowing direct data exchange between GPUs, effectively avoiding data transfer through the CPU and system memory, further improving data transfer efficiency between GPUs within the node. GPU modules in pairs of data interaction GPU units on different nodes establish GPUDirect RDMA connection channels, supporting remote direct memory access and achieving high-speed data transfer between GPU memory on different nodes, meeting the needs of large-scale distributed computing. This application, by adopting a three-segment PCIe structure design, decouples most functional areas from the CPU, solving the problem of insufficient CPU lanes while meeting the need for flexible adaptation to various components.

[0103] This application adopts a three-segment architecture, modularizing the data interaction GPU unit, external PCIe interconnect module, RDMA switch module, and CPU unit. Each module is relatively independent, which facilitates flexible configuration and expansion according to different application scenarios and needs. For example, the number of data interaction GPU units can be increased or decreased according to the scale of the computing task. A hot-swappable connection channel is established between the data interaction GPU unit and the CPU unit, which allows for easy plugging and unplugging of related components during system operation without shutting down the entire system, thus improving the maintainability and availability of the system.

[0104] In this application, different modules are connected using different types of PCIe protocols. This avoids the cost increase caused by uniformly adopting high-version PCIe standards. Appropriate PCIe protocol versions can be selected based on the actual needs of each module. Modules with low bandwidth requirements use lower-version PCIe protocols, reducing component costs. By rationally allocating PCIe protocol versions, the increased power consumption caused by low-speed devices operating in high-version PCIe mode is effectively avoided. For example, low-bandwidth devices using lower-version PCIe protocols have relatively lower power consumption, which helps reduce the overall system's heat dissipation pressure and energy consumption, conforming to the principles of green computing.

[0105] This application employs a three-segment intelligent computing architecture based on the PCIe protocol, distributing computing resources across multiple data-interacting GPU units. Unlike traditional CPU-centric designs where all devices rely on the CPU's PCIe channels, each data-interacting GPU unit has its own internal PCIe interconnect module, enabling communication between internal devices and reducing reliance on the CPU's PCIe channels. By establishing channels supporting P2P DMA, GPUDirect P2P connection channels, and GPUDirect RDMA connection channels, direct data transfer between devices is possible, bypassing the CPU and further alleviating the pressure on the CPU's PCIe channel allocation. For example, the P2P DMA channel between the GPU and the storage module allows them to exchange data directly without occupying the CPU's PCIe channels.

[0106] This application employs Type I, Type II, Type III, and Type IV PCIe protocols for connection based on the actual needs of different modules. For low-speed peripherals, a lower version of the PCIe protocol can be selected to improve bandwidth utilization and avoid the waste of physical layer resources caused by using a higher version of the PCIe interface. For example, for devices such as USB controllers and low-speed network cards, a lower version of the PCIe protocol can be used to meet their data transmission requirements without wasting the bandwidth of a higher version of the PCIe interface.

[0107] This application does not mandate the adoption of the latest PCIe standard. Instead, it selects appropriate PCIe protocol versions based on the bandwidth requirements of each module. For low-bandwidth devices, such as SATA controllers and 1GbE network cards, using lower versions of the PCIe protocol reduces component costs. For example, instead of using a PCIe 5.0 x4 network card to meet the requirement of only 1 Gb / s bandwidth, a more suitable low-bandwidth network card is chosen, avoiding resource waste and cost increases. Because different types of PCIe protocols are used, existing PCIe 3.0 / 4.0 devices can be made compatible by selecting appropriate PCIe protocols, reducing hardware iteration costs and enabling smooth upgrades. For example, some older PCIe 3.0 devices can be connected to modules using appropriate lower-version PCIe protocols to continue functioning. By rationally configuring the PCIe protocol version according to the device's performance and bandwidth requirements, the increased power consumption caused by uniformly adopting high-version PCIe standards is avoided. Low-speed devices operating in appropriate lower-version PCIe modes reduce energy consumption per unit of data throughput and correspondingly reduce overall system cooling pressure, conforming to green computing principles. For example, PCIe 4.0 devices consume less power than PCIe 5.0 devices. Using the PCIe 4.0 protocol can effectively reduce power consumption for some devices that do not have high bandwidth requirements.

[0108] In this application, the key components use a higher version of the PCIe protocol, the secondary components use a medium version of the PCIe protocol, and the scheduling and management functions use a general version of the PCIe protocol. Through this three-segment PCIe structure design, not only is the performance of the intelligent computing server improved by more than 30%, but the overall hardware cost of the intelligent computing server is also reduced by more than 20%.

[0109] The intelligent computing server, employing a three-segment PCIe architecture, supports the coexistence of different versions of the PCIe protocol, effectively avoiding hardware silos that may arise during technology iterations. Whether it's advanced PCIe 6.0 devices or widely used PCIe 4.0 or 3.0 components, this design enables seamless integration, ensuring efficient collaborative operation. This design feature guarantees that users do not need to completely replace old equipment when upgrading hardware, thus saving costs and extending the hardware's lifespan. Furthermore, through precise protocol management, conflicts between different signals and protocols can be effectively resolved, ensuring the stability and efficiency of data transmission. Support for hybrid protocols eliminates technology silos and effectively resolves conflicts between signals and protocols.

[0110] The three-segment PCIe architecture allows different functional areas to operate corresponding components according to different PCIe protocols, effectively reducing the overall power consumption of the intelligent computing system by 15%-25%, thereby reducing heat dissipation requirements. Simultaneously, high-frequency signal interference is effectively suppressed, thus improving the stability and reliability of the intelligent computing system. Furthermore, this design simplifies the maintenance and management process of the intelligent computing server, as the independence of each component makes troubleshooting and resolution more convenient.

[0111] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A three-stage AGC intelligent computing architecture system based on the PCIe protocol, characterized in that, include: The data interaction GPU unit includes an internal PCIe interconnect module, and a GPU module, a control module, a storage module, and a network module, all of which are connected to the internal PCIe interconnect module via a first type of PCIe protocol. A P2P DMA-enabled channel is established between the GPU module and the storage module. An external PCIe interconnect module is provided, and each of the data interaction GPU units on the same node is connected to the external PCIe interconnect module via the second type of PCIe protocol; a GPUDirect P2P connection channel is established between the GPU modules of each pair of data interaction GPU units on the same node. The RDMA switch module is connected to each of the data interaction GPU units on different nodes via the third type of PCIe protocol; a GPUDirect RDMA connection channel is established between the GPU modules of each pair of data interaction GPU units on different nodes. The CPU unit is not on the same board as the data interaction GPU unit, and is connected to the data interaction GPU unit via the fourth type PCIe protocol; a hot-swappable connection channel is established between the data interaction GPU unit and the CPU unit.

2. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The internal PCIe interconnect module, the GPU module, the control module, the storage module, and the network module in each of the data interaction GPU units are all integrated and packaged together. Each of the aforementioned data interaction GPU units is configured independently.

3. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 2, characterized in that, The CPU unit includes several servers, each server being connected to one or more of the data interaction GPU units; each of the data interaction GPU units is an independent hot-swappable unit. The network module in the data interaction GPU unit is connected to the corresponding OSFP interface in a single server, and the OSFP interfaces connected to different data interaction GPU units are all different.

4. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The version of the second type of PCIe protocol is lower than the version of the first type of PCIe protocol; The version of the third type of PCIe protocol is the same as the version of the first type of PCIe protocol. The version of the fourth type of PCIe protocol is no higher than the version of the second type of PCIe protocol.

5. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 4, characterized in that, The first type of PCIe protocol is the PCIe-Gen5 / 6 protocol; The second type of PCIe protocol is the PCIe-Gen4 / 5 protocol; The fourth type of PCIe protocol is the PCIe-GEN3 / 4 / 5 protocol.

6. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The transmission rate of the channel supporting P2P DMA is V1, where 100Gb / s ≤ V1 ≤ 400Gb / s.

7. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The transmission rate of the GPUDirect RDMA connection channel is V2; 2Tb / s≤V2≤8TbGb / s.

8. The three-stage AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The P2P DMA-supporting channel, the GPUDirect P2P connection channel, and the GPUDirect RDMA connection channel do not occupy CPU lanes.

9. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 1, characterized in that, The GPU module integrates a DMA controller. The storage module integrates a DMA controller. The channel supporting P2P DMA is the transmission channel between the DMA controller inside the GPU module and the DMA controller inside the storage module.

10. The three-segment AGC intelligent computing architecture system based on the PCIe protocol according to claim 9, characterized in that, The storage module is an NVMe SSD; Both the internal PCIe interconnect module and the external PCIe interconnect module are PCIe switch chips. The control module is an integrated system-on-a-chip; The network module is an RDMA network card.

Images