GPU BOX, GPU Control System

Abstract

This application discloses a GPU BOX and a GPU control system. The GPU BOX includes a housing with a slot for mounting a GPU module. A control motherboard, a network module, and a power module connected to the GPU module are installed within the housing. The control motherboard integrates a control module, a storage module, a PCIe interconnect module, and a slave control unit. The GPU module, control module, storage module, and network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module. A connector is mounted on the control motherboard, and the network module connects to a target server via the connector. The ends of the connector and the power module are matched to the insertion / removal slots of the target server. This solution can adapt to different GPU models, enabling rapid and convenient GPU replacement without server downtime, significantly improving system availability and efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a GPU BOX and a GPU control system. Background Technology

[0002] In today's rapidly developing AI data centers and machine learning fields, the demand for GPU swapping is becoming increasingly prominent. However, during large model pre-training and fine-tuning, the GPUs involved in the task execute only a part of the overall task. If a GPU experiences an alarm or malfunction, the current approach is that replacing the GPU requires interrupting part or the entire task, and powering off the device before it can be replaced. This not only leads to task failure and the need to restart, but also results in a significant waste of time and resources. For example, a long-running large model pre-training task might be completely wasted due to a GPU failure, requiring a significant amount of time and power to retrain.

[0003] Meanwhile, the process of inserting and removing GPU cards within the chassis involves opening the chassis, preparing the GPU power cable, checking slot space, powering on, testing, and then closing the chassis and placing it in the rack. This process is not only time-consuming but also requires professional operation to ensure that hardware damage or data loss is not caused by static electricity, power mismatch, or other factors during insertion and removal. Furthermore, traditional insertion and removal methods require system shutdown every time a GPU is replaced, which severely impacts system availability and efficiency. For large data centers or high-performance computing environments, frequent hardware replacements lead to a significant increase in operating costs. Utility Model Content

[0004] In view of this, the present disclosure provides a GPU BOX and a GPU control system, which can solve the problems of server power outages, long replacement cycles, low system efficiency, and poor availability caused by GPU replacement in the prior art.

[0005] In a first aspect, embodiments of this disclosure provide a GPU BOX, comprising:

[0006] The housing has a recess for mounting the GPU module;

[0007] The enclosure houses a control motherboard, a network module, and a power module connected to the GPU module. The control motherboard integrates a control module, a storage module, a PCIe interconnect module, and a slave control unit. The GPU module, the control module, the storage module, and the network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module.

[0008] A connector is installed on the control motherboard, and the network module is connected to the target server through the connector.

[0009] The connector and the power module have their ends extending out of the housing, and both the ends of the connector and the power module are matched with the plug-in slots of the target server.

[0010] Secondly, this application discloses a GPU control system, comprising:

[0011] An expansion backplane enclosure includes a first accommodating area and a second accommodating area. The first accommodating area includes N first mounting slots for mounting N target servers. At least one of the target server slots is equipped with a PCIe Switch expansion card, which has several downstream ports, and at least two of the downstream ports are Box mounting slots. The second accommodating area includes M second mounting slots for housing the GPU Box, which is used to mount GPUs.

[0012] Power supply groups; M power supply groups are configured, and each power supply group is matched with one GPUBOX.

[0013] The main control unit is located on the motherboard of the corresponding target server;

[0014] Each slave control unit is connected to the main control unit, and each slave control unit is connected to a hot-swap button of the GPU BOX. The slave control unit is used to obtain the status information of the corresponding hot-swap button in real time and send it to the corresponding target server through the main control unit.

[0015] The GPU BOX disclosed in this application includes a housing with a receiving slot for installing a GPU module. A control motherboard, a network module, and a power module connected to the GPU module are installed within the housing. The control motherboard integrates a control module, a storage module, a PCIe interconnect module, and a slave control unit. The GPU module, control module, storage module, and network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module. A connector is installed on the control motherboard, and the network module connects to a target server via the connector. The ends of the connector and power module protrude from the housing, and the ends of the connector and power module are matched with the insertion / removal slots of the target server. This structure can adapt to different GPU models; simply installing the GPU into the GPU BOX allows for seamless connection via the GPU module. The BOX enables rapid plug-and-play control with the target server without interrupting part or the entire task or shutting down the server, significantly improving system availability and efficiency. This solution effectively reduces reliance on professional personnel, lowers manpower and material costs, and greatly reduces the risk of hardware damage or data loss due to improper operation. Through the corresponding power supply group, it effectively solves the problem that traditional GPUs need to draw additional power supply lines from the server power supply due to limited PCIe power, ensuring stable power supply for each GPU.

[0016] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a three-dimensional schematic diagram of a GPU BOX provided in an embodiment of this disclosure.

[0019] Figure 2 for Figure 1 A schematic diagram of the second angle.

[0020] Figure 3 for Figure 1 A three-dimensional schematic diagram of the push-pull limiting device.

[0021] Figure 4 for Figure 1 A schematic diagram of the third angle.

[0022] Figure 5 This is a three-dimensional schematic diagram of a GPU control system provided in an embodiment of this disclosure.

[0023] Figure 6 for Figure 5 A magnified view of part A in the image.

[0024] Figure 7 This is a schematic diagram illustrating the application of the GPU control system provided in this embodiment.

[0025] Figure 8 This is a schematic diagram of the intelligent computing architecture of the GPU control system provided in an embodiment of this disclosure.

[0026] Explanation of reference numerals in the attached figures:

[0027] 100. Extended backplate box; 111. Second mounting position; 112. Limiting hole; 200. GPU BOX; 210. Box body; 211. Snap-fit ​​protrusion; 212. Through hole; 213. First ventilation hole; 214. Second ventilation hole; 220. GPU module; 230. Connector; 240. Power module; 250. Push-pull limiting device; 251. First rod segment; 252. Second rod segment; 253. Tightening component; 254. Pin. Detailed Implementation

[0028] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0029] Reference Figure 1 and Figure 2 The first aspect of this application discloses a GPU BOX that can install different models of GPU cards on the market without changing the existing GPU card structure, enabling the GPU card to be plugged in and replaced and powered on without shutting down or opening the target server. It is the first to propose an innovative GPU service structure that enables hot-swapping from outside the server.

[0030] Specifically, the GPU BOX includes a housing 210, which has an internal slot for mounting a GPU module 220.

[0031] The enclosure 210 houses a control motherboard, a network module, and a power module 240 connected to the GPU module 220. The control motherboard integrates a control module, a storage module, and a PCIe interconnect module. The GPU module 220, control module, storage module, and network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module 220, which allows the GPU card to be fixed in its original fixed mode.

[0032] A connector 230 is installed on the control motherboard, and the network module is connected to the target server through the connector 230. Furthermore, the ends of the connector 230 and the power module 240 are set to protrude from the housing 210, and the ends of the connector 230 and the power module 240 are matched with the plug-in slots of the target server.

[0033] A push-pull limiting device 250 is provided on the outside of the box 210. The push-pull limiting device 250 is used to assist in pushing the box 210 to engage and fix with the target server in the first state, to assist in pulling out the box 210 to disconnect from the target server in the second state, and to lock the relative position of the box 210 and the target server in the third state.

[0034] Specifically, the first state is the process of connecting the GPU BOX to the corresponding slot of the target server. The push-pull limiting device 250 applies force to the box 210 under the action of external force, pushing the box 210 along the corresponding channel to approach the target server until it is locked and fixed with the target server.

[0035] The second state is the process of disconnecting the GPU BOX from the corresponding slot of the target server. The push-pull limiting device 250 applies force to the box 210 under the action of external force, pulling the box 210 outward along the corresponding channel to disconnect it from the target server.

[0036] In the third state, when the GPU BOX is engaged and fixed with the target server, the push-pull limiting device 250 is in a vertical state, used to lock the relative position of the box 210 and the target server, ensuring the stability of the engagement and fixation with the target server.

[0037] A number of first ventilation holes 213 are provided on the first side, and a number of second ventilation holes 214 are provided on the opposite side of the first side. The number of first ventilation holes 213 and the number of second ventilation holes 214 are arranged opposite to each other, forming a good air convection channel inside the box 210. Cold air can be discharged from the ventilation holes on one side, and after passing through the inside of the box 210, it carries heat and is discharged from the ventilation holes on the other side, realizing effective heat dissipation of the internal components of the box 210 and accelerating the air circulation speed. This convection ventilation method is much better than natural heat dissipation or unidirectional ventilation, and can reduce the temperature around the GPU in time.

[0038] The ventilation area formed by several first ventilation holes 213 and the ventilation area formed by several second ventilation holes 214 are both matched and set to match the GPU module 220. That is, the ventilation area is determined according to the heat generation and heat dissipation requirements of the GPU module 220. This ensures that there is enough airflow through the GPU module 220 to provide it with just the right heat dissipation capacity, avoiding energy waste due to excessive ventilation area or insufficient heat dissipation due to insufficient ventilation area.

[0039] In this embodiment, through airflow and heat dissipation simulation experiments, the maximum contact surface of the internal airflow of the GPU BOX is designed to pass through the GPU module 220, so that the cooling air can directly and fully contact the heat source, maximize the heat exchange efficiency between the air and the GPU module 220, and quickly conduct the heat generated by the GPU card away, further improving the overall ventilation and heat dissipation effect.

[0040] The size of the receiving slot is not less than that of the GPU module 220, and the receiving slot is compatible with different types of GPU cards. The GPU BOX provided by this application has a receiving slot that is compatible with different types of GPU cards, and can be compatible with a variety of GPU cards. This allows users to choose the appropriate GPU card according to their actual needs without worrying about installation problems due to incompatibility of the receiving slot. In addition, the receiving slot with good compatibility can facilitate future GPU card upgrades, ensuring that the device can keep up with the pace of technological development and extend the service life of the device.

[0041] When users need to upgrade the system's graphics processing capabilities, they can simply remove the existing GPU card from its slot and replace it with a more powerful one. This convenient upgrade method allows the device to maintain a high performance level without requiring a major hardware replacement. When a GPU card malfunctions, the compatible slots for different types of GPU cards make replacing the faulty card easier. Repair personnel can quickly find and install a suitable replacement GPU card, reducing equipment downtime and improving maintenance efficiency.

[0042] The end of the power module 240 has a hole-like structure, which facilitates the connection and disconnection of the power line in one step when plugging and unplugging it on the server.

[0043] The connector 230 features a multi-row hole structure at its end, effectively reducing the difficulty of insertion and removal operations. Operators no longer need to spend significant time and effort manually aligning pins and holes, improving insertion and removal efficiency. This is especially beneficial during large-scale server deployments or maintenance, where frequent insertion and removal operations become easier and faster, saving labor and time costs. Connector 230 is used to route network signals from the network module to the OSFP interface on the target server's CPU control board. It effectively reduces contact resistance, helping to minimize signal loss and interference during transmission, ensuring stable and efficient network signal transmission, reducing data packet loss, and improving the quality and reliability of network communication.

[0044] In this embodiment, the housing is made of durable materials and has a self-alignment function; the ends of the power module and the connectors are made of flexible joint structure, which is easy to plug and unplug and will not damage the ends during the plugging and unplugging process, thus improving the service life of the equipment.

[0045] The GPU module integrates a DMA controller, and the storage module also integrates a DMA controller. A P2P DMA-enabled channel is established between the GPU module and the storage module. This channel is used for direct access between the storage device and the video memory in the GPU module, and supports direct read and write access to the disk.

[0046] Specifically, the channel that supports P2P DMA is the transmission channel between the DMA controller inside the GPU module and the DMA controller inside the storage module.

[0047] In this embodiment, the transmission rate of the P2P DMA-supporting channel is V1, where 100Gb / s ≤ V1 ≤ 400Gb / s. P2P DMA-supporting channels allow direct memory data transfer between devices without CPU intervention. The higher transmission rate (100Gb / s - 400Gb / s) enables large amounts of data to move rapidly between different devices (such as GPUs, storage devices, network interface cards, etc.), effectively accelerating data transfer between GPUs, reducing waiting time, and thus fully leveraging the parallel computing capabilities of multiple GPUs. The 100Gb / s-400Gb / s transmission rate provides sufficient bandwidth expansion space for the system, allowing it to adapt to the growth in data volume and computing demands over the next few years or even longer. This means that system upgrades and expansions do not require immediate replacement of the entire data transmission architecture, reducing upgrade costs and complexity.

[0048] The GPU module establishes a GPUDirect P2P connection channel with other GPU modules on the same target server; the GPU module establishes a GPUDirect RDMA connection channel with other GPU modules on different target servers.

[0049] The GPUDirect RDMA connection channel has a transfer rate of V2; 2Tb / s ≤ V2 ≤ 8TbGb / s. This high transfer rate allows data to move at lightning speed between different GPUs, and between GPUs and storage devices or other computing nodes, enabling rapid data sharing and synchronized computation results, thus enhancing the overall computing power and scalability of the cluster. For example, in a supercomputer cluster, multiple GPU nodes can work collaboratively through high-speed channels to solve more complex scientific computing and engineering simulation problems.

[0050] In this embodiment, the storage module is preferably an NVMe SSD, and the NVMe SSD interface is preferably an Edge 1.S interface, which can meet the data transmission requirements of high-performance storage devices, thereby improving the overall performance and efficiency of the GPU BOX.

[0051] The PCIe interconnect module is preferably a PCIe switch chip, which can relay and amplify signals from PCIe, control circuits, etc. that are attenuated due to length.

[0052] The network module is preferably an RDMA network card, which can achieve high-speed, low-latency network data transmission; in a distributed computing environment, RDMA network cards can significantly improve the data exchange efficiency between nodes.

[0053] Compared to traditional data access methods where data is first transferred from the disk to system memory and then processed and scheduled by the CPU to move the data to the GPU memory, the GPU BOX disclosed in this embodiment provides a direct access method that can significantly reduce data transmission latency and improve data processing efficiency. This solution bypasses the CPU, allowing direct communication between the GPU and storage devices, greatly improving the efficiency and speed of data transmission.

[0054] Specifically, during deep learning training, GPUs need to frequently read large amounts of training data from storage devices. Through the P2P DMA-enabled channel established in this application, GPUs can quickly acquire data, accelerating training speed and shortening model training time. In high-performance computing fields such as scientific computing and simulation, large amounts of data need to be processed. The established P2P DMA-enabled channel can improve data transfer efficiency, enabling GPUs to perform calculations more efficiently and improving computational performance. For data-intensive applications such as video processing and image analysis, the established P2P DMA-enabled channel can accelerate data read and write speeds, improving application processing power and response speed.

[0055] Each GPU BOX integrates its internal PCIe interconnect module, GPU module control module, storage module, and network module, preferably within a single GPU BOX. This GPU BOX is then mounted on the server via an expansion backplane enclosure. The assembled GPU BOX and the server form a smart computing server, effectively shortening the physical distance between components, reducing signal transmission path length, lowering signal latency and interference, improving data transmission stability and speed, and ensuring high-speed and accurate data interaction between modules, thereby enhancing the overall computing performance of the GPU unit. The integrated packaging of each module also makes communication between modules more efficient, reducing unnecessary energy consumption. Furthermore, unified power management is more convenient, intelligently adjusting power consumption based on the GPU unit's operating status, thereby reducing overall energy consumption and improving energy efficiency.

[0056] In this embodiment, each GPU BOX is independently configured. These independently configured GPU BOXes function like standardized components; during server or computing system deployment, they simply need to be installed onto the corresponding interfaces, eliminating the need for complex wiring and debugging. This plug-and-play feature significantly shortens system deployment time and improves efficiency. When computing demands increase, independent GPU BOXes can be easily added to enhance the system's computing power. This can be achieved easily, whether increasing the number of GPU units within a single server or expanding within a cluster of multiple servers, without requiring large-scale modifications to the existing system. Because each GPU BOX is independently configured, a failure in one unit does not affect the normal operation of other units. The system can quickly identify and isolate the faulty unit, continuing to utilize other functioning GPU units to complete computing tasks, thereby improving the overall system reliability and availability and reducing downtime due to hardware failures. The independently configured GPU BOXes also offer greater operability during maintenance. If a unit malfunctions, it can be directly removed from the system for repair or replacement without affecting other units or the normal operation of the entire system. This makes maintenance simpler, more efficient, and reduces maintenance costs and complexity. Since each GPU module is independent, its performance can be monitored and managed separately. Through monitoring software, information such as the working status and performance indicators of each unit can be obtained in real time, so as to promptly identify potential problems and make adjustments and optimizations to ensure that the entire system is always in the best operating state.

[0057] Reference Figure 1 and Figure 3 The push-pull limiting device 250 includes a first rod segment 251, a second rod segment 252, and a screwing component 253. The first rod segment 251 is connected to the outside of the box body 210 through a pin 254 and has the freedom to rotate around the pin 254, which facilitates effective control of the box body 210.

[0058] The housing 210 has a snap-fit ​​hole on its first side and a through hole 212 on the opposite side of the first side. The through hole 212 is used for the ends of the cantilever connector 230 and the power module 240.

[0059] The first segment 251 has an engaging part, specifically a C-shaped groove located at the end of the first segment 251.

[0060] The second segment 252 is located at the end of the first segment 251 away from the pin 254, and the second segment 252 has a mounting part for mounting the screwing component 253.

[0061] In the first and second states, the second rod segment 252 is both away from the housing 210 and does not contact the housing 210; in the third state, the second rod segment 252 or the first rod segment 251 rotates around the pin 254 under the action of external force until it fits against the side wall of the housing 210, the engaging part abuts against the preset position of the target server and the screwing part 253 is fixed with the engaging hole to lock the relative position of the housing 210 and the target server.

[0062] In the third state, the engaging part is located inside the engaging hole and abuts against the inner wall of the engaging hole, providing resistance to prevent the first rod segment 251 from continuing to rotate inward, so as to inform the operator that the rotation is in place, and the screw 253 can be operated to lock the box 210, which is simple and efficient.

[0063] The screw 253 is preferably a fixing screw, which facilitates the insertion, removal and fixing of the GPU BOX.

[0064] In actual operation, the first rod segment 251 can be rotated to a horizontal position and then pushed inward until it can no longer be pushed, indicating that it is properly connected to the target server. Then rotate the first rod segment 251 to a vertical position. At this time, the engaging part on the first rod segment 251 is located in the limiting hole 112 and abuts against the inner wall side of the limiting hole 112, so that the first rod segment 251 can no longer be controlled to rotate inward. Then tighten the fixing screw by hand to lock the second rod segment 252 to the box 210.

[0065] Reference Figure 4 The GPU BOX has a snap-fit ​​protrusion 211 on its side, and the inner wall of the second mounting position 111 has a snap-fit ​​groove that matches the snap-fit ​​protrusion 211.

[0066] The second mounting position may include a first receiving area and a second receiving area. The width between the inner walls on both sides of the first receiving area is smaller than the width between the inner walls on both sides of the second receiving area. Each inner wall has a locking groove located in the transition area between the first and second receiving areas. When the locking protrusion on the side of the GPU BOX engages with the locking groove inside the second mounting position, the GPU BOX cannot be pushed inward. At this point, the GPU BOX is successfully plugged into the target server. The locking protrusion and locking groove design ensures the installation accuracy of the GPU BOX and prevents external forces applied to the housing from damaging the corresponding interface of the target server.

[0067] The GPU BOX disclosed in this application includes a housing with a receiving slot for installing a GPU module. A control motherboard, a network module, and a power module connected to the GPU module are installed within the housing. The control motherboard integrates a control module, a storage module, a PCIe interconnect module, and a slave control unit. The GPU module, control module, storage module, and network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module. A connector is installed on the control motherboard, and the network module connects to a target server via the connector. The ends of the connector and power module protrude from the housing, and the ends of the connector and power module are matched with the insertion / removal slots of the target server. This structure can adapt to different GPU models; simply installing the GPU into the GPU BOX allows for seamless connection via the GPU module. The BOX enables rapid plug-and-play control with the target server without interrupting part or the entire task or shutting down the server, significantly improving system availability and efficiency. This solution effectively reduces reliance on professional personnel, lowers manpower and material costs, and greatly reduces the risk of hardware damage or data loss due to improper operation. Through the corresponding power supply group, it effectively solves the problem that traditional GPUs need to draw additional power supply lines from the server power supply due to limited PCIe power, ensuring stable power supply for each GPU.

[0068] Reference Figure 5 and Figure 6 This application discloses a GPU control system, specifically a PCIe-based GPU control system. The system includes an expansion backplane enclosure 100, a power supply group, a master control unit, and a slave control unit. The expansion backplane enclosure 100 is located on the back or side of the target server for user convenience and to avoid interference with other components. Specifically, the expansion backplane enclosure 100 includes a first accommodating area and a second accommodating area. The first accommodating area includes N first mounting positions for mounting N target servers. At least one target server slot houses a PCIe Switch expansion card, which has several downstream ports, with at least two downstream ports being Box mounting slots. The second accommodating area includes M second mounting positions 111 for housing the GPU Box 200, which is used to mount GPUs for easy insertion and removal.

[0069] Where 0 < N < M, the number of GPUs connected to the server can be effectively expanded through the settings of the PCIe Switch expansion card and GPU BOX.

[0070] In this embodiment, the second mounting position 111 is an independent channel. A limiting hole 112 is provided in the second mounting position 111 to assist in limiting the position of the GPU BOX after it is installed in place.

[0071] In this embodiment, there are M power supply groups, each power supply group is matched with a power module 240 of a GPU BOX, and is used to independently control the power supply of a single GPU BOX.

[0072] The main control unit is configured in the motherboard of the corresponding target server and is used to connect with the slave control units. Each slave control unit is connected to a hot-swap button of a GPU BOX and is used to obtain the status information of the corresponding hot-swap button in real time and send it to the CPU of the corresponding target server through the main control unit. The CPU of the target server controls the hot-swap operation of the corresponding GPU BOX according to the status information signal.

[0073] The slave control unit is connected to the main control unit, and each slave control unit is connected to a hot-swap button of a GPU BOX to obtain the status information of the corresponding hot-swap button in real time and send it to the corresponding target server through the main control unit.

[0074] Specifically, the CPU of the target server obtains the status type corresponding to the actual register value based on the preset mapping relationship between register values ​​and different status information of hot-swap buttons; obtains the slot information of the GPU BOX connected to the target server; determines the installation status of the GPU BOX based on the slot information; determines the target request of the GPU BOX based on the installation status and status type; and controls the running status of the GPU BOX based on the target request.

[0075] The target requests include plug-in power-on requests or power-off unplug requests.

[0076] Specifically, when the installation status is that the GPU BOX is plugged into the target server and the status type is "expected insertion", the target request for the corresponding GPU BOX is determined to be an insertion and power-on request; when the installation status is that the GPU BOX is plugged into the target server and the status type is "expected removal", the target request for the corresponding GPU BOX is determined to be a power-off and removal request; when the installation status is that the GPU BOX is not plugged into the target server and the status type is "expected insertion", the target request for the corresponding GPU BOX is determined to be an insertion and power-on request; when the installation status is that the GPU BOX is not plugged into the target server and the status type is "expected removal", the target request for the corresponding GPU BOX is determined to be a power-off and removal request.

[0077] In the "power-off unplugging" request, "power-off" refers to the GPU BOX wanting to disconnect from the server and no longer requiring additional power. Throughout this process, the target server does not need to be powered off, and the plugging or unplugging of a single GPU BOX does not affect the normal operation of other connected GPU BOXes. Determining the target request based on the installation status and status type allows the system to respond appropriately to the actual situation, improving the system's intelligence and operational security.

[0078] This system controls the GPU BOX's operating status based on target requests, achieving automated control of GPU BOX hot-swapping operations. Specifically, power checks and initialization are performed when the GPU BOX is plugged in and powered on, ensuring its normal operation; resource release and power-off operations are performed when the GPU BOX is unplugged, improving system stability and hardware lifespan.

[0079] With the solution disclosed in this embodiment, when inserting or removing the GPU card, that is, when inserting or removing the GPU BOX in this embodiment, there is no need to shut down the corresponding server. The GPU BOX can be directly inserted or removed to connect or disconnect from the target server.

[0080] Furthermore, in this embodiment, when the target request is a power-off and unplug request, the system will activate the corresponding protection program after receiving a signal that the hardware is about to be removed. The operating system will suspend all data transmission and processing tasks related to the GPU to ensure that data flow stops safely. For example, ongoing graphics rendering tasks will be temporarily interrupted to avoid data loss or corruption.

[0081] Furthermore, the GPU control system disclosed in this application can flexibly operate the GPU BOX according to actual needs, such as installation, removal, and self-testing. At the same time, the system can promptly detect installation problems and issue alarms, facilitating maintenance and management by administrators. During hot-swapping operations, the system will first perform necessary resource release and data saving operations to avoid data loss and system failure caused by sudden removal of the GPU BOX, thereby improving system security. The system can reasonably control the GPU BOX based on different status information, such as shutting it down when it is not needed to save energy, and starting it up in a timely manner when needed to improve resource utilization efficiency.

[0082] Furthermore, if the status is "request to unplug", the system indicator light of the GPU BOX will be set to flash rapidly (e.g., 5 times per second) to indicate that the request has been received. If the application-level hot-plug task processing software is not registered, the power to the control slot will be cut off, the current status of each slot will be saved, and all indicator lights will be turned off.

[0083] The system also includes: configuring hot-swap service software, which establishes network connections with different target servers to obtain GPU operating status information of each target server and triggers an early warning signal when the GPU operating status information is abnormal; the GPU operating status information includes GPU core processor utilization and GPU memory utilization.

[0084] Specifically, software registration and deregistration functions can be added to the hot-swap service software. The application-level hot-swap task processing software can be deployed on a server with an interconnected network to uniformly manage and process the hot-swap-related AI tasks and hot-swap events of each GPU server. That is, after the application-level hot-swap task processing software is deployed on a server with an interconnected network, SHPS establishes a network connection with each GPU server to achieve unified management.

[0085] The hot-swap service software monitors the GPU core processor utilization and GPU memory utilization of each GPU server in real time. When the utilization exceeds the threshold, it controls the system indicator light of the corresponding GPU BOX to flash. A GPU load balancing scheduling algorithm is developed based on the performance, power consumption, and strengths of each GPU card. Simultaneously, it supports manually specifying a server's GPU BOX for GPU task migration followed by power-off from the hot-swap service software interface; it can perform hibernation and wake-up operations on designated GPU BOXes on specific servers according to policies and set time periods; it continuously monitors the fault status of multiple server GPU BOXes and the utilization status of GPUs, memory, etc., and displays these statuses in real time on the software interface and GPU BOX indicator lights.

[0086] The process of controlling the running status of the GPU BOX based on the target request includes: in response to a power-off removal request, removing the GPU in the corresponding GPU BOX from the task scheduling list through the hot-swap service software; determining the target GPU from other GPUs according to the GPU load balancing scheduling algorithm, and having the target GPU execute the tasks to be processed by the removed GPU; in response to a power-on insertion request, powering on the newly inserted GPU in the GPU BOX through the hot-swap service software; and in response to a power-on completion instruction, adding the newly inserted GPU in the GPU BOX to the task scheduling list.

[0087] The hot-swap service software automatically responds to insertion power-up requests and performs power-up operations, eliminating the need for manual intervention in each power-up step and significantly improving operational efficiency. In large-scale data centers or scenarios requiring frequent GPU replacements, this automation can save substantial time and labor costs. The software-executed power-up operations follow a unified standard and process, ensuring consistency across all power-up operations. This guarantees that different GPUs receive the same treatment during power-up, avoiding problems caused by variations in human operation and improving system stability and maintainability.

[0088] Once power-on is complete, the newly inserted GPU in the GPU BOX is added to the task scheduling list. The system can immediately incorporate the new computing resources into the task allocation system. In subsequent task processing, the new GPU can share the load with other GPUs, improving the overall computing power and processing efficiency of the system. For example, during large-scale image rendering or deep learning training, the newly added GPU can accelerate task completion. As business grows and demands increase, it may be necessary to continuously expand the system's computing resources. Adding new GPUs to the task scheduling list in a timely manner allows the system to quickly adapt to business changes, better meet ever-growing business needs, and avoid business bottlenecks caused by insufficient resources.

[0089] The task scheduling list can rationally allocate tasks based on factors such as the performance and load of each GPU. Adding a new GPU to the list allows the system to reassess resource allocation and achieve more optimized task scheduling. For example, when a GPU is overloaded, the system can allocate some tasks to newly added GPUs, thus achieving dynamic resource balancing and improving resource utilization. This dynamic resource management approach enables the system to flexibly respond to various changes, such as hardware failures and peak business periods. When a GPU malfunctions, the system can promptly adjust task allocation, utilizing other GPUs to continue completing tasks. During peak business periods, new GPUs can be added to the task scheduling list at any time to enhance the system's processing capacity.

[0090] Furthermore, when multiple actual register values ​​sent from the control unit are received, the corresponding target requests are executed according to a preset priority order, allowing more important tasks to be processed first.

[0091] Specifically, when multiple actual register values ​​are received from the control unit (i.e., multiple buttons are pressed simultaneously), the control unit corresponding to each button generates a corresponding signal. The aggregated signals are then analyzed using an algorithm (such as a state machine or priority queue) to determine the priority and validity of the current operation. This method efficiently handles parallel operations of multiple buttons, ensuring the accuracy and reliability of the hot-swapping process.

[0092] The system also includes: in response to a request to introduce a new GPU BOX, scanning the corresponding PCIe slots through the ROM software on the target server, identifying the available slots dedicated to the GPU, and marking the available slots as independent slot resources; configuring the corresponding slot information for the new GPU BOX based on the independent slot resources; the slot information includes one or both of the physical address and priority of the corresponding slot.

[0093] By scanning the corresponding PCIe slots using ROM (Read-Only Memory) software on the target server, it is possible to accurately identify available slots dedicated to GPUs. Servers have numerous PCIe slots with diverse uses; manually identifying available GPU-dedicated slots is not only time-consuming and laborious but also prone to errors. Automated scanning using ROM software can quickly and accurately identify available slots, improving the efficiency and accuracy of resource identification. Marking the identified available slots as independent slot resources facilitates clear management of GPU slot resources. This marking method allows the system to intuitively distinguish which slots are available and which are occupied, facilitating subsequent resource allocation and scheduling. In large-scale data centers with numerous servers and complex PCIe slot resources, marking independent slot resources can greatly simplify resource management and improve management efficiency.

[0094] Configure the corresponding slot information for each new GPU BOX based on its independent slot resources, including the physical address and priority of the slot. Different GPU BOXes may have different performance and usage requirements. By configuring the physical address and priority, the GPU BOX can be matched with the most suitable slot. For example, for a GPU BOX with high performance requirements, a higher priority slot can be configured to ensure that it can obtain better resource support and improve operating efficiency.

[0095] The configuration of slot information provides the system with the ability to dynamically adjust resources. When the system load changes or new business needs arise, the slot information of GPU boxes can be reconfigured according to the actual situation, enabling flexible resource allocation. For example, during peak business periods, some GPU boxes can be moved to higher-priority slots to meet higher computing demands; during off-peak periods, the slot priority of some GPU boxes can be appropriately reduced to save resources.

[0096] Configuring accurate slot information for new GPU boxes can prevent resource conflicts. In multi-GPU systems, improper slot configuration can lead to multiple GPU boxes competing for the same resources, impacting system stability and performance. Proper slot configuration ensures each GPU box has independent resource space, reducing the likelihood of resource conflicts and improving system stability. Configuring slot information also enhances system compatibility with different types of GPU boxes. GPU boxes from different manufacturers may differ in interface standards, performance characteristics, etc. Configuring appropriate slot information allows the system to better adapt to these differences, ensuring that various types of GPU boxes function correctly within the system, thus improving system compatibility and versatility.

[0097] The GPU control system disclosed in this application can realize hardware-level hot-swapping, effectively solving the problem of needing to shut down, power off, and open the case when the server is not performing important tasks.

[0098] Specifically, this involves readjusting and implementing the ROM software on the target server motherboard to allocate specific, independent slot resources to the GPUs. Specifically, during the server production or initialization phase, technicians use specific programming tools (such as a programmer) to write configuration instructions to the motherboard ROM software. These instructions contain detailed information about GPU slot resource allocation, such as the physical address and priority of each GPU slot. Upon receiving the instructions, the ROM software updates its own configuration data. It rescans the PCIe slots on the motherboard, identifies slots specifically for the GPU BOX, and marks these slots as independent resources. Simultaneously, the ROM software establishes a slot resource mapping table, recording the status and related information of each GPU BOX slot.

[0099] This modifies the BIOS's monitoring and handling behavior during startup, detecting all critical hardware disconnections and connections. Specifically, technicians input commands to change the monitoring and handling behavior through the BIOS setup interface (usually accessed by pressing a specific key during server startup). These commands include setting the types of hardware to monitor (such as PCIe devices, memory modules, etc.) and event triggering conditions (such as time thresholds for hardware connection and disconnection). Upon receiving the commands, the BIOS starts a background monitoring program that periodically scans the status of hardware devices, monitoring the connection and disconnection of critical hardware in real time by detecting changes in PCIe bus signals and device insertion / removal detection pins. When an event is detected, the BIOS will handle it according to preset rules.

[0100] Furthermore, the application also includes: determining whether the monitored event is related to the GPU based on slot resources (if the slot where the event occurred is marked as a GPU slot in the slot resource mapping table, it is determined to be a GPU-related event). After detecting a hardware event, the monitoring program obtains information from the slot resource mapping table of the ROM software for judgment; if it is a GPU addition or removal event, it triggers a GPU hardware addition or removal notification to the CPU through the motherboard bus; if it is not a GPU addition or removal event, it triggers a major hardware failure notification to the CPU through the motherboard bus and triggers a protection mechanism for machine shutdown or restart.

[0101] By using a slot resource mapping table to determine whether monitored events are GPU-related, accurate classification of hardware events can be achieved. In complex server systems, there are numerous hardware events, and different types of events require different handling methods. Utilizing the slot resource mapping table for judgment allows the system to quickly and accurately distinguish between GPU-related events and other hardware events, providing a foundation for subsequent targeted processing.

[0102] Upon detecting a hardware event, the monitoring program directly retrieves information from the slot resource mapping table in the ROM software for judgment, avoiding complex analysis and detection processes and greatly improving the efficiency of event handling. This rapid response mechanism enables the system to react to events promptly, reducing potential risks caused by untimely event handling.

[0103] When an event is identified as a GPU addition or removal, a GPU hardware addition / removal notification is triggered to the CPU via the motherboard bus. This allows the CPU to be promptly informed of changes in GPU hardware and adjust system resource allocation and scheduling accordingly. For example, when a new GPU is added, the CPU can allocate more computing tasks to the new GPU, improving overall system performance; when a GPU is removed, the CPU can rebalance the load on other GPUs to ensure stable system operation. For events not involving GPU addition or removal, a protection mechanism is established to detect a major hardware failure and trigger a machine shutdown or restart. This timely protection prevents the hardware failure from escalating and avoids damage to other hardware components.

[0104] Precise event identification and targeted handling mechanisms help reduce the risk of system crashes. By promptly addressing GPU-related events and major hardware failures, the system can quickly adjust and protect itself when problems arise, maintaining stable operation. This stability is particularly important for mission-critical systems, reducing business interruptions and data loss caused by system failures.

[0105] The GPU control system disclosed in this application also includes configuration security processing logic. Specifically, a modification protection mechanism command is sent to the BIOS through the BIOS setup interface; in response to the modification protection mechanism command, the BIOS is triggered to execute preset modifications to its internal initial security processing logic. These preset modifications include: when the BIOS detects an addition or removal event of PCIe hardware, recording the event information and transmitting it to the target operating system, instead of controlling server restart or shutdown in the traditional way, thus achieving effective handling even when the server is powered on.

[0106] Modifying the protection mechanism through the BIOS settings interface allows the BIOS to log event information when it detects the addition or removal of PCIe hardware, providing the system with real-time monitoring capabilities for hardware changes. In environments with extremely high security requirements, such as financial institution data centers or military systems, any unauthorized hardware insertion or removal can pose security risks. Logging this event information helps administrators promptly detect abnormal hardware changes, enabling them to take appropriate security measures and prevent potential security threats, such as data leaks or system damage caused by malicious hardware implantation.

[0107] When a system malfunctions, recorded PCIe hardware addition or removal events can help administrators quickly pinpoint whether the problem is related to hardware changes. For example, if the system begins to malfunction after a specific point in time, and there are records of PCIe hardware insertions and removals at that time, the affected hardware can be prioritized for inspection and troubleshooting, significantly reducing troubleshooting time. This event information is also very helpful for hardware management and maintenance. Administrators can use the records to understand the frequency of hardware replacements, usage duration, and other information, allowing for the rational planning of hardware maintenance and update cycles. Furthermore, these records can be referenced during hardware upgrades or replacements to ensure the correctness and safety of the operations.

[0108] With technological advancements, the types and performance of PCIe hardware are constantly being updated. By configuring security processing logic, the system can better adapt to these hardware changes. When a new PCIe device is inserted or an old device is removed, the BIOS can promptly record this information and transmit it to the target operating system. The operating system can then adjust its configuration and drivers based on this information, ensuring the new hardware functions correctly and improving system compatibility with different hardware. By sending commands to the BIOS to modify the protection mechanism through the BIOS setup interface, users can flexibly adjust the system's security protection strategy according to their actual needs. Different application scenarios may have different requirements for managing hardware insertion and removal. Users can choose appropriate protection mechanisms based on their own security needs and business characteristics, achieving the optimal balance between security and flexibility.

[0109] The GPU control system disclosed in this application can realize reliable hot-swapping, effectively solving the problems of sudden changes in current and voltage and sparks caused by hot-swapping operations while the hardware is powered on, as well as electrostatic discharge caused by the power supply. This protects the motherboard and hardware chips and extends the lifespan of the GPU and the whole machine.

[0110] Specifically, when the status is "request to unplug," the GPU BOX system indicator light is first set to flash rapidly (e.g., 5 times per second) to indicate that the request has been received. If the system driver and hot-plug service software determine that the current status is "in use, request to unplug," they will first send a control signal to the GPU BOX system indicator light, requesting that the indicator light be set to flash rapidly (e.g., 5 times per second), indicating that the system has received the unplug request. After receiving the control signal, the GPU BOX system indicator light begins to flash rapidly at a frequency of 5 times per second. If the application-level hot-plug task processing software is not registered, the system driver and hot-plug service software will send a power-off command to the corresponding controller to control the power-off of that slot. The system saves the current status information of each slot and finally turns off all indicator lights, indicating that the hot-plug operation is complete.

[0111] Furthermore, the main control unit is preferably an I²C master control chip, whose input / output pins and interrupt pins are connected to the I²C control pins and interrupt pins of the CPU of the corresponding target server, respectively. The slave control unit is preferably an I²C slave controller, and different ranges of register values ​​of the I²C slave controller correspond to different state information of the hot-swap button. The I²C master control chip is connected to multiple I²C buses, and each I²C bus is connected to multiple I²C slave controllers.

[0112] Reference Figure 7 In this embodiment, the expansion of the number of GPUs connected to the server is implemented, therefore M is required to be greater than N, and each server must connect to at least two GPU groups. In the prior art, the number of PCIe slots directly supported by server CPUs and motherboards is generally limited, usually supporting 4-8. Taking a target server supporting 4 PCIe slots as an example, PCIe switch technology can be used for expansion. If the switch used supports 1 to 5, through expansion, a single server can connect 4*5 GPU boxes, that is, it can connect 4*5 GPUs. In specific operation, multiple GPU slots can be grouped according to switches, which can facilitate the grouping design and implementation of power supply, control, computing power, bus, etc.

[0113] Existing GPUs, due to their product structure and configuration, can only be plugged into server PCIe slots. Since PCIe power supply is limited and GPU power consumption can easily reach several hundred watts, additional power supplies from the server's power supply are required. In the solution disclosed in this embodiment, if a power supply group fails, such as due to a short circuit or overcurrent, only the individual GPU box it is paired with will be affected, without impacting other GPU boxes. This ensures that other GPU boxes in the entire server system can continue to operate normally, greatly improving system reliability. For example, in a large data center, hundreds of GPU boxes work collaboratively. Without independent power supply groups, a power failure could cause all GPUs to stop working, resulting in significant losses.

[0114] Different GPUs have fluctuating power requirements when running different tasks. Independent power supply groups can more precisely adjust power supply for the power changes of a single GPU box, reducing the impact of power fluctuations on other GPU boxes. For example, when a GPU box is performing complex deep learning calculations and its power consumption suddenly increases significantly, an independent power supply group can respond quickly to provide sufficient power without affecting the stable operation of other GPU boxes due to power fluctuations.

[0115] Furthermore, the power supply to a specific GPU box can be independently turned on or off according to actual needs. For example, during system maintenance, upgrades, or task scheduling, the power to a particular GPU box can be easily cut off without affecting the normal operation of other GPU boxes, effectively improving the flexibility and efficiency of system management. When a GPU box malfunctions, checking its matching power supply group can quickly pinpoint whether the fault is related to the power supply. Compared to multiple GPUs sharing a single power supply, this independent setting makes troubleshooting simpler and more accurate, shortening repair time.

[0116] Independent power supply groups can provide power on demand, tailored to the workload of different GPU boxes. When a GPU box is idle, its power supply can be reduced to minimize unnecessary power consumption; for example, at night when server load is low, energy can be saved by shutting down the corresponding power supply group for GPU boxes that are not currently in use. Different GPUs may have different power requirements, and independent power supply groups can be configured according to the specific power requirements of each GPU box, avoiding the problems of wasted or insufficient power resources caused by uniform power supply; for example, GPU boxes with lower power requirements can be equipped with smaller power supply groups, thereby optimizing the power resource allocation of the entire server system.

[0117] In this embodiment, a voltage monitoring chip, such as MAX6393, can be used to monitor the power supply voltage of the PCIe interface in real time. When an abnormal voltage is detected, measures can be taken in time to control the power on and off to avoid damage to the device. At the same time, a current sensor, such as INA219, can be used to monitor the current consumption of the PCIe device. The working status of the device can be determined based on the current magnitude. When the device is in standby or idle state, a power-off operation can be performed in a timely manner to save energy.

[0118] Hot-swap controller chips like the LTC4261 can also be used to safely control the power supply of the PCIe interface when the GPU is inserted or removed, preventing voltage spikes and current surges during insertion and removal, and protecting the server and PCIe devices from damage. When the GPU box is detected to be inserted, the hot-swap controller will gradually power on the PCIe interface according to a preset timing sequence; and when the GPU box is detected to be removed, it will smoothly power down.

[0119] In this application, when the GPU box is about to be removed, the detection circuit senses the insertion or removal action and quickly sends a signal to the system. For example, when the GPU box begins to be removed, the detection circuit detects a change in voltage or current at the interface and immediately notifies the system that a hardware removal operation is imminent. Upon receiving the signal that the hardware is about to be removed, the system activates the corresponding protection program. The operating system suspends all data transmission and processing tasks related to the GPU in the GPU box, ensuring that data flow stops safely. For example, ongoing graphics rendering tasks will be temporarily interrupted to prevent data loss or corruption.

[0120] A data buffer typically exists between the GPU and the system to temporarily store data being transferred. When a hot-plug operation is detected, the buffer saves any data that has not yet been processed. Like water flowing through a reservoir, the reservoir retains some water before the pipe is disconnected, awaiting appropriate processing. Before the GPU box is removed, the system ensures that the data in the buffer is synchronized with other storage devices or system components, guaranteeing data integrity and preventing the loss of important information even during a hot-plug operation. For example, data being transferred from the GPU to memory will be completed before removal, or at least safely saved to the buffer.

[0121] In this embodiment, the GPU BOX is connected to the computer system via a PCIe interface. The PCIe standard defines a complete hot-plug protocol, which specifies the behavior of hardware and software during hot-plugging, ensuring that the system can handle hot-plug events in an orderly manner. For example, the PCIe hot-plug protocol requires the system to release relevant resources in a specific order when it detects that the GPU BOX has been removed, to avoid system crashes or data corruption.

[0122] The phrase "different range register values ​​of the I²C slave controller correspond to different state information of the hot-swap button" can be understood as follows: the hot-swap button will have different states under different operations, such as not pressed, pressed, long press, etc. We reflect these states by setting the register values ​​of the I²C slave controller. When the button state changes, the register value changes accordingly and triggers an interrupt to notify the master controller.

[0123] Suppose we use an 8-bit register to store the status information of the hot-swappable button. The following are the different ranges of register values ​​and their corresponding button states: 1) Not pressed: When the button is not pressed, the register value remains 0x00. 2) Short press: When a button press signal is detected, a timer starts. If a button release signal is detected within a set short press time threshold (e.g., 200ms), the register value is set to a value in the range of 0x01-0x0F, for example, 0x01. 3) Long press: If the button press time exceeds a set long press time threshold (e.g., 1000ms), the register value is set to a value in the range of 0x10-0x1F, for example, 0x10. 4) Abnormal state: If a hardware failure or other abnormal situation occurs during the detection process, the register value is set to a value in the range of 0x20-0xFF, for example, 0x20. When a change in the button state causes a change in the register value, the I²C slave controller triggers an interrupt signal to the master controller. After receiving the interrupt signal, the master controller reads the register value through the I²C bus to obtain the current status information of the button. Using the above approach, we can implement different register values ​​corresponding to different state information of the hot-swappable button. When the button state changes, the register value changes accordingly and triggers an interrupt signal. The main controller can read the register value through the I²C bus to obtain the current state of the button.

[0124] The GPU control system disclosed in this application, through the structural design of the GPU BOX and the corresponding functional software design, proposes to achieve GPU hardware-level, system security-level, and application security-level hot-swapping based on practical needs. Hardware-level hot-swapping refers to the real-time monitoring and response to GPU insertion and removal through the hardware design of the I2C bus and the main controller. System reliability-level hot-swapping ensures the reliability of insertion and removal operations, preventing system instability or failure during hot-swapping. Application security-level hot-swapping ensures the reasonable scheduling of GPU resources at the application level, guaranteeing stable application operation and achieving load balancing.

[0125] During hot-plug event handling, BIOS upgrades and settings specifically include: reserving PCIe resources in the BIOS's PCI Bus Driver to allocate memory and I / O resources for all PCIe RPs supporting Hot Plugs. For example, the BIOS reserves 4KB of I / O, 16MB of Non-Prefetchable Memory, and 16MB of Prefetchable Memory resources for the PCIe RP. A Hot Plug Control has been added to the BIOS Setup Menu. This control option has multiple bits, each corresponding to a subnode. For example, Bit 0 controls the Hot Plug on PCIe Root Port D4F0 on Subnode 0. When reporting Hot Plug Events, the LID# Pin is used as the SCI input signal source to be transmitted to the CPU. Therefore, the LID# Pin SCI interrupt reporting registers need to be configured during BIOS initialization, such as LID# Pin SCIEnable and the interrupt triggering method.

[0126] When a PCIe Root Port is hot-plugged, an Uncorrectable Error is generated. This type of error may cause an MCE (Machine Check Architecture), which can cause the Host OS to stop working. Since Surprise Down Errors are unavoidable during the Hot Plug process, the BIOS is configured to disable Surprise Down Errors for PCIe RPs that support Hot Plugs, preventing them from being reported to the MCA (Machine Check Architecture).

[0127] The OS settings specifically include: adopting the HotPlug solution based on SCI Interrupt ACPI Hotplug; and the OS needs to support the handling of ACPI Hotplug events. When compiling the kernel, it is necessary to check and configure the kernel to support ACPI Hotplug.

[0128] Reference Figure 8 The GPU control system disclosed in this application also includes an RDMA switch module. Within a single GPU BOX, the GPU module, control module, storage module, and network module are all connected to the PCIe interconnect module via a first type of PCIe protocol. Each GPU BOX on the same node is connected to the PCIe switch via a second type of PCIe protocol. Each GPU BOX on different nodes is connected to the RDMA switch module via a third type of PCIe protocol. The version of the second type of PCIe protocol is lower than the version of the first type of PCIe protocol, the version of the third type of PCIe protocol is the same as the version of the first type of PCIe protocol, and the version of the fourth type of PCIe protocol is no higher than the version of the second type of PCIe protocol.

[0129] Among them, the first type of PCIe protocol is preferably the PCIe-Gen5 / 6 protocol, the second type of PCIe protocol is preferably the PCIe-Gen4 / 5 protocol, and the fourth type of PCIe protocol is preferably the PCIe-Gen3 / 4 / 5 protocol.

[0130] In this embodiment, the CPU unit may include several servers, each server connected to one or more GPU boxes; each GPU box is an independent hot-swappable unit; when at least two GPU boxes are installed on the same server, a single-server multi-box mode is formed. Through the solution provided in this application, users can flexibly adjust GPU resources according to actual business needs. For example, when performing large-scale deep learning training, more GPU units can be added to the server to enhance parallel computing capabilities; when the workload is small, the number of connected GPU units can be reduced to lower energy consumption and costs.

[0131] Each GPU BOX on the same node is connected to the PCIe Switch via the Type II PCIe protocol; the version of the Type II PCIe protocol is lower than the version of the Type I PCIe protocol; GPU modules in any two GPU BOXes on the same node establish a GPUDirect P2P connection channel, enabling GPUs to directly perform P2P DMA communication using the PCIe bus.

[0132] Specifically, the GPU module integrates a DMA controller connected to the GPU card, used to control data read / write operations on the GPU card and data interaction with the PCIe switch chip. The storage module integrates a DMA controller connected to the storage chip, used to control data read / write operations on the storage chip and data interaction with the PCIe switch chip. The P2PDMA supported channel is the transmission channel between the DMA controllers within the GPU module and the storage module.

[0133] Furthermore, the control module is preferably an integrated system-on-a-chip, i.e., SoC, which is responsible for the initialization and configuration of the DMA controller inside the GPU module and the DMA controller inside the storage module. It can support P2PDMA (i.e., point-to-point direct memory access) without the participation of the server CPU, that is, it can directly read data from the storage module without passing through the server CPU and memory.

[0134] Each GPU BOX on different nodes is connected to the RDMA switch module via the Type 3 PCIe protocol; the version of the Type 3 PCIe protocol is consistent with the version of the Type 1 PCIe protocol; GPU modules in each pair of GPU BOXes on different nodes establish GPUDirect RDMA (i.e., GDR) connection channels, and cross-node GPU interconnection is achieved through the RDMA switch module.

[0135] In this design, the CPU unit (i.e., the server) and the GPU BOX are not on the same board and are connected via Type 4 PCIe protocol, with the Type 4 PCIe protocol version no higher than that of Type 2 PCIe protocol. Specifically, the CPU unit manages the GPU module, network module, and storage module through the PCIe bus. This non-shared-board structure eliminates the dependence on PCIe interface switching devices and retimers.

[0136] The GPU BOX and CPU unit are connected via a hot-swappable interface. The GPU BOX is integrated within the server and connected via a hot-swappable interface based on a specific high-speed data transmission protocol. This interface provides high-speed, low-latency data transmission, enabling real-time data interaction and command transfer between the GPU BOX and CPU unit. Furthermore, the GPU BOX can be plugged in or unplugged at any time during normal server operation without causing electrical damage to the server system or GPU BOX, ensuring system stability and data integrity. The interface design ensures stable data transmission and prevents signal interference and electrical problems during plugging and unplugging.

[0137] In this embodiment, the CPU unit is only responsible for issuing initialization configuration instructions, transmission control, and monitoring tasks. In this architecture, tasks originally handled by the server CPU are now handled by the SoC chip inside the GPU BOX, eliminating the need for the server CPU and memory. The internal PCIe interconnect module primarily functions as a data routing and switching mechanism, ensuring accurate data transfer between the storage module and the GPU module. Therefore, in this simple one-to-one connection scenario, the DMA controllers of the storage module and the GPU module are mainly involved in the P2P DMA transfer process. Thus, through the internal settings of the GPU BOX, users and applications can be provided with the ability to directly transfer data from the storage device to the GPU memory. The preferred PCIe protocol is PCIe-GEN3 / 4 / 5.

[0138] In this embodiment, the channels supporting P2P DMA, GPUDirect P2P connection channels, and GPUDirect RDMA connection channels do not occupy CPU lanes, meaning they do not occupy the physical channels for data transmission between the CPU and other devices, i.e., the PCI-Express (PCIe) bus channels. The CPU exchanges data with these devices through PCIe channels. Each PCIe channel has a certain bandwidth for data transmission. The more channels there are, the greater the total data transmission bandwidth. However, the number of PCIe channels that the CPU can provide is limited. By setting up channels that do not occupy CPU lanes as described in this application, CPU intervention is effectively reduced, greatly improving the data transmission speed between local devices, while also effectively reducing data transmission latency.

[0139] To maximize the performance of the GPU card and disk, this three-segment intelligent computing architecture system preferentially uses a higher version of PCIe-GEN5 / 6 (i.e., the first type of PCIe protocol) for data transmission in this functional area inside the GPU BOX. Without occupying the CPU Lane, it supports each GPU card to establish a link with the high-speed network chip and realize direct read and write access to the disk.

[0140] The application discloses a three-segment intelligent computing architecture system based on the PCIe protocol, which enables communication and interaction between GPU cards in different GPU boxes without occupying CPU lanes.

[0141] The first segment of the intelligent computing architecture focuses on the interconnection of modules within the GPU BOX. This allows the GPU unit itself to be independently optimized for performance and expanded for functionality. For example, for specific computing tasks, such as convolution operations in deep learning, the computing cores and cache modules inside the GPU can be customized and upgraded without affecting other parts of the entire system.

[0142] The second segment of the intelligent computing architecture covers the interaction between different GPU BOXes on the same node and different nodes. This design allows the system to easily improve its overall computing power by increasing the number of GPU units. Whether it is adding GPU cards within a single server node or expanding the GPU cluster across multiple server nodes, it can be seamlessly expanded based on the existing interaction mode.

[0143] The third segment of the intelligent computing architecture handles the interaction between the GPU BOX and the CPU unit, allowing the system to flexibly adjust the GPU and CPU ratio according to actual needs. In scenarios requiring a large amount of parallel computing, the use of GPUs can be increased; while in tasks requiring complex logic control and serial processing, the CPU can play a greater role.

[0144] The disclosed solution supports diverse application scenarios. In practical applications, different scenarios have different requirements for computing resources. The three-stage intelligent computing architecture can flexibly adjust the configuration of each stage according to the characteristics of the specific application. For example, in the field of scientific computing, more emphasis may be placed on the parallel computing capabilities of GPUs, so the number of GPU units in the second stage architecture can be increased; while in the field of real-time data processing, closer collaboration between CPU and GPU may be required, in which case the interaction mode of the third stage architecture can be optimized.

[0145] Furthermore, the solution disclosed in this application can improve the portability of applications; through the relatively independent settings of each segment of the intelligent computing architecture, applications can be more easily ported between different system configurations. For example, a GPU-based deep learning application can be developed and tested on a single GPU system, and then easily scaled to a multi-GPU cluster environment by adjusting the second and third segment architectures.

[0146] This application adopts a three-segment architecture, modularizing the GPU BOX, PCIe Switch, RDMA switch module, and CPU unit. Each module is relatively independent, which facilitates flexible configuration and expansion according to different application scenarios and needs. For example, the number of GPU BOXes can be increased or decreased according to the scale of the computing task. A hot-swappable connection channel is established between the GPU BOX and the CPU unit, which allows for easy plugging and unplugging of related components during system operation without shutting down the entire system, thus improving the maintainability and availability of the system.

[0147] In this application, different modules are connected using different types of PCIe protocols. This avoids the cost increase caused by uniformly adopting high-version PCIe standards. Appropriate PCIe protocol versions can be selected based on the actual needs of each module. Modules with low bandwidth requirements use lower-version PCIe protocols, reducing component costs. By rationally allocating PCIe protocol versions, the increased power consumption caused by low-speed devices operating in high-version PCIe mode is effectively avoided. For example, low-bandwidth devices using lower-version PCIe protocols have relatively lower power consumption, which helps reduce the overall system's heat dissipation pressure and energy consumption, conforming to the principles of green computing.

[0148] This application employs a three-stage intelligent computing architecture based on the PCIe protocol, distributing computing resources across multiple GPU boxes, unlike traditional CPU-centric designs where all devices rely on the CPU's PCIe channels. Each GPU box has its own internal PCIe interconnect module, enabling communication between internal devices and reducing reliance on CPU-provided PCIe channels. By establishing channels supporting P2P DMA, GPUDirect P2P connection channels, and GPUDirect RDMA connection channels, direct data transfer between devices is possible, bypassing the CPU and further alleviating the pressure on CPU PCIe channel allocation. For example, the P2P DMA channel between the GPU and the storage module allows them to exchange data directly without occupying CPU PCIe channels.

[0149] This application employs Type I, Type II, Type III, and Type IV PCIe protocols for connection based on the actual needs of different modules. For low-speed peripherals, a lower version of the PCIe protocol can be selected to improve bandwidth utilization and avoid the waste of physical layer resources caused by using a higher version of the PCIe interface. For example, for devices such as USB controllers and low-speed network cards, a lower version of the PCIe protocol can be used to meet their data transmission requirements without wasting the bandwidth of a higher version of the PCIe interface.

[0150] This application does not mandate the adoption of the latest PCIe standard. Instead, it selects appropriate PCIe protocol versions based on the bandwidth requirements of each module. For low-bandwidth devices, such as SATA controllers and 1GbE network cards, using lower versions of the PCIe protocol reduces component costs. For example, instead of using a PCIe 5.0 x4 network card to meet the requirement of only 1 Gb / s bandwidth, a more suitable low-bandwidth network card is chosen, avoiding resource waste and cost increases. Because different types of PCIe protocols are used, existing PCIe 3.0 / 4.0 devices can be made compatible by selecting appropriate PCIe protocols, reducing hardware iteration costs and enabling smooth upgrades. For example, some older PCIe 3.0 devices can be connected to modules using appropriate lower-version PCIe protocols to continue functioning. By rationally configuring the PCIe protocol version according to the device's performance and bandwidth requirements, the increased power consumption caused by uniformly adopting high-version PCIe standards is avoided. Low-speed devices operating in appropriate lower-version PCIe modes reduce energy consumption per unit of data throughput and correspondingly reduce overall system cooling pressure, conforming to green computing principles. For example, PCIe 4.0 devices consume less power than PCIe 5.0 devices. Using the PCIe 4.0 protocol can effectively reduce power consumption for some devices that do not have high bandwidth requirements.

[0151] In this application, the key components use a higher version of the PCIe protocol, the secondary components use a medium version of the PCIe protocol, and the scheduling and management functions use a general version of the PCIe protocol. Through this three-segment PCIe structure design, not only is the performance of the intelligent computing server improved by more than 30%, but the overall hardware cost of the intelligent computing server is also reduced by more than 20%.

[0152] The intelligent computing server, employing a three-segment PCIe architecture, supports the coexistence of different versions of the PCIe protocol, effectively avoiding hardware silos that may arise during technology iterations. Whether it's advanced PCIe 6.0 devices or widely used PCIe 4.0 or 3.0 components, this design enables seamless integration, ensuring efficient collaborative operation. This design feature guarantees that users do not need to completely replace old equipment when upgrading hardware, thus saving costs and extending the hardware's lifespan. Furthermore, through precise protocol management, conflicts between different signals and protocols can be effectively resolved, ensuring the stability and efficiency of data transmission. Support for hybrid protocols eliminates technology silos and effectively resolves conflicts between signals and protocols.

[0153] The three-segment PCIe architecture allows different functional areas to operate corresponding components according to different PCIe protocols, effectively reducing the overall power consumption of the intelligent computing system by 15%-25%, thereby reducing heat dissipation requirements. Simultaneously, high-frequency signal interference is effectively suppressed, thus improving the stability and reliability of the intelligent computing system. Furthermore, this design simplifies the maintenance and management process of the intelligent computing server, as the independence of each component makes troubleshooting and resolution more convenient.

[0154] The GPU control system disclosed in this application includes a GPU BOX, a target server, a power supply group, an expansion backplane enclosure, a main control unit, and a slave control unit. The expansion backplane enclosure includes a first accommodating area and a second accommodating area. The first accommodating area includes N first mounting positions for mounting N target servers. The second accommodating area includes M second mounting positions, each first mounting position corresponding to P second mounting positions, and each second mounting position for mounting one GPU BOX. The GPU BOX has a locking protrusion on its side, and the inner wall of each second mounting position has a locking groove matching the locking protrusion. A limiting hole is provided in each second mounting position; in a third state, the locking part is located within the locking hole and abuts against the inner wall of the locking hole. M power supply groups are provided, each power supply group matching the power module of one GPU BOX. The main control unit is configured in the motherboard of the corresponding target server and is used to connect to the slave control unit. Each slave control unit is connected to a hot-swap button of a GPU BOX to obtain the status information of the corresponding hot-swap button in real time and send it to the CPU of the corresponding target server through the main control unit. The CPU of the target server controls the corresponding GPU based on the status information signal. The BOX features hot-swapping capabilities. This system is compatible with various GPU models, allowing for individual GPU replacements and power-offs without interrupting part or the entire task or requiring a power outage. This significantly improves system availability and efficiency. The GPU BOX allows for quick connection to different target server slots, offering simplicity and efficiency. It reduces reliance on professional personnel, lowers manpower and material costs, and greatly reduces the risk of hardware damage or data loss due to improper operation (such as static electricity or power mismatch). The corresponding power supply group effectively solves the problem of traditional GPUs requiring additional power lines from the server power supply due to limited PCIe power, ensuring stable power supply for each GPU.

[0155] For large data centers or high-performance computing environments, frequent hardware replacements can lead to a significant increase in operating costs. By reducing downtime and simplifying operational processes, this system can reduce operating costs and improve economic efficiency.

[0156] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A GPU BOX, characterized by, include: The housing has a recess for mounting the GPU module; The enclosure houses a control motherboard, a network module, and a power module connected to the GPU module. The control motherboard integrates a control module, a storage module, a PCIe interconnect module, and a slave control unit. The GPU module, the control module, the storage module, and the network module are all connected to the PCIe interconnect module. The control motherboard has a PCIe slot for inserting the GPU module. A connector is installed on the control motherboard, and the network module is connected to the target server through the connector. The connector and the power module have their ends extending out of the housing, and both the ends of the connector and the power module are matched with the plug-in slots of the target server.

2. The GPU BOX according to claim 1, wherein, The GPU module integrates a first DMA controller, and the storage module integrates a second DMA controller. A P2P DMA-enabled channel is established between the GPU module and the storage module; the P2P DMA-enabled channel is a transmission channel between the first DMA controller and the second DMA controller. The GPU module establishes a GPUDirect P2P connection channel with other GPU modules in the same target server. The GPU module establishes a GPUDirect RDMA connection channel with other GPU modules in different target servers.

3. The GPU BOX according to claim 1, wherein, The connector has a multi-row hole structure at its end, and the connector is used to lead the network signal of the network module to the OSFP interface on the target server. The end of the power module has a hole-like structure.

4. The GPU BOX according to claim 1, wherein, A push-pull limiting device is provided on the outside of the box; The push-pull limiting device is used to assist in pushing the box to engage and fix with the target server in the first state, to assist in disengaging the box from the target server in the second state, and to lock the relative position of the box and the target server in the third state.

5. The GPU BOX according to claim 4, characterized in that, The push-pull limiting device includes a first rod segment, a second rod segment, and a screwing component. The first rod segment is connected to the outside of the box body via a pin and has the freedom to rotate around the pin. The housing has a snap-fit ​​hole on its first side and a through hole on the opposite side of the first side. The through hole is used to suspend the ends of the connector and the power module. The first rod segment has a locking portion; the second rod segment is located at the end of the first rod segment away from the pin, and the second rod segment has a mounting portion for mounting the screwing component; In both the first and second states, the second rod segment is far from the box and does not contact the box. In the third state, the second rod segment or the first rod segment rotates around the pin under the action of external force until it fits against the side wall of the box body, the engaging part abuts against the preset position of the target server and the screwing part is fixed to the engaging hole.

6. The GPU BOX according to claim 5, characterized in that, A plurality of first ventilation holes are provided on the first side, and a plurality of second ventilation holes are provided on the opposite side of the first side, with the plurality of first ventilation holes and the plurality of second ventilation holes being arranged opposite to each other; The ventilation area formed by the first ventilation holes and the ventilation area formed by the second ventilation holes are both matched and configured to match the GPU module.

7. The GPU BOX according to claim 1, wherein, The size of the receiving slot is not less than the size of the GPU module, and the receiving slot is compatible with different types of GPU cards.

8. A GPU control system, comprising: include: The expansion backplane enclosure includes a first accommodating area and a second accommodating area. The first accommodating area includes N first mounting positions for corresponding installation of N target servers. At least one of the target server's slots is equipped with a PCIe Switch expansion card, the PCIe Switch expansion card having a plurality of downstream ports, and at least two of the downstream ports being Box mounting slots; the second accommodating area includes M second mounting positions, the second mounting positions being used to carry a GPU BOX as described in any one of claims 1-7, the GPU BOX being used to mount a GPU; Power supply groups; M power supply groups are configured, and each power supply group is matched with one of the GPU BOXes; The main control unit is located on the motherboard of the corresponding target server; Each slave control unit is connected to the main control unit, and each slave control unit is connected to a hot-swap button of the GPU BOX. The slave control unit is used to obtain the status information of the corresponding hot-swap button in real time and send it to the corresponding target server through the main control unit.

9. The GPU control system of claim 8, wherein, The main control unit is an I²C main control chip, and the input / output pins and interrupt pins of the I²C main control chip are respectively connected to the I²C control pins and interrupt pins of the CPU of the corresponding target server. The control unit is an I²C slave controller, and different ranges of register values ​​of the I²C slave controller correspond to different state information of the hot-swap button. The I²C master control chip is connected to multiple I²C buses, and each I²C bus is connected to multiple I²C slave controllers.

10. The GPU control system of claim 8, wherein, The GPU BOX has a snap-fit ​​protrusion on its side, and the inner wall of the second mounting position has a snap-fit ​​groove that matches the snap-fit ​​protrusion.

Images