AI server supernode-based gpu redundancy arrangement method and device, and equipment medium

Abstract

This disclosure provides a method, apparatus, and device medium for GPU redundancy deployment based on AI server supernodes. The method includes: acquiring and adjusting the internal and external interfaces of GPUs in a modular computing unit; configuring link paths for the nine GPUs in the modular computing unit to achieve pairwise interconnection of the nine GPUs; during the operation of the modular computing unit, setting eight of the nine GPUs as regular GPUs and setting the remaining one as a redundant GPU; when any one of the eight regular GPUs is detected to have failed, adjusting the remaining seven regular GPUs to achieve pairwise interconnection with the redundant GPU. In the embodiments of this disclosure, by adjusting GPU interface allocation, introducing a configurable optical switching module, and a redundancy management mechanism, the reliability and fault tolerance of the supernode system are improved without significantly increasing hardware costs.

Description

Technical Field

[0001] This disclosure relates to the field of supernode computing interconnection and communication, specifically to a method, apparatus, and device medium for redundant GPU deployment based on AI server supernodes. Background Technology

[0002] Currently, to meet the demands of parallel computing, supernode computers commonly employ fullmesh interconnect architectures to achieve all-to-all communication topologies. This results in extremely high requirements for the number and bandwidth of GPU interconnect interfaces. However, due to chip area limitations and the highly concentrated investment of computing resources, a single chip often cannot provide enough communication interfaces and corresponding bandwidth. Furthermore, a reasonable match must be maintained between computing power and communication bandwidth. For example, the current Blackwell GPU chip only provides 18 communication interfaces for scale-up interconnects. It should be understood that scale-up can be achieved through switching and fullmesh methods, and the fullmesh method inherently faces the challenge of flexibly adding redundant nodes during expansion. Under this constraint, how to utilize these limited 18 communication interfaces to build an efficient and scalable communication network to meet the communication needs of scale-up scenarios becomes a critical design problem.

[0003] In some existing solutions, such as some instance GPU chips, only 14 communication interfaces (PCIe G5X8) can be provided. The DF500 supernode, however, is designed with 8 GPUs per card forming an intra-blade mesh. Each GPU also has 7 communication interfaces to connect to the compute blades and can form meshes between 8 blade groups. This allows the entire supernode network to support up to 64 GPUs. However, such a supernode network lacks GPU redundancy. In actual network operation, GPUs, due to their high power consumption and complexity, are often the biggest weakness in reliability. Conventional reliability designs require redundant GPUs to improve the overall reliability of the supernode.

[0004] Therefore, existing technologies still need to be improved and enhanced.

[0005] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention

[0006] To address at least one of the aforementioned problems, as well as one or more other potential problems, this disclosure proposes a GPU redundancy deployment method, apparatus, and device medium based on AI server supernodes. By adjusting GPU interface allocation, introducing configurable optical switching modules, and redundancy management mechanisms, the reliability and fault tolerance of the supernode system are improved without significantly increasing hardware costs.

[0007] The first aspect of this disclosure proposes a GPU redundancy deployment method based on AI server supernodes, comprising: acquiring and adjusting the internal and external interfaces of GPUs in a modular computing unit; configuring link paths for the nine GPUs in the modular computing unit so that the nine GPUs are interconnected in pairs via link paths; during the operation of the modular computing unit, setting eight of the nine GPUs as regular GPUs and setting the remaining one of the nine GPUs as a redundant GPU; when any one of the eight GPUs set as regular GPUs is detected to have failed, adjusting the remaining seven GPUs set as regular GPUs to be interconnected in pairs with the redundant GPU via link paths.

[0008] Furthermore, in some embodiments, the above-mentioned acquisition and adjustment of the internal and external interfaces of the GPU in the modular computing unit includes: adjusting the 7 internal and 7 external interfaces of the GPU in the modular computing unit to 6 internal and 8 external interfaces.

[0009] Furthermore, in some embodiments, configuring the link paths of the nine GPUs in the modular computing unit to enable the interconnection of the link paths between the nine GPUs includes: configuring the link paths of eight GPUs in the first modular computing unit to enable the interconnection of the link paths between the eight GPUs; and configuring the link paths of eight GPUs in the second modular computing unit to enable the interconnection of the link paths between the eight GPUs.

[0010] Furthermore, in some embodiments, the above-mentioned link path configuration of the eight GPUs in the first modular computing unit of the modular computing unit to achieve pairwise interconnection of the link paths of the eight GPUs includes: connecting the six internal interfaces of the first GPU in the first modular computing unit to the third, fourth, fifth, sixth, seventh, and eighth GPUs respectively; connecting the eight external interfaces of the first GPU in the first modular computing unit to the eight OCS optical modules in the first OCS optical module group respectively, and connecting the eighth OCS optical module to one internal interface of the first to eighth GPUs; and configuring the first GPU to form an internal loopback with the second GPU via the eighth OCS optical module, thereby realizing the interconnection of the link paths of the first GPU and the second GPU.

[0011] Furthermore, in some embodiments, the above-mentioned link path configuration of the eight GPUs in the second modular computing unit of the modular computing unit to achieve pairwise interconnection of the link paths of the eight GPUs includes: connecting the six internal interfaces of the ninth GPU in the second modular computing unit to the eleventh, twelfth, thirteenth, fourteenth, fifteenth, and sixteenth GPUs respectively; connecting the eight external interfaces of the ninth GPU in the second modular computing unit to the eight OCS optical modules in the second OCS optical module group respectively, and connecting the sixteenth OCS optical module to one internal interface of the ninth to sixteenth GPUs; configuring the ninth GPU to form an internal loopback with the tenth GPU via the sixteenth OCS optical module, thereby realizing the interconnection of the link paths of the ninth GPU and the tenth GPU; and realizing pairwise interconnection of the link paths between the eight OCS optical modules in the second OCS optical module group and the eight OCS optical modules in the first OCS optical module group.

[0012] Furthermore, in some embodiments, setting 8 of the 9 GPUs as regular GPUs and setting the remaining 1 of the 9 GPUs as a redundant GPU includes: setting 8 GPUs in the first modular computing unit of the modular computing unit as regular GPUs, and setting any one of the 8 GPUs in the second modular computing unit of the modular computing unit as a redundant GPU.

[0013] Furthermore, in some embodiments, the above-mentioned adjustment of the remaining 7 GPUs among the 8 GPUs set as regular GPUs to establish a link path for pairwise interconnection with the redundant GPU when one GPU among the 8 GPUs set as regular GPUs is detected to fail includes: when any one of the 8 GPUs in the first modular computing unit is detected to fail, adjusting the remaining 7 GPUs in the first modular computing unit to establish a link path for pairwise interconnection with the redundant GPUs among the 8 GPUs in the second modular computing unit.

[0014] In a second aspect of this disclosure, a GPU redundancy deployment device based on AI server supernodes is also proposed, comprising: an interface adjustment module for configuring the number of internal and external interfaces of the GPU; a path configuration module for controlling the optical switching module to realize interconnection between GPUs; a status management module for setting the GPU working mode and monitoring its status; and a fault switching module for performing path switching when a GPU fails.

[0015] In a third aspect of this disclosure, an electronic device is proposed, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0016] In a fourth aspect of this disclosure, a computer-readable storage medium is further provided, wherein the storage medium stores at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by a processor to implement the steps of the method described above.

[0017] This disclosure has the following advantages over the prior art:

[0018] In some embodiments, GPU redundancy is achieved through network topology reconstruction without increasing the number of GPU chips, significantly improving system reliability. In some embodiments, real-time detection and automatic switching of GPU failures are supported, minimizing computing task downtime. In some embodiments, it is compatible with existing GPU hardware architectures, requiring no changes to chip design and resulting in low implementation costs. In some embodiments, it is highly scalable and applicable to GPU clusters of different sizes. Attached Figure Description

[0019] The above and other features, advantages and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description, wherein:

[0020] Figure 1 A schematic diagram of the GPU's internal and external interfaces is shown based on some examples;

[0021] Figure 2 Several examples are shown. Figure 1 A schematic diagram illustrating the interconnection between GPUs in the system;

[0022] Figure 3 A schematic diagram of the network connections of an 8×8 supernode network based on some examples is shown;

[0023] Figure 4 A schematic diagram of a redundant improved GPU internal / external interface according to some embodiments of the present disclosure is shown;

[0024] Figure 5 Several embodiments according to this disclosure are illustrated. Figure 4 A schematic diagram illustrating the interconnection between GPUs in the system;

[0025] Figure 6 A schematic diagram of a network connection with redundant nodes according to some embodiments of the present disclosure is shown;

[0026] Figure 7 A network connection diagram of an 8×9 supernode network according to some embodiments of the present disclosure is shown;

[0027] Figure 8 Some embodiments of the present disclosure are shown. Figure 7 A flowchart illustrating the process of managing and configuring GPU operating status and redundancy in a supernode network;

[0028] Figure 9 A GPU redundancy deployment method based on AI server supernodes according to some embodiments of the present disclosure is illustrated; and

[0029] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0030] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "this embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first state may also be referred to as a second state, and similarly, a second state may also be referred to as a first state. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] It should be understood that current supernode computers, due to the demands of parallel computing, mostly use a fully connected mesh interconnect structure to achieve an All-to-All communication topology (where All-to-All communication is generally referred to as All-to-All Communication, abbreviated as A2A, a collective communication mode). This generates a massive demand for GPU interconnect communication interfaces. However, chips, limited by area and concentrated resource allocation for computing, often cannot provide many communication interfaces and bandwidth, and there is also a certain mismatch between computing resources and communication bandwidth. For example, NVIDIA's Blackwell GPU chip currently only provides 18 communication interfaces for scale-up communication. So how can these 18 communication interfaces be used to build a reasonable communication network to meet the communication needs of the scale-up domain? Based on the first-generation GPU chip of the target product, an implementation scheme is proposed for the current scenario that only provides 14 communication interfaces (PCIE G5 X8), such as... Figure 1 As shown, the supernode is designed with 8 GPUs per card forming an intra-blade mesh. Each GPU also has 7 communication interfaces to communicate with the compute blades and can form meshes between 8 blade groups. Thus, the entire supernode network can support up to 64 GPUs. Figure 2-3As shown. However, such a supernode network does not have a GPU redundancy design. In actual network operation, GPUs are usually the biggest weakness in reliability due to their high power consumption and complexity. For example, if the GPU of one node fails, it will significantly reduce the operating performance of the entire supernode network. In conventional reliability design, redundant GPUs need to be designed to improve the reliability of the entire supernode.

[0034] The following will provide a more detailed explanation in conjunction with the accompanying drawings.

[0035] In some instances, such as Figure 1 As shown, each GPU was originally designed with 7 internal interfaces (represented by solid lines for distinction) and 7 external interfaces (represented by dashed lines for distinction), meaning the target GPU chip provides 14 high-speed SerDes interfaces, traditionally planned as 7 for blade-interconnect (internal) and 7 for blade-to-blade interconnect (external). Then, based on Figure 1 The GPU in the example, further, such as Figure 2 and Figure 3 As shown, a supernode consists of 8 modular computing units, each containing 8 GPUs, forming an 8×8 network structure without redundancy. However, such a supernode network lacks GPU redundancy design. In actual grid-connected operation, GPUs, due to their high power consumption and complexity, are usually the biggest weakness in reliability. In conventional reliability design, redundant GPUs are needed to improve the overall reliability of the supernode.

[0036] For redundancy, in some embodiments, such as Figure 4 As shown, the configuration is adjusted to have 6 internal interfaces and 8 external interfaces. This is equivalent to repurposing the 14 interfaces or wiring. The 6 internal interfaces are still represented by solid lines, while the rightmost solid line among the 8 external interfaces is used to more clearly indicate that it was originally an internal interface or wiring. That is, 6 interfaces are configured as internal interfaces to connect to other GPUs within the same compute blade; the remaining 8 interfaces are configured as external interfaces to connect to 8 optical modules (such as QSFP-DD) on the blade carrier, thereby accessing the rack-level optical circuit-switched (OCS) network. After the adjustment, the GPUs within each modular compute unit achieve internal interconnection through 6 interfaces, and the remaining 8 interfaces connect to the optical switching module (OCS). Further, in some embodiments, see... Figure 5 and Figure 6The supernode consists of eight modular computing units, each containing eight GPUs. These units are connected to a redundant GPU unit via an OCS module, forming an 8×9 redundant network structure. It should be understood that looping back one path on the computing switching blade still satisfies the requirement of an eight-GPU mesh within the machine. However, one GPU can also be taken offline and switched to a GPU on another computing blade via network software configuration. This network structure expands the entire GPU supernode from 8×8 to 8×9, thus providing 8+1 redundant GPUs. In other words, the system consists of nine such computing blades. Each blade carries eight GPUs. Following the 6+8 interface pattern described above, the eight GPUs within each blade first construct a partially connected internal network through six internal interfaces. To complete the full interconnection (mesh) of the eight GPUs within the blade, a special "internal loopback" configuration is performed using one external interface and its connected optical module (e.g., the eighth OCS optical module of each blade). See [link to relevant documentation] accordingly. Figure 7As shown, the OCS module supports two path configurations: Configuration A: Intra-unit loopback is achieved through the eighth OCS optical module, completing the link between GPU0 and GPU1 and maintaining full 8×8 interconnection. In some embodiments, its normal operating mode (Configuration A) is defined as follows: Taking computing blade A as an example, its six internal interfaces of GPU0 are connected to GPU2 through GPU7 respectively. The eight external interfaces of GPU0 are connected to the eight optical modules of OCS group A. Through software configuration, the link of the eighth optical module of OCS group A is physically looped back, connecting to a certain internal interface of GPU1 within this blade. Thus, although there is no direct internal interface connection between GPU0 and GPU1, the external path "GPU0 -> 8th OCS optical module -> GPU1" is logically equivalent to a high-speed internal link, thereby achieving full interconnection of the eight GPUs within the blade. The other eight blades are configured similarly. At this point, through the global configuration of the OCS network, the GPUs in these 9 blades can form an 8×9 physical connection foundation. However, we can configure 8 of the blades (64 GPUs in total) to work as a logically fully interconnected 8×8 cluster, while the 8 GPUs in the 9th blade are in standby status as redundant resources. Configuration B: When GPU1 fails, the OCS module switches the communication path to the corresponding GPU in the redundant unit to achieve fault replacement. In some implementations, it is defined in fault switching mode (Configuration B): When the monitoring software detects a failure of GPU1 in computing blade A, the switching process is initiated: Step 1), the monitoring system evaluates the status of redundant resources, for example, selecting GPU1' in the 9th blade (redundant blade) as the takeover GPU. Step 2), the software sends instructions to the OCS control system to change the relevant link configuration. It should be understood that: the path of the 8th optical module in OCS group A of computing blade A is reconfigured. Its original internal loopback connection (to the failed GPU1) is disconnected, and it is routed to the 8th optical module in OCS group B of the redundant blade, and then connected to GPU1'. Step 3) Simultaneously, other relevant paths in the OCS network may also need fine-tuning to ensure that all other GPUs in the cluster establish correct communication connections with the new "logical GPU1" (i.e., the physical GPU1'). Step 4) The system updates the GPU logical mapping table, redirecting tasks and data flows destined for the original GPU1 to GPU1'. The faulty GPU1 is marked offline. Through the above steps, the system logically maintains a fully interconnected 64-GPU work cluster, but one member has been replaced by the healthy redundant GPU1' from the faulty GPU1. The monitoring software can attempt to repair the faulty GPU1, and if successful, remark it as a redundant resource.

[0037] Furthermore, see Figure 7On the left is compute blade A. GPU0 has 6 internal interfaces, connecting to GPUs 2 / 3 / 4 / 5 / 6 / 7 respectively, and 8 external interfaces, connecting to 8 OCS optical modules, essentially forming an 8×9 interconnect network. Further, in some embodiments, the 6 internal interfaces of GPU0 in the first compute unit connect to GPUs 2-7 respectively, and the 8 external interfaces connect to the OCS modules; the eighth OCS module connects to GPU 1, enabling loopback or redundancy switching. In addition, in other embodiments, the OCS optical modules can be configured with the link path via software. Configuration A: Internal loopback, the internal network of the compute blade completes the switching path between GPU0 and GPU1 through the 8th OCS module. This configuration still results in an overall 8×8 GPU interconnect. Configuration B: When a GPU inside the compute blade fails and goes offline, it can be detected by software and then configured to switch to another backup GPU. For example, if GPU 1 on the left goes offline, GPU 1 on the right compute blade can be brought online, and the compute network switches to GPU 1 on the right, completing the backup switch.

[0038] Furthermore, in some embodiments, such as Figure 8 As shown, the GPU redundancy management process includes: Step S101: System initialization, configuring OCS to internal loopback mode; it should be understood that the initial configuration state is to complete the mesh interconnect structure inside the computing blades, and the actual operation is an 8×8 supernode. Step S102: Run the GPU fault detection and inspection program; for example, run GPU fault detection and inspection software. Step S103: If a fault is detected, assess the status of the backup GPU; for example, check if any GPU is faulty or offline. If a fault is found, report to the inspection software to prepare to switch to the backup GPU; otherwise, continue inspection. Step S104: Reconfigure the communication path through OCS and switch to the redundant GPU; for example, inspect and assess the GPU fault status, assess the backup GPU's working status, and after matching the status, configure the network to complete the switchover of the backup GPU. Step S105: Mark the faulty GPU and issue an alarm, and attempt to repair it; for example, mark the faulty GPU, issue an alarm to the inspection software, and the inspection software attempts to repair the GPU's operation. Step S106: If the repair is successful, switch it to backup status; for example, if the faulty GPU recovers, remove the fault mark, change it to backup GPU, and continue to be included in the inspection software management. Step S107: If repair is unsuccessful, prompt for GPU module replacement; for example, if the faulty GPU cannot self-recover, report an unrepairable status to the inspection software, issue a critical alarm, and prompt for replacement of the faulty GPU module. After replacing the GPU module and updating the GPU status, you can continue to access the inspection software.

[0039] It should also be understood that this disclosure also proposes a GPU redundancy deployment system based on AI server supernodes, including: multiple modular computing units (such as computing blades), each unit containing multiple GPUs; an optical switching module group for connecting GPUs between different modular computing units; and a redundancy monitoring and switching module for real-time monitoring of GPU status and controlling the optical switching module to perform path reconfiguration; wherein, some communication interfaces of each GPU are used for internal interconnection within the unit, and the remaining interfaces are connected to other units through the optical switching module to form a scalable redundant network structure.

[0040] Furthermore, see Figure 9 This disclosure also provides a GPU redundancy deployment method 100 based on AI server supernodes. The method 100 includes: step 110, acquiring and adjusting the internal and external interfaces of the GPUs in the modular computing unit; step 120, configuring the link paths of the 9 GPUs in the modular computing unit so that the 9 GPUs achieve pairwise interconnection of the link paths; step 130, when the modular computing unit is running, setting 8 of the 9 GPUs as regular GPUs and setting the remaining 1 of the 9 GPUs as a redundant GPU; step 140, when it is detected that any one of the 8 GPUs set as regular GPUs fails, adjusting the remaining 7 GPUs set as regular GPUs to achieve pairwise interconnection of the link paths with the redundant GPU.

[0041] Furthermore, in some alternative embodiments, this disclosure also addresses Figure 9Method 100 also provides a software control flow, specifically including: Step 210: System initialization. Configure the OCS network and the internal connections of each computing blade to start the system in a logically non-redundant N×N cluster mode (e.g., 8×8). Step 220: Run the background GPU fault detection and inspection software to periodically or event-triggeredly check the health status of each GPU (e.g., through heartbeat packets, ECC error counts, temperature performance, etc.). Step 330: Determine if any GPU is detected as faulty or abnormally offline. If not, return to step 220 to continue inspection; if so, proceed to step 240. Step 240: The inspection software assesses the severity of the fault and queries the redundant GPU resource pool, selecting a healthy and matching redundant GPU as the takeover target. Step 250: The inspection software, in conjunction with the OCS control software, executes network path switching configuration (as described in Example 1) to complete the communication link switch from the faulty GPU to the redundant GPU. Subsequently, update the resource configuration information of the cluster management software (e.g., the job scheduler). Step 260: Mark the faulty GPU as "faulty" and issue an alarm. The inspection software can attempt to perform recovery operations such as resetting and reloading the driver on the faulty GPU. Step 270: Determine if the faulty GPU has successfully self-recovered. If recovery is successful, proceed to step 280; if recovery is unsuccessful, proceed to step 290. Step 280: Clear the fault mark of the GPU, change its status to "standby," and reinstate it into the redundant resource pool. The process returns to the monitoring loop. Step 290: For GPUs that cannot be recovered, report a "serious fault, hardware replacement required" alarm to notify maintenance personnel. Step 299: After the maintenance personnel replace the faulty GPU module, update the GPU status to "ready / standby" in the management interface, and the system reinstates it into the redundant resource pool.

[0042] In some alternative embodiments, this disclosure also proposes a GPU redundancy deployment method based on AI server supernodes. The method includes: an interface configuration step: acquiring multiple GPUs in a modular computing unit, each GPU having a preset number of internal and external interfaces; adjusting the interface allocation of the GPUs to increase the number of external interfaces used for external connections; an interconnection construction step: interconnecting the multiple modular computing units and configuring the link paths of GPUs within and between units to construct a physically containing (N+1) groups of GPUs, where N is an integer greater than 1; a work initialization step: during the operation of the interconnection network, setting N groups of GPUs as regular working GPUs and at least one group of GPUs as redundant GPUs; configuring the switching module in the interconnection network to form a logically fully interconnected working cluster among the regular working GPUs; and a fault switching step: monitoring the operating status of the regular working GPUs; when any regular working GPU is detected to have failed, reconfiguring the link paths of the switching module, switching the communication link associated with the failed GPU to at least one of the redundant GPUs, and connecting the redundant GPU to the logical working cluster to replace the failed GPU.

[0043] In some alternative embodiments, the interface configuration step of adjusting the GPU interface allocation includes: changing the GPU interface allocation mode from a first number of internal interfaces and a second number of external interfaces to a third number of internal interfaces and a fourth number of external interfaces, wherein the third number is less than the first number and the fourth number is greater than the second number.

[0044] In some alternative embodiments, adjusting the GPU interface allocation includes changing the GPU's interface from 7 internal interfaces and 7 external interfaces to 6 internal interfaces and 8 external interfaces.

[0045] In some alternative embodiments, in the interconnect construction step, the switching module is an optical circuit switching (OCS) module; configuring the link path includes: configuring an internal loopback path for at least one modular computing unit, which completes the full interconnect link between all GPUs within the unit by connecting an OCS module to the unit.

[0046] In some alternative embodiments, during the initialization step, the internal loopback path is configured to take effect, so that N units, including the at least one modular computing unit, constitute a logically complete N×N fully interconnected cluster.

[0047] In some alternative embodiments, the fault-switching step of reconfiguring the link path of the switching module includes: disconnecting the internal loopback path and rerouting the link of the OCS module originally used for internal loopback to connect to the GPU located in another modular computing unit as a redundant GPU.

[0048] In some alternative embodiments, the fault switching step is followed by a recovery step: if the failed GPU is repaired, its status is updated to redundant GPU, and the resource configuration information of the interconnect network is updated.

[0049] In some embodiments, this disclosure further proposes a GPU redundancy deployment device based on AI server supernodes, which may include: an interface adjustment module for configuring the number of internal and external interfaces of the GPU; a path configuration module for controlling the optical switching module to realize interconnection between GPUs; a status management module for setting the GPU working mode and monitoring its status; and a fault switching module for performing path switching when the GPU fails.

[0050] In some embodiments, this disclosure further proposes an electronic device including a memory and a processor, wherein the memory stores a GPU redundancy management program; a communication interface connected to an OCS control module; the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0051] In some embodiments, this disclosure further proposes a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps of the method described above.

[0052] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0053] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A GPU redundancy deployment method based on AI server supernodes, characterized in that, include: The 7 internal interfaces and 7 external interfaces of the GPU in the modular computing unit are adjusted to 6 internal interfaces and 8 external interfaces; The nine GPUs in the modular computing unit are configured with link paths to achieve pairwise interconnection. Specifically, the eight GPUs in the first modular computing unit are configured with link paths to achieve pairwise interconnection, and the six internal interfaces of the first GPU in the first modular computing unit are connected to the third, fourth, fifth, sixth, seventh, and eighth GPUs, respectively. The eight external interfaces of the first GPU in the first modular computing unit are connected to the eight OCS optical modules in the first OCS optical module group, and the eighth OCS optical module is connected to one internal interface of the first to eighth GPUs. The first GPU is configured to form an internal loopback with the second GPU via the eighth OCS optical module, thereby achieving link path interconnection between the first GPU and the second GPU. The eight GPUs in the second modular computing unit are configured with link paths to achieve pairwise interconnection. When the modular computing unit is running, 8 of the 9 GPUs are set as regular GPUs, and the remaining 1 of the 9 GPUs is set as a redundant GPU. When any one of the eight GPUs configured as regular GPUs is detected to be faulty, the remaining seven GPUs will be adjusted to establish a pairwise interconnection link with the redundant GPU.

2. The method according to claim 1, characterized in that, The step of configuring link paths for the eight GPUs in the second modular computing unit of the modular computing unit to achieve pairwise interconnection of the eight GPUs includes: The six internal interfaces of the ninth GPU in the second modular computing unit are connected to the eleventh, twelfth, thirteenth, fourteenth, fifteenth, and sixteenth GPUs, respectively; the eight external interfaces of the ninth GPU in the second modular computing unit are connected to the eight OCS optical modules in the second OCS optical module group, and the sixteenth OCS optical module is connected to one internal interface of the ninth to sixteenth GPUs. The ninth GPU is configured to form an internal loop with the tenth GPU via the sixteenth OCS optical module, thereby realizing the interconnection of the link paths between the ninth GPU and the tenth GPU; The link paths of the eight OCS optical modules in the second OCS optical module group are interconnected with those of the eight OCS optical modules in the first OCS optical module group.

3. The method according to claim 2, characterized in that, The step of setting 8 out of the 9 GPUs as regular GPUs and setting the remaining 1 out of the 9 GPUs as a redundant GPU includes: The eight GPUs in the first modular computing unit of the modular computing unit are configured as regular GPUs, and any one of the eight GPUs in the second modular computing unit of the modular computing unit is configured as a redundant GPU.

4. The method according to claim 3, characterized in that, When it is detected that one of the eight GPUs configured as regular GPUs has failed, the remaining seven GPUs will be adjusted to establish pairwise interconnection links with the redundant GPU, including: When any one of the eight GPUs in the first modular computing unit fails, the remaining seven GPUs in the first modular computing unit are adjusted to establish a pairwise interconnection link path with the redundant GPUs in the second modular computing unit.

5. A GPU redundancy deployment device based on AI server supernodes, characterized in that, include: The interface adjustment module is used to adjust the 7 internal interfaces and 7 external interfaces of the GPU in the modular computing unit to 6 internal interfaces and 8 external interfaces. A path configuration module is used to configure the link paths of the nine GPUs in the modular computing unit, enabling pairwise interconnection of the nine GPUs. Specifically, the module configures the link paths of eight GPUs in the first modular computing unit, enabling pairwise interconnection of the eight GPUs. The six internal interfaces of the first GPU in the first modular computing unit are connected to the third, fourth, fifth, sixth, seventh, and eighth GPUs, respectively. The eight external interfaces of the first GPU in the first modular computing unit are connected to the eight OCS optical modules in the first OCS optical module group, and the eighth OCS optical module is connected to one internal interface of the first through eighth GPUs. The first GPU is configured to form an internal loopback with the second GPU via the eighth OCS optical module, thereby achieving link path interconnection between the first and second GPUs. The module also configures the link paths of the eight GPUs in the second modular computing unit, enabling pairwise interconnection of the eight GPUs. The status management module is used to set the GPU's working mode and monitor its status; The fault switching module is used to set 8 of the 9 GPUs as regular GPUs and the remaining 1 of the 9 GPUs as a redundant GPU when the modular computing unit is running; when any of the 8 GPUs set as regular GPUs is detected to fail, the remaining 7 GPUs set as regular GPUs are adjusted to achieve pairwise interconnection with the redundant GPU through link paths.

6. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the steps of the method as described in any one of claims 1 to 4.

Images