Server cabinet system and task processing method of server cabinet system

Abstract

The application discloses a server cabinet system and a task processing method of the server cabinet system, and relates to the technical field of servers. The server cabinet system comprises at least one first cabinet and a second cabinet, the second cabinet is interconnected with each first cabinet, the first cabinet comprises a first switching node and a plurality of first module interfaces, the plurality of first module interfaces of the first cabinet are used for connecting a plurality of first accelerator modules, the second cabinet comprises a plurality of second module interfaces, and the plurality of second module interfaces are used for connecting a plurality of second accelerator modules. The first switching node is used for reconfiguring a communication path for a processing task of a first accelerator module that is out of order in the plurality of first accelerator modules, so that a second accelerator module connected by the second cabinet takes over the processing task of the first accelerator module that is out of order.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a server rack system and a task processing method for the server rack system. Background Technology

[0002] In related technologies, server rack systems can be used to provide computing power support for large model training and inference. During the use of a server rack system, the accelerator modules within the system communicate in an all-to-all manner; if a single accelerator module fails, the entire system will stop.

[0003] This shows that the server rack system in the relevant technology has the problem of low system redundancy. Summary of the Invention

[0004] This application provides a server rack system and a task processing method for the server rack system, in order to at least solve the problem of low system redundancy in server rack systems in related technologies.

[0005] According to one aspect of the embodiments of this application, a server rack system is provided, comprising: at least one first rack and a second rack, the second rack being interconnected with each of the first racks, wherein the first rack includes a first switching node and a plurality of first module interfaces, the plurality of first module interfaces of the first rack being used to connect to a plurality of first accelerator modules, and the second rack includes a plurality of second module interfaces, the plurality of second module interfaces being used to connect to a plurality of second accelerator modules; wherein the first switching node is configured to, in the event that a first accelerator module among the plurality of first accelerator modules has failed, reconfigure the communication path for the processing task of the failed first accelerator module, so that the processing task of the failed first accelerator module is taken over by the second accelerator module connected to the second rack.

[0006] According to another aspect of the embodiments of this application, a server rack is also provided, including: an interconnect interface group and a plurality of module interfaces, the plurality of module interfaces being used to connect a plurality of accelerator modules, the interconnect interface group being used to interconnect the server rack with other racks besides the server rack; wherein, the interconnect interface in the interconnect interface group is used to receive processing task data sent by the other racks, wherein the processing task data is task data of a processing task of an accelerator module that has failed on the other racks, the processing task data being transmitted to a designated accelerator module among the plurality of accelerator modules; the designated accelerator module is used to take over the processing task in response to the received processing task data.

[0007] According to another aspect of the embodiments of this application, a task processing method for a server rack system is also provided. The server rack system includes at least one first rack and a second rack, the second rack being interconnected with each of the first racks. Each first rack includes a first switching node and a plurality of first module interfaces, the plurality of first module interfaces being used to connect to a plurality of first accelerator modules. The second rack includes a plurality of second module interfaces, the plurality of second module interfaces being used to connect to a plurality of second accelerator modules. The method includes: in the event that a first accelerator module among the plurality of first accelerator modules has failed, reconfiguring the communication path for the processing task of the failed first accelerator module through the first switching node, so that the second accelerator module connected to the second rack takes over the processing task of the failed first accelerator module.

[0008] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0009] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0010] This application employs a method of physical redundancy backup of accelerator modules connected to a first rack by using accelerator modules connected to a second rack. The server rack system includes at least one first rack and one second rack, with the second rack interconnected with each first rack. Each first rack includes a first switching node and multiple first module interfaces for connecting multiple first accelerator modules. Each second rack includes multiple second module interfaces for connecting multiple second accelerator modules. The first switching node is configured to handle situations where a first accelerator module fails. In the event of a failure, the communication path is reconfigured for the processing tasks of the first accelerator module that has malfunctioned, so that the second accelerator module connected to the second rack can take over the processing tasks of the first accelerator module that has malfunctioned. Since the second rack is interconnected with at least one first rack, when the accelerator module connected to the first rack malfunctions, the accelerator module connected to the second rack takes over the processing tasks of the malfunctioning accelerator module, ensuring that the system can operate normally when the accelerator module connected to the first rack malfunctions. This solves the problem of low system redundancy in server rack systems in related technologies and achieves the technical effect of improving system redundancy. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A structural block diagram of an optional server rack system provided for an embodiment of this application;

[0013] Figure 2 A schematic diagram of an optional AI cabinet design provided for an embodiment of this application;

[0014] Figure 3 A schematic diagram of the front layout of an optional cabinet provided in an embodiment of this application;

[0015] Figure 4 A schematic diagram of the rear layout of an optional cabinet provided in an embodiment of this application;

[0016] Figure 5 A schematic diagram illustrating an optional interconnection relationship between computing nodes and switching nodes, provided for an embodiment of this application;

[0017] Figure 6 A schematic diagram illustrating another optional interconnection relationship between computing nodes and switching nodes provided for an embodiment of this application;

[0018] Figure 7 A schematic diagram illustrating an optional second rack configuration provided for an embodiment of this application;

[0019] Figure 8 A schematic diagram of an optional server rack system provided for an embodiment of this application;

[0020] Figure 9 A schematic diagram of another optional server rack system provided for an embodiment of this application;

[0021] Figure 10 A schematic diagram of yet another optional server rack system provided in an embodiment of this application;

[0022] Figure 11 This is a schematic diagram of a server rack system provided in an embodiment of this application;

[0023] Figure 12 A schematic diagram illustrating another optional interconnection relationship between computing nodes and switching nodes provided in an embodiment of this application;

[0024] Figure 13 A schematic diagram illustrating another optional interconnection relationship between computing nodes and switching nodes provided in an embodiment of this application;

[0025] Figure 14 A schematic diagram illustrating a redundant connection method between a computing node and a switching node provided in an embodiment of this application;

[0026] Figure 15 A schematic diagram illustrating another redundant connection method between a computing node and a switching node provided in an embodiment of this application;

[0027] Figure 16 A schematic diagram illustrating another redundant connection method between a computing node and a switching node provided in an embodiment of this application;

[0028] Figure 17 A schematic diagram illustrating another redundant connection method between a computing node and a switching node provided in an embodiment of this application;

[0029] Figure 18 A schematic diagram illustrating another optional interconnection relationship between computing nodes and switching nodes provided in an embodiment of this application;

[0030] Figure 19 A schematic diagram illustrating another optional interconnection relationship between computing nodes and switching nodes provided in an embodiment of this application;

[0031] Figure 20 A schematic diagram of an optional task processing method for a server rack system provided in an embodiment of this application;

[0032] Figure 21 This is a computer system architecture block diagram of an optional electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0034] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] According to one aspect of an embodiment of this application, a server rack system is provided. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0037] Figure 1 This is a structural block diagram of an optional server rack system according to an embodiment of this application, such as... Figure 1 As shown, the server rack system includes at least one first rack 101 and a second rack 102, with the second rack 102 interconnected with each first rack 101. Each first rack 101 includes a first switching node 1011 and multiple first module interfaces 1012 for connecting multiple first accelerator modules. Each second rack 102 includes multiple second module interfaces 1021 for connecting multiple second accelerator modules. The first switching node 1011 is configured to reconfigure the communication path for the processing tasks of a faulty first accelerator module in the event of a fault, allowing the second accelerator modules connected to the second rack 102 to take over the processing tasks of the faulty first accelerator module.

[0038] With the continuous development of artificial intelligence (AI) technology, AI model training and inference are receiving increasing attention. As a key infrastructure supporting AI model training and inference, next-generation computing cluster architectures are facing severe performance challenges. The marginal benefits of individual server processor performance are diminishing, approaching physical limits and making it difficult to meet the continuously growing computing power demands of large AI models. Simultaneously, the strategy of simply increasing the number of servers to achieve horizontal scaling of computing clusters has also encountered bottlenecks in efficiency and scalability, facing challenges including cost, data synchronization, and energy consumption, hindering the efficient execution of large-scale parallel computing.

[0039] Against this backdrop, the AI ​​supernode server rack system has emerged. Through an innovative topology architecture, it systematically integrates accelerator modules and switching nodes, and comprehensively optimizes aspects such as multi-power computing fusion, high-speed advanced interconnection, heat dissipation, high power density power supply, and rack management. It provides scalable high-bandwidth domain (HBD) supernode systems with 32 cards or more to meet the cutting-edge needs of large model training and inference.

[0040] A server rack system may include at least one rack. Within a rack, multiple accelerator modules and one or more switching nodes can be configured. The nodes housing the accelerator modules can be connected to connectors (e.g., high-speed connectors) on the switching nodes via connectors. The aforementioned accelerator modules may be OAM (Open Accelerator Modules), and OAM may be GPU (Graphics Processing Unit) computing units. Switching nodes may also be referred to as switching units, switches, etc.

[0041] The node where the accelerator module is located can be a compute node (compute server). Correspondingly, a complete rack can include one or more compute nodes, and each compute node can include multiple accelerator modules. The connector on the compute node can be called a compute connector, and the connector on the switching node can be called a switching connector. The connection relationship between the compute connector and the switching connector can be one-to-one, many-to-one, or many-to-many. In this embodiment, the connection relationship between the compute node and the switching node is not limited.

[0042] For example, an AI rack design such as Figure 2As shown, the rack comprises N compute nodes and M switching nodes arranged in the middle, where N and M are both positive integers greater than or equal to 1. High-speed connectors are installed in both the compute nodes and switching nodes, and the compute nodes and switching nodes are connected via cables using a specific connection topology. Each compute node has 4 OAM2.0 (GPUs), and the entire system has 4*N GPUs.

[0043] For a server rack, one possible layout on the front of the rack is as follows: Figure 3 As shown, the computing nodes are placed on the top and bottom sides of the cabinet, while the switching nodes are placed in the middle, which allows for a more balanced distribution of power and communication lines. Figure 3 This only represents an optional configuration; the number of switching nodes and compute nodes can be adjusted according to the actual situation. The height of each U-position (i.e., the location where the node device is placed) can also be adjusted, and the square hole strips for fixing each node in the rack can be set to one or more optional sizes, such as OU (height of 48mm), RU (height of 44.45mm), and SU (height of 46.5mm).

[0044] One type of distribution on the back of the server rack can be as follows Figure 4 As shown, the rear of the cabinet includes three components: a busbar, a cable tray (a structure for supporting and managing cables), and a manifold. The busbar, located in the middle or on one side, is connected to the powershelf ("power rack" or "power module rack") nodes to supply power to the entire cabinet. The cable tray, located on either the left or right sides, connects the high-speed signals of the compute nodes and switching nodes according to the designed topology interconnection. The manifold, located on the left and right sides at the rear of the cabinet, provides water inlet and outlet, offering a liquid cooling path for the entire cabinet.

[0045] An interconnection relationship between compute nodes and switching nodes can be as follows: Figure 5 As shown, Figure 5The upper area of ​​the rack contains N compute nodes (#1 to #N), and the lower area contains M switching nodes. Each compute node has a CPU (Central Processing Unit), which can connect four OAM2.0 GPUs (i.e., GPUs) via a PCIe (Peripheral Component Interconnect Express) switch. If there are N compute nodes in the rack, the total number of GPUs is 4*N. For example, with 16 compute nodes, the total number of GPUs is 64 (16*4). The compute nodes also have a DPU (Data Processing Unit) network interface card, storage devices, and a BMC management module. The storage devices can be, but are not limited to, SSDs (Solid State Drives). Each switching node has a MAC (Media Access Control) unit. High-speed signals between the switching nodes and compute nodes are interconnected via a cable tray.

[0046] In AI rack servers, compute nodes and switching nodes can all be connected to the RackCable Tray (cable trays inside or near the rack) via high-density connectors, enabling high-speed interconnection of SerDes (Serializer / Deserializer) between all GPUs in the AI ​​rack server. For example... Figure 6 As shown, the high-speed signal resources of each OAM (GPU) in each computing node can be evenly distributed to each page switching node, realizing a scale-up topology.

[0047] The rack-connected accelerator modules can be used in the training and inference of large models. When used in large model training (Mixture of Experts), all-to-all communication is required, which means that each accelerator module must be able to function without failure. The hardware lacks redundancy, and if a single accelerator module fails, the entire system will stop.

[0048] For example, when training large models using technologies like MOE, all-to-all communication is required, which means that each GPU must be able to function without failure. This lack of hardware redundancy means that if a single GPU on a single node fails, the entire system will stop.

[0049] To at least partially solve the aforementioned technical problems, in this embodiment, in addition to the first rack serving as the work cabinet, a second rack can be set up in the server rack system, and the second rack is interconnected with the first rack. When the accelerator module connected to the first rack fails, the accelerator module connected to the second rack takes over the processing tasks of the failed accelerator module, ensuring that the system can operate normally when the accelerator module connected to the first rack fails. Here, the first rack and the second rack are connected to their respective accelerator modules through their respective module interfaces.

[0050] Both the first and second racks can be full rack servers. The architecture of the first and second racks can be the same or different. There can be one or more first racks and one or more second racks. In scenarios where there are multiple first racks and / or multiple second racks, the processing method of the processing tasks is the same as or similar to the processing method of the processing tasks in scenarios where there is one first rack and one second rack. The server rack system in this embodiment uses the same method.

[0051] The first rack may include a first switching node and multiple first module interfaces, which are used to connect multiple first accelerator modules. The second rack includes multiple second module interfaces, which are used to connect multiple second accelerator modules. The first switching node is used to reconfigure the communication path for the processing tasks of a faulty first accelerator module in the event of a fault, so that the processing tasks of the faulty first accelerator module can be taken over by the second accelerator modules connected to the second rack.

[0052] In this embodiment, in order for the second accelerator module connected to the second rack to take over the processing tasks of the first accelerator module connected to the first rack that has malfunctioned, the node where the accelerator module is located can reconfigure the communication path for the processing tasks of the malfunctioning accelerator module. However, considering the functions of different nodes and reducing interactions between nodes, the first switching node can reconfigure the communication path for the processing tasks of the malfunctioning first accelerator module.

[0053] Optionally, the communication path can be reconfigured by reconfiguring the destination address for the task data of the task being processed. The reconfigured destination address can be the rack address of the second rack, in which case the second accelerator module that takes over the task processing is specified by the second rack, or it can be the module address of the second accelerator module that takes over the task processing, in which case the second accelerator module that takes over the task processing is specified by the first switching node or other nodes in the first rack. In this embodiment, the method of reconfiguring the communication path is not limited.

[0054] It should be noted that redundant accelerator modules can also be installed within the rack to take over the processing tasks of a faulty accelerator module. However, installing redundant accelerator modules within the rack requires adjustments to the entire rack, which is costly and incompatible with existing racks. Adopting a configuration with at least one first rack and one second rack reduces the need for structural adjustments to the entire rack, lowers adjustment costs, and improves system compatibility.

[0055] According to the embodiments provided in this application, a server rack system includes at least one first rack and a second rack, with the second rack interconnected with each first rack. Each first rack includes a first switching node and multiple first module interfaces, which are used to connect multiple first accelerator modules. The second rack includes multiple second module interfaces, which are used to connect multiple second accelerator modules. The first switching node is used to reconfigure the communication path for the processing tasks of a faulty first accelerator module in the event of a fault, so that the second accelerator modules connected to the second rack can take over the processing tasks of the faulty first accelerator module. This solves the problem of low system redundancy in server rack systems in related technologies and improves system redundancy.

[0056] In one exemplary embodiment, to improve the utilization efficiency of the second rack, the server rack system can have multiple first racks and one second rack. Multiple first racks can share the resources of a second accelerator module connected to a single second rack. Different numbers of first racks can be set according to usage requirements; for example, the number of first racks can be 6, 8, 10, etc. This embodiment does not limit the number of first racks.

[0057] There are several possible strategies for using the second accelerator modules, including but not limited to the following: Each first rack can use a portion of the second accelerator modules from multiple second accelerator modules, and the second accelerator modules allowed to be used in different first racks are different. This strategy facilitates the management of the allocated second accelerator modules by the switching nodes on different first racks; Each first rack can use all of the second accelerator modules from multiple second accelerator modules, and one of the multiple second accelerator modules can only be used by one first rack at a time. This strategy can improve the utilization rate of the accelerator modules.

[0058] For example, such as Figure 7As shown, a redundant cabinet (an example of a second cabinet) is configured with 64 GPUs and interconnected with 8 master work cabinets (an example of a first cabinet, each with 64 GPUs). Each master work cabinet is allocated 8 redundant GPUs, forming a "1 redundant cabinet + 8 master work cabinets" topology.

[0059] In this embodiment, the server rack system is configured with multiple first racks, which share the accelerator modules connected to the second rack. The sharing method of the accelerator modules can be configured as needed, which can improve the utilization rate and flexibility of accelerator module resources.

[0060] In one exemplary embodiment, the second cabinet further includes a second switching node for relaying data between the second accelerator module connected to the first switching node and the second cabinet, wherein the transmission delay of the second switching node is less than the transmission delay of the first switching node.

[0061] The first computing node can be directly connected to the second accelerator module connected to the second rack via a cable. To enhance the reliability and fault tolerance of the AI ​​server rack system, a second switching node can be set up on the second rack. The second switching node acts as a relay to transfer data between the first switching node and the second accelerator module connected to the second rack.

[0062] Here, in addition to signal relay and routing, the second switching node can form a redundant cross-rack link with the first switching node, avoiding potential failure points and maintaining system stability. The second switching node also improves system scalability, allowing more accelerator module resources to be integrated into the system, while also providing room for future upgrades. Furthermore, the second switching node can manage and monitor the connected accelerator modules, facilitating system-level fault detection and management.

[0063] In large-scale distributed computing environments, communication latency between nodes directly impacts the overall system response time and computational efficiency. Therefore, selecting a switching node with low transmission latency as the second node ensures that its transmission latency is lower than that of the first switching node. Using a second switching node with lower latency than the first ensures that even during failover, data exchange between the second accelerator module connected to the second rack and the accelerator module connected to the first rack maintains extremely low latency, thereby guaranteeing system communication efficiency.

[0064] In this embodiment, the second switching node acts as a relay to transfer data between the accelerator modules connected to the first switching node and the second cabinet. The transmission delay of the second switching node is less than that of the first switching node, which can maintain the stability of the system operation and facilitate the management of the accelerator modules connected to the second cabinet.

[0065] In one exemplary embodiment, both the first and second switching nodes may be equipped with switching components. These switching components can dynamically adjust data flow paths based on network load and communication demands, ensuring the continuity and stability of data transmission through load balancing and redundant link mechanisms. Considering the high frequency of use and performance requirements of the switching nodes in the first rack, the switching component on the first switching node can be an Ethernet switching component (MAC Ethernet switching component). The switching component on the second switching node can also be an Ethernet switching component. An Ethernet switching component is a data switching device responsible for managing and coordinating data communication between accelerated computing modules such as GPU modules, enabling high-bandwidth data exchange required for large-scale parallel computing.

[0066] To reduce latency and lower costs, the switching components on the second switching node can be retimer components, such as PHY (Physical Layer) retimers. PHY retimers enhance the stability of high-speed signal transmission and extend transmission distance, ensuring the physical layer reliability of communication between redundant GPUs and the first rack. The switching components on the second switching node can also be re-driver components.

[0067] Here, compared to other switching components, the signal retiming and regeneration capabilities of the retimer component can optimize the transmission of high-speed signals, especially in long-distance communication. It can effectively reduce signal attenuation and distortion, ensure the quality of the signal upon arrival, and improve signal integrity and system fault tolerance. Similar to the retimer component, the re-driver component can capture the input signal, regenerate and enhance it to compensate for signal attenuation during transmission, ensure the quality of the signal upon arrival, and improve signal integrity.

[0068] In this embodiment, the switching nodes on the second rack select retimer components as switching components, which can effectively reduce signal attenuation and distortion, and improve signal integrity and system fault tolerance.

[0069] In one exemplary embodiment, the first switching node and the second switching node are horizontally interconnected using a pluggable interface.

[0070] To improve the flexibility of the connection between the first and second cabinets and facilitate cable routing, the first and second switching nodes can be horizontally interconnected using pluggable interfaces, for example, by using front-window-based OSFP (Octal Small Form Factor Pluggable) interfaces for horizontal interconnection.

[0071] For example, redundancy can be introduced at the GPU quantity level—a redundant architecture and physical connection design. The MAC components of the switching nodes in the entire rack can be replaced with 16 PHY Retimer components. Figure 8 An exemplary solution is provided for configuring an AI supernode server using a PHYRetimer switching component adapted to a general-purpose switch. The entire rack includes 16 compute nodes and 12 switching nodes, with each compute node having four GPUs and 16 Retimers. A scaling-up adapter board for a single switching node is also provided. Figure 8 As shown on the right, the entire rack can be horizontally interconnected through the OSFP interface based on the front window; Figure 8 On the left is a standard switch rack, which can be used to test or demonstrate the above-mentioned rack-mount end-to-end adaptation solution for general-purpose switches.

[0072] Will Figure 8 The Retimer switching node is interconnected with the Ethernet MAC switching node, enabling parallel cabinet operation, as shown in the topology. Figure 9 As shown (the two cabinets are cabinet 1 and cabinet 2).

[0073] Based on the aforementioned cabinet, a parallel 64+64 card (128 card) system can be constructed, such as... Figure 10 As shown. For a switch tray, one cabinet contains the MAC Ethernet switching component, and another cabinet contains the lower latency PHYRETIMER component.

[0074] A GPU redundancy cabinet is constructed using one Retimer switching node cabinet, while the other eight cabinets utilize Ethernet switching nodes, interconnected laterally via OSFP interfaces on the front end, forming... Figure 11 The server rack system consists of a redundant rack with 64 GPUs, interconnected with eight main racks (each with 64 GPUs) via a PHY Retimer relay unit (i.e., a Retimer unit). Each main rack is allocated eight redundant GPUs, forming a topology of "1 redundant rack + 8 main racks".

[0075] By using a pluggable interface to horizontally interconnect the switching nodes of the first cabinet and the second cabinet in this embodiment, the flexibility of the connection between the first cabinet and the second cabinet can be improved, and cable routing can be facilitated.

[0076] In one exemplary embodiment, the first switching node is further configured to receive processing task data to be transmitted to a first target accelerator module among a plurality of first accelerator modules; in response to the received processing task data, in the event of a failure of the first target accelerator module, reconfigure the communication path for the processing task data; and transmit the processing task data to a second target accelerator module connected to a second rack according to the reconfigured communication path, so that the second target accelerator module can process the processing task data in place of the first target accelerator module.

[0077] In this embodiment, the first switching node is also used for receiving task data, responding to faults, and reconfiguring communication paths. The first switching node can receive processing task data to be transmitted to the first target accelerator module among multiple first accelerator modules. The processing task data may include parameter updates, feature exchanges, intermediate results during inference, etc., of large model training, which need to be further processed on the first target accelerator module.

[0078] In response to the received processing task data, the first switching node can determine whether the first target accelerator module has malfunctioned. There are various ways to determine whether the first target accelerator module has malfunctioned. For example, the first switching node can monitor the status of the first accelerator module connected to it in real time and determine whether the first target accelerator module has malfunctioned based on the recorded status. Alternatively, after receiving the processing task data, the first switching node can learn whether the first target accelerator module has malfunctioned from the node where the first target accelerator module is located. Or, it can determine whether the first target accelerator module has malfunctioned by detecting the link status of the cable link from the first switching node to the first target accelerator module. Other determination methods are also possible, and this embodiment does not limit these methods. A malfunction of the first target accelerator module can occur in various ways, such as signal loss, computational anomalies, or communication delays exceeding a threshold.

[0079] In the event of a failure in the first target accelerator module, the first switching node can respond immediately by initiating a redundancy strategy to utilize a redundant accelerator module. Based on the redundancy strategy and the current network status, the first switching node can reconfigure the communication path for processing task data. This means that the first switching node recalculates the transmission path for processing task data, redirecting the data stream originally destined for the failed first accelerator module (i.e., the first target accelerator module) to a redundant second accelerator module (i.e., the second target accelerator module) connected to the second rack.

[0080] According to the reconfigured communication path, the first switching node can transmit the processing task data to the second target accelerator module connected to the second rack, so that the second target accelerator module can process the processing task data in place of the first target accelerator module. After receiving the transmitted processing task data, the second target accelerator module processes the processing task data in place of the first target accelerator module, which may include performing calculations for model training, data processing during inference, etc.

[0081] Optionally, the first switching node may include a baseboard management controller and a switching component. The operations performed by the first switching node may be executed by the baseboard management controller on the first switching node, wherein reconfiguring the communication path may be performed on the switching component. The control method is similar for other switching node structures, and will not be described in detail here.

[0082] For example, the redundant GPU's link is in standby mode when the first rack is operating normally, but is activated in case of a failure. When a GPU in the first rack fails, the BMC management module triggers fault detection and reconfigures the communication path via a switching component. The redundant GPU takes over the communication tasks of the failed GPU through the Retimer link, ensuring the continuity of data flow in all-to-all communication.

[0083] In this embodiment, when the switching node on the first rack detects a fault in the accelerator module connected to the first rack, it reconfigures the communication path to transmit task data to the accelerator module connected to the second rack for processing. This improves the reliability of data transmission. At the same time, having the switching node perform the above operation can improve the efficiency of data processing and avoid task interruption and loss in the server rack system.

[0084] In an exemplary embodiment, the first switching node is further configured to query the module status identifier of the first target accelerator module in response to the received processing task data; and to reconfigure the communication path for the processing task data if it is determined that the first target accelerator module has failed based on the module status identifier of the first target accelerator module.

[0085] To improve the timeliness and convenience of data processing, in this embodiment, the first switching node can record the module status of each first accelerator module connected to it. When receiving task data to be processed by a certain first accelerator module, it can determine whether the first accelerator module has malfunctioned by querying the recorded module status of the first accelerator module, and then perform subsequent processing operations based on the module status of the first accelerator module.

[0086] Upon receiving the processing task data, the first exchange node can first determine that the corresponding first accelerator module is the first target accelerator module; then, it queries the module status identifier of the first target accelerator module; based on the queried module status identifier, the first exchange node can analyze the operating status of the first target accelerator module and determine whether there is a fault.

[0087] Here, the module status identifier reflects the current health status and availability of the accelerator module. It can be a fault identifier; based on the fault identifier value, it can be directly determined whether the first target accelerator module has failed. The module status identifier can also include multiple indicators such as temperature, power consumption, computational load, error count, and communication status. By combining multiple indicators, it can be determined whether the corresponding accelerator module has failed. Alternatively, if an anomaly is detected in the module status identifier, such as excessively high temperature, power consumption exceeding the range, or an increase in error count, a fault diagnosis program can be initiated to confirm whether a fault has actually occurred, as well as the nature and severity of the fault.

[0088] If a failure is detected in the first target accelerator module, the first switching node will reconfigure the communication path, redirecting the processing task data originally sent to the failed first target accelerator module to the second target accelerator module connected to the second rack. This reconfiguration of the communication path may involve updating the routing table, re-encapsulating data packets, and adjusting the transmission protocol to ensure that data can be successfully transmitted to the second target accelerator module along the new communication path.

[0089] When reconfiguring communication paths, the first switching node considers low latency and high bandwidth requirements to ensure efficient data transmission and smooth computation. In AI training and inference scenarios, data transmission speed and consistency directly impact model convergence speed and prediction accuracy. Therefore, the first switching node needs to complete communication path reconfiguration in the shortest possible time to avoid data latency and packet loss, ensuring high-performance system operation.

[0090] In this embodiment, the module status of the corresponding accelerator module is identified by the module status identifier. This allows for the determination of whether an accelerator module has malfunctioned by directly querying its status. This can shorten the fault identification time, improve the efficiency of communication path reconfiguration, and ensure that the system can respond to faults and perform path configuration in a timely manner.

[0091] In one exemplary embodiment, the first rack may further include multiple first computing nodes. Each first computing node is connected to a portion of the multiple first accelerator modules. The first accelerator modules connected to different first computing nodes are different from each other. A first accelerator module can be directly connected to at least one connector of its respective first computing node, or it can be connected to at least one connector of its respective first computing node via a retimer. A retimer on the first computing node can be connected to one or more first accelerator modules or one or more connectors to improve topology flexibility and system redundancy. The first computing node may also include a baseboard management controller, and the operations performed by the first computing node can be controlled and executed by the baseboard management controller.

[0092] When there are multiple first switching nodes, a connector of a first computing node can be connected to one or more connectors of a first switching node, or it can be connected to at least two connectors of at least two first switching nodes. A connector on a switching node can be connected to multiple connectors of a first computing node, or it can be connected to multiple connectors of at least two computing nodes, or other connection methods can be used, which are not limited in this embodiment.

[0093] In this embodiment, the first target accelerator module is connected to a first target computing node. Regarding the aforementioned processing task data, the first switching node is further configured to transmit the processing task data to the first target computing node; upon receiving the processing task data transmitted back from the first target computing node, it reconfigures the communication path for the processing task data; the first target computing node is configured to receive the processing task data; and upon determining that the first target accelerator module has malfunctioned, it transmits the processing task data back to the first switching node.

[0094] In scenarios where the first switching node cannot determine whether the first target accelerator module has malfunctioned upon receiving processing task data, the first switching node can transmit the processing task data to the first target computing node. This transmission can occur via a connector between the two nodes. The first target computing node can receive the processing task data, determine whether the first target accelerator module has malfunctioned, and if so, transmit the processing task data back to the first switching node. Conversely, if the first switching node receives the processing task data transmitted back from the first target computing node, it can determine that the first target accelerator module has malfunctioned. In this case, it can reconfigure the communication path for the processing task data.

[0095] In this embodiment, the switching node on the first rack interacts with the computing node where the accelerator module is located, and determines whether the accelerator module has malfunctioned based on whether the task data has been transmitted back, which can improve the accuracy of status determination.

[0096] In an exemplary embodiment, the first switching node is further configured to query the module status identifier of the first target accelerator module to obtain a first identifier value before transmitting the processing task data to the first target computing node; and update the module status identifier of the first target accelerator module from the first identifier value to a second identifier value if it is determined that the first target accelerator module has failed.

[0097] To reduce task interruptions and improve system computational efficiency, the module status identifier of the first target accelerator module can be queried before transmitting the processing task data to the first target node. This ensures rapid GPU-level fault response and seamless rerouting of task data. If the query finds that the module status identifier of the first target accelerator module is at the first identifier value, it is determined that the first target accelerator module is not faulty, and the processing task data can then be transmitted to the first target node.

[0098] Here, the status identifier of a certain accelerator module has two values: a first identifier value and a second identifier value. The first identifier value indicates that the corresponding accelerator module has not malfunctioned, while the second identifier value indicates that the corresponding accelerator module has malfunctioned. For example, if the identifier of a certain accelerator module is 0 (the first identifier value), it means that the accelerator module is in normal condition (i.e., no malfunction has occurred).

[0099] After receiving the returned processing task data, the module status identifier of the first target accelerator module can be updated from the first identifier value to the second identifier value, indicating that the first target accelerator module has failed, so as to ensure that the identifier of the accelerator module and the fault detection can be matched in real time.

[0100] This embodiment improves the timely correspondence between module status identifiers and module status by querying module status and updating status identifiers, thus facilitating the timely and accurate transmission of processing task data.

[0101] In one exemplary embodiment, on the second rack, there may be multiple second accelerator modules that are not allowed to replace the first target accelerator module. For example, there may be a correspondence between the first and second accelerator modules, and the first accelerator module may only be replaced by the second accelerator module with which it has a corresponding relationship. For another example, in the case of multiple first racks, the multiple second accelerator modules may be divided into multiple groups, and a group of second accelerator modules may only be allowed to replace all the first accelerator modules connected to one first rack, or only be allowed to replace some of the first accelerator modules connected to one first rack. There may also be other situations where it is not allowed to replace the first target accelerator module (for example, the second accelerator modules connected to the second rack are not allowed to be used as redundant accelerator modules).

[0102] In this embodiment, a first switching node on a first rack can focus only on the second accelerator modules allocated to this first rack, and select a second accelerator module from the second accelerator modules allocated to this first rack that is allowed to replace the first target accelerator module. Correspondingly, the first switching node is also used to select the second target accelerator module from the second accelerator modules that are allowed to replace the first target accelerator module.

[0103] Here, the second accelerator module allowed to replace the first target accelerator module can be the second accelerator module allocated to the first rack from among multiple second accelerator modules. If the selected second accelerator module is already occupied, it is necessary to wait for the second accelerator module to be released before new task data can be processed, which prolongs the processing time of the task. Therefore, the second accelerator module allowed to replace the first target accelerator module can be the second accelerator module allocated to the first rack from among multiple second accelerator modules that is not currently occupied.

[0104] For example, for Figure 11 The server rack system shown has the 17th to 24th redundant GPUs allocated to the third main work rack. When a faulty GPU exists on the third main work rack, an unused redundant GPU from the 17th to 24th redundant GPUs can be used to perform the task.

[0105] It should be noted that within a first rack, the second accelerator module allocated to it can be indicated by configuration information, and the occupancy status of each allocated second accelerator module is recorded during operation. The occupancy status of each second accelerator module can be determined by whether it is used to replace a certain first accelerator module to perform processing tasks and whether the task processing results are received, or by interacting with the second rack, or by other determination methods. This embodiment does not limit the determination of these methods.

[0106] In this embodiment, by using the redundant accelerator modules allocated to the first rack that are not occupied, and allowing them to replace the accelerator modules that have failed on the first rack, the timeliness of task execution can be improved.

[0107] In an exemplary embodiment, similar to the architecture of the first rack described above, the second rack may include: a second switching node and a second computing node. There are multiple second computing nodes, and each second computing node is connected to a portion of the multiple second accelerator modules. The connection relationship between the second accelerator module and the connector on the connected second computing node is similar to the connection relationship between the first accelerator module and the connector on its corresponding first computing node. Similarly, the connection relationship between the connector of the second computing node and the connector of the second switching node is similar to the connection relationship between the connector of the first computing node and the connector of the first switching node, as previously described, and will not be repeated here. The second computing node connected to the second target accelerator module is the second target computing node.

[0108] Correspondingly, in this embodiment, the second switching node is used to receive processing task data sent by the first switching node; in response to the received processing task data, it transmits the processing task data to the second target computing node; the second target computing node is used to transmit the second target accelerator module from the second switching node, so that the second target accelerator module can replace the first target accelerator module to process the processing task data.

[0109] For the second cabinet, the second switching node can receive the processing task data sent by the first switching node. For the received processing task data, it determines that the second accelerator module to be transmitted is the second target accelerator module on the second target computing node, and transmits the processing task data to the second target computing node. Here, the parsing operation of the processing task data and the transmission operation of transmitting the processing task data to the second target computing node can be performed according to the data transmission protocol adopted, which will not be elaborated in this embodiment.

[0110] The second target computing node (which may be the baseboard management controller on it) can control the transmission of processing task data to the second target accelerator module. This involves the reception and parsing of processing task data, as well as the data transmission process within the second target computing node (e.g., parsing from a connector, parsing the data, and transmitting the data to the second target accelerator module via a retimer).

[0111] The second target accelerator module can replace the first target accelerator module to process task data. The data processing results can be transmitted to the first exchange node via the second exchange node. The first exchange node continues to relay data to execute subsequent task processing procedures.

[0112] In this embodiment, the second rack adopts an architecture design that combines computing nodes and switching nodes, which can maintain the stability of system operation and facilitate the management of the accelerator modules connected to the second rack.

[0113] In one exemplary embodiment, the second rack includes: a plurality of second computing nodes, one second computing node being connected to a portion of the plurality of second accelerator modules, and the second computing node connected to the second target accelerator module being a second target computing node; wherein, a first switching node is used to transmit processing task data to a target interface on the second rack corresponding to the second target computing node according to a reconfigured communication path, so as to transmit the processing task data to the second target computing node through a cable connecting the target interface and the second target computing node; the second target computing node is used to transmit the processing task data to the second target accelerator module, so that the second target accelerator module processes the processing task data in place of the first target accelerator module.

[0114] In this embodiment, similar to the previous embodiments, the first switching node can transmit the processing task data to the second rack according to the reconfigured communication path, and the interconnect interface (e.g., OSFP) on the second rack is directly connected to the connector of the second computing node via a cable. In this case, the first switching node can transmit the processing task data to the target interface on the second rack corresponding to the second target computing node according to the reconfigured communication path, so as to transmit the processing task data to the second target computing node via the cable connecting the target interface and the second target computing node.

[0115] The second price list calculation node can transmit the processing task data to the second target accelerator module, so that the second target accelerator module can process the processing task data instead of the first target accelerator module. This has already been explained and will not be repeated here.

[0116] In this embodiment, the second computing node, which connects the interconnection interface of the second cabinet and the second computing node via cables, can be adapted to the simple architecture of the cabinet, improve the efficiency of data transmission, thereby increasing the speed of task execution, and also reduce the cost of the second cabinet.

[0117] In one exemplary embodiment, to improve system redundancy, redundancy can be implemented at the connector level of the cable assembly. Here, the cable assembly is a component on the server rack that manages and organizes connectors and cables, and the connectors of compute nodes and switching nodes are supported by the cable assembly. The cable assembly can be a cable tray. In some examples of this embodiment, the cable assembly is explained using a cable tray as an example. It is understood that the server rack system provided in this embodiment is applicable to other types of cable assemblies.

[0118] To achieve redundancy at the connector level of the cable assembly, one of the multiple first accelerator modules is located on a first computing node in a first rack and connected to multiple connectors on that first computing node. If the first rack includes one cable assembly, the multiple connectors to which one first accelerator module is connected belong to the same cable assembly. If the first rack includes K cable assemblies, the multiple connectors to which one first accelerator module is connected can belong to the same cable assembly among the K cable assemblies, or they can be connected to at least two of the K cable assemblies, where K is a positive integer greater than or equal to 2.

[0119] Here, when multiple connectors connected to an accelerator module belong to the same cable assembly, even if one connector malfunctions, some signals from the accelerator module can still be transmitted through the other normal connectors. When multiple connectors connected to an accelerator module belong to at least two cable assemblies, in addition to partial signal transmission when a connector malfunctions, even if one cable assembly malfunctions, some signals from the accelerator module can still be transmitted through the other normal cable assemblies.

[0120] For example, in a server rack with two cable trays, considering redundancy, the connection from a single GPU on a single compute node to a single switch node can be evenly distributed across both cable trays. Figure 12 As shown. This way, even if the physical link between one GPU and a switching node fails, the other path remains usable. In this case, only a slowdown occurs, not a system crash. However, if... Figure 13 As shown in the connection diagram, if the connector at the corresponding position of a single Cable Tray fails, the signal from a certain GPU (OAM) to a certain switching node will be completely disconnected, making communication impossible and causing a system crash.

[0121] Taking node 5 as an example, Figure 14 , Figure 15 , Figure 16 and Figure 17The diagram illustrates the redundant connection between 16 compute nodes and switch node 5. Figure 14 This is a redundant connection method between compute nodes 1-4 and switching node 5. Figure 15 This is a redundant connection method between compute nodes 5-8 and switch node 5. Figure 16 This is a redundant connection method between compute nodes 9-12 and switching node 5. Figure 17 It is a redundant connection method between compute nodes 13-16 and switching node 5.

[0122] This embodiment achieves redundancy at the connector level of the cable assembly. An accelerator module is connected to at least two connectors, which ensures that signal transmission can still be carried out when a signal link is damaged, avoiding system downtime due to connector failure and improving the stability of system operation.

[0123] In an exemplary embodiment, the first cabinet further includes a first computing node. There are multiple first computing nodes. Each first computing node includes a portion of the first accelerator modules from the multiple first accelerator modules. The connection relationship between the first accelerator module and the connector on the first computing node, as well as the connection relationship between the connector of the first computing node and the connector of the first switching node, are similar to those in the previous embodiments and will not be described again here.

[0124] A first computing node may include multiple retimer components, and the correspondence between them and the first accelerator modules in the same first computing node is many-to-many. That is, a first accelerator module is connected to at least two retimer components in the same first computing node, and a retimer component is connected to at least two first accelerator modules in the same first computing node.

[0125] For a first accelerator module, it may include multiple signal channels (lanes) according to signals (corresponding to pins). In order to connect to at least two retimer components, the multiple signal channels can be divided into multiple signal channel groups. One signal channel group is connected to the uplink port of a corresponding retimer component. For a retimer component, its multiple uplink ports are divided into multiple uplink port groups according to port order. The uplink ports within the same uplink port group are sequentially adjacent. One uplink port group is connected to one signal channel group of a corresponding first accelerator module.

[0126] In this embodiment, by employing retimer redundancy, the accelerator modules are divided into multiple groups to be connected to multiple retimer components, and the uplink ports of the retimer components are divided into multiple groups to be connected to multiple accelerator modules. This improves the reliability of signal transmission and facilitates wiring.

[0127] In one exemplary embodiment, within a retimer component of a first computing node, the signal channels are reversed to adjust the order of the signal channels on the downlink port.

[0128] In this embodiment, to enhance the system's flexibility, reliability, and fault tolerance without sacrificing performance, the signal channel within a retimer component of a first computing node is reversed to adjust the order of the signal channels on the downlink port. This method ensures stable system operation even in the event of partial hardware failure or signal quality degradation, guaranteeing the continuity of critical computing tasks and data integrity.

[0129] For example, such as Figure 18 As shown, the signal for each OAM node can be divided into one lane. Assuming each GPU outputs 0-11 lanes, a single Retimer can support 16 lanes. Lane reversal can be adjusted internally within the Retimer. The mapping relationship from the Retimer's Down PORT to the BP (Backplane) connector is configured with the smallest possible granularity, as shown below. Figure 19 As shown. It should be noted that, Figure 19 This is an optional cable topology that can be adjusted as needed.

[0130] Combination Figure 18 and Figure 19 Let's take two Retimers as an example to illustrate the redundancy process when one Retimer fails. If Retimer 0 fails, Retimer 1 can still transmit the remaining half of the signal to the corresponding switching node.

[0131] This embodiment demonstrates how signal channel inversion can be performed within a single timer component, thereby enhancing system flexibility, reliability, and fault tolerance without sacrificing performance.

[0132] According to another aspect of the embodiments of this application, a server rack is also provided. Optionally, in this embodiment, the above-mentioned server rack may be applied to, but is not limited to, applications such as... Figure 1 In the server rack system shown, the server rack can be... Figure 1 The second rack in the server rack system shown has already been described and will not be repeated here.

[0133] In this embodiment, the server rack includes an interconnect interface group and multiple module interfaces. The multiple module interfaces are used to connect multiple accelerator modules. The interconnect interface group is used to interconnect the server rack with other racks besides the server rack. The interconnect interfaces in the interconnect interface group are used to receive processing task data sent by other racks. The processing task data is the task data of the processing task of the accelerator module that has failed on the other rack. The processing task data is transmitted to a designated accelerator module among the multiple accelerator modules.

[0134] The multiple module interfaces of the server rack can be the aforementioned second module interfaces, and the multiple accelerator modules connected can be the aforementioned multiple second accelerator modules.

[0135] According to the embodiments provided in this application, a server rack includes: an interconnect interface group and multiple module interfaces. The multiple module interfaces are used to connect multiple accelerator modules. The interconnect interface group is used to interconnect the server rack with other racks besides the server rack. The interconnect interfaces in the interconnect interface group are used to receive processing task data sent by other racks. The processing task data is task data of processing tasks of accelerator modules that have failed on other racks. The processing task data is transmitted to a designated accelerator module among the multiple accelerator modules. The designated accelerator module is used to take over the processing task in response to the received processing task data. This solves the problem of low system redundancy in server rack systems in related technologies and improves system redundancy.

[0136] In one exemplary embodiment, the server rack further includes at least one of the following: a rack switching node (corresponding to the aforementioned second switching node), a rack computing node (corresponding to the aforementioned second computing node), the functions of the rack switching node and the rack computing node are similar to those of the second switching node and the second computing node in the foregoing embodiments, and have been described before, so they will not be repeated here.

[0137] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0138] According to another aspect of the embodiments of this application, a task processing method for a server rack system is also provided. Optionally, in this embodiment, the above-described task processing method for a server rack system may be applied, but is not limited to, to applications such as... Figure 1In the server rack system shown, the server rack system includes at least one first rack and a second rack, which are interconnected with the first rack. The first rack includes a first switching node and multiple first module interfaces, which are used to connect multiple first accelerator modules. The second rack includes multiple second module interfaces, which are used to connect multiple second accelerator modules. The processing logic of the first rack, the second rack, the first switching node, the first accelerator modules, the second accelerator modules, and each part is similar to that in the previous embodiment, and will not be repeated hereafter.

[0139] Taking the server rack system's task processing method in this embodiment as an example, Figure 20 This is a flowchart illustrating an optional task processing method for a server rack system according to an embodiment of this application, as shown below. Figure 20 As shown, the process of this method may include the following steps:

[0140] In step S2002, if a faulty first accelerator module exists among multiple first accelerator modules, the communication path for the processing task of the faulty first accelerator module is reconfigured through the first switching node, so that the second accelerator module connected to the second cabinet can take over the processing task of the faulty first accelerator module.

[0141] In this embodiment, if a faulty first accelerator module is found among the multiple first accelerator modules, the first switching node can reconfigure the communication path for the processing task of the faulty first accelerator module. This allows one or more second accelerator modules on the second rack to take over the processing task of the faulty first accelerator module and complete the task execution. The above process is similar to that in the previous embodiments and will not be described in detail here.

[0142] Through the embodiments provided in this application, in the case where a faulty first accelerator module exists among multiple first accelerator modules, the communication path for the processing task of the faulty first accelerator module is reconfigured through the first switching node, so that the processing task of the faulty first accelerator module is taken over by the second accelerator module connected to the second rack. This solves the problem of low system redundancy in server rack systems in related technologies and improves system redundancy.

[0143] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0145] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0146] In one exemplary embodiment, the aforementioned computer-readable storage medium is a non-volatile storage medium, which may include, but is not limited to, various non-temporary storage media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0147] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0148] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0149] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 2109, and / or installed from removable medium 2111. When the computer program is executed by central processing unit 2101, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0150] Figure 21 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 21 As shown, the computer system 2100 includes a CPU (Central Processing Unit) 2101, which can perform various appropriate actions and processes based on programs stored in ROM 2102 or programs loaded into RAM 2103 from storage section 2108. Random access memory 2103 also stores various programs and data required for system operation. The CPU 2101, ROM 2102, and RAM 2103 are interconnected via bus 2104. An I / O (Input / Output) interface 2105 is also connected to bus 2104.

[0151] The following components are connected to I / O interface 2105: input section 2106 including keyboard, mouse, etc.; output section 2107 including CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), etc., and speakers, etc.; storage section 2108 including hard disk, etc.; and communication section 2109 including network interface card, modem, etc. Communication section 2109 performs communication processing via a network such as the Internet. Drive 2110 is also connected to I / O interface 2105 as needed. Removable media 2111, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 2110 as needed so that computer programs read from them can be installed into storage section 2108 as needed.

[0152] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 2109, and / or installed from removable medium 2111. When the computer program is executed by central processing unit 2101, it performs various functions defined in the system of this application.

[0153] It should be noted that, Figure 21 The computer system 2100 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0154] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0155] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A server rack system, characterized in that, include: At least one first rack and one second rack, the second rack being interconnected with each of the first racks, wherein each first rack includes a first switching node and a plurality of first module interfaces, the plurality of first module interfaces of the first rack being used to connect a plurality of first accelerator modules, and the second rack includes a plurality of second module interfaces, the plurality of second module interfaces being used to connect a plurality of second accelerator modules; wherein... The first switching node is configured to reconfigure the communication path for the processing task of the faulty first accelerator module in the event that a faulty first accelerator module exists among the plurality of first accelerator modules, so that the processing task of the faulty first accelerator module can be taken over by the second accelerator module connected to the second cabinet.

2. The server rack system according to claim 1, characterized in that, The second cabinet also includes: The second switching node is used as a relay to transfer data between the first switching node and the second accelerator module connected to the second cabinet, wherein the transmission delay of the second switching node is less than the transmission delay of the first switching node.

3. The server rack system according to claim 2, characterized in that, The switching component on the second switching node is a retimer component. The second cabinet is directly connected to the first switching node through the retimer component, forming an extended bandwidth domain across the cabinet.

4. The server rack system according to claim 3, characterized in that, The switching component on the first switching node is an Ethernet switching component, wherein a vertically extended architecture is adopted inside the first cabinet, and a bandwidth domain is constructed through Ethernet and the switching component.

5. The server rack system according to claim 2, characterized in that, The first switching node and the second switching node are horizontally interconnected using a pluggable interface.

6. The server rack system according to claim 1, characterized in that, The first switching node is further configured to receive processing task data to be transmitted to a first target accelerator module among the plurality of first accelerator modules; in response to the received processing task data, in the event of a failure of the first target accelerator module, reconfigure the communication path for the processing task data; and transmit the processing task data to a second target accelerator module connected to the second rack according to the reconfigured communication path, so that the second target accelerator module can process the processing task data in place of the first target accelerator module.

7. The server rack system according to claim 6, characterized in that, The first switching node is further configured to query the module status identifier of the first target accelerator module in response to the received processing task data; and to reconfigure the communication path for the processing task data if it is determined that the first target accelerator module has failed based on the module status identifier of the first target accelerator module.

8. The server rack system according to claim 6, characterized in that, The first rack also includes multiple first computing nodes. Each first computing node is connected to a portion of the multiple first accelerator modules. The first computing node to which the first target accelerator module is connected is the first target computing node. The first switching node is further configured to transmit the processing task data to the first target computing node; and, upon receiving the processing task data transmitted back from the first target computing node, to reconfigure the communication path for the processing task data. The first target computing node is used to receive the processing task data; and in the event that the first target accelerator module has failed, it transmits the processing task data back to the first switching node.

9. The server rack system according to claim 8, characterized in that, The first switching node is further configured to query the module status identifier of the first target accelerator module before transmitting the processing task data to the first target computing node, and obtain a first identifier value, wherein the first identifier value is used to indicate that the corresponding accelerator module has not failed; if it is determined that the first target accelerator module has failed, update the module status identifier of the first target accelerator module from the first identifier value to a second identifier value, wherein the second identifier value is used to indicate that the corresponding accelerator module has failed.

10. The server rack system according to claim 6, characterized in that, The first switching node is further configured to select the second target accelerator module from a group of second accelerator modules that are allowed to replace the first target accelerator module, wherein the second accelerator module that is allowed to replace the first target accelerator module is the second accelerator module that is allocated to the first rack and is not occupied among the plurality of second accelerator modules.

11. The server rack system according to claim 6, characterized in that, The second rack includes: a second switching node and a second computing node. There are multiple second computing nodes, and each second computing node is connected to a portion of the multiple second accelerator modules. The second computing node connected to the second target accelerator module is the second target computing node. The second switching node is configured to receive the processing task data sent by the first switching node; and in response to the received processing task data, transmit the processing task data to the second target computing node. The second target computing node is used to transmit the processing task data to the second target accelerator module, so that the second target accelerator module can process the processing task data in place of the first target accelerator module.

12. The server rack system according to claim 6, characterized in that, The second rack includes: multiple second computing nodes, one of which is connected to a portion of the multiple second accelerator modules; the second computing node connected to the second target accelerator module is a second target computing node; wherein... The first switching node is used to transmit the processing task data to the target interface on the second rack corresponding to the second target computing node according to the reconfigured communication path, so as to transmit the processing task data to the second target computing node through the cable connecting the target interface and the second target computing node; The second target computing node is used to transmit the processing task data to the second target accelerator module, so that the second target accelerator module can process the processing task data in place of the first target accelerator module.

13. The server rack system according to claim 1, characterized in that, One of the plurality of first accelerator modules is connected to a first computing node in the first cabinet and interconnected with a plurality of connectors of the first computing node. The first cabinet also includes K cable assemblies. The plurality of connectors connected to one of the plurality of first accelerator modules belong to the same cable assembly of the K cable assemblies, or at least two cable assemblies of the K cable assemblies, where K is a positive integer greater than or equal to 2.

14. The server rack system according to claim 1, characterized in that, The first cabinet also includes a first computing node, and there are multiple first computing nodes. Each first computing node includes multiple timer components and a portion of the multiple first accelerator modules. Within a first computing node, there is a many-to-many relationship between the timer component and the first accelerator module; Multiple signal channels of a first accelerator module are divided into multiple signal channel groups, and a signal channel group is connected to the uplink port of a corresponding retimer component. Multiple uplink ports of a retimer component are divided into multiple uplink port groups according to port order, and each uplink port group is connected to a corresponding signal channel group of a first accelerator module.

15. The server rack system according to claim 14, characterized in that, Within a retimer component of one of the first computing nodes, the signal channels are reversed to adjust the order of the signal channels on the downlink ports.

16. The server rack system according to any one of claims 1 to 15, characterized in that, There are multiple first cabinets and one second cabinet. Each first cabinet allows the use of a portion of the multiple second accelerator modules.

17. The server rack system according to any one of claims 1 to 15, characterized in that, There are multiple first cabinets and one second cabinet. Each first cabinet is allowed to use all the second accelerator modules from the multiple second accelerator modules, and one of the multiple second accelerator modules is allowed to be used by only one first cabinet at a time.

18. A server rack, characterized in that, include: The system includes an interconnect interface group and multiple module interfaces, wherein the multiple module interfaces are used to connect multiple accelerator modules, and the interconnect interface group is used to interconnect the server rack with other racks besides the server rack itself; wherein... The interconnect interface in the interconnect interface group is used to receive processing task data sent by the other racks, wherein the processing task data is the task data of the processing task of the accelerator module that has failed on the other racks, and the processing task data is transmitted to a designated accelerator module among the multiple accelerator modules. The designated accelerator module is used to take over the processing task in response to the received processing task data.

19. A task processing method for a server rack system, characterized in that, The server rack system includes at least one first rack and a second rack, the second rack being interconnected with each of the first racks. Each first rack includes a first switching node and multiple first module interfaces, the multiple first module interfaces being used to connect multiple first accelerator modules. The second rack includes multiple second module interfaces, the multiple second module interfaces being used to connect multiple second accelerator modules. The method includes: In the event that a faulty first accelerator module exists among the plurality of first accelerator modules, the communication path for the processing task of the faulty first accelerator module is reconfigured through the first switching node, so that the processing task of the faulty first accelerator module is taken over by the second accelerator module connected to the second cabinet.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method as described in claim 19.

Images