Fault recovery method and related system

Abstract

Provided in the present application is a fault recovery method, which is applied to a controller, the controller being used for performing fault recovery when a bus switching network in a computing cluster has had a fault, the bus switching network comprising at least one switching node, and the bus switching network being used for achieving memory access between computing nodes in the computing cluster. The method comprises: the controller acquiring fault information reported by at least one switching node, the fault information indicating a fault mode and a fault position, then determining, according to the fault information, from the at least one switching node an execution node used for fault recovery, determining, according to the fault information, a memory address communication path impacted by the fault, and then instructing the execution node to update the memory address communication path impacted by the fault. By means of updating the memory address communication path impacted by the fault, the method achieves fault recovery without shutdown, and does not require shielding of single or multiple computing nodes, thus avoiding waste of computing power.

Description

A fault recovery method and related system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 27, 2023, with application number 202310939855.X and invention name “A Fault Recovery Method and Related System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a fault recovery method, a controller, a fault recovery system, a computer-readable storage medium, and a computer program product. Background Art

[0003] As chips gradually reach the limits of Moore's Law, the computing power (i.e., computing power) of a single chip becomes difficult to scale up vertically. Increasing computing power typically requires scaling out horizontally to meet the growing demand for computing power. For example, computing nodes, including chips such as central processing units (CPUs) and neural processing units (NPUs), can be connected via bus-switched networks to form computing clusters. These computing clusters are also called super nodes or fat nodes.

[0004] As the scale of chip interconnection grows, the scale of bus switching networks is also growing. For example, 64 compute nodes are interconnected via a bus switching network to form a computing cluster. Each compute node can include four chips (such as two CPUs and two NPUs). Accordingly, the 64 compute nodes include a total of 256 chips. The bus switching network typically provides more than 4,000 ports to meet the high-bandwidth, low-latency chip interconnection requirements. As a result, the likelihood of bus switching network failures is increasing, and the impact of failures is also increasing.

[0005] When a bus switching network fails, how to efficiently repair the bus switching network failure becomes a key issue of concern in the industry.

[0006] Summary of the Invention

[0007] This application provides a fault recovery method that enables fault recovery of a bus switching network without shutting down the network, eliminating the need to block single or multiple computing nodes and avoiding waste of computing power. This application also provides a controller, a fault recovery system, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0008] In a first aspect, the present application provides a fault recovery method. The method is applied to a controller. The controller is used to perform fault recovery when a bus switching network in a computing cluster fails. The bus switching network includes at least one switching node, and the bus switching network is used to implement memory access between computing nodes in the computing cluster. The controller can be software, which can be deployed on a computing node, a switching node, or an independent node in the computing cluster. The computing node, the switching node, or the independent node executes the program code of the software, thereby executing the fault recovery method of the present application. The controller can also be hardware. For example, the controller can be a server with fault recovery capabilities, and the server executes the fault recovery method of the present application when running.

[0009] Specifically, the controller obtains fault information reported by at least one switching node, where the fault information indicates a fault mode and a fault location. The controller then determines an execution node for fault recovery from at least one switching node based on the fault information, and determines a memory address communication path affected by the fault based on the fault information. The controller then instructs the execution node to update the memory address communication path affected by the fault.

[0010] This method determines the memory address communication path affected by the fault by combining the fault information reported by the switching node, and instructs the execution node used for fault recovery to update the memory address communication path affected by the fault, for example, to delete the memory address communication path, thereby achieving fault recovery without shutting down the system, and without shielding single or multiple computing nodes, thus avoiding waste of computing power.

[0011] In some possible implementations, when determining the execution node, the controller may determine the execution node in different ways for different failure modes. Specifically, when the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, for ease of description, this failure mode may also be referred to as failure mode 1. The controller may determine that the faulty node indicated by the fault location is the execution node. Otherwise, the execution node is determined based on the fault location and the port connection relationship recorded in the topology table. The first switching node is one or more switching nodes in the bus switching network, the first chip is one or more chips in the first switching node, and the first computing node is any computing node in the computing cluster.

[0012] In this way, the execution node under a specific fault mode (such as fault mode 1) can be quickly determined, shortening the fault recovery time and improving the fault recovery efficiency.

[0013] In some possible implementations, for failure modes other than failure mode 1, the controller may determine the execution node in the following manner:

[0014] The failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node (for ease of description, this may also be referred to as failure mode 2), or a failure of the first chip in the first switching node (also referred to as failure mode 3). The controller may query the switching node information table based on the faulty node indicated by the fault location to determine the uplink port of the faulty node, and then query the topology table based on the uplink port of the faulty node to determine that the switching node connected to the uplink port of the faulty node is an execution node. Alternatively,

[0015] The failure mode is a partial link failure between the first chip in the first switching node and the second switching node (also referred to as failure mode 4). The controller may query the topology table based on the faulty node and port indicated by the fault location to determine the switching node to which the port is connected and the faulty node as the execution node, wherein the second switching node is one or more switching nodes in the bus switching network; or

[0016] The failure mode is a failure of all links connecting the first chip in the first switching node and the second switching node (also referred to as failure mode 5). The controller may query the switching node information table based on the faulty node indicated by the fault location to determine the uplink port of the faulty node, then query the topology table based on the uplink port of the faulty node to determine the switching node to which the uplink port of the faulty node is connected, and then query the topology table to determine that the switching node to which the chip uplinked to the same switching node as the chip indicated by the fault location belongs is the execution node; or

[0017] The failure mode is a second chip failure in the second switching node (also referred to as failure mode 6). The controller may query the topology table according to the failure node indicated by the failure location to determine that the switching node connected to the failure node is an execution node.

[0018] By combining the topology table (which may also include the switching node information table in some cases), this method accurately determines the switching nodes affected by the fault and requiring modification of the memory address communication path as execution nodes, thereby avoiding the expansion of the fault impact range and laying the foundation for fault recovery without downtime.

[0019] In some possible implementations, the memory address communication path affected by the fault includes a mapping between the address exchange tag of the compute node affected by the fault and the port associated with the fault. Accordingly, when determining the memory address communication path affected by the fault based on the fault information, the controller may first determine the compute node affected by the fault based on the fault information, then query the address translation table to determine the address exchange tag of the compute node affected by the fault, and then determine the memory address communication path affected by the fault based on the address exchange tag of the compute node affected by the fault.

[0020] In this way, the memory address communication path affected by the fault can be determined more accurately, thereby achieving precise fault recovery and reducing the impact on the business.

[0021] In some possible implementations, when the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node (failure mode 2), a failure of the first chip in the first switching node (failure mode 3), or a failure of part of the links connecting the first chip in the first switching node and the second switching node (failure mode 4), the memory address communication path affected by the failure includes a mapping of an address switching label of the computing node affected by the failure and a first failure-associated port, where the first failure-associated port is a port connected to the faulty node or chip indicated by the fault information.

[0022] The failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node (failure mode 1), or a partial link failure connecting the first chip in the first switching node and the second switching node (failure mode 4). The memory address communication path affected by the failure includes a mapping between the address switching label of the computing node affected by the failure and the second failure-associated port, and the second failure-associated port is the port indicated by the fault information.

[0023] When the failure mode is a failure of all links connecting the first chip in the first switching node and the second switching node (failure mode 5), or a failure of the second chip in the second switching node (failure mode 6), the memory address communication path affected by the failure includes a mapping of the address switching label of the computing node affected by the failure and a third fault-associated port, where the third fault-associated port is a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs.

[0024] In this way, the memory address communication paths affected by the fault under different fault modes can be determined in a fine-grained manner, ensuring the effectiveness of fault recovery under different fault modes.

[0025] In some possible implementations, the controller may use different methods to determine the compute nodes affected by the fault for different fault modes. Specifically, when the fault mode is a chip fault in an upper-layer switch node (e.g., fault mode 6), the controller may determine that the compute node in the computing cluster is the compute node affected by the fault. Otherwise, the controller may query the topology table based on the port or chip of the faulty node indicated by the fault location to determine that the compute node to which the port or chip of the faulty node is connected is the compute node affected by the fault.

[0026] In this way, the computing nodes affected by the fault under a specific fault mode (such as fault mode 6) can be quickly determined, and then the memory address communication path affected by the fault can be quickly determined, thereby shortening the fault recovery time and improving the fault recovery efficiency.

[0027] In some possible implementations, the fault location includes the node identifier and chip identifier of the faulty node. Furthermore, the fault location may also include a port identifier. Specifically, when the fault mode is Fault Mode 1, Fault Mode 2, Fault Mode 4, or Fault Mode 5, the fault location may include the aforementioned node identifier, chip identifier, and port identifier (e.g., port number). When the fault mode is Fault Mode 3 or Fault Mode 6, the fault location may include the aforementioned node identifier and chip identifier.

[0028] This can help determine the memory address communication path affected by the fault or determine the execution node, thereby providing assistance for efficient fault recovery.

[0029] In some possible implementations, the controller is independent of the computing cluster. This prevents the controller from occupying computing cluster resources during failure recovery, ensuring normal business operation.

[0030] In some possible implementations, the controller can be a node in a computing cluster, or the controller can be deployed on a node in a computing cluster. In other words, a node in a computing cluster (e.g., a computing node or a switching node) can assume the role of a controller and perform its functions. This allows for functional reuse and reduces additional resource overhead (such as network overhead).

[0031] In a second aspect, the present application provides a controller. The controller is used to perform fault recovery when a bus switching network in a computing cluster fails, the bus switching network including at least one switching node, the bus switching network being used to implement memory access between computing nodes in the computing cluster, and the controller includes:

[0032] a communication module, configured to obtain fault information reported by the at least one switching node, wherein the fault information indicates a fault mode and a fault location;

[0033] a fault recovery strategy determination module, configured to determine an execution node for fault recovery from the at least one switching node according to the fault information, and to determine a memory address communication path affected by the fault according to the fault information;

[0034] The communication module is further configured to instruct the execution node to update the memory address communication path affected by the fault.

[0035] In some possible implementations, the fault recovery strategy determination module is specifically configured to:

[0036] When the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, determining that the faulty node indicated by the fault location is an execution node; otherwise, determining the execution node based on the fault location and the port connection relationship recorded in the topology table;

[0037] The first switching node is one or more switching nodes in the bus switching network, the first chip is one or more chips in the first switching node, and the first computing node is any computing node in the computing cluster.

[0038] In some possible implementations, the fault recovery strategy determination module is specifically configured to:

[0039] The failure mode is that all links connecting the first chip in the first switching node and the first computing node fail, or the first chip in the first switching node fails, and the switching node information table is queried based on the faulty node indicated by the fault location to determine the uplink port of the faulty node, and the topology table is queried based on the uplink port of the faulty node to determine that the switching node connected to the uplink port of the faulty node is an execution node; or

[0040] The failure mode is a partial link failure connecting the first chip in the first switching node and the second switching node, and the topology table is queried based on the faulty node and port indicated by the fault location to determine that the switching node to which the port is connected and the faulty node is the execution node, and the second switching node is one or more switching nodes in the bus switching network; or

[0041] The failure mode is that all links connecting the first chip in the first switching node and the second switching node fail; the switching node information table is queried based on the faulty node indicated by the fault location to determine the uplink port of the faulty node; the topology table is queried based on the uplink port of the faulty node to determine the switching node to which the uplink port of the faulty node is connected; the topology table is queried to determine that the switching node to which the chip uplinked to the same switching node as the chip indicated by the fault location belongs is the execution node; or

[0042] The failure mode is a second chip failure in the second switching node. A topology table is queried according to the failure node indicated by the failure position to determine that the switching node connected to the failure node is the execution node.

[0043] In some possible implementations, the memory address communication path affected by the fault includes a mapping of an address exchange label of a computing node affected by the fault and a fault-associated port, and the fault recovery strategy determination module is specifically configured to:

[0044] Determining, based on the fault information, computing nodes affected by the fault;

[0045] The address decoding table is queried to determine the address exchange label of the computing node affected by the fault, and the memory address communication path affected by the fault is determined according to the address exchange label of the computing node affected by the fault.

[0046] In some possible implementations, the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node, a failure of the first chip in the first switching node, or a failure of part of the links connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of the address switching label of the computing node affected by the failure and the first failure-associated port, and the first failure-associated port is a port connected to the faulty node or chip indicated by the fault information; the failure mode is a failure of part of the links connecting the first chip in the first switching node and the first computing node, or a failure of the first chip in the first switching node and the second switching node. Part of the link connecting the nodes fails, the memory address communication path affected by the fault includes a mapping between the address exchange label of the computing node affected by the fault and the second fault-associated port, and the second fault-associated port is the port indicated by the fault information; when the fault mode is a complete link failure connecting the first chip in the first switching node and the second switching node, or a failure of the second chip in the second switching node, the memory address communication path affected by the fault includes a mapping between the address exchange label of the computing node affected by the fault and the third fault-associated port, and the third fault-associated port is a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs.

[0047] In some possible implementations, the fault recovery strategy determination module is specifically configured to:

[0048] When the fault mode is a chip failure in the upper-layer switching node, the computing node in the computing cluster is determined to be the computing node affected by the fault; otherwise, the topology table is queried according to the port or chip of the fault node indicated by the fault location to determine that the computing node to which the port or chip of the fault node is connected is the computing node affected by the fault.

[0049] In some possible implementations, the fault location includes a node identifier and a chip identifier of the faulty node.

[0050] In a third aspect, the present application provides a controller. The controller includes a processor and a memory, wherein the memory stores computer-readable instructions; at least one processor executes the computer-readable instructions to cause the controller to perform the fault recovery method described in any implementation of the first aspect of the present application.

[0051] In a fourth aspect, the present application provides a fault recovery system. The fault recovery system is used to perform fault recovery when a bus switching network in a computing cluster fails. The bus switching network includes at least one switching node, and the bus switching network is used to implement memory access between computing nodes in the computing cluster. The fault recovery system includes a controller and a fault recovery execution device deployed on the at least one switching node.

[0052] The controller is configured to obtain fault information reported by the at least one switching node, the fault information indicating a fault mode and a fault location, determine an execution node for fault recovery from the at least one switching node based on the fault information, and determine a memory address communication path affected by the fault based on the fault information, and instruct the execution node to update the memory address communication path affected by the fault;

[0053] The fault recovery execution device is used to update the memory address communication path affected by the fault according to the instruction of the controller.

[0054] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions instruct a controller to execute the fault recovery method described in the first aspect or any implementation of the first aspect.

[0055] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a controller, enables the controller to execute the fault recovery method described in the first aspect or any one of the implementations of the first aspect.

[0056] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0058] FIG1 is a schematic diagram of a process for performing fault recovery on a bus switching network according to an embodiment of the present application;

[0059] FIG2 is a flow chart of a fault recovery method in a fault recovery system provided in an embodiment of the present application;

[0060] FIG3 is a schematic diagram of a process for performing fault recovery under different fault modes provided by an embodiment of the present application;

[0061] FIG4 is a schematic diagram of an application scenario of fault recovery provided by an embodiment of the present application;

[0062] FIG5 is a hardware structure diagram of a controller provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0064] First, some technical terms involved in the embodiments of this application are introduced.

[0065] A bus switching network, also known as a bus network, is a network that implements data exchange via a bus. A bus switching network typically includes at least one switching node, such as a bus switch device (BSD). Bus switching networks can employ various network architectures, including but not limited to the Clos network architecture and the Dragonfly network architecture. For ease of description, the following example uses a bus switching network employing the Clos network architecture.

[0066] As shown in Figure 1, a simple Clos network consists of the following switching layers: leaf switching layer and spine switching layer. The leaf switching layer includes access switches (also called leaf switches), which aggregate traffic from compute nodes (such as servers) and are directly connected to the spine switching layer. The spine switching layer includes spine switches, which are directly connected to leaf switches.

[0067] Among them, the leaf switching layer is located at the bottom layer of the bus switching network. Therefore, the leaf switches in the leaf switching layer are also called level 1 (L1) switches and L1 switching nodes. The leaf switching layer is also called the L1 bus access layer. The spine switching layer is located at the upper layer of the bus switching network. Therefore, the spine switches in the spine switching layer are also called level 2 (L2) switches and L2 switching nodes. The spine switching layer is also called the L2 bus switching layer.

[0068] A bus failure may be a failure that occurs in the bus switching network. Bus failures can be divided into repairable failures and unrepairable failures. Hardware with repairable failures (such as switching nodes) can be automatically repaired and an exception log will be recorded after the repair is completed. Hardware with unrepairable failures will report the failure information to the computing nodes participating in the bus communication, specifically to the operating system (OS) or application software stack of the computing node, causing the operating system on the processing chip of the computing node to reset or the computing task to restart. When the computing task scheduler senses that the computing task on the processing chip is continuously restarted, the computing task can be rescheduled to other processing chips.

[0069] The processing chips of the computing nodes may include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), or neural network processing units (NPUs). For ease of description, the aforementioned processing chips may also be referred to as xPUs (for any processing unit). As shown in FIG1 , when an irreparable failure occurs in the bus switching network, a temporary recovery strategy may be implemented, such as shielding a single or multiple computing nodes and dispatching computing tasks to the xPUs of other computing nodes. A long-term recovery strategy may also be implemented, such as shutting down all nodes in the computing cluster (supernode) and replacing the hardware that has experienced the irreparable failure.

[0070] When an irreparable bus switching network failure occurs, even if no xPUs are faulty, several xPUs will be unavailable, resulting in wasted computing power. Implementing a temporary recovery strategy can result in wasted computing power for one or more blocked compute nodes, while implementing a long-term recovery strategy can result in hours of wasted computing power for the entire compute cluster (or supernode).

[0071] In view of this, the present application provides a fault recovery method. The method can be applied to a controller specifically used for fault recovery when a bus switching network in a computing cluster (such as a super node) fails. The bus switching network includes at least one switching node, and the bus switching network is used to implement memory access between computing nodes in the computing cluster. The controller can be software, and the software can be deployed on a computing node, a switching node, or an independent node in the computing cluster. The computing node, the switching node, or the independent node executes the program code of the software, thereby executing the fault recovery method of the present application. The controller can also be hardware, for example, the controller can be a server with a fault recovery function, and the server executes the fault recovery method of the present application when it is running. In other words, the controller can be independent of the computing cluster, or the controller can be deployed on a node in the computing cluster, or the controller can be a node in the computing cluster, for example, a computing node or a switching node that assumes the role of a controller and implements the function of a controller.

[0072] Specifically, the controller obtains fault information reported by at least one switching node, where the fault information indicates a fault mode and a fault location. The controller can then determine an execution node for fault recovery from at least one switching node based on the fault information, and determine a memory address communication path affected by the fault based on the fault information. The controller then instructs the execution node to update the memory address communication path affected by the fault.

[0073] This method determines the memory address communication path affected by the fault by combining the fault information reported by the switching node, and instructs the execution node used for fault recovery to update the memory address communication path affected by the fault, for example, to delete the memory address communication path, thereby achieving fault recovery without shutting down the system, and without shielding single or multiple computing nodes, thus avoiding waste of computing power.

[0074] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the fault recovery system of the embodiment of the present application is first introduced below with reference to the accompanying drawings.

[0075] 2 shows an architecture diagram of a fault recovery system. Fault recovery system 200 includes a controller 202, wherein controller 202 includes a communication module 2022 and a fault recovery strategy determination module 2024. Similar to controller 202, communication module 2022 and fault recovery strategy determination module 2024 can be software modules, such as software modules within controller 202 or software modules deployed by controller 202. Communication module 2022 and fault recovery strategy determination module 2024 can also be hardware modules.

[0076] Among them, the communication module 2022 is used to obtain fault information reported by at least one switching node, and the fault information indicates the fault mode and fault location. The fault recovery strategy determination module 2024 is used to determine the execution node for fault recovery from at least one switching node based on the fault information, and to determine the memory address communication path affected by the fault based on the fault information. Accordingly, the fault recovery strategy may be that the execution node updates the memory address communication path affected by the fault. The communication module 2022 is also used to instruct the execution node to update the memory address communication path affected by the fault. In other words, the communication module 2022 is also used to instruct the execution node to execute the above-mentioned fault recovery strategy for updating the memory address communication path affected by the fault. Among them, the communication module 2022 can instruct the execution node to execute the fault recovery strategy in a variety of ways. A typical way may be that the communication module 2022 sends the affected memory address communication path to the execution node.

[0077] In some possible implementations, the fault recovery system 200 may further include a fault sensing device 204 and a fault recovery execution device 206. Similar to the controller 202, the fault sensing device 204 and the fault recovery execution device 206 may be software, for example, software deployed in a switching node. The fault sensing device 204 and the fault recovery execution device 206 may also be hardware, for example, hardware in a switching node.

[0078] Fault sensing device 204 includes a fault sensing module 2042 and a communication module 2044. Fault sensing module 2042 and communication module 2044 can be software modules or hardware modules. Fault sensing module 2042 is used to perform fault detection, for example, by performing fault self-detection based on the fault detection logic built into the hardware to obtain fault information, which indicates the fault mode and fault location. Communication module 2044 is used to report fault information. It should be noted that fault recovery system 200 may also not include the above-mentioned fault sensing device 204. For example, fault recovery system 200 may obtain fault information including the fault location through other fault location methods.

[0079] Fault recovery execution device 206 includes a communication module 2062 and a fault recovery execution module 2064. Communication module 2062 is configured to receive instructions from controller 202, specifically instructing the execution node to update the memory address communication path affected by the fault. Fault recovery execution module 2064 is configured to update the memory address communication path affected by the fault according to the instructions from controller 202.

[0080] It should be noted that the system architecture shown in FIG2 is a schematic architecture of the fault recovery system 200. In other possible implementations of the embodiments of the present application, the fault recovery system 200 may also have other architectures. For example, the fault recovery strategy determination module 2024 may also be deployed in a switching node, or a module in a switching node. In other words, the switching node may also assume the responsibilities of the controller 202.

[0081] Based on the fault recovery system 200 shown in Figure 2, the present application further provides a fault recovery method. The fault recovery method provided by the present application will be introduced below in conjunction with an embodiment.

[0082] Referring to the flowchart of a fault recovery method shown in FIG2 , the method is applied to the fault recovery system 200 shown in FIG2 , and the method includes the following steps:

[0083] ① The controller 202 imports the networking information of the computing cluster.

[0084] Networking information is used to describe the nodes included in the network, the connection relationships between nodes, and the communication paths. In a computing cluster (i.e., a supernode) formed by a bus switching network, the networking information may include at least one item from the switching node information table, the memory address decoding table (MADT), the memory address switch label table (MASLT), and the topology (TOPO) table. The switching node information table records the information of the switching nodes and is therefore also called the bus switch device information table (BSDIT). The topology table records the topology information of the supernode (i.e., the computing cluster) and is therefore also called the super node bus switch topology table (SNBSTT).

[0085] The switching node information table is used to record the mapping relationship between switching nodes, connection nodes, and uplink ports. A connection node refers to a computing node connected to the switching node, and an uplink port refers to a port on the switching node that connects to an upper-layer switching node (such as an L2 switching node), also referred to as an uplink port. In some possible implementations, the switching node information table may store the node identifier of the switching node, the node identifier of the connection node, and the port number of the uplink port, as shown below:

[0086] Table 1 Example of exchange node information table

[0087] It should be noted that the switching node information table may store the node identifier of the L1 switching node and the corresponding node identifier of the connection node and the port number of the uplink port. In some possible implementations, the switching node information table may also store the node identifier of the L2 switching node. For an L2 switching node, the node identifier of the connection node and the port number of the uplink port may be null.

[0088] The memory address decoding table is used to record the mapping relationship between the memory address of a computing node (e.g., the processing chip of the computing node) and the address exchange label. The address exchange label is a simple, fixed-length label formed by address mapping and is used for fast forwarding or exchanging data. In this embodiment, the address exchange label can be a memory address exchange label formed by memory address mapping.

[0089] When a computing node includes multiple processing chips, each chip may include its own memory address segment. The memory address decoding table may record the mapping relationship between the memory address and the address exchange label of the computing node's processing chip (e.g., xPU). In some possible implementations, the memory address decoding table may store the xPU identifier, memory address, and address exchange label, as shown below:

[0090] Table 2 Example of memory address decoding table

[0091] The memory address label switching table is used to record the mapping relationship between address switching labels and destination ports. The address switching label can be a memory address switching label, and the destination port can be a port of a destination node. The destination node can be a computing node or a switching node. In some possible implementations, the memory address label switching table can store the address switching label and the port number of the destination port as shown below:

[0092] Table 3 Example of memory address label switching table

[0093] The topology table is used to record the mapping relationship between source nodes and destination nodes. Both source nodes and destination nodes have ports. The source nodes and destination nodes establish a mapping relationship through the ports. In some possible implementations, the topology table is used to store the port number of the source node and the corresponding port number of the destination node, as shown below:

[0094] Table 4 Example of a topology table

[0095] It should be noted that step ① above is an optional step and may not be performed when executing the fault recovery method of the embodiment of the present application. For example, the controller 202 may also obtain networking information through detection. For another example, the switching node maintains the memory address decoding table and memory address exchange label table of the node. Based on this, the controller 202 may receive the memory address decoding table and memory address exchange label table reported by the switching node, thereby obtaining networking information. It should be noted that the memory address decoding table and memory address exchange label table maintained by the switching node are usually local, for example, the memory address decoding table and memory address exchange label table related to the switching node. The controller 202 receives the memory address decoding table and memory address exchange label table reported by each switching node and aggregates them, so as to maintain a global memory address decoding table and memory address exchange label table.

[0096] ② The fault sensing device 204 senses the fault of the switching node and obtains fault information.

[0097] The fault sensing device 204 deployed on a switching node can sense hardware faults in the node (the switching device). This hardware fault can include an unrepairable fault, specifically a fault that the hardware cannot automatically repair. When an unrepairable fault occurs on a switching node, the fault sensing device 204 on the switching node can sense the unrepairable fault and obtain fault information.

[0098] Fault information indicates the fault mode and fault location. The fault mode is used to describe the fault type, which can be classified according to the scope of the fault impact. For a bus switching network, the fault mode may include the following modes:

[0099] Failure mode 1: A partial link failure occurs between the first chip in the first switching node and the first computing node (for example, the xPU of the first computing node); the first computing node may be any computing node in the computing cluster, and the first switching node may be one or more switching nodes in the bus switching network. The first chip may be one or more chips in the first switching node. It should be noted that the first switching node may be a first-level switching node (L1 switching node). An L1 switching node refers to a switching node directly connected to a computing node. The L1 switching node detects a partial link failure between the chip in this node and the computing node xPU, indicating that the failure mode of the bus switching network failure is Failure mode 1.

[0100] Failure Mode 2: All links connecting the first chip in the first switching node to the first compute node (e.g., the xPU of the first compute node) fail. For example, if the L1 switching node detects that all links connecting the chips in the node to the compute node xPU have failed, this indicates that the bus switching network has experienced a failure mode 2.

[0101] Fault Mode 3: The first chip in the first switching node fails. For example, the L1 switching node detects a chip failure in the node, indicating that the fault mode of the bus switching network is Fault Mode 3.

[0102] Failure mode 4: Partial link failure between the first chip in the first switching node and the second switching node (for example, the second chip in the second switching node). The second switching node is one or more switching nodes in the bus switching network, for example, the second switching node may be an L2 switching node. An L2 switching node refers to a switching node that is indirectly connected to at least one computing node. The second chip may be one or more chips in the second switching node. If the L1 switching node detects a partial link failure between the chip in the node and the L2 switching node, the failure mode of the bus switching network is failure mode 4. It should be noted that the first switching node may also be an L2 switching node, and the second switching node may be an L1 switching node. If the L2 switching node detects a partial link failure between the chip in the node (such as the second chip) and the L1 switching node, the failure mode of the bus switching network is failure mode 4. In other words, failure mode 4 may be a partial link failure between different switching nodes (for example, chips in different switching nodes).

[0103] Failure mode 5: All links connecting the first chip in the first switching node and the second switching node (for example, the second chip in the second switching node) fail. The first switching node may be an L1 switching node, and the second switching node may be an L2 switching node. If the L1 switching node detects that all links connecting the chips in this node and the L2 switching node fail, then the failure mode of the bus switching network failure is Failure mode 5. It should be noted that the first switching node may also be an L2 switching node, and the second switching node may be an L1 switching node. If the L2 switching node detects that all links connecting the chips in this node (such as the second chip) and the L1 switching node fail, then the failure mode of the bus switching network failure is Failure mode 5. In other words, Failure mode 5 may be the failure of all links connecting different switching nodes (for example, chips in different switching nodes).

[0104] Failure Mode 6: Failure of the second chip in the second switching node. For example, the L2 switching node detects a chip failure in the node, indicating that the failure mode of the bus switching network is Failure Mode 6.

[0105] The fault location is used to describe the location where the fault occurred. The granularity of the fault location can vary depending on the fault mode. For example, when the fault mode is Fault Mode 1, Fault Mode 2, Fault Mode 4, or Fault Mode 5, the granularity of the fault location can be port-level. For example, the fault location can include a node identifier, a chip identifier, and a port identifier to indicate the node, chip, and port where the fault occurred. When the fault mode is Fault Mode 3 or Fault Mode 6, the granularity of the fault location can be chip-level. For example, the fault location can include a node identifier and a chip identifier to indicate the node and chip where the fault occurred.

[0106] It should be noted that step ② is an optional step and may not be performed when executing the fault recovery method of the embodiment of the present application. For example, the fault information may also be obtained through other fault detection methods or fault location methods.

[0107] ③ The controller 202 obtains fault information reported by at least one switching node.

[0108] The fault information indicates the fault mode and fault location. The fault location can be represented by the node identifier and chip identifier of the switching node. The controller 202 can also maintain a mapping relationship between the fault mode identifier and the fault mode. For example, the controller 202 can maintain a fault mode table as shown below:

[0109] Table 5 Failure mode table

[0110] Accordingly, the fault information can indicate the fault mode through the fault mode identifier, and indicate the fault location through the node identifier (such as SwitchId) and the chip identifier (such as ChipId). Furthermore, the fault information can also indicate the fault location in combination with the port number (also called the port identifier PortId). For example, when the fault mode identifier is fault mode 1, fault mode 2, fault mode 4, or fault mode 5, the fault information indicates the fault location through the node identifier, chip identifier, and port number. Furthermore, in a multi-layer bus switching network, the fault information can also indicate the fault location in combination with the level identifier (LevelId) of the switching node.

[0111] Based on this, fault information can include a fault mode identifier, a node identifier, and a chip identifier. Furthermore, the fault information can also include at least one of a level identifier and a port number. In some possible implementations, the fault information format can be: [ModeId-LevelId-SwitchId-ChipId-PortId], or [ModeId, LevelId-SwitchId-ChipId PortId]. For ease of understanding, the following example illustrates this. For example, the fault information reported by a switching node can be: [Fault Mode 1, L1-16-2 Port 6].

[0112] In specific implementation, the switching node may generate a fault notification, and the fault notification may carry the above fault information. The controller 202 receives the fault notification sent by the switching node, thereby obtaining the fault information reported by the switching node.

[0113] ④ The controller 202 determines an execution node for fault recovery from at least one switching node according to the fault information.

[0114] ⑤ The controller 202 determines the memory address communication path affected by the fault based on the fault information.

[0115] Based on the fault information, the controller 202 can determine the execution nodes and the memory address communication paths affected by the fault through the fault recovery strategy determination module 2024. Specifically, the controller 202 can parse the fault mode field in the fault information to determine the fault mode. For different fault modes, the controller 202 can use the fault recovery strategy calculation method corresponding to the fault mode to determine the execution nodes and the memory address communication paths affected by the fault.

[0116] The following first describes how to implement the determination of the execution node.

[0117] In some possible implementations, when the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, that is, the failure mode is failure mode 1, the controller 202 may determine that the faulty node indicated by the fault location is an execution node. Otherwise, the controller 202 determines the execution node based on the fault location in the fault information and the port connection relationship recorded in the topology table. The faulty node indicated by the fault location may be a fault reporting node.

[0118] The following describes the process of determining the execution node according to the fault location and the port connection relationship recorded in the topology table in fault modes other than fault mode 1.

[0119] When the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node, or a failure of the first chip in the first switching node, that is, the failure mode is failure mode 2 or failure mode 3, the controller 202 can query the switching node information table based on the faulty node indicated by the fault location to determine the uplink port of the faulty node, and query the topology table based on the uplink port of the faulty node to determine whether the switching node to which the uplink port of the faulty node is connected is an execution node. For example, the controller 202 can query the topology table to determine all L2 switching nodes to which the uplink port of the faulty node is connected as execution nodes.

[0120] When the failure mode is a partial link failure connecting the first chip in the first switching node and the second switching node, that is, the failure mode is failure mode 4, the controller 202 can query the topology table based on the faulty node and port indicated by the fault location to determine whether the port is connected to the switching node and the faulty node is an execution node. In other words, the execution node can be the switching node (which can be an L2 switching node) to which the faulty port (the port of the faulty node indicated by the fault location) is connected, and the switching node to which the faulty port belongs (that is, the faulty node itself indicated by the fault location, which can be an L1 switching node).

[0121] When the failure mode is failure of all links connecting the first chip in the first switching node and the second switching node, that is, the failure mode is failure mode 5, the controller 202 can query the switching node information table according to the failure node indicated by the failure location to determine the uplink port of the failure node, query the topology table according to the uplink port of the failure node to determine the switching node to which the uplink port of the failure node is connected, and query the topology table to determine that the switching node to which the chip connected to the same switching node as the chip indicated by the failure location belongs is the execution node.

[0122] When the fault mode is a second chip fault in the second switching node, that is, the fault mode is fault mode 6, the controller 202 can query the topology table based on the fault node indicated by the fault location to determine that the switching node connected to the fault node is an execution node. In other words, the execution node can be all L1 switching nodes connected to the faulty L2 switching node.

[0123] Next, the implementation method of determining the memory address communication path affected by the fault is described.

[0124] The memory address communication path affected by the fault includes a mapping between the address exchange labels of the affected compute nodes and the fault-associated ports. Depending on the fault mode, the fault-associated ports in the memory address communication path affected by the fault can be faulty ports of different nodes. The following describes the fault-associated ports and affected memory address communication paths for different fault modes.

[0125] When the failure mode is a failure of all links connecting the first chip in the first switching node and the computing node, a failure of the first chip in the first switching node, or a failure of some links connecting the first chip in the first switching node and the second switching node, that is, when the failure mode is failure mode 2, failure mode 3, or failure mode 4, the fault-associated port may include a port connected to the faulty node or chip indicated by the fault information, also referred to as a first fault-associated port. Accordingly, the memory address communication path affected by the fault includes a mapping of the address switching label of the computing node affected by the fault and the first fault-associated port.

[0126] The failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, or a partial link failure connecting the first chip in the first switching node and the second switching node, that is, when the failure mode is failure mode 1 or failure mode 4, the fault-associated port may include a port indicated by the fault information, also referred to as a second fault-associated port, and the memory address communication path affected by the fault includes a mapping between an address switching label of the computing node affected by the fault and the second fault-associated port.

[0127] When the failure mode is a failure of all links connecting the first chip in the first switching node and the second switching node, or a failure of the second chip in the second switching node, that is, the failure mode is failure mode 5 or failure mode 6, the fault-associated port may include a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs, also referred to as a third fault-associated port, and the memory address communication path affected by the fault includes a mapping of the address switching label of the computing node affected by the fault and the third fault-associated port.

[0128] In specific implementation, the controller 202 can determine the computing node affected by the fault based on the fault information, and then the controller 202 can query the address decoding table to determine the address exchange label of the computing node affected by the fault, and then the controller 202 determines the memory address communication path affected by the fault based on the address exchange label of the computing node affected by the fault.

[0129] When determining the computing nodes affected by the fault, the controller 202 can make a determination based on the fault mode in the fault information. Specifically, when the fault mode is a chip fault in the upper-layer switching node (for example, an L2 switching node), that is, when the fault mode is fault mode 6, the controller 202 can determine that the computing node in the computing cluster is a computing node affected by the fault. Otherwise, the controller 202 queries the topology table based on the port or chip of the fault node indicated by the fault location to determine that the computing node to which the port or chip of the fault node is connected is a computing node affected by the fault. Among them, when the fault mode is fault mode 1, fault mode 2, or fault mode 4, the controller 202 queries the topology table to determine that the computing node to which the port of the fault node is connected is a computing node affected by the fault. When the fault mode is fault mode 3 or fault mode 5, the controller 202 queries the topology table to determine that the computing node to which the chip of the fault node is connected is a computing node affected by the fault, so as to improve query efficiency.

[0130] It should be noted that the above ④ and ⑤ can be executed in parallel or in sequence, and the embodiments of the present application do not limit this.

[0131] ⑥ The controller 202 sends the memory address communication path affected by the fault to the execution node.

[0132] ⑦ The fault recovery execution device 206 updates the memory address communication path affected by the fault.

[0133] The memory address communication path affected by the fault includes a mapping between the address switching label of the computing node affected by the fault and the faulty port. This mapping may be an entry in a memory address switching label table. Fault recovery device 206 may update the entry in the memory address switching label table. In some possible implementations, fault recovery execution device 206 may delete the entry from the memory address switching label table, that is, delete the mapping between the address switching label of the computing node affected by the fault and the faulty port, thereby reducing packet loss.

[0134] It should be noted that when the fault mode is fault mode 6, the computing nodes affected by the fault may include all computing nodes in the computing cluster, and the execution node is the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs. The computing nodes affected by the fault connected to each execution node may be different, and their corresponding memory address exchange labels are also different. Therefore, the memory address communication path affected by the fault may also include a mapping of the memory address of the computing node affected by the fault connected to the execution node to the address exchange label. The mapping may be a table entry in the memory address decoding table. The fault recovery device 206 may update the above-mentioned table entry in the memory address decoding table, for example, by deleting the mapping of the memory address of the computing node affected by the fault to the address exchange label from the memory address decoding table, thereby reducing packet loss.

[0135] It should be noted that the above ⑥⑦ is an implementation method in which the controller 202 instructs the execution node to update the memory address communication path affected by the fault. In other possible implementation methods of the embodiment of the present application, the controller 202 may also instruct the execution node to update the memory address communication path affected by the fault in other ways.

[0136] Based on the above description, the fault recovery method of the present application determines the memory address communication path affected by the fault by combining the fault information reported by the switching node, and instructs the relevant execution node to update the memory address communication path affected by the fault, for example, deleting the memory address communication path, thereby realizing fault recovery without shutting down, and there is no need to shield single or multiple computing nodes, thereby avoiding waste of computing power.

[0137] FIG3 also shows a flowchart of determining an execution node and a communication path of a memory address affected by a fault under different fault modes, and then notifying the execution node to perform fault recovery, which are described below respectively.

[0138] When the controller 202 parses that the fault mode is fault mode 1, it determines that the execution node is the fault node indicated by the fault information, such as the fault reporting node. Then the controller 202 can query the topology table to obtain the computing node xPU connected to the port reported by the fault reporting node. The controller 202 can query the memory address decoding table to obtain the address switching label. The controller 202 can notify the execution node to delete the mapping of the address switching label to the port indicated by the fault information (such as the port of the L1 switching node).

[0139] When controller 202 determines that the fault mode is fault mode 2, it queries the switch node information table to obtain the uplink port of the faulty node indicated by the fault information. It then queries the topology table to determine the execution node. Controller 202 also queries the switch node information table to obtain the compute node xPU connected to the port indicated by the fault information. Controller 202 queries the memory address translation table to obtain the address switch label of the affected compute node xPU and instructs the execution node to delete the mapping of the address switch label to the port of the switch node connected to the uplink port (e.g., the faulty port of the L2 switch node).

[0140] When controller 202 determines that the fault mode is fault mode 3, it queries the switching node information table to obtain the uplink port of the faulty node indicated by the fault information. It then queries the topology table to determine the execution node. Controller 202 also queries the switching node information table to obtain the compute node xPU connected to the chip indicated by the fault information. Controller 202 queries the memory address translation table to obtain the address switch label of the affected compute node xPU and instructs the execution node to delete the mapping of the address switch label to the port of the switching node connected to the uplink port.

[0141] When the controller 202 determines that the fault mode is fault mode 4, it queries the topology table to obtain the ports of the switch nodes connected to the faulty node indicated in the fault information. This determines that the execution nodes include the faulty node indicated in the fault information and the switch nodes to which it is connected via the indicated ports. The controller 202 queries the switch node information table to obtain all compute nodes xPU connected to the faulty node indicated in the fault information. Next, the controller 202 queries the memory address translation table to obtain the address switch labels of the affected compute nodes xPUs and notifies the execution nodes to delete the mappings of these address switch labels to their respective ports associated with the fault.

[0142] When the controller 202 determines that the fault mode is fault mode 5, it queries the switch node information table to obtain the uplink port. Based on the uplink port, it queries the topology table to determine the port of the switch node to which it is connected (e.g., a port of an L2 switch node). The controller 202 then traverses the topology table to determine the chip of the L1 switch node to which the port of the L2 switch node is connected, and determines that the execution node is the switch node to which the chip connected to the same switch node as the chip indicated by the fault information belongs. The controller 202 queries the switch node information table to obtain the compute node xPU connected to the faulty node indicated by the fault information. The controller 202 queries the memory address decoding table to obtain the address switch label of the compute node xPU affected by the fault, and notifies the execution node to delete the mapping of the address switch label to the port of the switch node to which the chip connected to the same switch node as the chip indicated by the fault location belongs.

[0143] When controller 202 parses the fault mode as fault mode 6, it determines that the execution node is the switch node connected to the faulty node indicated in the fault information, for example, all L1 switch nodes. Controller 202 queries the switch node information table to obtain the compute nodes xPU connected to each execution node. Controller 202 queries the memory address translation table to obtain the address switch labels of the compute nodes xPU connected to each execution node, and instructs the execution node to delete the mapping of the address switch labels to the ports on each execution node connected to the faulty node indicated in the fault information.

[0144] Based on the aforementioned method embodiments, the fault recovery method of the present application is introduced below with reference to specific scenarios.

[0145] FIG4 shows a schematic diagram of the structure of a computing cluster, which is composed of 16 computing nodes connected via a two-layer bus switching network. Each computing node includes two CPUs and two NPUs, each L1 switching node includes two chips, and each L2 switching node includes two chips. Each endpoint of each connection between nodes (between a computing node and a switching node, and between switching nodes) represents two ports. In this example, each xPU includes four ports, and each xPU is connected to a chip in the L1 switching node via two ports. Each chip in the L1 switching node includes eight input ports and eight uplink ports (uplink ports). Each chip in the L1 switching node is connected to two ports of a chip in the L2 switching node via two ports. Each chip in the L2 switching node is connected to 32 chips in the 16 L1 switching nodes, for a total of 64 ports.

[0146] Taking the memory communication between nodes 1.14 and 1.F4 as an example, the fault recovery solutions under various fault modes are described respectively.

[0147] The controller 202 maintains the networking information of the computing cluster, as shown below:

[0148] Table 6 Topology table

[0149] Table 7 Switching Node Information Table

[0150] Table 8 Switching Node Information Table

[0151] Table 9 Memory address label exchange table

[0152] Table 10 Memory address label exchange table

[0153] The first scenario involves a partial link failure connecting a chip in an L1 switch node to a compute node. Specifically, controller 202 receives the fault information [fault mode 1, L1-16-2 port 6], queries the topology table, and finds that the port is connected to node 16-1.F4. It then queries the memory address translation table and finds that the address switching label for node 16-1.F4 is 64. It then queries the address switching label table and deletes the mapping from the address switching label to L1-16-2 port 6. This means that the mapping from [64, L1-16-2 port 6-7] in the address switching label table is updated to [64, L1-16-2 port 7]. In this fault mode, the execution node is the faulty node indicated by the fault information, namely, fault reporting node L1-16, specifically chip 2. Therefore, controller 202 notifies L1-16 to update the mapping from [64, L1-16-2 port 6-7] in the address switching label table to [64, L1-16-2 port 7].

[0154] The second scenario involves a complete failure of all links connecting the chips in the L1 switch node to the compute nodes. Specifically, controller 202 receives the fault information [failure mode 2, L1-16-2 port 6-7], queries the topology table, and finds that the port is connected to node 16-1.F4. It then queries the memory address translation table and finds that the address switching label for node 16-1.F4 is 64. Controller 202 also queries the switch node information table to obtain the uplink port for L1-16-2. The controller 202 queries the topology table according to the uplink port of L1-16-2 to obtain the port of the L2 switching node to which it is connected and the switching node and chip to which it belongs, specifically ports 62 and 63 of chip No. 1 and chip No. 2 of the L2-1 switching node (recorded as L2-1-1 port 62-63 and L2-1-2 port 62-63) and ports 62 and 63 of chip No. 1 and chip No. 2 of the L2-2 switching node (recorded as L2-2-1 port 62-63 and L2-2-2 port 62-63). Controller 202 notifies chip 1 of the L2-1 switch node to update the address switching label table from [64, L2-1-1 port 60-63] to [64, L2-1-1 port 60-61]. Controller 202 also notifies chip 2 of the L2-1 switch node to change the address switching label table from [64, L2-1-2 port 60-63] to [64, L2-1-2 port 60-61]. Controller 202 notifies chip 1 of the L2-2 switch node to update the address switching label table from [64, L2-2-1 port 60-63] to [64, L2-2-1 port 60-61]. Controller 202 also notifies chip 2 of the L2-2 switch node to change the address switching label table from [64, L2-2-2 port 60-63] to [64, L2-2-2 port 60-61].

[0155] The third scenario involves a chip failure in the L1 switch node. Specifically, controller 202 receives the fault information [failure mode 3, L1-16-2 chip] and queries the switch node information table to determine that the chip's connections are nodes 16-1.F1 through 16-1.F4. Controller 202 queries the memory address translation table and determines that the address switching labels for nodes 16-1.F1 through 16-1.F4 are 61 through 64. The controller 202 queries the switching node information table to obtain the uplink port of L1-16. Based on the uplink port of L1-16, it queries the topology table to obtain the port of the L2 switching node to which it is connected and the switching node and chip to which it belongs, specifically ports 62 and 63 of chip No. 1 and chip No. 2 of the L2-1 switching node (recorded as L2-1-1 port 62-63 and L2-1-2 port 62-63) and ports 62 and 63 of chip No. 1 and chip No. 2 of the L2-2 switching node (recorded as L2-2-1 port 62-63 and L2-2-2 port 62-63). Controller 202 notifies chip No. 1 of the L2-1 switching node to update [61-64, L2-1-1 port 60-63] in the address switching label table to [61-64, L2-1-1 port 60-61]; and notifies chip No. 2 of the L2-1 switching node to change [61-64, L2-1-2 port 60-63] in the address switching label table to [61-64, L2-1-2 port 60-61]. Controller 202 notifies chip No. 1 of the L2-2 switching node to update [61-64, L2-2-1 port 60-63] in the address switching label table to [61-64, L2-2-1 port 60-61], and notifies chip No. 2 of the L2-2 switching node to update [61-64, L2-2-2 port 60-63] in the address switching label table to [61-64, L2-2-2 port 60-61].

[0156] The fourth scenario involves a partial link failure between the chip in the L1 switch node and the L2 switch node. Specifically, controller 202 receives the fault information [fault mode 4, L1-16-2 port 14], queries the TOPO table, and finds that the port's connection object is L2-2-2 port 62. Controller 202 queries the switch node information table and finds that L1-16-2 connects nodes 16-1.F1 to 16-1.F4. Controller 202 queries the memory address decoding table and finds that the address switching labels for nodes 16-1.F1 to 16-1.F4 are 61 to 64. In this fault mode, the execution nodes are L1-16 (specifically, chip 2 of L1-16, i.e., L1-16-2) and L2-2 (specifically, chip 2 of L2-2, i.e., L2-2-2). Therefore, the controller 202 notifies the chip No. 2 of the L1-16 switching node to update the [61~64, L1-16-2 port 8-15] in the address switching label table to [61~64, L1-16-2 port 8-13, 15], and notifies the chip No. 2 of the L2-2 switching node to update the [61~64, L2-2-2 port 60-63] in the address switching label table to [61~64, L2-2-2 port 60-61, 63].

[0157] The fifth scenario involves a complete failure of all links connecting the chip in the L1 switch node to the L2 switch node. Specifically, controller 202 receives the fault information [failure mode 5, L1-16-2 ports 14-15] and queries the switch node information table to determine that the chip is connected to nodes 16-1.F1 through 16-1.F4. Controller 202 queries the memory address translation table to determine that the address switching labels for nodes 16-1.F1 through 16-1.F4 are 61 through 64. Controller 202 queries the switch node information table to obtain the uplink port of L1-16. Based on the uplink port of L1-16, controller 202 queries the topology table to obtain the port of the L2 switch node to which it is connected, as well as the corresponding switch node and chip L2-2. Controller 202 traverses the topology table to obtain the chips of other L1 switch nodes connected to L2-2, such as L1-1-1, L1-1-2, L1-2-1, L1-2-2, ..., L16-1-1. Controller 202 notifies chips 1 and 2 of the L1-1~15 switching node to update [61~64, L1-1-1 port 8-15] in the address switching label table to [61~64, L1-1-1 port 8-13], and notifies chip 1 of the L1-16 switching node to update [61~64, L1-1-1 port 8-15] in the address switching label table to [61~64, L1-1-1 port 8-13].

[0158] The sixth case is a chip failure in an L2 switch node. Specifically, controller 202 receives the fault information [failure mode 6, L2-2-2 chip] and traverses the topology table to obtain the chips and ports in all L1 switch nodes connected to L2-2, such as L1-1-1, L1-1-2, L1-2-1, L1-2-2 and their ports 14 and 15, ... L16-1-1, L16-1-2 and their ports 62 and 63. Controller 202 queries the memory address decoding table and obtains the address switching labels of all nodes, which are 1 to 64. Among them, the address switching labels of L1-1-1, L1-1-2 and their ports 14 and 15 connected to node xPU are 1-4, and the address switching labels of L1-2-1, L1-2-2 and their ports 14 and 15 connected to node xPU are 5-8. Controller 202 notifies chips No. 1 and No. 2 of the L1-1 switching node to change the address switching label table from [5 to 64, L1-1-1 port 8-15] to [5 to 64, L1-1-1 port 8-13], and notifies chips No. 1 and No. 2 of the L1-2 switching node to change the address switching label table from [1 to 4, 9 to 64, L1-1-1 port 8-15] to [1 to 4, 9 to 64, L1-1-1 port 8-13], until all chips in the L1 switching nodes connected to L2-2 have been notified to update the address switching label table.

[0159] Based on the aforementioned embodiments, the present application further provides a controller 202. The controller 202 is configured to execute the fault recovery method of the aforementioned embodiments to recover from a fault in a bus switching network within a computing cluster. The controller 202 provided in the present embodiment is described below from the perspective of functional modularization.

[0160] 2 shows a schematic diagram of a controller 202. The controller 202 is used to recover from a fault in a bus switching network in a computing cluster. The bus switching network includes at least one switching node. The bus switching network is used to implement memory access between computing nodes in the computing cluster. The controller 202 includes:

[0161] The communication module 2022 is configured to obtain fault information reported by the at least one switching node, where the fault information indicates a fault mode and a fault location;

[0162] a fault recovery strategy determination module 2024, configured to determine an execution node for fault recovery from the at least one switching node based on the fault information, and to determine a memory address communication path affected by the fault based on the fault information;

[0163] The communication module 2022 is further configured to instruct the execution node to update the memory address communication path affected by the fault.

[0164] Exemplarily, the communication module 2022 and the fault recovery strategy determination module 2024 may be implemented by hardware or software.

[0165] When implemented via software, the communication module 2022 and the fault recovery strategy determination module 2024 can be applications running on a computer device, such as a computing engine. Furthermore, the application can be virtualized via virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can be services that use virtualization technology to create a virtual machine (VM) resource pool on multiple physical hosts to provide users with VMs on demand. BMS services are services that create a virtual BMS resource pool on multiple physical hosts to provide users with BMSs on demand. Container services are services that create a virtual container resource pool on multiple physical hosts to provide users with containers on demand. A VM is a simulated virtual computer, or logically a single computer. BMS is a scalable, high-performance computing service with computing performance comparable to that of a traditional physical machine and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0166] When implemented in hardware, the communication module 2022 may be a transceiver, and the fault recovery strategy determination module 2024 may include at least one computing device, such as a server. Alternatively, the fault recovery strategy determination module 2024 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0167] In some possible implementations, the fault recovery strategy determination module 2024 is specifically configured to:

[0168] When the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, determining that the faulty node indicated by the fault location is an execution node; otherwise, determining the execution node based on the fault location and the port connection relationship recorded in the topology table;

[0169] The first switching node is one or more switching nodes in the bus switching network, the first chip is one or more chips in the first switching node, and the first computing node is any computing node in the computing cluster.

[0170] In some possible implementations, the fault recovery strategy determination module 2024 is specifically configured to:

[0171] The failure mode is that all links connecting the first chip in the first switching node and the first computing node fail, or the first chip in the first switching node fails, and the switching node information table is queried based on the faulty node indicated by the fault location to determine the uplink port of the faulty node, and the topology table is queried based on the uplink port of the faulty node to determine that the switching node connected to the uplink port of the faulty node is an execution node; or

[0172] The failure mode is a partial link failure connecting the first chip in the first switching node and the second switching node, and the topology table is queried based on the faulty node and port indicated by the fault location to determine that the switching node to which the port is connected and the faulty node is the execution node, and the second switching node is one or more switching nodes in the bus switching network; or

[0173] The failure mode is that all links connecting the first chip in the first switching node and the second switching node fail; the switching node information table is queried based on the faulty node indicated by the fault location to determine the uplink port of the faulty node; the topology table is queried based on the uplink port of the faulty node to determine the switching node to which the uplink port of the faulty node is connected; the topology table is queried to determine that the switching node to which the chip uplinked to the same switching node as the chip indicated by the fault location belongs is the execution node; or

[0174] The failure mode is a second chip failure in the second switching node. A topology table is queried according to the failure node indicated by the failure position to determine that the switching node connected to the failure node is the execution node.

[0175] In some possible implementations, the memory address communication path affected by the fault includes a mapping of an address exchange tag of a computing node affected by the fault and a fault-associated port, and the fault recovery strategy determination module 2024 is specifically configured to:

[0176] Determining, based on the fault information, computing nodes affected by the fault;

[0177] The address decoding table is queried to determine the address exchange label of the computing node affected by the fault, and the memory address communication path affected by the fault is determined according to the address exchange label of the computing node affected by the fault.

[0178] In some possible implementations, the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node, a failure of the first chip in the first switching node, or a failure of part of the links connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of the address switching label of the computing node affected by the failure and the first failure-associated port, and the first failure-associated port is a port connected to the faulty node or chip indicated by the fault information; the failure mode is a failure of part of the links connecting the first chip in the first switching node and the first computing node, or a failure of the first chip in the first switching node and the second switching node. Part of the link connecting the nodes fails, the memory address communication path affected by the fault includes a mapping between the address exchange label of the computing node affected by the fault and the second fault-associated port, and the second fault-associated port is the port indicated by the fault information; when the fault mode is a complete link failure connecting the first chip in the first switching node and the second switching node, or a failure of the second chip in the second switching node, the memory address communication path affected by the fault includes a mapping between the address exchange label of the computing node affected by the fault and the third fault-associated port, and the third fault-associated port is a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs.

[0179] In some possible implementations, the fault recovery strategy determination module 2024 is specifically configured to:

[0180] When the fault mode is a chip failure in the upper-layer switching node, the computing node in the computing cluster is determined to be the computing node affected by the fault; otherwise, the topology table is queried according to the port or chip of the fault node indicated by the fault location to determine that the computing node to which the port or chip of the fault node is connected is the computing node affected by the fault.

[0181] In some possible implementations, the fault location includes a node identifier and a chip identifier of the faulty node.

[0182] The controller 202 is described above from the perspective of functional modularization. Next, the controller 202 is described from the perspective of hardware implementation.

[0183] This application also provides a computing device 202. As shown in Figure 5, controller 202 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. Controller 202 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in controller 202.

[0184] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG2 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 502 may include a path for transmitting information between various components of controller 202 (e.g., memory 506, processor 504, and communication interface 508).

[0185] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 506 may include a volatile memory, such as a random access memory (RAM). The memory 506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 506 stores executable program code, and the processor 504 executes the executable program code to implement the aforementioned fault recovery method. Specifically, the memory 506 stores instructions for the controller 202 to execute the fault recovery method.

[0187] The communication interface 508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the controller 202 and other devices or a communication network.

[0188] Based on the aforementioned controller 202, the present application also provides a fault recovery system 200. As shown in Figure 2, fault recovery system 200 is used to perform fault recovery when a bus switching network in a computing cluster fails. The bus switching network includes at least one switching node, which is used to implement memory access between computing nodes in the computing cluster. The fault recovery system includes controller 202 and a fault recovery execution device 206 deployed on at least one switching node.

[0189] In specific implementation, the controller 202 is used to obtain fault information reported by at least one switching node, where the fault information indicates the fault mode and fault location, and then determine the execution node for fault recovery from at least one switching node based on the fault information, and determine the memory address communication path affected by the fault based on the fault information, and then instruct the execution node to update the memory address communication path affected by the fault; the fault recovery execution device 206 is used to update the memory address communication path affected by the fault according to the instruction of the controller 202.

[0190] In some possible implementations, fault recovery system 200 may further include a fault sensing device 204. Fault sensing device 204 may be deployed in at least one switching node. Upon detecting a fault, fault sensing device 204 is configured to report fault information to controller 202. For example, fault sensing device 204 may perform fault detection using built-in detection logic and, upon detecting an unrecoverable fault, report the fault information to controller 202.

[0191] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that the controller can store, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the controller 202 to execute the fault recovery method.

[0192] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a controller or stored on any available medium. When the computer program product is run on the controller, the controller 202 executes the aforementioned fault recovery method.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A fault recovery method, characterized in that: Applied to a controller, the controller is used to perform fault recovery when a bus switching network in a computing cluster fails, the bus switching network includes at least one switching node, and the bus switching network is used to implement memory access between computing nodes in the computing cluster, the method comprising: The controller acquires fault information reported by the at least one switching node, wherein the fault information indicates a fault mode and a fault location; The controller determines an execution node for fault recovery from the at least one switching node according to the fault information, and determines a memory address communication path affected by the fault according to the fault information; The controller instructs the execution node to update the memory address communication path affected by the fault.

2. The method according to claim 1, characterized in that The controller determines an execution node from the at least one switching node according to the fault information, including: When the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, the controller determines that the faulty node indicated by the fault location is an execution node, otherwise, the controller determines the execution node according to the fault location and the port connection relationship recorded in the topology table; The first switching node is one or more switching nodes in the bus switching network, the first chip is one or more chips in the first switching node, and the first computing node is any computing node in the computing cluster.

3. The method according to claim 2, characterized in that The controller determines the execution node according to the fault location and the port connection relationship recorded in the topology table, including: The failure mode is that all links connecting the first chip in the first switching node and the first computing node fail, or the first chip in the first switching node fails, the controller queries the switching node information table according to the faulty node indicated by the faulty location to determine the uplink port of the faulty node, and queries the topology table according to the uplink port of the faulty node to determine that the switching node connected to the uplink port of the faulty node is an execution node; or The failure mode is a partial link failure connecting the first chip in the first switching node and the second switching node, the controller queries the topology table according to the faulty node and port indicated by the faulty location to determine that the switching node connected to the port and the faulty node are the execution node, and the second switching node is one or more switching nodes in the bus switching network; or The failure mode is that all links connecting the first chip in the first switching node and the second switching node fail, the controller queries the switching node information table according to the faulty node indicated by the faulty location to determine the uplink port of the faulty node, queries the topology table according to the uplink port of the faulty node to determine the switching node to which the uplink port of the faulty node is connected, queries the topology table to determine that the switching node to which the chip uplinked to the same switching node as the chip indicated by the faulty location belongs is the execution node; or, The failure mode is a second chip failure in the second switching node, and the controller queries a topology table according to the failure node indicated by the failure position to determine that the switching node connected to the failure node is the execution node.

4. The method according to any one of claims 1 to 3, characterized in that: The memory address communication path affected by the fault includes a mapping of an address exchange tag of a computing node affected by the fault and a fault-associated port, and the controller determines the memory address communication path affected by the fault according to the fault information, including: The controller determines the computing nodes affected by the fault according to the fault information; The controller queries the address decoding table to determine the address exchange label of the computing node affected by the fault, and determines the memory address communication path affected by the fault according to the address exchange label of the computing node affected by the fault.

5. The method according to claim 4, characterized in that When the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node, a failure of the first chip in the first switching node, or a failure of part of the links connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of an address switching label of the computing node affected by the failure and a first failure-associated port, and the first failure-associated port is a port connected to the failed node or chip indicated by the failure information; The failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, or a partial link failure connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of an address switching label of the computing node affected by the failure and a second failure-associated port, and the second failure-associated port is a port indicated by the failure information; When the failure mode is a failure of all links connecting the first chip in the first switching node and the second switching node, or a failure of the second chip in the second switching node, the memory address communication path affected by the failure includes a mapping of the address switching label of the computing node affected by the failure and a third failure associated port, and the third failure associated port is a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs.

6. The method according to claim 4, characterized in that The controller determines, according to the fault information, a computing node affected by the fault, including: When the fault mode is a chip failure in an upper-layer switching node, the controller determines that a computing node in the computing cluster is a computing node affected by the fault; otherwise, the controller queries a topology table based on the port or chip of the faulty node indicated by the fault location to determine that the computing node to which the port or chip of the faulty node is connected is a computing node affected by the fault.

7. The method according to any one of claims 1 to 6, characterized in that: The fault location includes a node identifier and a chip identifier of the faulty node.

8. The method according to any one of claims 1 to 7, characterized in that: The controller is independent of the computing cluster.

9. The method according to any one of claims 1 to 7, characterized in that: The controller is a node in the computing cluster, or the controller is deployed on a node in the computing cluster.

10. A controller, characterized in that: The controller is used to perform fault recovery when a bus switching network in the computing cluster fails, the bus switching network includes at least one switching node, the bus switching network is used to implement memory access between computing nodes in the computing cluster, and the controller includes: A communication module, configured to obtain fault information reported by the at least one switching node, wherein the fault information indicates a fault mode and a fault location; a fault recovery strategy determination module, configured to determine an execution node for fault recovery from the at least one switching node according to the fault information, and to determine a memory address communication path affected by the fault according to the fault information; The communication module is further used to instruct the execution node to update the memory address communication path affected by the fault.

11. The controller according to claim 10, characterized in that: The fault recovery strategy determination module is specifically used for: When the failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, determining the faulty node indicated by the fault location as an execution node; otherwise, determining the execution node according to the fault location and the port connection relationship recorded in the topology table; The first switching node is one or more switching nodes in the bus switching network, the first chip is one or more chips in the first switching node, and the first computing node is any computing node in the computing cluster.

12. The controller according to claim 11, characterized in that: The fault recovery strategy determination module is specifically used for: The failure mode is that all links connecting the first chip in the first switching node and the first computing node fail, or the first chip in the first switching node fails, and the switching node information table is queried according to the faulty node indicated by the faulty location to determine the uplink port of the faulty node, and the topology table is queried according to the uplink port of the faulty node to determine that the switching node connected to the uplink port of the faulty node is an execution node; or The failure mode is a partial link failure connecting the first chip in the first switching node and the second switching node, the topology table is queried according to the failed node and port indicated by the failure position to determine that the switching node connected to the port and the failed node are the execution node, and the second switching node is one or more switching nodes in the bus switching network; or, The failure mode is that all links connecting the first chip in the first switching node and the second switching node fail, query the switching node information table according to the faulty node indicated by the fault location to determine the uplink port of the faulty node, query the topology table according to the uplink port of the faulty node to determine the switching node to which the uplink port of the faulty node is connected, query the topology table to determine that the switching node to which the chip uplinked to the same switching node as the chip indicated by the fault location belongs is the execution node; or, The failure mode is a second chip failure in the second switching node, and a topology table is queried according to the failure node indicated by the failure position to determine that the switching node connected to the failure node is the execution node.

13. The controller according to any one of claims 10 to 12, characterized in that: The memory address communication path affected by the fault includes a mapping between an address exchange label of a computing node affected by the fault and a fault-associated port, and the fault recovery strategy determination module is specifically used to: Determine the computing nodes affected by the fault according to the fault information; The address decoding table is queried to determine the address exchange label of the computing node affected by the fault, and the memory address communication path affected by the fault is determined according to the address exchange label of the computing node affected by the fault.

14. The controller according to claim 13, characterized in that: When the failure mode is a failure of all links connecting the first chip in the first switching node and the first computing node, a failure of the first chip in the first switching node, or a failure of part of the links connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of an address switching label of the computing node affected by the failure and a first failure-associated port, and the first failure-associated port is a port connected to the failed node or chip indicated by the failure information; The failure mode is a partial link failure connecting the first chip in the first switching node and the first computing node, or a partial link failure connecting the first chip in the first switching node and the second switching node, the memory address communication path affected by the failure includes a mapping of an address switching label of the computing node affected by the failure and a second failure-associated port, and the second failure-associated port is a port indicated by the failure information; When the failure mode is a failure of all links connecting the first chip in the first switching node and the second switching node, or a failure of the second chip in the second switching node, the memory address communication path affected by the failure includes a mapping of the address switching label of the computing node affected by the failure and a third failure associated port, and the third failure associated port is a port of the switching node to which the chip connected to the same switching node as the chip indicated by the fault location belongs.

15. The controller according to claim 13, characterized in that: The fault recovery strategy determination module is specifically used for: When the fault mode is a chip failure in an upper-layer switching node, the computing node in the computing cluster is determined to be a computing node affected by the fault; otherwise, a topology table is queried according to the port or chip of the faulty node indicated by the fault location to determine that the computing node to which the port or chip of the faulty node is connected is the computing node affected by the fault.

16. The controller according to any one of claims 10 to 15, characterized in that: The fault location includes a node identifier and a chip identifier of the faulty node.

17. A controller, characterized in that: The controller includes a processor and a memory, wherein computer-readable instructions are stored in the memory; the processor executes the computer-readable instructions so that the controller performs the method according to any one of claims 1 to 9.

18. A fault recovery system, characterized in that: The fault recovery system is used to perform fault recovery when a bus switching network in a computing cluster fails, the bus switching network includes at least one switching node, the bus switching network is used to implement memory access between computing nodes in the computing cluster, and the fault recovery system includes a controller and a fault recovery execution device deployed on the at least one switching node; The controller is configured to obtain fault information reported by the at least one switching node, the fault information indicating a fault mode and a fault location, determine an execution node for fault recovery from the at least one switching node according to the fault information, and determine a memory address communication path affected by the fault according to the fault information, and the controller instructs the execution node to update the memory address communication path affected by the fault; The fault recovery execution device is used to update the memory address communication path affected by the fault according to the instruction of the controller.

19. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

20. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images