Distributed computing system and failure handling method for distributed computing system

Abstract

The present application relates to the technical field of computers, and discloses a distributed computing system and a failure handling method for the distributed computing system. In the distributed computing system provided by the present application, a computing node group and a backup node both access an exchange device, and therefore, when detecting that a failure occurs in the computing node group, a management node of the distributed computing system can configure the exchange device to establish a data exchange path between the backup node and the computing node group, such that the backup node can perform data exchange with the computing node group via the data exchange path to replace a failed device in the computing node group to execute a computing task. In this way, the failure handling costs are greatly reduced while maintaining the original system performance and the network topology.

Description

Distributed computing system and fault handling method for distributed computing system

[0001] This application claims priority to Chinese patent application number 202311094187.1 filed on August 25, 2023, entitled “Distributed computing system and fault handling method for distributed computing system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a distributed computing system and a fault handling method for the distributed computing system. Background Art

[0003] As the scale of computing tasks in scenarios such as artificial intelligence (AI) and high-performance computing (HPC) continues to grow, distributed computing systems have emerged. In some cases, by grouping the computing nodes in the system and providing high-bandwidth interconnection between the computing nodes within each computing node group, the high-bandwidth interconnection within the node can be expanded to between nodes, improving cross-node communication performance and, in turn, the efficiency of computing task execution.

[0004] In related technologies, in order to prevent the performance of the entire computing node group from being affected by node failure or accelerator failure within the node, redundant backup is usually performed within the group. For example, redundant backup is performed for the nodes within the group. If a node within the group fails, the backup node is used to replace the failure. For another example, redundant backup is performed for the accelerator within the node. If an accelerator within the node fails, the backup accelerator is used to replace the failure.

[0005] However, since the scale of computing nodes in distributed computing systems in scenarios such as AI and HPC is often large, the above approach is costly, has a large amount of redundancy, and is prone to waste of resources.

[0006] Summary of the Invention

[0007] The present invention provides a distributed computing system and a method for troubleshooting a distributed computing system, which can effectively reduce troubleshooting costs. The technical solution is as follows:

[0008] In a first aspect, a distributed computing system is provided, comprising a management node, a switching device, a computing node group, and a backup node, wherein the switching device is physically connected to the computing node group and the backup node, respectively, each computing node in the computing node group includes multiple accelerators, and the backup node includes multiple backup accelerators;

[0009] The management node is used to:

[0010] detecting that a fault occurs in the computing node group, and determining a faulty device where the fault occurs, wherein the faulty device is a faulty computing node or a faulty accelerator in the computing node group;

[0011] The switching device is configured to establish a data exchange path between the backup node and the computing node group through the switching device, wherein the data exchange path is used for data exchange between the backup node and the computing node group so that the backup node replaces the faulty device to perform computing tasks.

[0012] In the distributed computing system provided in the present application, the computing task is a distributed parallel computing task. Since both the computing node group and the backup node are connected to the switching device, when the management node of the distributed computing system detects a fault in the computing node group, the switching device can be configured to establish a data exchange path between the backup node and the computing node group, so that the backup node can exchange data with the computing node group through this data exchange path to replace the faulty device in the computing node group to perform the computing task, thereby greatly reducing the fault handling cost while maintaining the original performance of the system and the network topology.

[0013] In some embodiments, the switching device is an optical switch, and the optical switch includes a plurality of optical fiber ports, and the plurality of optical fiber ports are used to access the backup node and the computing node group.

[0014] In some embodiments, the management node is configured to:

[0015] configuring an optical path between a first optical fiber port and a second optical fiber port on the optical switch to establish an optical path between the backup node and the computing node group through the optical switch;

[0016] In which, the optical path is used to exchange optical signals between the backup node and the computing node group, so that the backup node replaces the faulty device to perform the computing task, the first optical fiber port is the optical fiber port connected to the backup node, and the second optical fiber port is the optical fiber port connected to the computing node group, and the optical signal is used to carry the data corresponding to the computing task.

[0017] Through the above method, an optical switch is used as a switching device. Since the optical switch is a data switching device that directly exchanges optical signals without the need for electrical-optical-photoelectric conversion, it can ensure that the network topology and communication performance of the computing node cluster in the system remain unchanged after the fault is replaced, ensuring that the execution efficiency of the computing tasks before and after the fault occurs remains unchanged.

[0018] In some embodiments, the computing nodes in the computing node group are connected via a first switching component, at least one port of the first switching component is used to access the optical switch, and the first switching component is a switch or a switching chip.

[0019] It should be understood that since failures occurring within a computing node group are usually single-point failures, such as a single accelerator failure or a single computing node failure, connecting a small number of ports corresponding to the computing node group to the optical switch can reduce the requirements for switching ports while meeting fault replacement requirements, thereby significantly reducing the cost of infrastructure such as optical modules, optical fibers, and optical switches.

[0020] In some embodiments, the backup accelerators in the backup node are connected via a second switching component, at least one port of the second switching component is used to access the optical switch, and the second switching component is a switch or a switching chip.

[0021] In some embodiments, the management node is used for at least one of the following:

[0022] If the faulty device includes a first number of faulty computing nodes, configuring the switching device to establish, through the switching device, a first data exchange path between the first number of backup nodes and at least one first node, wherein the first data exchange path is used for exchanging data between the first number of backup nodes and the at least one first node, so that the first number of backup nodes replace the first number of faulty computing nodes to perform the computing task, the first nodes being nodes in the computing node group other than the faulty computing node;

[0023] If the faulty device includes a second number of faulty accelerators, configuring the switching device to establish, through the switching device, a second data exchange path between the second number of backup accelerators in the at least one backup node and the at least one first accelerator, wherein the second data exchange path is used for exchanging data between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform the computing task, where the first accelerator refers to the accelerators in the computing node group other than the faulty accelerator;

[0024] If the fault device includes a third number of fault accelerators, determine a fourth number of nodes to which the third number of fault accelerators belong, and configure the switching device to establish a third data exchange path between the fourth number of backup nodes and the at least one first node through the switching device, wherein the third data exchange path is used for data exchange between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of fault accelerators belong to perform the computing task.

[0025] Through the above method, the management node configures the switching device based on the number of faulty devices to connect the backup node with the computing node group through the switching device. By dynamically allocating backup nodes in this way, the flexibility of fault replacement is improved on the basis of realizing dynamic fault replacement.

[0026] In some embodiments, the computing nodes in the computing node group exchange data based on a first bandwidth, and the backup node is used to exchange data with the computing node group based on the first bandwidth through the data exchange path to replace the faulty device to perform the computing task.

[0027] In this way, after a failure occurs, the network topology and communication performance of the computing node cluster in the system can remain unchanged after the failure is replaced, ensuring that the execution efficiency of the computing tasks before and after the failure remains unchanged.

[0028] In some embodiments, the system includes multiple computing node groups, each computing node group has the same number of computing nodes, and each computing node group exchanges data based on a second bandwidth, and the first bandwidth is greater than or equal to the second bandwidth.

[0029] In some embodiments, the bandwidths for data exchange between computing nodes in different computing node groups are different, and the bandwidths for data exchange between computing nodes in different computing node groups are all greater than or equal to the second bandwidth.

[0030] In some embodiments, the number of accelerators in each computing node in the computing node group is the same, and data is exchanged between the accelerators based on a third bandwidth, and the first bandwidth is less than or equal to the third bandwidth.

[0031] In some embodiments, data is exchanged between backup accelerators in the backup node based on the third bandwidth.

[0032] Through the above method, a cluster composed of multiple computing node groups can adopt a hierarchical design with different bandwidths to improve the execution efficiency of computing tasks.

[0033] In some embodiments, the distributed computing system is used to execute a distributed parallel training task of an artificial intelligence (AI) model, and the faulty device is used to execute a subtask of the distributed parallel training task.

[0034] In a second aspect, an embodiment of the present application provides a fault handling method for a distributed computing system, which is applied to a management node of the distributed computing system. The system also includes a switching device, a computing node group, and a backup node. The switching device is physically connected to the computing node group and the backup node, respectively. Each computing node in the computing node group includes multiple accelerators, and the backup node includes multiple backup accelerators. The method includes:

[0035] detecting that a fault occurs in the computing node group, and determining a faulty device where the fault occurs, wherein the faulty device is a faulty computing node or a faulty accelerator in the computing node group;

[0036] The switching device is configured to establish a data exchange path between the backup node and the computing node group through the switching device, wherein the data exchange path is used for data exchange between the backup node and the computing node group so that the backup node replaces the faulty device to perform computing tasks.

[0037] In some embodiments, the switching device is an optical switch, the optical switch includes multiple optical fiber ports, the multiple optical fiber ports are used to access the backup node and the computing node group, and the configuring the switching device to establish a data exchange path between the backup node and the computing node group through the switching device includes:

[0038] configuring an optical path between a first optical fiber port and a second optical fiber port on the optical switch to establish an optical path between the backup node and the computing node group through the optical switch;

[0039] In which, the optical path is used to exchange optical signals between the backup node and the computing node group, so that the backup node replaces the faulty device to perform the computing task, the first optical fiber port is the optical fiber port connected to the backup node, and the second optical fiber port is the optical fiber port connected to the computing node group, and the optical signal is used to carry the data corresponding to the computing task.

[0040] In some embodiments, configuring the switching device to establish a data exchange path between the backup node and the computing node group through the switching device includes at least one of the following:

[0041] If the faulty device includes a first number of faulty computing nodes, configuring the switching device to establish, through the switching device, a first data exchange path between the first number of backup nodes and at least one first node, the first data exchange path being used for exchanging data between the first number of backup nodes and the at least one first node, so that the first number of backup nodes replace the first number of faulty computing nodes to perform the computing task, the first nodes being nodes in the computing node group other than the faulty computing node;

[0042] If the faulty device includes a second number of faulty accelerators, the switching device is configured to establish, through the switching device, a second data exchange path between the second number of backup accelerators in the at least one backup node and the at least one first accelerator, the second data exchange path being used for data exchange between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform the computing task, the first accelerator being the accelerator in the computing node group other than the faulty accelerator;

[0043] If the fault device includes a third number of fault accelerators, determine a fourth number of nodes to which the third number of fault accelerators belong, configure the switching device to establish a third data exchange path between the fourth number of backup nodes and the at least one first node through the switching device, and the third data exchange path is used for data exchange between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of fault accelerators belong to perform the computing task.

[0044] In a third aspect, an embodiment of the present application provides a fault handling device for a distributed computing system, which is applied to a management node in a distributed computing system. The system also includes a switching device, a computing node group, and a backup node. The switching device is physically connected to the computing node group and the backup node, respectively. Each computing node in the computing node group includes multiple accelerators, and the backup node includes multiple backup accelerators. The device includes at least one functional module for executing a fault handling method for a distributed computing system as provided in the aforementioned second aspect or any possible implementation of the second aspect.

[0045] In a fourth aspect, an embodiment of the present application provides a management device, which includes a processor and a memory, and the processor is used to execute at least one piece of program code stored in the memory so that the management device executes the fault handling method for a distributed computing system provided by the aforementioned second aspect or any possible implementation of the second aspect.

[0046] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium for storing at least one program code. When the at least one program code is executed on a management device, the management device executes the fault handling method for a distributed computing system provided in the second aspect or any possible implementation of the second aspect. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0047] In a sixth aspect, embodiments of the present application provide a computer program product that, when executed on a management device, causes the management device to execute the fault handling method for a distributed computing system as provided in the aforementioned second aspect or any possible implementation of the second aspect. The computer program product may be a software installation package, and when the aforementioned method is to be implemented, the computer program product may be downloaded and executed on the management device.

[0048] In the seventh aspect, an embodiment of the present application provides a device cluster, which includes a management device, a switching device and at least one computing device, each device including a processor and a memory, and the processor of each device is used to execute at least one piece of program code stored in the memory of each device, so that the device cluster can realize the functions of a distributed computing system provided by the first aspect or any possible implementation method of the first aspect.

[0049] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when running on a device cluster, enables the device cluster to implement the functions of a distributed computing system provided by the aforementioned first aspect or any possible implementation of the first aspect.

[0050] In the ninth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store at least one segment of program code. When the at least one segment of program code is executed by a device cluster, the device cluster implements the functions of a distributed computing system provided by the aforementioned first aspect or any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG1 is a schematic diagram of distributed training of an AI model provided by a related technology;

[0052] FIG2 is a schematic diagram of a distributed computing system provided by a related art;

[0053] FIG3 is a schematic diagram of a failure of a distributed computing system provided by a related art;

[0054] FIG4 is a schematic diagram of a distributed computing system provided in an embodiment of the present application;

[0055] FIG5 is a schematic diagram of another distributed computing system provided in an embodiment of the present application;

[0056] FIG6 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0057] FIG7 is a schematic structural diagram of another device provided in an embodiment of the present application;

[0058] FIG8 is a flowchart of a fault handling method for a distributed computing system provided in an embodiment of the present application;

[0059] FIG9 is a schematic diagram of a fault replacement provided in an embodiment of the present application;

[0060] FIG10 is a schematic diagram of another fault replacement provided in an embodiment of the present application;

[0061] FIG11 is a schematic diagram of another fault replacement provided in an embodiment of the present application;

[0062] FIG12 is a schematic structural diagram of a fault handling device for a distributed computing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that the information involved in this application (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the fault information generated by the computing node group involved in this application, the configuration information of the switch, etc. are all obtained with full authorization.

[0064] For ease of understanding, the key terms and key concepts involved in this application are explained below.

[0065] High performance computing (HPC) is a high-performance computing architecture in which hundreds or thousands of computing nodes (such as servers) are interconnected through a network to form at least one HPC cluster. The computing nodes in the same HPC cluster work in parallel to increase data processing speed and achieve high-performance computing.

[0066] Artificial intelligence (AI) models are a type of mathematical algorithm model that uses machine learning ideas to solve practical problems. AI models include a large number of parameters and calculation formulas (or calculation rules).

[0067] An accelerator, or acceleration device, or accelerator card, is a specialized hardware accelerator or computer system designed to accelerate computing processes in AI and HPC scenarios. Illustratively, accelerators include graphics processing units (GPUs), neural network processing units (XPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, and so on. This application does not limit the type of accelerator.

[0068] Distributed training of AI models refers to the process of dividing the model into multiple accelerators in multiple computing nodes (such as servers) in a distributed computing system through parallel splitting methods such as model parallelism, pipeline parallelism, and data parallelism for distributed parallel training. During the entire training process, different distributed parallel instances communicate with each other through the network between accelerators in the computing node and the network between computing nodes for real-time data exchange. For example, referring to Figure 1, Figure 1 is a schematic diagram of distributed training of AI models provided by a related technology. As shown in Figure 1, model parallelism refers to dividing different parts of the same layer of an AI model into multiple accelerators for parallel training; pipeline parallelism refers to dividing the different layers before and after an AI model into multiple accelerators for parallel training; data parallelism refers to dividing the training data set into multiple accelerators for parallel training. It should be understood that the parallel splitting strategy and the allocation of computing nodes are matched with the network topology of the computing node cluster in the distributed computing system. In some embodiments, model parallelism, pipeline parallelism, and data parallelism have different requirements for communication bandwidth. Among them, model parallelism has the highest requirement for communication bandwidth, pipeline parallelism has lower communication bandwidth requirements due to the small amount of data, and data parallelism has relatively low requirements for communication bandwidth because it can be hidden by computing time. Therefore, in related technologies, when designing computing nodes for performing AI computing tasks, high-bandwidth interconnection is adopted between the accelerators within the computing nodes to meet the large bandwidth requirements of model parallelism, while pipeline parallelism and data parallelism are implemented through the network between computing nodes. Accordingly, during model training, model parallel instances are usually placed inside the same computing node to match the large bandwidth design between accelerators within the computing node, while pipeline parallelism and data parallelism instances are allocated to different computing nodes, and the requirements for communication bandwidth are relatively low.

[0069] A mixture of experts (MoE) is an ensemble learning technique developed in the field of neural networks. MoE breaks down the predictive modeling task into several subtasks, trains an expert model on each subtask, and develops a gating model that learns which expert to trust based on the input to be predicted and combines the predictions.

[0070] A high-bandwidth Internet refers to a computer Internet network that can provide large bandwidth, high-speed, and high-quality network transmission services. It has the characteristics of large capacity, high speed, and low latency, and can provide support for large-scale data transmission, high-definition video transmission, collaborative work, cloud computing and other applications.

[0071] An optical switch is a data exchange device that directly exchanges optical signals without the need for electrical-optical-photoelectrical conversion. It includes multiple fiber optic ports and can directly switch the optical signal from the input port to any output port.

[0072] The following is an introduction to the application scenarios and implementation environment involved in this application.

[0073] The technical solution provided by this application can be applied to scenarios such as AI and HPC. It provides a new networking architecture for distributed computing systems used to perform distributed parallel computing tasks, which can reduce the troubleshooting costs caused by computing node failures or accelerator failures within computing nodes. The following first introduces the application scenarios of this application with reference to the distributed computing systems provided by related technologies.

[0074] In related technologies, taking AI scenarios as an example, as the scale of the model grows, the number of accelerators in a single computing node can no longer meet the requirements of model parallel slicing, and model parallelism often needs to span multiple computing nodes. At the same time, with the gradual evolution of large models, MoE sparse models and new expert parallel methods have emerged. Expert parallelism introduces all-to-all communication between each training instance, and also brings the requirement for higher bandwidth interconnection across multiple computing nodes. Based on this, in order to improve the training performance of AI models, related technologies group the computing nodes in the system and provide high-bandwidth interconnection between each computing node in each computing node group. In this way, the high-bandwidth interconnection within the node can be expanded to between nodes. That is, the computing nodes within a computing node group have the same bandwidth interconnection as the computing node (or interconnection with similar bandwidth), which improves the cross-node communication performance and thus improves the training performance of the AI ​​model. For example, for collective communication such as model parallelism, which has high bandwidth requirements, the above method can effectively improve the collective communication performance between nodes. In some embodiments, the computing node group may also be referred to as an affinity group, a superpod (a supercomputer cluster), a supernode, etc., which are collectively referred to as a computing node group in this application and will not be repeated hereafter.

[0075] For example, with reference to FIG2, FIG2 is a schematic diagram of a distributed computing system provided by a related art. As shown in FIG2, in the related art, the distributed computing system includes a plurality of computing node groups, and the computing node groups exchange data through an inter-node network. The inter-node network is, for example, Ethernet, Fibre Channel, remote direct memory access (RDMA), RDMA over converged Ethernet (RoCE) network, InfiniBand (IB) network, etc., which are only illustrated by way of example and are not limited thereto. High-bandwidth interconnection is adopted between each computing node group and between each accelerator in each computing node, for example, NVIDIA link (NvLink), peripheral component interconnect express (PCIe), compute express link (CXL), universal chiplet interconnect express (UCIe), Huawei cache coherent system (HCCS), cache coherent interconnect for accelerators (CCIX), etc., without limitation thereto.

[0076] In a distributed computing system such as that shown in Figure 2 above, since AI model distributed training requires that the model parallel partitioning strictly matches the computing node cluster network topology, the topological structures of different instances at all levels of the cluster network are strictly isomorphic. For example, the number of available accelerators in different computing nodes is strictly consistent, and the number of available computing nodes in different computing node groups is strictly consistent. If an accelerator fails or a computing node fails, the number of accelerators in the computing node and computing node group will be inconsistent with other computing nodes and other computing node groups, resulting in the failed computing node and computing node group no longer meeting the model parallel partitioning requirements, and thus cannot be scheduled for use in the entire computing task, as shown in Figure 3. Figure 3 is a failure diagram of a distributed computing system provided by a related art. However, since the scale of the computing node group design is generally large (for example, a computing node group includes dozens of computing nodes and hundreds of accelerators), the inability of the entire computing node group to be used in the computing task will result in serious waste or resource loss. In addition, if training resources are insufficient, model parallel training requires re-planning and re-partitioning of parallel training, which has a significant impact on the overall model training time.

[0077] Based on this, the present application provides a new networking architecture, which, on the one hand, can prevent the performance of the entire group from being affected by computing node failures or accelerator failures within the computing node group, and on the other hand, can significantly reduce the cost of fault handling.

[0078] The distributed computing system provided by this application is introduced below with reference to FIG4 .

[0079] FIG4 is a schematic diagram of a distributed computing system provided in an embodiment of the present application. As shown in FIG4 , the distributed computing system 400 provided in the present application includes a management node 410, a switching device 420, a computing node group 430, and a backup node 440. The switching device 420 is physically connected to the computing node group 430 and the backup node 440, respectively. In some embodiments, a node is also referred to as a device, which is not limited here.

[0080] In an embodiment of the present application, the distributed computing system 400 is a distributed computing system with cross-node communication capabilities. In the distributed computing system 400, the management node 410, the switching device 420, the computing node group 430 and the backup node 440 are interconnected through a network (such as the inter-node network interconnection method RDMA, RoCE, IB, etc. mentioned in Figure 2 above, which will not be repeated here. It should be noted that the network interconnection method between each node in the distributed computing system 400 can also adopt other methods, which are not limited here), so that the distributed computing system 400 can perform various distributed parallel computing tasks. For example, distributed parallel computing tasks include distributed parallel training tasks in AI scenarios, big data parallel computing tasks in HPC scenarios, and distributed cloud computing tasks, etc., but the present application is not limited to this. Among them, distributed cloud computing tasks refer to computing tasks performed by computing nodes in virtualized forms such as virtual machines or containers in AI, HPC, image computing or other scenarios.

[0081] The management node 410 is used to manage the distributed computing system 400, for example, to perform cluster management on the distributed computing system 400, control data exchange, detect faulty computing nodes and fault accelerators, configure the data exchange path between each switching port on the switching device 420, and so on, but the present application is not limited thereto. In some embodiments, the management node 410 is a server, a desktop computer, etc., but the present application is not limited thereto. Taking the management node 410 as an example of a server, the management node 410 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The management node 410 can also be a node in a virtualized form such as a virtual machine or a container. In this case, the virtual machine or container can be deployed in at least one computing device (e.g., a server). In addition, the number of management nodes 410 can be one or more, and the present application does not limit this.

[0082] Switching device 420 is a device capable of data exchange, capable of connecting computing node group 430 with backup node 440. Specifically, it establishes a data exchange path between backup node 440 and computing node group 430, enabling data exchange between computing node group 430 and backup node 440 via this data exchange path. For example, switching device 420 may be an optical switch or other device capable of dynamic direct connection at the physical layer, although this application is not limited thereto. Furthermore, the number of switching devices 420 may be one or more, and this application is not limiting in this regard.

[0083] The computing node group 430 is used to execute the computing task of the distributed computing system 400, which is a distributed parallel computing task, such as a distributed parallel training task of an AI model, an HPC distributed parallel computing task, etc., but the present application is not limited thereto. Schematically, if the computing task is a distributed parallel training task of an AI model, each computing node in the computing node group 430 is used to execute a subtask of the distributed parallel training task. In addition, the number of computing node groups 430 can be one or more, and the present application does not limit this. Among them, for any computing node group 430, the computing node group 430 includes multiple computing nodes, and each computing node includes multiple accelerators. In an embodiment of the present application, the physical connection between the computing nodes in the computing node group 430 and the physical connection between the accelerators in each computing node are, for example, in the form of a fixed backplane high-speed bus (such as NvLink, HCCS, etc. mentioned in Figure 2 above, which will not be repeated here), or through a physical connection such as a switch board, switch or switch chip. It should be noted that other methods can also be used in the distributed computing system 400 to achieve high-bandwidth interconnection, which is not limited here. In this way, each computing node in the computing node group 430 has an interconnection with the same bandwidth as the computing nodes within the computing node group (it should be understood that the bandwidth here is not strictly the same, there may be certain differences, and it can be configured according to actual needs), thereby effectively improving the communication performance between the computing nodes in the computing node group 430, that is, improving the execution efficiency of computing tasks.

[0084] In some embodiments, if there are multiple computing node groups 430, the number of computing nodes in each computing node group 430 is the same, the number of accelerators in each computing node is the same, and the computing node cluster composed of multiple computing node groups 430 can adopt a hierarchical design with different bandwidths to improve the execution efficiency of computing tasks. For example, for any computing node group 430, the computing nodes in the computing node group 430 exchange data based on the first bandwidth to perform computing tasks; the computing node groups 430 exchange data based on the second bandwidth to perform computing tasks, and the first bandwidth is greater than or equal to the second bandwidth; for any computing node, the accelerators in the computing node exchange data based on the third bandwidth to perform computing tasks, and the first bandwidth is less than or equal to the third bandwidth. Of course, the bandwidths for data exchange between computing nodes in different computing node groups can be the same or different. If different, the bandwidths for data exchange between computing nodes in different computing node groups are all greater than or equal to the second bandwidth, that is, the bandwidth for data exchange between nodes in the group is greater than or equal to the bandwidth for component data exchange. In this way, the execution efficiency of computing tasks can be improved.

[0085] The backup node 440 is used to replace the faulty device in the computing node group 430 to perform computing tasks. The faulty device is a faulty computing node or a faulty accelerator in the computing node group 430. In addition, the number of backup nodes 440 can be one or more, and this application does not limit this. For any backup node 440, the backup node 440 includes multiple backup accelerators, and high-bandwidth interconnection is used between the backup accelerators in the backup node 440, which is the same as the computing node group 430 and will not be repeated here. In some embodiments, a cluster composed of multiple backup nodes 440 is called a backup node group, or a backup pool, and this application is not limited to this. In addition, if the number of backup nodes 440 is multiple, high-bandwidth interconnection can be used between the backup nodes 440, or it may not be used, and this application is not limited to this. In some embodiments, the bandwidth for data exchange between the backup accelerators in the backup node 440 is the same as the bandwidth for data exchange between the accelerators in the computing node group 430, and this application does not limit this.

[0086] In addition, the computing nodes and backup nodes involved in the above-mentioned distributed computing system 400 can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers, or they can be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms. These nodes can also be nodes in virtualized forms such as virtual machines or containers. In this case, the virtual machines or containers can be deployed in at least one computing device (e.g., a server). The accelerators and backup accelerators in the above-mentioned computing nodes and backup nodes are, for example, GPUs, XPUs, IPUs, TPUs, DSA chips, etc., and this application is not limited thereto.

[0087] Illustratively, in the distributed computing system 400 described above, management node 410 can monitor the operating status of computing node group 430. If a fault is detected in computing node group 430, the management node 410 identifies the faulty device, which is either a faulty computing node or a faulty accelerator within computing node group 430. Switching device 420 is configured to establish a data exchange path between backup node 440 and computing node group 430. This data exchange path is used to exchange data between backup node 440 and computing node group 430, allowing backup node 440 to perform computing tasks in place of the faulty device. This significantly reduces fault handling costs while maintaining the original system performance and network topology. For example, if the failed devices in computing node group 430 include two failed computing nodes, data exchange paths are established between the two backup nodes 440 and computing node group 430 via switch device 420. For another example, if the failed devices in computing node group 430 include two failed accelerators, data exchange paths are established between the two backup accelerators in backup node 440 and computing node group 430 via switch device 420, and so on. This process can also be understood as the process by which management node 410 dynamically allocates backup nodes when a failed device is detected. The detailed allocation method will be described in subsequent method embodiments and will not be repeated here.

[0088] In some embodiments, the above-mentioned distributed computing system 400 can also be connected to a wireless network or other wired network, for example, the wireless network or other wired network uses standard communication technology and / or protocol. The network is usually a transmission control protocol / internet protocol (TCP / IP) network and an RDMA network in a data center network. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology. For example, there are multi-plane network interconnections (shared or independent) between the various nodes involved in the above-mentioned distributed computing system 400: a management plane network (for cluster management and transmission of control data), a data plane network (for downloading data such as AI training data sets, HPC computing data sets, etc.), a parameter plane network (for parameter synchronization across nodes or across computing node groups), etc., but the present application is not limited to this.

[0089] 5 , the distributed computing system 400 will be described below by taking the switching device 420 as an optical switch as an example.

[0090] FIG5 is a schematic diagram of another distributed computing system provided by an embodiment of the present application. As shown in FIG5 , the distributed computing system 500 provided by the present application includes a management node 510, an optical switch 520, a computing node group 530, and a backup node 540. The optical switch 520 is physically connected to the computing node group 530 and the backup node 540, respectively.

[0091] The management node 510 is used to manage the distributed computing system 500 . Please refer to the contents shown in FIG. 4 above and will not be described in detail here.

[0092] The optical switch 520 includes multiple optical fiber ports, which are used to access the computing node group 530 and the backup node 540.

[0093] The computing nodes in the computing node group 530 are connected through a first switching component to achieve data exchange between the computing nodes, wherein the first switching component is a switch, a switching chip or a switching board, and the present application is not limited thereto. In some embodiments, at least one port of the first switching component is used to access the optical switch 520. That is, a small number of ports corresponding to the computing node group 530 are connected to the optical switch 520. It should be understood that since the failure occurring in the computing node group 530 is usually a single point failure, for example, a single accelerator failure or a single computing node failure, connecting a small number of ports corresponding to the computing node group 530 to the optical switch 520 can reduce the requirements for switching ports on the basis of meeting fault replacement, and significantly reduce the cost of infrastructure such as optical modules, optical fibers, and optical switches. Of course, a preset number of ports of the first switching component can also be connected to the optical switch 520. The preset number can be configured according to actual needs or historical experience, and the present application is not limited thereto. In addition, for any computing node in the computing node group 530, the accelerators in the computing node are connected through a third switching component to realize data exchange between the accelerators. The third switching component is a switching chip or a switching board, but the present application is not limited thereto.

[0094] The backup accelerators in the backup node 540 are connected via a second switching component to enable data exchange between the backup accelerators. The second switching component is a switching chip or a switching board, etc., but the present application is not limited thereto. In some embodiments, at least one port of the second switching component is used to access the optical switch 520, that is, a small number of ports corresponding to the backup node 540 are connected to the optical switch 520, thereby reducing the requirements for switching ports on the basis of meeting fault replacement requirements, and significantly reducing the cost of infrastructure such as optical modules, optical fibers, and optical switches. In addition, if there are multiple backup nodes 540, each backup node 540 is connected to the optical switch 520 to meet fault replacement requirements.

[0095] In the above-mentioned distributed computing system 500, an optical switch is used as a switching device. Since the optical switch is a data switching device that directly exchanges optical signals without the need for electro-optical-to-photoelectric conversion, it can ensure that the network topology and communication performance of the computing node cluster in the system remain unchanged after a fault is replaced, thereby ensuring that the execution efficiency of the computing tasks before and after the fault occurs remains unchanged.

[0096] The following introduces the hardware structure of the nodes involved in the distributed computing system 400 and the distributed computing system 500.

[0097] Figure 6 is a schematic diagram of the structure of a device provided in an embodiment of the present application. As shown in Figure 6, the device 600 includes a memory 601, a processor 602, a communication interface 603, and a bus 604. It can be configured as a management node involved in the above-mentioned distributed computing system 400 and distributed computing system 500. Schematically, the device 600 is also referred to as a management device. Among them, the memory 601, the processor 602, and the communication interface 603 are connected to each other through the bus 604.

[0098] The memory 601 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Schematically, the memory 601 is used to store at least one section of program code. When the program code stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to perform the steps involved in the management node in the following method embodiment.

[0099] The processor 602 may be a network processor (NP), a central processing unit (CPU), an application-specific integrated circuit (ASIC), or an integrated circuit for controlling the execution of the program of the present application. The processor 602 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The number of processors 602 may be one or more. The communication interface 603 uses a transceiver module such as a transceiver to implement communication between the computing device 600 and other devices or a communication network. For example, data may be obtained through the communication interface 603.

[0100] The memory 601 and the processor 602 may be separately provided or integrated together.

[0101] Bus 604 may include a path for transmitting information between various components of computing device 600 (eg, memory 601 , processor 602 , communication interface 603 ).

[0102] Figure 7 is a schematic diagram of the structure of another device provided in an embodiment of the present application. As shown in Figure 7, the device 700 includes a memory 701, a processor 702, a communication interface 703, an accelerator 704, and a bus 705. It can be configured as a computing node or backup node involved in the above-mentioned distributed computing system 400 and distributed computing system 500. Schematically, the device 700 is also referred to as a computing device. Among them, the memory 701, the processor 702, the communication interface 703, and the accelerator 704 are connected to each other through the bus 705.

[0103] The memory 701 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. Schematically, the memory 701 is used to store at least one section of program code. When the program code stored in the memory 701 is executed by the processor 702, the processor 702 and the communication interface 703 are used to perform the functions involved in the computing node and the backup node in the following method embodiments.

[0104] The processor 702 may be an NP, a CPU, an ASIC, or an integrated circuit for controlling the execution of the program of the present application. The processor 702 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. The number of processors 702 may be one or more. The communication interface 703 uses a transceiver module such as a transceiver to implement communication between the computing device 700 and other devices or communication networks. For example, data can be obtained through the communication interface 703.

[0105] The accelerator 704 can be a GPU, XPU, IPU, TPU, DSA chip, etc. Illustratively, the accelerator 704 is used to provide computing power for computing tasks running on the computing device 700, thereby accelerating the computing process. Furthermore, the number of accelerators 704 can be one or more, and this application does not limit this.

[0106] The memory 701 , the processor 702 , and the accelerator 704 may be provided separately or integrated together.

[0107] The bus 705 may include a path for transmitting information between various components of the computing device 700 (eg, the memory 701 , the processor 702 , the communication interface 703 , and the accelerator 704 ).

[0108] Based on the above introduction to the distributed computing system, the following describes the fault handling method for a distributed computing system provided by this application with reference to Figure 8. Figure 8 is a flow chart of a fault handling method for a distributed computing system provided by an embodiment of this application. As shown in Figure 8, this method is applied to the distributed computing system shown in Figure 4 or Figure 5, and includes the following steps 801 to 803.

[0109] 801. The management node detects the operating status of each computing node group in the distributed computing system.

[0110] Among them, in the process of each computing node group in the distributed computing system performing computing tasks, the management node can determine whether each computing node group has failed based on the operating status of each computing node group. This application does not limit the way in which the management node detects the operating status of the computing node group. For example, an agent module can be configured in the computing node group (for example, an agent module is configured in each computing node), and the agent module is used to report the relevant fault situation to the management node when a fault is detected in the computing node group. For another example, the management node counts the cache errors reported by each computing node group. If the number of cache errors reported by a computing node group reaches a threshold, it is determined that the computing node group has failed. For another example, the management node performs a polling mechanism detection on each computing node in the computing node group. When an abnormality is detected, it is determined that the corresponding computing node has failed, and so on.

[0111] 802. The management node detects that a computing node group has failed, and determines a failed device.

[0112] Among them, the computing node group refers to any computing node group in a distributed computing system. The faulty device is a faulty computing node or a faulty accelerator in the computing node group. It should be understood that the present application does not limit the type of fault that occurs in the computing node group. For example, the fault is a network communication failure, a hardware failure, and so on. In this step, if the management node detects that a computing node group has a fault, it determines the faulty device that has failed based on the fault information. For example, the fault information includes the number of faulty devices, identification ID, etc., but the present application is not limited to this. Schematically, taking the distributed parallel training task of an AI model as an example, the faulty device is used to execute subtasks of the distributed parallel training task.

[0113] 803. The management node configures the switching device to establish a data exchange path between the backup node and the computing node group through the switching device. The data exchange path is used for data exchange between the backup node and the computing node group, so that the backup node replaces the failed device to perform computing tasks.

[0114] Among them, the management node configures the switching device based on the faulty device to establish a data exchange path between the backup node and the computing node group through the switching device. That is, the backup node is connected to the computing node group through the switching device, so that data can be exchanged between the backup node and the computing node group through the switching device. In this way, the backup node can replace the faulty device to perform computing tasks.

[0115] Taking the switching device as an optical switch as an example, the management node configures the optical path between the first optical fiber port and the second optical fiber port on the optical switch to establish an optical path between the backup node and the computing node group through the optical switch. This allows the backup node to exchange optical signals with the computing node group via this optical path to replace the failed device in performing computing tasks. The first optical fiber port is the optical fiber port connected to the backup node, and the second optical fiber port is the optical fiber port connected to the computing node group. The optical signal is used to carry data corresponding to the computing task. It should be understood that the optical switch includes an optical switching structure that routes optical signals from input ports to output ports. By configuring the optical switch, the management node can exchange optical signals from the input port of the optical switch to any output port, thereby achieving connectivity between the backup node and the computing node group, that is, establishing a data exchange path between the backup node and the computing node group. It should be noted that if the switching device is another device capable of dynamic direct connection at the physical layer, the management node can similarly configure the device to exchange signals from the input port on the device to any output port, thereby establishing a data exchange path between the backup node and the computing node group.

[0116] In some embodiments, the management node configures the switching device based on the number of failed devices to establish a data exchange path between the backup node and the computing node group through the switching device. This dynamic allocation of backup nodes improves the flexibility of failover while enabling dynamic failover. This configuration is described below, including the following scenarios:

[0117] Case 1: If the faulty device includes a first number of faulty computing nodes, the switching device is configured to establish a first data exchange path between the first number of backup nodes and at least one first node through the switching device. The first data exchange path is used to exchange data between the first number of backup nodes and at least one first node, so that the first number of backup nodes replace the first number of faulty computing nodes to perform computing tasks.

[0118] The first nodes are nodes in the computing node group excluding the failed computing node, and the first number is a positive integer. The method illustrated in Case 1 is to schedule an equal number of backup nodes from the backup pool as replacements for the failed computing node group based on the number of failed computing nodes in the group, so that the network topology of the computing node cluster in the distributed computing system remains the same before and after the failure.

[0119] For example, referring to Figure 9, Figure 9 is a schematic diagram of a fault replacement provided by an embodiment of the present application. As shown in Figure 9, the distributed computing system includes computing node group 1 (including computing node 1a and computing node 1b), computing node group 2 (including computing node 2a and computing node 2b), backup nodes A and B. The management node detects that both computing node group 1 and computing node group 2 have failed, and determines that the failed devices are computing node 1a in computing node group 1 and computing node 2a in computing node group 2. The management node configures the switching device to establish a data exchange path between computing node 1a and backup node A, and to establish a data exchange path between computing node 2a and backup node B. In this way, the network topology structure of the computing node cluster in the distributed computing system is equivalent before and after the failure.

[0120] Case 2: If the faulty device includes a second number of faulty accelerators, the switching device is configured to establish a second data exchange path between the second number of backup accelerators in at least one backup node and the at least one first accelerator through the switching device. The second data exchange path is used to exchange data between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform computing tasks.

[0121] The first accelerator refers to the accelerators in the computing node group excluding the failed accelerator, and the second number is a positive integer. The method illustrated in Case 2 is to dispatch an equal number of backup accelerators from the backup pool as replacements for the failed computing node group based on the number of failed accelerators in the group, so that the network topology of the computing node cluster in the distributed computing system remains consistent before and after the failure.

[0122] For example, referring to Figure 10, Figure 10 is a schematic diagram of another fault replacement provided by an embodiment of the present application. As shown in Figure 10, the distributed computing system includes computing node group 1 (including computing node 1a and computing node 1b), computing node group 2 (including computing node 2a and computing node 2b), backup nodes A and B, and the management node detects that both computing node group 1 and computing node group 2 have failed, and determines that the failed faulty devices are accelerator 1a-1 of computing node 1a in computing node group 1 and accelerator 2a-1 of computing node 2a in computing node group 2. The management node configures the switching device to establish a data exchange path between accelerator 1a-1 and backup accelerator A-1 in backup node A, and to establish a data exchange path between accelerator 2a-1 and backup accelerator A-2 in backup node A. In this way, the network topology structure of the computing node cluster in the distributed computing system is equivalent before and after the failure.

[0123] Case 3: If the faulty device includes a third number of faulty accelerators, determine a fourth number of nodes to which the third number of faulty accelerators belong, configure the switching device to establish, through the switching device, a third data exchange path between the fourth number of backup nodes and the at least one first node. The third data exchange path is used for data exchange between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of faulty accelerators belong to perform computing tasks.

[0124] The first node refers to the nodes in the computing node group excluding the failed computing node, and the third and fourth quantities are both positive integers. The method illustrated in Case 3 is to determine, for the failed computing node group, the number of nodes to which the failed accelerators belong based on the number of failed accelerators in the group, and then dispatch an equal number of backup nodes from the backup pool as replacements. This means that the nodes to which the failed accelerators belong are completely replaced as failed computing nodes, ensuring that the network topology of the computing node cluster in the distributed computing system remains the same before and after the failure.

[0125] For example, referring to Figure 11, Figure 11 is a schematic diagram of another fault replacement provided by an embodiment of the present application. As shown in Figure 11, the distributed computing system includes computing node group 1 (including computing node 1a and computing node 1b), computing node group 2 (including computing node 2a and computing node 2b), backup nodes A and B. The management node detects that both computing node group 1 and computing node group 2 have failed, and determines that the failed faulty devices are accelerator 1a-1 of computing node 1a in computing node group 1 and accelerator 2a-1 of computing node 2a in computing node group 2. The management node configures the switching device to establish a data exchange path between computing node a1 and backup node A, and to establish a data exchange path between computing node 2a and backup node B. In this way, the network topology structure of the computing node cluster in the distributed computing system is equivalent before and after the failure.

[0126] It should be understood that the above-mentioned situations 1 to 3 and the fault replacement methods shown in Figures 9 to 11 are only examples and do not constitute a limitation of this application. In actual applications, backup nodes and backup accelerators within backup nodes can be dynamically allocated according to needs.

[0127] In some embodiments, the management node configures the switching device based on the communication bandwidth within the computing node group before the failure occurs, so as to establish a data exchange path between the backup node and the computing node group through the switching device, and enable the backup node and the computing node group to use the same communication bandwidth as before the failure to exchange data. For example, if before the failure, the computing nodes in the computing node group exchange data based on the first bandwidth to perform computing tasks, then after the backup node is connected to the computing node group, the backup node and the computing node group exchange data based on the first bandwidth to perform computing tasks. Similarly, if before the failure, the accelerators in the computing node exchange data based on the third bandwidth to perform computing tasks, and the first bandwidth is less than or equal to the third bandwidth, then after the backup node is connected to the computing node group, the backup accelerators in the backup node exchange data based on the third bandwidth to perform computing tasks. In this way, after a failure occurs, the fault handling method provided by the present application can be used to ensure that the network topology and communication performance of the computing node cluster in the system remain unchanged after the fault is replaced, thereby ensuring that the execution efficiency of the computing tasks before and after the failure remains unchanged.

[0128] In summary, in the fault handling method for a distributed computing system provided by the present application, since both the computing node group and the backup node are connected to a switching device in the distributed computing system provided by the present application, when the management node detects a fault in the computing node group, it can configure the switching device to establish a data exchange path between the backup node and the computing node group. Thus, the backup node can exchange data with the computing node group through this data exchange path to replace the faulty device in the computing node group to perform computing tasks. This greatly reduces the fault handling cost while maintaining the original system performance and network topology. For example, if the distributed computing system includes 10 computing node groups, each computing node group includes 10 computing nodes, and each computing node includes 8 accelerators, that is, the entire system includes 100 computing nodes (including 800 accelerators), by additionally configuring a switching device and 2 backup nodes in the distributed computing system, each backup node includes 8 backup accelerators (i.e., one switching device and 16 backup accelerators), a 1:N backup of the faulty device can be achieved, effectively reducing the networking cost. Moreover, in this manner, the redundancy of the backup device is small, which is less likely to cause resource waste. Using related technologies, a distributed computing system also includes 10 computing node groups, each computing node group includes 10 computing nodes, and each computing node includes 8 accelerators. If redundant backup is performed for the nodes in the group, at least 10 additional backup nodes (i.e., 80 accelerators) need to be configured; if redundant backup is performed for the accelerators within the nodes, at least 100 additional accelerators need to be configured. It can be seen that the use of related technologies is costly, has a large amount of redundancy, and is prone to waste of resources.

[0129] Figure 12 is a schematic diagram of the structure of a fault handling device for a distributed computing system provided by an embodiment of the present application. The device can realize some or all of the functions of the aforementioned management node through software, hardware, or a combination of both. As shown in Figure 12, the device is applied to the management node of a distributed computing system, which also includes a switching device, a computing node group, and a backup node. The switching device is physically connected to the computing node group and the backup node respectively. Each computing node in the computing node group includes multiple accelerators, and the backup node includes multiple backup accelerators. The device includes a determination module 1201 and a configuration module 1202.

[0130] A determination module 1201 is configured to detect a failure in a computing node group and determine a faulty device, where the faulty device is a faulty computing node or a faulty accelerator in the computing node group.

[0131] Configuration module 1202 is used to configure the switching device to establish a data exchange path between the backup node and the computing node group through the switching device, wherein the data exchange path is used to exchange data between the backup node and the computing node group so that the backup node can replace the faulty device to perform computing tasks.

[0132] In some embodiments, the switching device is an optical switch, which includes multiple optical fiber ports for accessing backup nodes and computing node groups. The configuration module 1202 is configured to:

[0133] An optical path between a first optical fiber port and a second optical fiber port on an optical switch is configured to establish an optical path between a backup node and a computing node group through the optical switch, wherein the optical path is used to exchange optical signals between the backup node and the computing node group so that the backup node replaces the failed device to perform computing tasks. The first optical fiber port is an optical fiber port for accessing the backup node, and the second optical fiber port is an optical fiber port for accessing the computing node group. The optical signal is used to carry data corresponding to the computing task.

[0134] In some embodiments, the configuration module 1202 is configured to:

[0135] If the failed device includes a first number of failed computing nodes, configuring the switching device to establish, through the switching device, a first data exchange path between the first number of backup nodes and the at least one first node, the first data exchange path being used for exchanging data between the first number of backup nodes and the at least one first node, so that the first number of backup nodes replace the first number of failed computing nodes to perform computing tasks, the first nodes being nodes in the computing node group other than the failed computing node;

[0136] If the faulty device includes a second number of faulty accelerators, configuring the switching device to establish, through the switching device, a second data exchange path between the second number of backup accelerators in the at least one backup node and the at least one first accelerator, the second data exchange path being used for exchanging data between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform computing tasks, where the first accelerator refers to an accelerator in the computing node group other than the faulty accelerator;

[0137] If the faulty device includes a third number of faulty accelerators, determine a fourth number of nodes to which the third number of faulty accelerators belong, configure the switching device to establish a third data exchange path between the fourth number of backup nodes and the at least one first node through the switching device, and use the third data exchange path to exchange data between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of faulty accelerators belong to perform computing tasks.

[0138] Through the above-mentioned device, since in the distributed computing system provided in the present application, both the computing node group and the backup node are connected to the switching device, when the management node of the distributed computing system detects a fault in the computing node group, it can configure the switching device to establish a data exchange path between the backup node and the computing node group, so that the backup node can exchange data with the computing node group through this data exchange path to replace the faulty device in the computing node group to perform computing tasks, thereby greatly reducing the fault handling cost while maintaining the original performance of the system and the network topology.

[0139] It should be noted that the fault handling device for a distributed computing system provided in the above embodiment only uses the division of the above functional modules as an example to illustrate fault handling. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the fault handling device for a distributed computing system provided in the above embodiment and the fault handling method embodiment for a distributed computing system are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0140] In this application, the words such as term "first" and "second" are used to distinguish the same items or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second" and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various examples described, the first node can be referred to as the second node, and similarly, the second node can be referred to as the first node. The first node and the second node can both be nodes, and in some cases, can be separate and different nodes.

[0141] In this application, the term "at least one" means one or more, and the term "plurality" means two or more. For example, a plurality of nodes means two or more nodes.

[0142] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0143] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of program structure information. The program structure information includes one or more program instructions. When the program instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of the present application are generated.

[0144] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0145] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A distributed computing system, characterized in that: The system includes a management node, a switching device, a computing node group and a backup node, wherein the switching device is physically connected to the computing node group and the backup node respectively, each computing node in the computing node group includes a plurality of accelerators, and the backup node includes a plurality of backup accelerators; The management node is used to: Detecting that a fault occurs in the computing node group, and determining a faulty device where the fault occurs, wherein the faulty device is a faulty computing node or a faulty accelerator in the computing node group; The switching device is configured to establish a data exchange path between the backup node and the computing node group through the switching device, wherein the data exchange path is used for data exchange between the backup node and the computing node group so that the backup node replaces the failed device to perform computing tasks.

2. The system according to claim 1, characterized in that The switching device is an optical switch, and the optical switch includes a plurality of optical fiber ports, and the plurality of optical fiber ports are used to access the backup node and the computing node group.

3. The system according to claim 2, characterized in that The management node is used to: configuring an optical path between a first optical fiber port and a second optical fiber port on the optical switch to establish an optical path between the backup node and the computing node group through the optical switch; Among them, the optical path is used for exchanging optical signals between the backup node and the computing node group so that the backup node replaces the faulty device to perform the computing task, the first optical fiber port is the optical fiber port connected to the backup node, and the second optical fiber port is the optical fiber port connected to the computing node group, and the optical signal is used to carry data corresponding to the computing task.

4. The system according to claim 2 or 3, characterized in that: The computing nodes in the computing node group are connected via a first switching component, at least one port of the first switching component is used to access the optical switch, and the first switching component is a switch or a switching chip.

5. The system according to any one of claims 2 to 4, characterized in that The backup accelerators in the backup node are connected via a second switching component, at least one port of the second switching component is used to access the optical switch, and the second switching component is a switch or a switching chip.

6. The system according to any one of claims 1 to 5, characterized in that The management node is used for at least one of the following: If the faulty device includes a first number of faulty computing nodes, the switching device is configured to establish a first data exchange path between the first number of backup nodes and at least one first node through the switching device, wherein the first data exchange path is used for data exchange between the first number of backup nodes and the at least one first node, so that the first number of backup nodes replace the first number of faulty computing nodes to perform the computing task, and the first node refers to a node in the computing node group other than the faulty computing node; If the faulty device includes a second number of faulty accelerators, the switching device is configured to establish a second data exchange path between the second number of backup accelerators in at least one backup node and at least one first accelerator through the switching device, wherein the second data exchange path is used for data exchange between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform the computing task, and the first accelerator refers to an accelerator in the computing node group other than the faulty accelerator; If the fault device includes a third number of fault accelerators, determine a fourth number of nodes to which the third number of fault accelerators belong, and configure the switching device to establish a third data exchange path between the fourth number of backup nodes and the at least one first node through the switching device, wherein the third data exchange path is used for data exchange between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of fault accelerators belong to perform the computing task.

7. The system according to any one of claims 1 to 6, characterized in that The computing nodes in the computing node group exchange data based on the first bandwidth; The backup node is used to exchange data with the computing node group based on the first bandwidth through the data exchange path, so as to replace the failed device to perform the computing task.

8. The system according to claim 7, characterized in that The system includes multiple computing node groups, the number of computing nodes in each computing node group is the same, and data is exchanged between the computing node groups based on a second bandwidth, and the first bandwidth is greater than or equal to the second bandwidth.

9. The system according to claim 8, characterized in that The bandwidths for data exchange between computing nodes in different computing node groups are different, and the bandwidths for data exchange between computing nodes in different computing node groups are greater than or equal to the second bandwidth.

10. The system according to any one of claims 7 to 9, characterized in that The number of the accelerators in each computing node in the computing node group is the same, and the accelerators exchange data based on a third bandwidth, and the first bandwidth is less than or equal to the third bandwidth.

11. The system according to claim 10, characterized in that The backup accelerators in the backup node exchange data based on the third bandwidth.

12. The system according to any one of claims 1 to 11, characterized in that The distributed computing system is used to execute a distributed parallel training task of an AI model, and the faulty device is used to execute a subtask of the distributed parallel training task.

13. A fault handling method for a distributed computing system, characterized in that: The method is applied to a management node of the distributed computing system, wherein the system further comprises a switching device, a computing node group and a backup node, wherein the switching device is physically connected to the computing node group and the backup node respectively, each computing node in the computing node group comprises a plurality of accelerators, and the backup node comprises a plurality of backup accelerators. Detecting that a fault occurs in the computing node group, and determining a faulty device where the fault occurs, wherein the faulty device is a faulty computing node or a faulty accelerator in the computing node group; The switching device is configured to establish a data exchange path between the backup node and the computing node group through the switching device, wherein the data exchange path is used for data exchange between the backup node and the computing node group so that the backup node replaces the failed device to perform computing tasks.

14. The method according to claim 13, characterized in that The switching device is an optical switch, the optical switch includes a plurality of optical fiber ports, the plurality of optical fiber ports are used to access the backup node and the computing node group, and the configuring of the switching device to establish a data exchange path between the backup node and the computing node group through the switching device includes: configuring an optical path between a first optical fiber port and a second optical fiber port on the optical switch to establish an optical path between the backup node and the computing node group through the optical switch; Among them, the optical path is used for exchanging optical signals between the backup node and the computing node group so that the backup node replaces the faulty device to perform the computing task, the first optical fiber port is the optical fiber port connected to the backup node, and the second optical fiber port is the optical fiber port connected to the computing node group, and the optical signal is used to carry data corresponding to the computing task.

15. The method according to claim 13 or 14, characterized in that The configuring the switching device to establish a data exchange path between the backup node and the computing node group through the switching device includes at least one of the following: If the faulty device includes a first number of faulty computing nodes, the switching device is configured to establish a first data exchange path between the first number of backup nodes and at least one first node through the switching device, the first data exchange path being used for data exchange between the first number of backup nodes and the at least one first node, so that the first number of backup nodes replace the first number of faulty computing nodes to perform the computing task, the first node being the computing node group other than the faulty computing node node; If the faulty device includes a second number of faulty accelerators, the switching device is configured to establish a second data exchange path between the second number of backup accelerators in at least one backup node and at least one first accelerator through the switching device, the second data exchange path being used for data exchange between the second number of backup accelerators and the at least one first accelerator, so that the second number of backup accelerators replace the second number of faulty accelerators to perform the computing task, the first accelerator being an accelerator in the computing node group other than the faulty accelerator; If the fault device includes a third number of fault accelerators, determine a fourth number of nodes to which the third number of fault accelerators belong, configure the switching device to establish a third data exchange path between a fourth number of backup nodes and the at least one first node through the switching device, and the third data exchange path is used for data exchange between the fourth number of backup nodes and the at least one first node, so that the fourth number of backup nodes replace the nodes to which the third number of fault accelerators belong to perform the computing task.

16. A management device, characterized in that: The management device includes a processor and a memory, and the processor is used to execute at least one program code stored in the memory, so that the management device executes the fault handling method for a distributed computing system as described in any one of claims 13 to 15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one section of program code. When the at least one section of program code is executed on a management device, the management device executes the fault handling method for a distributed computing system as described in any one of claims 13 to 15.

18. A computer program product, characterized in that When the computer program product is executed on a management device, the management device is enabled to execute the fault handling method for a distributed computing system according to any one of claims 13 to 15.

19. A device cluster, characterized in that: The device cluster includes a management device, a switching device and at least one computing device, each device includes a processor and a memory, and the processor of each device is used to execute at least one program code stored in the memory of each device, so that the device cluster can realize the function of a distributed computing system as described in any one of claims 1 to 12.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one section of program code. When the at least one section of program code is executed by a device cluster, the device cluster implements the function of the distributed computing system according to any one of claims 1 to 12.

21. A computer program product, characterized in that The method comprises: when the computer program product is executed on a device cluster, causing the device cluster to implement the function of the distributed computing system according to any one of claims 1 to 12.

Images