Training fault tolerance method and device applied to distributed training system and chip product

Abstract

The invention provides a training fault tolerance method and device applied to a distributed training system and a chip product, and relates to the technical field of distributed training. The method comprises the steps that under the condition that execution of a distributed training task is interrupted, a fault node is determined from a plurality of computing nodes, and the fault node refers to a computing node which cannot continue to participate in execution of the distributed training task; determining a first network domain based on the network domains to which the plurality of computing nodes belong, the first network domain being the network domain to which the fault node belongs; and under the condition that the standby node exists in the first network domain, selecting the standby node from the first network domain to replace the fault node. According to the embodiment of the invention, the standby node belonging to the same network domain as the fault node is adopted to replace the fault node, so that the execution stability and the execution efficiency of the distributed training task can be maintained.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of distributed training technology, and in particular to a training fault-tolerant method, device, and chip product applied to a distributed training system. Background Art

[0002] Distributed training is a method for training machine learning models, which distributes the training process of machine learning models to multiple computing nodes in a distributed system to achieve parallel training and improve the training efficiency of machine learning models. Distributed training relies on the collaborative work between multiple computing nodes, but it is easy to be interrupted due to various software and hardware problems of computing nodes.

[0003] When the distributed training is interrupted, the related technology identifies and isolates the faulty node, and randomly selects a backup node to replace the faulty node to resume the distributed training. However, this easily causes fluctuations in the communication link of the distributed system, thereby causing the training efficiency of the distributed training to fluctuate and become unstable. Summary of the invention

[0004] The embodiment of the present application provides a training fault tolerance method, device and chip product applied to a distributed training system. The technical solution provided by the embodiment of the present application includes the following contents.

[0005] According to one aspect of an embodiment of the present application, a training fault tolerance method applied to a distributed training system is provided, wherein the distributed training system includes a plurality of computing nodes and at least one spare node, and the plurality of computing nodes are used to perform distributed training tasks; the method includes:

[0006] In the case where the execution of the distributed training task is interrupted, determining a faulty node from the multiple computing nodes, wherein the faulty node refers to a computing node that can no longer participate in the execution of the distributed training task;

[0007] Determine a first network domain based on the network domains to which the multiple computing nodes respectively belong, where the first network domain is the network domain to which the faulty node belongs;

[0008] In the case that the backup node exists in the first network domain, the backup node is selected from the first network domain to replace the failed node.

[0009] According to one aspect of an embodiment of the present application, a training fault-tolerant device applied to a distributed training system is provided, wherein the distributed training system includes a plurality of computing nodes and at least one spare node, wherein the plurality of computing nodes are used to perform distributed training tasks; the device includes:

[0010] A fault node determination module, used to determine a fault node from the multiple computing nodes when the execution of the distributed training task is interrupted, wherein the fault node refers to a computing node that can no longer participate in the execution of the distributed training task;

[0011] A network domain determination module, configured to determine a first network domain based on the network domains to which the plurality of computing nodes respectively belong, wherein the first network domain is the network domain to which the faulty node belongs;

[0012] The standby node selection module is used to select the standby node from the first network domain to replace the failed node if the standby node exists in the first network domain.

[0013] According to one aspect of an embodiment of the present application, a chip product is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned training fault-tolerant method applied to a distributed training system.

[0014] According to one aspect of an embodiment of the present application, a computer device is provided, which includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned training fault tolerance method applied to a distributed training system.

[0015] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned training fault-tolerant method applied to a distributed training system.

[0016] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned training fault tolerance method applied to a distributed training system.

[0017] The technical solution provided by the embodiments of the present application may include the following beneficial effects.

[0018] In the process of using a distributed training system to execute distributed training tasks, if a faulty node occurs, the faulty node is replaced by a backup node located in the same network domain as the faulty node to restore the execution of the distributed training task, thereby avoiding the problem of increased communication links due to the backup node and the faulty node not belonging to the same network domain, thereby avoiding a decrease in the execution efficiency of the distributed training task, which is beneficial to maintaining the execution efficiency of the distributed training task.

[0019] In addition, since the network domain to which the backup node selected in the present application belongs is fixed relative to the faulty node, it is not easy to cause fluctuations in communication efficiency, thereby avoiding fluctuations in the execution efficiency of distributed training tasks, and further improving the stability of the execution efficiency of distributed training tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a schematic diagram of a distributed training system provided in a possible implementation of the present application;

[0022] Figure 2 is a schematic diagram of data parallelism provided in a possible implementation of the present application;

[0023] Figure 3 It is a schematic diagram of pipeline parallelism provided in a possible implementation of the present application;

[0024] Figure 4 is a schematic diagram of tensor parallelism provided in a possible implementation of the present application;

[0025] Figure 5 is a schematic diagram of hybrid parallelism provided in a possible implementation of the present application;

[0026] Figure 6 It is a schematic diagram of a method for replacing a faulty node provided by the related art;

[0027] Figure 7 is a flowchart of a training fault-tolerant method applied to a distributed training system provided in a possible implementation of the present application;

[0028] Figure 8 is a schematic diagram of a method for replacing a faulty node provided in a possible implementation of the present application;

[0029] Fig. 9 It is a schematic diagram of replacing a faulty node and scaling down and up a distributed training system provided in a possible implementation of the present application;

[0030] Fig.10 is a flowchart of a training fault-tolerant method applied to a distributed training system provided in another possible implementation of the present application;

[0031] Fig.11is a block diagram of a training fault-tolerant device applied to a distributed training system provided in a possible implementation of the present application;

[0032] Fig.12 is a block diagram of a training fault-tolerant device applied to a distributed training system provided in another possible implementation of the present application;

[0033] Fig.13 It is a simplified structural block diagram of a computer device provided in a possible implementation of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0035] Please refer to Figure 1 , which shows a schematic diagram of a distributed training system provided in a possible implementation of the present application.

[0036] The distributed training system 100 may refer to a distributed system for implementing distributed training of machine learning models. Among them, the machine learning model is an algorithm that automatically learns and generates prediction results based on given input data (also called features), such as a machine learning model that can be a neural network model built based on a neural network. Distributed training is a method for training a machine learning model, which distributes the training process of the machine learning model to multiple computing nodes in a distributed system to achieve parallel training to improve the training efficiency of the machine learning model. A distributed system is a system composed of a group of computing nodes that communicate through a network and work in coordination to complete a common task. In an embodiment of the present application, the distributed training of a machine learning model may be referred to as a distributed training task.

[0037] In one example, the distributed training system 100 may include a plurality of computing nodes 101 and at least one standby node 102. The computing node 101 may be used to independently process data to provide computing resources. The computing node 101 may be implemented as a server. Exemplarily, the computing node 101 may be implemented as at least one of the following: a server provided with a central processing unit (CPU), a GPU server provided with a graphics processing unit (GPU), a server provided with a general-purpose graphics processor (GPGPU), a server provided with a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a server provided with a tensor processor (TPU), a field programmable gate array (FPGA), a server provided with a tensor core (Tensor Core), and a server provided with an NPU (Neural Processing Unit, Neural Network Processor).

[0038] Multiple computing nodes 101 are used to execute distributed training tasks, such as multiple computing nodes 101 can collaborate to complete the execution of distributed training tasks. Exemplarily, the distributed training task can be divided into multiple subtasks, each of which can be executed by one or more computing nodes 101. Optionally, in the process of executing the distributed training task, multiple computing nodes 101 run in parallel.

[0039] For example, the above distributed training task includes t subtasks, multiple computing nodes 101 are divided into t computing node groups, the t computing node groups are used to execute t subtasks in parallel, each computing node group is used to execute one subtask, each computing node group includes at least one computing node 101, and t is an integer greater than 1. Optionally, the t computing node groups include the same number of computing nodes 101, which is not limited in the embodiment of the present application.

[0040] The standby node 102 may refer to a standby computing node 101, for example, a computing node 101 that has not executed a task may be determined as the standby node 102. The standby node 102 may be used to replace the computing node 101 to continue to execute the subtask corresponding to the computing node 101 when the computing node 101 fails. The capability and implementation form of the standby node 102 are the same as those of the computing node 101, so that the standby node 102 may directly replace the computing node 101.

[0041] Optionally, multiple computing nodes 101 can communicate with each other, and computing nodes 101 and standby nodes 102 can also communicate with each other. For example, communication links are established between computing nodes 101, so that computing nodes 101 can communicate with each other, and communication links are established between each standby node 102 and each computing node 101, so that each standby node 102 and each computing node 101 can communicate with each other, wherein the communication links can be implemented through a network, such as a wired network or a wireless network.

[0042] Exemplarily, the network architecture of the distributed training system 100 is a leaf-spine network architecture, which includes a leaf switch 103 and a spine switch 104. The leaf switch 103 can be implemented as a layer 2 switch for connecting the computing node 101 or the standby node 102. The spine switch 104 can be implemented as a layer 3 switch responsible for routing and forwarding, which is connected to the leaf switch 103. Some computing nodes 101 can form a communication link only through the leaf switch 103, and some computing nodes 101 can form a communication link through the leaf switch 103 and the spine switch 104 at the same time. For example, computing node H0 and computing node H1 can form a communication link only through leaf switch 0, and computing node H0 and computing node H4 can form a communication link through leaf switch 0, spine switch 104 and leaf switch N. The embodiment of the present application does not limit the value of N, which can be set and adjusted according to actual use requirements. Optionally, the embodiment of the present application does not limit the number of layers of switches, such as the distributed training system 100 can also include 3 layers or more than 3 layers of switches.

[0043] Optionally, the computing nodes 101 and the backup nodes 102 connected to the same leaf switch 103 belong to the same network domain. Two computing nodes 101 connected to different leaf switches 103 belong to different network domains, that is, each computing node 101 in the same network domain is connected to the same leaf switch 103. In the implementation of this application, a network domain may refer to a set of computing nodes with a security boundary, and a trust relationship is established between each computing node in the same network domain, so that when accessing other computing nodes within the domain, the permission of the accessed computing node is no longer required. For example, computing nodes H0, H1, H2, H3, and backup nodes B1 and B2 can form network domain 0 through leaf switch 0.

[0044] In one example, the distributed training system 100 further includes a master computing node ( Figure 1 The master computing node 101 is a node that is used to coordinate and schedule computing nodes 101 and backup nodes 102, as well as the subtasks of distributed training tasks, so that multiple computing nodes 101 can collaboratively perform distributed training tasks. The master computing node ensures that all computing nodes 101 work together to optimize training efficiency by centrally managing and allocating tasks. Optionally, computing nodes 101 and backup nodes 102 can be collectively referred to as workers.

[0045] The main computing node can be implemented as at least one of the following: a server with a CPU, a GPU server, a server with a GPGPU, a server with a DSP, an ASIC, a server with a TPU, an FPGA, a server with a Tensor Core, and a server with an NPU. Exemplarily, a computing node 101 can be randomly selected from the distributed training system 100 as the main computing node, and a main computing node can also be separately set outside the distributed training system 100, which is not limited in the embodiments of the present application.

[0046] Optionally, the master computing node may also be responsible for monitoring distributed training tasks, partitioning of sample data of machine learning models, monitoring of computing node 101 status, network configuration management, and fault log analysis. In a feasible example, the master computing node may be implemented as a functional module and deployed on computing node 101.

[0047] Optionally, each computing node 101 may be provided with a plurality of computing devices, and the computing devices include at least one of the following: CPU, GPU, GPGPU, DSP, ASIC, TPU, FPGA, Tensor Core, NPU. For example, in the case where the computing node 101 is implemented as a GPU server, the computing node 101 may be provided with a plurality of GPUs, each of which may be implemented as a GPU card (i.e., a chip), and the GPU card is a computing device. Each computing node 101 may run a plurality of subprocesses to utilize the plurality of computing devices on the computing node 101 to accelerate computing tasks.

[0048] In one example, the distributed training of the machine learning model is an iterative process, and each iterative process may correspond to setting a distributed training task of one stage. The distributed training system 100 can complete the iterative training of the machine learning model by executing the distributed training tasks of each stage in sequence. Distributed training usually requires multiple parallel strategies to improve the efficiency of training. Based on the parallelism of different parallel strategies, the distributed training task can be divided into subtasks with different parallelisms.

[0049] Exemplarily, the parallel strategy includes at least one of the following:

[0050] Data Parallel (DP): The sample data used to train the machine learning model is divided into multiple data subsets, and different data subsets are assigned to different computing node groups. Each computing node group has the same and complete model parameters. Among them, the computing node group includes at least one computing node. For example, refer to Figure 2 , the sample data 201 is divided into two data subsets, and the two data subsets are respectively processed by a computing node group to achieve distributed training.

[0051] Pipeline Parallel (PP): The machine learning model is divided into layers, that is, the data of some model layers of the machine learning model (such as model parameters) are divided into some computing nodes, and the data of other model layers are divided into other computing nodes. Among them, the model layer is the basic data structure of the machine learning model. Different model layers can process the data differently, such as convolution, pooling, normalization, linear transformation, logistic regression, etc. For example, refer to Figure 3 The model parameters of some model layers of the machine learning model are divided into computing node 302, and the model parameters of the remaining model layers are divided into computing node 303. Computing node 302 and computing node 303 complete the distributed training of the machine learning model based on sample data 301.

[0052] Tensor Parallel (TP): divides the machine learning model into layers, that is, divides the data of the same model layer (such as model parameters) into different computing nodes. For example, refer to Figure 4 For each model layer of the machine learning model, the model parameters of the model layer are evenly divided into computing nodes 402 and 403, and computing nodes 402 and 403 are trained based on sample data 401 at the same time. Among them, PP and TP belong to model parallelism (ModelParallel).

[0053] Hybrid parallelism (3D Parallel): DP, PP, and TP are applied simultaneously. For example, refer to Figure 5, taking DP=N, PP=4, TP=8 as an example, the sample data is split into N data subsets according to the parallelism of DP, and each data subset will be used to train a complete machine learning model, that is, the multiple computing nodes 101 in the distributed training system 100 are divided into N computing node groups, and each computing node group is used to process a data subset. For any complete machine learning model, it is split according to the parallelism of PP so that different model layers are assigned to different computing nodes 101. For example, taking computing node group 501 as an example, the model parameters of the four model layers of the machine learning model can be respectively assigned to the four computing nodes 502 in computing node group 501. For each model layer, it can continue to be split according to the parallelism of TP, so that different model parameters are assigned to different computing devices (such as GPU) on the same computing node 101. For example, taking computing node 502 as an example, the model parameters of model layer 1 can be divided into 8 parts, each of which is assigned to a computing device 503. The numerical values ​​in the above embodiments are merely exemplary and explanatory. The embodiments of the present application do not limit the number of computing node groups, the number of computing nodes in a computing node group, and the number of computing devices used on the computing nodes.

[0054] During the distributed training process of the machine learning model, sample data is sharded to different computing node groups for training. After each computing node group completes a cycle of training, they need to exchange training status data (such as model parameters, parameter gradients, etc.) with each other and update the overall model parameters of the machine learning model to complete the entire distributed training.

[0055] Optionally, during distributed training, each computing device (such as a GPU) has a number, which is often referred to as a rank. The number of the computing device on each computing node is called a local rank, and each computing device in the entire distributed training system has a global number, which is referred to as a global rank.

[0056] In one example, when the execution of a distributed training task is interrupted, a training fault tolerance solution can be used to restore the execution of the distributed training task. The training fault tolerance solution mainly involves how to promptly detect problems and take appropriate measures to recover when problems such as failures or network delays occur in the distributed training system, so as to ensure the normal operation of the distributed training system.

[0057] In the related art, when a distributed training interruption is found, the related art identifies and isolates the faulty node, and randomly selects a spare node to replace the faulty node to resume the distributed training. Figure 6, the distributed training system 600 includes computing nodes H0 to H7, and standby nodes B1 to B4, H2 and H3 are a computing node group, and in the case of a failure of computing node H3, the related technology randomly selects a standby node from the standby nodes B1 to B4 to replace computing node H3. In the case of selecting standby node B1, since B1 and computing node H3 are under the same leaf switch (i.e., leaf switch 0), the communication link between H2 and B1 is equivalent to the communication link between H2 and H3, so that the overall communication efficiency of the distributed training system 600 remains unchanged, so the efficiency of distributed training will not be affected. In the case of selecting standby node B3, since standby node B3 is under leaf switch 1, if H2 and B3 need to communicate, it is necessary to cross the spine switch to achieve it (in a three-layer or more layer network architecture, more switches may need to be crossed), then there will be a problem of increasing the overall communication link of the distributed training system 600, which will reduce the overall communication efficiency of the distributed training system 600, and thus affect the efficiency of distributed training. Therefore, by randomly selecting backup nodes to replace faulty nodes, the relevant technology will lead to uncertainty in the network domain to which the selected backup nodes belong, which may easily cause fluctuations in the overall communication efficiency of the distributed training system 600, and further lead to unstable training efficiency of distributed training.

[0058] The technical solution provided in the embodiment of the present application can first sense the network topology of the distributed training system for any faulty node to determine the network domain of the faulty node, and then preferentially use a backup node located in the same network domain as the faulty node to replace the faulty node to restore the execution of the distributed training task, thereby avoiding the problem of increased communication links due to the backup node and the faulty node not belonging to the same network domain, thereby avoiding a decrease in the execution efficiency of the distributed training task, and thus being beneficial to maintaining the execution efficiency of the distributed training task.

[0059] In addition, since the network domain to which the backup node selected in the present application belongs is the same as the network domain to which the faulty node belongs, it is not easy to cause fluctuations in communication efficiency, thereby avoiding fluctuations in the execution efficiency of distributed training tasks, and further improving the stability of the execution efficiency of distributed training tasks.

[0060] The following will use a method embodiment to explain in detail the technical solution provided by the embodiment of the present application. The content not described in the method embodiment can be referred to the above embodiment and will not be repeated here.

[0061] Please refer to Figure 7 , which shows a flowchart of a training fault tolerance method applied to a distributed training system provided in a possible implementation of the present application. The execution subject of each step of the method may be Figure 1The main computing node (training master) in the distributed training system 100 is shown. The method may include at least one of the following steps (701-703).

[0062] Step 701, when the execution of the distributed training task is interrupted, determine the faulty node from multiple computing nodes, where the faulty node refers to a computing node that can no longer participate in the execution of the distributed training task.

[0063] During the process of the distributed training system executing distributed training tasks, the main computing node can be used to continuously monitor the execution process of the distributed training tasks. When the execution of the distributed training tasks is interrupted, the main computing node enters the fault diagnosis process. If the fault is unrecoverable, the main computing node determines the faulty node from multiple computing nodes.

[0064] Optionally, the execution of the distributed training task may be interrupted due to at least one of the following: hardware failure, software failure, network interruption, insufficient memory, and excessive CPU resource usage. Among them, for software failure, network interruption and other failures, the distributed training system can directly repair the failure and resume the execution of the distributed training task. For hardware failures such as damaged computing nodes and computing devices (such as GPU cards), the distributed training system cannot directly repair them.

[0065] Exemplarily, the failure of any computing node (such as a GPU server) in a distributed training system may cause the interruption of a distributed training task; the failure of any computing device (such as a GPU card) in a distributed training system may cause the interruption of a distributed training task. In an embodiment of the present application, a computing node that is unable to continue to participate in the execution of a distributed training task due to a failure is referred to as a failed node, such as a computing node with a damaged GPU card, a computing node with a damaged hard disk, a computing node with a power outage, a computing node with an operating system crash, or a computing node that is down. Optionally, in the event of an interruption in the execution of a distributed training task, the main computing node may determine one or more failed nodes, and the above-mentioned failed node may refer to any failed node determined by the main computing node.

[0066] In one example, a method for determining a faulty node includes at least one of the following:

[0067] 1. The main computing node can determine the specific cause of the failure by analyzing the logs of the distributed training system, such as the time, location, and cause of the failure, to identify the failed node.

[0068] 2. Each computing node will send its own status data to the main computing node, and the main computing node can determine the faulty node by analyzing the status data of each computing node.

[0069] 3. The main computing node can determine the faulty node by executing the detection script. If it is detected that the computing node fails the inspection, the computing node can be determined as a faulty node.

[0070] 4. When a failure occurs, the computing node will actively send abnormal information to the main computing node, and the main computing node can determine the faulty node based on the abnormal information.

[0071] 5. Compute nodes can identify faulty nodes through the heartbeat mechanism.

[0072] 6. The main computing node can monitor the status indicators of the distributed training system (such as CPU utilization, network latency, etc.) in real time to monitor the status of each computing node to determine the faulty node.

[0073] Optionally, the faulty node needs to be isolated so as not to affect the normal operation of other computing nodes. The method of isolating the faulty node includes at least one of the following: functional module isolation, resource isolation, thread-level isolation, service degradation, and fuse mode.

[0074] Step 702: Determine a first network domain based on network domains to which multiple computing nodes respectively belong, where the first network domain is the network domain to which the faulty node belongs.

[0075] A network domain refers to a collection of computing nodes with a security boundary. In an embodiment of the present application, for each computing node that can form a communication link through only one switch, the domain it constructs is a network domain. Exemplarily, the network topology of the distributed training system can be first perceived, and then the network domains to which multiple computing nodes belong are determined based on the network topology of the distributed training system. Among them, the network topology of the distributed training system can be used to indicate the distribution of network domains corresponding to the distributed training system, as well as the composition of the network domains.

[0076] For example, refer to Figure 1 , the distributed training system 100 includes N+1 network domains: network domain 0 to network domain N. Among them, network domain 0 is constructed by computing nodes H0, H1, H2 and H3, and backup nodes B1 and B2. Network domain N is constructed by computing nodes H4, H5, H6 and H7. Each computing node 101 is used to perform distributed training tasks. Optionally, each leaf switch 103 corresponds to a network domain, and each computing node 101 in the same network domain can complete communication through the corresponding leaf switch 103, that is, data transmission can be completed through "2 hops", such as computing node H0 sends data to leaf switch 103, completing 1 hop, and leaf switch 103 sends data to computing node H1, completing 2 hops.

[0077] In one example, the network domain to which the faulty node belongs can be determined from the network domains to which the multiple computing nodes belong respectively, and the network domain to which the faulty node belongs can be determined as the first network domain. Figure 1 , when the computing node H0 is a faulty node, the network domain 0 may be determined as the first network domain. It should be noted that each computing node in the computing node group belongs to the same network domain.

[0078] In one example, after the failed node is determined, the first network domain may be determined based on the switch to which the failed node is first connected. Figure 1 In the case where the computing node H0 is a faulty node, the network domain 0 corresponding to the leaf switch 0 connected to the computing node H0 may be determined as the first network domain.

[0079] In a feasible example, when the network domain to which the faulty node belongs is different from the network domain to which the coordinated node of the faulty node belongs, the network domain to which the coordinated node belongs may be determined as the first network domain. The coordinated node may refer to other computing nodes in the computing node group to which the faulty node belongs. Figure 1 If computing nodes H0 and H4 are a computing node group, when computing node H0 is a faulty node, the network domain N to which computing node H4 belongs can be determined as the first network domain. In this way, each computing node in the computing node group can be restored to belong to the same network domain, which is beneficial to reduce the length of the communication link and further improve the execution efficiency of distributed training tasks.

[0080] Step 703: If there is a spare node in the first network domain, select a spare node from the first network domain to replace the failed node.

[0081] After the failed node is replaced by a spare node, the spare node is updated and set as a computing node to continue to participate in the execution of distributed training tasks instead of the failed node, thereby restoring the distributed training of the machine learning model. Among them, the spare node can be used to replace the failed node to continue to execute the subtasks corresponding to the failed node. For example, in the case where the execution of the distributed training task of the p+1th stage is interrupted, the spare node can inherit the state data of the failed node after completing the distributed training task of the pth stage (such as model parameters, parameter gradients, etc.), so that the distributed training system can re-execute the distributed training task of the p+1th stage, where p is an integer.

[0082] The at least one standby node included in the distributed training system may be distributed in different network domains. A single network domain may include one or more standby nodes, or a single network domain may not include any standby nodes. Figure 1 , network domain 0 includes two spare nodes, while network domain N includes no spare node.

[0083] In one example, the number of standby nodes in the first network domain is m, the number of faulty nodes in the first network domain is n, the n faulty nodes are distributed in w faulty node groups, the faulty node group refers to a computing node group in which the faulty node exists, m is an integer, n and w are positive integers. Each faulty node group may include at least one faulty node, that is, w is less than or equal to n.

[0084] Exemplarily, the selection process of the standby node may include the following contents:

[0085] 1. When m is less than n, for the i-th fault node group among the w fault node groups, if the number of spare nodes in the first network domain is greater than or equal to the number of fault nodes in the i-th fault node group, the spare nodes are used to replace the fault nodes in the i-th fault node group, where i is an integer less than or equal to w.

[0086] The i-th faulty node group may refer to any faulty node group among the w faulty node groups. When m is less than n, the spare nodes in the first network domain are not enough to completely replace the n faulty nodes, that is, the w faulty node groups cannot be completely repaired. Therefore, some faulty node groups may be selected for repair to maintain the execution efficiency of the distributed training task as much as possible, thereby reducing the fluctuation range of the execution efficiency. The repaired faulty node group may be restored to the computing node group.

[0087] In one example, the faulty nodes in each faulty node group can be replaced one by one, or the faulty nodes in some faulty node groups can be replaced simultaneously, and the embodiments of the present application do not limit this. For example, since the spare nodes are updated and set as computing nodes after being used, the number of spare nodes decreases as the spare nodes are used. In the case of repairing the faulty node groups one by one, for each faulty node group, the number of spare nodes in the first network domain is compared with the number of faulty nodes in the current faulty node group. If the number of spare nodes in the first network domain is greater than or equal to the number of faulty nodes in the current faulty node group, the spare nodes are used to replace the faulty nodes in the current faulty node group to repair the current faulty nodes.

[0088] For example, the w faulty node groups are sorted in order of the number of faulty nodes from small to large to obtain a faulty node group sequence, and the faulty node groups are repaired in sequence starting from the head of the faulty node group sequence. Alternatively, the total number of faulty nodes in the first j faulty node groups in the faulty node group sequence is counted, and when the total number of faulty nodes is a maximum integer less than or equal to m, the faulty nodes in the first j faulty node groups are replaced simultaneously with spare nodes, j is a positive integer less than or equal to w, and the total number of faulty nodes is the sum of the number of faulty nodes in the j faulty node groups.

[0089] For example, refer to Figure 8 , the first network domain 801 includes computing nodes H0 to H3, and a standby node B1. The computing nodes H0 and H1 form computing node group 1, and the computing nodes H2 and H3 form computing node group 2. In the case that computing node H3 fails (i.e., changes to a failed node), the training master selects the standby node B1 in the first network domain 801 to replace computing node H3, and isolates computing node H3. The standby node B1 and computing node H2 recover computing node group 2 to continue to execute the subtasks corresponding to computing node group 2.

[0090] Optionally, if the number of standby nodes in the first network domain is less than the number of faulty nodes in the i-th faulty node group, the i-th faulty node group is controlled to cancel execution of a corresponding subtask.

[0091] If the number of spare nodes in the first network domain is less than the number of faulty nodes in the i-th faulty node group, the faulty nodes in the i-th faulty node group cannot be completely replaced, resulting in the i-th faulty node group being unable to recover to a computing node group and unable to continue to execute the corresponding subtask. In this case, the distributed training system can be scaled down, that is, the i-th faulty node group is removed from the execution process of the distributed training task. For computing nodes other than the faulty nodes in the i-th faulty node group (i.e., healthy computing nodes), the distributed training system is directed to stop using them.

[0092] For example, refer to Figure 8 After computing node group 2 is restored using backup node B1 and computing node H2, if computing node H2 also fails, since there is no backup node in the first network domain 801, that is, the number of backup nodes in the first network domain 801 is less than the number of failed nodes in computing node group 2, the training master controls computing node group 2 to cancel the execution of the corresponding subtask, that is, to isolate computing node H2 and stop using backup node B1 (which has been converted into a computing node at this time).

[0093] 2. When m is greater than or equal to n, select n backup nodes from the first network domain to replace the n failed nodes in the first network domain.

[0094] When m is greater than or equal to n, the spare nodes in the first network domain are sufficient to completely replace n faulty nodes, that is, to completely repair w faulty node groups. Therefore, n spare nodes can be randomly selected from the first network domain to replace the n faulty nodes in the first network domain, so that the distributed training system can resume executing distributed training tasks, thereby maintaining the execution efficiency of distributed training tasks and avoiding fluctuations in execution efficiency.

[0095] For example, refer to Fig. 9Taking the leaf-spine network architecture as an example, the distributed training system 900 corresponds to three network domains: network domain 0, network domain 1, and network domain 2. Network domain 0 includes 4 computing nodes and 2 standby nodes, network domain 1 includes 4 computing nodes, and network domain 2 includes 2 computing nodes and 2 standby nodes. The 10 computing nodes are divided into 5 computing node groups, and the 5 computing node groups are used to execute distributed training tasks in parallel.

[0096] In the case of a distributed training task execution interruption, the main computing node determines the faulty nodes H3 and H7 from the 10 computing nodes. For the faulty node H3, the number of faulty nodes in the faulty node group to which the faulty node H3 belongs is 1, which is less than the number of spare nodes in the network domain 0 (i.e., 2). Then, a spare node (such as spare node B1) can be randomly selected from spare nodes B1 and B2 to replace the faulty node H3, so as to repair the faulty node group to which the faulty node H3 belongs and isolate the faulty node H3.

[0097] For the faulty node H7, the number of faulty nodes in the faulty node group to which the faulty node H7 belongs is 1, which is greater than the number of spare nodes in the network domain 1 (i.e. 0). Then the faulty node group to which the faulty node H7 belongs is controlled to cancel the execution of the corresponding subtask, i.e. stop using the computing node H6 and isolate the faulty node H7.

[0098] In the embodiment of the present application, the distributed training task can be divided into multiple subtasks according to the parallelism of DP, so that the lack of a computing node group does not affect the execution of the corresponding subtasks by other computing node groups. If it is divided according to the parallelism of TP and PP, the lack of a computing node will cause the machine learning model to be unable to obtain complete distributed training, such as some model parameters cannot be trained.

[0099] In one example, considering the parallel strategy, the embodiment of the present application may also include the following content: when the distributed training task is divided according to the parallelism of TP, since the distributed training task can be completed by a single computing node, there is no need to consider the communication link between the computing nodes. A backup node can be directly selected from at least one backup node to replace the faulty node, or a backup node can be preferentially selected from the first network domain to replace the faulty node. When the distributed training task is divided according to the parallelism of PP, if there is a backup node in the first network domain, a backup node is selected from the first network domain to replace the faulty node. If there is no backup node in the first network domain, a new backup node can be waited for to appear in the first network domain to replace the faulty node, or a backup node in other network domains can be used to replace the faulty node first, and when a new backup node appears in the first network domain, the backup nodes in other network domains are replaced with new backup nodes in the first network domain to restore the communication link length between the computing nodes used to execute the distributed training task. When the distributed training task is divided according to the parallelism of DP, the technical solution provided in the embodiment of the present application can be used for training fault tolerance, and the embodiment of the present application is not limited to this.

[0100] Among them, when the distributed training tasks are divided according to the parallelism of TP or PP, the technical solution provided by the embodiment of the present application only includes the replacement of faulty nodes, and does not involve the expansion and reduction of the distributed training system. When the distributed training tasks are divided according to the parallelism of DP, the technical solution provided by the embodiment of the present application may include at least one of the replacement of faulty nodes, the expansion of the distributed training system, and the reduction of the distributed training system.

[0101] To sum up, the technical solution provided by the embodiment of the present application is that in the process of executing distributed training tasks using a distributed training system, if a faulty node occurs, the faulty node is replaced by a backup node located in the same network domain as the faulty node to restore the execution of the distributed training task, thereby avoiding the problem of increased communication links caused by the backup node and the faulty node not belonging to the same network domain, thereby avoiding the decrease in the execution efficiency of the distributed training task, and further facilitating maintaining the execution efficiency of the distributed training task.

[0102] In addition, since the network domain to which the backup node selected in the present application belongs is fixed relative to the faulty node, it is not easy to cause fluctuations in communication efficiency, thereby avoiding fluctuations in the execution efficiency of distributed training tasks, and further improving the stability of the execution efficiency of distributed training tasks.

[0103] In some embodiments, the embodiments of the present application also support expanding the capacity of the distributed training system to improve the execution efficiency of distributed training tasks. The process may include the following contents.

[0104] 1. Select a spare node from at least one spare node included in the distributed training system to build at least one expansion node group.

[0105] An expansion node group refers to a computing node group composed of spare nodes, which is used to increase the scale of computing nodes participating in the execution of distributed training tasks. Each computing node group can be used to train a complete machine learning model. As the number of computing node groups participating in the execution of distributed training tasks increases, the execution efficiency of distributed training tasks also increases.

[0106] Optionally, when the expansion node group includes at least two backup nodes, at least two backup nodes belong to the same network domain, so that the communication link between each backup node in the expansion node group can be minimized, thereby maximizing the efficiency of the expansion node group in executing the corresponding subtasks.

[0107] In one example, when the first condition is met, the step of selecting a backup node from at least one backup node included in the distributed training system to construct at least one expansion node group is performed. The first condition includes at least one of the following:

[0108] 1. The number of computing node groups in the distributed training system is less than the number threshold.

[0109] Optionally, the above-mentioned quantity threshold may refer to the number of computing node groups in the distributed training system in the initial state, and the distributed training system when executing the distributed training task of the first stage may be determined as the distributed training system in the initial state.

[0110] For example, in the initial state, the number of computing node groups in the distributed training system is t, and t can be determined as the number threshold. Fig. 9 In the initial state, the distributed training system 900 includes 5 computing node groups. Due to the faulty node H7, the number of computing node groups is reduced to 4. The number of computing node groups in the distributed training system 900 becomes less than the quantity threshold. In this case, a spare node can be used to build an expansion node group to supplement the distributed training system 900, so as to improve the efficiency of the distributed training system 900 in executing distributed training tasks and restore the execution efficiency of distributed training tasks.

[0111] Optionally, in the case where an expansion node group needs to be constructed due to capacity reduction, the capacity can be expanded after the distributed training system has been reduced in capacity and a stage of distributed training tasks has been executed; or when it is determined that the distributed training system needs to be reduced in capacity, an expansion node group can be constructed to expand the capacity. The embodiments of the present application are not limited to this.

[0112] Optionally, the above-mentioned quantity threshold may also refer to a preset value, which can be set and adjusted according to actual usage requirements. The above-mentioned preset value is used to enable the distributed training system to devote as much as possible to the execution process of the distributed training task, and to leave enough spare nodes for fault tolerance. For example, 80% of the maximum number of computing node groups that can be constructed by the distributed training system for the distributed training task can be determined as the quantity threshold, and the remaining computing nodes are set as spare nodes.

[0113] 2. New backup nodes appear in the distributed training system.

[0114] Optionally, the distributed training system includes not only the backup nodes set in the initial state, but also newly appeared backup nodes. In the case of new backup nodes, if the number of new backup nodes in the same network domain meets the number required to build an expanded node group, the new backup nodes in the same network domain can be built into an expanded node group.

[0115] Exemplarily, the process of determining a new standby node may include the following contents:

[0116] (1) When the faulty node group to which the faulty node belongs is canceled from executing the corresponding subtask, the computing nodes in the faulty node group except the faulty node are set as standby nodes.

[0117] For example, refer to Fig. 9 In the case where the faulty node group where the faulty node H7 is located is cancelled from executing the corresponding subtask, the computing node H6 can be adjusted to be set as a spare node, so that the computing node H6 is a newly added spare node.

[0118] If computing node H5 also fails, computing node H4 can be adjusted to be a new backup node. At this time, computing node H4 and computing node H6 can be constructed into an expansion node group to avoid a large drop in the execution efficiency of distributed training tasks.

[0119] (2) In the case where a computing node used to execute tasks other than the distributed training task in the distributed training system is released, the released computing node is set as a standby node.

[0120] Other tasks may refer to tasks different from distributed training tasks, and the released computing node may mean that the computing node is no longer occupied by other tasks, such as the computing node is cancelled from executing other tasks, or the computing node is automatically released after completing other tasks. The embodiments of the present application do not limit this.

[0121] For example, refer to Fig. 9, if there is a first computing node in network domain 2 that is being used to execute other tasks, then after the first computing node is automatically released after executing other tasks, the first computing node can be set as a new standby node in the network domain. For another example, if there is a first computing node in network domain 2 that is being used by a user and cannot participate in the execution of distributed training tasks, then after the user finishes using the first computing node, the first computing node can be set as a new standby node in the network domain.

[0122] (3) When a new computing node appears in the distributed training system, the new computing node is set as a standby node.

[0123] That is, after adding a computing node outside the distributed training system to the distributed training system, the computing node can be set as a standby node. Fig. 9 When the leaf switch 1 is connected to a new computing node, the new computing node can be set as a new standby node in the network domain 1.

[0124] 3. Enter the preset time period.

[0125] Optionally, the preset time period can be set and adjusted according to actual usage requirements, and the embodiments of the present application are not limited to this. For example, a time period with low computing node usage can be set as a preset time period, such as after entering the time period from 0:00 to 3:00 in the morning, a backup node can be selected from at least one backup node included in the distributed training system to form at least one expansion node group.

[0126] In one example, when the multiple computing nodes included in the above-mentioned distributed training system are divided into t computing node groups, the number of nodes s of the computing node group can be obtained first, where s is a positive integer; from at least one backup node, s backup nodes are selected each time to build an expansion node group.

[0127] For example, refer to Fig. 9 Each computing node group includes 2 computing nodes. When the faulty node group to which the faulty node H7 belongs is canceled from executing the corresponding subtask, the main computing node can cross the network domain and find 2 backup nodes in network domain 2: B3 and B4, and build an expansion node group based on the backup nodes B3 and B4 to replace the faulty node group to which the faulty node H7 belongs to execute the corresponding subtask, so as to maintain the execution efficiency of the distributed training task.

[0128] For the distributed training system 900, by adopting the technical solution provided in the embodiment of the present application, when the execution of the distributed training task is interrupted, the distributed training resources can be adjusted from the initial {(H0, H1), (H2, H3), (H4, H5), (H6, H7), (H8, H9)} to {(H0, H1), (H2, B1), (H4, H5), (H8, H9), (B3, B4)}, so that the execution efficiency of the distributed training task remains unchanged, thereby maintaining the stability of the execution efficiency of the distributed training task.

[0129] 2. Control the expansion node group to participate in the execution of distributed training tasks.

[0130] Optionally, after building the expansion node group, the main computing node can update the expansion node group to the computing node group, and then count the total number of computing node groups in the distributed training system, and then divide the distributed training task into a total number of subtasks, and finally execute the total number of subtasks in parallel through all computing node groups to resume execution of the distributed training task.

[0131] For example, refer to Fig. 9 , in the initial state, the sample data of the machine learning model can be divided into 5 parts, and each of them is processed in parallel by 5 computing node groups. In the case where the faulty node group to which the faulty node H7 belongs is cancelled from executing the corresponding subtask, the sample data of the machine learning model can be divided into 4 parts, and each of them is processed in parallel by the remaining 4 computing node groups. After adding the expansion node group composed of backup nodes B3 and B4, the sample data of the machine learning model can be restored to being divided into 5 parts, and each of them is processed in parallel by the adjusted 5 computing node groups.

[0132] To sum up, the technical solution provided in the embodiment of the present application can expand the capacity of the distributed training system through the spare nodes after the distributed training system is reduced in capacity to restore the execution efficiency of the distributed training tasks, or can expand the capacity of the distributed training system through the spare nodes to accelerate the execution of the distributed training tasks, thereby helping to improve the stability and performance of the distributed training system.

[0133] In some embodiments, reference Fig.10 , the embodiments of the present application may also include the following contents.

[0134] 1.Practice pull-ups.

[0135] The distributed training system executes distributed training tasks for machine learning models to launch distributed training.

[0136] For example, refer to Fig. 9The distributed training system 900 can execute distributed training tasks in parallel through 5 computing node groups. For example, each computing node group trains the complete machine learning model with a part of sample data to obtain training results (such as model parameters, parameter gradients, etc.), and then summarizes the training results corresponding to the 5 computing node groups to update the model parameters of the machine learning model to obtain updated model parameters. Finally, the updated model parameters are completely distributed to the 5 computing node groups for the next round of distributed training until the training of the machine learning model is completed.

[0137] 2. Training monitoring.

[0138] The master computing node will continue to monitor the distributed training task, and exit the distributed training task after determining that the distributed training task has been completed. Optionally, the master computing node will also monitor the status of each computing node in the distributed training system.

[0139] For example, refer to Fig. 9 , the main computing node in the distributed training system 900 can continuously monitor the status of the 10 computing nodes in the distributed training system 900.

[0140] 3. Fault diagnosis.

[0141] In the case where the execution of the distributed training task is interrupted, that is, when a fault occurs in the distributed training system and causes the execution of the distributed training task to be interrupted, the main computing node enters the fault diagnosis process.

[0142] For example, refer to Fig. 9 , the main computing node in the distributed training system 900 can analyze the cause of the fault according to the log of the distributed training system 900 to determine whether the fault is repairable. If the fault is repairable, go to step 4; if the fault is not repairable, go to step 5.

[0143] 4. Fault repair.

[0144] If the fault is repairable, the main computing node can directly guide the distributed training system to resume executing the distributed training task after the fault is repaired, and go to step 1 after the fault is repaired.

[0145] For example, refer to Fig. 9 In the case where the execution of the distributed training task is interrupted due to a network interruption of the computing node H0, the main computing node can directly guide the distributed training system 900 to resume the execution of the distributed training task after restoring the network of the computing node H0.

[0146] 5. Find a backup node.

[0147] If the fault is unrepairable, for each faulty node, the main computing node determines the first network domain to which the faulty node belongs, and then searches for a spare node in the first network domain to replace the faulty node, and then goes to step 1. Optionally, if there are not enough spare nodes in the first network domain, go to step 6.

[0148] When there are spare nodes in the first network domain and the number of spare nodes is sufficient to replace all faulty nodes in the first network domain, the spare nodes are used to replace all faulty nodes in the first network domain so that the spare nodes execute corresponding subtasks instead of the faulty nodes, thereby restoring the execution of the distributed training task.

[0149] For example, refer to Fig. 9 If there is only a faulty node H3 in the distributed training system 900, the faulty node H3 is replaced with the backup node B1, and after the faulty node H3 is isolated, the distributed training system 900 can be controlled to restart the distributed training of the machine learning model.

[0150] 6. Try to reduce the capacity.

[0151] When there are not enough spare nodes in the first network domain, the main computing node guides the distributed training system to scale down, that is, directly controls the faulty node group to which the faulty node belongs to cancel the execution of the corresponding subtask.

[0152] For example, refer to Fig. 9 If there is only faulty node H7 in the distributed training system 900 and there is no spare node in the network domain 1, the main computing node can guide the distributed training system 900 to exclude the faulty node group to which the faulty node H7 belongs from the execution process of the distributed training task to achieve capacity reduction.

[0153] When the distributed training system is successfully scaled down, go to step 1, that is, resume executing the distributed training task through the scaled-down distributed training system, and the execution efficiency of the distributed training task is reduced.

[0154] 7. Try to expand capacity.

[0155] After the distributed training system is scaled down, if there are enough spare nodes in another network domain, the main computing node can try to build an expansion node group through the spare nodes in the other network domain to expand the distributed training system and restore the execution efficiency of the distributed training tasks.

[0156] For example, refer to Fig. 9 , when the faulty node group to which the faulty node H7 belongs is excluded from the execution process of the distributed training task, the main computing node can build an expansion node group through the two backup nodes in network domain 2 to replace the faulty node group to which the faulty node H7 belongs to participate in the execution of the distributed training task.

[0157] Alternatively, when the backup nodes in the same network domain are insufficient to build an expanded node group, if a new backup node appears in the distributed system, the main computing node can try to build an expanded node group through the new backup node to expand the capacity of the distributed training system and restore the execution efficiency of the distributed training tasks.

[0158] For example, refer to Fig. 9 , when the faulty node group to which the faulty node H7 belongs is excluded from the execution process of the distributed training task, if two new backup nodes appear in the network domain 1, the main computing node can build an expanded node group based on the two new backup nodes to replace the faulty node group to which the faulty node H7 belongs to participate in the execution of the distributed training task.

[0159] When the capacity expansion of the distributed training system is successful, go to step 1, that is, resume executing the distributed training task through the expanded distributed training system, and the execution efficiency of the distributed training task returns to normal.

[0160] To sum up, the technical solution provided by the embodiment of the present application is that in the process of using a distributed training system to execute distributed training tasks, in the event of a faulty node, you can first try to use a spare node located in the same network domain as the faulty node to replace the faulty node to maintain the execution efficiency of the distributed training task. If there are not enough spare nodes in the network domain to which the faulty node belongs, try to reduce the capacity of the distributed training system. After reducing the capacity of the distributed training system, continue to try to expand the capacity of the distributed training system through the spare nodes, so that the execution efficiency of the distributed training task is always maintained within a stable range, which is beneficial to maintaining the stability and performance of distributed training.

[0161] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0162] refer to Fig.11 , which shows a block diagram of a training fault-tolerant device for a distributed training system provided in a possible implementation of the present application. The device has the function of implementing the above method example, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the computer device introduced above (such as the main computing node), or it can be set in a computer device. Fig.11 As shown, the device 1100 includes: a fault node determination module 1101, a network domain determination module 1102 and a backup node selection module 1103.

[0163] The faulty node determination module 1101 is used to determine a faulty node from the multiple computing nodes when the execution of the distributed training task is interrupted. The faulty node refers to a computing node that can no longer participate in the execution of the distributed training task.

[0164] The network domain determining module 1102 is configured to determine a first network domain based on the network domains to which the multiple computing nodes respectively belong, where the first network domain is the network domain to which the faulty node belongs.

[0165] The backup node selection module 1103 is configured to select the backup node from the first network domain to replace the failed node if the backup node exists in the first network domain.

[0166] In some embodiments, the distributed training task includes t subtasks, the multiple computing nodes are divided into t computing node groups, the t computing node groups are used to execute the t subtasks in parallel, the number of standby nodes in the first network domain is m, the number of faulty nodes in the first network domain is n, the n faulty nodes are distributed in w faulty node groups, the faulty node group refers to the computing node group in which the faulty node exists, m is an integer, n and w are positive integers, and t is an integer greater than 1;

[0167] The backup node selection module 1103 is used to, when m is less than n, for the i-th fault node group among the w fault node groups, if the number of backup nodes in the first network domain is greater than or equal to the number of fault nodes in the i-th fault node group, use the backup node to replace the faulty nodes in the i-th fault node group, where i is an integer less than or equal to w.

[0168] In some embodiments, the backup node selection module 1103 is also used to control the i-th faulty node group to cancel the execution of the corresponding subtask if the number of backup nodes in the first network domain is less than the number of faulty nodes in the i-th faulty node group.

[0169] In some embodiments, the backup node selection module 1103 is further used to select n backup nodes from the first network domain to replace n failed nodes in the first network domain when m is greater than or equal to n.

[0170] In some embodiments, Fig.12 As shown, the device 1100 also includes: an expansion node group construction module 1104 and an expansion node group adding module 1105.

[0171] The expansion node group construction module 1104 is used to select the backup node from the at least one backup node included in the distributed training system to construct at least one expansion node group. When the expansion node group includes at least two backup nodes, the at least two backup nodes belong to the same network domain.

[0172] The expansion node group adding module 1105 is used to control the expansion node group to participate in executing the distributed training task.

[0173] In some embodiments, the expansion node group construction module 1104 is also used to execute the step of selecting the backup node from the at least one backup node included in the distributed training system to construct at least one expansion node group when a first condition is met; wherein the first condition includes at least one of the following: the number of computing node groups in the distributed training system is less than a quantity threshold, a new backup node appears in the distributed training system, and a preset time period is entered.

[0174] In some embodiments, the multiple computing nodes are divided into t computing node groups; the expansion node group construction module 1104 is also used to obtain the number of nodes s of the computing node group, s is a positive integer; from the at least one backup node, each s backup nodes are selected to construct an expansion node group.

[0175] In some embodiments, Fig.12 As shown, the apparatus 1100 further includes: a standby node setting module 1106. The standby node setting module 1106 is used to:

[0176] In the case that the faulty node group to which the faulty node belongs is cancelled from executing the corresponding subtask, setting the computing nodes in the faulty node group except the faulty node as the standby nodes;

[0177] Alternatively, in a case where a computing node in the distributed training system that is used to execute tasks other than the distributed training task is released, setting the released computing node as the standby node;

[0178] Alternatively, when a new computing node appears in the distributed training system, the new computing node is set as the standby node.

[0179] In some embodiments, when the network structure of the distributed training system is a leaf-spine network structure, each computing node in the same network domain is connected to the same leaf switch.

[0180] To sum up, the technical solution provided by the embodiment of the present application is that in the process of executing distributed training tasks using a distributed training system, if a faulty node occurs, the faulty node is replaced by a backup node located in the same network domain as the faulty node to restore the execution of the distributed training task, thereby avoiding the problem of increased communication links caused by the backup node and the faulty node not belonging to the same network domain, thereby avoiding the decrease in the execution efficiency of the distributed training task, and further facilitating maintaining the execution efficiency of the distributed training task.

[0181] In addition, since the network domain to which the backup node selected in the present application belongs is fixed relative to the faulty node, it is not easy to cause fluctuations in communication efficiency, thereby avoiding fluctuations in the execution efficiency of distributed training tasks, and further improving the stability of the execution efficiency of distributed training tasks.

[0182] Please refer to Fig.13 , which is a simplified structural block diagram of a computer device 1300 provided in a possible implementation of the present application. The computer device 1300 can be implemented as the above-mentioned computing node or backup node or main computing node. The computer device 1300 can be used to implement the training fault tolerance method applied to the distributed training system provided in the above-mentioned embodiment.

[0183] Typically, the computer device 1300 includes a processor 1301 and a memory 1302 .

[0184] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI processor, which is used to process computing operations related to machine learning.

[0185] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned training fault tolerance method applied to the distributed training system.

[0186] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the computer device 1300, and the computer device 1300 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0187] Optionally, the computer device 1300 may be a server, a server cluster, an artificial intelligence computing cluster, a cloud computing cluster, etc., wherein the artificial intelligence computing cluster may also be referred to as an intelligent computing cluster or a smart computing cluster, which is not limited in this application.

[0188] In some embodiments, a chip product is also provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned training fault-tolerant method applied to a distributed training system.

[0189] In some embodiments, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor of a computer device, the computer program implements the above-mentioned training fault tolerance method applied to a distributed training system.

[0190] Optionally, the computer readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives) or optical disks, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0191] In some embodiments, a computer program product is further provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned training fault tolerance method applied to a distributed training system.

[0192] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0193] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training fault tolerance method applied to a distributed training system, characterized in that: The distributed training system includes a plurality of computing nodes and at least one standby node, wherein the plurality of computing nodes are used to execute distributed training tasks; the method includes: In the case where the execution of the distributed training task is interrupted, determining a faulty node from the multiple computing nodes, wherein the faulty node refers to a computing node that can no longer participate in the execution of the distributed training task; Determine a first network domain based on the network domains to which the multiple computing nodes respectively belong, where the first network domain is the network domain to which the faulty node belongs; In the case that the backup node exists in the first network domain, the backup node is selected from the first network domain to replace the failed node.

2. The method according to claim 1, characterized in that The distributed training task includes t subtasks, the multiple computing nodes are divided into t computing node groups, the t computing node groups are used to execute the t subtasks in parallel, the number of standby nodes in the first network domain is m, the number of faulty nodes in the first network domain is n, the n faulty nodes are distributed in w faulty node groups, the faulty node group refers to the computing node group where the faulty node exists, m is an integer, n and w are positive integers, and t is an integer greater than 1; In the case where the backup node exists in the first network domain, selecting the backup node from the first network domain to replace the failed node includes: When m is less than n, for the i-th fault node group among the w fault node groups, if the number of spare nodes in the first network domain is greater than or equal to the number of fault nodes in the i-th fault node group, the spare nodes are used to replace the fault nodes in the i-th fault node group, where i is an integer less than or equal to w.

3. The method according to claim 2, characterized in that The method further comprises: If the number of standby nodes in the first network domain is less than the number of faulty nodes in the i-th faulty node group, the i-th faulty node group is controlled to cancel execution of a corresponding subtask.

4. The method according to claim 2, characterized in that: The method further comprises: In the case that m is greater than or equal to n, n backup nodes are selected from the first network domain to replace the n failed nodes in the first network domain.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: From the at least one standby node included in the distributed training system, select the standby node to build at least one expansion node group, and when the expansion node group includes at least two standby nodes, the at least two standby nodes belong to the same network domain; Control the expansion node group to participate in executing the distributed training task.

6. The method according to claim 5, characterized in that The method further comprises: When the first condition is met, the step of selecting the at least one backup node from the at least one backup node included in the distributed training system to build at least one expansion node group is performed; Among them, the first condition includes at least one of the following: the number of computing node groups in the distributed training system is less than the number threshold, a new standby node appears in the distributed training system, and a preset time period begins.

7. The method according to claim 5, characterized in that The plurality of computing nodes are divided into t computing node groups; The selecting the at least one spare node from the at least one spare node included in the distributed training system to build at least one expansion node group includes: Get the number of nodes s of the computing node group, where s is a positive integer; From the at least one spare node, s spare nodes are selected at a time to construct an expansion node group.

8. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In the case that the faulty node group to which the faulty node belongs is cancelled from executing the corresponding subtask, setting the computing nodes in the faulty node group except the faulty node as the standby nodes; or, In the case where a computing node in the distributed training system used to execute other tasks except the distributed training task is released, setting the released computing node as the standby node; or, When a new computing node appears in the distributed training system, the new computing node is set as the standby node.

9. The method according to any one of claims 1 to 4, characterized in that: When the network structure of the distributed training system is a leaf-spine network structure, each computing node in the same network domain is connected to the same leaf switch.

10. A training fault-tolerant device applied to a distributed training system, characterized in that: The distributed training system includes a plurality of computing nodes and at least one standby node, wherein the plurality of computing nodes are used to execute distributed training tasks; The device comprises: A fault node determination module, used to determine a fault node from the multiple computing nodes when the execution of the distributed training task is interrupted, wherein the fault node refers to a computing node that can no longer participate in the execution of the distributed training task; A network domain determination module, configured to determine a first network domain based on the network domains to which the plurality of computing nodes respectively belong, wherein the first network domain is the network domain to which the faulty node belongs; The standby node selection module is used to select the standby node from the first network domain to replace the failed node if the standby node exists in the first network domain.

11. A chip product, characterized in that: The chip product stores a computer program, which is loaded and executed by a processor to implement the training fault-tolerant method applied to a distributed training system as described in any one of claims 1 to 9.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the training fault tolerance method applied to a distributed training system as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the training fault-tolerant method applied to a distributed training system as described in any one of claims 1 to 9.

14. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium. The processor reads and executes the computer program from the computer-readable storage medium to implement the training fault-tolerant method applied to a distributed training system as described in any one of claims 1 to 9.

Images