Fault recovery method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a fault recovery method and device, electronic equipment and a storage medium, and the method comprises the steps: stopping executing a local training process corresponding to a training subtask under the condition that the training process is detected to be abnormal or a remote node fault notification is received, checking the health state of the local hardware to obtain a checking result; and performing dynamic networking based on the check result, and restarting the local training process after the networking is successful to recover the execution of the training sub-task. According to the method, the fault discovery time can be shortened and the fault recovery efficiency can be improved by introducing the remote node fault sensing capability, and potential hardware problems can be identified before the training process is restarted by checking the health state of local hardware, so that the occurrence of repeated faults is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a fault recovery method, device, electronic device and storage medium. Background Art

[0002] In the training process of artificial intelligence (AI) models, training data, AI models, and hardware computing power are three core elements. The training process of AI models is to input a large amount of training data into the AI ​​model deployed on the hardware, and the AI ​​model uses the computing power of the hardware to process and learn the training data. Generally speaking, the more training data, the better the learning effect and the higher the accuracy of the AI ​​model.

[0003] As the scale of problems solved by AI models increases, the model parameter scale and the amount of training data also continue to increase, resulting in an increasing demand for hardware computing power. Therefore, distributed training has become an indispensable training method. Distributed training significantly improves training efficiency by splitting large-scale training tasks into multiple subtasks and executing them in parallel on multiple computing nodes.

[0004] However, as the scale of models and training data increases, the number of nodes in distributed tasks also increases, resulting in a significant increase in the probability of hardware and software failures during the training process. This will not only cause model training to be interrupted or fail, but also increase the difficulty and time cost of recovery after the failure. Summary of the invention

[0005] The present invention provides a fault recovery method, device, electronic device and storage medium, which are used to solve the defects of long fault recovery time and low efficiency in the model training process.

[0006] The present invention provides a fault recovery method, which is applied to a node, and the node is used to perform a training subtask of distributed training of an artificial intelligence model, and the method comprises: When a training process abnormality is detected or a remote node failure notification is received, the local training process corresponding to the executed training subtask is stopped, and the health status of the local hardware is checked to obtain the inspection result; Based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

[0007] According to a fault recovery method provided by the present invention, the dynamic networking is performed based on the inspection result, including: If the inspection result is healthy, dynamic networking is directly performed; When the inspection result indicates that a hardware fault exists, dynamic networking is performed based on a repair type of the hardware fault.

[0008] According to a fault recovery method provided by the present invention, the dynamic networking based on the repair type of the hardware fault includes: If the repair type of the hardware fault is repairable, repair the hardware fault and perform dynamic networking based on the repair result; When the repair type of the hardware fault is unrepairable, the system exits with a predetermined fault code, which is used to instruct the platform to identify the node as a faulty node for isolation and reschedule the Pod on the node to a healthy node.

[0009] According to a fault recovery method provided by the present invention, the dynamic networking based on the repair result includes: When the repair result is successful, dynamic networking is performed; When the repair result is a repair failure, the system exits with the agreed fault code.

[0010] According to a fault recovery method provided by the present invention, the training process abnormality includes any one of a local training process abnormality and a remote training process abnormality.

[0011] The present invention also provides a fault recovery method, which is applied to a platform, the platform is associated with a computing resource pool, the computing resource pool includes multiple nodes for distributed training of artificial intelligence models, each of the multiple nodes is used to perform a training subtask of the distributed training of the artificial intelligence model, and the method includes: In the case of detecting that a first node fails, marking the first node as a failed node for isolation, the first node being a failed node among the multiple nodes; Sending a remote node failure notification to a second node, where the second node is a node that has not failed among the multiple nodes, the remote node failure notification is used to instruct the second node to stop executing a local training process corresponding to the training subtask, and to check the health status of the local hardware, the second node is used to perform dynamic networking based on the inspection result, and after the networking is successful, restart the local training process to resume the execution of the training subtask; The Pod on the first node is rescheduled to a third node, where the third node is a healthy node in the computing resource pool other than the multiple nodes, and the third node is used to replace the first node to execute the training subtask.

[0012] According to a fault recovery method provided by the present invention, when any one of the multiple nodes detects that the local training process corresponding to the executed training subtask is abnormal, the node sends a training process abnormality notification to other nodes among the multiple nodes except the any one node, and the training process abnormality notification is used to instruct the other nodes to stop the local training process corresponding to the executed training subtask and check the health status of the local hardware.

[0013] The present invention also provides a fault recovery device, which is applied to a node, and the node is used to perform a training subtask of distributed training of an artificial intelligence model, and the device includes: The inspection unit is used to stop the local training process corresponding to the executed training subtask and inspect the health status of the local hardware to obtain the inspection result when an abnormality in the training process is detected or a remote node failure notification is received; The recovery unit is used to perform dynamic networking based on the inspection result, and restart the local training process after the networking is successful to resume the execution of the training subtask.

[0014] The present invention also provides a fault recovery device, which is applied to a platform, the platform is associated with a computing resource pool, the computing resource pool includes a plurality of nodes for distributed training of an artificial intelligence model, each of the plurality of nodes is used to perform a training subtask of the distributed training of the artificial intelligence model, and the device includes: an isolation unit, configured to, when detecting that a first node fails, identify the first node as a failed node for isolation, wherein the first node is a node that fails among the multiple nodes; a sending unit, configured to send a remote node fault notification to a second node, wherein the second node is a node that has not failed among the multiple nodes, wherein the remote node fault notification is used to instruct the second node to stop executing a local training process corresponding to the training subtask, and to check the health status of the local hardware, wherein the second node is used to dynamically network based on the check result, and after the networking is successful, restart the local training process to resume the execution of the training subtask; A rescheduling unit is used to reschedule the Pod on the first node to a third node, where the third node is a healthy node in the computing resource pool other than the multiple nodes, and the third node is used to replace the first node to execute the training subtask.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned fault recovery methods when executing the computer program.

[0016] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the fault recovery method described in any one of the above methods is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned fault recovery methods.

[0018] The fault recovery method, device, electronic device and storage medium provided by the present invention, when a node detects an abnormality in the training process or receives a remote node fault notification, immediately enters the fault recovery process, that is, stops the local training process, and checks the health status of the local hardware. This immediate response can minimize the impact of the fault on the overall training progress and quickly try to restore the training, thereby saving time and computing resources. By checking the health status of the local hardware, potential hardware problems can be identified before restarting the training process, thereby avoiding the occurrence of repeated failures and improving the stability and reliability of model training. Dynamic networking is performed according to the hardware inspection results, and the local training process is restarted after the networking is successful, which can ensure that the training task continues to be executed in the new or adjusted network environment. In addition, the present invention introduces remote node fault perception capabilities. Once a node in a distributed training task fails, other nodes can immediately receive notifications and quickly enter the fault recovery process, thereby greatly shortening the fault discovery time and improving the fault recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a schematic diagram of the architecture of the dynamic networking of the training framework provided by the present invention; Figure 2 This is one of the flowcharts of the fault recovery method provided by the present invention; Figure 3 It is a schematic diagram of Pod rescheduling provided by the present invention; Figure 4 This is the second flow chart of the fault recovery method provided by the present invention; Figure 5 It is a flow chart of a fault recovery method based on Pod rescheduling provided by the present invention; Figure 6 It is one of the structural schematic diagrams of the fault recovery device provided by the present invention; Figure 7 This is the second structural schematic diagram of the fault recovery device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In order to facilitate understanding of the technical solutions and embodiments provided by the present invention, the concepts of AI model, AI model training, distributed training, nodes, platform, etc. are described in detail below: AI model is a kind of mathematical algorithm model that uses machine learning ideas to solve practical problems. AI model includes a large number of parameters and calculation formulas (or calculation rules). The parameters in AI model are numerical values ​​that can be obtained by training AI model through training data sets. For example, the parameters of AI model are the calculation formulas or weights of calculation factors in AI model. It should be noted that the AI ​​model mentioned in the present invention is a general term, and AI model includes deep learning model, machine learning model, etc. General AI models need to be trained before they are used to complete a task.

[0023] Training an AI model means using existing data to fit the AI ​​model to the rules of the existing data through certain methods and determine the parameters in the AI ​​model. Training an AI model requires preparing a training data set, using the training data in the training data set as the input of the AI ​​model, and the AI ​​model calculates and processes the input training data to adjust the parameters in the AI ​​model.

[0024] Distributed training is one of the commonly used acceleration methods in the AI ​​model training process. Distributed training means splitting the training into multiple independent computing nodes for independent calculation, and then periodically aggregating and redistributing the results, thereby accelerating the AI ​​model training process.

[0025] The platform provides AI developers and users with a convenient AI development environment and convenient development tools. It is responsible for managing and optimizing resource allocation and task scheduling during model training. It can monitor and manage the computing resources, storage resources, and network resources required for training, and ensure that each training task can obtain sufficient resource support through reasonable resource allocation.

[0026] A node, also known as a computing node, refers to the basic building block in distributed training, which may include multiple computing devices, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-purpose computing on graphics processing units), TPU (Tensor Processing Unit), etc.

[0027] Pod refers to the basic unit of management and deployment in distributed model training. It is a relatively independent and isolated environment for process running constructed using a virtualization technology in the computer operating system. The environment can contain independent file systems, namespaces, resource views, etc. It should be understood that Pod is a relatively independent and isolated environment constructed on a node using virtualization technology. Each Pod will be deployed on a specific node and use the computing resources of the node to train the model. In distributed training, AI models are usually split into multiple parts (or shards), and each Pod is responsible for managing and deploying one or more of these shards. Pod provides a relatively independent and isolated environment to ensure that the model shards can run stably in this environment.

[0028] A training task refers to a set of programs that need to be executed to complete the training of an initial AI model. The completion of a training task means completing the training of an initial AI model and obtaining a trained AI model.

[0029] A training subtask refers to a single program in a set of programs corresponding to a training task. In distributed training, a complex training task will be split into multiple training subtasks, which will be distributed to multiple nodes and trained on the nodes. A training task includes at least one training subtask. It should be understood that Pod is a management and deployment unit for model shards. Each Pod will carry one or more model shards and be responsible for the training subtasks corresponding to these shards. Therefore, when a Pod exits due to software or hardware failure, the training subtasks on it are usually interrupted.

[0030] A training process is a process of a training subtask executed on a computing node. One training process corresponds to one training subtask, and one computing node can execute one or more training subtasks, so there are one or more training processes on one computing node.

[0031] The training framework is a toolkit or function package that is required during the AI ​​model training process. It is also a running program framework that each training subtask in the training task needs to rely on. Currently, the more common training frameworks include PyTorch, Tensorflow, etc.

[0032] The computing resource pool consists of computing resources that can be used for AI model training. Computing resources can be computing nodes. For a training task, computing resources refer to all computing nodes used in the training process. Each computing node can include multiple computing devices (such as GPUs), and each computing device has a training process.

[0033] As the scale of AI model parameters and the amount of training data increase, the demand for storage resources and hardware computing power has exceeded the upper limit that a single computing device can provide. Therefore, in order to break through this limitation and improve the training throughput, a 3D parallel strategy is often adopted, that is, to implement distributed training of the model through data parallelism, tensor parallelism, and pipeline parallelism. The following is a brief introduction to the 3D parallel strategy.

[0034] Data parallel distributed training means deploying the same AI model on multiple nodes, distributing the training data in the training data set to multiple nodes for simultaneous calculation, executing AI model training on each node, and synchronizing the gradients of the model parameters generated on each node before updating the model parameters. By increasing the dimension of data parallelism, the training throughput can be effectively improved.

[0035] Tensor parallel distributed training is achieved by dividing the model weight matrix into multiple parts and distributing the calculation logic corresponding to the weights to multiple nodes in parallel. In this way, the video memory limitation of tensor calculations for model training on a single computing device can be overcome.

[0036] Pipeline parallel distributed training means dividing the model layers into multiple stages, distributing the model networks at different stages to different nodes for processing, and processing each stage in parallel. This not only reduces the memory requirements for each computing device, but also improves the overall training throughput.

[0037] 3D parallel distributed training can effectively solve the memory limitation of model training and improve throughput. 3D parallel has become a basic method for model training. As the scale of parameters and training data increases, the number of nodes involved in distributed training increases accordingly. For example, for an AI model with 175 billion parameters, hundreds of nodes (each node is equipped with 8 computing devices) are required to train for about 1 month. Large-scale distributed training tasks face more challenges in resuming training once a failure occurs than model training completed on a single computing device.

[0038] If a model training task fails, the process of resuming training from a breakpoint is as follows: 1) During the training process, regularly save breakpoints to the storage system. Here, breakpoints refer to the training status that is regularly saved during the model training process in order to be able to resume training after a failure. This training status usually includes information such as the current weight of the model, the state of the optimizer (such as learning rate, momentum, etc.), the current training round or batch index, etc. The purpose of this step is to be able to restore to the most recent valid state in the event of a failure, avoiding training from scratch.

[0039] 2) Individual Pods exit due to software or hardware failures. During model training, if a Pod stops running due to software errors, hardware failures, or other reasons, the training subtask on it will also be interrupted. This is because the exit of the Pod means that the computing resources and storage resources it relies on will no longer be available, resulting in the inability to continue the training subtask.

[0040] 3) Restart the entire training task through the platform or manual intervention. When a training subtask is detected to be interrupted, the entire training task can be restarted through the platform or manual operation.

[0041] 4) After the task is restarted, load the breakpoints and resume training. After the entire training task is restarted, load the previously saved breakpoint information from the storage system and restore the training status based on this information.

[0042] According to the above process, the first failure in a distributed training task is usually a single node. When the Pod corresponding to the failed node terminates abnormally, the restart mechanism of the entire training task is triggered. When executing the task restart, all Pods related to the task need to be terminated and then rescheduled to nodes that meet the resource requirements. Then, pull the necessary images on the newly allocated nodes and start the corresponding containers. This solution has shown good performance in small and medium-scale model training fault recovery. However, in the scenario of large model training, as the number of task nodes increases, this task-level fault recovery method will also significantly increase in processing difficulty, and each link of fault recovery is prone to cause system-level performance and stability problems. For example, the distributed training of a model is performed on 1,000 GPUs. If one of the GPUs fails, all processes on these 1,000 GPUs need to be restarted, resulting in difficulty and low efficiency in fault recovery.

[0043] The above task-level fault recovery method is not suitable for distributed training of large models. As the scale of distributed training task nodes increases, the above method exposes a series of problems: (1) Increase the time consumption of fault recovery. Task restart includes Pod destruction, resource rescheduling, image pulling, container initialization, etc. These processing processes will introduce several minutes or even more than ten minutes of time overhead, affecting the efficiency of fault recovery.

[0044] (2) Poor scalability. Task rescheduling puts pressure on the platform, and the pressure increases with the increase in node scale. Large-scale task restarts require the platform to be able to quickly respond to large-scale resource scheduling and complete the release and application of resources such as GPUs and networks. Services such as mirrors and data storage also need to meet high-concurrency requests. Once a link cannot be met, it will lead to restart failure or introduce a huge time consumption. This task-level fault handling method is not conducive to the expansion of the scale of distributed tasks.

[0045] (3) It is not conducive to data localization. The training task may cache data locally, for example, storing the training status (i.e., breakpoints) in the main memory. Once the Pod is destroyed, the local data will be lost.

[0046] (4) Not conducive to task stability. The performance, stability and other indicators of distributed training tasks may be affected by the environment. For example, the performance of training tasks may be different under different network topologies. The new resource environment after the task is restarted may cause fluctuations in indicators or introduce new problems, which is not conducive to task stability and problem troubleshooting.

[0047] In this regard, the present invention proposes a fault recovery method based on Pod rescheduling. By optimizing the processing of each link in the fault recovery process into operations at the Pod granularity, the system risks introduced by the task-level fault handling method are avoided, the efficiency of fault recovery is improved, and the fault recovery function has good scalability relative to the distributed task scale.

[0048] The task-level fault handling method faces a series of problems in the distributed training scenario of large models. In order to avoid the stability and performance problems introduced by task-level operations, this paper optimizes each link of fault recovery. The optimization goal is to reduce task-level operations, thereby reducing or avoiding the impact of distributed task scale on fault recovery. The specific optimization goals are as follows: (1) If there is no hardware failure, no rescheduling is performed; (2) If there is a hardware failure, only the Pod on the failed node is rescheduled.

[0049] In order to achieve the above objectives, the present invention is designed and implemented in the following aspects: 1. Dynamic Networking The Pytorch training framework achieves elastic scaling and fault tolerance capabilities through TorchElastic. During elastic training, the address and number of task nodes change dynamically. TorchElastic uses dynamic networking to eliminate the need for task-level rescheduling of training tasks when expanding or shrinking, but only scheduling incremental Pods or expelling some Pods. This approach makes elastic training lightweight and efficient. It should be understood that TorchElastic is a component in the Pytorch training framework, designed specifically for distributed training. It allows distributed training tasks to automatically adjust the number of training processes and resume training from the most recent breakpoint when facing node failures or resource changes, thereby ensuring the continuity and stability of training.

[0050] It is understandable that elastic scaling allows training tasks to dynamically increase training processes when resources are available to improve training efficiency. When some resources need to be released, TorchElastic can also reduce the number of training processes accordingly to optimize resource usage. When a training process fails, TorchElastic can capture the exception and restart the process. The restarted process will resume training from the most recent breakpoint, thereby ensuring the continuity of the training process.

[0051] Figure 1 It is a schematic diagram of the architecture of the dynamic networking of the training framework provided by the present invention, such as Figure 1 As shown in the figure, TorchElastic introduces Elastic Agent and Rendezvous mechanisms. An Elastic Agent process is deployed on each node, which is responsible for starting and managing all training processes on the local node. Elastic Agents communicate with each other through shared storage to complete the Rendezvous mechanism and restart the training process. When a training process fails, the corresponding Elastic Agent is responsible for restarting the process and resuming training from the most recent breakpoint. TorchElastic uses Rendezvous as a distributed synchronization mechanism for peer discovery. Before training starts, each training process will synchronize information through the Rendezvous mechanism, including the node list, the role of each node training process, etc. When the number of training processes changes (such as increase or decrease), the Elastic Agent processes of all nodes will renegotiate, establish a new process group, realize dynamic networking, and assign a number to each training process. It should be understood that dynamic networking means that in a distributed training task, when a node or training process becomes abnormal and is successfully restarted, the ElasticAgent on all nodes needs to renegotiate and establish a new network to ensure information synchronization between nodes and the continuity of training tasks.

[0052] TorchElastic provides a basis for Pod-level fault handling at the training framework level. The present invention is based on the dynamic networking function design of TorchElastic, and improves and enhances the distributed protocol or function of TorchElastic. The specific improvements are as follows: 1) In the implementation of the original protocol, each Elastic Agent can only perceive the status of the training process on the local node. On this basis, the present invention adds the ability of Elastic Agent to perceive the status of the training process on other remote nodes. Here, for a certain Elastic Agent, the local node refers to the node where the Elastic Agent is located, and the remote node refers to the other nodes other than the local node among the multiple nodes participating in the distributed training task. Once the training process of a node in the entire distributed task exits abnormally due to a fault, the Elastic Agents of other nodes can quickly know and immediately enter the fault recovery process.

[0053] It should be understood that in a distributed training task, when an Elastic Agent senses that the training process of a local node is abnormal, it will restart the training process of the local node and start re-networking after the restart is complete. Re-networking means that the Elastic Agent processes of all nodes need to join the process group, but other nodes have not received notifications to enter the fault recovery process. These nodes need to wait until they detect that a node has initiated a new round of networking before they can start the fault recovery process. In this way, the efficiency of fault recovery will be greatly reduced. The present invention adds the state perception capability of the remote node training process to the new Elastic Agent. When a training process on one of the nodes fails, all other nodes can quickly perceive it, so that all nodes can enter the fault recovery process at the same time, thereby improving the efficiency of fault recovery.

[0054] 2) Added the ability for Elastic Agent to sense remote node failures (such as node crashes or node disconnection). Once a node in the entire distributed task crashes, the Elastic Agent on other nodes can be quickly notified and enter the fault recovery process. It should be understood that Elastic Agent runs on the node. If the node it is on crashes, the Elastic Agent process will no longer exist. Therefore, it cannot sense the crash of the node it is on, but it can sense the crash of other nodes (i.e. remote nodes).

[0055] 3) Added the ability of Elastic Agent to perceive hardware failures. Elastic Agent can obtain the hardware health status of the node by integrating hardware failure detection tools or linking with the platform. Here, the hardware health status of a node refers to the operating status of all key hardware devices on the node (such as CPU, GPU, memory, hard disk, network interface, etc.). It should be understood that in a distributed training task, if there is a hardware failure in a certain node, and the failure is not discovered and handled in time, then even if the training process is restarted, the failure is likely to occur again, resulting in frequent interruptions of the training task. By adding the ability of Elastic Agent to perceive hardware failures, the present invention can identify potential hardware problems before restarting the process, thereby avoiding the occurrence of repeated failures.

[0056] 2. Pod Rescheduling In distributed training tasks, usually all Pods are organized into a PodGroup and scheduled uniformly to ensure the atomicity of resource scheduling. Here, ensuring the atomicity of resource scheduling means ensuring the consistency and integrity of the resource scheduling process, that is, the resources required by all Pods must remain consistent during the scheduling process. When a Pod exits abnormally, the training task will be set to fail, or the PodGroup of the entire task will be rescheduled, that is, all Pods of the entire task will be scheduled. However, distributed training task anomalies are often only due to business failures or hardware failures of some of the nodes, so there is no need to reschedule the entire PodGroup, but only the Pods on the faulty nodes need to be rescheduled. Here, business failures refer to pure business logic failures, that is, failures caused by software. In order to realize the function of rescheduling only the Pods on the faulty nodes, the present invention designs a Pod rescheduling mechanism based on exit codes, and the nodes and platforms coordinate subsequent operations through exit codes. The definition of exit codes is as follows: 1) 0: indicates that the Pod exits successfully, that is, the task is completed successfully, and the platform sets the task as successful; 2) Agreed fault code: indicates that the node where the Pod is located fails. The platform needs to identify the node as a faulty node and isolate it, and migrate the Pod on the node to a healthy node. 3) Other exit codes: Indicates that the Pod failed to exit and the platform sets the task to failure.

[0057] It is understandable that the exit code refers to a numerical value returned after the command execution is completed, which is used to indicate the execution status of the command. In distributed training tasks, when Pod scheduling is performed, the exit code is used to indicate the execution result of the Pod. The agreed fault code refers to a set of exit codes pre-defined to indicate a specific type of error or failure. The embodiment of the present invention uses an agreed fault code to indicate that the node where the Pod is located has a failure. For example, when a Pod exits abnormally due to a failure of the node where it is located, a specific exit code (i.e., the agreed fault code) can be returned so that the platform can recognize that this is a Pod exit caused by a node failure, and take corresponding measures, such as migrating the Pod to a healthy node and isolating the faulty node.

[0058] 3. Hardware Fault Detection The node notifies the platform to execute the corresponding fault handling logic through the exit code. If the training task exits only due to a business failure and the hardware is not faulty, the Pod does not need to be rescheduled and the training process can be restarted; if there is a hardware failure, you can try to quickly repair it. If it cannot be repaired, the platform is notified to perform fault migration through the specified exit code (i.e., the agreed fault code). Therefore, in order to make correct fault handling decisions, the Elastic Agent on the node needs to be able to perceive hardware failures. The specific implementation of the fault perception capability is not specifically limited here. The platform can monitor the health status of the hardware and notify the Elastic Agent on the node, or the Elastic Agent can be integrated with a hardware fault detection tool to achieve it. It should be understood that the types of failures in the distributed training process can include business failures, GPU failures, node failures, etc., among which GPU failures refer to failures of the GPU chip, and node failures refer to downtime or disconnection failures of the node. For GPU failures, some failures can be repaired by some simple commands. After the repair is successful, the training process can be directly restarted; some failures are more serious and cannot be repaired, and Pod rescheduling is required at this time.

[0059] Based on the above designs, the present invention proposes an efficient distributed training fault recovery method. The specific process of implementing this method will be introduced in detail below.

[0060] Figure 2 It is one of the flowcharts of the fault recovery method provided by the present invention, such as Figure 2 As shown, the method is applied to a node, and the node is used to perform a training subtask of distributed training of an artificial intelligence model. The method includes: Step 210, when a training process abnormality is detected or a remote node failure notification is received, the local training process corresponding to the training subtask being executed is stopped, and the health status of the local hardware is checked to obtain the inspection result.

[0061] Specifically, the method provided by the embodiment of the present invention can be applied to each node participating in the distributed training task of the model. An Elastic Agent process is deployed on each node, and during the model training process, it monitors all training processes on the node. Assuming that node A is any node participating in the distributed training task, for node A, when the Elastic Agent of the node detects that one or more local training processes are abnormal, the Elastic Agent of other nodes in the distributed training task can perceive the abnormal event of the training process through the shared storage mechanism, so that the Elastic Agent on all nodes can synchronously execute the subsequent fault recovery process. Here, one or more local training processes refer to one or more training processes on node A.

[0062] Similarly, when the Elastic Agent of node A senses the abnormality of the remote training process through the shared storage mechanism, the Elastic Agent of the node will immediately enter the fault recovery process. Here, the remote training process refers to the training process on other nodes except node A in the distributed training task.

[0063] In addition, when the platform senses that a node has crashed, it can notify the Elastic Agents on other nodes in the distributed training task through a pre-set protocol so that they can receive remote node failure notifications, allowing the Elastic Agents on these nodes to synchronously execute subsequent failure recovery processes.

[0064] For each node, after entering the fault recovery process, first stop the local training process through Elastic Agent, and then check the health status of the local hardware. Here, the local training process refers to all training processes on the node where the ElasticAgent is located. The local hardware refers to the key hardware devices on the node where the Elastic Agent is located.

[0065] It is understandable that in distributed training tasks, each node is usually in an independent but collaborative state. Traditionally, if a node fails, other nodes cannot know it immediately. The embodiment of the present invention introduces remote node fault perception capabilities, so that once a node fails, the platform can immediately send a fault notification to other nodes. This instant notification mechanism greatly shortens the fault discovery time and wins a valuable time window for subsequent recovery operations, thereby improving fault recovery efficiency.

[0066] Step 220, based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

[0067] Specifically, the inspection result refers to the result obtained after checking the health status of the local hardware. Different fault handling decisions can be executed according to the inspection result. Specifically, when the hardware health status check result of a node is healthy, it indicates that the hardware equipment on the node is running normally, and there is no need to replace the node at this time; when the hardware health status check result of a node is a hardware failure, it indicates that the hardware equipment on the node is faulty. At this time, you can try to repair it quickly. If the repair is successful, there is no need to replace the node. If the repair fails or the fault cannot be repaired, you can exit according to the agreed fault code to notify the platform to isolate the faulty node and reschedule the Pod on the faulty node to a healthy node.

[0068] After all nodes have completed fault handling, the Elastic Agents on these nodes will dynamically form a network. After the network is successfully formed, all training processes on the nodes will be started. These training processes will load the most recently saved breakpoints to resume the execution of the training subtasks, thereby resuming model training.

[0069] The method provided by the embodiment of the present invention is that when the node detects an abnormality in the training process or receives a remote node fault notification, it immediately enters the fault recovery process, that is, stops the local training process, and checks the health status of the local hardware. This immediate response can minimize the impact of the fault on the overall training progress and quickly try to restore the training, thereby saving time and computing resources. By checking the health status of the local hardware, potential hardware problems can be identified before restarting the training process, thereby avoiding the occurrence of repeated failures and improving the stability and reliability of model training. Dynamic networking is performed according to the hardware inspection results, and the local training process is restarted after the networking is successful, which can ensure that the training task continues to be executed in the new or adjusted network environment. In addition, the present invention introduces remote node fault perception capabilities. Once a node in a distributed training task fails, other nodes can immediately receive notifications and quickly enter the fault recovery process, thereby greatly shortening the fault discovery time and improving fault recovery efficiency.

[0070] Based on any of the above embodiments, in step 220, the performing dynamic networking based on the inspection result includes: Step 221, when the inspection result is healthy, directly perform dynamic networking; Step 222: When the inspection result indicates that a hardware fault exists, dynamic networking is performed based on the repair type of the hardware fault.

[0071] It should be noted that the traditional fault recovery method usually directly restarts the abnormal training process after the training process is abnormal, while ignoring the detection of hardware failures. This method will cause the restart process to fail when a hardware failure exists, thereby extending the fault recovery time. In this regard, the embodiment of the present invention introduces a hardware health status check, which can immediately perform a hardware check when an abnormal training process is detected. Once a problem is found, corresponding measures can be taken quickly, for example, quickly repairing the fault, or rescheduling the Pod on the faulty node to a healthy node, thereby greatly shortening the fault recovery time and improving training efficiency.

[0072] Specifically, when the hardware health status check result of a node is healthy, it indicates that the hardware device on the node is running normally. At this time, there is no need to replace the node, and the Elastic Agent of the node can directly enter the dynamic networking process.

[0073] When the hardware health status check result of a node is a hardware failure, it indicates that there is an abnormality in the hardware device on the node, causing the device to fail to work properly. At this time, if the training process on the node is directly restarted, it may cause the process to restart. Failure. In this case, you can further decide what kind of fault handling strategy to adopt based on the repair type of the hardware fault. Here, the repair type refers to the way or classification of handling hardware faults, which can be divided into repairable and unrepairable. Among them, repairable means that the fault can be quickly repaired by the corresponding command, and unrepairable means that the fault cannot be repaired by the command. It should be understood that for a hardware fault on a node, when determining the repair type of the hardware fault, it can be based on a pre-set rule judgment, or it can be obtained by classification through a machine learning model. The embodiment of the present invention does not specifically limit this.

[0074] The method provided by the embodiment of the present invention can effectively avoid task-level rescheduling. When there is no hardware fault or the hardware fault is repairable, the corresponding Pod does not need to be rescheduled, thereby avoiding the impact of large-scale Pod rescheduling on the system.

[0075] Based on any of the above embodiments, in step 222, the dynamic networking based on the repair type of the hardware fault includes: Step 2221: When the repair type of the hardware fault is repairable, repair the hardware fault, and perform dynamic networking based on the repair result; Step 2222, when the repair type of the hardware fault is unrepairable, exit with a predetermined fault code, wherein the predetermined fault code is used to instruct the platform to identify the node as a faulty node for isolation, and reschedule the Pod on the node to a healthy node.

[0076] Specifically, for a node, when a hardware fault is detected, if the hardware fault can be repaired, you can try to quickly repair it through the corresponding command, and decide which treatment measures to take next based on the repair results. For example, if the fault is repaired successfully, it means that the hardware health status on the node has returned to normal. At this time, the node can continue to participate in the model training task, so the Elastic Agent on the node can enter the dynamic networking process. If the repair fails, it means that the hardware device on the node is still abnormal and cannot work normally. At this time, the node needs to be replaced.

[0077] If the repair type of the hardware fault on the node is unrepairable, it means that the hardware fault cannot be repaired quickly and the node needs to be replaced. Specifically, the Pod on the node can exit according to the agreed fault code. When the platform finds that a Pod on a node exits with an agreed fault code, it will mark the node indicated by the agreed fault code as a faulty node and isolate it, and reschedule the Pod on the faulty node to a healthy node.

[0078] It can be understood that isolating a faulty node includes two meanings: the first meaning is that a training task is executed by multiple nodes. When there is a faulty node among the multiple nodes, the faulty node is removed from the multiple nodes so that the faulty node is no longer used to execute the corresponding training subtask; the second meaning is that after the fault is isolated and before the fault is recovered, the faulty node will not be used to execute any training subtask of any training task.

[0079] Figure 3 Schematic diagram of Pod rescheduling provided by the present invention, such as Figure 3As shown in the figure, the platform is equipped with training frameworks (such as Pytorch, Tensorflow, etc.), MPI (Message Passing Interface), and controllers for controlling and managing Pods, services, and nodes respectively. Here, MPI supports parallel computing on a large number of nodes and can be used to run distributed model training tasks that require high-performance computing. The controllers include Pod Control, Service Control, and Node Control. Among them, Pod Control is used to manage the creation, update, deletion, scheduling, and health check of Pods. For example, the number of Pods can be increased or decreased as needed, and Pods can be rescheduled as needed. Service Control is used to manage the creation, update, deletion, and load balancing strategies of services. Requests can be distributed to multiple Pods on the backend to achieve high availability and load balancing. In distributed training, services serve as a bridge for model training tasks to communicate with the outside world, allowing users to access a set of applications running on multiple Pods. Node Control is used to manage operations such as the creation, update, deletion, status monitoring, fault isolation, and fault recovery of nodes.

[0080] like Figure 3 As shown in the figure, in the distributed training of the model, when there is a hardware fault on Node A and the fault is unrepairable or fails to be repaired, Pod1 on Node A can exit according to the agreed fault code. When Pod Control on the platform detects that Pod1 on Node A exits with the agreed fault code, Node Control on the platform will mark Node A as a faulty node and isolate Node A. At the same time, Pod Control will reschedule Pod1 on Node A to the healthy Node B, so that Node B replaces Node A to perform the corresponding training subtask.

[0081] Based on any of the above embodiments, in step 2221, the dynamic networking based on the repair result includes: When the repair result is successful, dynamic networking is performed; When the repair result is a repair failure, the system exits with the agreed fault code.

[0082] Specifically, when a hardware fault is detected on a node, you can try to repair the fault. If the repair is successful, it means that there is no hardware fault on the node, that is, the hardware device on the node resumes normal operation, and the Elastic Agent of the node can enter the dynamic networking process. If the fault repair fails, it means that the hardware fault on the node still exists, that is, the hardware device cannot work properly. In this case, the Pod on the node can exit according to the agreed fault code to notify the platform to reschedule the Pod on the node.

[0083] In the embodiment of the present invention, the Pod of the node with the hardware fault will be rescheduled only when the hardware fault is unrepairable or the repair fails. This Pod-level fault recovery method makes the fault recovery difficulty or cost of large-scale distributed training unaffected by the task scale, avoids large-scale node migration, and is conducive to improving the efficiency of fault recovery.

[0084] Based on any of the above embodiments, the training process abnormality includes any one of a local training process abnormality and a remote training process abnormality.

[0085] It should be noted that in a distributed training environment, each node is usually independent of each other but works together. Traditionally, if the training process on a node exits abnormally, other nodes cannot know it immediately. In this regard, the embodiment of the present invention introduces the ability to perceive abnormalities in remote training processes, so that once an abnormality occurs in the training process on a node, other nodes can immediately perceive it. This instant perception mechanism greatly shortens the time to discover faults.

[0086] Specifically, for the Elastic Agent of a certain node, whether it detects an abnormality in the local training process, detects an abnormality in the remote training process, or receives a remote node failure notification, it will immediately enter the subsequent fault recovery process, thereby shortening the fault discovery time and improving the fault recovery efficiency.

[0087] Based on any of the above embodiments, Figure 4 FIG. 2 is a flow chart of the fault recovery method provided by the present invention. Figure 4 As shown, the method is applied to a platform, the platform is associated with a computing resource pool, the computing resource pool includes multiple nodes for distributed training of artificial intelligence models, each of the multiple nodes is used to perform a training subtask of the distributed training of the artificial intelligence model, and the method includes: Step 410: When a fault is detected in a first node, the first node is identified as a faulty node for isolation, and the first node is a node in which the fault occurs among the multiple nodes.

[0088] Specifically, the method provided in the embodiment of the present invention can be applied to the platform, and the platform can perform fault monitoring on each node participating in the training task, and after detecting that the node has a downtime, isolate the first node that has failed. Here, the first node refers to a type of node that has failed, which can be one or more nodes.

[0089] Step 420, sending a remote node fault notification to the second node, where the second node is a node that has not failed among the multiple nodes, and the remote node fault notification is used to instruct the second node to stop executing the local training process corresponding to the training subtask, and check the health status of the local hardware. The second node is used to perform dynamic networking based on the inspection results, and after the networking is successful, restart the local training process to resume the execution of the training subtask.

[0090] Specifically, after detecting that a node has crashed, the platform will send a remote node failure notification to the second node that has not failed, in order to notify the second node to stop executing the training subtask, that is, to notify the second node to stop the local training process. Here, the second node refers to a type of node that has not crashed, which can be one or more nodes, and the second node and the first node are used to execute the same training task.

[0091] After receiving the remote node failure notification, the second node will first stop the local training process and check the health status of the local hardware. If there is no hardware failure in the local node, the Elastic Agent of the node can perform dynamic networking; if there is a hardware failure in the local node, you can first try to quickly repair the failure. If the repair fails or the failure cannot be quickly repaired, you can exit according to the agreed fault code to notify the platform to reschedule the Pod on the node.

[0092] Step 430: reschedule the Pod on the first node to a third node, where the third node is a healthy node in the computing resource pool other than the multiple nodes, and the third node is used to replace the first node to execute the training subtask.

[0093] Specifically, when the platform detects that the first node has a downtime failure, in order to ensure the smooth progress of the training task, the Pod on the first node needs to be rescheduled to the third node so that the third node can replace the first node to perform the corresponding training subtask. Here, the third node refers to a type of node used to replace the first node, which can be one or more nodes. The third node is used to execute the training subtask originally performed by the first node, and the third node can be a healthy node in the computing resource pool that is not used to execute the training task.

[0094] After the Pod on the first node is rescheduled to the third node, the Elastic Agent on the third node can also be dynamically networked. After the Elastic Agent on all nodes is successfully networked, all training processes can be started. These training processes will load the most recently saved breakpoints to resume the execution of the training subtasks.

[0095] The method provided by the embodiment of the present invention can enable the node that has not experienced a downtime to receive the notification in the first place and enter the subsequent fault recovery process by sending a remote node fault notification to the second node. This instant notification mechanism greatly shortens the time to discover the fault, and wins a valuable time window for subsequent recovery operations, thereby improving the efficiency of fault recovery. By rescheduling the Pod on the first node to the third node, only the Pod on the node where the fault occurs is rescheduled, thereby effectively avoiding task-level rescheduling, so that the difficulty and cost of fault recovery are not affected by the task scale.

[0096] Based on any of the above embodiments, when any one of the multiple nodes detects that the local training process corresponding to the executed training subtask is abnormal, the node sends a training process abnormality notification to other nodes among the multiple nodes except the node, and the training process abnormality notification is used to instruct the other nodes to stop the local training process corresponding to the executed training subtask and check the health status of the local hardware.

[0097] It should be noted that, considering that the Elastic Agent on each node can only perceive the status of the training process on the local node, it cannot perceive the abnormal training process on other nodes, which will greatly extend the time for fault discovery and reduce the efficiency of subsequent fault recovery. In this regard, the embodiment of the present invention adds the ability of Elastic Agent to perceive the status of training processes on other remote nodes. Once the training process of a node in the entire distributed task exits due to an abnormal failure, the Elastic Agent of other nodes can quickly know and immediately enter the fault recovery process.

[0098] Specifically, for any node, when the Elastic Agent of the node detects that the local training process exits abnormally, it will report the abnormal event and enable other nodes to perceive the abnormal event through the shared storage mechanism so that other nodes can enter the fault recovery process. Subsequently, these nodes will stop the local training process corresponding to the executed training subtask and check the health status of the local hardware so as to adopt different fault handling strategies according to the inspection results.

[0099] Based on any of the above embodiments, Figure 5It is a flow chart of the fault recovery method based on Pod rescheduling provided by the present invention, such as Figure 5 As shown, the method includes: Step S1: Fault discovery: Elastic Agent or the platform first discovers the fault.

[0100] a) Training process abnormality. When a hardware failure or business failure occurs on a node, these failures will be fed back to the training process, causing the training process to exit abnormally. At this time, the Elastic Agent on the node will immediately perceive the local training process abnormality and notify the Elastic Agents of other nodes.

[0101] b) Node downtime. When one or more nodes fail, the platform will sense the node downtime event, and the Elastic Agent on other nodes of the distributed training task can also sense the failure of the remote node through a pre-set protocol.

[0102] Step S2: Fault handling: After the Elastic Agent or platform detects the fault, it executes the subsequent fault handling process.

[0103] a) When the Elastic Agent of a node senses a fault (including any of the following: abnormal local training process, abnormal remote training process, or remote node downtime), it first stops the local training process and then checks the hardware health status. If the hardware is not faulty, there is no need to replace the node. If the hardware is faulty, you can try to repair it first. If the repair fails or the fault cannot be repaired quickly, it exits according to the agreed fault code to notify the platform to migrate the Pod.

[0104] b) When the platform senses that a node is down or finds that the Pod on the node has exited with an agreed fault code, it marks the corresponding node as a faulty node for isolation and reschedules the Pod on the faulty node to a healthy node.

[0105] Step S3, resume training. The Elastic Agents on all nodes perform dynamic networking. After the networking is successful, the training process is started, and the training process loads the breakpoint to resume training.

[0106] The method provided by the embodiment of the present invention enables efficient recovery when a failure occurs in model distributed training. The method specifically includes the following advantages: 1) It can effectively avoid task-level rescheduling. If the hardware is fault-free or can be quickly repaired, the corresponding Pod does not need to be rescheduled, thus avoiding the impact of large-scale task rescheduling on various modules of the system; 2) It can improve the efficiency of fault recovery. Large-scale task rescheduling, involving PodGroup rescheduling, resource release and application, image pulling, etc., often takes several minutes or even more than ten minutes. The present invention can effectively optimize this time consumption; 3) Pod-level fault recovery makes the difficulty or cost of fault recovery in large-scale distributed training unaffected by the task scale, and can support fault recovery of larger-scale distributed training tasks, with the cost or difficulty remaining basically constant.

[0107] 4) Improves the stability of the training task environment and avoids large-scale migration of nodes. Since there is no need to reschedule all Pods, the original topology structure will not be destroyed, thereby avoiding fluctuations in training indicators before and after failures.

[0108] Based on any of the above embodiments, Figure 6 is one of the structural diagrams of the fault recovery device provided by the present invention, such as Figure 6 As shown, the device is applied to a node, and the node is used to perform a training subtask of distributed training of an artificial intelligence model. The device includes: The checking unit 610 is used to stop the local training process corresponding to the executed training subtask and check the health status of the local hardware to obtain the inspection result when detecting that the training process is abnormal or receiving the remote node failure notification; The recovery unit 620 is used to perform dynamic networking based on the inspection result, and restart the local training process after the networking is successful to resume the execution of the training subtask.

[0109] The device provided by the embodiment of the present invention, when the node detects an abnormality in the training process or receives a remote node fault notification, immediately enters the fault recovery process, that is, stops the local training process, and checks the health status of the local hardware. This immediate response can minimize the impact of the fault on the overall training progress and quickly try to restore the training, thereby saving time and computing resources. By checking the health status of the local hardware, potential hardware problems can be identified before restarting the training process, thereby avoiding the occurrence of repeated failures and improving the stability and reliability of model training. Dynamic networking is performed according to the hardware inspection results, and the local training process is restarted after the networking is successful, which can ensure that the training task continues to be executed in the new or adjusted network environment. In addition, the present invention introduces remote node fault perception capabilities. Once a node in a distributed training task fails, other nodes can immediately receive notifications and quickly enter the fault recovery process, thereby greatly shortening the fault discovery time and improving the fault recovery efficiency.

[0110] Based on any of the above embodiments, the recovery unit 620 includes: A first networking subunit is used to directly perform dynamic networking when the inspection result is healthy; The second networking subunit is used to perform dynamic networking based on the repair type of the hardware fault when the inspection result shows that there is a hardware fault.

[0111] Based on any of the above embodiments, the second group of network subunits includes: A repair subunit, configured to repair the hardware fault if the repair type of the hardware fault is repairable, and to perform dynamic networking based on the repair result; The exit subunit is used to exit with a predetermined fault code when the repair type of the hardware fault is irreparable. The predetermined fault code is used to instruct the platform to identify the node as a faulty node for isolation and reschedule the Pod on the node to a healthy node.

[0112] Based on any of the above embodiments, the repair subunit is specifically used for: When the repair result is successful, dynamic networking is performed; When the repair result is a repair failure, the system exits with the agreed fault code.

[0113] Based on any of the above embodiments, the training process abnormality includes any one of a local training process abnormality and a remote training process abnormality.

[0114] Based on any of the above embodiments, Figure 7 This is the second structural diagram of the fault recovery device provided by the present invention, such as Figure 7 As shown, the device is applied to a platform, the platform is associated with a computing resource pool, the computing resource pool includes multiple nodes for distributed training of artificial intelligence models, each of the multiple nodes is used to perform a training subtask of the distributed training of the artificial intelligence model, and the device includes: The isolation unit 710 is configured to, when a fault occurs on a first node, identify the first node as a faulty node for isolation, wherein the first node is a node on which the fault occurs among the multiple nodes; A sending unit 720 is used to send a remote node fault notification to a second node, where the second node is a node that has not failed among the multiple nodes, and the remote node fault notification is used to instruct the second node to stop executing a local training process corresponding to the training subtask, and check the health status of the local hardware, and the second node is used to perform dynamic networking based on the inspection result, and restart the local training process after the networking is successful to resume the execution of the training subtask; The rescheduling unit 730 is used to reschedule the Pod on the first node to a third node, where the third node is a healthy node in the computing resource pool other than the multiple nodes, and the third node is used to replace the first node to execute the training subtask.

[0115] The device provided by the embodiment of the present invention can send a remote node fault notification to the second node, so that the node that has not experienced a downtime fault can receive the notification in the first time and enter the subsequent fault recovery process. This instant notification mechanism greatly shortens the fault discovery time and wins a valuable time window for subsequent recovery operations, thereby improving the fault recovery efficiency. By rescheduling the Pod on the first node to the third node, only the Pod on the node where the fault occurs is rescheduled, thereby effectively avoiding task-level rescheduling, so that the difficulty and cost of fault recovery are not affected by the task scale.

[0116] Based on any of the above embodiments, when any one of the multiple nodes detects that the local training process corresponding to the executed training subtask is abnormal, the node sends a training process abnormality notification to other nodes among the multiple nodes except the node, and the training process abnormality notification is used to instruct the other nodes to stop the local training process corresponding to the executed training subtask and check the health status of the local hardware.

[0117] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a fault recovery method, which is applied to a node, and the node is used to execute a training subtask of distributed training of an artificial intelligence model. The method includes: when an abnormality in the training process is detected or a remote node fault notification is received, the local training process corresponding to the executed training subtask is stopped, and the health status of the local hardware is checked to obtain the inspection result; based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

[0118] The processor 810 can call the logic instructions in the memory 830 to execute a fault recovery method, which is applied to a platform, and the platform is associated with a computing resource pool, and the computing resource pool includes multiple nodes for distributed training of artificial intelligence models, and each of the multiple nodes is used to execute a training subtask of the distributed training of the artificial intelligence model. The method includes: when a fault is detected in a first node, the first node is identified as a faulty node for isolation, and the first node is a node that has failed among the multiple nodes; a remote node fault notification is sent to a second node, and the second node is a node that has not failed among the multiple nodes. The remote node fault notification is used to instruct the second node to stop the local training process corresponding to the training subtask executed, and check the health status of the local hardware. The second node is used to perform dynamic networking based on the inspection results, and after the networking is successful, restart the local training process to resume the execution of the training subtask; reschedule the Pod on the first node to a third node, and the third node is a healthy node other than the multiple nodes in the computing resource pool, and the third node is used to replace the first node to execute the training subtask.

[0119] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fault recovery method provided by the above methods. The method is applied to a node, and the node is used to execute a training subtask of distributed training of an artificial intelligence model. The method includes: when an abnormality in the training process is detected or a remote node fault notification is received, the local training process corresponding to the executed training subtask is stopped, and the health status of the local hardware is checked to obtain the inspection result; based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

[0121] In addition, when the computer program is executed by the processor, the computer can execute the fault recovery method provided by the above methods, which is applied to a platform, and the platform is associated with a computing resource pool, and the computing resource pool includes multiple nodes for distributed training of artificial intelligence models, and each of the multiple nodes is used to execute a training subtask of the distributed training of the artificial intelligence model. The method includes: when a failure of a first node is detected, the first node is identified as a faulty node for isolation, and the first node is a node that has failed among the multiple nodes; a remote node fault notification is sent to a second node, and the second node is a node that has not failed among the multiple nodes, and the remote node fault notification is used to instruct the second node to stop the local training process corresponding to the training subtask executed, and check the health status of the local hardware, and the second node is used to perform dynamic networking based on the inspection results, and after the networking is successful, restart the local training process to resume the execution of the training subtask; reschedule the Pod on the first node to a third node, and the third node is a healthy node other than the multiple nodes in the computing resource pool, and the third node is used to replace the first node to execute the training subtask.

[0122] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fault recovery method provided by the above-mentioned methods. The method is applied to a node, and the node is used to execute a training subtask of distributed training of an artificial intelligence model. The method includes: when an abnormality in the training process is detected or a remote node fault notification is received, the local training process corresponding to the executed training subtask is stopped, and the health status of the local hardware is checked to obtain the inspection result; based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

[0123] In addition, when the computer program is executed by the processor, it is implemented to execute the fault recovery method provided by the above methods. The method is applied to a platform, and the platform is associated with a computing resource pool. The computing resource pool includes multiple nodes for distributed training of artificial intelligence models, and each of the multiple nodes is used to execute a training subtask of the distributed training of the artificial intelligence model. The method includes: when a failure of a first node is detected, the first node is identified as a faulty node for isolation, and the first node is a node that has failed among the multiple nodes; a remote node fault notification is sent to a second node, and the second node is a node that has not failed among the multiple nodes. The remote node fault notification is used to instruct the second node to stop the local training process corresponding to the training subtask executed, and check the health status of the local hardware. The second node is used to perform dynamic networking based on the inspection results, and after the networking is successful, restart the local training process to resume the execution of the training subtask; the Pod on the first node is rescheduled to a third node, and the third node is a healthy node other than the multiple nodes in the computing resource pool, and the third node is used to replace the first node to execute the training subtask.

[0124] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault recovery method, characterized in that: The method is applied to a node, and the node is used to perform a training subtask of distributed training of an artificial intelligence model, and the method includes: When a training process abnormality is detected or a remote node failure notification is received, the local training process corresponding to the executed training subtask is stopped, and the health status of the local hardware is checked to obtain the inspection result; Based on the inspection result, dynamic networking is performed, and after the networking is successful, the local training process is restarted to resume the execution of the training subtask.

2. The fault recovery method according to claim 1, characterized in that: The performing dynamic networking based on the inspection result includes: If the inspection result is healthy, dynamic networking is directly performed; When the inspection result indicates that a hardware fault exists, dynamic networking is performed based on a repair type of the hardware fault.

3. The fault recovery method according to claim 2, characterized in that: The performing of dynamic networking based on the repair type of the hardware fault includes: If the repair type of the hardware fault is repairable, repair the hardware fault and perform dynamic networking based on the repair result; When the repair type of the hardware fault is unrepairable, the system exits with a predetermined fault code, which is used to instruct the platform to identify the node as a faulty node for isolation and reschedule the Pod on the node to a healthy node.

4. The fault recovery method according to claim 3, characterized in that: The dynamic networking based on the repair result includes: When the repair result is successful, dynamic networking is performed; When the repair result is a repair failure, the system exits with the agreed fault code.

5. The fault recovery method according to any one of claims 1 to 4, characterized in that: The training process abnormality includes any one of a local training process abnormality and a remote training process abnormality.

6. A fault recovery method, characterized in that: The method is applied to a platform, the platform is associated with a computing resource pool, the computing resource pool includes a plurality of nodes for distributed training of an artificial intelligence model, each of the plurality of nodes is used to perform a training subtask of the distributed training of the artificial intelligence model, and the method includes: In the case of detecting that a first node fails, marking the first node as a failed node for isolation, the first node being a failed node among the multiple nodes; Sending a remote node failure notification to a second node, where the second node is a node that has not failed among the multiple nodes, the remote node failure notification is used to instruct the second node to stop executing a local training process corresponding to the training subtask, and to check the health status of the local hardware, the second node is used to perform dynamic networking based on the inspection result, and after the networking is successful, restart the local training process to resume the execution of the training subtask; The Pod on the first node is rescheduled to a third node, where the third node is a healthy node in the computing resource pool other than the multiple nodes, and the third node is used to replace the first node to execute the training subtask.

7. The fault recovery method according to claim 6, characterized in that: When any one of the multiple nodes detects that the local training process corresponding to the executed training subtask is abnormal, the node sends a training process abnormality notification to other nodes among the multiple nodes except the node, and the training process abnormality notification is used to instruct the other nodes to stop the local training process corresponding to the executed training subtask and check the health status of the local hardware.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the fault recovery method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fault recovery method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the fault recovery method according to any one of claims 1 to 7 is implemented.

Images