Fault processing method and related device

Abstract

Provided in the present application is a fault processing method, which is applied to a training system, wherein the training system comprises a first chip on a host side and a plurality of second chips on a device side, the first chip and the plurality of second chips are used for collaboratively executing a training task, the training task comprises a first sub-task and a plurality of second sub-tasks, and the execution of the second sub-tasks depend on an execution result of the first sub-task. The method comprises: a first chip executing a first sub-task; when a fault occurs, the first chip saving a faulty file before a second chip in a normal state on a device side stops executing a second sub-task; and the first chip synchronizing the faulty file to a rescheduled chip on the device side, such that the rescheduled chip continues executing the second sub-task. In the method, a first chip on a host side does not have to stop executing a second sub-task. In this way, an execution result of the first sub-task on the host side can be reused to restore a training task, thereby shortening the time for restoring the training task, and thus improving the efficiency of restoring the training task.

Description

A fault handling method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 28, 2023, with application number 202310784685.2, entitled “A Fault Recovery Method for Distributed Model Training”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 28, 2023, with application number 202311281352.4, entitled “A Fault Handling Method and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a fault handling method, device, scheduler, chip, computer-readable storage medium, and computer program product. Background Art

[0003] With the continuous development of AI technology, more and more industries and fields are adopting AI models (for ease of description, sometimes referred to simply as models) to achieve intelligent and automated business operations. For example, in the e-commerce industry, more and more businesses are using AI customer service built based on AI models to replace human customer service to provide pre-sales and after-sales consulting services. Another example is that in social networks, platforms are using AI models to replace manual review of user content, saving labor costs.

[0004] An AI model is a mathematical model built using AI technology and used to make predictions about unknown data. For example, an AI model could be an object detection model or an image classification model built using a neural network. AI models typically require large amounts of data to train. To improve AI model training efficiency, distributed training methods have emerged.

[0005] The distributed training method is to distribute the training tasks to multiple training nodes for execution, and multiple training nodes train the model in parallel. Among them, executing the training task is the process of using the data set to train the model and obtain the parameters of the model (such as weights). Multiple training nodes can use a synchronous update mechanism to update the parameters of the model. The synchronous update mechanism refers to accumulating the gradients obtained by each training node to calculate the mean, and updating the parameters of the model based on the mean. When an individual training node, training algorithm or network fails, the entire distributed training task will be interrupted. As the number of training nodes increases, the possibility of interruption becomes higher and higher. Therefore, it is necessary to provide a fault handling mechanism to resume the training task.

[0006] Currently, the mainstream fault handling solutions in the industry take a long time to restore training tasks, are inefficient, and are difficult to meet business needs.

[0007] Summary of the Invention

[0008] The present application provides a fault handling method. This method supports the second chip on the device side stopping the execution of the second subtask when a fault occurs, and the first chip on the host side can continue to execute the second subtask. In this way, the execution result of the first subtask on the host side can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task. The present application also provides a device, scheduler, chip, training system, computer-readable storage medium and computer program product corresponding to the above-mentioned processing method.

[0009] In a first aspect, the present application provides a fault handling method. The method is applied to a training system. The training system includes a first chip on a host side and multiple second chips on a device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtasks depends on the execution results of the first subtask.

[0010] Specifically, the first chip executes the first subtask. When a fault occurs, the first chip saves the fault file before the second chip, which is in a normal state on the device side, stops executing the second subtask. The first chip then synchronizes the fault file with the rescheduled chip on the device side, allowing the rescheduled chip to continue executing the second subtask.

[0011] In this method, when a fault occurs, the second chip on the device side can stop executing the second subtask, while the first chip on the host side can continue executing the first subtask. This allows the execution results of the first subtask on the host side (for example, compilation results, including but not limited to the compilation results of the computation graph) to be reused to resume the training task, shortening the time it takes to resume the training task and improving its efficiency.

[0012] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault. This method supports fine-grained recovery solutions based on different fault types, which can maximize the efficiency of resuming training tasks.

[0013] In some possible implementations, if the fault is a recoverable chip failure, the device prioritizes scheduling the second chip, which has returned to normal after reset. This allows training tasks to be resumed on the original node without requiring additional node resources, improving resource utilization.

[0014] In some possible implementations, the training system includes a training node, each of which includes a first chip and multiple second chips. Specifically, the training system may have a single-machine, multi-card architecture. The fault is a target chip failure among the multiple second chips. Based on this, when the first chip synchronizes the faulty file to a chip that is being scheduled by the device, it can also synchronize the faulty file to the target chip that has returned to a normal state after being reset. This allows the training task to be resumed at the original node when a recoverable chip failure occurs in a single training node.

[0015] In some possible implementations, the failure is an unrecoverable node failure, and the device prioritizes scheduling the chip including the newly added third chip. This allows the training task to be resumed on the new node, minimizing the interruption time of the training task and improving the recovery efficiency of the training task.

[0016] In some possible implementations, the training system includes a first training node and a second training node, each of which includes at least one first chip and at least one second chip. That is, the training system can have a multi-machine, multi-card architecture. If an unrecoverable node failure occurs in the first training node, the third chip becomes part of the newly added third training node. Accordingly, the first chip of the first training node can store the execution results of the first subtask, and the first chip of the third training node can load the execution results of the first subtask.

[0017] This method caches the execution result of the first subtask and then loads the cached execution result on the new node, thereby reusing the execution result, thereby improving the efficiency of resuming the training task.

[0018] In some possible implementations, the first subtask may be a compilation subtask (e.g., compiling a computation graph). The first chip does not stop executing the first subtask, thereby achieving compilation-free execution. Alternatively, the first chip of the first training node caches the compilation result, and the first chip of the newly added third training node loads the compilation result cached by the first chip of the first training node, thereby reusing the compilation result. This shortens the time to resume the training task and improves the efficiency of resuming the training task.

[0019] In some possible implementations, the first chip can retrieve the fault file from memory and synchronize it with the device-focused scheduling chip via a cluster communication interface. This eliminates the need for re-deserialization, shortens synchronization time, and improves fault recovery efficiency.

[0020] In some possible implementations, subtasks are executed as processes or threads. The fault handling method of the present application can achieve the process or thread exit on the device side, while the process or thread on the host side can be kept exited. The host side does not need to reapply resources to repeat the same subtask, further shortening the time to resume training tasks and improving the efficiency of resuming training tasks.

[0021] In a second aspect, the present application provides a fault handling method. The method is applied to a scheduler, which is used to handle faults when a training system fails. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The method includes:

[0022] performing fault detection on the training system;

[0023] When a fault is detected, the first chip is notified to save a fault file before the second chip in normal state on the device side stops executing the second subtask. The fault file is used to be synchronized by the first chip to the chip that is heavily scheduled on the device side.

[0024] When the scheduler detects a fault, this method notifies the first chip to save the fault file before the second chip on the device, which is in a normal state, stops executing the second subtask. The second chip on the device can then stop executing the second subtask, while the first chip on the host can continue executing the second subtask. This allows the execution results of the first subtask on the host to be reused to resume the training task, shortening the time required to resume the training task and improving its efficiency.

[0025] In some possible implementations, if the fault is a recoverable chip fault, the scheduler can also reset the faulty chip on the device side. The device-side prioritized chips for scheduling include a second chip whose state returns to normal after reset. This method supports fine-grained recovery solutions based on different fault types. For recoverable chip faults, an online hot reset can be performed to recover, and the training state can be restored by synchronizing the faulty file with a functioning chip.

[0026] In some possible implementations, a training system includes a training node, each comprising a first chip and multiple second chips. A target chip among the multiple second chips is faulty, and a fault file is synchronized by the first chip to the target chip, which has returned to a normal state after being reset. This allows for the resumption of training tasks at the original node when a recoverable chip fault occurs on a single training node.

[0027] In some possible implementations, the fault is an unrecoverable node failure, and the device prioritizes scheduling the chip, including the newly added third chip. Accordingly, the scheduler can also synchronize information about the newly added third chip with the healthy second chip. This can help establish a link between the second chips, thereby helping to resume training tasks.

[0028] In some possible implementations, a training system includes a first training node and a second training node, each training node including at least one first chip and at least one second chip. If an unrecoverable node failure occurs in the first training node, a scheduler may synchronize information about a third chip to at least one second chip in the second training node. The information about the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

[0029] This method can enable the third chip to resume executing the second subtask as soon as possible by synchronizing the address, resource configuration and other information of the third chip to at least one second chip in the second training node, thereby improving the efficiency of training task recovery.

[0030] In some possible implementations, the subtask is executed as a process or thread. This method allows the process or thread on the device to exit without exiting the process or thread on the host. The host does not need to reapply resources to repeat the same subtask, further shortening the time to resume training tasks and improving the efficiency of resuming training tasks.

[0031] In a third aspect, the present application provides a fault handling device. The device is deployed on a first chip on the host side of a training system, and the first chip and multiple second chips on the device side of the training system are used to collaboratively execute a training task, wherein the training task includes a first subtask and multiple second subtasks, and the execution of the second subtasks depends on the execution result of the first subtask. The device includes:

[0032] A task execution module, configured to execute the first subtask;

[0033] A file saving module is configured to save a fault file when a fault occurs, before the first chip in a normal state on the device side stops executing the second subtask;

[0034] The file synchronization module is used to synchronize the fault file to the chip that is rescheduled by the device, so that the rescheduled chip can continue to execute the second subtask.

[0035] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault.

[0036] In some possible implementations, the fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after reset.

[0037] In some possible implementations, the training system includes a training node, the training node includes a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips;

[0038] The file synchronization module is specifically used for:

[0039] The fault file is synchronized to the target chip whose state has returned to normal after reset.

[0040] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

[0041] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the unrecoverable node failure occurs in the first training node, and the third chip belongs to a newly added third training node, and the apparatus further includes an execution result saving module deployed in the first chip in the first training node and an execution result loading module deployed in the first chip in the third training node;

[0042] The execution result saving module is used to save the execution result of the first subtask;

[0043] The execution result loading module is used to load the execution result of the first subtask.

[0044] In some possible implementations, the file synchronization module is specifically configured to:

[0045] The fault file is obtained from the memory, and the fault file is synchronized with the chip for focusing on scheduling of the device through the cluster communication interface.

[0046] In some possible implementations, the subtask is executed in a process or thread manner.

[0047] In a fourth aspect, the present application provides a scheduler. The scheduler is used to handle a fault in a training system, wherein the training system includes a first chip on a host side and multiple second chips on a device side, wherein the first chip and the multiple second chips are used to collaboratively execute a training task, wherein the training task includes a first subtask and multiple second subtasks, wherein the execution of the second subtask depends on the execution result of the first subtask, and the scheduler includes:

[0048] A fault detection module, configured to perform fault detection on the training system;

[0049] The notification module is used to notify the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask when a fault is detected. The fault file is used to be synchronized by the first chip to the chip that is heavily scheduled on the device side.

[0050] In some possible implementations, the fault is a recoverable chip fault, and the scheduler further includes:

[0051] A reset module, used to reset the faulty chip on the device side;

[0052] The chip that the device focuses on scheduling includes a second chip whose state returns to normal after reset.

[0053] In some possible implementations, the training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to synchronize the first chip to the target chip whose state has returned to normal after reset.

[0054] In some possible implementations, the fault is an unrecoverable node fault, the chip that the device prioritizes scheduling includes a newly added third chip, and the scheduler further includes:

[0055] The synchronization module is used to synchronize information of the newly added third chip with the second chip in normal state.

[0056] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, and the unrecoverable node failure occurs in the first training node;

[0057] The synchronization module is specifically used for:

[0058] The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

[0059] In some possible implementations, the subtask is executed in a process or thread manner.

[0060] In a fifth aspect, the present application provides a chip. The chip includes a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory, so that the chip performs the fault handling method described in the first aspect or any implementation of the first aspect.

[0061] In a sixth aspect, the present application provides a scheduler. The scheduler includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions to cause the scheduler to perform the fault handling method described in the first aspect or any implementation of the first aspect.

[0062] In a seventh aspect, the present application provides a training system. The training system includes a first chip on a host side and multiple second chips on a device side. The first chip is configured to execute computer-readable instructions to perform the fault handling method described in the first aspect or any implementation of the first aspect.

[0063] In an eighth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the fault handling method described in the first aspect or any one of the implementations of the first aspect.

[0064] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when run on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the fault handling method described in the first aspect or any one of the implementations of the first aspect.

[0065] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0067] FIG1A is a schematic diagram of the architecture of a training system provided in an embodiment of the present application;

[0068] FIG1B is a schematic diagram of the architecture of another training system provided in an embodiment of the present application;

[0069] FIG2A is a schematic diagram of a fault handling solution provided in an embodiment of the present application;

[0070] FIG2B is a schematic diagram of another fault handling solution provided in an embodiment of the present application;

[0071] FIG3 is a hardware structure diagram of a server provided in an embodiment of the present application;

[0072] FIG4 is a framework diagram of a server deployment software provided in an embodiment of the present application;

[0073] FIG5 is a flowchart of a fault handling method provided in an embodiment of the present application;

[0074] FIG6 is a flowchart of a fault handling method provided in an embodiment of the present application;

[0075] FIG7 is a schematic diagram of recovering a training task from an original node according to an embodiment of the present application;

[0076] FIG8 is a schematic diagram of resuming a training task from a new node according to an embodiment of the present application;

[0077] FIG9 is a schematic structural diagram of a fault handling device provided in an embodiment of the present application;

[0078] FIG10 is a schematic structural diagram of another fault handling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0080] First, some technical terms involved in the embodiments of this application are introduced.

[0081] Artificial intelligence (AI), also known as machine intelligence, refers to the intelligence exhibited by machines (such as computers) by mimicking human thinking and behavior (such as learning, reasoning, thinking, and planning). AI typically mimics human thinking and behavior based on knowledge to achieve specific goals or complete specific tasks. This knowledge can be derived from experience or data.

[0082] Deep learning (DL), a branch of AI, uses deep neural network models (also called deep learning models, sometimes referred to simply as models for ease of description) to process massive amounts of data. This allows the model to learn knowledge (such as multiple nonlinear transformations) from the data and then process and analyze the data based on this knowledge. Trained deep learning models can be applied to perception and decision-making scenarios in the AI ​​field, such as image recognition, speech recognition, natural language translation, and computer game playing.

[0083] Deep learning models have a high number of parameters, often reaching hundreds of billions or even trillions. For example, large models in the field of natural language processing (NLP) can have hundreds of billions of parameters. These large-scale deep learning models typically require massive datasets for training. A typical training method is distributed training.

[0084] Distributed training can be performed by a training system. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip can be a central processing unit (CPU), also known as a central processing unit. The host side can include not only a CPU, but also host memory. The second chip can be an external processing unit used to assist the CPU in completing tasks and achieve computing acceleration. Therefore, the second chip can also be called an acceleration card or accelerator. The second chip can include a neural processing unit (NPU) or a graphics processing unit (GPU). Different types of processors / processing units can implement different types of instruction set architectures. Similar to the host side, the device side can also include device memory. In some examples, the device memory can be integrated with the second chip on the device side.

[0085] The training system can adopt a single-machine, multi-card architecture or a multi-machine, multi-card architecture for distributed training. It should be noted that in the above "x machine x card" structure, "machine" refers to the host computer, and "card" refers to devices such as accelerator cards. The following describes the system architecture of the training system with reference to the accompanying figures.

[0086] Referring to the architectural diagrams of the training system shown in Figures 1A and 1B , in Figure 1A , the training system may be a training node, wherein the training node 100 includes a host 102 and multiple devices 104. The host 102 includes a first chip 1022 and a host memory 1024, and the device 104 may be a second chip. In Figure 1B , the training system may include multiple training nodes 100, for example, a first training node and a second training node. Each training node 100 includes a host 102 and at least one device 104. The host 102 includes a first chip 1022 and a host memory 1024, and the device 104 may be a second chip.

[0087] The first chip 1022 may be a CPU, and the second chip may be an NPU or a GPU. The first chip 1022 and the second chip may be connected via a bus. The first chip 1022 and multiple second chips may collaborate to complete a training task. For example, a training task may be split into multiple subtasks, with different chips performing different subtasks. Based on this, the above training task is also referred to as a distributed training task.

[0088] It should be noted that the system architectures shown in Figures 1A and 1B are merely exemplary, and other possible implementations of the present application may employ other architectures. For example, the host 102 may include multiple CPUs, which may be connected to form a mesh.

[0089] The training system can use an iterative method to update the parameters of the model to achieve model training. Each iteration updates the parameters of the model once, where an iteration can also be called a training step, or simply step. The sample size used in each iteration is called the batch size. During the training process, the process of using all the sample data in a data set (such as a training set) once is called an epoch. For ease of understanding, an example is given below. In this example, the training set includes 1,000 sample data, and the batch size can be 100. Then, 100 sample data can be used for one iteration each time, and 10 iterations of the 1,000 sample data in the training set complete the training of one epoch.

[0090] In a distributed training mechanism, a synchronous update mechanism can be used between training nodes (or between training chips) to update model parameters. This mechanism involves accumulating the gradients obtained by each training node (or between each chip) and calculating the mean, then updating the model parameters based on this mean. Compared to an asynchronous update mechanism, where each training node updates the model parameters based on its own gradients, the synchronous update mechanism ensures a more stable reduction in loss and avoids large fluctuations.

[0091] The synchronous update mechanism binds computation and communication to each gradient synchronization. When using this mechanism, failures in individual training nodes, the training algorithm, or the network can interrupt the entire distributed training task. As the number of training nodes increases, the likelihood of interruption increases. Therefore, related technologies offer solutions for backing up failed files to restore training tasks.

[0092] As shown in Figure 2A, a typical fault handling solution uses an elastic agent for fault detection. Upon detecting a fault, it notifies the corresponding process in the worker process group within the same process group to stop. The elastic agent then reinitializes the cluster. The training task then creates a new process group on a non-faulty cluster node, loads a periodic fault file, such as a checkpoint (CKPT) file, and resumes the training job based on the CKPT file. This prevents catastrophic failures caused by server maintenance or network issues, preventing loss of training progress.

[0093] As shown in Figure 2B, another typical fault handling solution is to schedule components for fault detection. The fault types detected include chip faults and node faults (including parameter-level faults, such as cluster communication failures). Once a fault is detected, the faulty node is isolated, the training process is stopped, and the AI ​​framework saves the final CKPT file (the CKPT file at the time of the fault, also known as the breakpoint CKPT file). The training task can be rescheduled to a non-faulty cluster node, which loads the final CKPT file and continues the training operation.

[0094] However, the above solution causes all processes to exit (also known as stop, specifically closing the file descriptors opened by the process to release the resources occupied by the process) upon detecting a fault, regardless of the fault type. This results in the training system needing to restart the process to perform the corresponding task, such as recompiling, when different types of faults occur, which greatly affects the efficiency of resuming training tasks.

[0095] In view of this, the present application provides a fault handling method. The method is applied to a training system, which includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. For example, the first subtask can be a subtask that compiles a computation graph, and the second subtask can be a subtask that executes an operator sequence based on the compiled result of the computation graph.

[0096] Specifically, the first chip executes the first subtask. When a fault occurs, the first chip saves the fault file before the second chip in normal status on the device side stops executing the second subtask. Then the first chip synchronizes the fault file to the rescheduled chip on the device side so that the rescheduled chip can continue to execute the second subtask.

[0097] In this method, when a fault occurs, the second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the first subtask. In this way, the execution result of the first subtask on the host side (for example, the compilation result, including but not limited to the compilation result of the calculation graph) can be reused to resume the training task, shortening the time for resuming the training task and improving the efficiency of resuming the training task. Among them, the subtask can be executed by a process or a thread. In other words, the fault handling method of the present application can realize the exit of the process or thread on the device side, and the process or thread on the host side may not exit. The host side does not need to reapply for resources to repeatedly execute the same subtask, further shortening the time for resuming the training task and improving the efficiency of resuming the training task.

[0098] Furthermore, this method supports fine-grained recovery solutions based on different fault types. For recoverable chip faults, an online hot reset can be performed to recover, and the training state is restored by synchronizing the faulty file with a healthy chip. For unrecoverable node faults, only the subtasks on the failed node are stopped (processes or threads are stopped) and resumed on a new training node. The training state is synchronized by synchronizing the faulty file with a healthy training node.

[0099] The fault handling method of the present application can be applied to various distributed training scenarios. For example, the fault handling method can be used in the scenario of distributed training of an image recognition model, in which different network structures of the image recognition model (for example, different subnetworks or different network layers) can be trained by different training nodes or multiple second chips of a training node. When a training node fails or a second chip (certain second chips) fails, the second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the second subtask, wherein the first chip saves the fault file before the second chip in normal state stops executing the second subtask, and then synchronizes the fault file to the chip that is rescheduled on the device side, so that the rescheduled chip continues to execute the second subtask.

[0100] This fault handling method can also be used in scenarios involving distributed training of text recognition models, where different network structures can be trained separately by different training nodes or second chips. The fault handling mechanism for a fault can be referenced in the scenario involving distributed training of image recognition models. This will not be further elaborated here.

[0101] Figures 1A and 1B introduce the framework of the training system. The following describes the hardware structure and software logic of the training system using a single-machine, multi-card architecture as an example.

[0102] A training system with a single-machine, multi-GPU architecture can be a server, while a training system with a multi-machine, multi-GPU architecture can be a server cluster. A server cluster is a collection of multiple servers, and the hardware structure of a server cluster can be similar to that of a single server. Specifically, users can purchase or lease servers, which can be cloud servers or physical servers.

[0103] Referring to the hardware structure diagram of a server shown in FIG3 , the server 30 includes a host 32 and a device 34 . The host 32 and the device 34 are connected. The host 32 includes a processor and memory (i.e., host memory). The processor may be a CPU, and the memory may be a Dual In-line Memory Module (DIMM). The DIMM may be a double data rate (DDR) type, for example, the memory may be a DDR4 DIMM. In the example of FIG3 , the host 32 includes four CPUs and four DDR4 DIMM groups. Each CPU is connected to one DDR4 DIMM group, and each DDR4 DIMM group includes eight DDR4 DIMMs. Multiple CPUs of the host 32 can be connected to form a hydra mesh.

[0104] Optionally, the host 32 further includes an interface, such as one or more of a Serial Advanced Technology Attachment (SATA) interface, a next-generation non-volatile memory express (NVMe) interface, and a Gigabit Ethernet (GE) interface. The host 32 may further include a memory. The memory may include a SATA-supported memory or an NVMe-supported memory, such as a SATA-supported mechanical hard disk or an NVMe-supported solid-state drive (SSD).

[0105] Device 34 includes an accelerator card. In the example of FIG3 , the accelerator included in device 34 may be an NPU. FIG3 illustrates an example in which device 34 includes eight NPUs. In other possible implementations of the embodiment of the present application, device 34 may also include more accelerator cards, for example, more types of accelerator cards, or a greater number of accelerator cards.

[0106] Then, referring to the framework diagram of the software deployed on the server shown in Figure 4, the user can install firmware 302 and driver 304 on the server 30. Among them, firmware 302 is usually a program written into read-only memory that can directly control the hardware, interact with the hardware, and check whether the hardware has any errors. Driver 304 is specifically a small piece of code added to the operating system that contains information about the hardware. When a computer program requests to interact with a certain hardware, driver 304 can act as a converter of instructions between the hardware and the program that uses it. For example, firmware 302 can control device 34, interact with device 34, and check whether the device 34 has any errors. Driver 304 can act as a converter of instructions between device 34 and the program that uses it.

[0107] Furthermore, when the hardware architecture of the server 30 adopts a heterogeneous computing architecture (including a computing architecture of processing units using different types of instruction sets), the user can also install a heterogeneous computing framework 306 on the server 30. In a distributed training scenario, the heterogeneous computing framework 306 can be a heterogeneous computing framework for neural networks (Compute Architecture for Neuro Net, CANN). CANN can support users to quickly build AI applications by providing a multi-level programming interface. Among them, AI applications refer to applications built based on trained AI models. It should be noted that the heterogeneous computing framework 306 is an optional framework. The above framework is not installed on the server 30, and the fault handling method of the embodiment of the present application can also be executed. The role of the above framework is to improve the efficiency of building AI applications.

[0108] Then, the user can install the deep learning framework 308 on the server 30. The deep learning framework 308 is used to compile the computational graph of the model and automatically implement the gradient calculation in the computational graph. In this way, when performing distributed training, the graph compilation results (such as operator sequences) can be executed to perform related calculations for distributed training. Depending on the compilation method, deep learning frameworks can be divided into frameworks that support static compilation and frameworks that support dynamic compilation. Users can choose to install one or more deep learning frameworks 308 on the server 30 according to business needs. In some embodiments, the deep learning framework 308 may not be installed on the server 30. In this case, the server 30 can implement the model from scratch using a programming language such as Python.

[0109] The user can also install a scheduler 310 on the server 30. The scheduler 310 is used to schedule training tasks to achieve distributed training. Among them, the scheduler 310 can be a distributed scheduling component (distributed scheduling framework) or an AI development platform. In some embodiments, the distributed scheduling component or AI development platform can be a self-developed component or development platform, or it can be a third-party distributed scheduling component or a third-party AI development platform.

[0110] Furthermore, users can also deploy a model library 312 on the server 30. This model library 312 includes AI models implemented using a unified framework, with standardized parameters and APIs. The AI ​​models include reusable configuration items defined in the unified framework. This reduces the configuration work for the AI ​​models.

[0111] The scheduler 310 can initiate a training task on the training node, the training code can be downloaded from the model library 312, and the configuration file is read from the training code downloaded from the model 312. When a training node fails or one or more second chips (such as NPU) in the training node fails, the scheduler 310 can, upon detecting a failure, notify the first chip (such as CPU) to save a fault file before the second chip in a normal state on the device side stops executing the second subtask. The fault file is used to be synchronized by the first chip to the chip that is heavily scheduled on the device side.

[0112] The scheduler 310 can be provided to the user in the form of a code package, which the user can install or deploy. The scheduler 310 can also be provided to the user as a cloud service. Specifically, the cloud service provider can provide an application programming interface (API) for fault handling, and the training system can call the API to implement the fault handling method of the present application.

[0113] In order to make the technical solution of the present application clearer and easier to understand, the fault handling method of the present application is introduced below with reference to the accompanying drawings.

[0114] Referring to the flowchart of a fault handling method shown in FIG5 , the method is applied to a training system, the training system comprising a first chip on the host side and a plurality of second chips on the device side. The first chip and the plurality of second chips are used to collaboratively execute a training task, the training task comprising a first subtask and a plurality of second subtasks. The execution of the second subtask depends on the execution result of the first subtask. For example, the first subtask may be a compilation subtask, and the execution result of the first subtask may be a compilation result. The second subtask is an operator sequence (kernel sequence) execution subtask. The execution of the operator sequence depends on the compilation result on the host side. The method comprises the following steps:

[0115] S502: The first chip executes the first subtask. If a fault occurs, execute S504.

[0116] The first subtask is the subtask that executes the host-side logic within the training task. Specifically, the training task is a computation graph that executes the model in a loop. The computation graph is a high-level representation that provides a global view of operators without specifying implementation details. To maximize the reuse of computing resources, improve computational efficiency, and shorten model training time, the computation graph is typically compiled and optimized. Compilation optimization can include high-level data flow rewriting to generate an optimized computation graph, followed by operator-level optimization to generate efficient code for the fused operators in the computation graph. Operators are declared using a tensor expression language (TEL). A cost model is then used to search for the optimal code for the operators from a set of potential optimizations. The generated code is packaged into a deployable module, such as a kernel sequence. This compilation optimization process can be performed on the host, while the kernel sequence can be executed on the device. Therefore, the first subtask can include a compilation optimization subtask (also called a compilation subtask). The second subtask can include a subtask that executes the kernel sequence.

[0117] When looping through the model's computation graph, it's often necessary to read a dataset and perform data preprocessing on the data in the dataset. The data preprocessing operations can vary depending on the data type of the dataset. For example, if the dataset includes image data, the data preprocessing operations may include cropping, scaling, and rotation. Another example is if the dataset includes text data, the data preprocessing operations may include word segmentation. This data preprocessing can be performed on the host. Accordingly, the first subtask may also include a data preprocessing subtask.

[0118] The first chip can execute the first subtask through a process or thread. A process or thread is an application running in memory. A process is an encapsulation of a runtime program and is the basic unit of resource scheduling and allocation. A thread usually depends on a process and is used to achieve concurrency within the process. A process has independent memory, and threads share the memory of a process. Taking the compilation optimization subtask as an example, the first chip can execute the compilation optimization subtask through one or more processes. In some possible implementations, the first chip can also execute the compilation optimization subtask through one or more threads.

[0119] S504: The first chip saves the fault file before the second chip in a normal state on the device side stops executing the second subtask.

[0120] Failures in the training system can be recoverable chip failures or unrecoverable node failures. A recoverable chip failure is one that can be recovered through a warm reset. An unrecoverable node failure is one that is difficult to recover from on the original node. Unrecoverable node failures include, but are not limited to, failures of all chips on the training node's device side or parameter plane failures. Parameter plane failures can include cluster communication failures.

[0121] Because the training system includes multiple second chips. When a second chip fails, the first chip can save the fault file before the second chip in a normal state stops executing the second subtask. In some possible implementations, for a multi-machine, multi-card training system, if a recoverable chip fault or an unrecoverable node fault occurs in a training node in the training system, the first chip of a normal node (a training node that has not failed) in the training system can save the fault file before the second chip of the normal node stops executing the subtask. In a specific implementation, the first chip of the normal node calls the interface of the heterogeneous computing architecture component through an AI framework (such as a deep learning framework), causing the heterogeneous computing architecture component to exit, the process or thread based on the heterogeneous computing architecture component to stop, and the second subtask to stop executing. Before the second subtask stops executing, the AI ​​framework can save the fault file. In other possible implementations, for a single-machine, multi-card training system, if an unrecoverable chip fault occurs in a training node, the first chip of the training node can save the fault file before the second chip in the node that has not failed stops executing the second subtask. In some possible implementations, the first subtask may not stop executing. In other words, the process or thread used to execute the first subtask may not exit.

[0122] Among them, when saving the fault file, the first chip can read the fault file from the device side and save the fault file in the memory of the host side without persisting the fault file. In other words, the first chip can not write the fault file to the disk, for example, the first chip does not write the fault file to the Object Storage Service (OBS) or the Open Computing Kit (OCK). Furthermore, when saving the fault file, the first chip can save the final fault file, such as the final CKPT (also known as the breakpoint CKPT), to avoid the loss of training progress.

[0123] S506: The first chip synchronizes the fault file with the chip that is rescheduled by the device, so that the rescheduled chip can continue to execute the second subtask.

[0124] When a training task is interrupted due to a fault, it can be rescheduled to a normal chip. This chip is called the rescheduled chip. The first chip synchronizes the fault file with the rescheduled chip, allowing the rescheduled chip to continue executing the second subtask, thereby resuming the training task.

[0125] For recoverable chip failures, the rescheduled chip can be the second chip whose state has returned to normal after reset. For example, the training system is a training node with a single-machine multi-card architecture, and the training node includes a first chip and multiple second chips, and a target chip among the multiple second chips fails. The rescheduled chip can be the target chip whose state has returned to normal after reset. Accordingly, the first chip can synchronize the fault file with the second chip (such as the target chip) whose state has returned to normal after reset.

[0126] For an unrecoverable node failure, the rescheduled chip can be a newly added third chip. For example, the training system includes a first training node and a second training node, and each training node includes at least one first chip and at least one second chip. If an unrecoverable node failure occurs in the first training node, the training task can be rescheduled to the newly added third training node, and the third chip belongs to the newly added third training node. Accordingly, the first chip can synchronize the fault file to the third chip (or third training node).

[0127] It should be noted that the first chip of the first training node can also save the execution result of the first subtask, for example, cache the compilation result, and the first chip of the third training node can load the execution result of the above-mentioned first subtask without recompiling.

[0128] When the fault file is stored in memory (such as the host's memory), the first chip can read the fault file from memory and then synchronize it with the device-focused chip through the cluster interface. Model information (such as model weights) is no longer serialized and deserialized through CKPT, avoiding the impact of model scale, input and output (IO), and bandwidth on training task recovery. Instead, point-to-point state synchronization is performed through the cluster mechanism.

[0129] When the first chip saves the terminal fault file such as the terminal CKPT, the first chip can synchronize the training state (or training progress) at the moment of the fault, thereby avoiding the loss of the training process caused by the periodic CKPT.

[0130] Based on the above description, it can be seen that the fault handling method of the present application supports that when a fault occurs, the second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the second subtask. In this way, the execution result of the first subtask on the host side (for example, the compilation result, including but not limited to the compilation result of the calculation graph) can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0131] The above describes the fault handling method from the perspective of the first chip. The following describes the fault handling method of the present application from the perspective of the scheduler.

[0132] Referring to FIG6 , a flowchart of a fault handling method is shown. This method is applied to a scheduler, which is used to handle faults in a training system. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The method includes the following steps:

[0133] S602: The scheduler performs fault detection on the training system.

[0134] Specifically, the scheduler can perform fault detection on the second chip in the training system or the training node where the second chip is located. Specifically, the scheduler can perform fault detection on the second chip through heartbeat messages. The second chip in the training system can periodically send heartbeat messages to the scheduler, and the scheduler determines whether the second chip has failed based on the reception of the heartbeat messages. For example, if the scheduler does not receive the heartbeat message from the second chip for N consecutive cycles, it means that the second chip or the node where the second chip is located has failed. The scheduler can determine the type of fault based on the reception of heartbeat messages from other second chips in the same training node.

[0135] For the K second chips of the same training node, if the scheduler does not receive heartbeat messages from the K second chips of the training node, for example, if it does not receive heartbeat messages from the K second chips for N consecutive cycles, the scheduler may determine that the training node has failed. For the K second chips of the same training node, if the scheduler does not receive heartbeat messages from some of the second chips but receives heartbeat messages from the remaining second chips, the scheduler may determine that the second chip that did not receive the corresponding heartbeat message has failed.

[0136] It should be noted that the heartbeat message is only one implementation method for fault detection of the training system. In other possible implementation methods of the embodiment of the present application, the scheduler can also implement fault detection in other ways. For example, the second chip can be detected by the scheduler by reading the status code of the second chip.

[0137] S604: When a fault is detected, the scheduler notifies the first chip to save the fault file before the second chip in a normal state on the device side stops executing the second subtask.

[0138] The fault file is used to be synchronized by the first chip to the chip that is rescheduled on the device side. In this way, the rescheduled chip can continue to execute the second subtask based on the fault file synchronized by the first chip.

[0139] In some possible implementations, for recoverable chip failures, the scheduler can also reset the faulty chip on the device side, such as performing an online hot reset. Accordingly, the chip that the device prioritizes scheduling includes the second chip that has returned to normal after reset.

[0140] In other possible implementations, for an unrecoverable node failure, the device prioritizes scheduling a chip including a newly added third chip. The scheduler may synchronize information about the newly added third chip with a normal second chip. This information about the third chip is used to establish a link with other second chips participating in training. In a specific implementation, the training system includes a first training node and a second training node, each of which includes at least one first chip and at least one second chip. If an unrecoverable node failure occurs in the first training node, the training task may be rescheduled to the third training node. The third chip is the device-side chip of the third training node. The scheduler may synchronize information about the third chip with at least one second chip in the second training node. The information about the third chip includes at least one of address information and resource configuration information of the third training node to which the third chip belongs. The address information of the third training node may be the IP address of the third training node or another address that can be used for link establishment. The resource configuration information may be a ranking table for the third chip, which records the sequence numbers of the third chips occupied by the processes or threads executing the subtasks.

[0141] Based on the above description, it can be seen that the fault handling method of the present application notifies the first chip to save the fault file before the second chip on the device side, which is in a normal state, stops executing the second subtask when the scheduler detects a fault. The second chip on the device side can stop executing the second subtask, while the first chip on the host side does not stop executing the second subtask. In this way, the execution result of the first subtask on the host side can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0142] The above describes the fault detection method of the present application from the perspectives of the first chip and the scheduler. The following describes the fault handling method of the present application in combination with specific application scenarios.

[0143] First, refer to FIG7 , a schematic diagram of a scenario for using an original node to recover a training task. As shown in FIG7 , the training system includes multiple training nodes, each of which includes a first chip on the host side and a second chip on the device side (such as an NPU / GPU). FIG7 illustrates an example in which the device side of each training node includes 8 second chips. When a recoverable chip failure occurs in the second chip numbered 0 to 3 of a training node, the training task can be recovered on the original node.

[0144] The faulty training node is called an abnormal node, and the healthy node is called a normal node. The abnormal node and the normal node perform corresponding operations to restore the training task on the original node.

[0145] For abnormal nodes, the distributed scheduling framework (such as a scheduler) deployed on the host side detects a chip failure and resets the faulty chip, such as the second chip (rank 0 to 3). Among them, the chip failure or chip reset can trigger the exit of the heterogeneous computing architecture component on the device side. Accordingly, the process on the device side can exit and the second subtask stops executing. The corresponding process of the host-side AI framework does not exit, so there is no need to recompile the calculation graph and wait for the chip to be reset and the heterogeneous computing architecture component to exit. The host-side AI framework detects that the status of the second chip has returned to normal after the reset, and can trigger the link building process to re-establish the link with the normal node, for example, it can re-establish the link through the collective communication library. The host-side AI framework then re-executes the calculation graph and waits for the fault file synchronization to be completed. When the synchronization is completed, the abnormal node can continue the training job. For example, the second chip of the abnormal node can continue to execute the second subtask.

[0146] For normal nodes, the distributed scheduling framework deployed on the host side can notify the AI ​​framework to call the heterogeneous computing architecture component interface, so that the heterogeneous computing architecture component exits, the process or thread based on the heterogeneous computing architecture component stops, and the second subtask stops executing. Among them, the AI ​​framework saves the fault file before the process or thread based on the heterogeneous computing architecture component stops (the second subtask stops executing). After saving the fault file, the heterogeneous computing architecture component can be quickly launched. The process or thread corresponding to the AI ​​framework does not exit, so there is no need to recompile. The host-side AI framework detects that the chip status is normal and can re-establish the chain. The AI ​​framework then re-executes the calculation graph, and the AI ​​framework synchronizes the fault file to the recovered node through the cluster communication interface according to the fault file recovery strategy. Normal nodes can continue training jobs.

[0147] Next, referring to FIG8 , a schematic diagram of a scenario for using a new node to resume a training task is shown. As shown in FIG8 , the training system includes multiple training nodes, and the training node includes a first chip on the host side and a second chip on the device side (such as an NPU / GPU). FIG8 illustrates an example in which the device side of each training node includes 8 second chips. When an unrecoverable node failure occurs on a training node (all second chips of the training node fail), the training task can be resumed on the new node.

[0148] The abnormal nodes, normal nodes, and new nodes perform corresponding operations respectively to resume the training task on the new node.

[0149] For abnormal nodes, the distributed scheduling framework detects a failure and the heterogeneous computing architecture component can exit. The AI ​​framework can cache the compilation results based on the return value (abnormal return value) of the heterogeneous computing architecture component exit. Then, the AI ​​framework process can exit.

[0150] For normal nodes, the distributed scheduling framework notifies the AI ​​framework to call the heterogeneous computing architecture component interface to exit the heterogeneous computing architecture component. Before the heterogeneous computing architecture component exits, the AI ​​framework saves the fault file to the host side. After saving the file, the heterogeneous computing architecture component can exit quickly. The process or thread based on the heterogeneous computing architecture component can exit. The second subtask stops executing. The AI ​​framework on the host side does not exit and does not need to be recompiled. The AI ​​framework on the host side detects that the chip status is normal and can re-establish the chain. The AI ​​framework then re-executes the calculation graph and synchronizes the fault file to the recovered node through the cluster communication interface based on the fault file recovery strategy. Normal nodes can continue training jobs.

[0151] For new nodes, the distributed scheduling framework synchronizes the new node's IP address and rank table to every container group POD in each cluster. The AI ​​framework then obtains the new node's IP address and updates the rank table. The AI ​​framework then loads the compilation results cached from the faulty node, using, for example, OCK to accelerate the loading of cached compilation results from the faulty node. The AI ​​framework then verifies that the chip is functioning normally and reestablishes the chain. The AI ​​framework then re-executes the computation graph and waits for the faulty file to be synchronized. Once the faulty file is synchronized, the newly joined node can resume training.

[0152] Based on the aforementioned fault handling method, the present application also provides a fault handling device. As shown in FIG9 , the fault handling device 900 is deployed on the first chip on the host side of the training system. The first chip and multiple second chips on the device side of the training system are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The fault handling device 900 includes:

[0153] A task execution module 902, configured to execute the first subtask;

[0154] The file saving module 904 is configured to save the fault file when a fault occurs and before the second chip in a normal state on the device side stops executing the second subtask;

[0155] The file synchronization module 906 is configured to synchronize the fault file to the rescheduled chip of the device, so that the rescheduled chip can continue to execute the second subtask.

[0156] Exemplarily, the task execution module 902 , the file saving module 904 , and the file synchronization module 906 may be implemented by hardware or software.

[0157] Among them, when implemented by software, the task execution module 902, the file saving module 904, and the file synchronization module 906 can be applications running on a computing device, such as a computing engine. Applications can also be virtualized through virtualization services to be provided to users. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that uses virtualization technology to virtualize a virtual machine (VM) resource pool on multiple physical hosts to provide users with VMs for use on demand. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMSs for use on demand. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers for use on demand. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable. Its computing performance is no different from that of a traditional physical machine and has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user spaces, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely specific examples. In actual applications, virtualization services can also include other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0158] When implemented through hardware, the task execution module 902, the file storage module 904, and the file synchronization module 906 may include at least one computing device, such as a server. Alternatively, the task execution module 902, the file storage module 904, and the file synchronization module 906 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0159] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault.

[0160] In some possible implementations, the fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after reset.

[0161] In some possible implementations, the training system includes a training node, the training node includes a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips;

[0162] The file synchronization module 906 is specifically used to:

[0163] The fault file is synchronized to the target chip whose state has returned to normal after reset.

[0164] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

[0165] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the unrecoverable node failure occurs in the first training node, and the third chip belongs to a newly added third training node. The apparatus 900 further includes an execution result saving module 908 deployed in the first training node for the first chip and an execution result loading module 909 deployed in the third training node for the first chip.

[0166] The execution result saving module 908 is used to save the execution result of the first subtask;

[0167] The execution result loading module 909 is used to load the execution result of the first subtask.

[0168] The execution result saving module 908 and the execution result loading module 909 can be implemented via software or hardware. When implemented via software, the execution result saving module 908 and the execution result loading module 909 can be applications running on a computing device, such as a computing engine. When implemented via hardware, the execution result saving module 908 and the execution result loading module 909 can include at least one computing device, such as a server. Alternatively, the execution result saving module 908 and the execution result loading module 909 can be implemented as an ASIC or a PLD.

[0169] In some possible implementations, the file synchronization module 906 is specifically configured to:

[0170] The fault file is obtained from the memory, and the fault file is synchronized with the chip for focusing on scheduling of the device through the cluster communication interface.

[0171] In some possible implementations, the subtask is executed in a process or thread manner.

[0172] Based on the aforementioned fault handling method, the present application also provides a scheduler. The scheduler is used to handle faults when a training system fails. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. As shown in Figure 10, the scheduler 1000 includes:

[0173] A fault detection module 1002 is configured to perform fault detection on the training system;

[0174] Notification module 1004 is used to notify the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask when a fault is detected. The fault file is used to be synchronized by the first chip to the chip that is heavily scheduled on the device side.

[0175] Exemplarily, the fault detection module 1002 and the notification module 1004 may be implemented by hardware or software.

[0176] When implemented via software, the fault detection module 1002 and notification module 1004 may be application programs running on a computing device. When implemented via hardware, the fault detection module 1002 and notification module 1004 may include at least one computing device, such as a server. Alternatively, the fault detection module 1002 and notification module 1004 may be implemented as an ASIC or a PLD. Alternatively, the fault detection module 1002 and notification module 1004 may be a processing unit within the aforementioned devices.

[0177] In some possible implementations, the fault is a recoverable chip fault, and the scheduler 1000 further includes:

[0178] A reset module 1006 is configured to reset the faulty chip on the device side;

[0179] The chip that the device focuses on scheduling includes a second chip whose state returns to normal after reset.

[0180] The reset module 1006 may be implemented by hardware or software.

[0181] When implemented via software, reset module 1006 can be an application running on a computing device. When implemented via hardware, reset module 1006 can include at least one computing device, such as a server. Alternatively, reset module 1006 can be implemented as an ASIC or PLD. Alternatively, reset module 1006 can be a processing unit within any of the aforementioned devices.

[0182] In some possible implementations, the training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to synchronize the first chip to the target chip whose state has returned to normal after reset.

[0183] In some possible implementations, the fault is an unrecoverable node fault, the chip that the device prioritizes scheduling includes a newly added third chip, and the scheduler 1000 further includes:

[0184] The synchronization module 1008 is configured to synchronize information of the newly added third chip with the second chip in normal state.

[0185] Similarly, the synchronization module 1008 may be implemented by hardware, or may be implemented by software.

[0186] When implemented via software, synchronization module 1008 can be an application running on a computing device. When implemented via hardware, synchronization module 1008 can include at least one computing device, such as a server. Alternatively, synchronization module 1008 can be implemented as an ASIC or PLD. Alternatively, synchronization module 1008 can be a processing unit within any of the aforementioned devices.

[0187] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, and the unrecoverable node failure occurs in the first training node;

[0188] The synchronization module 1008 is specifically used for:

[0189] The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

[0190] This application also provides a chip. The chip may include a processor and a memory. The memory stores computer-readable instructions; the processor executes the computer-readable instructions to perform the fault handling method described in the aforementioned embodiment. For example, the chip may be a first chip, and the processor of the first chip executes the computer-readable instructions to perform the fault handling method described in the aforementioned embodiment performed by the first chip.

[0191] This application also provides a scheduler. The scheduler may include a processor and a memory. The memory stores computer-readable instructions; the processor executes the computer-readable instructions to perform the fault handling method performed by the scheduler in the aforementioned embodiment, such as the method in the embodiment shown in FIG6 .

[0192] The present application also provides a training system. The training system may include a first chip on the host side and multiple second chips on the device side. Among them, the training system may be a single-machine multi-card architecture, that is, the training system may be a training node, such as a training server. The training node may include a host (the host includes a first chip) and multiple accelerator cards (i.e., a second chip), and the first chip on the host side and the multiple second chips on the device side in the training node can collaboratively perform distributed training tasks. In some examples, the training system may also be a multi-machine multi-card architecture, that is, the training system may be a cluster formed by multiple training nodes. Each of the multiple training nodes includes a first chip on the host side and at least one second chip on the device side. Multiple training nodes can collaboratively perform distributed training tasks. Among them, the first chip in the training system is used to execute computer-readable instructions to execute the fault handling method of the aforementioned embodiment.

[0193] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the execution of the fault handling method of the aforementioned embodiment.

[0194] Embodiments of the present application also provide a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the aforementioned fault handling method.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A fault handling method, characterized in that: Applied to a training system, the training system includes a first chip on a host side and multiple second chips on a device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the method includes: The first chip performs the first subtask; When a fault occurs, the first chip saves the fault file before the second chip in a normal state on the device side stops executing the second subtask; The first chip synchronizes the fault file to the rescheduled chip of the device, so that the rescheduled chip continues to execute the second subtask.

2. The method according to claim 1, characterized in that The fault includes a recoverable chip fault or an unrecoverable node fault.

3. The method according to claim 1 or 2, characterized in that: The fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

4. The method according to any one of claims 1 to 3, characterized in that: The training system comprises a training node, the training node comprises a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips; The first chip synchronizes the fault file to the chip that focuses on scheduling the device, including: The first chip synchronizes the fault file with the target chip whose state is restored to normal after reset.

5. The method according to claim 1 or 2, characterized in that: The fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

6. The method according to claim 5, characterized in that The training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the method further includes: The first chip of the first training node stores the execution result of the first subtask; The first chip of the third training node loads the execution result of the first subtask.

7. The method according to any one of claims 1 to 6, characterized in that: The first chip synchronizes the fault file to the chip that focuses on scheduling the device, including: The first chip obtains the fault file from the memory, and synchronizes the fault file with the chip focused on scheduling of the device through the cluster communication interface.

8. The method according to any one of claims 1 to 7, characterized in that: The subtasks are executed in a process or thread manner.

9. A fault handling method, characterized in that: Applied to a scheduler, the scheduler is used to perform fault processing when a training system fails, the training system includes a first chip on a host side and multiple second chips on a device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the method includes: Performing fault detection on the training system; When a fault is detected, the first chip is notified to save a fault file before the second chip in normal state on the device side stops executing the second subtask. The fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

10. The method according to claim 9, characterized in that The fault is a recoverable chip fault, and the method further includes: Resetting the faulty chip on the device side; The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

11. The method according to claim 9 or 10, characterized in that: The training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to be synchronized by the first chip to the target chip whose state is restored to normal after reset.

12. The method according to claim 9, characterized in that The fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the method further includes: The information of the newly added third chip is synchronized with the second chip in normal state.

13. The method according to claim 12, characterized in that The training system comprises a first training node and a second training node, each training node comprises at least one first chip and at least one second chip, and the first training node has the unrecoverable node failure; The step of synchronizing information of the newly added third chip to the second chip in a normal state includes: The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

14. The method according to any one of claims 9 to 13, characterized in that: The subtasks are executed in a process or thread manner.

15. A fault handling device, characterized in that: The apparatus is deployed on a first chip on a host side of a training system, and the first chip and a plurality of second chips on a device side of the training system are used to collaboratively execute a training task, wherein the training task includes a first subtask and a plurality of second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. The apparatus includes: A task execution module, used for executing the first subtask; A file saving module, configured to save the fault file when a fault occurs and before the second chip in a normal state on the device side stops executing the second subtask; The file synchronization module is used to synchronize the fault file to the rescheduled chip of the device, so that the rescheduled chip can continue to execute the second subtask.

16. The device according to claim 15, characterized in that The fault includes a recoverable chip fault or an unrecoverable node fault.

17. The device according to claim 15 or 16, characterized in that The fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

18. The device according to any one of claims 15 to 17, characterized in that The training system comprises a training node, the training node comprises a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips; The file synchronization module is specifically used for: The fault file is synchronized to the target chip whose state is restored to normal after reset.

19. The device according to claim 15 or 16, characterized in that The fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

20. The device according to claim 19, characterized in that The training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the device also includes an execution result saving module of the first chip deployed in the first training node and an execution result loading module of the first chip deployed in the third training node; The execution result saving module is used to save the execution result of the first subtask; The execution result loading module is used to load the execution result of the first subtask.

21. The device according to any one of claims 15 to 20, characterized in that The file synchronization module is specifically used for: The fault file is obtained from the memory, and the fault file is synchronized to the chip focused on scheduling of the device through the cluster communication interface.

22. The device according to any one of claims 15 to 21, characterized in that The subtasks are executed in a process or thread manner.

23. A scheduler, characterized in that: The scheduler is used to handle a fault when a training system fails. The training system includes a first chip on a host side and multiple second chips on a device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The scheduler includes: A fault detection module, used for performing fault detection on the training system; The notification module is used to, when a fault is detected, notify the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask, and the fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

24. The scheduler according to claim 23, characterized in that The fault is a recoverable chip fault, and the scheduler further includes: A reset module, used to reset the faulty chip on the device side; The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

25. The scheduler according to claim 23 or 24, characterized in that: The training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to be synchronized by the first chip to the target chip whose state is restored to normal after reset.

26. The scheduler according to claim 23, characterized in that The fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the scheduler further includes: The synchronization module is used to synchronize the information of the newly added third chip with the second chip in a normal state.

27. The scheduler according to claim 26, characterized in that The training system comprises a first training node and a second training node, each training node comprises at least one first chip and at least one second chip, and the first training node has the unrecoverable node failure; The synchronization module is specifically used for: The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

28. The scheduler according to any one of claims 23 to 27, characterized in that: The subtasks are executed in a process or thread manner.

29. A chip, characterized in that: The chip comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the chip executes the method according to any one of claims 1 to 8.

30. A scheduler, characterized in that: The system comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the scheduler executes the method according to any one of claims 9 to 14.

31. A training system, characterized in that: The training system comprises a first chip on a host side and a plurality of second chips on a device side, wherein the first chip is used to execute the method as claimed in any one of claims 1 to 8.

32. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 14.

Images