Fault processing method and related equipment

Abstract

The invention provides a fault processing method, which is applied to a training system, the training system comprises a first chip at a host side and a plurality of second chips at an equipment side, the first chip and the plurality of second chips are used for cooperatively executing a training task, and the training task comprises a first subtask and a plurality of second subtasks. The execution of the second sub-task depends on the execution result of the first sub-task, and the method comprises the following steps: the first chip executes the first sub-task, when a fault occurs, the first chip stores a fault file before the second chip in a normal state on the device side stops executing the second sub-task, the first chip synchronizes the fault file to the chip rescheduling on the device side, and the first chip performs the fault file synchronization on the chip rescheduling on the device side. And the rescheduling chip continues to execute the second subtask. In the method, the first chip on the host side can not stop executing the second subtask. Therefore, the execution result of the first subtask on the host side can be reused to recover the training task, the time for recovering the training task is shortened, and the efficiency for recovering the training task is improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on June 28, 2023, with application number 202310784685.2 and invention name “A Fault Recovery Method for Distributed Model Training”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a fault handling method, device, scheduler, chip, computer-readable storage medium, and computer program product. Background Art

[0003] With the continuous development of AI technology, more and more industries and fields are using AI models (for the sake of ease of description, sometimes referred to as models) to realize the intelligence and automation of business. For example, in the e-commerce industry, more and more merchants use AI customer service built based on AI models to replace manual customer service to provide pre-sales and after-sales consulting services. For another example, in social networks, platforms use AI models instead of manual review of user-posted content to save labor costs.

[0004] An AI model refers to a mathematical model built based on AI technology and used to predict unknown data. For example, an AI model can be an object detection model or an image classification model built based on a neural network. AI models usually need to be trained with a large amount of data. In order to improve the training efficiency of AI models, distributed training methods have emerged.

[0005] The distributed training method is to distribute the training tasks to multiple training nodes for execution, and multiple training nodes train the model in parallel. Among them, executing the training task is the process of training the model with the data set to obtain the parameters of the model (such as weights). Multiple training nodes can use a synchronous update mechanism to update the parameters of the model. The synchronous update mechanism refers to accumulating the gradients obtained by each training node to calculate the mean, and updating the parameters of the model based on the mean. When an individual training node, training algorithm or network fails, the entire distributed training task will be interrupted. As the number of training nodes increases, the possibility of interruption becomes higher and higher. Therefore, it is necessary to provide a fault handling mechanism to resume the training task.

[0006] At present, the mainstream fault handling solutions in the industry take a long time to restore training tasks, are inefficient, and are difficult to meet business needs. Summary of the invention

[0007] The present application provides a fault handling method. The method supports that when a fault occurs, the second chip on the device side stops executing the second subtask, and the first chip on the host side does not stop executing the second subtask. In this way, the execution result of the first subtask on the host side can be reused to restore the training task, shortening the time to restore the training task and improving the efficiency of restoring the training task. The present application also provides a device, a scheduler, a chip, a training system, a computer-readable storage medium, and a computer program product corresponding to the above-mentioned processing method.

[0008] In a first aspect, the present application provides a fault handling method. The method is applied to a training system. The training system includes a first chip on a host side and multiple second chips on a device side, and the first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask.

[0009] Specifically, the first chip executes the first subtask, and when a fault occurs, the first chip saves the fault file before the second chip in the normal state on the device side stops executing the second subtask. Then the first chip synchronizes the fault file to the rescheduled chip on the device side, so that the rescheduled chip continues to execute the second subtask.

[0010] In this method, when a fault occurs, the second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the first subtask. In this way, the execution result of the first subtask on the host side (for example, the compilation result, including but not limited to the compilation result of the calculation graph) can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0011] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault. The method supports implementing fine-grained recovery schemes according to different fault types, and the efficiency of recovering training tasks can be improved as much as possible based on the fine-grained recovery scheme.

[0012] In some possible implementations, the fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes the second chip whose state returns to normal after reset. In this way, the training task can be restored on the original node without applying for new node resources, thereby improving resource utilization.

[0013] In some possible implementations, the training system includes a training node, and the training node includes a first chip and multiple second chips, that is, the training system can be a single-machine multi-card architecture. The fault is a target chip fault among multiple second chips. Based on this, when the first chip synchronizes the fault file to the chip that the device focuses on scheduling, the fault file can be synchronized to the target chip that has recovered to a normal state after reset. In this way, when a recoverable chip fault occurs in a single training node, the training task is restored at the original node.

[0014] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes the newly added third chip. In this way, the training task can be restored on the new node to shorten the interruption time of the training task as much as possible and improve the recovery efficiency of the training task.

[0015] In some possible implementations, the training system includes a first training node and a second training node, and each training node includes at least one first chip and at least one second chip. That is, the training system can be a multi-machine multi-card architecture. Among them, an unrecoverable node failure occurs in the first training node, and the third chip belongs to the newly added third training node. Accordingly, the first chip of the first training node can save the execution result of the first subtask, and the first chip of the third training node can load the execution result of the first subtask.

[0016] This method caches the execution result of the first subtask and then loads the cached execution result on the new node, thereby realizing the reuse of the execution result, thereby improving the efficiency of resuming the training task.

[0017] In some possible implementations, the first subtask may be a compilation subtask (for example, a compilation calculation graph). The first chip does not stop executing the first subtask, and compilation-free operation can be achieved, or the first chip of the first training node caches the compilation result, and the first chip of the newly added third training node loads the compilation result cached by the first chip of the first training node, and reuses the compilation result. In this way, the time to resume the training task is shortened and the efficiency of resuming the training task is improved.

[0018] In some possible implementations, the first chip can obtain the fault file from the memory and synchronize the fault file to the device-focused scheduling chip through the cluster communication interface, thereby eliminating the need for re-deserialization, shortening synchronization time, and improving fault recovery efficiency.

[0019] In some possible implementations, the subtask is executed in a process or thread mode. The fault handling method of the present application can realize the process or thread exit on the device side, and the process or thread on the host side may not exit, and the host side does not need to reapply for resources to repeat the same subtask, further shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0020] In a second aspect, the present application provides a fault handling method. The method is applied to a scheduler, the scheduler is used to handle faults when a training system fails, the training system includes a first chip on the host side and multiple second chips on the device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the method includes:

[0021] Performing fault detection on the training system;

[0022] When a fault is detected, the first chip is notified to save a fault file before the second chip in normal state on the device side stops executing the second subtask. The fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

[0023] When the scheduler detects a fault, the method notifies the first chip to save the fault file before the second chip on the device side stops executing the second subtask. The second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the second subtask. In this way, the execution result of the first subtask on the host side can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0024] In some possible implementations, the fault is a recoverable chip fault, and the scheduler can also reset the faulty chip on the device side. The chip that the device focuses on scheduling includes a second chip whose state is restored to normal after reset. The method supports fine-grained recovery solutions based on different fault types. For recoverable chip faults, hot reset can be performed online to recover, and the training state can be restored by synchronizing the fault file with the normal chip.

[0025] In some possible implementations, the training system includes a training node, and the training node includes a first chip and a plurality of second chips. The fault is a target chip fault among the plurality of second chips, and the fault file is used to synchronize the target chip with the first chip to restore the normal state after reset. In this way, when a recoverable chip fault occurs in a single training node, the training task is restored at the original node.

[0026] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes the newly added third chip. Accordingly, the scheduler can also synchronize the information of the newly added third chip to the second chip in normal state. This can help establish a link between the second chip and the second chip, and then help restore the training task.

[0027] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip. The first training node has the unrecoverable node failure. Accordingly, the scheduler may synchronize the information of the third chip to at least one second chip in the second training node, the information of the third chip including at least one of the address information and resource configuration information of the third training node to which the third chip belongs.

[0028] The method can enable the third chip to resume execution of the second subtask as soon as possible, thereby improving the recovery efficiency of the training task, by synchronizing the address, resource configuration and other information of the third chip to at least one second chip in the second training node.

[0029] In some possible implementations, the subtask is executed in a process or thread mode. This method can realize that the process or thread on the device side exits, and the process or thread on the host side does not exit. The host side does not need to reapply for resources to repeatedly execute the same subtask, further shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0030] In a third aspect, the present application provides a fault handling device. The device is deployed on a first chip on the host side of a training system, the first chip and multiple second chips on the device side of the training system are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the device includes:

[0031] A task execution module, used for executing the first subtask;

[0032] A file saving module, configured to save the fault file when a fault occurs and before the second chip in a normal state on the device side stops executing the second subtask;

[0033] The file synchronization module is used to synchronize the fault file to the rescheduled chip of the device, so that the rescheduled chip can continue to execute the second subtask.

[0034] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault.

[0035] In some possible implementations, the fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

[0036] In some possible implementations, the training system includes a training node, the training node includes a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips;

[0037] The file synchronization module is specifically used for:

[0038] The fault file is synchronized to the target chip whose state is restored to normal after reset.

[0039] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

[0040] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the device further includes an execution result saving module of the first chip deployed in the first training node and an execution result loading module of the first chip deployed in the third training node;

[0041] The execution result saving module is used to save the execution result of the first subtask;

[0042] The execution result loading module is used to load the execution result of the first subtask.

[0043] In some possible implementations, the file synchronization module is specifically used to:

[0044] The fault file is obtained from the memory, and the fault file is synchronized to the chip focused on scheduling of the device through the cluster communication interface.

[0045] In some possible implementations, the subtask is executed in a process or thread manner.

[0046] In a fourth aspect, the present application provides a scheduler. The scheduler is used to handle a fault when a training system fails, the training system includes a first chip on a host side and multiple second chips on a device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the scheduler includes:

[0047] A fault detection module, used for performing fault detection on the training system;

[0048] The notification module is used to, when a fault is detected, notify the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask, and the fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

[0049] In some possible implementations, the fault is a recoverable chip fault, and the scheduler further includes:

[0050] A reset module, used to reset the faulty chip on the device side;

[0051] The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

[0052] In some possible implementations, the training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to synchronize the first chip to the target chip whose state is restored to normal after reset.

[0053] In some possible implementations, the fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the scheduler further includes:

[0054] The synchronization module is used to synchronize the information of the newly added third chip with the second chip in a normal state.

[0055] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, and the unrecoverable node failure occurs in the first training node;

[0056] The synchronization module is specifically used for:

[0057] The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

[0058] In some possible implementations, the subtask is executed in a process or thread manner.

[0059] In a fifth aspect, the present application provides a chip. The chip includes a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory, so that the chip executes the fault handling method described in the first aspect or any implementation of the first aspect.

[0060] In a sixth aspect, the present application provides a scheduler. The scheduler includes a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the scheduler executes the fault handling method described in the first aspect or any implementation of the first aspect.

[0061] In a seventh aspect, the present application provides a training system. The training system includes a first chip on a host side and a plurality of second chips on a device side. The first chip is used to execute computer-readable instructions to execute the fault handling method as described in the first aspect or any implementation of the first aspect.

[0062] In an eighth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the fault handling method described in the first aspect or any one of the implementations of the first aspect.

[0063] In a ninth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the fault handling method described in the first aspect or any one of the implementations of the first aspect.

[0064] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0066] Figure 1A A schematic diagram of the architecture of a training system provided in an embodiment of the present application;

[0067] Figure 1B A schematic diagram of the architecture of another training system provided in an embodiment of the present application;

[0068] Figure 2A A schematic diagram of a fault handling solution provided in an embodiment of the present application;

[0069] Figure 2B A schematic diagram of another fault handling solution provided in an embodiment of the present application;

[0070] Figure 3 A hardware structure diagram of a server provided in an embodiment of the present application;

[0071] Figure 4 A framework diagram of server deployment software provided in an embodiment of the present application;

[0072] Figure 5 A flowchart of a fault handling method provided in an embodiment of the present application;

[0073] Figure 6 A flowchart of a fault handling method provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of restoring a training task from an original node provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of resuming a training task from a new node provided in an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of the structure of a fault handling device provided in an embodiment of the present application;

[0077] Fig.10 A schematic diagram of the structure of another fault handling device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0079] First, some technical terms involved in the embodiments of the present application are introduced.

[0080] Artificial intelligence (AI), also known as machine intelligence, refers to the intelligence displayed by machines (such as computers) by imitating human thinking and behavior (such as learning, reasoning, thinking, planning, etc.). Artificial intelligence usually imitates human thinking and behavior based on knowledge to achieve specific goals or complete specific tasks. Among them, knowledge can come from experience or data.

[0081] Deep learning (DL), as a branch of AI, specifically uses deep neural network models (also called deep learning models, and for ease of description, sometimes referred to as models) to process massive amounts of data, learn knowledge (such as multiple nonlinear transformation structures) from massive amounts of data, and process and analyze data based on this knowledge. Trained deep learning models can be applied to perception, decision-making and other scenarios in the field of AI, such as image recognition, speech recognition, natural language translation, computer games and other scenarios.

[0082] The number of parameters in deep learning models is relatively high, usually reaching hundreds of billions or trillions. For example, the number of parameters in large models in the field of natural language processing (NLP) can reach hundreds of billions. Such large-scale deep learning models usually require huge data sets for training. A typical training method is distributed training.

[0083] Distributed training can be performed by a training system. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip can be a central processing unit (CPU), also known as a central processing unit. The host side can include not only a CPU but also a host memory. The second chip can be an external processing unit used to assist the CPU in completing tasks and achieve computing acceleration. Therefore, the second chip can also be called an acceleration card or an accelerator. The second chip can include a neural processing unit (NPU) or a graphics processing unit (GPU). Different types of processors / processing units can implement different types of instruction set architectures. Similar to the host side, the device side can also include device memory. In some examples, the device memory can be integrated with the second chip on the device side.

[0084] The training system can adopt a single-machine multi-card architecture or a multi-machine multi-card architecture to achieve distributed training. It should be noted that the "machine" in the above x machine x card refers to the host, and the "card" refers to the accelerator card and other devices. The system architecture of the training system is described below in conjunction with the accompanying drawings.

[0085] See also Figure 1A , Figure 1B The architecture diagram of the training system shown in Figure 1A In the training system, the training node 100 may include a host 102 and multiple devices 104. The host 102 includes a first chip 1022 and a host memory 1024, and the device 104 may be a second chip. Figure 1B In the embodiment, the training system may include multiple training nodes 100, for example, a first training node and a second training node. Each training node 100 includes a host 102 and at least one device 104. The host 102 includes a first chip 1022 and a host memory 1024, and the device 104 may be a second chip.

[0086] Among them, the first chip 1022 can be a CPU, and the second chip can be an NPU or a GPU. The first chip 1022 and the second chip can be connected via a bus. The first chip 1022 and multiple second chips can work together to complete the training task. For example, the training task can be divided into multiple subtasks, and different chips perform different subtasks respectively. Based on this, the above training task is also called a distributed training task.

[0087] It should be noted that Figure 1A , Figure 1BThe system architecture shown is only exemplary, and other architectures may also be used in other possible implementations of the embodiments of the present application. For example, the host 102 may also include multiple CPUs, and the multiple CPUs may be connected to form a network (mesh).

[0088] The training system can use an iterative method to update the parameters of the model to achieve model training. Each iteration updates the parameters of the model, where an iteration can also be called a training step trainstep, or step for short. The sample size used in each iteration is called the batch size. During the training process, the process in which all sample data in a data set (such as a training set) are used once is called an epoch. For ease of understanding, an example is used below to illustrate. In this example, the training set includes 1,000 sample data, and the batch size can be 100, so 100 sample data can be used for one iteration each time, and 10 iterations of the 1,000 sample data in the training set complete the training of one epoch.

[0089] In the distributed training mechanism, the training nodes (or chips used for training) can use a synchronous update mechanism to update the model parameters. The synchronous update mechanism refers to accumulating the gradients obtained by each training node (or between each chip) to calculate the mean, and updating the model parameters based on the mean. Compared with the asynchronous update mechanism, that is, each training node updates the model parameters based on its own gradient, the synchronous update mechanism can ensure that the loss decreases more steadily and avoid large jitters.

[0090] The synchronous update mechanism binds computation and communication to each gradient synchronization. When the synchronous update mechanism is adopted, when a single training node, training algorithm or network fails, the entire distributed training task will be interrupted. As the number of training nodes increases, the possibility of interruption increases. To this end, the relevant technology provides a solution for backing up fault files to restore training tasks.

[0091] like Figure 2A As shown in the figure, a typical fault handling solution is that the elastic agent performs fault detection. When a fault is detected, the corresponding process of the working process group in the same process group is notified to stop. The elastic agent reinitializes the cluster, and the training task creates a new ProcessGroup on the non-faulty cluster node, loads periodic fault files, such as checkpoint (CKPT) files, and continues the training job based on the CKPT file. This can prevent catastrophic failures caused by server maintenance or network problems, and will not lose training progress.

[0092] like Figure 2B As shown in the figure, another typical fault handling solution is to schedule components for fault detection. The fault types detected include chip faults and node faults (including parameter surface faults, such as cluster communication faults). When a fault is detected, the node where the fault is located is isolated, the training process is stopped, and the AI ​​framework saves the final CKPT file (the CKPT file when the fault occurs, also called the breakpoint CKPT file). The training task can be rescheduled to a non-faulty cluster node, the final CKPT file is loaded, and the training job continues.

[0093] However, in the above solution, after a fault is detected, no matter what kind of fault it is, all processes will exit (also called stopping, specifically closing the file descriptors opened by the process to release the resources occupied by the process). This results in the training system needing to restart the process to perform the corresponding task, such as recompiling, when different types of faults occur, which greatly affects the efficiency of resuming training tasks.

[0094] In view of this, the present application provides a fault handling method. The method is applied to a training system, the training system includes a first chip on the host side and multiple second chips on the device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. For example, the first subtask can be a subtask of compiling a computation graph, and the second subtask can be a subtask of executing an operator sequence based on the compilation result of the computation graph.

[0095] Specifically, the first chip executes the first subtask. When a fault occurs, the first chip saves the fault file before the second chip in normal status on the device side stops executing the second subtask. Then the first chip synchronizes the fault file to the rescheduled chip on the device side so that the rescheduled chip continues to execute the second subtask.

[0096] In this method, when a fault occurs, the second chip on the device side can stop executing the second subtask, and the first chip on the host side may not stop executing the first subtask. In this way, the execution result of the first subtask on the host side (for example, the compilation result, including but not limited to the compilation result of the calculation graph) can be reused to restore the training task, shortening the time to restore the training task and improving the efficiency of restoring the training task. Among them, the subtask can be executed by a process or a thread. In other words, the fault handling method of the present application can realize the exit of the process or thread on the device side, and the process or thread on the host side may not exit. The host side does not need to reapply for resources to repeat the same subtask, further shortening the time to restore the training task and improving the efficiency of restoring the training task.

[0097] Moreover, the method supports fine-grained recovery schemes according to different fault types. For recoverable chip faults, hot reset can be performed online to recover, and the training status can be restored by synchronizing the fault file with the normal chip; for unrecoverable node faults, only the subtasks on the faulty node are stopped (process or thread is stopped) and the subtasks are resumed on the new training node, and the training status is synchronized by synchronizing the fault file with the normal training node.

[0098] The fault handling method of the present application can be applied to various distributed training scenarios. For example, the fault handling method can be used in the scenario of distributed training of image recognition models, and different network structures (for example, different subnetworks or different network layers) of the image recognition model can be trained by different training nodes or multiple second chips of a training node. When a training node fails or a second chip (certain second chips) fails, the second chip on the device side can stop executing the second subtask, and the first chip on the host side may not stop executing the second subtask, wherein the first chip saves the fault file before the second chip in a normal state stops executing the second subtask, and then synchronizes the fault file to the rescheduled chip on the device side, so that the rescheduled chip continues to execute the second subtask.

[0099] The fault handling method can also be used in the scenario of distributed training text recognition model, and different network structures of the text recognition model can be trained by different training nodes or second chips respectively. The fault handling mechanism when a fault occurs can refer to the scenario of distributed training image recognition model. It will not be repeated here.

[0100] Figure 1A , Figure 1B The framework of the training system is introduced. The following uses the single-machine multi-card architecture as an example to introduce the hardware structure and software logic of the training system.

[0101] The training system of a single-machine multi-card architecture can be a server, and the training system of a multi-machine multi-card architecture can be a server cluster. A server cluster is a cluster formed by multiple servers, and the hardware structure of a server cluster can refer to a single server. Specifically, users can purchase or rent a server, which can be a cloud server or a physical server.

[0102] See also Figure 3The hardware structure diagram of a server shown in FIG. 3 includes a host 32 and a device 34. The host 32 and the device 34 are connected. The host 32 includes a processor and a memory (i.e., a host memory). The processor may be a CPU, and the memory may be a dual in-line memory module (DIMM). The DIMM may be a double data rate (DDR) type, for example, the memory may be a DDR4 DIMM. Figure 3 In the example, the host 32 includes 4 CPUs and 4 DDR4 DIMM groups, each CPU is connected to 1 DDR4 DIMM group, and each DDR4 DIMM group includes 8 DDR4 DIMMs. Multiple CPUs of the host 32 can be connected to form a hydra mesh.

[0103] Optionally, the host 32 further includes an interface, such as one or more of a Serial Advanced Technology Attachment (SATA) interface, a new generation non-volatile memory express (NVMe) interface, and a Gigabit Ethernet (GE) interface. The host 32 may also include a memory. The memory may include a SATA-supported memory or an NVMe-supported memory, such as a mechanical hard disk that supports SATA, or a solid state drive (SSD) that supports NVMe.

[0104] Device 34 includes an accelerator card. Figure 3 In the example of , the accelerator included in the device 34 may be an NPU. Figure 3 The device 34 includes 8 NPUs for illustration. In other possible implementations of the embodiment of the present application, the device 34 may also include more accelerator cards, such as more types of accelerator cards, or a greater number of accelerator cards.

[0105] Then, see Figure 4The framework diagram of the software deployed on the server shown in the figure, the user can install firmware 302 and driver 304 on the server 30. Among them, the firmware 302 is usually a program written in a read-only memory, which can directly control the hardware, interact with the hardware, and check whether the hardware has any errors. Driver 304 is specifically a small piece of code added to the operating system, which contains information about the hardware. When a computer program requests to interact with a certain hardware, the driver 304 can act as a converter of instructions between the hardware and the program that uses it. For example, the firmware 302 can control the device 34, interact with the device 34, and check whether the device 34 has any errors. The driver 304 can act as a converter of instructions between the device 34 and the program that uses it.

[0106] Furthermore, when the hardware architecture of the server 30 adopts a heterogeneous computing architecture (including a computing architecture of processing units using different types of instruction sets), the user can also install a heterogeneous computing framework 306 on the server 30. In a distributed training scenario, the heterogeneous computing framework 306 can be a heterogeneous computing framework for neural networks (Compute Architecture for Neuro Net, CANN). CANN can support users to quickly build AI applications by providing a multi-level programming interface. Among them, AI applications refer to applications built based on trained AI models. It should be noted that the heterogeneous computing framework 306 is an optional framework. The above framework is not installed on the server 30, and the fault handling method of the embodiment of the present application can also be executed. The role of the above framework is to improve the efficiency of building AI applications.

[0107] Then, the user can install the deep learning framework 308 on the server 30. The deep learning framework 308 is used to compile the computational graph of the model and automatically implement the gradient calculation in the computational graph. In this way, when performing distributed training, the graph compilation results (such as operator sequences) can be executed to perform related calculations for distributed training. Depending on the compilation method, deep learning frameworks can be divided into frameworks that support static compilation and frameworks that support dynamic compilation. Users can choose to install one or more deep learning frameworks 308 on the server 30 according to business needs. In some embodiments, the deep learning framework 308 may not be installed on the server 30. In this case, the server 30 can implement the model from scratch using a programming language such as Python.

[0108] The user can also install a scheduler 310 on the server 30. The scheduler 310 is used to schedule training tasks to achieve distributed training. Among them, the scheduler 310 can be a distributed scheduling component (distributed scheduling framework) or an AI development platform. In some embodiments, the distributed scheduling component or AI development platform can be a self-developed component or development platform, or it can be a third-party distributed scheduling component or a third-party AI development platform.

[0109] Furthermore, the user can also deploy a model library 312 on the server 30. The model library 312 includes an AI model implemented using a unified framework, and the AI ​​model has standardized parameters and APIs. Among them, the AI ​​model includes configuration items that can be reused, and the above-mentioned configuration items that can be reused are defined in the unified framework. This can reduce the configuration work of the AI ​​model.

[0110] The scheduler 310 may start a training task on the training node, the training code may be downloaded from the model library 312, and the configuration file may be read from the training code downloaded from the model 312. When a training node fails or one or some second chips (such as NPU) in the training node fails, the scheduler 310 may, upon detecting a failure, notify the first chip (such as CPU) to save a failure file before the second chip in a normal state on the device side stops executing the second subtask, and the failure file is used to be synchronized by the first chip to the chip that is heavily scheduled on the device side.

[0111] The scheduler 310 may be provided to the user in the form of a code package, and installed or deployed by the user. The scheduler 310 may also be provided to the user in the form of a cloud service. Specifically, the cloud service provider may provide an application programming interface (API) for fault handling, and the training system may call the API to implement the fault handling method of the present application.

[0112] In order to make the technical solution of the present application clearer and easier to understand, the fault handling method of the present application is introduced below with reference to the accompanying drawings.

[0113] See also Figure 5 A flowchart of a fault handling method is shown, and the method is applied to a training system, and the training system includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute a training task, and the training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. For example, the first subtask can be a compilation subtask, and the execution result of the first subtask can be a compilation result. The second subtask is an operator sequence (kernel sequence) execution subtask. The execution of the operator sequence depends on the compilation result on the host side. The method includes the following steps:

[0114] S502, the first chip executes the first subtask. When a failure occurs, execute S504.

[0115] The first subtask is a subtask that executes host-side logic in the training task. The training task is specifically a computational graph of a loop-executed model. The computational graph is a high-level representation that provides a global view of the operator without specifying the implementation details. In order to reuse computing resources as much as possible, improve computing efficiency, and shorten model training time, it is usually necessary to compile and optimize the computational graph. Compilation optimization may include high-level data writing to generate an optimized computational graph, and then perform operator-level optimization to generate efficient code for the fused operators in the computational graph. The operator is declared by a tensor expression language, and then the cost model is used to search for the optimal code of the operator from the potential optimization set, and the generated code is packaged into a deployable module, such as a kernel sequence. The above-mentioned compilation optimization process can be executed on the host side, and the kernel sequence can be executed on the device side. Based on this, the first subtask may include a compilation optimization subtask (also called a compilation subtask). The second subtask may include a subtask that executes a kernel sequence.

[0116] When looping the computation graph of the model, it is usually necessary to read the data set and perform data preprocessing on the data in the data set. The data preprocessing operation may be different depending on the data type of the data set. For example, the data set includes image data, and the data preprocessing operation may include cropping, scaling, and rotation. For another example, the data set includes text data, and the data preprocessing operation may include word segmentation. The above data preprocessing may be performed on the host side. Accordingly, the first subtask may also include a data preprocessing subtask.

[0117] The first chip can execute the first subtask through a process or a thread. Among them, a process or a thread is an application running in memory. A process is an encapsulation of a runtime program and is the basic unit of resource scheduling and allocation. A thread usually depends on a process and is used to achieve concurrency within the process. A process has independent memory, and threads share the memory of a process. Taking the first subtask as a compilation optimization subtask as an example, the first chip can execute the compilation optimization subtask through one or more processes. In some possible implementations, the first chip can also execute the compilation optimization subtask through one or more threads.

[0118] S504: The first chip saves the fault file before the second chip in a normal state on the device side stops executing the second subtask.

[0119] The failure of the training system can be a recoverable chip failure or an unrecoverable node failure. Among them, a recoverable chip failure refers to a chip failure that can be recovered by hot reset. An unrecoverable node failure refers to a failure that is difficult to recover on the original node. An unrecoverable node failure may include but is not limited to all chip failures or parameter plane failures on the device side of the training node, among which a parameter plane failure may be a cluster communication failure.

[0120] Since the training system includes multiple second chips. When there is a second chip failure, the first chip can save the fault file before the second chip in a normal state stops executing the second subtask. In some possible implementations, for a multi-machine multi-card training system, a recoverable chip failure or an unrecoverable node failure occurs in a training node in the training system, and the first chip of a normal node (a training node that has not failed) in the training system can save the fault file before the second chip of the normal node stops executing the subtask. In a specific implementation, the first chip of the normal node calls the interface of the heterogeneous computing architecture component through an AI framework (such as a deep learning framework), so that the heterogeneous computing architecture component exits, the process or thread based on the heterogeneous computing architecture component stops, and the second subtask stops executing. Before the second subtask stops executing, the AI ​​framework can save the fault file. In other possible implementations, for a single-machine multi-card training system, an unrecoverable chip failure occurs in a training node, and the first chip of the training node can save the fault file before the second chip that has not failed in the node stops executing the second subtask. In some possible implementations, the first subtask may not stop executing, in other words, the process or thread used to execute the first subtask may not exit.

[0121] Among them, when saving the fault file, the first chip can read the fault file from the device side and save the fault file in the memory of the host side without persisting the fault file. In other words, the first chip may not write the fault file to the disk, for example, the first chip does not write the fault file to the Object Storage Service (OBS) or the Open Computing Kit (OCK). Furthermore, when saving the fault file, the first chip can save the terminal fault file, such as the terminal CKPT (also called the breakpoint CKPT), so as to avoid the loss of training progress.

[0122] S506: The first chip synchronizes the fault file with the chip that is rescheduled by the device, so that the rescheduled chip continues to execute the second subtask.

[0123] When a training task is interrupted due to a fault, the training task can be rescheduled to a chip in a normal state, which is the rescheduled chip. The first chip synchronizes the fault file to the rescheduled chip so that the rescheduled chip continues to execute the second subtask, thereby restoring the training task.

[0124] For recoverable chip failures, the rescheduled chip can be the second chip whose state is restored to normal after reset. For example, the training system is a training node of a single-machine multi-card architecture, and the training node includes a first chip and multiple second chips, and a target chip among the multiple second chips fails. The rescheduled chip can be the target chip whose state is restored to normal after reset. Accordingly, the first chip can synchronize the fault file to the second chip (such as the target chip) whose state is restored to normal after reset.

[0125] For an unrecoverable node failure, the rescheduled chip can be a newly added third chip. For example, the training system includes a first training node and a second training node, and each training node includes at least one first chip and at least one second chip. If an unrecoverable node failure occurs in the first training node, the training task can be rescheduled to the newly added third training node, and the third chip belongs to the newly added third training node. Accordingly, the first chip can synchronize the fault file to the third chip (or the third training node).

[0126] It should be noted that the first chip of the first training node can also save the execution result of the first subtask, for example, cache the compilation result, and the first chip of the third training node can load the execution result of the first subtask without recompiling.

[0127] When the fault file is stored in memory (such as host memory), the first chip can read the fault file from the memory, and then synchronize the fault file to the chip scheduled by the device through the cluster interface. Model information (such as model weights) is no longer serialized and deserialized through CKPT, avoiding the impact of model scale, input output (IO) and bandwidth on training task recovery, but point-to-point state synchronization is performed through the cluster mechanism.

[0128] When the first chip saves the terminal fault file such as the terminal CKPT, the first chip can synchronize the training state (or training progress) at the time of the fault, thereby avoiding the loss of the training process caused by the periodic CKPT.

[0129] Based on the above description, it can be known that the fault handling method of the present application supports that when a fault occurs, the second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the second subtask. In this way, the execution result of the first subtask on the host side (for example, the compilation result, including but not limited to the compilation result of the calculation graph) can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0130] The above introduces the fault handling method from the perspective of the first chip. The following introduces the fault handling method of the present application from the perspective of the scheduler.

[0131] See also Figure 6 A flowchart of a fault handling method is shown, the method is applied to a scheduler, the scheduler is used to handle faults when a training system fails, the training system includes a first chip on the host side and multiple second chips on the device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. The method includes the following steps:

[0132] S602: The scheduler performs fault detection on the training system.

[0133] Specifically, the scheduler can perform fault detection on the second chip in the training system or the training node where the second chip is located. Specifically, the scheduler can perform fault detection on the second chip through a heartbeat message. The second chip in the training system can periodically send a heartbeat message to the scheduler, and the scheduler determines whether the second chip has a fault based on the reception of the heartbeat message. For example, if the scheduler does not receive a heartbeat message from the second chip for N consecutive cycles, it means that the second chip or the node where the second chip is located has a fault. Among them, the scheduler can determine the fault type in combination with the reception of the heartbeat message of other second chips in the same training node.

[0134] For the K second chips of the same training node, if the scheduler does not receive the heartbeat messages of the K second chips of the training node, for example, if the heartbeat messages of the K second chips are not received for N consecutive cycles, the scheduler can determine that the training node has failed. For the K second chips of the same training node, if the scheduler does not receive the heartbeat messages of some second chips, but receives the heartbeat messages of the remaining second chips, the scheduler can determine that the second chip that did not receive the corresponding heartbeat message has failed.

[0135] It should be noted that the heartbeat message is only one implementation method of fault detection for the training system. In other possible implementation methods of the embodiment of the present application, the scheduler can also implement fault detection in other ways. For example, the second chip can be detected by the scheduler by reading the status code of the second chip.

[0136] S604: When a fault is detected, the scheduler notifies the first chip to save the fault file before the second chip in a normal state on the device side stops executing the second subtask.

[0137] The fault file is used to be synchronized by the first chip to the chip rescheduled by the device side. In this way, the rescheduled chip can continue to execute the second subtask based on the fault file synchronized by the first chip.

[0138] In some possible implementations, for recoverable chip failures, the scheduler can also reset the faulty chip on the device side, for example, by performing a hot reset online. Accordingly, the chip that the device focuses on scheduling includes the second chip whose state returns to normal after reset.

[0139] In some other possible implementations, for an unrecoverable node failure, the chip that the device focuses on scheduling includes a newly added third chip. The scheduler can synchronize the information of the newly added third chip to the second chip in normal state. The information of the third chip is used to establish a link with other second chips participating in the training. In a specific implementation, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, and an unrecoverable node failure occurs in the first training node. The training task can be rescheduled to the third training node. The third chip is a device-side chip of the third training node. The scheduler can synchronize the information of the third chip to at least one second chip in the second training node. Among them, the information of the third chip includes at least one of the address information and resource configuration information of the third training node to which the third chip belongs. The address information of the third training node may be the IP address of the third training node, or other addresses that can be used to build a link. The resource configuration information may be a sorting table of the third chip, which is used to record the sequence number of the third chip occupied by the process or thread that executes the subtask.

[0140] Based on the above description, it can be known that the fault handling method of the present application notifies the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask when the scheduler detects a fault. The second chip on the device side can stop executing the second subtask, and the first chip on the host side can continue to execute the second subtask. In this way, the execution result of the first subtask on the host side can be reused to resume the training task, shortening the time to resume the training task and improving the efficiency of resuming the training task.

[0141] The fault detection method of the present application is introduced above from the perspectives of the first chip and the scheduler respectively. The fault handling method of the present application is described below in combination with specific application scenarios.

[0142] First, see Figure 7 A schematic diagram of a scenario of using the original node to resume the training task is shown in Figure 7 As shown, the training system includes multiple training nodes, and the training nodes include a first chip on the host side and a second chip (such as NPU / GPU) on the device side. Figure 7 Taking the example that the device side of each training node includes 8 second chips, when a recoverable chip failure occurs in the second chips numbered 0 to 3 of a training node, the training task can be restored at the original node.

[0143] Among them, the training node that fails is called an abnormal node, and the node that does not fail is called a normal node. The abnormal node and the normal node perform corresponding operations respectively to restore the training task on the original node.

[0144] For abnormal nodes, the distributed scheduling framework (such as a scheduler) deployed on the host side detects a chip failure and resets the faulty chip, such as the second chip (rank 0 to 3). Among them, chip failure or chip reset can trigger the exit of the heterogeneous computing architecture component on the device side. Correspondingly, the process on the device side can exit and the second subtask stops executing. The process corresponding to the host-side AI framework does not exit, so there is no need to recompile the calculation graph and wait for the chip to be reset and the heterogeneous computing architecture component to exit. The host-side AI framework detects that the state of the second chip has returned to normal after being reset, and can trigger the link building process to re-establish the link with the normal node, for example, through the collective communication library. Then the host-side AI framework re-executes the calculation graph and waits for the fault file synchronization to complete. When the synchronization is completed, the abnormal node can continue the training job. For example, the second chip of the abnormal node can continue to execute the second subtask.

[0145] For normal nodes, the distributed scheduling framework deployed on the host side can notify the AI ​​framework to call the heterogeneous computing architecture component interface, so that the heterogeneous computing architecture component exits, the process or thread based on the heterogeneous computing architecture component stops, and the second subtask stops executing. Among them, the AI ​​framework saves the fault file before the process or thread based on the heterogeneous computing architecture component stops (the second subtask stops executing). After saving the fault file, the heterogeneous computing architecture component can be quickly launched. The process or thread corresponding to the AI ​​framework does not exit, so there is no need to recompile. The host-side AI framework detects that the chip status is normal and can rebuild the chain. Then the AI ​​framework re-executes the calculation graph, and the AI ​​framework synchronizes the fault file to the recovered node through the cluster communication interface according to the fault file recovery strategy. Normal nodes can continue training jobs.

[0146] Next, see Figure 8 A schematic diagram of a scenario of using a new node to resume a training task is shown in Figure 8 As shown, the training system includes multiple training nodes, and the training nodes include a first chip on the host side and a second chip (such as NPU / GPU) on the device side. Figure 8 The device side of each training node includes 8 second chips as an example. When an unrecoverable node failure occurs in a training node (all second chips of the training node fail), the training task can be restored in a new node.

[0147] The abnormal nodes, normal nodes, and new nodes perform corresponding operations respectively to resume the training task on the new node.

[0148] For abnormal nodes, the distributed scheduling framework detects a failure and the heterogeneous computing architecture component can exit. The AI ​​framework can cache the compilation results based on the return value (abnormal return value) of the exit of the heterogeneous computing architecture component. Then the process of the AI ​​framework can exit.

[0149] For normal nodes, the distributed scheduling framework notifies the AI ​​framework to call the heterogeneous computing architecture component interface to exit the heterogeneous computing architecture component. Before the heterogeneous computing architecture component exits, the AI ​​framework saves the fault file to the host side. After saving the file, the heterogeneous computing architecture component can exit quickly. The process or thread based on the heterogeneous computing architecture component can exit. The second subtask stops executing. The AI ​​framework on the host side does not exit, so there is no need to recompile. The AI ​​framework on the host side checks that the chip status is normal and can rebuild the chain. Then the AI ​​framework re-executes the calculation graph. The AI ​​framework synchronizes the fault file to the recovered node through the cluster communication interface according to the fault file recovery strategy. Normal nodes can continue training jobs.

[0150] For new nodes, the distributed scheduling framework synchronizes the IP and rank table of the new node to each container group POD in each cluster. The AI ​​framework obtains the IP of the new node and updates the rank table. Then, the AI ​​framework loads the compilation results cached by the abnormal node, for example, using OCK to accelerate the loading of the compilation results cached by the abnormal node. Next, the AI ​​framework checks that the chip status is normal and rebuilds the chain. The AI ​​framework can then re-execute the calculation graph and wait for the fault file synchronization to complete. After the fault file synchronization is complete, the newly added node can continue the training job.

[0151] Based on the above fault handling method, the present application also provides a fault handling device. Fig. 9As shown, the fault handling device 900 is deployed on the first chip on the host side of the training system. The first chip and multiple second chips on the device side of the training system are used to collaboratively execute the training task, which includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The fault handling device 900 includes:

[0152] A task execution module 902, configured to execute the first subtask;

[0153] A file saving module 904, configured to save the fault file when a fault occurs and before the second chip in a normal state on the device side stops executing the second subtask;

[0154] The file synchronization module 906 is used to synchronize the fault file to the rescheduled chip of the device, so that the rescheduled chip can continue to execute the second subtask.

[0155] Exemplarily, the task execution module 902, the file saving module 904, and the file synchronization module 906 may be implemented by hardware or by software.

[0156] Among them, when implemented by software, the task execution module 902, the file saving module 904, and the file synchronization module 906 can be applications running on a computing device, such as a computing engine, etc. The application can also be virtualized through a virtualization service to provide it to users. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only used as specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0157] When implemented by hardware, the task execution module 902, the file saving module 904, and the file synchronization module 906 may include at least one computing device, such as a server, etc. Alternatively, the task execution module 902, the file saving module 904, and the file synchronization module 906 may also be implemented by using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0158] In some possible implementations, the fault includes a recoverable chip fault or an unrecoverable node fault.

[0159] In some possible implementations, the fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

[0160] In some possible implementations, the training system includes a training node, the training node includes a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips;

[0161] The file synchronization module 906 is specifically used for:

[0162] The fault file is synchronized to the target chip whose state is restored to normal after reset.

[0163] In some possible implementations, the fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

[0164] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the device 900 further includes an execution result saving module 908 of the first chip deployed in the first training node and an execution result loading module 909 of the first chip deployed in the third training node;

[0165] The execution result saving module 908 is used to save the execution result of the first subtask;

[0166] The execution result loading module 909 is used to load the execution result of the first subtask.

[0167] The execution result saving module 908 and the execution result loading module 909 can be implemented by software or hardware. When implemented by software, the execution result saving module 908 and the execution result loading module 909 can be applications running on a computing device, such as a computing engine, etc. When implemented by hardware, the execution result saving module 908 and the execution result loading module 909 can include at least one computing device, such as a server, etc. Alternatively, the execution result saving module 908 and the execution result loading module 909 can also be devices implemented by ASIC or PLD, etc.

[0168] In some possible implementations, the file synchronization module 906 is specifically used to:

[0169] The fault file is obtained from the memory, and the fault file is synchronized to the chip focused on scheduling of the device through the cluster communication interface.

[0170] In some possible implementations, the subtask is executed in a process or thread manner.

[0171] Based on the above fault handling method, the present application also provides a scheduler. The scheduler is used to handle faults when a training system fails. The training system includes a first chip on the host side and multiple second chips on the device side. The first chip and the multiple second chips are used to collaboratively execute training tasks. The training task includes a first subtask and multiple second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. Fig.10 As shown, the scheduler 1000 includes:

[0172] A fault detection module 1002 is used to perform fault detection on the training system;

[0173] Notification module 1004 is used to, when a fault is detected, notify the first chip to save a fault file before the second chip in normal state on the device side stops executing the second subtask, and the fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

[0174] Exemplarily, the fault detection module 1002 and the notification module 1004 may be implemented by hardware or software.

[0175] When implemented by software, the fault detection module 1002 and the notification module 1004 may be application programs running on a computing device. When implemented by hardware, the fault detection module 1002 and the notification module 1004 may include at least one computing device, such as a server, etc. Alternatively, the fault detection module 1002 and the notification module 1004 may also be devices implemented by ASIC or PLD, etc. Alternatively, the fault detection module 1002 and the notification module 1004 may also be a processing unit in the above devices.

[0176] In some possible implementations, the fault is a recoverable chip fault, and the scheduler 1000 further includes:

[0177] A reset module 1006, used to reset the faulty chip on the device side;

[0178] The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

[0179] The reset module 1006 may be implemented by hardware or software.

[0180] When implemented by software, the reset module 1006 may be an application program running on a computing device. When implemented by hardware, the reset module 1006 may include at least one computing device, such as a server, etc. Alternatively, the reset module 1006 may also be a device implemented by ASIC or PLD, etc. Alternatively, the reset module 1006 may also be a processing unit in the above devices.

[0181] In some possible implementations, the training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to synchronize the first chip to the target chip whose state is restored to normal after reset.

[0182] In some possible implementations, the fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the scheduler 1000 further includes:

[0183] The synchronization module 1008 is used to synchronize the information of the newly added third chip with the second chip in normal state.

[0184] Similarly, the synchronization module 1008 may be implemented by hardware, or may be implemented by software.

[0185] When implemented by software, the synchronization module 1008 may be an application program running on a computing device. When implemented by hardware, the synchronization module 1008 may include at least one computing device, such as a server, etc. Alternatively, the synchronization module 1008 may also be a device implemented by ASIC or PLD, etc. Alternatively, the synchronization module 1008 may also be a processing unit in the above devices.

[0186] In some possible implementations, the training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, and the unrecoverable node failure occurs in the first training node;

[0187] The synchronization module 1008 is specifically used for:

[0188] The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

[0189] The present application also provides a chip. The chip may include a processor and a memory. The memory stores computer-readable instructions; the processor executes the computer-readable instructions to execute the fault handling method in the aforementioned embodiment. For example, the chip may be a first chip, and the processor of the first chip executes the computer-readable instructions to execute the fault handling method executed by the first chip in the aforementioned embodiment.

[0190] The present application also provides a scheduler. The scheduler may include a processor and a memory. The memory stores computer-readable instructions; the processor executes the computer-readable instructions to execute the fault handling method executed by the scheduler in the above embodiment, for example Figure 6 The method of the illustrated embodiment.

[0191] The present application also provides a training system. The training system may include a first chip on the host side and multiple second chips on the device side. Among them, the training system may be a single-machine multi-card architecture, that is, the training system may be a training node, such as a training server. The training node may include a host (the host includes a first chip) and multiple acceleration cards (i.e., a second chip), and the first chip on the host side and the multiple second chips on the device side in the training node can collaboratively perform distributed training tasks. In some examples, the training system may also be a multi-machine multi-card architecture, that is, the training system may be a cluster formed by multiple training nodes. Each of the multiple training nodes includes a first chip on the host side and at least one second chip on the device side. Multiple training nodes can collaboratively perform distributed training tasks. Among them, the first chip in the training system is used to execute computer-readable instructions to execute the fault handling method of the aforementioned embodiment.

[0192] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the execution of the fault handling method of the aforementioned embodiment.

[0193] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned fault handling method.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault handling method, characterized in that: Applied to a training system, the training system includes a first chip on a host side and multiple second chips on a device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the method includes: The first chip performs the first subtask; When a fault occurs, the first chip saves the fault file before the second chip in a normal state on the device side stops executing the second subtask; The first chip synchronizes the fault file to the rescheduled chip of the device, so that the rescheduled chip continues to execute the second subtask.

2. The method according to claim 1, characterized in that The fault includes a recoverable chip fault or an unrecoverable node fault.

3. The method according to claim 1 or 2, characterized in that: The fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

4. The method according to any one of claims 1 to 3, characterized in that: The training system comprises a training node, the training node comprises a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips; The first chip synchronizes the fault file to the chip that focuses on scheduling the device, including: The first chip synchronizes the fault file with the target chip whose state is restored to normal after reset.

5. The method according to claim 1 or 2, characterized in that: The fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

6. The method according to claim 5, characterized in that The training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the method further includes: The first chip of the first training node stores the execution result of the first subtask; The first chip of the third training node loads the execution result of the first subtask.

7. The method according to any one of claims 1 to 6, characterized in that: The first chip synchronizes the fault file to the chip that focuses on scheduling the device, including: The first chip obtains the fault file from the memory, and synchronizes the fault file with the chip focused on scheduling of the device through the cluster communication interface.

8. The method according to any one of claims 1 to 7, characterized in that: The subtasks are executed in a process or thread manner.

9. A fault handling method, characterized in that: Applied to a scheduler, the scheduler is used to perform fault processing when a training system fails, the training system includes a first chip on a host side and multiple second chips on a device side, the first chip and the multiple second chips are used to collaboratively execute a training task, the training task includes a first subtask and multiple second subtasks, the execution of the second subtask depends on the execution result of the first subtask, and the method includes: Performing fault detection on the training system; When a fault is detected, the first chip is notified to save a fault file before the second chip in normal state on the device side stops executing the second subtask. The fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

10. The method according to claim 9, characterized in that The fault is a recoverable chip fault, and the method further includes: Resetting the faulty chip on the device side; The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

11. The method according to claim 9 or 10, characterized in that: The training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to be synchronized by the first chip to the target chip whose state is restored to normal after reset.

12. The method according to claim 9, characterized in that The fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the method further includes: The information of the newly added third chip is synchronized with the second chip in normal state.

13. The method according to claim 12, characterized in that The training system comprises a first training node and a second training node, each training node comprises at least one first chip and at least one second chip, and the first training node has the unrecoverable node failure; The step of synchronizing information of the newly added third chip to the second chip in a normal state includes: The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

14. The method according to any one of claims 9 to 13, characterized in that: The subtasks are executed in a process or thread manner.

15. A fault handling device, characterized in that: The apparatus is deployed on a first chip on a host side of a training system, and the first chip and a plurality of second chips on a device side of the training system are used to collaboratively execute a training task, wherein the training task includes a first subtask and a plurality of second subtasks, and the execution of the second subtask depends on the execution result of the first subtask. The apparatus includes: A task execution module, used for executing the first subtask; A file saving module, configured to save the fault file when a fault occurs and before the second chip in a normal state on the device side stops executing the second subtask; The file synchronization module is used to synchronize the fault file to the rescheduled chip of the device, so that the rescheduled chip can continue to execute the second subtask.

16. The device according to claim 15, characterized in that The fault includes a recoverable chip fault or an unrecoverable node fault.

17. The device according to claim 15 or 16, characterized in that The fault is a recoverable chip fault, and the chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

18. The device according to any one of claims 15 to 17, characterized in that The training system comprises a training node, the training node comprises a first chip and a plurality of second chips, and the fault is a target chip fault among the plurality of second chips; The file synchronization module is specifically used for: The fault file is synchronized to the target chip whose state is restored to normal after reset.

19. The device according to claim 15 or 16, characterized in that The fault is an unrecoverable node fault, and the chip that the device focuses on scheduling includes a newly added third chip.

20. The device according to claim 19, characterized in that The training system includes a first training node and a second training node, each training node includes at least one first chip and at least one second chip, the first training node has the unrecoverable node failure, the third chip belongs to a newly added third training node, and the device also includes an execution result saving module of the first chip deployed in the first training node and an execution result loading module of the first chip deployed in the third training node; The execution result saving module is used to save the execution result of the first subtask; The execution result loading module is used to load the execution result of the first subtask.

21. The device according to any one of claims 15 to 20, characterized in that The file synchronization module is specifically used for: The fault file is obtained from the memory, and the fault file is synchronized to the chip focused on scheduling of the device through the cluster communication interface.

22. The device according to any one of claims 15 to 21, characterized in that The subtasks are executed in a process or thread manner.

23. A scheduler, characterized in that: The scheduler is used to handle a fault when a training system fails. The training system includes a first chip on a host side and multiple second chips on a device side. The first chip and the multiple second chips are used to collaboratively execute a training task. The training task includes a first subtask and multiple second subtasks. The execution of the second subtask depends on the execution result of the first subtask. The scheduler includes: A fault detection module, used for performing fault detection on the training system; The notification module is used to, when a fault is detected, notify the first chip to save the fault file before the second chip in normal state on the device side stops executing the second subtask, and the fault file is used to be synchronized by the first chip to the chip that is re-scheduled on the device side.

24. The scheduler according to claim 23, characterized in that The fault is a recoverable chip fault, and the scheduler further includes: A reset module, used to reset the faulty chip on the device side; The chip that the device focuses on scheduling includes a second chip whose state returns to normal after being reset.

25. The scheduler according to claim 23 or 24, characterized in that: The training system includes a training node, the training node includes a first chip and multiple second chips, the fault is a target chip fault among the multiple second chips, and the fault file is used to be synchronized by the first chip to the target chip whose state is restored to normal after reset.

26. The scheduler according to claim 23, characterized in that The fault is an unrecoverable node fault, the chip that the device focuses on scheduling includes a newly added third chip, and the scheduler further includes: The synchronization module is used to synchronize the information of the newly added third chip with the second chip in a normal state.

27. The scheduler according to claim 26, characterized in that The training system comprises a first training node and a second training node, each training node comprises at least one first chip and at least one second chip, and the first training node has the unrecoverable node failure; The synchronization module is specifically used for: The information of the third chip is synchronized to at least one second chip in the second training node, where the information of the third chip includes at least one of address information and resource configuration information of a third training node to which the third chip belongs.

28. The scheduler according to any one of claims 23 to 27, characterized in that: The subtasks are executed in a process or thread manner.

29. A chip, characterized in that: The chip comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the chip executes the method according to any one of claims 1 to 8.

30. A scheduler, characterized in that: The system comprises a processor and a memory, wherein the memory stores computer-readable instructions; the processor executes the computer-readable instructions so that the scheduler executes the method according to any one of claims 9 to 14.

31. A training system, characterized in that: The training system comprises a first chip on a host side and a plurality of second chips on a device side, wherein the first chip is used to execute the method as claimed in any one of claims 1 to 8.

32. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 14.

Images