Distributed training method and device of AI model and computing equipment cluster

Abstract

The embodiment of the invention provides a distributed training method and device of an AI model and a computing device cluster, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a fault reason of a faulted first computing node, and matching a corresponding training task recovery mode according to the fault reason of the faulted first computing node; and through at least one training task recovery mode of training task restarting, node rescheduling or model restarting, AI model distributed training is continued. Thus, corresponding training task recovery modes are selected according to different fault reasons, and the rescheduling time after operation failure caused by direct isolation of fault nodes is avoided. Therefore, the training time of the AI model can be shortened. Besides, based on the fault information, the training task recovery mode matched with the fault reason is automatically identified, the dependence on manual recovery is reduced, and thus the training task recovery efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and more specifically to a distributed training method, apparatus and computing device cluster for an AI model. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI models are becoming increasingly larger, and the amount of training data for training AI models is also increasing.

[0003] As the scale and duration of AI model training increase, the probability of hardware and software failures during training also increases significantly, leading to a greater frequency of training interruptions. Manually restarting tasks after training interruptions begins to reveal many drawbacks. For example, training interruptions may not be detected promptly at night, and overall resource utilization may decrease considerably.

[0004] Currently, faulty training nodes are identified through management nodes. Then, node isolation is achieved manually, and a new node is restarted to replace the faulty one, thus resuming the training task. However, because this technology relies on manual repair, it requires a significant amount of manpower to handle faults, resulting in a long job recovery time and increasing model training duration. Summary of the Invention

[0005] This application provides a distributed training method, apparatus, and computing device cluster for AI models to improve the problem of increased job recovery time caused by reliance on manual repair in AI model training tasks.

[0006] Firstly, embodiments of this application provide a distributed training method for an AI model. During AI model training, a first subset of a computing cluster is used for distributed training of the AI ​​model. Each computing node in the first subset executes a training task for distributed AI model training. During distributed training, when a first computing node in the first subset fails, the cause of the failure is obtained. The cause of failure includes at least one of hardware failure or software failure. Model training is resumed according to the training task recovery method corresponding to the cause of failure of the first computing node. Optionally, the training task recovery method indicates at least one model training recovery scheme, either model restart or task restart.

[0007] Compared to directly isolating the faulty node, this embodiment matches the corresponding training task recovery method based on the cause of the first faulty computing node. Furthermore, it selects at least one model training recovery scheme—either model restart or task restart—based on different fault causes to restore the training task. This approach selects appropriate training task recovery methods for different fault causes, avoiding the rescheduling time caused by directly isolating the faulty node. This shortens the training time of the AI ​​model. In addition, automatically identifying the training task recovery method matching the fault cause reduces reliance on manual repair, thereby improving the efficiency of training task recovery.

[0008] In one possible implementation, the hardware failure includes accelerator card failure or other hardware failures besides accelerator card failure; the software failure includes failures other than hardware failures.

[0009] Based on this possible implementation method, corresponding training task recovery methods are obtained by considering the different fault ranges of accelerator card failure, hardware failure other than accelerator card failure, and software failure. This improves the compatibility between training task recovery methods and faults, thereby increasing job recovery efficiency.

[0010] In one possible implementation, model training is resumed according to the training task recovery method corresponding to the cause of failure of the first computing node. Specifically, when the cause of failure indicates a hardware failure, the training task executed by the first computing node is restarted on the second computing node or the first computing node. The second computing node is a computing node in the computing cluster other than the first subset.

[0011] In this possible implementation, when a hardware failure occurs on the first computing node, the training task can be restarted either on the first computing node or on the second computing node. This reduces job recovery time, avoids manually detected job interruptions, and thus minimizes the time required to recover interrupted jobs.

[0012] In one possible implementation, the specific implementation is as follows: before restarting the training task executed by the first computing node on the second computing node or the first computing node, a stop training notification is sent to the first computing node and a stop training notification is sent to the third computing node; the stop training notification is used to indicate that the training task to be stopped is stopped; the third computing node is a computing node in the first subset that has not experienced a failure.

[0013] In this way, the consistency of the training process is ensured by pausing the training process on computing nodes that have not experienced failures.

[0014] In one possible implementation, the specific implementation is as follows: after sending a stop training notification to the first computing node and a stop training notification to the third computing node, when the hardware failure is an accelerator card failure, a chip repair instruction is sent to the first computing node; the chip repair instruction is used to instruct the accelerator card in the first computing node to perform a chip repair operation; when the accelerator card in the first computing node successfully performs the chip repair operation, a continue training notification is sent to both the first computing node and the third computing node; the continue training notification is used to instruct the resumption of the distributed training task of the AI ​​model.

[0015] Based on this possible implementation, since the first computing node retains the environment information of the training task, this environment information can be directly used after restarting the training task on the first computing node, resulting in minimal recovery time. This avoids the problem of increased training job interruption time caused by reconfiguring the training task environment. It can minimize the time required to resume interrupted training jobs.

[0016] In one possible implementation, the specific implementation is as follows: after sending a chip repair instruction to the first computing node, if the chip repair operation performed by the accelerator card in the first computing node fails, the training task executed by the first computing node is restarted on the second computing node.

[0017] Thus, if the chip repair operation in the first computing node fails, the training task executed on the first computing node is restarted on the second computing node. This improves the task recovery rate and ensures that the training job can be resumed.

[0018] In one possible implementation, the specific implementation is as follows: after sending a stop training notification to the first computing node and a stop training notification to the third computing node, if the hardware failure is a hardware failure other than an accelerator card failure, the training task executed by the first computing node is restarted on the second computing node; and a continue training notification is sent to the second computing node and the third computing node.

[0019] Based on this possible implementation, when a hardware failure occurs on the first computing node other than an accelerator card failure, the job containers on the failed first computing node are rescheduled, and the CKPT (check point) file and data are re-downloaded, thus avoiding human-perceived job interruption and minimizing the time required to recover the interrupted job.

[0020] In one possible implementation, the specific implementation is as follows: after sending the continue training notification to the second and third computing nodes, the first computing node is isolated.

[0021] In this way, by isolating the faulty computing node, the problem of increased network overhead caused by communication between the node and the faulty node can be avoided.

[0022] In one possible implementation, the process is as follows: After restarting the training task executed by the first computing node on the second computing node, if the second computing node fails to restart, the training tasks executed in the first subset are restarted on a second subset. The second subset includes multiple computing nodes in the computing cluster other than the first subset. When the second computing node restarts successfully, a notification to continue training is sent to both the second and third computing nodes.

[0023] Based on this possible implementation, if the second computing node fails to restart, the model is restarted. This ensures that the training job can be resumed.

[0024] In one possible implementation, model training is resumed according to the training task recovery method corresponding to the cause of failure of the first computing node. Specifically, when the cause of failure indicates a software failure, the training tasks executed in the first subset are restarted on the second subset. The second subset includes multiple computing nodes in the computing cluster other than the first subset.

[0025] In this way, if a software failure occurs on the first computing node, the training job can be automatically restored by reallocating the computing node to execute the training job.

[0026] In one possible implementation, the training task executed in the first subset is restarted on the second subset. Specifically, this is achieved by sending a stop training notification to each computing node in the first subset. The stop training notification is used to indicate that the training task to be executed has been stopped.

[0027] A "Continue Training" notification is sent to each computing node in the second subset to restart the training task executed in the first subset on the second subset. The "Continue Training" notification is used to instruct each computing node in the second subset to resume the training process corresponding to the distributed training task of the AI ​​model.

[0028] In this way, the consistency of the training process is ensured by pausing the training process on computing nodes that have not experienced failures.

[0029] In one possible implementation, the specific implementation is as follows: The first subset includes fault handling nodes. The computing cluster includes management nodes; wherein, the management nodes are the computing nodes in the computing cluster other than the first subset. The fault cause sent by the first computing node is obtained through the fault handling nodes. The exit code sent by the first computing node is obtained through the management node, and the fault cause of the first computing node is obtained based on the exit code. The exit code includes a first exit code or a second exit code. Optionally, the first exit code indicates a hardware fault, and the second exit code indicates a software fault.

[0030] In this way, the management node and fault handling node can better monitor the running status of the computing node, and promptly restart the task or model to ensure the recovery of training jobs.

[0031] Secondly, embodiments of this application provide a distributed training apparatus for an AI model. The computing cluster includes multiple computing nodes for distributed training of the AI ​​model; each computing node executes a training task for the distributed training of the AI ​​model. The apparatus includes a fault perception module and a management module.

[0032] The fault detection module is used to obtain the cause of the first computing node's failure when the first computing node among multiple computing nodes fails. Optionally, the cause of the failure may include at least one of a hardware failure or a software failure.

[0033] The management module is used to resume model training based on the training task recovery method corresponding to the cause of the failure of the first computing node. Optionally, the training task recovery method is used to indicate at least one model training recovery scheme, either model restart or task restart.

[0034] In one possible implementation, the fault perception module obtains the fault cause, where hardware faults include accelerator card faults or other hardware faults besides accelerator card faults; and software faults include other faults besides hardware faults.

[0035] In one possible implementation, the specific implementation is as follows: a management module is used to restart the training task executed by the first computing node on the second computing node or the first computing node when the fault cause indicates a hardware failure; the second computing node is a computing node in the computing cluster other than the first subset.

[0036] In one possible implementation, the specific implementation is as follows: a management module is used to send a stop training notification to the first computing node and to the third computing node; the stop training notification is used to indicate that the training task to be stopped; the third computing node is a computing node in the first subset that has not experienced a failure.

[0037] In one possible implementation, the specific implementation is as follows: a management module is used to send a chip repair instruction to the first computing node when the hardware failure is an accelerator card failure; the chip repair instruction is used to instruct the accelerator card in the first computing node to perform a chip repair operation; when the accelerator card in the first computing node successfully performs the chip repair operation, a continue training notification is sent to the first computing node and the third computing node; the continue training notification is used to instruct the resumption of the distributed training task of the AI ​​model.

[0038] In one possible implementation, the specific implementation is as follows: a management module is used to restart the training task executed by the first computing node on the second computing node if it receives a first exit code sent by the first computing node; the first exit code indicates that the chip repair operation performed by the accelerator card in the first computing node has failed, and the first computing node exits the training process corresponding to the training task.

[0039] In one possible implementation, the specific implementation is as follows: a management module is used to restart the training task executed by the first computing node on the second computing node when the hardware failure is a hardware failure other than an accelerator card failure; and to send a notification to the second computing node and the third computing node to continue training.

[0040] In one possible implementation, the specific implementation is as follows: a management module is used to isolate the first computing node.

[0041] In one possible implementation, the specific implementation is as follows: a management module is used to restart the training task executed in the first subset on the second subset if a second exit code is received from the second computing node; the second subset includes multiple computing nodes in the computing cluster other than the first subset; the second exit code is used to indicate that the second computing node exits the training task;

[0042] If the second exit code is not received from the second computing node, a continue training notification is sent to both the second and third computing nodes.

[0043] In one possible implementation, the specific implementation is as follows: a management module is used to restart the training tasks executed in the first subset on the second subset when the cause of the fault indicates a software failure; the second subset includes multiple computing nodes in the computing cluster other than the first subset.

[0044] In one possible implementation, the specific implementation is as follows: a management module is used to send a stop training notification to each computing node in the first subset; the stop training notification is used to indicate that the training task to be stopped is stopped; a continue training notification is sent to each computing node in the second subset to restart the training task executed in the first subset on the second subset; the continue training notification is used to instruct each computing node in the second subset to resume the training process corresponding to the training task of the distributed training of the AI ​​model.

[0045] In one possible implementation, the specific implementation is as follows: the first subset includes fault handling nodes, wherein the fault handling nodes are the computing nodes in the first subset. The computing cluster includes management nodes, wherein the management nodes are the computing nodes in the computing cluster other than the first subset. The fault awareness module is used to obtain the fault cause sent by the first computing node through the fault handling nodes, and the management node obtains the exit code sent by the first computing node. Based on the exit code, the fault cause of the first computing node is obtained; the exit code includes a first exit code or a second exit code; the first exit code indicates a hardware fault, and the second exit code indicates a software fault.

[0046] Thirdly, embodiments of this application provide a computing device cluster, which includes at least one computing device. Each computing device includes a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method as described in the first aspect or any possible implementation thereof.

[0047] Fourthly, embodiments of this application provide a computer program product containing instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method described in the first aspect or any possible implementation thereof.

[0048] Fifthly, embodiments of this application provide a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the instructions in the computing program stored in the computer-readable storage medium to perform the method in the first aspect or any possible implementation of the first aspect.

[0049] The technical effects of any of the implementation methods in aspects two through five can be found in the technical effects of different implementation methods in aspects one through two. They will not be repeated here.

[0050] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the structure of a distributed training system for AI models in the existing technology;

[0052] Figure 2 This is a schematic diagram of the compression process in the prior art;

[0053] Figure 3 This is a schematic diagram of fault machine detection in the existing technology;

[0054] Figure 4A schematic diagram illustrating an application scenario of the distributed training method for the AI ​​model provided in this application embodiment;

[0055] Figure 5 A flowchart illustrating the distributed training method for the AI ​​model provided in this application embodiment;

[0056] Figure 6 This is a schematic diagram of the system architecture of the computing node provided in the embodiments of this application;

[0057] Figure 7 This is a schematic diagram of the connection between computing nodes provided in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of the first computing node restart task provided in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the training task recovery process in the scenario of restarting the first computing node, provided in an embodiment of this application.

[0060] Figure 10 A schematic diagram illustrating the process of restarting a task on a first computing node, provided as an embodiment of this application;

[0061] Figure 11 A schematic diagram of the distributed training process of an AI model in a scenario where a task is restarted on a first computing node, provided as an embodiment of this application;

[0062] Figure 12 A schematic diagram illustrating the training task executed by restarting the first computing node on the second computing node, as provided in an embodiment of this application.

[0063] Figure 13 This is a schematic diagram of the training task recovery process for restarting a task on a second computing node, provided in an embodiment of this application.

[0064] Figure 14 This is a schematic diagram of a process for restarting a training task executed by a first computing node on a second computing node, as provided in an embodiment of this application.

[0065] Figure 15 This is a schematic diagram illustrating another process for restarting the training task executed by the first computing node on the second computing node, as provided in an embodiment of this application.

[0066] Figure 16 This is a schematic diagram of the distributed training process of an AI model in a scenario where the training task is restarted on the first computing node on the second computing node, as provided in an embodiment of this application.

[0067] Figure 17 This is a schematic diagram of model restart provided in an embodiment of this application;

[0068] Figure 18 This is a schematic diagram of the training task recovery process in the model restart scenario provided in this application embodiment;

[0069] Figure 19 This application provides a schematic diagram of a model restart process.

[0070] Figure 20 A schematic diagram of the distributed training process of the AI ​​model in the model restart scenario provided in this application embodiment;

[0071] Figure 21 This application provides a schematic diagram of a task recovery process that combines different fault causes in its embodiments.

[0072] Figure 22 This is a schematic diagram of the structure of the distributed training device 23 for the AI ​​model provided in the embodiments of this application;

[0073] Figure 23 This is a schematic diagram of the structure of the distributed training system 24 for the AI ​​model provided in an embodiment of this application;

[0074] Figure 24 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0075] Figure 25 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0076] Figure 26 This is a schematic diagram of the network connection between computing devices in a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0077] First, the terms used in the embodiments of this application will be introduced.

[0078] AI models are mathematical algorithm models that use machine learning concepts to solve practical problems. The term "AI model" in this application is a general term and includes deep learning models, machine learning models, etc.

[0079] AI models include a large number of parameters and calculation formulas (or calculation rules). The parameters in an AI model are numerical values ​​obtained by training the AI ​​model using a training dataset.

[0080] For example, the parameters of an AI model include the weights of the calculation formulas or factors in the AI ​​model. The parameters of an AI model also include some hyperparameters.

[0081] Hyperparameters are parameters that cannot be obtained by training an AI model using a training dataset. Hyperparameters can be used to guide the construction or training of AI models.

[0082] There are many types of hyperparameters. For example, the number of iterations in AI model training, learning rate, batch size, number of layers in the AI ​​model, and number of neurons in each layer.

[0083] Generally, AI models need to be trained before they can be used to complete a task.

[0084] Training an AI model refers to using existing data and certain methods to make the AI ​​model fit the patterns in the existing data, and to determine the parameters in the AI ​​model.

[0085] Typically, during AI model training, the training loss is calculated using a loss function based on the training dataset and the AI ​​model's output values ​​based on the training dataset. The AI ​​model's parameters are then adjusted based on the training loss until the training loss is less than or equal to a loss threshold, or the number of training iterations is greater than or equal to a threshold, at which point AI model training stops.

[0086] A loss function is a function used to measure how well an AI model has been trained (that is, to calculate the difference between the AI ​​model's predictions and the true target).

[0087] During the training of an AI model, the model's parameters need to be adjusted to minimize the loss function in the next iteration. Gradient descent is typically used to update the model's parameters; therefore, in each iteration, the gradient of the loss function corresponding to the current training data needs to be calculated, and the AI ​​model's parameters are then updated based on this gradient.

[0088] The gradient is a vector of partial derivatives of the loss function.

[0089] Distributed training involves splitting the training process across multiple independent computing nodes, then periodically aggregating and redistributing the results, thereby accelerating the training process of AI models. Distributed training can include data-parallel distributed training.

[0090] Data-parallel distributed training involves deploying the same AI model across multiple computing nodes, distributing the training data across these nodes for simultaneous computation. Training the AI ​​model is performed on each node, and the gradients of the model parameters generated on each node are aggregated before updating the model parameters.

[0091] The data-parallel distributed training process consists of two stages. The first stage involves computation by multiple computing nodes, and the second stage involves gradient synchronization.

[0092] For example, taking m computing nodes participating in distributed training as an example, in each round of distributed training, in the first stage, the training samples on each computing node are different, resulting in m batches of samples across the m computing nodes, and each of the m computing nodes calculates a gradient. In the second stage, these m gradients are averaged to obtain the average value of the m gradients. The parameters of the AI ​​model are updated based on the average value of the m gradients, and then the next round of training is performed based on the AI ​​model with updated parameters.

[0093] An AI platform is a platform that provides AI developers and users with a convenient AI development environment and convenient development tools.

[0094] The AI ​​platform contains various pre-trained AI models or AI sub-models built-in to solve different problems. The platform can search for and build suitable AI models based on user needs. Users simply need to define their requirements within the platform and prepare and upload their training dataset as prompted. The platform can then train an AI model tailored to the user's needs. Alternatively, users can prepare their own algorithm (also known as an initial AI model) and training dataset as prompted and upload them to the platform. Based on the user's algorithm and training dataset, the platform can train an AI model to meet the user's requirements. Users can then use the trained AI model to complete their specific tasks. It should be understood that the AI ​​model before it is trained by the AI ​​platform (e.g., the algorithm uploaded by the user, the algorithm pre-built into the AI ​​platform, or a pre-trained model) is referred to as the initial AI model.

[0095] An AI cloud service platform refers to an AI platform deployed in a cloud environment.

[0096] In this context, the cloud environment refers to the entity that provides cloud services to users using basic resources under the cloud computing model. The cloud environment includes cloud data centers and cloud service platforms.

[0097] Cloud data centers include a large number of basic resources (including computing clusters, storage resources, and network resources) owned by cloud service providers. The computing clusters included in cloud data centers can be a large number of computing nodes (such as servers).

[0098] A container is a relatively independent and isolated environment for running processes, constructed using virtualization technology within a computer operating system. This environment can contain independent file systems, namespaces, resource views, etc. Using containers simplifies software deployment, enhances software portability and security, and improves system resource utilization.

[0099] A job is a set of programs that need to be executed to complete a specific computational task. Typically, a job corresponds to a group of processes, containers, or other running entities on one or more computers. For example, AI model training in this application embodiment. A job includes at least one task.

[0100] A task is a single program within a set of programs that correspond to a job. It typically corresponds to a process, container, or other runtime entity on a computer.

[0101] A training job is a set of procedures that need to be executed to train an initial AI model. Completion of a training job represents the completion of training an initial AI model, resulting in a trained AI model. A training job includes at least one training task.

[0102] A training task is a single program in a set of programs corresponding to a training job.

[0103] A training process is a process executing a training task on a compute node. One training process corresponds to one training task, and a compute node can execute one or more training tasks; that is, one compute node can have one or more training processes.

[0104] A training framework is a toolkit or function package that is required during the training of an AI model. It is the runtime framework that each training task in a training job depends on. Training frameworks include Tensorflow, Caffe, Theano, MXNet, Torch, and PyTorch.

[0105] A computing cluster consists of computing nodes (Pods) that can be used to train AI models. Each computing node can be a computing device, such as a server. Alternatively, each computing node can be a computing card, such as a graphics processing unit (GPU) or a neural network processing unit (NPU). In this embodiment, the computing cluster can be a cluster managed via Kubernetes (K8S).

[0106] The application scenarios involved in the embodiments of this application will be described below.

[0107] In distributed training of AI models, the training job is interrupted when a computing node fails. It is necessary to automatically restart the training job to continue training the AI ​​model.

[0108] Currently, the main approach is to add a management node to the training cluster. This management node manages the nodes performing parameter training tasks within the training cluster and is aware of potential faults. For example... Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of a distributed AI model training system in existing technology. The structure of the distributed AI model training system shown includes a master node (CPU node) and multiple training nodes connected to the master node. Among them, the master node is the central processing unit (CPU) in the cluster and does not participate in the model training task.

[0109] like Figure 1 As shown, each training node is equipped with a networking component (RendezvousHandler), a creation component, and a monitoring component.

[0110] The networking component receives data from the master node and reports data to the master node. The creation component creates training processes based on the node identifier (Pod ID, Pod Identity document) and the cluster size. The monitoring component monitors the execution status of the training process on the training node and performs resilient training in the event of a training node failure.

[0111] When a training node starts, each training node is assigned a node identifier (Pod ID) and configured in its environment variables. The networking component of the training node reports its Pod ID and the number of GPUs to the master node. The networking component then continuously requests communication from the master node, i.e., information about all nodes. The master node stores the received training node information in a list. When the number of nodes in the list reaches the network requirement, the master node sends communication to all training nodes. Communication is sorted by Pod ID, such as {0:8,1:8,2:8,3:8}. The key represents the Pod ID, and the value represents the number of GPUs in the Pod. For example, "0:8" means that the compute node with Pod ID 0 has 8 GPUs. The order of the Pod IDs in the compute cluster is its rank.

[0112] When a training node fails, the master node restarts the training process running on all training nodes. It also identifies the failed training node through fault detection and automatically scales down the training nodes participating in the training job. Furthermore, it notifies administrators to isolate the failed training node. After a new training node is started, the master node automatically scales up.

[0113] For example, taking a training job with 6 training nodes as an example, such as Figure 2As shown, when the master node detects a failure in training node Pod-5 and the new training node Pod-6 has not yet started, the node information received by the master node is {0:8, 1:8, 2:8, 3:8, 4:8}. However, if the user requests a multiple of 2 training nodes, the master node will remove Pod-4 from the cluster, achieving automatic scaling down. Pod-4 will then wait for Pod-6 to start before joining the training cluster to achieve scaling up.

[0114] During fault detection, before the master node restarts the training processes running on all training nodes, it first divides all training nodes into multiple node groups, pairwise. Nodes within each group execute a many-to-many communication (Allgather) task and report its success or failure to the master node. If at least one node group has failed its Allgather task, the training nodes within that group are considered potential faulty machines. Then, in the second round of testing, the master node again pairs the potential faulty machines and healthy nodes into node groups. Nodes within each group continue executing Allgather, thus identifying the faulty nodes.

[0115] For example, taking a training job with 6 training nodes as an example, such as Figure 3 As shown, in the first round of fault detection, the partitioning result is [{1,2},{3,4},{5,6}]. Since {5,6} failed to execute Allgather, training nodes 5 and 6 are potential faulty nodes. In the second round of fault detection, the partitioning is [{1,2},{3,5},{4,6}]. If {4,6} fails, then node 6 is the faulty node.

[0116] As can be seen from the existing AI model distribution training described above, when a node failure is detected, the manual isolation method relies on manual repair, which consumes a significant amount of manpower to handle the failure and results in a long job recovery time. Furthermore, for all failures, the training task is restarted on a new node. Changing nodes and restarting the training task on the new node increases the rescheduling time after job failures, thus increasing the model training duration. Moreover, restarting the training process on all training nodes further increases the rescheduling time after job failures, leading to an increase in the overall model training duration.

[0117] Therefore, to address the issue of increased job recovery time caused by reliance on manual repair in AI model training tasks, this application provides a distributed training method for AI models. Instead of directly isolating faulty nodes, this application matches the cause of the fault in the first computing node with an appropriate training task recovery method. The distributed training of the AI ​​model continues through at least one of the training task restart or model restart methods. This approach selects appropriate training task recovery methods for different fault causes, avoiding the rescheduling time caused by directly isolating faulty nodes, thus shortening the training time of the AI ​​model. Furthermore, based on fault information, the application automatically identifies the training task recovery method matching the fault cause, reducing reliance on manual repair and improving training task recovery efficiency.

[0118] The distributed training method for the AI ​​model provided in this application will be described below with reference to specific embodiments.

[0119] like Figure 4 As shown, Figure 4 This is a schematic diagram illustrating an application scenario of the distributed training method for the AI ​​model provided in this application embodiment. The application scenario shown includes a cloud data center 10, a cloud service platform 30, and a client 20. The cloud data center 10 houses an AI platform 101.

[0120] In some implementations, AI platform 101 can be deployed independently within an instance of cloud data center 10. Alternatively, AI platform 101 can be deployed in a distributed manner across multiple instances of cloud data center 10.

[0121] In this context, an instance can refer to a virtual machine, container, bare metal server, physical server, etc., which contain computing components such as CPU, memory, operating system, network, and disk.

[0122] like Figure 4As shown, the AI ​​platform 101 is abstracted into an AI cloud service by the cloud service provider on the cloud service platform 30 and provided to users. After the user purchases the cloud service through the client 20 on the cloud service platform 30 (pre-payment is possible, with settlement based on the final resource usage), the cloud environment utilizes the AI ​​platform 101 deployed in the cloud data center 10 to provide the AI ​​platform cloud service to the user. When using the AI ​​platform 101 cloud service, the user can specify the task to be completed by the AI ​​model and upload the training dataset to the cloud environment through the application programming interface (API) or graphical user interface (GUI) in the client 20. The AI ​​platform 101 in the cloud environment receives the user's task information and training dataset, and performs data preprocessing and AI model training. The AI ​​platform 101 returns the status information of the AI ​​model training process to the user through the API or GUI. The trained AI model can be downloaded by the client 20 or used online to complete specific tasks, such as image recognition and image generation.

[0123] In one possible implementation, the AI ​​platform 101 is associated with a computing cluster 1014. The computing cluster 1014 includes multiple computing nodes, and the AI ​​platform 101 can schedule these computing nodes for AI model training. For example... Figure 4 As shown, the AI ​​platform 101 includes a data storage module 1011, a resource management module 1012, and a task scheduling module 1013. It should be understood that... Figure 4 The diagram below is merely an example illustrating one structural configuration of the AI ​​platform 101. This application does not limit the module division or naming of the AI ​​platform 101.

[0124] The task scheduling module 1013 communicates with the resource management module 1012 and the computing cluster 1014. The computing cluster 1014 can communicate with the task scheduling module 1013, the resource management module 1012, and the data storage module 1011. The computing nodes in the computing cluster 1014 communicate with each other.

[0125] In one example, when managing the computing cluster 1014 via Kubernetes (K8S), the resource management module 1012 and the task scheduling module 1013 can be deployed within the API server in K8S. Alternatively, the functionality implemented by the resource management module 1012 and the task scheduling module 1013 can be implemented by the API server in K8S. The API server provides CRUD (Create, Read, Update, Delete) interfaces for various K8S resource objects.

[0126] In another example, computing cluster 1014 includes a management node and worker nodes. Worker nodes are used to execute AI model training tasks. The management node schedules the worker nodes. The management node communicates with the data storage module 1011, resource management module 1012, and task scheduling module 1013. For example, an API server is deployed in the management node. The data storage module 1011, resource management module 1012, and task scheduling module 1013 each communicate with the API server.

[0127] The task scheduling module 1013 is used to configure training jobs, schedule training jobs, and receive training jobs submitted by users.

[0128] For example, the task scheduling module 1013 divides the training tasks based on the number of computing nodes participating in the training job, resulting in multiple training tasks included in the training job. Then, based on the computing nodes allocated by the resource management module 1012, it configures the training tasks in the training job to be executed on the computing nodes.

[0129] Resource management module 1012 is used for computing node allocation, management and scheduling.

[0130] For example, the resource management module 1012 allocates computing nodes participating in the training job according to the computing resource requirements of the training job.

[0131] In one possible implementation, computing cluster 1014 can execute one or more training jobs simultaneously. Each training job trains an AI model, and the training of an AI model is based on the same training framework. The training framework is the runtime framework that each training task in the training job depends on. Each training job includes one or more training tasks, and all training tasks in each training job depend on the same runtime framework. For example, if computing cluster 1014 executes n training jobs, then computing cluster 1014 is used to train n AI models. For any one of the n training jobs, all training tasks within it depend on the same runtime framework, and this training framework can be obtained from data storage module 1011.

[0132] When a user submits a training job to the AI ​​platform 101, the task scheduling module 1013, upon receiving the submitted training job, requests computing resources from the resource management module 1012 to execute multiple training tasks within that training job. The resource management module 1012 allocates multiple computing nodes for this training job or these multiple training tasks and returns the allocation results to the task scheduling module 1013. The task scheduling module 1013 sends the training framework, training dataset, initial AI model, etc., to these multiple computing nodes; alternatively, each of these computing nodes can obtain the training framework, training dataset, initial AI model, etc., from the data storage module 1011 to deploy the training framework on each of the multiple computing nodes. The task scheduling module 1013 then configures these multiple training tasks on these multiple computing nodes, thereby initiating training.

[0133] In this embodiment of the application, when a computing node fails during the execution of a training task, the AI ​​platform 101 executes the distributed training method of the AI ​​model provided in this embodiment of the application to restore the training job and avoid interruption of the training job.

[0134] It should be noted that, Figure 4 The naming of modules in the provided application scenarios is illustrative. In practical applications, there can be more... Figure 4 More or fewer modules.

[0135] based on Figure 4 The provided application scenario, in this application embodiment, offers a distributed training method for an AI model. This distributed training method for the AI ​​model can be performed by… Figure 4 The AI ​​platform 101 is executed. (For example...) Figure 5 As shown, Figure 5 This is a flowchart illustrating the distributed training method for an AI model provided in this application embodiment. The distributed training method for the AI ​​model shown includes steps S510 to S540.

[0136] S510 distributes training tasks to the first subset of tasks to be executed.

[0137] In one possible implementation, a user submits a training job for an AI model to a cloud service platform 30 via client 20. The cloud service platform 30 then distributes the training job to the AI ​​platform. The AI ​​platform performs task scheduling based on the training job, determining multiple computing nodes to execute the training job and the training tasks to be performed on each computing node. The AI ​​platform then distributes the training job to multiple computing nodes based on a first subset of the executed training jobs and the training tasks performed on each computing node within that first subset.

[0138] For example, the resource management module 1012 in the AI ​​platform allocates resources, assigning a first subset of training jobs to be executed. This first subset includes multiple computing nodes. The task scheduling module 1013 in the AI ​​platform allocates tasks based on the multiple computing nodes executing the training jobs, determining the training tasks to be executed by each computing node and distributing the training tasks to the corresponding computing nodes.

[0139] In this embodiment, after receiving a computing task, the computing node executes the task process within that task. The AI ​​platform monitors the state of the computing node while it is executing the training task.

[0140] In one possible implementation, the AI ​​platform can perform fault monitoring through the resource management module 1012, or the computing nodes can perform self-fault monitoring. Alternatively, the management node in the computing cluster can perform fault monitoring.

[0141] In the first example, the AI ​​platform uses the resource management module 1012 to monitor the faults of each computing node in the first subset of the training jobs, and determines whether a fault occurs in each computing node in the first subset when executing the training task.

[0142] For example, the resource management module 1012 can periodically monitor the computing nodes. It determines that a computing node has failed when a hardware failure occurs on the computing node and / or the training process on the computing node exits.

[0143] Among them, hardware failures include accelerator card failures or hardware failures other than accelerator card failures.

[0144] Accelerator card failures can refer to failures that do not cause the training process on the computing node to exit or stop, but only affect the computing performance of the computing node. Examples include insufficient chip memory failures and multi-bit error correction code (ECC) failures.

[0145] Hardware failures, excluding accelerator card malfunctions, can refer to hardware failures that cause the training process on a compute node to exit or stop. Examples include power outages of the compute node, network disconnections between the compute node and other compute nodes that share the same training job.

[0146] In this embodiment, hardware failure of the computing node is only one possible cause of the training process exiting on the computing node. This application does not specifically limit the reasons for the training process exiting on the computing node. When the training process on the computing node exits for any reason, it can be detected by the resource management module 1012. For example, when the training process on the computing node exits, the computing node reports an exit code to the resource management module 1012. When the resource management module 1012 identifies the exit code sent by any computing node, it determines that the training process on that computing node has exited and identifies that the computing node has failed.

[0147] In the second example, each of the multiple computing nodes monitors the training process for any operational failures while executing a training task. For example, each computing node's monitoring program checks for operational failures in the training process corresponding to the training task.

[0148] In this embodiment, when a computing node detects a failure in the corresponding training process, it can report it to the resource management module 1012. Alternatively, the computing node can exit the training process and report an exit code to the resource management module 1012, or report an exit code to the management node.

[0149] In the third example, the management node in the computing cluster determines whether each computing node in the first subset has failed while executing the training task by identifying whether the computing nodes report an exit code.

[0150] S520: When a faulty first computing node is detected in the first subset, the cause of the fault in the first computing node is obtained.

[0151] In related technologies, when a computing node fails, the node is directly isolated, and either the node itself is reassigned or all computing nodes for the training job are reassigned. However, in practical applications, the causes of computing node failures are diverse. If the same recovery method is used for all failures, the method will be incompatible with the cause of the failure, increasing training time overhead. Therefore, in this embodiment, when the AI ​​platform detects a failed computing node, it selects the appropriate training task recovery method based on the cause of the failure.

[0152] The cause of failure is used to indicate the type of failure. In this embodiment, the cause of failure includes at least one of hardware failure or software failure.

[0153] In this context, software failures can refer to failures that cause the training process on a computing node to exit. Examples include process startup failures and unresponsive operating systems. In this embodiment, software failures include failures other than hardware failures that cause the training process on a computing node to exit.

[0154] In this embodiment of the application, the cause of the failure of the first computing node can be obtained based on the fault information of the running task process of the first computing node.

[0155] In some implementations, the fault information includes at least one of the following: alarm information, running status information, or exit code when the computing node is running a task process.

[0156] Alarm information is used to indicate faults occurring in computing nodes. For example, alarm information can be one or more of the following: insufficient processor memory alarm, network disconnection alarm, insufficient network bandwidth alarm, insufficient processor utilization alarm, and multi-bit ECC alarm.

[0157] The runtime status information indicates the health status of the compute node running task processes and the execution status of the task processes. Health status includes, but is not limited to, node normal, node abnormal, and node inactive. Execution status includes task process running normally, task process completed, task process running abnormally, and task process exited. Node abnormality can be due to hardware failure, such as power loss or network connection loss. Node abnormality can also be due to software failure. Task process running abnormality can refer to one or more of the following: task process execution time exceeding a duration threshold, task process execution interruption, and task process memory usage exceeding a usage threshold.

[0158] The exit code indicates the reason for the task process exiting the compute node. For example, the reason for the task process exiting could be a hardware failure or a software failure on the compute node.

[0159] In one possible implementation, the cause of a computing node failure can be identified based on at least one of the following: alarm information, running status information, or exit code during the execution of a task process by the computing node.

[0160] For example, taking alarm information as an example, if the fault indicated by the alarm information includes one or more of the following: insufficient processor memory alarm, multi-bit ECC alarm, the cause of the first computing node's fault is determined to be an accelerator card fault. As another example, if the fault indicated by the alarm information includes one or more of the following: network disconnection alarm, insufficient network bandwidth alarm, insufficient processor utilization alarm, the cause of the first computing node's fault is determined to be a hardware fault other than an accelerator card fault.

[0161] For example, taking runtime status information as an example, when the runtime status information indicates that the task process has exited and the computing node has experienced a hardware failure, the cause of the failure of the first computing node is determined to be a hardware failure other than an accelerator card failure. As another example, when the runtime status information indicates that the task process has exited and the computing node has not experienced a hardware failure, the cause of the failure of the first computing node is determined to be a software failure.

[0162] For example, taking the exit code as an example, when the exit code matches the first exit code, the cause of the failure of the first computing node is determined to be a hardware failure other than an accelerator card failure. When the exit code matches the second exit code, the cause of the failure of the first computing node is determined to be a software failure.

[0163] In another possible implementation, the health status of a node and the execution status of a task process can be determined based on the running status information. Faults occurring on the node can then be identified based on the health status and the execution status of the task process. Furthermore, the cause of the fault can be identified based on the fault's scope.

[0164] In this application embodiment, there are multiple ways to obtain the fault information of the first computing node. In practical applications, the appropriate acquisition method can be selected according to the specific application scenario. This application embodiment does not limit this. For example, in a first possible implementation, the fault information of the first computing node is obtained through the driver interface of the first computing node. For example, alarm information when the first computing node is running a task process is obtained through the driver interface of the first computing node. Another example, in a second possible implementation, the first computing node executes a node inspection script to obtain the fault information of the first computing node. For example, the running status information of the first computing node is obtained by executing a node inspection script. Yet another example, in a third possible implementation, the AI ​​platform receives the exit code sent by the computing node. For example, the resource management module 1012 receives the exit code sent by the computing node. Yet another example, the management node in the computing cluster receives the exit code sent by the computing node.

[0165] S530: Resume model training according to the training task recovery method corresponding to the cause of failure of the first computing node.

[0166] Optionally, the training task recovery method is used to indicate at least one recovery scheme for model training, either model restart or task restart.

[0167] Task restart can refer to restarting the training task executed by the first computing node on the first computing node. Alternatively, task restart can also refer to restarting the training task executed by the first computing node on the second computing node. The second computing node is any computing node in the computing cluster other than the first subset.

[0168] Model restart can refer to restarting the training task executed by the computing nodes in the first subset on the second subset. The second subset consists of multiple computing nodes in the computing cluster other than the first subset. In one example, the number of computing nodes in the second subset is equal to or greater than the number of computing nodes in the first subset.

[0169] In this embodiment, the recovery times required for task restart and model restart are different. Since restarting the training task does not require rescheduling all nodes, the recovery time required for restarting the training task is shorter than that for restarting the model.

[0170] Furthermore, task restart and model restart have different causes of failure. For example, task restart can resume training tasks when a hardware failure occurs on the first computing node, while model restart can resume training jobs when a software failure occurs on the first computing node.

[0171] Therefore, in this embodiment, a corresponding training task recovery method is selected based on the cause of the first computing node's failure. This improves the compatibility between the training recovery method and the cause of the failure, thereby enabling the training job to be recovered in the shortest possible time. It avoids problems such as training task recovery failure or excessively long recovery times caused by selecting an unsuitable training task method.

[0172] In this embodiment, the AI ​​platform performs at least one of training task restart or model restart to continue distributed training of the AI ​​model.

[0173] In the first possible implementation, when the training recovery task includes either training task restart or model restart, the AI ​​platform performs training task restart or model restart.

[0174] In the second possible implementation, when the training recovery task includes at least two of the following: training task restart and model restart, the training task recovery method is executed according to priority.

[0175] For example, when the training recovery task includes both training task restart and model restart, the priorities for training task restart and model restart are assigned according to their respective recovery durations. Since the recovery duration for training task restart is shorter than that for model restart, the priority for training task restart is higher than that for model restart. The AI ​​platform executes the task restart. If the training task restart in the first computing node is successful, the AI ​​platform continues to monitor each computing node for fault detection. If the training task restart in the first computing node fails, the AI ​​platform executes the model restart.

[0176] based on Figure 5The provided embodiments, compared to directly isolating the failed node, match the corresponding training task recovery method based on the cause of the first failed computing node's failure. The distributed training of the AI ​​model continues through at least one of the following methods: task restart or model restart. This approach selects appropriate training task recovery methods for different failure causes, avoiding the rescheduling time caused by directly isolating the failed node. This shortens the training time of the AI ​​model. Furthermore, automatically identifying the matching training task recovery method based on the failure cause reduces reliance on manual repair, thereby improving training task recovery efficiency.

[0177] Next, we will introduce the methods for fault monitoring of computing nodes.

[0178] Current technologies typically monitor the computing nodes involved in training by using a master node that does not participate in the training task. This results in a complex inter-node structure within the computing cluster, and adding a master node that does not participate in the training task increases the overhead of the computing nodes. Therefore, in this embodiment, a process management module is deployed in the computing nodes to monitor for node failures. This eliminates the need for additional master nodes that do not participate in the training task, reducing node overhead and lowering the complexity of the computing cluster.

[0179] The process management module is used to start the training process in the computing node, stop the training process in the computing node, and monitor at least one of the following: the hardware status of the computing node or the training process status corresponding to the training task executed by the computing node.

[0180] Hardware status includes, but is not limited to, network connection status, memory status, chip status, and power supply status. Training process status includes process exit, process completion, process execution, and process shutdown.

[0181] In the first possible implementation, when the process management module in the computing node detects at least one of the following: a hardware failure of the computing node where the process management module is located, or the training process corresponding to the training task being executed exits, the computing node where the process management module is located is determined to be the first computing node.

[0182] For example, the process management module obtains information about whether the accelerator card in the compute node has malfunctioned through the driver interface. Another example is that the process management module checks for hardware failures in the compute node by executing inspection scripts. Yet another example is that the process management module monitors whether the training process in the compute node has abnormally terminated.

[0183] In one example, such as Figure 6 As shown, Figure 6This is a schematic diagram of the system architecture of a computing node provided in an embodiment of this application. The computing node deploys a container 61 (Pod), a training accelerator card 62 (XPU device), a training accelerator card management component 63 (XPU-device-plugin), and a node management component 64 (node-agent).

[0184] Container 61 contains a process management module 611. Container 61 is used to run training tasks.

[0185] Training Accelerator Card 62 is used to perform training tasks.

[0186] The training accelerator card management component 63 is used to manage the training accelerator card 62, monitor whether the training accelerator card 62 has malfunctioned, and identify the type of malfunction that has occurred in the training accelerator card 62.

[0187] The node management component 64 is used to run inspection scripts on a regular basis to detect whether computing nodes have failed.

[0188] When the training task is running in container 61, if the training accelerator card management component 63 detects a failure in the training accelerator card 62 through the subscription interface, the training accelerator card 62 reports the accelerator card failure to the process management module 611. The node management component 64 periodically runs a patrol script to detect whether the compute nodes have failed. When a compute node fails, the node management component 64 reports the node patrol failure to the process management module 611. The node patrol failure includes hardware failures other than accelerator card failures.

[0189] In the second possible implementation, to facilitate the management of computing nodes, a fault handling node can be elected from among the multiple computing nodes included in the first subset. The process management module 611 deployed in the fault handling node is designated as the master management module. The process management modules 611 deployed in the other computing nodes in the first subset besides the fault handling node are designated as slave management modules. The slave management modules report fault information to the master management module when they detect at least one of the following: a hardware failure in the computing node where the process management module 611 resides, or the training process corresponding to the executed training task exits. The master management module monitors the fault handling node for faults, receives the fault information reported by the slave management modules, obtains the cause of the first computing node's failure based on the fault information, and executes the aforementioned S530 recovery training task based on the cause of the failure.

[0190] In one example, any one of the multiple computing nodes executing the training job can be used as a fault handling node.

[0191] In another example, a node priority can be configured for each of the multiple computing nodes, and a fault handling node can be selected based on this priority. For instance, computing nodes can be assigned sequence numbers, with the node having the lowest sequence number selected as the fault handling node. Alternatively, the computing node with the fewest computing resources can be selected based on the computational resources required for each computing node's training task.

[0192] Optionally, if the first computing node that fails is the fault handling node, a new fault handling node can be selected in accordance with the above-mentioned method for selecting fault handling nodes.

[0193] For example, taking the computing nodes executing the training job as including computing node 1, computing node 2, and computing node 3 as an example, Figure 7 As shown, compute node 1 is the fault handling node, and compute nodes 2 and 3 are both slave nodes. Compute nodes 1, 2, and 3 all have... Figure 6 The structure is shown. During the training task, computation node 1, computation node 2, and computation node 3 operate according to the above... Figure 6 The process management modules 611 in computing nodes 2 and 3 report the fault information to the process management module 611 in computing node 1 when a fault is detected.

[0194] In this embodiment, to better monitor the operating status of computing nodes and promptly restart tasks or models to ensure the recovery of training jobs, a management node can be set up in the computing cluster. The management node and the process management module 611 deployed on the computing nodes monitor whether any computing nodes have failed.

[0195] The management node can be any computing node in the computing cluster other than the first subset. Alternatively, the management node can be any node in the computing cluster used to manage the computing nodes; for example, the management node can be the master node in the computing cluster.

[0196] In one possible implementation, the process management module 611 exits the training process on the computing node and reports the exit code to the management node. The management node identifies the cause of the failure of the first computing node based on the exit code.

[0197] For example, when the process management module 611 receives a fault reported by the node management module, and the fault is a hardware fault, the process management module 611 exits the training process on the computing node and reports a first exit code to the management node. The management node obtains the first exit code and determines that the computing node that reported the first exit code has experienced a hardware fault other than the accelerator card.

[0198] For example, when the process management module 611 receives a fault reported by the node management module, and the fault is not a hardware fault, the process management module 611 exits the training process on the computing node and reports a second exit code to the management node. The management node obtains the second exit code and determines that the computing node that reported the second exit code has experienced a software fault.

[0199] In one example, an API server is deployed on the management node. The process management module 611 reports exit codes to the API server.

[0200] In this embodiment of the application, in order to better monitor the running status of computing nodes, perform timely node rescheduling or model restart, and ensure the recovery of training jobs, the resource management module 1012 and the process management module 611 deployed in the computing node are used to monitor whether the computing node has failed.

[0201] For example, when the resource management module 1012 receives the first exit code reported by the process management module 611, it determines that the computing node that reported the first exit code has experienced a hardware failure other than the accelerator card. As another example, when the resource management module 1012 receives the second exit code reported by the process management module 611, it determines that the computing node that reported the second exit code has experienced a software failure.

[0202] In this embodiment of the application, after the AI ​​platform detects a first computing node that has failed among multiple computing nodes, it obtains the cause of the failure of the first computing node according to the above-described S510, and determines a training task recovery method that matches the cause of the failure of the first computing node.

[0203] Next, we will introduce how to determine the training task recovery method.

[0204] In the first possible implementation, a training task recovery method matching the fault cause can be determined based on the mapping relationship between the fault cause and the training task recovery method. This improves the fit between the training recovery method and the fault cause, enabling training tasks to be recovered in the shortest possible time. It avoids problems such as training task recovery failure or excessively long recovery times caused by selecting an unsuitable training task method.

[0205] The mapping relationship between the cause of the fault and the training task recovery method is used to indicate the mapping relationship between the cause of the fault and the corresponding training task recovery method.

[0206] In one example, when the failure of the first compute node is due to a hardware failure, the training task executed by the first compute node is restarted on either the second compute node or the first compute node.

[0207] In another example, when the failure of the first computing node is due to a software failure, the training task executed in the first subset is restarted on the second subset.

[0208] In the second possible implementation, the probability of restarting the training task based on task restart and model restart probability can be predicted according to the cause of failure of the first computing node. The method with the highest probability is determined as the training task recovery method that matches the cause of failure.

[0209] Optionally, probabilities can be obtained through probabilistic prediction models. These probabilistic prediction models include machine learning-based prediction models or neural network-based prediction models.

[0210] In the third possible implementation, the fault level of the first computing node can be determined based on the cause of the fault. The training task recovery method that matches the fault level of the first computing node in the task restart and model restart is determined as the training task recovery method that matches the cause of the fault.

[0211] In one example, the fault level of the first computing node can be determined based on the mapping relationship between fault levels and fault causes. This mapping relationship indicates the correspondence between fault levels and their respective causes. For example, a hardware fault corresponds to a first-level fault, and a software fault corresponds to a second-level fault.

[0212] Specifically, the training task recovery method whose fault level matches the fault level of the first computing node can refer to a training task recovery method whose fault level is the same as the fault level of the first computing node, or a training task recovery method whose fault level is higher than the fault level of the first computing node.

[0213] For example, if the failure of the first computing node is a hardware failure, either task restart or model restart can be used as the training task recovery method matching the failure cause. As another example, if the failure of the first computing node is a software failure, model restart can be used as the training task recovery method matching the failure cause.

[0214] Next, we will introduce the training task recovery method provided in this application embodiment, taking task restart as an example.

[0215] In this embodiment, when a hardware failure is determined to have occurred in the first computing node, the failure handling node sends a stop training notification to both the first and third computing nodes. In this way, by suspending the training process in the computing nodes that have not experienced a failure, the consistency of the training process is ensured.

[0216] The "Stop Training" notification is used to instruct the first and third computing nodes to stop executing the training task.

[0217] Optionally, the third computing node is a computing node in the first subset that has not experienced a failure.

[0218] In one possible implementation, after the first and third computing nodes stop the training task, the task is restarted by restarting the training task executed by the first computing node on the first computing node, or by restarting the training task of the first computing node on the second computing node.

[0219] Next, we will take restarting the training task executed on the first computing node as an example to introduce the task restart.

[0220] In one possible implementation, when the hardware failure is an accelerator card failure, the training task executed by the first computing node is restarted on the first computing node.

[0221] For example, in the AI ​​platform, the task scheduling module 1013 issues a task restart command to the first computing node. The first computing node executes the task restart command and restarts the training task on the first computing node. As another example, the process management module 611 deployed in the fault handling node issues a task restart command to the process management module 611 deployed in the first computing node.

[0222] For example, such as Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the restart task of the first computing node provided in an embodiment of this application. Computing nodes 1, 2, and 3 execute training job 1. When an accelerator card failure occurs in computing node 3, the training process in computing node 3 is paused. After the training process in computing node 3 is paused, computing node 3 restarts the training task. Training job 1 continues to execute.

[0223] like Figure 9 As shown, Figure 9 This is a schematic diagram of the training task recovery process in the scenario of restarting the first computing node, as provided in an embodiment of this application. The training task recovery process shown includes steps S91 to S92.

[0224] S91 sends a chip repair command to the first computing node.

[0225] The chip repair instruction is used to instruct the accelerator card in the first computing node to perform a chip repair operation.

[0226] Optionally, the chip repair operation is used to instruct the accelerator card configured in the first compute node to perform a restart.

[0227] In this embodiment, after receiving a notification to stop the training process, the process management module 611 deployed on the first computing node shuts down the training process. Upon receiving a chip repair instruction, the accelerator card management component on the first computing node detects that the training process has been shut down, performs a chip repair operation, and monitors the chip repair result. The chip repair result is then reported to the process management module 611 in the first computing node. The chip repair result includes whether the chip repair was successful or failed.

[0228] S92, when the chip repair operation in the first computing node is successfully executed, a notification to continue training is sent to the first computing node and the third computing node.

[0229] The "Continue Training" notification is used to instruct computing nodes to resume the training process corresponding to the distributed training task of the AI ​​model.

[0230] In one possible implementation, when the process management module 611 deployed in the first computing node receives the chip repair result and the chip repair is successful, the process management module 611 reports the chip status to the fault handling node. The fault handling node then sends a continue training notification to both the first and third computing nodes. For example, the process management module 611 deployed in the first computing node reports the chip status to the process management module 611 deployed in the fault handling node, and the process management module 611 deployed in the fault handling node sends a continue training notification to both the first and third computing nodes.

[0231] In one possible implementation, when the process management module 611 deployed in the first computing node receives a notification to continue training, it restarts the training process and continues to execute the training task.

[0232] For example, with Figure 6 Taking the provided computing node system architecture as an example, such as Figure 10 As shown, Figure 10This is a flowchart illustrating the process of restarting the task on the first computing node. The training accelerator card management component 63 detects a malfunction in the accelerator card through the subscription interface (S101). The training accelerator card management component 63 reports the accelerator card malfunction to the process management module 611 (S102). The process management module 611 reports the accelerator card malfunction to the process management module 611 in the fault handling node (S103). The process management module 611 in the fault handling node sends a stop training process notification to both the first and third computing nodes (S104). The process management module 611 in each computing node of the first subset shuts down the training process (S105). After the training accelerator card management component 63 in the first computing node detects that all task processes on the training accelerator card 62 have been shut down, it executes a chip recovery command (S106). The training accelerator card management component 63 reports the successfully repaired chip status to the process management module 611, and the process management module 611 reports the successful chip repair to the process management module 611 in the fault handling node (S107). In the fault handling node, the process management module 611 sends a continue training notification to all computing nodes in the first subset (S108). In the computing nodes of the first subset, the process management module 611 restarts the training process (S109).

[0233] In one possible implementation, to improve the task recovery rate, when it is determined that the chip repair operation in the first computing node has failed, the training task executed on the first computing node is restarted on the second computing node, or the training task executed on the computing nodes in the first subset is restarted on the second subset.

[0234] For example, when the process management module 611 in the first computing node receives a chip repair result indicating that the chip repair operation failed, the process management module 611 in the first computing node exits the training process and reports an exit code to the task scheduling module 1013 or resource management module 1012 in the AI ​​platform, or to the API server in the management node of the computing cluster. When the AI ​​platform or management node receives the exit code sent by the computing node, it determines that the chip repair operation in the first computing node has failed. For example, the process management module 611 in the first computing node reports either a first exit code or a second exit code. When the first exit code is received, the training task executed on the first computing node is restarted on the second computing node. When the second exit code is received, the training task executed on the computing nodes in the first subset is restarted on the second subset.

[0235] For example, taking the execution model restart as an example, such as Figure 11 As shown, Figure 11This is a schematic diagram of the distributed training process of an AI model in a scenario where the task is restarted on the first computing node, as provided in an embodiment of this application. The AI ​​platform distributes training jobs to the computing nodes, and the computing nodes execute the corresponding training jobs (S111). The process is then determined whether the job is complete. If the job is complete, the process ends. If the job is not complete, the computing node monitors the process status and hardware status on the computing node (S112). When a hardware failure is detected on the first computing node, it is determined whether the hardware failure is an accelerator card failure (S113). If the hardware failure is not an accelerator card failure, the first computing node is isolated, and the model is restarted (S114). If the hardware failure is an accelerator card failure, the training task executed by the first computing node is restarted on the first computing node (S115). The process is then determined whether the chip recovery of the first computing node was successful (S116). If the chip recovery was successful, the training job is restarted, and the process status and hardware status on the computing node are monitored again (S117). If the chip recovery failed, node isolation is performed (S118), isolating the first computing node and restarting the model.

[0236] Isolating the first computing node can refer to removing the first computing node from among multiple computing nodes executing the training job, so that the first computing node no longer executes the training task of the training job. Alternatively, isolating the first computing node can also mean that the first computing node will not be used to execute the training task of the task training job until the first computing node recovers from the failure.

[0237] In this embodiment of the application, after isolating the first computing node, node failure information is sent to the operation and maintenance personnel so that the operation and maintenance personnel can repair the failed node.

[0238] based on Figure 9 In the provided embodiment, when restarting the training task on the first computing node, the environment information of the training task is retained in the first computing node. The task process can directly use this environment information after restarting, resulting in minimal recovery time. This avoids the problem of increased training job interruption time caused by reconfiguring the training task's environment. It can minimize the time required to resume interrupted training jobs.

[0239] Next, we will take restarting the training task executed by the first computing node on the second computing node as an example to introduce the task restart.

[0240] For example, such as Figure 12 As shown, Figure 12This is a schematic diagram illustrating the restart of a training task executed by a first computing node on a second computing node, as provided in an embodiment of this application. Training job 1 is divided into four training tasks (training task 1, training task 2, training task 3, and training task 4), which are executed by computing nodes 1, 2, 3, and 4, respectively. If computing nodes 1, 2, and 3 do not fail (i.e., they are the third computing nodes), and computing node 4 fails (i.e., it becomes the first computing node), computing node 5 is reassigned as the second computing node, and training task 4 executed by computing node 4 is restarted on computing node 5. Computing nodes 1, 2, and 3 disconnect from computing node 4, isolating computing node 4, and connect to computing node 5 for communication. The communication network is updated to consist of computing nodes 1, 2, 3, and 5. Thus, compute nodes 1, 2, 3 and 5 can continue training, meaning that training job 1 will be executed by compute nodes 1, 2, 3 and 5.

[0241] like Figure 13 As shown, Figure 13 This is a schematic diagram of the training task recovery process on the second computing node provided in an embodiment of this application. The training task recovery process shown includes steps S131 to S132.

[0242] S131, determine the second computing node.

[0243] The second computing node is a computing node in the computing cluster that is not among the multiple computing nodes.

[0244] In one possible implementation, the AI ​​platform determines the second computing node from the computing cluster. Alternatively, the management node in the computing cluster determines the second computing node.

[0245] For example, taking the AI ​​platform determining a second computing node from a computing cluster as an example, the task scheduling module 1013 in the AI ​​platform sends a node application request to the resource management module 1012. The resource management module 1012 receives the node application request, reallocates the second computing node from the computing cluster, and returns the result of the reallocation of the computing node to the task scheduling module 1013.

[0246] The result of the reallocation of compute nodes can indicate that a second compute node has been allocated. The result of the reallocation of compute nodes includes the name, identifier, and specifications of the second compute node.

[0247] In one example, the task scheduling module 1013 determines the second computing node based on the name and identifier included in the result of the reallocation of computing nodes.

[0248] S132, restart the training task executed by the first computing node on the second computing node.

[0249] In one possible implementation, the task scheduling module 1013 sends the training framework, training dataset, and initial AI model to the second computing node, which then configures itself. Alternatively, the task scheduling module 1013 obtains the training framework, training dataset, and initial AI model from the first computing node. Or, the task scheduling module 1013 sends information from the third computing node to the second computing node, which then deploys the training framework and constructs a communication topology within the deployed training framework based on its own information and the information from the third computing node. The task scheduling module 1013 configures training tasks originally executed by the first computing node to be executed on the second computing node; that is, the second computing node runs a training process for the training tasks originally executed by the first computing node.

[0250] In one possible implementation, after configuring the second computing node, the AI ​​platform sends a notification to the third and second computing nodes to continue training, restarting the training task executed by the first computing node on the second computing node, thereby restarting the training job.

[0251] In one example, after configuring the second compute node, the AI ​​platform also isolates the first compute node.

[0252] For example, the AI ​​platform sends a node isolation notification to a third computing node. This notification instructs the third computing node to disconnect from the first computing node and to establish communication with the second computing node. Upon receiving the node isolation notification, the third computing node disconnects from the first computing node and establishes communication with the second computing node. Thus, through node isolation, the first computing node is removed from the first subset, and a second computing node is added to the first subset.

[0253] In one possible implementation, restarting the training task executed by the first computing node on the second computing node could be due to a hardware failure on the first computing node, other than an accelerator card malfunction, or a failure to repair the first computing node's chip.

[0254] For example, taking the failure of the first computing node chip repair operation as an example, such as... Figure 14 As shown, Figure 14 This is a schematic diagram of a process for restarting a training task executed by a first computing node on a second computing node, as provided in an embodiment of this application.

[0255] In the above Figure 10 Based on this, the training accelerator card management component 63 in the first computing node reports the status of the failed repair chip to the process management module 611 in the first computing node (S151). The process management module 611 in the first computing node exits the training process with the first exit code, and the API server obtains the first exit code of the first computing node (S152). The API server determines the second computing node, restarts the training task executed by the first computing node on the second computing node, and isolates the first computing node (S153). When the second computing node restarts successfully, the process management module 611 in the second computing node reports the information to the process management module 611 in the fault handling node (S154). The process management module 611 in the fault handling node sends a continue training notification to all computing nodes (S155). The process management module 611 in the computing node restarts the training process (S156).

[0256] For example, taking a hardware failure other than an accelerator card malfunction as an example, such as... Figure 15 As shown, Figure 15 This is another schematic diagram illustrating the process of restarting the training task executed by the first computing node on the second computing node, as provided in an embodiment of this application.

[0257] When the node management component 64 detects a hardware fault (excluding accelerator card failure) in the first computing node through the inspection script, it reports the fault to the process management module 611 in the first computing node (S161). The process management module 611 in the first computing node reports the fault to the process management module 611 in the fault handling node (S162). The process management module 611 in the fault handling node sends a stop training process notification to both the first and third computing nodes (S163). The first computing node shuts down the running task process (S164). The process management module 611 in the first computing node exits the training process with a first exit code, and the API server obtains this first exit code (S164). The API server determines the second computing node, restarts the training task executed by the first computing node on the second computing node, and isolates the first computing node (S165). When the second computing node restarts successfully, the process management module 611 in the second computing node reports the information to the process management module 611 in the fault handling node (S166). In the fault handling node, the process management module 611 sends a continue training notification to all computing nodes (S167). In the computing node, the process management module 611 starts the training process (S168).

[0258] In one possible implementation, to improve task recovery rate, the model is restarted when the second computing node fails to restart. This ensures that the training job can be resumed.

[0259] For example, if the second computing node fails to start, the process management module 611 deployed in the second computing node exits the training process and reports the second exit code to the API server in the management node of the computing cluster, or the task scheduling module 1013 or resource management module 1012 in the AI ​​platform, or reports the exit code upwards.

[0260] For example, such as Figure 16 As shown, Figure 16 This is a schematic diagram illustrating the distributed training process of an AI model in a scenario where the training task executed by the first computing node is restarted on the second computing node, as provided in an embodiment of this application. Figure 11 Based on this, determine if a software failure has occurred (S173). If a software failure is detected on the first computing node, execute a model restart (S174). If a hardware failure (not a software failure) is detected on the first computing node, isolate the first computing node (S175) and restart the training task executed on the first computing node on the second computing node. Determine if the restart of the second computing node was successful (S175). If the second computing node restarted successfully, the training job restarted (S176), and determine if the job was completed. If no job was completed, the process ends. If the job was not completed, continue monitoring the process status and hardware status on the computing nodes. If the restart of the second computing node failed, execute a model restart (S177).

[0261] based on Figure 13 In the provided embodiment, when restarting the training task executed by the first computing node on the second computing node, the job container, CKPT (check point) file, and data are only re-scheduled on the first computing node where the failure occurred, avoiding human-perceived job interruption and thus minimizing the time required to recover the interrupted job. The CKPT file is a binary file saved during the execution of the training task, containing all variables such as weights, biases, and gradients, used to resume training progress after a training task failure.

[0262] Next, taking model restart as an example, we will introduce the training task recovery method provided in the embodiments of this application.

[0263] For example, such as Figure 17 As shown, Figure 17 This is a schematic diagram of model restart provided in an embodiment of this application. In the above... Figure 12Based on this, when compute node 4 experiences a software failure, compute nodes 1, 2, 3, and 4 are isolated. Compute nodes 5, 6, 7, and 8 are selected from the compute cluster, forming a second subset. The training tasks executed by compute nodes 1, 2, 3, and 4 are restarted on this second subset. Thus, compute nodes 5, 6, 7, and 8 can continue training; that is, training job 1 will subsequently be executed by compute nodes 5, 6, 7, and 8.

[0264] like Figure 18 As shown, Figure 18 This is a schematic diagram of the training task recovery process in the model restart scenario provided in this application embodiment. A stop training notification is sent to each computing node in the first subset (S191). A second subset is determined (S192). A continue training notification is sent to each computing node in the second subset, and the training task executed in the first subset is restarted on the second subset (S193).

[0265] In this embodiment, after determining the second subset, each computing node in the second subset is configured. For example, each computing node in the second subset can be configured according to the configuration method of the second computing node described above, which will not be repeated here.

[0266] In one possible implementation, the AI ​​cloud platform's model restart could be due to a software failure on the first computing node, or a failure to restart the training task on the first computing node, or a failure to restart on the second computing node.

[0267] For example, taking the failure of the second compute node to restart as an example, such as Figure 19 As shown, Figure 19 This is a schematic diagram of a model restart process provided in an embodiment of this application.

[0268] In the above Figure 14 Based on this, the second computing node fails to start the training process (S201). The process management module 611 in the second computing node exits the training process with the second exit code, and the API server obtains the second exit code (S202). The API server determines the second subset and restarts the training tasks executed in the first subset on the second subset (S203).

[0269] like Figure 20 As shown, Figure 20 This is a schematic diagram of the distributed training process of the AI ​​model in a model restart scenario provided in this application embodiment. In the above... Figure 16Based on this, when a hardware failure occurs, the first compute node is isolated and a model restart is performed (S213). If a hardware failure is detected in the first compute node, a model restart is performed (S214). After the model restart, the training job continues. The system then determines whether the job is complete. If the job is not complete, the process terminates. If the job is not complete, the compute node monitors the process status and hardware status on the compute node.

[0270] based on Figure 18 The provided embodiment ensures the automatic recovery of training jobs by reallocating the computing nodes that execute training jobs when the model restarts.

[0271] In this embodiment, to better ensure the recovery of training jobs and avoid job interruptions caused by training task restart failures, a model restart is initiated when the training task restart fails during task recovery. Model restart serves as a fallback recovery mechanism for training jobs, allowing the job to unconditionally retry model restart without manual intervention. Thus, by combining task recovery methods for different fault levels, job recovery is guaranteed.

[0272] like Figure 21 As shown, Figure 21 This is a schematic diagram of a task recovery process for different causes, provided as an embodiment of this application.

[0273] In the above Figure 11 Based on this, determine whether a software failure has occurred on the first computing node (S223). If a software failure occurs on the first computing node, restart the model (S224). If a hardware failure is detected on the first computing node (but no software failure), determine whether the hardware failure is an accelerator card failure (S225). If the hardware failure is a hardware failure other than an accelerator card failure, isolate the first computing node and restart the training task executed by the first computing node on the second computing node (S226). If the hardware failure is an accelerator card failure, restart the training task executed by the first computing node on the first computing node (S227). Determine whether the chip recovery of the first computing node was successful (S228). If the chip recovery was successful, restart the training job and continue to monitor the process status and hardware status on the computing node (S229). If the chip recovery failed, execute S226.

[0274] Optionally, after restarting the training task executed by the first computing node on the second computing node, it is determined whether the restart of the second computing node was successful. If the second computing node restarts successfully, then S229 is executed. If the second computing node fails to restart, then S224 is executed.

[0275] The above description primarily focuses on the interaction between various modules within an AI platform, illustrating the methods provided in this application's embodiments. It is understood that the AI ​​platform includes corresponding hardware structures and / or software modules for executing various functions. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed through hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0276] This application embodiment can group the AI ​​platform into functional modules according to the above method example. For example, each functional group can be assigned to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the naming and grouping of modules in this application embodiment are illustrative and only represent one logical functional grouping. In actual implementation, there may be other grouping methods.

[0277] For example, an AI platform can also be named a distributed training device for AI models. Figure 22 As shown, Figure 22 This is a schematic diagram of the structure of the distributed training device 23 for the AI ​​model. The distributed training device 23 for the AI ​​model includes a fault perception module 231 and a management module 232.

[0278] The fault detection module 231 is used to obtain the cause of the first computing node's failure when the first computing node in the first subset fails. The cause of the failure includes at least one of a hardware failure or a software failure. For example, the fault detection module 231 performs the above... Figure 5 The S520 in the middle.

[0279] The management module 232 is used to resume model training according to the training task recovery method corresponding to the cause of the first computing node's failure. The training task recovery method indicates at least one model training recovery scheme, either model restart or task restart. For example, the management module 232 executes the above... Figure 5 The S530 in the middle.

[0280] The fault perception module 231 and the management module 232 can be implemented in software or hardware. For example, the implementation of the fault perception module 231 will be described below. Similarly, the implementation of the management module 232 can refer to the implementation method of the fault perception module 231.

[0281] As an example of a software functional unit, the fault detection module 231 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the computing instance may be one or more. For example, the fault detection module 231 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0282] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0283] As an example of a hardware functional unit, the fault detection module 231 may include at least one computing device, such as a server. Alternatively, the fault detection module 231 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0284] The fault detection module 231 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the communication module 121 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the communication module 121 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0285] It should be noted that, in other embodiments, the fault perception module 231 can be used to execute any step in the distributed training method of the AI ​​model. The management module 232 can be used to execute any step in the distributed training method of the AI ​​model. The steps implemented by the fault perception module 231 and the management module 232 can be specified as needed. By implementing different steps in the distributed training method of the AI ​​model through the fault perception module 231 and the management module 232, all functions of the distributed training device 23 of the AI ​​model can be realized.

[0286] This application also provides a distributed training system 24 for an AI model equipped with the aforementioned distributed training device 23. For example... Figure 23 As shown, the distributed training system 24 for the AI ​​model includes a client 20 and a distributed training device 23 for the AI ​​model.

[0287] The client 20 is used to send task information and training datasets to the distributed training device 23 of the AI ​​model, and to download the trained AI model.

[0288] The distributed training device 23 for the AI ​​model receives task information and training datasets to form training jobs. These training jobs are then distributed to multiple computing nodes that execute them. When a failure is detected in the first computing node among the multiple nodes, the cause of the failure is determined. Based on the training task recovery method corresponding to the cause of the failure in the first computing node, model training is resumed.

[0289] The client 20 and the distributed training device 23 for the AI ​​model can be implemented in software or hardware. The implementation method of the client 20 and the distributed training device 23 for the AI ​​model can refer to the implementation method of the fault perception module 231 described above. Further details are not provided in this embodiment.

[0290] This application embodiment also provides a computing device 25 for executing the distributed training method of the above-described AI model.

[0291] In one example, the computing device 25 may include, for example, Figure 22The distributed training device 23 for the AI ​​model shown includes a fault perception module 231 and a management module 232.

[0292] In another instance, such as Figure 24 As shown, the computing device 25 includes a bus 252, a processor 254, a memory 256, and a communication interface 258. The processor 254, the memory 256, and the communication interface 258 communicate with each other via the bus 252. The computing device 25 can be a server or a terminal device. It should be understood that this application does not limit the number of processors 254 and memory 256 in the computing device 25.

[0293] Bus 252 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 24 The bus 252 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 252 may include a path for transmitting information between various components of the computing device 25 (e.g., memory 256, processor 254, communication interface 258).

[0294] The processor 254 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0295] In this application, processor 254 performs the above-described... Figure 5 The method is illustrated. For example, the training job is distributed to multiple computing nodes that execute the training job. When a first computing node is detected to have failed, the cause of the failure of the first computing node is obtained. Based on the training task recovery method corresponding to the cause of the failure of the first computing node, model training is resumed.

[0296] Memory 256 may include volatile memory, such as random access memory (RAM). Processor 254 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0297] The memory 256 stores executable program code, which the processor 254 executes to implement the functions of the aforementioned fault perception module 231 and management module 232, thereby realizing the distributed training method of the AI ​​model. That is, the memory 256 stores instructions for executing the distributed training method of the AI ​​model.

[0298] In this embodiment, the memory 256 stores the training framework, task information, training data, and trained AI model.

[0299] The communication interface 258 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 25 and other devices or communication networks.

[0300] The distributed training method for AI models disclosed in the above embodiments can be applied to, or implemented by, processor 254. Processor 254 can be an integrated circuit chip with signal processor capabilities.

[0301] In implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 254 or by instructions in software form. The processor 254 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete vacuum tubes or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of the hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 256, and the processor 254 reads the information in memory 256 and completes the steps of the above method in combination with its hardware.

[0302] In one possible implementation, the processor 254 can also be used to execute a distributed training method for the AI ​​model. For specific implementation, please refer to the embodiments provided above for the distributed training method of the AI ​​model. The embodiments in this application will not be repeated here.

[0303] In this embodiment of the application, the chip system may be composed of chips or may include chips and other discrete devices.

[0304] This application embodiment also provides a computing device cluster 26 for executing the distributed training method of the above-described AI model.

[0305] In one example, the computing device cluster 26 may include, for example: Figure 22 The distributed training device 23 for the AI ​​model shown includes a fault perception module 231 and a management module 232.

[0306] In another example, the computing device cluster 26 may include, for example, Figure 23 The distributed training system 24 for the AI ​​model shown includes a client 20 and a distributed training device 23 for the AI ​​model.

[0307] In another instance, such as Figure 25 As shown, the computing device cluster 26 includes at least one such Figure 24The computing device 25 shown includes a bus 252, a processor 254, a memory 256, and a communication interface 258. The processor 254, memory 256, and communication interface 258 communicate with each other via the bus 252. The computing device 25 can be a server or a terminal device.

[0308] In one possible implementation, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 26 One possible implementation is shown. For example... Figure 26 As shown, the two computing devices 25A and 25B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, the memory 256 in computing device 25A stores instructions for executing the functions of the management module 232. Simultaneously, the memory 256 in computing device 25B stores instructions for executing the functions of the fault detection module 231.

[0309] Figure 26 The connection method between the computing device clusters shown can take into account the distributed training method of the AI ​​model provided in this application. This requires the fault detection module 231 to monitor whether computing nodes will fail, and the management module 232 to restore the job. The fault detection module 231 and the management module 232 can be deployed in different devices. Therefore, it is considered that the functions implemented by the management module 232 are executed by computing device 25A, and the functions implemented by the fault detection module 231 are executed by computing device 25B.

[0310] It should be understood that Figure 26 The functions of computing device 25A shown can also be performed by multiple computing devices 25. Similarly, the functions of computing device 25B can also be performed by multiple computing devices 25.

[0311] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the distributed training method of the AI ​​model described above.

[0312] For example, when a computer program product is run on at least one computing device, it causes the at least one computing device to perform... Figure 5 The distributed training method for the AI ​​model shown.

[0313] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. This program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a terminal of any of the foregoing embodiments, such as an internal storage unit including a data transmission end and / or a data receiving end, like a hard disk or memory of the terminal. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal. Further, the computer-readable storage medium can include both the internal storage unit and the external storage device of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0314] It should be understood that the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with relevant laws and regulations and do not violate public order and good morals. For example, in the technical solution of this application, the processing of user personal information is carried out with the user's authorization, and this will not be repeated hereafter.

[0315] It should be noted that the terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0316] It should be understood that in this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0317] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. For example, B can be determined based on A. It should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information. Furthermore, the term "connection" in the embodiments of this application refers to various connection methods, such as direct connection or indirect connection, to achieve communication between devices, and the embodiments of this application do not impose any limitations on this.

[0318] Unless otherwise specified, the term "transmission" in the embodiments of this application refers to bidirectional transmission, encompassing the actions of sending and / or receiving. Specifically, "transmission" in the embodiments of this application includes sending data, receiving data, or both sending and receiving data. In other words, data transmission here includes uplink and / or downlink data transmission. Data may include channels and / or signals; uplink data transmission refers to uplink channel and / or uplink signal transmission, and downlink data transmission refers to downlink channel and / or downlink signal transmission. The terms "network" and "system" in the embodiments of this application refer to the same concept; an all-optical network is an all-optical system.

[0319] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the grouping of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0320] In the several embodiments provided in this application, it should be understood that the disclosed distributed training device 23 and method for AI models can be implemented in other ways. For example, the embodiments of the distributed training device 23 for AI models described above are merely illustrative. For instance, the grouping of modules or units is only a logical functional grouping, and in actual implementation, there may be other grouping methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms.

[0321] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0322] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0323] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device, such as a microcontroller, chip, or processor, to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media for storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0324] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distributed training method for an artificial intelligence (AI) model, characterized in that, A first subset of the computing cluster is used for distributed training of the AI ​​model; each of the multiple computing nodes in the first subset executes a training task for the distributed training of the AI ​​model; the method includes: When the first computing node in the first subset fails, the cause of the failure of the first computing node is obtained, and the cause of failure includes at least one of hardware failure or software failure. Based on the training task recovery method corresponding to the cause of failure of the first computing node, model training is resumed. The training task recovery method is used to indicate at least one model training recovery scheme, either model restart or task restart.

2. The method according to claim 1, characterized in that, The hardware failure includes accelerator card failure or other hardware failures besides accelerator card failure; the software failure includes failures besides hardware failures.

3. The method according to claim 1 or 2, characterized in that, The step of resuming model training according to the training task recovery method corresponding to the cause of the first computing node failure includes: When the cause of the fault indicates a hardware failure, the training task executed by the first computing node is restarted on the second computing node or the first computing node; the second computing node is a computing node in the computing cluster other than the first subset.

4. The method according to claim 3, characterized in that, Before restarting the training task executed by the first computing node on the second computing node or the first computing node, the method further includes: A stop training notification is sent to the first computing node and to the third computing node; the stop training notification is used to indicate that the training task to be stopped; the third computing node is a computing node in the first subset that has not experienced a failure.

5. The method according to claim 4, characterized in that, After sending the stop training notification to the first computing node and the stop training notification to the third computing node, the method further includes: When the hardware failure is an accelerator card failure, a chip repair command is sent to the first computing node; the chip repair command is used to instruct the accelerator card in the first computing node to perform a chip repair operation. When the accelerator card in the first computing node successfully performs the chip repair operation, a continue training notification is sent to the first computing node and the third computing node; the continue training notification is used to instruct the resumption of the distributed training task of the AI ​​model.

6. The method according to claim 5, characterized in that, After sending the chip repair command to the first computing node, the method further includes: If the chip repair operation fails to be performed on the accelerator card in the first computing node, the training task executed by the first computing node will be restarted on the second computing node.

7. The method according to claim 4, characterized in that, After sending the stop training notification to the first computing node and the stop training notification to the third computing node, the method further includes: When the hardware failure is a hardware failure other than the accelerator card failure, the training task executed by the first computing node is restarted on the second computing node; Send a continue training notification to the second computing node and the third computing node.

8. The method according to claim 7, characterized in that, After sending the continue training notification to the second computing node and the third computing node, the method further includes: Perform node isolation on the first computing node.

9. The method according to claim 7 or 8, characterized in that, After restarting the training task executed by the first computing node on the second computing node, the method further includes: If the second computing node fails to restart, the training tasks executed in the first subset are restarted on the second subset; the second subset includes multiple computing nodes in the computing cluster other than the first subset; When the second computing node restarts successfully, the continued training notification is sent to both the second computing node and the third computing node.

10. The method according to claim 1 or 2, characterized in that, The step of resuming model training according to the training task recovery method corresponding to the cause of the first computing node failure includes: When the cause of the failure indicates a software failure, the training task executed in the first subset is restarted on the second subset; the second subset includes multiple computing nodes in the computing cluster other than the first subset.

11. The method according to claim 10, characterized in that, Restarting the training task executed in the first subset on the second subset includes: A stop training notification is sent to each computing node in the first subset; the stop training notification is used to indicate that the training task to be executed should be stopped. A continue training notification is sent to each computing node in the second subset to restart the training task executed in the first subset on the second subset; the continue training notification is used to instruct each computing node in the second subset to resume the training process corresponding to the training task of the distributed training of the AI ​​model.

12. The method according to any one of claims 1 to 11, characterized in that, The first subset includes the fault handling node; the computing cluster includes a management node; The management node is a computing node in the computing cluster other than the first subset; The process of obtaining the cause of failure of the first computing node includes: The fault handling node obtains the fault cause sent by the first computing node; The management node obtains the exit code sent by the first computing node, and based on the exit code, obtains the cause of the failure of the first computing node; the exit code includes a first exit code or a second exit code; the first exit code indicates a hardware failure, and the second exit code indicates a software failure.

13. A distributed training device for an AI model, characterized in that, The computing cluster includes multiple computing nodes for distributed training of the AI ​​model; each of the multiple computing nodes executes a training task for the distributed training of the AI ​​model. The device includes: The fault detection module is used to obtain the cause of the fault of the first computing node when the first computing node among the plurality of computing nodes fails, and the cause of the fault includes at least one of hardware fault or software fault. The management module is used to resume model training according to the training task recovery method corresponding to the cause of the failure of the first computing node. The training task recovery method is used to indicate at least one model training recovery scheme, namely model restart or task restart.

14. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device uses instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 12.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 12.

Images