Distributed training system, method, and controller for artificial intelligence models

Abstract

The application discloses a distributed training system and method of an artificial intelligence model and a controller, and belongs to the technical field of artificial intelligence infrastructure. The system comprises a controller, a monitor, n working nodes and m hot standby nodes, wherein n and m are positive integers; the controller is configured to deploy a training program corresponding to a training task to the n working nodes and the m hot standby nodes; the n working nodes are configured to cooperatively execute the training task; the monitor is configured to monitor the health status of the n working nodes; and the controller is configured to, in the case where a first working node in the n working nodes fails, switch a training subtask corresponding to the first working node to a first hot standby node in the m hot standby nodes for execution.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence infrastructure technology, and in particular to a distributed training system, method and controller for an artificial intelligence model. Background Technology

[0002] As the parameters of artificial intelligence (AI) models become increasingly large—for example, large language models have hundreds of millions of parameters—distributed training methods are commonly used to train AI models. Distributed training methods utilize multiple computing nodes to complete the training task. However, during the collaborative training process across multiple computing nodes, hardware and software issues can occur, leading to training interruptions.

[0003] In related technologies, checkpoints are periodically set for the training tasks of AI models. When a training task is interrupted due to a failure, a backup node is found from the backup node pool. The training program corresponding to the failed node is deployed to the backup node, and the training task is started on the backup node.

[0004] The replacement time for the aforementioned faulty nodes is too long, which will slow down the training speed. Summary of the Invention

[0005] This application provides a distributed training system, method, and controller for artificial intelligence models, which can be used to reduce the impact of faults on training tasks. The technical solution is as follows:

[0006] On the one hand, embodiments of this application provide a distributed training system for an artificial intelligence model, the system comprising: a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers;

[0007] The controller is used to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0008] The n working nodes are used to collaboratively execute the training task;

[0009] The monitor is used to monitor the health status of the n working nodes;

[0010] The controller is configured to, in the event that a first working node fails among the n working nodes, switch the training subtask corresponding to the first working node to be executed on the first hot standby node among the m hot standby nodes.

[0011] In one possible implementation, the monitor is configured to select the first hot standby node from the m hot standby nodes if a first working node fails among the n working nodes.

[0012] The controller is configured to set the working role of the first hot standby node as the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0013] In one possible implementation, the controller is configured to send the context information of the training subtask corresponding to the first working node to the second working node;

[0014] The second working node is used to re-execute or continue executing the training subtask corresponding to the first working node through the training program based on the context information.

[0015] In one possible implementation, the context information includes at least one of the following:

[0016] The task role of the first working node in the training task;

[0017] The identifier of the first working node in the training task;

[0018] The Internet Protocol IP address of the master node of the training task;

[0019] The port address of the master node for the training task;

[0020] The address of the training dataset for the training task;

[0021] The storage address of the last checkpoint of the training task;

[0022] The model configuration of the AI ​​model trained by the training task.

[0023] In one possible implementation, the second working node further includes: a training process control unit;

[0024] The training process control unit is used to start the training process corresponding to the training program; and through the training process, based on the context information, to re-execute or continue the training sub-task corresponding to the first working node.

[0025] In one possible implementation, the controller is further configured to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training sub-task, the restart training task message being used to trigger the various working nodes to re-coordinate the execution of each training sub-task of the training task.

[0026] In one possible implementation, the monitor is configured to select a first hot standby node from a pool of m hot standby nodes based on selection factors, provided that a first working node among the n working nodes fails; the selection factors include at least one of the following:

[0027] The communication speed between the first working node and the m hot standby nodes;

[0028] The communication bandwidth between the first working node and the m hot standby nodes;

[0029] The communication speed between the third working node and the m hot standby nodes;

[0030] The third working node is a working node that has a collaborative requirement with the first working node.

[0031] In one possible implementation, the n working nodes are used to report their own resource usage and training status information to the monitor;

[0032] The monitor is used to determine the first working node that needs to be switched due to a fault, based on the resource usage and training status information of the n working nodes.

[0033] In one possible implementation, the controller is used to deploy the training program corresponding to the training task to multiple computing nodes; and to divide the multiple computing nodes into the n working nodes and the m hot standby nodes.

[0034] In one possible implementation, the controller is configured to adjust the number of the m hot standby nodes based on the number of fault exchanges in the most recent time period.

[0035] The number of fault exchanges refers to the number of exchanges between working nodes and hot standby nodes caused by faults.

[0036] On the other hand, embodiments of this application provide a distributed training method applied to a distributed training system, the distributed training system including a controller, a monitor, n worker nodes and m hot standby nodes, where n and m are positive integers, the method including:

[0037] The controller deploys the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0038] The n working nodes collaboratively execute the training task;

[0039] The monitor monitors the health status of the n working nodes;

[0040] If the first working node among the n working nodes fails, the controller will switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution.

[0041] On the other hand, embodiments of this application provide a controller applied to a distributed training system. The distributed training system includes the controller, a monitor, n worker nodes, and m hot standby nodes, where n and m are positive integers. The controller includes:

[0042] The deployment module is used to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0043] The switching module is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when the first working node fails among the n working nodes.

[0044] On the other hand, embodiments of this application provide a distributed training system, the system including: a controller and a monitor;

[0045] The controller is used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes, where n and m are positive integers;

[0046] The monitor is used to monitor the health status of the n working nodes;

[0047] The controller is configured to, in the event that a first working node fails among the n working nodes, switch the training subtask corresponding to the first working node to be executed on the first hot standby node among the m hot standby nodes.

[0048] The technical solution provided in this application has at least the following beneficial effects:

[0049] The hot standby node mechanism in this application greatly reduces the replacement time of faulty nodes. When the monitor detects a fault in a working node (such as the first working node), the controller can switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution, avoiding long waiting times for other nodes, thereby effectively improving the fault handling speed and reducing the impact of faults on training tasks. Since the hot standby node can quickly take over the tasks of the faulty node, the working nodes can focus more on the training task during normal operation without reserving too many resources or processing capabilities for possible faults, thus providing a more stable environment for distributed training. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the structure of a distributed training system provided in an embodiment of this application;

[0052] Figure 2 This is a flowchart of a distributed training method provided in an embodiment of this application;

[0053] Figure 3 This is an overall flowchart of a distributed training method provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the structure of a controller provided in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of a distributed training system provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of the structure of a computing node virtualization system provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] First, a brief introduction to the terms used in the embodiments of this application:

[0061] Distributed training is a training method that divides and distributes computational tasks across multiple computing devices (such as multiple graphics processing units, multiple machines, etc.) for simultaneous computation. Its core idea is to accelerate the model training process through parallel computing.

[0062] In related technologies, large AI models are typically trained in a distributed environment, relying on the collaboration of multiple computing nodes. Distributed training often experiences interruptions due to various hardware and software issues. The current approach to this problem involves: first, periodically setting checkpoints for the AI ​​model's training task; second, identifying and isolating the faulty node and replacing it with a healthy backup node when a training failure occurs; and finally, restarting the training task from the previous checkpoint. However, during this replacement process, other computing nodes are in a waiting state and cannot continue effective training, resulting in excessively long replacement times for faulty nodes.

[0063] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0064] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0065] Figure 1This diagram illustrates the structure of a distributed training system according to an embodiment of this application. The system includes: a controller 101, a monitor 102, n worker nodes 103, and m hot standby nodes 104, where n and m are positive integers. The controller 101 is used to deploy the training program corresponding to the training task to multiple computing nodes; divide the multiple computing nodes into n worker nodes 103 and m hot standby nodes 104; and adjust the number of m hot standby nodes 104 based on the number of fault exchanges in the recent time period; wherein, the number of fault exchanges is the number of exchanges between worker nodes and hot standby nodes caused by faults; the n worker nodes 103 are used for coordination... The system performs training tasks and reports its resource usage and training status information to monitor 102. Monitor 102 monitors the health status of n working nodes 103. Based on the resource usage and training status information of the n working nodes 103, it determines the first working node that needs to be switched due to a failure. If the first working node among the n working nodes 103 fails, it selects a first hot standby node from m hot standby nodes 104 based on selection factors, where the selection factors include at least one of the following: communication speed between the first working node and the m hot standby nodes 104; communication speed between the first working node and the m hot standby nodes 104; and communication speed between the first working node and the m hot standby nodes 104. The communication bandwidth between the 4 nodes; the communication speed between the third working node and the m hot standby nodes 104; wherein the third working node is a working node that has a collaborative requirement with the first working node; the controller 101 is used to set the working role of the first hot standby node as the second working node when the first working node fails among the n working nodes 103; to send the context information of the training subtask corresponding to the first working node to the second working node, wherein the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; the model configuration of the AI ​​model trained by the training task; to send a restart training task message to each working node corresponding to the training task, the restart training task message is used to trigger each working node to re-coordinate and execute each training subtask of the training task; the second working node also includes: a training process control unit; the training process control unit is used to start the training process corresponding to the training program; and to re-execute or continue to execute the training subtask corresponding to the first working node based on the context information through the training process.

[0066] like Figure 1As shown in the embodiment of this application, a distributed training system for an artificial intelligence model is provided. The system includes: a controller 101, a monitor 102, n worker nodes 103 and m hot standby nodes 104, where n and m are positive integers. The controller 101 is used to deploy the training program corresponding to the training task to the n worker nodes 103 and the m hot standby nodes 104. The n worker nodes 103 are used to collaboratively execute the training task. The monitor 102 is used to monitor the health status of the n worker nodes 103. The controller 101 is used to switch the training subtask corresponding to the first worker node to the first hot standby node among the m hot standby nodes 104 for execution when the first worker node among the n worker nodes 103 fails.

[0067] The controller 101 occupies a core control position in the distributed training system, and is responsible for coordinating and managing various operations throughout the distributed training process.

[0068] The training task is the purpose of the entire distributed training system; it can be understood as a specific goal, such as training a large-scale neural network model for image recognition, natural language processing, or other artificial intelligence applications. The training task has clearly defined input and output requirements. The input might be a large dataset, such as a large amount of image or text data, while the output is the trained model parameters. A large training task is usually decomposed into multiple subtasks. In a distributed training system, these subtasks are assigned to different worker nodes for collaborative execution. For example, when training a deep neural network, the computational tasks corresponding to different layers or different subsets of data can be considered subtasks. These subtasks are computed in parallel on various worker nodes and finally combined through a pre-defined mechanism (such as parameter synchronization) to obtain the final training result.

[0069] The training program corresponding to a training task can be understood as the algorithmic implementation to achieve the training task's objective. It includes various algorithms used to train the model, such as stochastic gradient descent and its variants in neural network training. These algorithms are implemented in the training program through programming to process the input data, calculate gradients based on the loss function, and update the model's parameters. For example, in a training program written in the Python programming language...

[0070] The training procedure also defines the model structure to be trained, including the number of layers, the number of neurons in each layer, and the type of activation function. For example, in a training procedure for building a convolutional neural network, the parameters and connection methods of convolutional layers, pooling layers, and fully connected layers are defined. Furthermore, the training procedure also includes data processing logic. For example, how to read input data, how to preprocess the data (such as normalization, data augmentation, etc.), and how to distribute the data to different computing nodes for processing.

[0071] The training program interacts with hardware, particularly with hardware devices such as graphics processing units (GPUs) on the computing nodes. It leverages the parallel computing capabilities of the GPUs by calling relevant libraries and interfaces to accelerate the training process.

[0072] Worker nodes can be understood as execution units in a distributed training system. They are responsible for actually executing training tasks, and multiple worker nodes work together to complete the entire training task. For example, in a large-scale neural network training task, worker nodes will perform calculations on the subtasks assigned to them, including forward propagation of data, calculation of the loss function, backpropagation to calculate gradients, and other operations. They will also cooperate with other worker nodes to update the model parameters.

[0073] A hot standby node can be understood as a backup mechanism. While the working node is operating normally, it does not initiate training-related processes and remains in a ready-to-wait state for the training program. This ready-to-wait state means that the hot standby node is prepared to accept tasks at any time. For example, it has already installed and configured the training program, and its resources have been allocated; as long as the controller 101 issues a command, it can quickly start the training task.

[0074] As an optional implementation, the n working nodes 103 and the m hot standby nodes 104 are identical in terms of software environment, hardware environment, and the running configuration file of the training program.

[0075] Since the hot standby nodes and working nodes have the same resources and hardware / software environment, when the monitor detects a failure in a working node (such as the first working node), the controller can quickly switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes 104 for execution, thus improving the efficiency of fault handling.

[0076] Worker nodes and hot standby nodes maintain consistency in hardware resources. Whether existing as bare-metal physical machines, or as Pods in the open-source container orchestration system Kubernetes, or other resource management components in a cluster scheduling system, they possess similar computing resources, storage resources (such as memory and hard drives), and network resources (such as network interfaces and bandwidth). For example, in a distributed training system, if worker nodes are equipped with a pre-defined model and number of graphics processors (GPUs) to accelerate deep learning training tasks, then the hot standby node will also be equipped with the corresponding number of GPUs. On the software side, the operating systems, drivers, deep learning frameworks, and other related software libraries are identical for both worker and hot standby nodes. Taking PyTorch for deep learning training as an example, both worker and hot standby nodes have the same version of the PyTorch framework and its dependent libraries installed. Furthermore, their operating system versions and configurations are identical, including kernel parameter settings and file system formats. This ensures that during failover, the training program can run on the hot standby node without additional software adaptation.

[0077] During normal operation, although hot standby nodes do not participate in the execution of training tasks, they are related to worker nodes in terms of resource allocation. When allocating resources, controller 101 needs to consider the needs of both worker and hot standby nodes. For example, in a resource-constrained system, to ensure sufficient hot standby nodes, hardware resources need to be allocated reasonably between worker and hot standby nodes. Allocating too many resources to hot standby nodes will reduce the available resources for worker nodes, affecting the execution efficiency of training tasks; conversely, if hot standby nodes have too few resources, they may not be able to meet the needs during failover.

[0078] While hot standby nodes do not directly participate in training tasks, they do have a certain impact on the overall system performance, and this impact is closely related to the worker nodes. For example, hot standby nodes consume certain system resources (such as storage resources for storing training program-related files), which reduces the total resources available to worker nodes. Simultaneously, the system consumes certain computing resources when managing hot standby nodes (such as the controller 101 monitoring the status of hot standby nodes). Therefore, when designing a distributed training system, it is necessary to balance the number and resource allocation of hot standby nodes and worker nodes to optimize the overall system performance.

[0079] The process of deploying the training program for controller 101 is described below.

[0080] As an optional implementation, the controller 101 is used to create training tasks, deploy the training program corresponding to the training tasks to multiple computing nodes; divide the multiple computing nodes into n working nodes 103 and m hot standby nodes 104, with the m hot standby nodes 104 in a ready-to-wait state; set the start command and start parameters for the training tasks; and coordinate the n working nodes 103 to start training based on the start command and start parameters.

[0081] The process by which the controller 101 deploys the training program corresponding to the training task to the working node and the hot standby node demonstrates the role of the controller 101 in allocating resources and scheduling training tasks, thereby achieving full utilization of resources.

[0082] Compute nodes are the basic building blocks of a distributed training system. From a hardware perspective, a compute node typically refers to a graphics processing unit (GPU) server. A GPU server contains several key hardware components, the most important of which are multiple GPUs used for training. In distributed training scenarios (such as training with thousands or tens of thousands of GPUs), multiple compute nodes work collaboratively. Each compute node undertakes a portion of the training task, and they communicate with each other via a network, exchanging information such as data and model parameters. For example, when training a large deep learning model, different compute nodes are responsible for processing different batches of data or updating parameters for different parts of the model.

[0083] The controller 101 first conducts a comprehensive evaluation of the hardware resources and software environment of the entire distributed training system; based on the evaluation results of the hardware resources and software environment, it lays out a node list, which includes the computing nodes required for the training task.

[0084] The node list layout can be, but is not limited to: determining the computing nodes in the node list based on the controller 101's autonomous management functions (when the computing nodes are hardware, the controller 101 autonomously searches for and allocates each computing node; when the computing nodes are containers, the controller 101 autonomously creates each computing node), and performing subsequent maintenance and recycling of the created computing nodes; or, the distributed system includes a resource management system for managing computing nodes, and the controller 101 requests a node list from the resource management system. This request process involves sending a request containing resource requirement information to the resource management system. For example, the request might include the need for a certain number of nodes with graphics processor models and sufficient memory, and require that the nodes have a preset version of the operating system and a deep learning framework installed; the resource management system processes the controller 101's request based on its own resource allocation strategy and the current available resources. If there are enough available resources to meet the controller 101's needs, the resource management system will return a node list containing the required node information to the controller 101.

[0085] After receiving the node list returned by the resource management system, the controller 101 first parses the node information in the computing node list. The node information includes, but is not limited to: the node's unique identifier (such as hostname), hardware configuration information (such as the number of CPU cores), and software environment information (such as operating system version, installed software packages, etc.). By parsing the node information, the controller 101 gains a comprehensive understanding of the basic situation of each computing node, preparing for subsequent deployment and role assignment.

[0086] When deploying the role information (worker nodes or hot standby nodes) of computing nodes, controller 101 first considers the trade-off between reliability and performance. To improve the reliability of the distributed training system, a sufficient number of hot standby nodes are needed. However, too many hot standby nodes will consume resources and affect the overall performance of the system. Controller 101 can set a reliability metric (such as the maximum allowable failure probability) and a performance metric (such as the expected percentage increase in training speed). Then, based on these two metrics and the analysis of the training task, it determines the appropriate ratio of worker nodes to hot standby nodes.

[0087] Furthermore, when the controller creates training tasks, different training tasks have different characteristics, which also affect the allocation of computing node roles. For computationally intensive training tasks, such as large-scale natural language processing model training, more worker nodes are needed to accelerate the computation process. For data-intensive training tasks, such as processing massive amounts of image data, data storage and transmission factors need to be considered in the allocation of worker and hot standby nodes. If the training task has high real-time requirements for data, the controller 101 tends to allocate more computing nodes as worker nodes to speed up data processing. At the same time, to ensure system reliability, a certain proportion of hot standby nodes will be reserved. For example, for a real-time video stream analysis training task, worker nodes need to process each frame of video data quickly; in this case, 80% of the computing nodes can be allocated as worker nodes, and 20% as hot standby nodes.

[0088] Controller 101 also needs to consider the balanced allocation of resources between worker nodes and hot standby nodes. In terms of hardware resources, it must ensure that both worker nodes and hot standby nodes receive relatively balanced computing resources, storage resources (such as memory and hard drives), and network resources (such as bandwidth). For example, in a system consisting of 100 compute nodes, if 60 compute nodes are allocated as worker nodes and 40 as hot standby nodes, it must be ensured that there is no significant difference in average graphics processing unit (GPU) computing power, memory capacity, and network bandwidth between worker nodes and hot standby nodes.

[0089] Based on the strategy for assigning role information to compute nodes, controller 101 marks the role information of compute nodes in the node list. For nodes identified as worker nodes, controller 101 adds a worker node identifier to their node information, such as setting a target flag or writing a "worker node" label in the node configuration file. Similarly, corresponding identifiers are added for hot standby nodes.

[0090] In a distributed training system, resources include computing resources (such as graphics processing units and central processing units) and storage resources (such as hard disk space). Controller 101 needs to understand the resource status of each computing node (working node and hot standby node), such as the model, number, and memory size of the graphics processing units on each computing node; the number of cores and clock speed of the central processing unit; and the available storage capacity. When deploying the training program corresponding to the training task to the computing node, controller 101 can allocate appropriate computing nodes according to the resource requirements of the training program (such as how much graphics processing unit memory is needed to store model parameters).

[0091] Furthermore, the controller 101 is responsible for decomposing the training task into subtasks and assigning these subtasks to different worker nodes. For a complex training task, such as training a large deep learning model, the controller 101 must consider the structural characteristics of the model (e.g., some layers can be computed in parallel, while others have a specific computational order) and the characteristics of the data (e.g., the distribution of the data) to rationally allocate the subtasks. When allocating subtasks, the controller 101 also considers factors such as the computing power of the worker nodes and the network connectivity. For example, subtasks with higher computational demands are assigned to worker nodes with stronger computing power, while subtasks with higher network communication requirements are assigned to worker nodes with better network connectivity to improve the overall efficiency of the training task.

[0092] The method for determining the values ​​of n and m is explained below.

[0093] The methods for determining the values ​​of n and m include, but are not limited to: determining them based on historical data; or, determining them based on real-time monitoring and feedback adjustments; or, determining them based on cost-effectiveness.

[0094] When the values ​​of n and m are determined based on historical data, the controller 101 determines their values ​​by analyzing the failure rate in the historical data. It collects failure data of worker nodes from past training tasks and calculates the failure rate. For example, if a worker node failed 10 times in the past 100 training tasks, the failure rate is 10%. The number of hot standby nodes, m, is determined based on the failure rate. If it is desired to be able to quickly resume training in the event of a failure and to tolerate a certain level of failure risk, a hot standby node ratio can be set based on the failure rate. For example, if the failure rate is 10%, m can be set to 10%-20% of n; that is, if there are 100 worker nodes (n=100), the number of hot standby nodes, m, can be set to 10-20. Furthermore, it examines the resource utilization of worker nodes and hot standby nodes in past training tasks, including GPU utilization, CPU utilization, and memory utilization. If it is found that worker nodes have high resource utilization in most cases, while hot standby nodes have low resource utilization, the ratio of n and m needs to be adjusted. Increase the number of worker nodes (n) and decrease the number of hot standby nodes (m) to improve the overall resource utilization of the system. For example, if the average GPU utilization of worker nodes reaches 80%, while the GPU utilization of hot standby nodes is only 10%, the number of hot standby nodes can be reduced, and the freed-up resources can be allocated to worker nodes, thereby improving training efficiency.

[0095] When the values ​​of n and m are adjusted based on real-time monitoring and feedback, during training, the controller 101 monitors the resource usage of worker nodes and hot standby nodes in real time through the monitor 102. For example, it acquires information such as the utilization rate of the graphics processor, the utilization rate of the central processing unit, and the memory usage of the nodes at regular intervals (e.g., every minute). Based on the real-time monitoring results, if it is found that the worker nodes are experiencing resource shortages while the hot standby nodes have more idle resources, the controller 101 can adjust the values ​​of n and m. This involves converting some hot standby nodes into worker nodes, increasing the number of worker nodes n, and simultaneously decreasing the number of hot standby nodes m to meet the resource requirements of the training task.

[0096] The startup command is the basic instruction to initiate a training task. In a large-scale distributed training scenario, it is used to start the entire training process. The basic command tells the compute nodes the specific operations to perform, i.e., to run the training script. It is a key identifier that clearly defines the entry point of the training task, allowing worker nodes to know where to begin executing training-related code.

[0097] Startup parameters provide more detailed configuration information for the training task. These parameters can adjust various characteristics of the training task. Taking image classification training as an example, startup parameters include batch size. Batch size determines the number of data samples used for training in each iteration. If the batch size is set to 32, it means that the model will process 32 image samples simultaneously in each training step.

[0098] After the controller completes the startup command and startup parameters, it needs to distribute this information to the n worker nodes 103. First, it must be ensured that each worker node receives the same startup command and startup parameters to guarantee the consistency of the training task across all nodes. For example, in a distributed training system consisting of 10 worker nodes 103, the controller will send the startup command and parameters to each worker node.

[0099] The controller ensures that all worker nodes train according to the same rules and configuration in a large-scale distributed training environment by setting and distributing unified startup commands and parameters, and coordinating the startup of worker nodes.

[0100] As an optional implementation, when the controller starts the training task, m hot standby nodes are started simultaneously.

[0101] When the controller initiates a training task, simultaneously starting m hot standby nodes 104 avoids the need to create an initial runtime environment in the event of a failure. Since the m hot standby nodes 104 are already in a ready-to-wait state, they can immediately take over the tasks of the failed worker node without requiring additional operations or events to trigger them. This significantly reduces fault recovery time and improves training efficiency.

[0102] As an optional implementation, the controller 101 is used to adjust the number of m hot standby nodes 104 based on the number of fault exchanges in the most recent time period; wherein, the number of fault exchanges is the number of exchanges between working nodes and hot standby nodes caused by faults.

[0103] The number of fault exchanges within a recent time period reflects the current stability and reliability of the distributed training system. By analyzing the number of fault exchanges within this period, potential problems in the distributed training system can be identified in a timely manner, or adjustments to the number of hot standby nodes can be made. This enables a dynamic and adaptive system management mechanism to adapt to the ever-changing operating environment and training task requirements of the distributed training system.

[0104] The recent time period is a relative time concept; it refers to a time window within the distributed training process. The length of this time window can be set according to the situation. For example, it can be set to the past hour, two hours, or one day. Within this time window, the distributed training system counts and analyzes the number of failure exchanges in order to adjust the number of hot standby nodes in a timely manner based on the failure exchange situation.

[0105] Faults include, but are not limited to, at least one of the following: graphics processing card failure, central processing unit failure, memory failure, network device failure, deep learning framework failure, operating system failure, or driver failure. This application does not limit the types of faults.

[0106] For example, graphics processing units (GPUs) may experience hardware failures, such as damaged video memory, overheating of the core chip, or short circuits. Damaged video memory can lead to data storage and retrieval errors, affecting data processing during training. For instance, during deep learning model training, when a large amount of model parameters and data needs to be loaded into video memory, a memory failure can result in data loss or the model failing to train properly. Overheating of the core chip can be caused by prolonged high-load operation, cooling system malfunctions, etc. Overheating can cause a decrease in GPU performance or even automatic frequency reduction, thereby slowing down training or interrupting training.

[0107] For example, a central processing unit (CPU) failure may manifest as a core failure, a cache failure, or an instruction set execution error. A core failure can affect task scheduling and data processing capabilities, a cache failure can cause a significant decrease in data read speed, and an instruction set execution error can cause program malfunctions.

[0108] Fault swap count is a metric in the aforementioned distributed training system. It refers to the number of swaps between worker nodes and hot standby nodes due to faults. When one of the n worker nodes 103 fails, the controller 101 switches the training subtask corresponding to the failed worker node to one of the m hot standby nodes 104 for execution. This operation is counted as one fault swap. For example, in a distributed training process, if in a system with 10 worker nodes and 5 hot standby nodes, 3 worker nodes fail successively, and each failure successfully switches the task from the failed worker node to a hot standby node, then the fault swap count is 3. This metric reflects the frequency of task transfers between worker nodes and hot standby nodes due to faults during the operation of the distributed training system. It is a dynamic statistic that is continuously updated as the training process progresses, providing a clear indication of the system's stability and reliability.

[0109] Optionally, the method for adjusting the number of m hot standby nodes 104 based on the number of fault exchanges includes, but is not limited to: linearly increasing or decreasing the number of m hot standby nodes 104 according to a certain proportion based on the number of fault exchanges. For example, if it is set that one hot standby node is added for every 10 fault exchanges, the number of hot standby nodes is gradually increased according to this proportion after the number of fault exchanges reaches a threshold (e.g., 50 times); or, adjusting the number of m hot standby nodes 104 according to the growth trend of the number of fault exchanges. For example, when the number of fault exchanges begins to increase, the number of hot standby nodes can be increased slowly, such as adding 0.5 hot standby nodes each time when the number of fault exchanges is between 1 and 10 times (a similar increase effect can be achieved by adding some hardware resources or adjusting software configuration). When the number of fault exchanges increases rapidly (e.g., after exceeding 10 times, 2 hot standby nodes are added for every 5 additional fault exchanges).

[0110] The following section describes how controller 101 deploys the training program corresponding to the training task to the worker node.

[0111] The controller 101 deploys the training program corresponding to the training task to the working node, including at least one of the following: resource adaptability check, training task decomposition and allocation, and program installation and configuration.

[0112] Before deploying the training program corresponding to the training task to the worker node, the controller 101 first performs a hardware resource compatibility check. This check includes, but is not limited to, verifying whether the hardware resources on the worker node, such as the graphics processor, central processing unit, and memory, meet the requirements of the training program. For example, for a training program that requires a large amount of GPU memory to store model parameters, the controller 101 must ensure that the GPU memory of the worker node is sufficiently large. If the hardware resources of the worker node do not meet the requirements of the training program, the controller 101 will take measures such as adjusting the parameters of the training program (e.g., reducing the batch size to reduce GPU memory usage) or replacing the worker node (if other suitable worker nodes are available).

[0113] Controller 101 also checks the software resource compatibility of the worker nodes. This check includes, but is not limited to, verifying whether the worker nodes have the necessary operating system, drivers, deep learning frameworks, and other relevant software libraries installed. For example, if the training program is based on PyTorch, then the corresponding version of PyTorch and its dependencies must be installed on the worker nodes. If the software resources are incompatible, controller 101 will install or update the relevant software resources on the worker nodes to ensure that the training program can run correctly on the worker nodes.

[0114] The controller 101 decomposes the training program corresponding to the training task into subtasks based on the parallelism characteristics of the training task. For data-parallel training tasks, the controller 101 can divide the data into several subsets according to the size of the dataset, with the computation corresponding to each subset being a subtask. For example, a dataset containing 1 million images can be decomposed into 10 subtasks, each consisting of 100,000 images. For model-parallel training tasks, the controller 101 can divide the model into several parts according to the model structure, with the computation corresponding to each part being a subtask. For example, for a multi-layer neural network, different layers can be assigned to different worker nodes as subtasks. The controller 101 then allocates the decomposed subtasks to different worker nodes based on their resource availability and performance characteristics. For example, subtasks with higher computational demands are assigned to worker nodes with stronger computational capabilities, while subtasks with higher network communication requirements are assigned to worker nodes with better network connectivity. When allocating subtasks, the controller 101 also considers load balancing among worker nodes. Try to make the load of each worker node relatively balanced, and avoid situations where one worker node is overloaded while other worker nodes are underloaded, so as to improve the execution efficiency of the entire training task.

[0115] The controller 101 installs the training program corresponding to the training task on the worker node. This may involve copying the training program's code files to a specified directory on the worker node, and then compiling or interpreting it according to the requirements of the training program. For example, if the training program is written in Python, the controller 101 may need to install a Python interpreter on the worker node and copy the training program's .py files to a suitable directory. After installing the training program, the controller 101 needs to configure it. This includes setting the input and output paths of the training program, the initial parameters of the model, and the training hyperparameters (such as learning rate, batch size, etc.). For example, specifying which directory the training program reads training data from, which directory the trained model parameters will be saved to, and setting the initial learning rate.

[0116] Optionally, when there is only one training task, the deployment method of the controller 101 includes, but is not limited to: dividing the training task into several sub-tasks, and allocating a training program corresponding to a sub-task to each of the n working nodes 103; or, deploying all the training programs corresponding to the training task to each of the n working nodes 103.

[0117] Optionally, when there are at least two training tasks, the deployment method of controller 101 includes, but is not limited to: determining the priority of each training task; and allocating and deploying the resources of n worker nodes 103 according to the priority of the training tasks. The method for determining the priority includes, but is not limited to, based on at least one of the following: the urgency of the task, the importance of the task, or the expected benefit of the task. For example, a task related to the company's core business and requiring training results as soon as possible will have a higher priority; while a task used for exploratory research and with less urgent time requirements will have a relatively lower priority.

[0118] Optionally, more worker node resources can be allocated to higher-priority training tasks. For example, if there are 10 worker nodes, 6 worker nodes can be allocated to execute the corresponding training program for the highest-priority training task, while only 2-3 worker nodes can be allocated for lower-priority training tasks.

[0119] Furthermore, on each worker node, the training program is deployed based on the training task. For high-priority tasks assigned to more worker nodes, a more granular subtask decomposition and allocation method may be adopted to fully utilize the resources of these worker nodes. For low-priority tasks, a relatively simple deployment method may be used, such as deploying the entire training program on fewer worker nodes, to reduce the complexity of resource allocation and management.

[0120] Optionally, the controller 101 deploys the training programs corresponding to all training tasks on each of the n worker nodes 103. In this deployment, each worker node deploys the entire training program. For example, for a small deep learning model training task, each worker node has a complete model structure definition, data loading module, etc. Each worker node trains independently, using its own subset of data or different sampling methods on the entire dataset. During training, each worker node independently calculates the update of its model parameters. Finally, the controller 101 needs to integrate the training results obtained from the n worker nodes 103. For example, for model parameters, an average or weighted average method can be used to merge the model parameters obtained from each worker node to obtain the final training result. This method improves training efficiency, especially when the communication cost between worker nodes is high, reducing the need for frequent communication between worker nodes.

[0121] Optionally, the controller 101 can adopt a hybrid deployment approach, where some training tasks are deployed as a whole on n worker nodes 103, while other training tasks are decomposed into sub-tasks and deployed separately. For example, for a complex training task containing multiple sub-models, some sub-models have relatively simple structures and have little impact on the overall training results, so the training programs corresponding to these sub-models can be deployed as a whole on certain worker nodes. However, for the core sub-model, which has a complex structure and a significant impact on the training results, it can be decomposed into sub-tasks and allocated to other worker nodes.

[0122] Another hybrid deployment approach dynamically adjusts the deployment method based on model characteristics and worker node resources. During training, the controller 101 can change the deployment method of the training program on the n worker nodes 103 according to the model's convergence and the remaining resources of the worker nodes. For example, in the early stages of training, the entire training program is deployed to the n worker nodes 103 as subtasks to accelerate the initial convergence speed of the model. As training progresses, when some worker nodes have surplus resources, and certain parts of the model require more computing resources for optimization, the controller 101 deploys the entire training program corresponding to this part of the model to these worker nodes with surplus resources to improve the final training effect of the model.

[0123] The following section describes how controller 101 deploys the training program corresponding to the training task to m hot standby nodes 104.

[0124] Optionally, the controller 101 may deploy the training program corresponding to the training task to the m hot standby nodes 104 in the following ways: each of the m hot standby nodes 104 deploys the training program corresponding to all training tasks; or, each of the m hot standby nodes 104 deploys the training program corresponding to the sub-training task in the training task; or, based on a microservice architecture, deploy the training program corresponding to the training task to the m hot standby nodes 104; or, based on incremental deployment, deploy the training program corresponding to the training task to the m hot standby nodes 104; or, based on dynamically loaded deployment, deploy the training program corresponding to the training task to the m hot standby nodes 104.

[0125] When each of the m hot standby nodes 104 deploys the training program corresponding to all training tasks, the controller 101 first obtains the complete training program and its related configuration files, dependency libraries, etc., and also obtains a predefined configuration file containing information such as training parameters (learning rate, batch size, etc.) and data paths. Then, the controller 101 distributes the complete training program package to each of the m hot standby nodes 104. After receiving the complete training program package, each of the m hot standby nodes 104 stores it in a preset local storage location. The preset storage location can be pre-defined according to the file system structure of the hot standby nodes. At the same time, the hot standby nodes need to decompress the received training program (if it is in compressed package form) and perform basic environment configuration, such as setting the environment variables required for runtime.

[0126] Optionally, when any worker node fails, a hot standby node needs to take over. Since any hot standby node deploys the entire training program, it may contain some redundant training programs (relative to the subtasks executed by the failed worker node). In this case, the hot standby node needs to determine the parts to retain and execute based on the task identifier of the failed worker node, and delete the redundant training programs to save computing resources. For example, if the failed worker node is responsible for processing a portion of the training dataset (such as data block 1-100), the hot standby node needs to filter out the modules and code segments related to processing this portion of data from the complete training program, delete other irrelevant parts, and then start executing that subtask.

[0127] When each of the m hot standby nodes 104 deploys the training program corresponding to a sub-training task in the training task, before the training task begins, the controller 101 divides the entire training task into sub-tasks according to preset rules. The preset rules can be based on the partitioning of the training dataset, for example, dividing the training dataset into n subsets of equal size (assuming there are n worker nodes 103), with each subset corresponding to a sub-task. Alternatively, they can be based on the partitioning of the model structure, for example, assigning different layers of the deep learning model to different sub-tasks for training. Then, based on the number of hot standby nodes m, these sub-tasks are further distributed to the m hot standby nodes 104. The controller 101 deploys the training program for the corresponding sub-task to each hot standby node according to the sub-task partitioning result. The deployment process includes distributing the code modules, configuration files, and data subsets (if the sub-task is based on dataset partitioning) related to the sub-task to the m hot standby nodes 104. After receiving the training program for the sub-task corresponding to itself, the m hot standby nodes 104 perform local installation and configuration.

[0128] When deploying the training program corresponding to a training task to m hot standby nodes 104 based on a microservice architecture, the training program is divided into multiple microservices according to their functions. For example, for a deep learning training task, functions such as data reading and preprocessing, model building, loss function calculation, and optimizer application can be divided into independent microservices. Each microservice has a clear functional boundary and input / output interface. For each decomposed microservice, an independent running unit is built. The controller 101 deploys the built microservices to the m hot standby nodes 104. The microservices can be allocated to the m hot standby nodes 104 according to their functional importance, resource requirements, and the resource status of the hot standby nodes. For example, the data reading and preprocessing microservice can be deployed to a hot standby node with abundant storage resources, and the model building microservice can be deployed to a hot standby node with abundant computing resources.

[0129] When the training program corresponding to a training task is deployed to m hot standby nodes 104 using incremental deployment, the controller 101 deploys the core part of the training program to the m hot standby nodes 104. The core part includes the basic framework of the training task, such as the basic structural definition of the deep learning model and the basic training algorithm. During training, as the training task progresses or the training strategy is adjusted, new code, data, or configuration information will need to be updated to the m hot standby nodes 104. The controller 101 uses an incremental approach to deploy these updates to the m hot standby nodes 104. For example, if a new data augmentation strategy is added, the controller 101 only needs to deploy the code and configuration files related to the data augmentation strategy to the m hot standby nodes 104, without redeploying the entire training program.

[0130] Optionally, during incremental update deployment, the m hot standby nodes 104 correctly integrate the new update content. The m hot standby nodes 104 rationally adjust their operating status according to the type and scope of the update content to adapt to new training requirements. Incremental deployment reduces deployment workload and resource consumption. Since only the core components and incremental update content are deployed, compared to deploying the entire training program each time, network transmission bandwidth and storage space on the hot standby nodes are saved. Especially when training tasks are long-lasting and updates are frequent, it can effectively improve the system's resource utilization efficiency.

[0131] When the training program corresponding to a training task is deployed to m hot standby nodes 104 based on dynamic loading, the controller 101 first deploys the basic part of the training program to the m hot standby nodes 104. The basic part includes some common code modules, such as basic operating system calls, the use of standard library functions, and some basic configuration information of the training task, such as the name and identifier of the training task. These basic parts provide a running framework for subsequent dynamic loading. On the hot standby nodes, a dynamic loading framework environment is established. The framework environment can be based on the operating system's dynamic link library mechanism or the dynamic loading function of the programming language. The framework environment must be able to recognize and load new code modules, data files, and configuration information. During training, when it is necessary to update or supplement the training program on the m hot standby nodes 104, the controller 101 sends the new content to the hot standby nodes in a dynamically loadable form. For example, if a new training algorithm needs to be added to the training program, the controller 101 sends the code module containing the new algorithm to the hot standby nodes in the form of a dynamic library. After receiving new content, the hot standby node uses the dynamic loading framework environment to load it into the running training program (if the hot standby node is in a ready-to-wait state, it is loaded into the pre-running framework).

[0132] Optionally, the dynamic loading-based deployment method offers high flexibility and real-time performance. New features, algorithms, or data can be added to the m hot standby nodes 104 as needed during training without redeploying the entire training program. This is highly advantageous for large-scale distributed training scenarios requiring continuous adjustment and optimization of training strategies.

[0133] Optionally, the controller 101 is used to send the basic context information of the training subtasks corresponding to each of the n working nodes 103 to the m hot standby nodes 104 when the n working nodes 103 are working normally; the m hot standby nodes 104 are used to receive the basic context information.

[0134] Optionally, the controller 101 is configured to send the basic context information of the training subtasks corresponding to each of the n working nodes 103 to the m hot standby nodes 104 when the check result at each checkpoint is normal. That is, if the check result at each checkpoint is normal, it proves that the n working nodes 103 are in a healthy state. At this time, the check result of the checkpoint is synchronized to the m hot standby nodes 104 once.

[0135] Optionally, the basic context information includes at least one of the following: the task role of any one of the n worker nodes 103 in the training task; the identifier of any one of the n worker nodes 103 in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0136] Hot standby nodes are in a ready-to-train state, but this state is not entirely static. Hot standby nodes perform pre-operations based on changes in the state of worker nodes. For example, when the model parameters of a worker node are updated, the hot standby node receives basic context information (although it doesn't perform actual model training). This allows the first hot standby node out of the m hot standby nodes 104 to obtain the latest updated basic context information from the n worker nodes 103 at the previous normal checkpoint when a worker node needs to be replaced. This enables faster startup of training using the latest model parameters and shortens the resource integration time during the first hot standby node's replacement process. Furthermore, the task status of hot standby nodes includes their own health monitoring. Although hot standby nodes do not perform training tasks, they still need to maintain the health of their hardware and software environments so that they can normally take over the tasks of worker nodes when needed.

[0137] In the above manner, the controller 101 determines the role information and training program corresponding to each of the n working nodes 103; and the role information and training program corresponding to each of the m hot standby nodes 104. The controller 101 needs to configure the corresponding parameters for the computing nodes with different roles.

[0138] For the worker nodes, parameters related to the execution of training tasks need to be configured, such as the subtask scope assigned to each of the n worker nodes 103, the data reading path, and the port number for communication with other worker nodes. For the m hot standby nodes 104, although they do not start training-related processes, some basic parameters also need to be configured, such as the port number for communication with the controller 101 and how to receive failover notifications from worker nodes.

[0139] The n worker nodes 103 and m hot standby nodes 104 share hardware resources, meaning that at the hardware level, hot standby nodes can seamlessly take over the tasks of worker nodes when they fail, as the consistency of the hardware environment ensures the normal operation of the training task, and network resources are also shared. Both worker nodes and hot standby nodes are connected to the same high-speed network environment to ensure rapid data transmission during training. Furthermore, the n worker nodes 103 and m hot standby nodes 104 run the same operating system and related software stack. This allows hot standby nodes to quickly enter a working state without needing to reinstall or configure the software environment when replacing worker nodes.

[0140] Optionally, when deploying the training program corresponding to the training task, the controller 101 considers the overall situation of n worker nodes 103 and m hot standby nodes 104. Although the hot standby nodes do not directly participate in the initial execution of the training task, they are part of the task allocation system of the entire distributed training system. For example, when allocating a subset of training data to worker nodes, the amount of data for each worker node is determined based on the total number of worker nodes and hot standby nodes to ensure load balancing of the entire training task even when hot standby nodes are present.

[0141] During normal training, the task states of the n worker nodes 103 are constantly changing. For example, as training progresses, the model parameters of the worker nodes are continuously updated, and the amount of data they process gradually increases. The task state information of the worker nodes (such as the number of training epochs completed, the current model accuracy, etc.) is monitored by the monitor 102 and fed back to the controller 101. This task state information of the worker nodes is a very important reference for the hot standby node. This is because the hot standby node needs to adjust its own state according to the state information of the worker nodes so that it can take over the task as seamlessly as possible in the event of a failure. For example, if the worker nodes have reached the later stage of training, the hot standby node needs to load the corresponding initial parameters of the model for the later stage to reduce the retraining time.

[0142] Hot standby nodes are in a ready-to-train state, but this state is not entirely static. They perform pre-operations based on changes in the worker node's state. For example, when the worker node's model parameters are updated, the hot standby node may receive these updates (even without actually training the model) so that it can quickly start training with the latest model parameters when the worker node needs to be replaced. Furthermore, the hot standby node's task status includes monitoring its own health. Although hot standby nodes do not perform training tasks, they still need to maintain their own hardware and software environment in good condition so that they can properly take over the worker node's tasks when needed.

[0143] The monitor 102 will be described below.

[0144] Monitor 102 is primarily responsible for monitoring the operational status of the n worker nodes 103 to ensure the smooth progress of the entire training process. In a distributed training system, its monitoring scope covers everything from the hardware's operational status to the software's execution.

[0145] The health status of the n working nodes 103 includes at least one of hardware health status or software health status. Hardware health status includes, but is not limited to, at least one of: healthy temperature of the graphics management card, sufficient memory, a healthy central processing unit, or a healthy disk. Software health status includes, but is not limited to, at least one of: correct operation of the training program, compatibility and synergy between the training program and other software components, or software stability.

[0146] The monitored content includes, but is not limited to, at least one of the following: the hardware status of n working nodes 103 or the software status of n working nodes 103.

[0147] Optionally, for the graphics processing unit (GPU) servers among the n worker nodes 103, the monitor 102 monitors the status of the GPUs, including key indicators such as GPU temperature, power consumption, and memory usage. For example, during deep learning training, GPUs generate heat due to high-intensity computational tasks. Excessive temperature can lead to frequency throttling or even hardware damage, affecting the normal progress of training. The monitor 102 acquires GPU temperature data in real time and issues an alert when the temperature approaches or exceeds a preset safety threshold (e.g., 80°C). Furthermore, the monitor 102 also monitors server hardware indicators such as CPU usage and disk I / O. In large-scale distributed training, although the GPU undertakes the main computational tasks, the CPU also plays a crucial role in data preprocessing and model parameter management. Disk I / O speed affects the efficiency of reading training data and storing model parameters.

[0148] Optionally, monitor 102 monitors the training program running on n worker nodes 103. Monitoring includes, but is not limited to, at least one of the following: whether the training program starts normally or whether a process crash occurs. For example, if the training program unexpectedly terminates due to an error in the code (such as a division-by-zero error, null pointer dereference, etc.), monitor 102 can quickly detect the abnormal process exit and send a corresponding notification to controller 101. Furthermore, monitor 102 also monitors the progress of the training program, including the number of training epochs completed, the current model accuracy, and the loss function value. By monitoring these metrics, the progress of the training task can be understood, and it can be determined whether the training is proceeding as expected. If the model accuracy does not improve for a long time or the loss function value no longer decreases, it may mean that there is a problem with the training algorithm or that the model has fallen into a local optimum, requiring adjustments to the training strategy.

[0149] The monitor 102 monitors the health status of n working nodes 103 in ways including, but not limited to: for temperature monitoring of the graphics processor card, obtaining real-time temperature data from these sensors by interacting with the graphics processor driver or related hardware management interface; or, monitoring various hardware components inside the computer by integrating hardware monitoring tools; or, monitoring the process status of the training program through the operating system's process management mechanism. This application does not limit the monitoring method.

[0150] As an optional implementation, n working nodes 103 are used to report their own resource usage and training status information to the monitor 102.

[0151] The system promptly reports resource usage and training status information to monitor 102, enabling monitor 102 to comprehensively and timely monitor the real-time status of n working nodes 103. In case of faults, it can detect and handle them in a timely manner.

[0152] Resource usage includes hardware resource usage and software resource usage. Hardware resource usage includes, but is not limited to, at least one of the following: resource utilization rate indicators, resource saturation indicators, or resource error indicators.

[0153] Among them, resource utilization-related indicators include, but are not limited to, at least one of the following: graphics processing unit (GPU) utilization, central processing unit (CPU) utilization, memory usage, disk usage, disk I / O performance, such as the number of input / output operations per second (I / OPS), network card bandwidth utilization, or computer bus interface standard (PCIE) bandwidth utilization.

[0154] Resource saturation related indicators include, but are not limited to, at least one of the following: video memory saturation, memory saturation, or disk saturation.

[0155] Resource error related indicators include, but are not limited to, at least one of the following: network card malfunction, disk write (including remote storage) error, or system load (current number of queued processes).

[0156] Training state information refers to the various data and metrics used to describe the execution status of training tasks during distributed training. This information is crucial for monitoring the training process, detecting anomalies, optimizing performance, and recovering from failures.

[0157] Training state information includes, but is not limited to: the iteration time of each training task, i.e., the time spent on each iteration. For example, in the training of an image recognition model, if the iteration time increases from 10 seconds to 20 seconds, it means that the model complexity has increased or the computing resources are insufficient; or, the loss function value of the training task, i.e., the difference between the model's prediction and the actual result. For example, in a regression task, if the loss function value continues to decrease, it indicates that the model is learning the pattern in the data. If the loss function value suddenly increases, it may indicate that the model is overfitting or that there is a problem with the data; or, log information generated during training, i.e., various information recorded during training, including program running status, error messages, warning messages, etc.; or, detailed records of function call order, data flow, etc., during task execution; or, the update status of model parameters; or, the data processing status; or, the convergence status of the training task. This application does not limit the type of training state information.

[0158] The process of n working nodes 103 reporting their resource usage and training status information to the monitor 102 is described below.

[0159] Reporting methods include, but are not limited to: reporting based on a message passing interface (MPI); or reporting based on a network socket. This application does not restrict the reporting method.

[0160] When n worker nodes 103 report resource usage and training status information to monitor 102 using the MPI communication mechanism, each worker node can be considered an MPI process. They can send and receive messages using functions such as MPI_Send and MPI_Recv. Worker nodes need to encapsulate the information to be reported. For hardware resource usage, information such as GPU core utilization, video memory usage, memory usage, and network bandwidth usage can be combined into a structure according to a certain format. Then, this structure is sent as a message to monitor 102. Training status information can also be encapsulated using a similar structure, such as a structure containing information like the number of completed training epochs, model accuracy, and loss function value.

[0161] n worker nodes 103 can create a Socket connection to the server port where monitor 102 is located, and then send resource usage and training status information through this connection. When reporting information via the Socket, a suitable data format is determined.

[0162] As an optional implementation, the monitor 102 is used to determine the first working node that needs to be switched due to a fault, based on the resource usage and training status information of the n working nodes 103.

[0163] In a distributed training system, n worker nodes 103 are the main entities that actually execute the training task. Each worker node undertakes a portion of the training sub-tasks, and these sub-tasks together constitute the entire large-scale training task. The first worker node, as one of these worker nodes, normally collaborates with the other worker nodes. For example, in the distributed training of a neural network, different worker nodes are responsible for processing different batches of data or computing different parts of the neural network. The first worker node also participates in this collaborative data processing and computation process, exchanging intermediate results (such as gradient information) with other nodes to gradually optimize the entire neural network model.

[0164] Due to various factors such as hardware and software, worker nodes may fail during training. The first worker node is the computing node detected as having failed among the n worker nodes 103. Once the first worker node fails, it will be unable to continue executing the training subtasks it is responsible for, which will affect the entire training task. This is because the training subtasks of each worker node are interconnected, and the failure of one node may lead to inaccurate overall training results or interruption of the training process.

[0165] Monitor 102 is responsible for monitoring the health status of n worker nodes 103, with the first worker node being one of the monitored objects. Monitor 102 continuously collects resource usage and training status information reported by the first worker node. By analyzing this information, monitor 102 can determine whether the first worker node is operating normally. For example, if the GPU utilization of the first worker node suddenly drops to 0, while other nodes are normal, this may be a signal that the GPU of the first worker node has malfunctioned; or if the training task loss function value of the first worker node fluctuates abnormally (such as suddenly increasing significantly), it may also indicate that there is a problem with the first worker node.

[0166] As an optional implementation, the monitor 102 is used to mark the first working node as faulty.

[0167] The monitor 102 marks the selected first working node as faulty to clearly indicate to other components in the distributed training system (such as the controller 101, other working nodes, etc.) which node has a problem. Once the monitor 102 marks this node as faulty, the controller 101 can take appropriate measures based on this fault mark, such as transferring the node's tasks to other nodes, thereby ensuring the continued stable operation of the entire training system.

[0168] As an optional implementation, the monitor 102 is used to select a first hot standby node from m hot standby nodes 104 in the event that a first working node fails among the n working nodes 103.

[0169] When the monitor 102 detects a failure in the first working node, it needs to promptly select the first hot standby node from the m hot standby nodes 104 to replace the first working node in the subsequent training tasks, so as to ensure the smoothness of the training process.

[0170] The following section further elaborates on how the monitor 102 selects the first hot standby node from the m hot standby nodes 104.

[0171] As an optional implementation, the monitor 102 is used to select a first hot standby node from m hot standby nodes 104 based on selection factors when a first working node fails among n working nodes 103. The selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes 104; the communication bandwidth between the first working node and the m hot standby nodes 104; and the communication speed between the third working node and the m hot standby nodes 104. The third working node is a working node that has a collaborative requirement with the first working node.

[0172] By determining the first hot standby node based on multiple selection factors, a more comprehensive and smooth replacement of the first working node with the first hot standby node is achieved, thereby greatly reducing the replacement time of the faulty node.

[0173] The communication speed between the first working node and the m hot standby nodes 104 refers to the speed at which data is transmitted between the first working node and the m hot standby nodes 104, which can be measured by the number of bytes transmitted per second or the number of data packets transmitted per second. In a distributed training system, data transmission between nodes is continuous, such as updating model parameters and sharing intermediate calculation results. The communication speed reflects the actual efficiency of transmitting this data from the first working node to the m hot standby nodes 104.

[0174] The communication bandwidth between the first working node and the m hot standby nodes 104 refers to the data transmission capacity between the first working node and the m hot standby nodes 104 per unit time. The larger the communication bandwidth, the more data can be transmitted simultaneously.

[0175] The third working node is a node that has a collaborative requirement with the first working node. In distributed training, there are complex collaborative relationships between the working nodes. The first working node may be responsible for a certain part of the computational task, while the third working node may generate data related to the first working node or depend on the computational results of the first working node. For example, in the distributed training of an image recognition model, the first working node may be responsible for extracting features from a certain region of the image, while the third working node may be responsible for integrating these extracted features or performing further classification calculations based on the results of the first working node. Because the third working node has a collaborative requirement with the first working node, the communication relationship between the third working node and the hot standby node becomes important when the first working node fails and needs to switch to a hot standby node. If the communication speed between the third working node and the hot standby node is slow or the communication bandwidth is small, it will affect the efficiency of the entire collaborative training task.

[0176] The communication speed between the third working node and the m hot standby nodes 104 refers to the speed at which data is transmitted between the third working node and the m hot standby nodes 104. The data transmission speed between the third working node and the hot standby nodes in the distributed training system also affects the overall system performance.

[0177] When the first working node fails, monitor 102 evaluates the communication speed between the first working node and the m hot standby nodes 104. If the communication speed between a hot standby node and the first working node is faster, it means that during failover, data related to the first working node (such as intermediate results of training subtasks, model parameters, etc.) can be quickly transmitted to that hot standby node. For example, in a large-scale speech recognition model training, some acoustic model parameters of the first working node need to be quickly transmitted to the hot standby nodes for training to continue. If the communication speed between hot standby node A and the first working node is 80 bits per second (Mbps), while the communication speed of hot standby node B is 20 Mbps, all other things being equal, hot standby node A is more likely to be selected as the first hot standby node.

[0178] Monitor 102 also considers the communication speed between the third working node and the m hot standby nodes 104. Because the third working node has a close relationship with the first working node in collaborative training, a faster communication speed between the third working node and a hot standby node gives that hot standby node an advantage in selection. For example, in the distributed training of a recommender system, the first working node is responsible for processing part of the user behavior data, and the third working node is responsible for matching product features based on the results of the first working node. When the first working node fails and switches to a hot standby node, the fast communication between the third working node and the hot standby node ensures the continuity of the entire recommender system training task.

[0179] In addition, monitor 102 checks the communication bandwidth between the first working node and the m hot standby nodes 104. Larger communication bandwidth means faster transmission of large amounts of data during failover. For example, in the distributed training of a video analytics model, the first working node generates a large amount of video frame feature data, which needs to be transmitted to the hot standby nodes. If the communication bandwidth between hot standby node C and the first working node is 1 gigabit per second (Gbps), while the communication bandwidth of hot standby node D is 500 Mbps, hot standby node C is more likely to be selected when a large amount of data needs to be transmitted.

[0180] Optionally, the distributed training system includes a shared storage location, which is a storage area established within the distributed training system to enable data exchange and sharing among the n worker nodes 103 during the training process. The shared storage location is a public data warehouse where each worker node can send its training-related data (such as the context information, global training parameters, and some intermediate results of the model) to this location, and can also obtain data from other worker nodes or shared global information from this location.

[0181] Optionally, the first hot standby node can be determined based on network distance. Network distance can be measured in several ways, such as by the number of network hops from the hot standby node to the shared storage location. Network hops refer to the number of intermediate devices (such as routers, switches, etc.) that data passes through when transmitting data from one node to another in the network. Generally, the fewer the network hops, the lower the data transmission latency, and the faster the hot standby node can obtain context information. Another way to measure network distance is based on network latency. This is done by sending probe packets from the hot standby node to the shared storage location and measuring the round-trip time.

[0182] Optionally, the first hot standby node is determined based on physical distance and network topology. The hot standby node and the shared storage location may be located in different physical racks or server clusters. If the physical distance is short, such as within the same rack, the physical link for data transmission is shorter, and the signal transmission latency is relatively low. Meanwhile, network topology also affects the data transmission path. For example, in a tree-like network topology, a hot standby node located near the root node and close to the shared storage location may have a shorter data transmission path, thus enabling it to obtain context information faster.

[0183] Optionally, when selecting the first hot standby node from the m hot standby nodes 104 based on selection factors, the monitor 102 can comprehensively consider communication speed and communication bandwidth. Sometimes, although one hot standby node has a faster communication speed, its communication bandwidth is smaller, which may cause a bottleneck when transmitting large amounts of data; while another hot standby node has a larger communication bandwidth but a relatively slower communication speed, which is inefficient when transmitting small amounts of data. The monitor 102 needs to weigh the importance of communication speed and communication bandwidth according to the data transmission requirements between the first working node and the third working node and the hot standby node in order to select the most suitable first hot standby node.

[0184] After determining the first hot standby node among the m hot standby nodes 104, the controller 101 will be further described below.

[0185] When monitor 102 detects a failure in the first working node among n working nodes 103, controller 101 activates the fault response mechanism. It coordinates the switching of the training subtask corresponding to the first working node to the first hot standby node among m hot standby nodes 104. This process requires effective communication between controller 101, monitor 102, working nodes, and hot standby nodes to ensure a smooth switchover. When the first working node fails, controller 101 manages the working nodes, such as isolating the failed node to prevent adverse effects on other working nodes and the entire training task. Simultaneously, controller 101 also ensures that other working nodes can work normally with the first hot standby node that is about to be switched over.

[0186] The controller 101 switches the training subtask corresponding to the first working node to the first hot standby node by sending a start command to the first hot standby node and transmitting necessary information related to the training subtask, such as the current training parameters and model weights.

[0187] As an optional implementation, the controller 101 is used to set the working role of the first hot standby node as the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0188] The hot standby node shares the same resources and hardware / software environment as the working node, but does not initiate training-related processes. The hot standby node is in a ready-to-wait state for the training program. This means that the hot standby node has the necessary conditions to execute the training task, such as having installed the required software environment and configured the necessary hardware resources, but it does not participate in actual training computations under normal circumstances. When the first working node fails, the role of the first hot standby node will change. The first hot standby node will transition from a ready-to-wait state to a second working node that actually executes the training task.

[0189] The second worker node is relative to the first worker node. When the first worker node fails, the first hot standby node will be set up as the second worker node to take over the tasks of the first worker node. This name is used to clearly describe the role transformation during fault handling, that is, from a node that was originally a hot standby node to a worker node that undertakes training subtasks.

[0190] When controller 101 sets the working role of the first hot standby node to the second working node, it first needs to perform configuration adjustments. This includes modifying the working role identifier in the system. For example, in the node management database of the distributed training system, the role identifier of the first hot standby node is updated from "hot standby node" to "second working node". In addition, controller 101 also needs to adjust the network configuration to ensure that the second working node (formerly the first hot standby node) can communicate normally with other working nodes.

[0191] As an optional implementation, the controller 101 is used to send the context information of the training subtask corresponding to the first working node to the second working node. The second working node is used to re-execute or continue executing the training subtask corresponding to the first working node through the training program based on the context information.

[0192] When the first working node fails, the controller 101 sends the context information of its training subtask to the second working node. The second working node can accurately re-execute or continue the training subtask corresponding to the first working node, maintaining the continuity of the training task and avoiding confusion or interruption of the training process due to node failure. Since the second working node can continue the task based on the context information of the first working node, it does not need to start the entire training subtask from the beginning, significantly improving the overall training efficiency.

[0193] Controller 101 needs to notify the second worker node (formerly the first hot standby node) that its role has changed. This notification includes some necessary instructions and information. For example, controller 101 will notify the second worker node to start preparing to execute the training subtask corresponding to the first worker node, and inform it of the context information about the first worker node so that the second worker node can start working from the appropriate state.

[0194] Context information is a set of crucial information related to a training subtask in a distributed training system. This information is essential for switching the training subtask corresponding to the first worker node to the second worker node and ensuring its correct execution in the event of a failure. Context information provides the second worker node with a comprehensive description of the training task from the first worker node, enabling the second worker node to accurately re-execute or continue the task.

[0195] As an optional implementation, the context information includes at least one of the following: the task role of the first worker node in the training task; the identifier of the first worker node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0196] The task role of the first working node refers to its function or task type in the entire training task. For example, in the distributed training of a deep learning model, there are different task roles such as working nodes responsible for data preprocessing, working nodes responsible for computing some layers of the model, and working nodes responsible for model evaluation. The task role of the first working node determines its specific operation in the training process; for example, it might be a node responsible for normalizing the input image data. The task role of the first working node also defines its relationship with other nodes. Working nodes with different task roles may have data dependencies, communication relationships, etc.

[0197] When the first worker node fails and its training subtasks need to be switched to the second worker node, the task role information of the first worker node guides the second worker node. The second worker node adjusts its operations based on the task role of the first worker node to ensure that it can correctly take over the work of the first worker node. For example, if the first worker node is responsible for data distribution, then the second worker node needs to understand the distribution rules, the source and destination of the data, and other information in order to continue the data distribution task.

[0198] The identifier of the first worker node in a training task is a unique identifier for that node within that task. For example, in a cluster with 100 worker nodes, only 20 worker nodes might be used for a specific training task. Each of these 20 worker nodes has an identifier, which serves as the first worker node's identity within that training task. This identifier distinguishes the first worker node from other participating worker nodes.

[0199] When switching the training subtask from the first worker node to the second worker node, the identifier of the first worker node helps locate and match relevant resources and operations. In fault handling scenarios, the identifier of the first worker node can accurately identify the faulty node. When it is necessary to transfer the training subtask from the first worker node to another node (such as the second worker node), the identifier ensures the accurate transmission of context information. For example, the master node of the training task can find the context information related to the identifier of the first worker node from the storage system and send it to the second worker node so that the second worker node can correctly continue or re-execute the task of the first worker node.

[0200] The IP address of the master node for a training task is its unique identifier within the network, used to locate the master node in the network environment. In a distributed training system, the master node plays a crucial role in coordinating and managing the entire training task. Other worker nodes and hot standby nodes need to communicate with the master node through its IP address. When the first worker node fails and its training subtask is switched to the second worker node, the second worker node needs to re-establish communication with the master node. The master node's IP address provides a clear communication target for the second worker node.

[0201] The port address of the master node for a training task is an address that further identifies the service or application based on the master node's IP address. Multiple services or applications may run on a single master node, and the port address is used to distinguish these different services. For example, a data distribution service and a model parameter synchronization service may run simultaneously on the master node, and different services will be assigned different port addresses.

[0202] When the first worker node fails and the task is switched to the second worker node, the second worker node needs to connect to the target service of the master node to continue executing the training subtasks. The master node's port address provides the second worker node with a way to connect to the target service. For example, if the first worker node was interacting with the master node's data distribution service before the failure, the second worker node needs to connect to that service using the same port address to obtain the correct data for training.

[0203] The address of the training dataset for a training task refers to the specific location of the training data in the storage system. This address can be a path in the local file system or a target storage location identifier in a distributed storage system.

[0204] When the first worker node fails and its training subtask is switched to the second worker node, the address of the training dataset ensures that the second worker node can continue to obtain the necessary training data. For example, if the first worker node had already processed a portion of the training dataset before the failure, the second worker node needs to find the remaining unprocessed data based on the address of the training dataset to continue training.

[0205] The storage address of the last checkpoint in a training task is the recorded position of the training state at a certain moment during training. Checkpoints are set up to allow training to resume from the last recorded state if training is interrupted. This storage address can be a folder on the local disk or a target location in a distributed storage system. For example, in a deep learning training task, a checkpoint is created after a certain number of training epochs, and the storage address of the last checkpoint records the storage location of information such as model parameters and optimizer state at that time.

[0206] For the second worker node, the storage address of the last checkpoint serves as the starting point for its state recovery. Based on this address, the second worker node retrieves the training state of the first worker node before its failure, including the amount of data processed and the model's current parameters. This allows the second worker node to continue executing training subtasks on the same basis, maintaining training continuity and efficiency. For example, in a distributed training task for speech recognition, if the first worker node fails after 50 training iterations, the second worker node can retrieve the model state after those 50 iterations from the storage address of the last checkpoint and continue training for subsequent iterations.

[0207] The model configuration of the AI ​​model trained in the training task includes a description of the model structure, such as the number of layers, the number of neurons in each layer, and the type of activation function. For example, in a convolutional neural network model, the model configuration describes the number of convolutional layers, the size of the convolutional kernels, and the location and type of pooling layers.

[0208] When the first worker node fails and its training subtask is switched to the second worker node, the model configuration serves as the basis for the second worker node to rebuild the model and continue training. The second worker node must build the same model according to the model structure description in the model configuration and continue training according to the training parameter settings in the model configuration.

[0209] Re-executing the training subtask means that after receiving the context information of the training subtask from the first worker node, the second worker node starts the training operation from the initial state of the training subtask. That is, it ignores the results of some operations that the first worker node may have already performed.

[0210] The circumstances under which the training subtask is re-executed include, but are not limited to, at least one of the following: the data related to the training subtask of the first working node has an integrity problem; or the type of failure that occurs to the first working node makes it difficult to accurately restore the state of its training subtask.

[0211] Continuing to execute the training subtask means that after the first working node fails, the second working node continues the training operation from the state at the time of the failure, based on the partial training results already completed by the first working node.

[0212] The conditions under which the training subtask can continue to be executed include, but are not limited to: the first working node fully saving the state of its training subtask before the failure occurs; or, for certain recoverable failure types, such as temporary errors in the software or brief network interruptions, as long as the state of the first working node's training subtask is not severely corrupted.

[0213] As an optional implementation, the second working node further includes: a training process control unit; the training process control unit is used to start the training process corresponding to the training program; and to re-execute or continue to execute the training sub-task corresponding to the first working node based on context information through the training process.

[0214] The system can quickly initiate the training process corresponding to the training program, improving effective training time. The training process control unit can re-execute or continue the training subtask corresponding to the first working node based on context information. This functionality allows the training task to quickly recover after a failed node is replaced, avoiding the need to start training from scratch. This continuity is particularly important for distributed training, ensuring the accuracy and effectiveness of the training.

[0215] The training process control unit is a functional unit used to manage and operate the training process. It is responsible for scheduling, starting, pausing, and resuming processes related to the training task. The training process control unit contains multiple sub-modules that work together to achieve effective control of the training process. These include, but are not limited to: a process startup module, responsible for initializing the training process; a process scheduling module, which rationally arranges the execution order of training processes based on system resource conditions (such as CPU and GPU utilization) and the priority of training tasks; and a process monitoring module, which monitors the running status of the training process in real time, such as whether memory leaks or other problems occur.

[0216] The training process control unit has a close interaction with the training program. The training program is the code that actually executes the training task, while the training process control unit manages the execution of the training program. The training process control unit is responsible for loading the training program into memory and starting the corresponding training process.

[0217] When the first worker node fails, all worker nodes suspend operation to ensure the consistency and accuracy of the entire distributed training system. n worker nodes 103 (especially the second worker node) begin waiting for a restart or continuation signal. Simultaneously, the controller 101 prepares the context information for the training subtask corresponding to the first worker node. When the second worker node receives the restart or continuation signal, its internal training process control unit first starts the training process corresponding to the training program. Once the training process starts, the training process control unit re-executes or continues executing the training subtask corresponding to the first worker node based on the received context information.

[0218] When the second worker node re-executes or continues the training subtask corresponding to the first worker node based on context information during the training process, other worker nodes remain in a paused and waiting state. Once the second worker node completes the initialization or recovery operation for the task of the first worker node and reaches a state where it can resume collaborative work with other worker nodes, it sends a ready signal to the controller 101.

[0219] After receiving the signal from the second worker node, the controller 101 sends a signal to all worker nodes to restart collaborative training. At this time, all worker nodes (including the second worker node) will continue to perform large-scale distributed training tasks in the previous collaborative manner, thereby achieving efficient recovery in the event of a failure, reducing the replacement time of the failed node, and improving the effective training time.

[0220] As an optional implementation, the controller 101 is also configured to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training sub-task. The restart training task message is used to trigger each working node to re-coordinate and execute each training sub-task of the training task.

[0221] The "Restart Training Task" message is an instruction sent by the controller 101. Its purpose is to coordinate the n worker nodes 103 in the distributed training system, enabling them to re-execute the various training sub-tasks of the training task collaboratively. It prompts each worker node to stop its current waiting or temporary state and restart the execution of the training task according to predetermined rules, thereby ensuring the state consistency and training continuity of the entire distributed training system.

[0222] The training task restart message contains several information elements. First, it includes the training task's identifier, specifying which training task needs to be restarted. This can be a unique task identifier or task name. Second, it contains global configuration information related to the training task, such as the overall goals of the training task (e.g., the target accuracy, recall, etc.) and a basic description of the dataset used (e.g., dataset size, data type, etc.). Additionally, it may contain timestamp information to record the time the training task restart message was sent, enabling the n worker nodes 103 to determine the message's timeliness and coordinate their own operation order.

[0223] When the second worker node begins executing the training subtask of the first worker node, the state of the entire distributed training system changes. To enable other worker nodes to adapt to this change and continue to work effectively together, the restart training task message acts as a synchronization signal, allowing all worker nodes to readjust their state and operational processes.

[0224] The following section explains how to send restart training task messages to each worker node corresponding to the training task.

[0225] When sending a restart training task message to each worker node via network communication, a suitable network communication protocol must first be selected; then, the specific communication method must be determined. One method is for the controller 101 to establish a one-to-one communication connection with each worker node and then send the restart training task message to each one sequentially. Another method is to use broadcast or multicast communication. Broadcast sends the message to all nodes in the network, while multicast sends the message to a predefined group of nodes.

[0226] Before sending the restart training task message, controller 101 needs to check the current status of n worker nodes 103. Worker nodes may be in different states, such as processing data, in an idle waiting state, or undergoing fault recovery. Depending on the node's state, controller 101 can adopt different sending strategies. To ensure that the restart training task message is sent accurately and orderly to the n worker nodes 103, controller 101 can use a message queue management approach.

[0227] Once a worker node receives the message, it sends an acknowledgment message to the controller 101. The acknowledgment message contains the worker node's identification information so that the controller 101 can identify which worker node sent the acknowledgment. Upon receiving the acknowledgment message, the controller 101 knows that the worker node has received the message to restart the training task and can continue with subsequent operations.

[0228] The hot standby node mechanism in this application significantly reduces the replacement time for failed nodes. Since hot standby nodes share the same resources and hardware / software environment as worker nodes and are in a ready-to-wait state for the training program, once the monitor detects a failure in a worker node (such as the first worker node), the controller can quickly switch the training subtask corresponding to the first worker node to the first hot standby node among the m hot standby nodes for execution. This avoids other nodes waiting for extended periods, effectively improving fault handling speed and reducing the impact of failures on training tasks. Furthermore, hot standby nodes are in a ready-to-wait state in the system; although they do not start the training process, they can take over the tasks of failed nodes at any time, making system resources more effectively utilized. Because hot standby nodes can quickly take over the tasks of failed nodes, worker nodes can focus more on training tasks during normal operation without reserving excessive resources or processing capacity for potential failures, thus providing a more stable environment for distributed training.

[0229] This application provides a distributed training method, applied to the above-mentioned... Figure 1The distributed training system in the example includes a controller, a monitor, n worker nodes, and m hot standby nodes, where n and m are positive integers. For example... Figure 2 As shown, the distributed training method provided in this application embodiment may include the following steps 201 to 204.

[0230] In step 201, the controller deploys the training program corresponding to the training task to n working nodes and m hot standby nodes.

[0231] In step 202, the n worker nodes collaboratively execute the training task.

[0232] In step 203, the monitor monitors the health status of n working nodes.

[0233] In step 204, if the first working node among the n working nodes fails, the controller switches the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution.

[0234] In this application, the hot standby node has the same resources and environment as the working node and is in a ready-to-wait state. Once the working node (first working node) fails, the controller can quickly switch its training subtask to the hot standby node (first hot standby node) for execution, which greatly reduces the replacement time of the failed node. Other nodes do not need to wait for a long time, so that the training task can be quickly restored and the speed of fault recovery can be effectively improved.

[0235] See Figure 3 The diagram shows an overall flowchart of a distributed training method. This method is applied to a distributed training system, which includes a controller, a monitor, n worker nodes, and m hot standby nodes, where n and m are positive integers.

[0236] Step 1: The controller deploys the training program corresponding to the training task to multiple computing nodes; the multiple computing nodes are divided into n working nodes and m hot standby nodes.

[0237] Step 2: n worker nodes collaboratively execute the training task.

[0238] Step 3: Each of the n worker nodes reports its resource usage and training status information to the monitor.

[0239] Step 4: Based on the resource usage and training status information of the n working nodes, the monitor determines the first working node that needs to be switched due to a fault.

[0240] Step 5: If the first working node among the n working nodes fails, the monitor selects the first hot standby node from the m hot standby nodes based on selection factors.

[0241] Optionally, the selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes; the communication bandwidth between the first working node and the m hot standby nodes; the communication speed between the third working node and the m hot standby nodes; and the communication speed between the third working node and the m hot standby nodes; wherein the third working node is a working node that has a collaborative requirement with the first working node.

[0242] Step 6: If the first working node among the n working nodes fails, the controller sets the working role of the first hot standby node to the second working node and switches the training subtask corresponding to the first working node to the second working node for execution.

[0243] Step 7: The controller sends the context information of the training subtask corresponding to the first working node to the second working node.

[0244] Optionally, the context information includes at least one of the following: the task role of the first worker node in the training task; the identifier of the first worker node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0245] Step 8: The controller sends a restart training task message to each working node corresponding to the training task.

[0246] The restart training task message is used to trigger all worker nodes to re-coordinate and execute the various training subtasks of the training task.

[0247] Step 9: The training process control unit starts the training process corresponding to the training program; based on the context information, the training process re-executes or continues to execute the training sub-task corresponding to the first working node.

[0248] The second working node also includes: the training process control unit.

[0249] Step 10: The controller adjusts the number of m hot standby nodes based on the number of fault exchanges in the most recent time period.

[0250] Optionally, the number of fault swaps is the number of swaps between the working node and the hot standby node caused by a fault.

[0251] See Figure 4 This application provides a controller for a distributed training system. The distributed training system includes a controller, a monitor, n worker nodes, and m hot standby nodes, where n and m are positive integers. The controller includes:

[0252] Deployment module 401 is used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes;

[0253] The switching module 402 is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when the first working node fails among the n working nodes.

[0254] In one possible implementation, the switching module 402 is used to set the working role of the first hot standby node as the second working node when the first working node fails among the n working nodes, and to switch the training subtask corresponding to the first working node to the second working node for execution.

[0255] In one possible implementation, the deployment module 401 is further configured to send the context information of the training subtask corresponding to the first working node to the second working node, and the second working node is configured to re-execute or continue to execute the training subtask corresponding to the first working node based on the context information through the training program.

[0256] In one possible implementation, the context information includes at least one of the following: the task role of the first worker node in the training task; the identifier of the first worker node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0257] In one possible implementation, the switching module 402 is further configured to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training sub-task. The restart training task message is used to trigger each working node to re-coordinate and execute each training sub-task of the training task.

[0258] In one possible implementation, the deployment module 401 is used to create a training task, deploy the training program corresponding to the training task to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, with the m hot standby nodes in a ready-to-wait state; set the start command and start parameters for the training task; and coordinate the start of training on the n working nodes based on the start command and start parameters.

[0259] In one possible implementation, the deployment module 401 is further configured to adjust the number of m hot standby nodes based on the number of fault exchanges in the most recent time period; wherein the number of fault exchanges is the number of exchanges between working nodes and hot standby nodes caused by faults.

[0260] In one possible implementation, the n working nodes and m hot standby nodes have the same software environment, hardware environment, and training program configuration file.

[0261] It should be understood that the controller described above is only illustrated by the division of the functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the controller and method embodiments provided above belong to the same concept; their specific implementation process and beneficial effects are detailed in the method embodiments, and will not be repeated here.

[0262] See Figure 5 This application provides a distributed training system, which includes a controller 501 and a monitor 502.

[0263] Controller 501 is used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes, where n and m are positive integers;

[0264] Monitor 502 is used to monitor the health status of n working nodes;

[0265] The controller 501 is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when the first working node fails among the n working nodes.

[0266] In one possible implementation, monitor 502 is used to select a first hot standby node from m hot standby nodes if a first working node fails among n working nodes.

[0267] The controller 501 is used to set the working role of the first hot standby node to the second working node and switch the training subtasks corresponding to the first working node to the second working node for execution.

[0268] In one possible implementation, controller 501 is used to send the context information of the training subtask corresponding to the first working node to the second working node.

[0269] In one possible implementation, the context information includes at least one of the following: the task role of the first worker node in the training task; the identifier of the first worker node in the training task; the Internet Protocol IP address of the master node of the training task; the port address of the master node of the training task; the address of the training dataset of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0270] In one possible implementation, the controller 501 is further configured to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training sub-task. The restart training task message is used to trigger each working node to re-coordinate the execution of each training sub-task of the training task.

[0271] In one possible implementation, monitor 502 is used to select a first hot standby node from m hot standby nodes based on selection factors when a first working node fails among n working nodes; the selection factors include at least one of the following: communication speed between the first working node and the m hot standby nodes; communication bandwidth between the first working node and the m hot standby nodes; communication speed between the third working node and the m hot standby nodes; wherein the third working node is a working node that has a collaborative requirement with the first working node.

[0272] In one possible implementation, monitor 502 is used to determine the first working node that needs to be switched due to a failure, based on the resource usage and training status information of the n working nodes.

[0273] In one possible implementation, controller 501 is used to create training tasks, deploy the training program corresponding to the training tasks to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, with the m hot standby nodes in a ready-to-wait state; set the start command and start parameters for the training tasks; and coordinate the start of training on the n working nodes based on the start command and start parameters.

[0274] In one possible implementation, controller 501 is used to adjust the number of m hot standby nodes based on the number of fault exchanges in the most recent time period; wherein the number of fault exchanges is the number of exchanges between working nodes and hot standby nodes caused by faults.

[0275] In one possible implementation, the n working nodes and m hot standby nodes have the same software environment, hardware environment, and training program configuration file.

[0276] It should be understood that the distributed training system described above is only illustrated by the division of the functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the distributed training system and method embodiments provided above belong to the same concept; their specific implementation process and beneficial effects are detailed in the method embodiments, and will not be repeated here.

[0277] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory storing at least one computer program. The at least one computer program is loaded and executed by one or more processors to enable the computer device to implement any of the distributed training methods described above. The computer device can be a server or a terminal device. The structures of the server and the terminal device will be described below.

[0278] Figure 6 This is a schematic diagram of a server structure provided in an embodiment of this application. The server can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 601 and one or more memories 602. The one or more memories 602 store at least one computer program, which is loaded and executed by the one or more processors 601 to enable the server to implement the distributed training methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0279] Figure 7 This is a schematic diagram of the structure of a computing node virtualization system provided in an embodiment of this application; the computing node virtualization system can be used in various networks, such as data center networks, carrier networks, or local area networks. Figure 7 As shown, the computing node virtualization system includes: a computing node infrastructure layer 701, n computing nodes 702, and a distributed training system 703.

[0280] The compute node infrastructure layer 701 includes a hardware resource layer 704, a virtualization layer 705, and a virtual resource layer 706. The compute node infrastructure layer 701 utilizes hardware or software resources to deploy a virtualized environment. Specifically, the hardware resource layer 704 and the virtual resource layer 706 provide virtualization resources, such as virtual machines and other forms of virtual containers, for compute node virtualization. The hardware resource layer 704 includes compute hardware, storage hardware, and network hardware. The compute hardware provides processing and computing resources; the storage hardware can be storage capacity provided within the network or storage capacity residing within the storage hardware itself; and the network hardware can be switches, routers, or any network device configured with switching capabilities.

[0281] The virtualization layer 705 abstracts hardware resources from the physical layer and decouples compute nodes, providing virtualized resources to the compute nodes. The virtual resource layer 706 includes virtual computing, virtual storage, and virtual networking. Virtual computing and virtual storage can be provided to compute nodes in the form of virtual machines or other virtual containers. For example, multiple compute nodes can be deployed on a single virtual machine. The virtualization layer 705 abstracts network hardware to form a virtual network, which can include virtual switches to provide connectivity between multiple virtual machines.

[0282] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described distributed training methods.

[0283] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0284] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions loaded and executed by a processor to enable the computer to implement any of the above-described distributed training methods.

[0285] Optionally, all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the resource usage and training status information of the working nodes involved in this application were obtained with full authorization.

[0286] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0287] The above description is merely an exemplary embodiment of this application and does not limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A distributed training system for an artificial intelligence model, characterized in that, The system includes: a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers; The controller is used to create training tasks and deploy the training program corresponding to the training tasks to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, with the m hot standby nodes in a ready-to-wait state; the controller is also used to allocate the same software environment, hardware environment, and running configuration file of the training program to the n working nodes and the m hot standby nodes. The n working nodes are used to collaboratively execute the training task; The controller is further configured to, when the check result at each checkpoint is normal, send the basic context information of the training subtask corresponding to each of the n working nodes to the m hot standby nodes; the basic context information includes at least one of the following: the task role of the working node in the training task, the identifier of the working node in the training task, the IP address of the master node of the training task, and the port address of the master node of the training task. The monitor is used to monitor the health status of the n working nodes; The controller is configured to, in the event that a first working node fails among the n working nodes, switch the training subtask corresponding to the first working node to be executed on the first hot standby node among the m hot standby nodes; the first hot standby node further includes: a training process control unit; the training process control unit is configured to start the training process corresponding to the training program; and execute the training subtask corresponding to the first working node through the training process.

2. The system according to claim 1, characterized in that, The monitor is used to select the first hot standby node from the m hot standby nodes when a first working node fails among the n working nodes. The controller is configured to set the working role of the first hot standby node as the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

3. The system according to claim 2, characterized in that, The controller is used to send the context information of the training subtask corresponding to the first working node to the second working node; The second working node is used to re-execute or continue executing the training subtask corresponding to the first working node through the training program based on the context information.

4. The system according to claim 3, characterized in that, The context information includes at least one of the following: The task role of the first working node in the training task; The identifier of the first working node in the training task; The Internet Protocol IP address of the master node of the training task; The port address of the master node for the training task; The address of the training dataset for the training task; The storage address of the last checkpoint of the training task; The model configuration of the AI ​​model trained by the training task.

5. The system according to claim 3, characterized in that, The training process control unit is used to re-execute or continue executing the training sub-task corresponding to the first working node based on the context information through the training process.

6. The system according to any one of claims 2 to 5, characterized in that, The controller is further configured to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training sub-task, the restart training task message being used to trigger the various working nodes to re-coordinate the execution of each training sub-task of the training task.

7. The system according to any one of claims 2 to 5, characterized in that, The monitor is configured to select the first hot standby node from the m hot standby nodes based on selection factors, provided that a first working node among the n working nodes fails, the selection factors including at least one of the following: The communication speed between the first working node and the m hot standby nodes; The communication bandwidth between the first working node and the m hot standby nodes; The communication speed between the third working node and the m hot standby nodes; The third working node is a working node that has a collaborative requirement with the first working node.

8. The system according to claim 1, characterized in that, The n working nodes are used to report their own resource usage and training status information to the monitor; The monitor is used to determine the first working node that needs to be switched due to a fault, based on the resource usage and training status information of the n working nodes.

9. The system according to claim 1, characterized in that, The controller is used to set the start command and start parameters for the training task; and to coordinate the start of training by the n working nodes based on the start command and start parameters.

10. The system according to claim 9, characterized in that, The controller is used to adjust the number of the m hot standby nodes based on the number of fault exchanges in the most recent time period. The number of fault exchanges refers to the number of exchanges between working nodes and hot standby nodes caused by faults.

11. A distributed training method, characterized in that, Applied to a distributed training system, the distributed training system including a controller, a monitor, n worker nodes and m hot standby nodes, where n and m are positive integers, the method includes: The controller creates training tasks and deploys the training programs corresponding to the training tasks to multiple computing nodes; it divides the multiple computing nodes into n working nodes and m hot standby nodes, with the m hot standby nodes in a ready-to-wait state; and it allocates the same software environment, hardware environment, and running configuration files for the training programs to the n working nodes and the m hot standby nodes. The n working nodes collaboratively execute the training task; If the check result at each checkpoint is normal, the controller sends the basic context information of the training subtask corresponding to each of the n working nodes to the m hot standby nodes; the basic context information includes at least one of the following: the task role of the working node in the training task, the identifier of the working node in the training task, the IP address of the master node of the training task, and the port address of the master node of the training task. The monitor monitors the health status of the n working nodes; If a first working node among the n working nodes fails, the controller switches the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution; the first hot standby node also includes a training process control unit. The training process control unit starts the training process corresponding to the training program; and executes the training sub-task corresponding to the first working node through the training process.

12. A controller, characterized in that, This is applied to a distributed training system, which includes a controller, a monitor, n worker nodes, and m hot standby nodes, where n and m are positive integers. The controller includes: The deployment module is used to create training tasks and deploy the training program corresponding to the training tasks to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, with the m hot standby nodes in a ready-to-wait state; the controller is also used to allocate the same software environment, hardware environment, and running configuration file of the training program to the n working nodes and the m hot standby nodes; the controller is also used to send the basic context information of the training subtasks corresponding to the n working nodes to the m hot standby nodes when the check result at each checkpoint is normal; the basic context information includes at least one of the following: the task role of the working node in the training task, the identifier of the working node in the training task, the IP address of the master node of the training task, and the port address of the master node of the training task; A switching module is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when the first working node fails among the n working nodes; the first hot standby node further includes: a training process control unit; the training process control unit is used to start the training process corresponding to the training program; and execute the training subtask corresponding to the first working node through the training process.

13. A distributed training system, characterized in that, The system includes: a controller and a monitor; The controller is used to create training tasks and deploy the training program corresponding to the training tasks to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, wherein the m hot standby nodes are in a ready-to-wait state; the controller is also used to allocate the same software environment, hardware environment and running configuration file of the training program to the n working nodes and the m hot standby nodes, where n and m are positive integers; The controller is configured to send the basic context information of the training subtasks corresponding to each of the n working nodes to the m hot standby nodes when the check result at each checkpoint is normal; the basic context information includes at least one of the following: the task role of the working node in the training task, the identifier of the working node in the training task, the IP address of the master node of the training task, and the port address of the master node of the training task. The monitor is used to monitor the health status of the n working nodes; The controller is configured to, in the event that a first working node fails among the n working nodes, switch the training subtask corresponding to the first working node to be executed on the first hot standby node among the m hot standby nodes; the first hot standby node further includes: a training process control unit; the training process control unit is configured to start the training process corresponding to the training program; and execute the training subtask corresponding to the first working node through the training process.

Images