Distributed training system and method of artificial intelligence model, and controller

Abstract

The invention discloses a distributed training system and method for an artificial intelligence model, and a controller, and belongs to the technical field of artificial intelligence infrastructures. The system comprises a controller, a monitor, n working nodes and m hot standby nodes, wherein n and m are positive integers; the controller is used for deploying training programs corresponding to the training tasks to the n working nodes and the m hot standby nodes; the n working nodes are used for cooperatively executing training tasks; the monitor is used for monitoring health states of the n working nodes; and the controller is used for switching the training sub-task corresponding to a first working node to a first hot standby node in the m hot standby nodes for execution under the condition that the first working node in the n working nodes has a fault.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence infrastructure technology, and in particular to a distributed training system, method, and controller for artificial intelligence models. Background Art

[0002] As the parameters of artificial intelligence (AI) models continue to grow, with large language models typically requiring hundreds of millions of parameters, distributed training methods are often used to train AI models. Distributed training utilizes multiple computing nodes to complete training tasks. However, as these nodes collaborate to complete training tasks, software and hardware issues can lead to training interruptions.

[0003] In related technologies, AI model training tasks are regularly checkpointed. If a training task is interrupted by a failure, a backup node is found from the backup pool. The training program corresponding to the failed node is deployed to the backup node, and the training task is restarted on the backup node.

[0004] The replacement time of the above faulty nodes is too long, which will slow down the training speed. Summary of the Invention

[0005] The present invention provides a distributed training system, method, and controller for artificial intelligence models, which can be used to reduce the impact of failures on training tasks. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides a distributed training system for an artificial intelligence model, the system comprising: a controller, a monitor, n working nodes, and m hot standby nodes, where n and m are positive integers;

[0007] The controller is configured to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0008] The n working nodes are used to collaboratively execute the training task;

[0009] The monitor is used to monitor the health status of the n working nodes;

[0010] The controller is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

[0011] In a possible implementation, the monitor is configured to select, when a first working node among the n working nodes fails, the first hot standby node from the m hot standby nodes;

[0012] The controller is used to set the working role of the first hot standby node to the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0013] In a possible implementation, the controller is configured to send context information of the training subtask corresponding to the first working node to the second working node;

[0014] The second working node is used to re-execute or continue to execute the training subtask corresponding to the first working node through the training program based on the context information.

[0015] In one possible implementation, the context information includes at least one of the following:

[0016] The task role of the first working node in the training task;

[0017] An identifier of the first working node in the training task;

[0018] The Internet Protocol (IP) address of the master node of the training task;

[0019] The port address of the master node of the training task;

[0020] The address of the training data set for the training task;

[0021] The storage address of the last checkpoint of the training task;

[0022] The model configuration of the AI ​​model trained by the training task.

[0023] In a possible implementation, the second working node further includes: a training process control unit;

[0024] The training process control unit is used to start the training process corresponding to the training program; and re-execute or continue to execute the training subtask corresponding to the first working node based on the context information through the training process.

[0025] In one possible implementation, the controller is also used to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training subtask, and the restart training task message is used to trigger the each working node to re-coordinate and execute the each training subtask of the training task.

[0026] In one possible implementation, the monitor is configured to, when a first working node among the n working nodes fails, select the first hot standby node from the m hot standby nodes based on a selection factor; the selection factor includes at least one of the following:

[0027] Communication speed between the first working node and the m hot standby nodes;

[0028] The communication bandwidth between the first working node and the m hot standby nodes;

[0029] Communication speed between the third working node and the m hot standby nodes;

[0030] The third working node is a working node that has collaboration requirements with the first working node.

[0031] In a possible implementation, the n working nodes are configured to report their own resource usage and training status information to the monitor;

[0032] The monitor is used to determine the first working node that needs to be switched due to a fault based on the resource usage and training status information of the n working nodes.

[0033] In a possible implementation, the controller is configured to deploy the training program corresponding to the training task to a plurality of computing nodes; and divide the plurality of computing nodes into the n working nodes and the m hot standby nodes.

[0034] In a possible implementation, the controller is configured to adjust the number of the m hot standby nodes based on the number of failure switches in a recent time period;

[0035] The number of fault exchanges is the number of exchanges between the working node and the hot standby node caused by the fault.

[0036] On the other hand, an embodiment of the present application provides a distributed training method, which is applied to a distributed training system. The distributed training system includes a controller, a monitor, n working nodes, and m hot standby nodes, where n and m are positive integers. The method includes:

[0037] The controller deploys the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0038] The n working nodes collaboratively perform the training task;

[0039] The monitor monitors the health status of the n working nodes;

[0040] When a first working node among the n working nodes fails, the controller switches the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution.

[0041] On the other hand, an embodiment of the present application provides a controller applied to a distributed training system, wherein the distributed training system includes the controller, a monitor, n working nodes, and m hot standby nodes, where n and m are positive integers, and the controller includes:

[0042] A deployment module, configured to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes;

[0043] The switching module is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

[0044] On the other hand, an embodiment of the present application provides a distributed training system, the system comprising: a controller and a monitor;

[0045] The controller is used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes, where n and m are positive integers;

[0046] The monitor is used to monitor the health status of the n working nodes;

[0047] The controller is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

[0048] The technical solutions provided by the embodiments of the present application bring at least the following beneficial effects:

[0049] The hot standby node mechanism in the present application greatly reduces the time of replacing faulty nodes. When the monitor detects a fault in a working node (such as the first working node), the controller can switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution, avoiding long waiting times for other nodes, thereby effectively improving the fault handling speed and reducing the impact of faults on training tasks. Since the hot standby node can quickly take over the tasks of the faulty node, the working node can focus more on the training task during normal operation without reserving too many resources or processing power for possible faults. This provides a more stable environment for distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 This is a structural diagram of a distributed training system provided in an embodiment of the present application;

[0052] Figure 2 This is a flow chart of a distributed training method provided in an embodiment of the present application;

[0053] Figure 3 This is an overall flow chart of a distributed training method provided in an embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of the structure of a controller provided in an embodiment of the present application;

[0055] Figure 5 This is a structural diagram of a distributed training system provided in an embodiment of the present application;

[0056] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0057] Figure 7 This is a structural diagram of a computing node virtualization system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0059] It should be noted that the terms "first," "second," and the like in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application as detailed in the appended claims.

[0060] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0061] Distributed training: This is a training method that divides and distributes computing tasks to multiple computing devices (such as multiple graphics processing units, multiple machines, etc.) for simultaneous operation. The core idea is to accelerate the model training process through parallel computing.

[0062] In related technologies, large AI models are usually trained in a distributed environment. Distributed training relies on the collaboration between multiple computing nodes to complete the work. Distributed training often encounters training interruptions due to various software and hardware problems. In related technologies, the method to deal with this problem is: first, regularly set checkpoints for the AI ​​model training tasks; second, when a training failure interruption occurs, identify the faulty node, isolate it, and replace it with a healthy backup node; finally, restart from the previous checkpoint and resume the training task. During the replacement process, other computing nodes are in a waiting state and cannot continue with effective training tasks, resulting in a long replacement time for the faulty node.

[0063] To further illustrate the technical solutions provided by the embodiments of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. During the actual processing process or when the control device is executed, the method can be executed in the order of the methods shown in the embodiments or drawings or in parallel.

[0064] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0065] Figure 1The present invention provides a structural diagram of a distributed training system, which includes a controller 101, a monitor 102, n working nodes 103, and m hot standby nodes 104, where n and m are positive integers; the controller 101 is used to deploy the training program corresponding to the training task to multiple computing nodes; the multiple computing nodes are divided into n working nodes 103 and m hot standby nodes 104; based on the number of fault exchanges in the recent time period, the number of m hot standby nodes 104 is adjusted; wherein the number of fault exchanges is the number of exchanges between the working node and the hot standby node due to a fault; the n working nodes 103 are used to coordinate The monitor 102 performs the training task and reports its own resource usage and training status information to the monitor 102; the monitor 102 is used to monitor the health status of the n working nodes 103, and based on the resource usage and training status information of the n working nodes 103, determines the first working node to be switched due to a fault. In the case that the first working node fails among the n working nodes 103, the first hot standby node is selected from the m hot standby nodes 104 based on the selection factors, wherein the selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes 104; the communication speed between the first working node and the m hot standby nodes 104; 4; the communication speed between the third working node and the m hot standby nodes 104; wherein the third working node is a working node that has a collaborative requirement with the first working node; the controller 101, used to set the working role of the first hot standby node to the second working node when the first working node fails among the n working nodes 103; send the context information of the training subtask corresponding to the first working node to the second working node, wherein the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; the model configuration of the AI ​​model trained by the training task; sending a restart training task message to each working node corresponding to the training task, the restart training task message is used to trigger each working node to re-coordinate and execute each training subtask of the training task; the second working node also includes: a training process control unit; a training process control unit, used to start the training process corresponding to the training program; re-execute or continue to execute the training subtask corresponding to the first working node based on the context information through the training process.

[0066] like Figure 1As shown, an embodiment of the present application provides a distributed training system for an artificial intelligence model, and the system includes: a controller 101, a monitor 102, n working nodes 103 and m hot standby nodes 104, where n and m are positive integers; the controller 101 is used to deploy the training program corresponding to the training task to the n working nodes 103 and the m hot standby nodes 104; the n working nodes 103 are used to collaboratively execute the training task; the monitor 102 is used to monitor the health status of the n working nodes 103; the controller 101 is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes 104 for execution when a first working node among the n working nodes 103 fails.

[0067] The controller 101 plays a core control role in the distributed training system and is responsible for coordinating and managing various operations in the entire distributed training process.

[0068] A training task is the purpose of the entire distributed training system and can be understood as a clear goal, such as training a large-scale neural network model for image recognition, natural language processing, or other AI applications. A training task has clear input and output requirements. The input may be a large dataset, such as large amounts of image data or text data, while the output is the trained model parameters. A large training task is typically broken down into multiple subtasks. In a distributed training system, these subtasks are distributed to different worker nodes for collaborative execution. For example, when training a deep neural network, the computational tasks corresponding to different layers or different data subsets can be considered subtasks. These subtasks are computed in parallel on each worker node and then combined through pre-defined mechanisms (such as parameter synchronization) to produce the final training results.

[0069] The training program corresponding to a training task can be understood as the algorithmic implementation that achieves the training task's objectives. It encompasses various algorithms used to train models, such as the stochastic gradient descent algorithm and its variants used in neural network training. These algorithms are implemented programmatically in the training program to process input data, calculate gradients based on the loss function, and update model parameters. For example, consider a training program written in the computer programming language Python.

[0070] The training program also defines the structure of the model to be trained, including the number of layers, the number of neurons in each layer, the type of activation function, and so on. For example, in a training program for building a convolutional neural network, the parameters and connection methods of the convolutional layer, pooling layer, and fully connected layer are defined. Furthermore, the training program also includes data processing logic. For example, how to read input data, how to preprocess the data (such as normalization and data augmentation), and how to distribute the data to different computing nodes for processing.

[0071] The training program interacts with the hardware, especially with hardware devices such as the graphics processors on the computing nodes. It uses the parallel computing capabilities of the graphics processors to accelerate the training process by calling corresponding libraries and interfaces.

[0072] A worker node can be thought of as the execution unit in a distributed training system. It's responsible for actually executing training tasks, and multiple worker nodes work together to complete the entire training task. For example, in a large-scale neural network training task, a worker node will compute its assigned subtasks, including forward propagation of data, calculation of loss functions, and backpropagation of gradients. It also collaborates with other worker nodes to update model parameters.

[0073] The hot standby node can be understood as a backup mechanism. While the active node is operating normally, it does not initiate training-related processes and remains in a ready-to-follow state for the training program. This ready-to-follow state means the hot standby node is ready to accept tasks at any time. For example, the training program has been installed and configured, and its resources have been allocated. As soon as the controller 101 issues a command, the hot standby node can quickly initiate a training task.

[0074] As an optional implementation, the n working nodes 103 and the m hot standby nodes 104 have the same software environment, hardware environment and running configuration files of the training program.

[0075] Since the hot standby node has the same resources, software and hardware environment as the working node, when the monitor detects a failure of the working node (such as the first working node), the controller can quickly switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes 104 for execution, thereby improving the efficiency of fault handling.

[0076] Worker nodes and hot standby nodes share the same hardware resources. Whether implemented as bare metal machines, pods within the open-source container orchestration system Kubernetes, or other resource management components within cluster scheduling systems, they all share similar compute resources, storage resources (such as memory and hard drives), and network resources (such as network interfaces and bandwidth). For example, in a distributed training system, if worker nodes are equipped with a preset model and number of GPUs to accelerate deep learning training tasks, the hot standby nodes should also be equipped with a corresponding number of GPUs. Regarding software, the operating system, drivers, deep learning frameworks, and other related libraries are identical between worker and hot standby nodes. For example, when using PyTorch for deep learning training, both worker and hot standby nodes have the same version of the PyTorch framework and its dependent libraries installed. Furthermore, their operating system versions and configurations, including kernel parameter settings and file system formats, are identical. This ensures that, in the event of a failover, the training program can run on the hot standby node without requiring additional software adaptation.

[0077] During normal operation, although the hot standby nodes do not participate in the execution of training tasks, they are associated with the working nodes in terms of resource allocation. When allocating resources, the controller 101 needs to consider the needs of both the working nodes and the hot standby nodes. For example, in a system with limited resources, in order to ensure that there are enough hot standby nodes, it is necessary to reasonably allocate hardware resources between the working nodes and the hot standby nodes. If too many resources are allocated to the hot standby node, the available resources of the working node will be reduced, affecting the execution efficiency of the training task; conversely, if the hot standby node resources are too few, it may not be able to meet the needs during failover.

[0078] Although the existence of hot standby nodes does not directly participate in the training task, it has a certain impact on the overall performance of the system, and this impact is closely related to the working nodes. For example, hot standby nodes occupy certain system resources (such as storage resources for storing training program-related files, etc.), which will reduce the total amount of resources available to the working nodes. At the same time, the system will also consume certain computing resources when managing hot standby nodes (such as the status monitoring of hot standby nodes by the controller 101, etc.). Therefore, when designing a distributed training system, it is necessary to balance the number and resource allocation of hot standby nodes and working nodes to achieve the optimization of the overall performance of the system.

[0079] The following describes the process of deploying the training program by the controller 101.

[0080] As an optional implementation, the controller 101 is used to create a training task and deploy the training program corresponding to the training task to multiple computing nodes; divide the multiple computing nodes into n working nodes 103 and m hot standby nodes 104, and the m hot standby nodes 104 are in a ready waiting state; set the start command and start parameters of the training task; coordinate n working nodes 103 to start training based on the start command and start parameters.

[0081] The process of the controller 101 deploying the training program corresponding to the training task to the working node and the hot standby node reflects the role of the controller 101 in allocating resources and scheduling training tasks, thereby achieving full utilization of resources.

[0082] Compute nodes are the fundamental building blocks of distributed training systems. From a hardware perspective, a compute node typically refers to a graphics processing unit (GPU) server. A GPU server contains several key hardware components, the most important of which are multiple GPU cards used for training. In distributed training scenarios (such as 1000- and 10,000-GPU training), multiple compute nodes work together. Each compute node is responsible for a portion of the training task, and they communicate with each other over the network, exchanging information such as data and model parameters. For example, when training a large deep learning model, different compute nodes are responsible for processing different batches of data or computing parameter updates for different parts of the model.

[0083] The controller 101 first comprehensively evaluates the hardware resources and software environment of the entire distributed training system; based on the evaluation results of the hardware resources and software environment, it lays out a node list, which includes the computing nodes required for the training task.

[0084] The layout of the node list includes, but is not limited to: determining the computing nodes in the node list based on the autonomous management function of the controller 101 itself (when the computing nodes are hardware, the controller 101 autonomously searches and allocates each computing node; when the computing nodes are containers, the controller 101 autonomously creates each computing node), and performing subsequent maintenance and recycling of the created computing nodes; or, if the distributed system includes a resource management system for managing computing nodes, the controller 101 requests the node list from the resource management system. This request process involves sending a request containing resource requirement information to the resource management system. For example, the request may include a request for a certain number of nodes with graphics processor models and sufficient memory, and a requirement that a preset version of an operating system and a deep learning framework be installed on the nodes; the resource management system processes the controller 101's request based on its own resource allocation strategy and the current available resources. If there are sufficient available resources to meet the controller 101's requirements, the resource management system will return a node list containing information about nodes that meet the requirements to the controller 101.

[0085] After receiving the node list from the resource management system, controller 101 first parses the node information in the compute node list. Node information includes, but is not limited to, the node's unique identifier (such as the host name), hardware configuration information (such as the number of CPU cores), and software environment information (such as the operating system version and installed software packages). By parsing this node information, controller 101 fully understands the basic information of each compute node, preparing for subsequent deployment and role assignment.

[0086] When deploying the role information of the computing node (working node or hot standby node), the controller 101 must first consider the trade-off between reliability and performance. In order to improve the reliability of the distributed training system, a sufficient number of hot standby nodes are required. However, too many hot standby nodes will occupy resources and affect the overall performance of the system. The controller 101 can set a reliability index (such as the maximum allowable failure probability) and a performance index (such as the expected training speed improvement ratio). Then, based on these two indicators and the analysis of the training task, the appropriate ratio of working nodes and hot standby nodes is determined.

[0087] In addition, when the controller creates a training task, different training tasks have different characteristics, and these characteristics will also affect the allocation of role information of the computing nodes. For computationally intensive training tasks, such as large-scale natural language processing model training, more working nodes are required to speed up the computing process. For data-intensive training tasks, such as processing massive amounts of image data, it is necessary to consider data storage and transmission factors in the allocation of working nodes and hot standby nodes. If the training task has high real-time requirements for data, the controller 101 tends to allocate more computing nodes as working nodes to speed up data processing. At the same time, in order to ensure the reliability of the system, a certain proportion of hot standby nodes will be retained. For example, for a real-time video stream analysis training task, the working nodes need to quickly process each frame of the video data. At this time, 80% of the computing nodes can be allocated as working nodes and 20% of the computing nodes can be allocated as hot standby nodes.

[0088] The controller 101 also needs to consider the balanced allocation of resources between working nodes and hot standby nodes. In terms of hardware resources, it is necessary to ensure that both working nodes and hot standby nodes have access to relatively balanced computing resources, storage resources (such as memory, hard disk, etc.), and network resources (such as bandwidth). For example, in a system consisting of 100 computing nodes, if 60 computing nodes are assigned as working nodes and 40 computing nodes are assigned as hot standby nodes, it is necessary to ensure that there is not too large a gap between the working nodes and the hot standby nodes in terms of average graphics processor computing power, memory capacity, and network bandwidth.

[0089] Based on the policy for assigning compute node roles, controller 101 tags the compute nodes in the node list with their roles. For nodes identified as active nodes, controller 101 adds active node identification to their node information, such as by setting a target flag or writing a "active node" tag in the node configuration file. Similarly, hot standby nodes are also identified.

[0090] In a distributed training system, resources include computing resources (such as graphics processors and central processing units), storage resources (such as hard disk space), etc. The controller 101 needs to understand the resource status of each computing node (active nodes and hot standby nodes), such as the model, quantity, and memory size of the graphics processor on each computing node; the number of cores, main frequency, and available storage capacity of the central processing unit; when deploying the training program corresponding to the training task to the computing node, the controller 101 can allocate the appropriate computing node based on the resource requirements of the training program (such as how much graphics processor memory is needed to store model parameters, etc.).

[0091] In addition, the controller 101 is responsible for decomposing the training task into subtasks and assigning these subtasks to different work nodes. For a complex training task, such as training a large deep learning model, the controller 101 must consider the structural characteristics of the model (such as some layers can be calculated in parallel, and some layers have a sequence of calculations) and the characteristics of the data (such as the distribution of the data) to reasonably arrange the allocation of subtasks. When allocating subtasks, the controller 101 must also consider factors such as the computing power of the work nodes and the network connection status. For example, subtasks with larger computational requirements are assigned to work nodes with stronger computing power, and subtasks with higher network communication requirements are assigned to work nodes with better network connections to improve the execution efficiency of the entire training task.

[0092] The following describes how to determine the values ​​of n and m.

[0093] Methods for determining the values ​​of n and m include, but are not limited to: determining based on historical data; or, adjusting based on real-time monitoring and feedback; or, determining based on cost-effectiveness.

[0094] When the values ​​of n and m are determined based on historical data, the controller 101 determines the values ​​of n and m by analyzing the failure rate in the historical data. Collect the failure conditions of the working nodes in previous training tasks and calculate the failure rate. For example, in the past 100 training tasks, the working node has a total of 10 failures, so the failure rate is 10%. Determine the number of hot standby nodes m based on the failure rate. If you want to be able to quickly resume training when a failure occurs and can withstand a certain risk of failure, then you can set a hot standby node ratio based on the failure rate. For example, if the failure rate is 10%, you can consider setting m to 10%-20% of n, that is, if there are 100 working nodes (n=100), then the number of hot standby nodes m can be set to 10-20. In addition, check the resource utilization of the working nodes and hot standby nodes in past training tasks, including the utilization of the graphics processor, the utilization of the central processing unit, the memory utilization, etc. If you find that worker nodes often have high resource utilization, while hot standby nodes have low resource utilization, you can adjust the ratio of n to m by increasing the number of worker nodes (n) and reducing the number of hot standby nodes (m) to improve overall system resource utilization. For example, if the average GPU utilization of worker nodes reaches 80%, while the GPU utilization of hot standby nodes is only 10%, you can reduce the number of hot standby nodes and allocate the freed-up resources to worker nodes, thereby improving training efficiency.

[0095] When the values ​​of n and m are adjusted based on real-time monitoring and feedback, during training, controller 101 monitors the resource usage of active and standby nodes in real time via monitor 102. For example, information such as the GPU usage, CPU usage, and memory usage of each node is obtained at regular intervals (e.g., 1 minute). Based on the real-time monitoring results, if active nodes are resource-constrained while standby nodes have more idle resources, controller 101 can adjust the values ​​of n and m. Some standby nodes can be converted into active nodes, increasing the number of active nodes n while reducing the number of standby nodes m to meet the resource requirements of the training task.

[0096] The start command is the basic instruction for launching a training task. In the context of large-scale distributed training, it is used to initiate the entire training process. This basic instruction tells the compute nodes to perform specific operations, namely, to run the training script. It is a key identifier for the training task entry point, allowing the worker nodes to know where to begin executing training-related code.

[0097] Launch parameters provide more detailed configuration information for training tasks. These parameters can adjust various characteristics of the training task. For example, for image classification training, launch parameters include the batch size. The batch size determines the number of data samples used for training in each iteration. If the batch size is set to 32, the model will process 32 image samples simultaneously during each training step.

[0098] After the controller sets the startup command and parameters, it needs to distribute this information to n worker nodes 103. First, it must ensure that each worker node receives the same startup command and parameters to ensure consistency of training tasks across all nodes. For example, in a distributed training system consisting of 10 worker nodes 103, the controller will send the startup command and parameters to each worker node.

[0099] The controller sets and distributes unified startup commands and startup parameters, and coordinates the worker nodes to start training, ensuring that all worker nodes are trained according to the same rules and configurations in a large-scale distributed training environment.

[0100] As an optional implementation, when the controller starts a training task, m hot standby nodes are started at the same time.

[0101] When the controller initiates a training task, it simultaneously starts m hot standby nodes 104, eliminating the need to create an initial runtime environment when a failure occurs. Because the m hot standby nodes 104 are already in a ready state, they can immediately take over the tasks of the failed active node without requiring additional operations or events to trigger. This significantly reduces failure recovery time and improves training efficiency.

[0102] As an optional implementation, the controller 101 is configured to adjust the number of m hot standby nodes 104 based on the number of fault swaps in a recent time period; wherein the number of fault swaps is the number of swaps between the working node and the hot standby node due to a fault.

[0103] The number of failovers in a recent period of time can reflect the current stability and reliability of the distributed training system. By analyzing the number of failovers in this period, we can promptly identify potential issues with the distributed training system and the need to adjust the number of hot standby nodes. This allows us to implement a dynamic, adaptive system management mechanism to adapt to the ever-changing distributed training system operating environment and training task requirements.

[0104] The recent time period is a relative term, referring to a time window during the distributed training process. The length of this time window can be customized. For example, it can be set to the past hour, two hours, or a day. Within this time window, the distributed training system counts and analyzes the number of failovers, allowing it to adjust the number of hot standby nodes accordingly.

[0105] Failures include, but are not limited to, at least one of: graphics processor card failure, central processing unit failure, memory failure, network device failure, deep learning framework failure, operating system failure, or driver failure. This application does not limit the type of failure.

[0106] For example, a graphics processor card may experience hardware failures such as video memory damage, core chip overheating, circuit short circuits, and other problems. Video memory damage may cause data storage and reading errors, affecting data processing during training. For example, when training a deep learning model, when a large amount of model parameters and data needs to be loaded into the video memory, if the video memory fails, data loss will occur or the model cannot be trained normally. Core chip overheating may be caused by long-term high-load operation, cooling system failure, etc. Overheating can cause the graphics processor card performance to decline or even automatically reduce the frequency, thereby slowing down the training speed or interrupting the training.

[0107] For example, a CPU failure may manifest as a core failure, cache failure, or instruction set execution error. A core failure can affect task scheduling and data processing capabilities, a cache failure can significantly reduce data read speeds, and an instruction set execution error can cause program execution to malfunction.

[0108] The number of fault swaps is a metric in the distributed training system described above. It refers to the number of swaps between working nodes and hot standby nodes due to faults. When a working node among the n working nodes 103 fails, the controller 101 switches the training subtask corresponding to the failed working node to one of the m hot standby nodes 104 for execution. This operation is counted as one fault swap. For example, in a distributed training process, if three working nodes fail successively in a system with 10 working nodes and 5 hot standby nodes, and each failure successfully switches the task of the failed working node to the hot standby node, then the number of fault swaps is 3. This metric reflects the frequency of task transfers between working nodes and hot standby nodes due to faults during the operation of the distributed training system. It is a dynamic statistic that is continuously updated as the training process progresses and can intuitively reflect the stability and reliability of the system.

[0109] Optionally, the method of adjusting the number of m hot standby nodes 104 based on the number of fault exchanges includes but is not limited to: linearly increasing or decreasing the number of m hot standby nodes 104 according to a certain ratio based on the number of fault exchanges. For example, if it is set that 1 hot standby node is added every 10 fault exchanges, when the number of fault exchanges reaches a threshold (such as 50 times), the number of hot standby nodes is gradually increased according to this ratio; or, adjusting the number of m hot standby nodes 104 according to the growth trend of the number of fault exchanges. For example, when the number of fault exchanges begins to increase, the number of hot standby nodes can be slowly increased. For example, when the number of fault exchanges is between 1 and 10 times, 0.5 hot standby nodes are added each time (similar increase effects can be achieved by adding some hardware resources or adjusting software configurations). When the number of fault exchanges increases rapidly (such as after exceeding 10 times, 2 hot standby nodes are added for every 5 fault exchanges).

[0110] The following describes how the controller 101 deploys the training program corresponding to the training task to the working node.

[0111] The controller 101 deploys the training program corresponding to the training task to the working node, including at least one of: resource adaptability check, training task decomposition and allocation, and program installation and configuration.

[0112] Before deploying the training program corresponding to the training task to the working node, the controller 101 must first perform a hardware resource adaptability check, which includes but is not limited to: checking whether the hardware resources such as the graphics processor, central processing unit, and memory on the working node meet the requirements of the training program. For example, for a training program that requires a large amount of graphics processor video memory to store model parameters, the controller 101 must ensure that the graphics processor of the working node has a large enough video memory. If the hardware resources of the working node do not meet the requirements of the training program, the controller 101 will take some measures, such as adjusting the parameters of the training program (such as reducing the batch size to reduce video memory usage) or replacing the working node (if there are other working nodes that meet the requirements to choose from).

[0113] Controller 101 also checks the software resource compatibility of the worker nodes. This includes, but is not limited to, checking whether the operating system, drivers, deep learning framework, and other relevant software libraries required by the training program are installed on the worker nodes. For example, if the training program is developed based on PyTorch, the corresponding version of PyTorch and its dependent libraries must be installed on the worker nodes. If the software resources do not match, controller 101 will be responsible for installing or updating the relevant software resources on the worker nodes to ensure that the training program can run normally on the worker nodes.

[0114] Controller 101 decomposes the training program corresponding to the training task into subtasks based on the parallel characteristics of the training task. For data-parallel training tasks, controller 101 can divide the data into several subsets based on the size of the dataset, with the computation corresponding to each subset being a subtask. For example, a dataset containing 1 million images can be decomposed into 10 subtasks, with each subset consisting of 100,000 images. For model-parallel training tasks, controller 101 divides the model into several parts based on its structure, with the computation corresponding to each part being a subtask. For example, for a multi-layer neural network, different layers can be assigned to different worker nodes as subtasks. Controller 101 assigns the decomposed subtasks to different worker nodes based on the resource availability and performance characteristics of the worker nodes. For example, subtasks with higher computational requirements can be assigned to worker nodes with greater computing power, while subtasks with higher network communication requirements can be assigned to worker nodes with better network connectivity. When assigning subtasks, controller 101 also considers load balancing between worker nodes. Try to make the load of each working node relatively balanced to avoid the situation where one working node is overloaded while other working nodes are underloaded, so as to improve the execution efficiency of the entire training task.

[0115] The controller 101 installs the training program corresponding to the training task on the working node, which may involve copying the code file of the training program to the specified directory of the working node, and then compiling or interpreting it according to the requirements of the training program. For example, if the training program is written in Python, the controller 101 may need to install the Python interpreter on the working node and copy the .py file of the training program to the appropriate directory. After installing the training program, the controller 101 must configure the training program. This includes setting the input and output paths of the training program, the initial parameters of the model, the hyperparameters of the training (such as learning rate, batch size, etc.), etc. For example, specify which directory the training program reads the training data from, which directory the trained model parameters are saved to, and setting the initial learning rate.

[0116] Optionally, when there is one training task, the deployment method of the controller 101 includes but is not limited to: dividing the training task into several subtasks, and assigning a training program corresponding to a subtask to each of the n working nodes 103; or, deploying all training programs corresponding to the training task to each of the n working nodes 103.

[0117] Optionally, when there are at least two training tasks, the controller 101 deployment method includes but is not limited to: determining the priority of each training task; and allocating and deploying the resources of n work nodes 103 according to the priority of the training task. The priority determination method includes but is not limited to: based on at least one of the urgency of the task, the importance of the task, or the expected benefits of the task. For example, a task related to a company's core business and requiring training results as quickly as possible will have a higher priority; while a task used for exploratory research and with less urgent time requirements will have a relatively lower priority.

[0118] Optionally, more worker nodes are allocated to higher-priority training tasks. For example, if there are 10 worker nodes, 6 worker nodes are allocated to execute the training program for the highest-priority training task, while only 2-3 worker nodes are allocated to lower-priority training tasks.

[0119] Furthermore, on each worker node, the training program is deployed based on the training task. For high-priority tasks assigned to a large number of worker nodes, a more refined subtask decomposition and allocation approach may be adopted to fully utilize the resources of these worker nodes. For low-priority tasks, a relatively simple deployment approach may be adopted, such as deploying the entire training program to a smaller number of worker nodes to reduce the complexity of resource allocation and management.

[0120] Optionally, the controller 101 deploys the training programs corresponding to all training tasks on n work nodes 103 respectively. In this deployment mode, the entire training program is deployed on each work node. For example, for a small deep learning model training task, each work node has a complete model structure definition, data loading module, etc. Each work node is trained independently, using its own data subset or performing different sampling methods on the entire data set for training. During the training process, the work nodes independently calculate the update of the model parameters. Finally, the controller 101 needs to integrate the training results obtained by the n work nodes 103. For example, for model parameters, the model parameters obtained by each work node can be merged by averaging or weighted averaging to obtain the final training results. This method improves the efficiency of training, especially when the communication cost between work nodes is high, reducing the need for frequent communication between work nodes.

[0121] Optionally, the controller 101 may adopt a hybrid deployment approach, where some training tasks are deployed as a whole on n worker nodes 103, while other training tasks are broken down into subtasks for deployment. For example, for a complex training task containing multiple sub-models, some of which have relatively simple structures and have little impact on the overall training results, the training programs corresponding to these sub-models can be deployed as a whole on certain worker nodes. As for the core sub-model, which has a complex structure and has a greater impact on the training results, it is broken down into subtasks and assigned to other worker nodes.

[0122] Another hybrid deployment method is to dynamically adjust the deployment method based on model characteristics and work node resources. During the training process, the controller 101 can change the deployment method of the training program on the n work nodes 103 according to the convergence of the model and the remaining resources of the work nodes. For example, in the early stage of training, the entire training program is deployed to the n work nodes 103 in the form of subtasks to speed up the initial convergence of the model. As the training progresses, when some work nodes have surplus resources and some parts of the model require more computing resources for optimization, the controller 101 deploys the training program corresponding to this part of the model as a whole to these work nodes with surplus resources to improve the final training effect of the model.

[0123] The following describes how the controller 101 deploys the training program corresponding to the training task to the m hot standby nodes 104 .

[0124] Optionally, the controller 101 deploys the training program corresponding to the training task to m hot standby nodes 104 in a manner including but not limited to: each of the m hot standby nodes 104 deploys the training program corresponding to all training tasks; or, each of the m hot standby nodes 104 deploys the training program corresponding to the sub-training task in the training task; or, based on the microservice architecture, the training program corresponding to the training task is deployed to m hot standby nodes 104; or, based on incremental deployment, the training program corresponding to the training task is deployed to m hot standby nodes 104; or, based on dynamic loading deployment, the training program corresponding to the training task is deployed to m hot standby nodes 104.

[0125] When each of the m hot standby nodes 104 deploys the training program corresponding to all training tasks, the controller 101 first obtains the complete training program and its related configuration files, dependency libraries, etc., and at the same time obtains the predefined configuration files, which contain information such as training parameters (learning rate, batch size, etc.), data path, etc. Then, the controller 101 distributes the complete training program package to each of the m hot standby nodes 104. After the m hot standby nodes 104 receive the complete training program package, they store it in a local preset storage location. The preset storage location can be pre-set according to the file system structure of the hot standby node. At the same time, the hot standby node must decompress the received training program (if it is in the form of a compressed package) and perform basic environmental configuration, such as setting the environmental variables required for runtime.

[0126] Optionally, when any working node fails, the hot standby node needs to replace the failed working node. Since any hot standby node deploys all training programs, there may be some redundant training programs in any hot standby node (relative to the subtasks executed by the failed working node). At this time, the hot standby node needs to determine the part that needs to be retained and executed based on the task identifier of the failed working node, and delete the redundant training programs to save computing resources. For example, if the failed working node is responsible for processing part of the data in the training data set (such as data blocks 1-100), any hot standby node needs to filter out the modules and code segments related to processing this part of the data from the complete training program, delete other irrelevant parts, and then start executing the subtask.

[0127] When each of the m hot standby nodes 104 deploys the training program corresponding to the sub-training task in the training task, before the training task begins, the controller 101 divides the entire training task into sub-tasks according to the preset rules. The preset rules can be based on the segmentation of the training data set, for example, the training data set is evenly divided into n subsets according to the amount of data (assuming there are n working nodes 103 in total), and each subset corresponds to a subtask. It can also be based on the segmentation of the model structure, for example, different layers of the deep learning model are assigned to different subtasks for training. Then, according to the number of hot standby nodes m, these subtasks are further distributed to the m hot standby nodes 104. The controller 101 deploys the training program of the corresponding subtask to each hot standby node based on the division results of the subtask. The deployment process includes distributing the code modules, configuration files, data subsets (if it is a subtask based on data set segmentation) related to the subtask to the m hot standby nodes 104. After receiving the training program of the subtask corresponding to itself, the m hot standby nodes 104 perform local installation and configuration.

[0128] When the training program corresponding to the training task is deployed to m hot standby nodes 104 based on the microservice architecture, the training program is split into multiple microservices according to function. For example, for deep learning training tasks, functions such as data reading and preprocessing, model building, loss function calculation, and optimizer application can be split into independent microservices. Each microservice has a clear functional boundary and input and output interface. For each split microservice, an independent operating unit is constructed. The controller 101 deploys the constructed microservice to m hot standby nodes 104. Microservices can be allocated to m hot standby nodes 104 based on the functional importance, resource requirements, and resource status of the hot standby nodes of the microservices. For example, the data reading and preprocessing microservices can be deployed on a hot standby node with relatively abundant storage resources, and the model building microservices can be deployed on a hot standby node with relatively abundant computing resources.

[0129] When the training program corresponding to the training task is deployed to m hot standby nodes 104 based on incremental deployment, the controller 101 deploys the core part of the training program to the m hot standby nodes 104. The core part includes the basic framework of the training task, such as the basic structure definition of the deep learning model, the basic training algorithm, etc. During the training process, as the training task progresses or the training strategy is adjusted, new code, data or configuration information will need to be updated to the m hot standby nodes 104. The controller 101 deploys these updated contents to the m hot standby nodes 104 in an incremental manner. For example, if a new data enhancement strategy is added, the controller 101 only needs to deploy the code and configuration files related to the data enhancement strategy to the m hot standby nodes 104 without redeploying the entire training program.

[0130] Optionally, during the incremental update deployment process, the m hot standby nodes 104 correctly integrate the new update content. Based on the type and impact of the update content, the m hot standby nodes 104 rationally adjust their operating states to accommodate new training requirements. The incremental deployment method reduces deployment workload and resource consumption. By deploying only the core components and incremental update content, network transmission bandwidth and hot standby node storage space are saved compared to deploying the entire training program each time. This can effectively improve the system's resource utilization efficiency, especially when training tasks last for a long time and updates are frequent.

[0131] When the training program corresponding to the training task is deployed to m hot standby nodes 104 based on dynamic loading, the controller 101 first deploys the basic part of the training program to the m hot standby nodes 104. The basic part includes some common code modules, such as basic operating system calls, the use of standard library functions, etc., as well as some basic configuration information of the training task, such as the name and identifier of the training task. These basic parts provide an operating framework for subsequent dynamic loading. On the hot standby node, a dynamic loading framework environment is established. The framework environment can be based on the dynamic link library mechanism of the operating system or the dynamic loading function of the programming language. The framework environment must be able to identify and load new code modules, data files and configuration information. During the training process, when the training programs of the m hot standby nodes 104 need to be updated or supplemented, the controller 101 sends the new content to the hot standby node in a dynamically loadable form. For example, if a new training algorithm needs to be added to the training program, the controller 101 sends the code module containing the new algorithm to the hot standby node in the form of a dynamic library. After receiving the new content, the hot standby node uses the dynamic loading framework environment to load it into the running training program (if the hot standby node is in the ready waiting state, it is loaded into the pre-running framework).

[0132] Alternatively, a dynamic loading-based deployment approach offers high flexibility and real-time performance. New functions, algorithms, or data can be added to the m hot standby nodes 104 as needed during training, without requiring the entire training program to be redeployed. This is particularly advantageous for large-scale distributed training where continuous adjustment and optimization of training strategies are required.

[0133] Optionally, the controller 101 is used to send basic context information of the training subtasks corresponding to the n working nodes 103 to the m hot standby nodes 104 when the n working nodes 103 are working normally; the m hot standby nodes 104 are used to receive the basic context information.

[0134] Optionally, the controller 101 is configured to send the basic context information of the training subtask corresponding to each of the n working nodes 103 to the m hot standby nodes 104 when the check result of each checkpoint is normal. That is, when the check result of each checkpoint is normal, it proves that the n working nodes 103 are in a healthy state. At this time, the checkpoint check result is synchronized once to the m hot standby nodes 104.

[0135] Optionally, the basic context information includes at least one of the following: the task role of any one of the n work nodes 103 in the training task; the identifier of any one of the n work nodes 103 in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0136] The hot standby node is in a ready waiting state for the training program, but this state is not completely static. The hot standby node will perform some pre-operations according to the state changes of the working node. For example, when the model parameters of the working node are updated, the hot standby node will receive basic context information (although no actual model training is performed), so that when the working node needs to be replaced, the first hot standby node among the m hot standby nodes 104 can obtain the basic context information of the latest updated n working nodes 103 in the last normal checkpoint, thereby enabling the training to be started more quickly using the latest model parameters and shortening the resource docking time during the replacement process of the first hot standby node. In addition, the task status of the hot standby node also includes its own health status monitoring. Although the hot standby nodes do not perform training tasks, they also need to keep their own hardware and software environment healthy so that they can normally take over the tasks of the working nodes when needed.

[0137] In the above manner, the controller 101 determines the role information and training programs corresponding to the n working nodes 103 and the role information and training programs corresponding to the m hot standby nodes 104. The controller 101 configures corresponding parameters for computing nodes of different roles.

[0138] For the worker nodes, parameters related to training task execution need to be configured, such as the subtask range assigned to each of the n worker nodes 103, the data reading path, and the port number for communicating with other worker nodes. For the m hot standby nodes 104, although they do not start training-related processes, they also need to configure some basic parameters, such as the port number for communicating with the controller 101 and how to receive worker node failover notifications.

[0139] The n working nodes 103 and the m hot standby nodes 104 share a common hardware resource base. This means that at the hardware level, the hot standby node can seamlessly take over the tasks of the working node when a working node fails, because the consistency of the hardware environment ensures the normal operation of the training task, and network resources are also shared. Both the working node and the hot standby node are connected to the same high-speed network environment to ensure rapid data transmission during training. In addition, the n working nodes 103 and the m hot standby nodes 104 run the same operating system and related software stack. This means that when the hot standby node replaces the working node, there is no need to reinstall or configure the software environment and it can quickly enter the working state.

[0140] Optionally, when deploying the training program corresponding to a training task, the controller 101 considers the overall availability of the n worker nodes 103 and the m hot standby nodes 104. Although the hot standby nodes do not directly participate in the initial training task execution, they are part of the overall distributed training system's task allocation architecture. For example, when assigning a subset of training data to a worker node, the amount of data allocated to each worker node is determined based on the total number of worker nodes and hot standby nodes to ensure load balancing across the entire training task while hot standby nodes are present.

[0141] During normal training, the task states of the n working nodes 103 are constantly changing. For example, as the training progresses, the model parameters of the working nodes will be continuously updated, and the amount of data processed by them will gradually increase. The task status information of the working nodes (such as the number of training rounds completed, the current model accuracy, etc.) will be monitored by the monitor 102 and fed back to the controller 101. The task status information of these working nodes is a very important reference for the hot standby node. Because the hot standby node needs to adjust its own state according to the status information of the working node, so that it can take over the task as seamlessly as possible when a fault is replaced. For example, if the working node has reached the later stage of training, the hot standby node needs to load the initial model parameters of the corresponding later stage to reduce the time for retraining.

[0142] The hot standby node is in a ready-to-wait state for the training program, but this state is not completely static. The hot standby node will perform some pre-operations based on the status changes of the working node. For example, when the model parameters of the working node are updated, the hot standby node may receive these update information (although it does not perform actual model training) so that when the working node needs to be replaced, it can start training with the latest model parameters more quickly. In addition, the task status of the hot standby node also includes its own health status monitoring. Although hot standby nodes do not perform training tasks, they also need to keep their own hardware and software environment healthy so that they can normally take over the tasks of the working node when needed.

[0143] The monitor 102 is introduced and explained below.

[0144] Monitor 102 is mainly responsible for monitoring the operating status of n working nodes 103 to ensure the smooth progress of the entire training process. In a distributed training system, everything from the operating status of the hardware to the execution of the software is within its monitoring range.

[0145] The health status of the n worker nodes 103 includes at least one of hardware health or software health. The hardware health includes, but is not limited to, at least one of the following: a healthy temperature of a graphics card, sufficient memory, a healthy CPU, or a healthy disk. The software health includes, but is not limited to, at least one of the following: correct operation of the training program, compatibility and synergy between the training program and other software components, or software stability.

[0146] The monitored content includes but is not limited to: at least one of the hardware status of the n working nodes 103 or the software status of the n working nodes 103 .

[0147] Optionally, for the graphics processor servers in the n working nodes 103, the monitor 102 will pay attention to the status of the graphics processor card. This includes key indicators such as the temperature, power consumption, and video memory usage of the graphics processor card. For example, during deep learning training, the graphics processor card will generate heat due to high-intensity computing tasks. If the temperature is too high, it may cause frequency reduction or even hardware damage, thereby affecting the normal progress of the training task. The monitor 102 will obtain the temperature data of the graphics processor card in real time, and when the temperature approaches or exceeds the preset safety threshold (such as 80°C), it will issue an alarm in time. In addition, the monitor 102 will also monitor the server's CPU usage, disk I / O and other hardware indicators. In large-scale distributed training, although the graphics processor undertakes the main computing tasks, the CPU also plays an important role in data preprocessing, model parameter management, etc. The speed of disk I / O will affect the reading of training data and the storage efficiency of model parameters.

[0148] Optionally, the monitor 102 monitors the training programs running on the n working nodes 103. The monitoring content includes but is not limited to at least one of whether the training program is started normally or whether there is a process crash. For example, during the training process, if the training program is unexpectedly terminated due to an error in the code (such as a division by zero error, a null pointer reference, etc.), the monitor 102 can quickly detect the abnormal exit of the process and send a corresponding notification to the controller 101. In addition, the monitor 102 also pays attention to the running progress of the training program. Including indicators such as the number of training rounds completed, the current model accuracy, and the loss function value. By monitoring the indicators, the progress of the training task can be understood and whether the training is proceeding as expected. If the model accuracy does not improve for a long time or the loss function value no longer decreases, it may mean that there is a problem with the training algorithm or the model has fallen into a local optimum and the training strategy needs to be adjusted.

[0149] Monitor 102 monitors the health status of n worker nodes 103 by, but is not limited to, monitoring the temperature of graphics processor cards by interacting with the graphics processor driver or related hardware management interfaces to obtain real-time temperature data from these sensors; or by monitoring various hardware components within the computer through integrated hardware monitoring tools; or by monitoring the process status of the training program through the operating system's process management mechanism. This application does not impose any restrictions on the monitoring method.

[0150] As an optional implementation, n working nodes 103 are configured to report their own resource usage and training status information to the monitor 102 .

[0151] Report resource usage and training status information to the monitor 102 in a timely manner, so that the monitor 102 can comprehensively and timely monitor the real-time status of the n working nodes 103, and can promptly discover and handle any failures.

[0152] Resource usage includes hardware resource usage and software resource usage, wherein the hardware resource usage includes but is not limited to at least one of resource utilization-related indicators, resource saturation-related indicators, or resource error-related indicators.

[0153] Among them, resource utilization-related indicators include but are not limited to: graphics processing unit (GPU) utilization, central processing unit (CPU) utilization, memory usage, disk usage, disk I / O performance, such as the number of read / write (I / O) operations per second (Input / Output Operations Per Second, I / OPS), etc., network card bandwidth utilization or computer bus interface standard (Peripheral Component Interconnect Express, PCIE) bandwidth utilization, etc. At least one of the following.

[0154] Resource saturation related indicators include but are not limited to: at least one of video memory saturation, memory saturation, or disk saturation.

[0155] Resource error related indicators include, but are not limited to: at least one of: network card abnormality, disk write (including remote storage) error, or system load (current number of queued processes).

[0156] Training status information refers to the various data and metrics used to describe the execution status of training tasks during distributed training. This information is crucial for monitoring the training process, detecting anomalies, optimizing performance, and recovering from failures.

[0157] Training status information includes but is not limited to: the time of each iteration of the training task, that is, the time spent on each iteration, for example: in the training of an image recognition model, if the time of each iteration increases from 10 seconds to 20 seconds, it means that the model complexity has increased or the computing resources are insufficient; or, the loss function value of the training task, that is, the difference between the model prediction result and the actual result, for example: in a regression task, if the loss function value continues to decrease, it means that the model is learning the pattern in the data. If the loss function value suddenly rises, it may indicate that the model is overfitting or there is a problem with the data; or, the log information generated during the training process, that is, various information recorded during the training process, including program running status, error information, warning information, etc.; or, detailed records of the function call order, data flow direction and other information during the execution of the task in the training task; or, the update status of the model parameters; or, the data processing status; or, the convergence status of the training task. This application does not limit the type of training status information.

[0158] The following describes the process in which the n working nodes 103 report their own resource usage and training status information to the monitor 102 .

[0159] The reporting method includes but is not limited to: reporting based on the Message Passing Interface (MPI); or reporting based on a network socket. This application does not impose any restrictions on the reporting method.

[0160] When n worker nodes 103 use the MPI communication mechanism to report resource usage and training status information to the monitor 102, each worker node can be regarded as an MPI process, and they can send and receive messages through functions such as MPI_Send and MPI_Recv. The worker nodes need to encapsulate the information to be reported. For hardware resource usage, information such as GPU core utilization, video memory usage, memory utilization, network bandwidth utilization, etc. can be combined into a structure according to a certain format. Then, this structure is sent to the monitor 102 as a message. For training status information, similar structures can also be used for encapsulation, such as a structure containing information such as the number of completed training rounds, model accuracy, and loss function value.

[0161] The n worker nodes 103 can create a socket connection to the server port where the monitor 102 is located, and then send resource usage and training status information through this connection. When reporting information through the socket, determine the appropriate data format.

[0162] As an optional implementation, the monitor 102 is configured to determine the first working node to be switched due to a failure based on resource usage and training status information of the n working nodes 103 .

[0163] In the distributed training system, n working nodes 103 are the main bodies that actually execute the training tasks. Each working node undertakes a part of the training subtasks, and these subtasks together constitute the entire large-scale training task. The first working node is one of the working nodes. Under normal circumstances, it works in coordination with other working nodes. For example, in the distributed training of neural networks, different working nodes are responsible for processing different batches of data or calculating different parts of the neural network. The first working node also participates in this collaborative process of data processing and calculation, and exchanges intermediate results (such as gradient information) with other nodes to gradually optimize the entire neural network model.

[0164] Due to various factors, including software and hardware, worker nodes may fail during the training process. The first worker node is the compute node detected as failing among the n worker nodes 103. Once the first worker node fails, it will be unable to continue executing the training subtask it is responsible for, which will affect the entire training task. Because the training subtasks of each worker node are interrelated, the failure of one node may lead to inaccurate overall training results or interrupt the training process.

[0165] The monitor 102 is responsible for monitoring the health status of n working nodes 103, and the first working node is one of the monitored objects. The monitor 102 continuously collects the resource usage and training status information reported by the first working node. By analyzing this information, the monitor 102 can determine whether the first working node is operating normally. For example, if the GPU usage of the first working node suddenly drops to 0, while other nodes are normal, this may be a signal that the GPU of the first working node has failed; or if the loss function value of the training task of the first working node fluctuates abnormally (such as suddenly increasing a lot), it may also indicate that there is a problem with the first working node.

[0166] As an optional implementation, the monitor 102 is configured to mark the first working node as faulty.

[0167] Monitor 102 marks the selected first working node as faulty to clearly indicate to other components in the distributed training system (e.g., controller 101 and other working nodes) which node has experienced a problem. Monitor 102 marks this node as faulty, and controller 101 can then take appropriate measures based on this fault flag, such as migrating tasks from this node to another node, thereby ensuring continued stable operation of the entire training system.

[0168] As an optional implementation, the monitor 102 is configured to select a first hot standby node from the m hot standby nodes 104 when a first working node among the n working nodes 103 fails.

[0169] When the monitor 102 detects that the first working node has failed, it needs to promptly select the first hot standby node from the m hot standby nodes 104 to replace the next training task of the first working node to ensure a smooth training process.

[0170] The following further describes the manner in which the monitor 102 selects the first hot standby node from the m hot standby nodes 104 .

[0171] As an optional implementation, the monitor 102 is used to select a first hot standby node from the m hot standby nodes 104 based on selection factors when a first working node among the n working nodes 103 fails; the selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes 104; the communication bandwidth between the first working node and the m hot standby nodes 104; the communication speed between the third working node and the m hot standby nodes 104; wherein the third working node is a working node that has a collaboration requirement with the first working node.

[0172] The first hot standby node is determined based on multiple selection factors, which enables a more comprehensive and smooth replacement of the first working node by the first hot standby node, thereby greatly reducing the time required to replace a faulty node.

[0173] The communication speed between the first working node and the m hot standby nodes 104 refers to the speed at which data is transmitted between the first working node and the m hot standby nodes 104. It can be measured by the number of bytes transmitted per second or the number of packets transmitted per second. In a distributed training system, data is continuously transmitted between nodes, for example, to update model parameters and share intermediate calculation results. The communication speed reflects the actual efficiency of data transmission from the first working node to the m hot standby nodes 104.

[0174] The communication bandwidth between the first working node and the m hot standby nodes 104 refers to the capacity of data that can be transmitted between the first working node and the m hot standby nodes 104 in a unit time. The larger the communication bandwidth, the more data can be transmitted simultaneously.

[0175] The third working node is a working node that has a collaborative need with the first working node. In distributed training, there is a complex collaborative relationship between the various working nodes. The first working node may be responsible for a certain part of the computing task, while the third working node may generate data related to the first working node or rely on the computing results of the first working node. For example, in the distributed training of an image recognition model, the first working node may be responsible for processing the feature extraction of a certain area of ​​the image, while the third working node may be responsible for integrating these extracted features or performing further classification calculations based on the results of the first working node. Since the third working node has a collaborative need with the first working node, when the first working node fails and needs to switch to the hot standby node, the communication relationship between the third working node and the hot standby node becomes important. If the communication speed between the third working node and the hot standby node is slow or the communication bandwidth is small, it will affect the efficiency of the entire collaborative training task.

[0176] The communication speed between the third working node and the m hot standby nodes 104 refers to the speed at which data is transmitted between the third working node and the m hot standby nodes 104. The data transmission speed between the third working node and the hot standby nodes in the distributed training system also affects the performance of the entire system.

[0177] When the first working node fails, the monitor 102 evaluates the communication speed between the first working node and the m hot standby nodes 104. If the communication speed between a hot standby node and the first working node is faster, this means that during failover, the data related to the first working node (such as the intermediate results of the training subtasks, model parameters, etc.) can be quickly transmitted to the hot standby node. For example, in a large-scale speech recognition model training, some acoustic model parameters of the first working node need to be quickly transmitted to the hot standby node in order to continue training. If the communication speed between the hot standby node A and the first working node is 80 bits per second (Megabits per second, Mbps), and the communication speed of the hot standby node B is 20Mbps, under the same conditions, the hot standby node A is more likely to be selected as the first hot standby node.

[0178] Monitor 102 also considers the communication speed between the third working node and the m hot standby nodes 104. Because the third working node is closely connected to the first working node during collaborative training, if the communication speed between the third working node and a hot standby node is fast, then the hot standby node has an advantage in selection. For example, in the distributed training of a recommendation system, the first working node is responsible for processing part of the user behavior data, and the third working node is responsible for matching product features based on the results of the first working node. If the first working node fails and switches to the hot standby node, the fast communication between the third working node and the hot standby node can ensure the continuity of the entire recommendation system training task.

[0179] In addition, the monitor 102 checks the communication bandwidth between the first working node and the m hot standby nodes 104. A larger communication bandwidth means that large amounts of data can be transmitted faster during a failover. For example, in the distributed training of a video analysis model, the first working node generates a large amount of video frame feature data, which needs to be transmitted to the hot standby node. If the communication bandwidth between the hot standby node C and the first working node is 1 Gigabit per second (Gbps), and the communication bandwidth of the hot standby node D is 500 Mbps, the hot standby node C is more likely to be selected when a large amount of data needs to be transmitted.

[0180] Optionally, the distributed training system includes a shared storage location. The shared storage location is a storage area established in the distributed training system to enable data exchange and sharing among the n worker nodes 103 during the training process. The shared storage location is a common data warehouse. Each worker node can send its own training-related data (such as the context information described below, global training parameters, and some intermediate results of the model) to this location and can also obtain data from other worker nodes or shared global information from this location.

[0181] Optionally, the first hot standby node is determined based on the network distance. The network distance can be measured in a variety of ways, for example, based on the number of network hops from the hot standby node to the shared storage location. The number of network hops refers to the number of intermediate devices (such as routers, switches, etc.) that data passes through when it is transmitted from one node to another in the network. Generally speaking, the fewer the number of network hops, the lower the delay in data transmission, and the faster the hot standby node obtains context information. Another way to measure network distance is based on the measurement of network delay. The network distance is determined by sending a probe packet from the hot standby node to the shared storage location and measuring the round-trip time.

[0182] Optionally, the first hot standby node is determined based on physical distance and network topology. The hot standby node and the shared storage location may be located in different physical cabinets or server clusters. If the physical distance is close, such as within the same cabinet, the physical link for data transmission is shorter, and the signal transmission delay is relatively small. Furthermore, the network topology can also affect the data transmission path. For example, in a tree-like network topology, a hot standby node located near the root node and close to the shared storage location may have a shorter data transmission path, enabling faster acquisition of context information.

[0183] Optionally, monitor 102 may comprehensively consider communication speed and bandwidth when selecting the first hot standby node from among m hot standby nodes 104 based on selection factors. Sometimes, while one hot standby node has a fast communication speed, its communication bandwidth is small, potentially creating a bottleneck when transmitting large amounts of data. Meanwhile, another hot standby node has a larger communication bandwidth but a relatively slow communication speed, making it inefficient when transmitting small amounts of data. Monitor 102 needs to balance the importance of communication speed and bandwidth based on the data transmission requirements between the first and third working nodes and the hot standby node to select the most appropriate first hot standby node.

[0184] After the first hot standby node is determined among the m hot standby nodes 104 , the controller 101 is further described below.

[0185] When the monitor 102 detects that the first working node among the n working nodes 103 has failed, the controller 101 will start the fault response mechanism. It is responsible for coordinating the switching of the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes 104 for execution. This process requires the controller 101 to conduct effective information interaction with the monitor 102, the working node and the hot standby node to ensure the smooth progress of the switching process. When the first working node fails, the controller 101 needs to manage the working node, such as isolating the faulty first working node to prevent it from causing adverse effects on other working nodes and the entire training task. At the same time, the controller 101 also needs to ensure that other working nodes can work normally with the first hot standby node that is about to be switched.

[0186] The controller 101 switches the training subtask corresponding to the first working node to the first hot standby node, including: sending a startup instruction to the first hot standby node, and transmitting necessary information related to the training subtask, such as current training parameters, model weights, etc.

[0187] As an optional implementation, the controller 101 is used to set the working role of the first hot standby node to the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0188] The hot standby node has the same resources, software, and hardware environment as the active node, but does not start any training-related processes. It is in a ready-to-run state for the training program. This means the hot standby node has all the necessary conditions to execute training tasks, such as having the required software environment installed and the necessary hardware resources configured. However, under normal circumstances, it does not participate in the actual training calculations. If the primary active node fails, the first hot standby node's role will be reversed. The first hot standby node will transition from a ready-to-run state to become the second active node that actually executes the training tasks.

[0189] The second worker node is relative to the first worker node. When the first worker node fails, the first hot standby node will be set as the second worker node to take over the first worker node's tasks. This name is used to clearly describe the role transition during failure handling, that is, the node originally serving as a hot standby node will be transformed into a worker node responsible for training subtasks.

[0190] When the controller 101 sets the working role of the first hot standby node to the second working node, it must first make configuration adjustments. This includes modifying the working role identifier in the system. For example, in the node management database of the distributed training system, the role identifier of the first hot standby node is updated from "hot standby node" to "second working node." In addition, the controller 101 also needs to adjust the network configuration to ensure that the second working node (formerly the first hot standby node) can communicate normally with other working nodes.

[0191] As an optional embodiment, the controller 101 is configured to send context information of the training subtask corresponding to the first working node to the second working node. The second working node is configured to re-execute or continue to execute the training subtask corresponding to the first working node through the training program based on the context information.

[0192] When the first working node fails, the controller 101 sends the context information of its training subtask to the second working node. The second working node can accurately re-execute or continue the training subtask corresponding to the first working node, maintaining the consistency of the training task and avoiding confusion or interruption of the training process due to node failure. Because the second working node can continue the task based on the context information of the first working node, it does not need to start the entire training subtask from scratch, significantly improving overall training efficiency.

[0193] Controller 101 needs to notify the second worker node (formerly the first hot standby node) that its role has changed. This notification includes necessary instructions and information. For example, controller 101 will notify the second worker node to begin preparing to execute the training subtask corresponding to the first worker node and provide it with context information about the first worker node so that the second worker node can start working from the appropriate state.

[0194] Context information is a set of critical information related to training subtasks in a distributed training system. This information is crucial for switching training subtasks from the first worker node to the second worker node and ensuring their correct execution when a failure occurs. Context information provides the second worker node with a comprehensive description of the training task on the first worker node, enabling it to accurately re-execute or continue the task.

[0195] As an optional implementation, the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0196] The task role of a first worker node refers to the function or task type it performs within the entire training task. For example, in the distributed training of a deep learning model, there may be different task roles, such as worker nodes responsible for data preprocessing, worker nodes responsible for computing certain layers of the model, and worker nodes responsible for model evaluation. The task role of a first worker node determines its specific operations within the training process. For example, it may be a node responsible for normalizing input image data. The task role of a first worker node also defines its relationship with other nodes. Workers with different task roles may have data dependencies, communication relationships, and other relationships.

[0197] When a first worker node fails and needs to switch its training subtask to a second worker node, the first worker node's task role information guides the second worker node. The second worker node adjusts its operations based on the first worker node's task role to ensure it can correctly take over the first worker node's work. For example, if the first worker node is responsible for data distribution, the second worker node needs to understand the distribution rules, the source and destination of the data, and other information before it can continue the data distribution task.

[0198] The ID of the first worker node in a training task identifies that node within the task. For example, in a cluster containing 100 worker nodes, only 20 worker nodes are used for a particular training task. Each of these 20 worker nodes has an ID that identifies the first worker node within the task. The ID of the first worker node is used to distinguish it from the other worker nodes participating in the task.

[0199] When switching the training subtask of the first working node to the second working node, the identifier of the first working node helps to locate and match related resources and operations. In a fault handling scenario, the identifier of the first working node can accurately refer to the node where the fault occurs. When the training subtask of the first working node needs to be transferred to other nodes (such as the second working node), the identifier ensures the accurate transmission of context information. For example, the master node of the training task can find the context information related to the identifier from the storage system based on the identifier of the first working node, and send it to the second working node so that the second working node can correctly continue or re-execute the task of the first working node.

[0200] The IP address of the master node for a training task is a unique identifier for the master node within the network and is used to locate the master node within the network environment. In a distributed training system, the master node plays a crucial role in coordinating and managing the entire training task. Other worker nodes and hot standby nodes communicate with the master node through the master node's IP address. If the primary worker node fails and its training subtask is switched to a secondary worker node, the secondary worker node must reestablish communication with the master node. The master node's IP address provides a clear communication target for the secondary worker node.

[0201] The port address of the training task's master node further identifies the address of a service or application based on the master node's IP address. Multiple services or applications may be running on a single master node, and the port address is used to distinguish between these services. For example, if a data distribution service and a model parameter synchronization service are running simultaneously on the master node, different port addresses will be assigned to these different services.

[0202] If the first worker node fails and the task is switched to the second worker node, the second worker node needs to connect to the target service of the master node to continue executing the training subtask. The port address of the master node provides the second worker node with a way to connect to the target service. For example, if the first worker node was interacting with the data distribution service of the master node before the failure, the second worker node needs to connect to this service using the same port address to obtain the correct data for training.

[0203] The address of the training dataset for a training task refers to the specific location of the training data in the storage system. The address can be a path in the local file system or the target storage location in a distributed storage system.

[0204] If the first worker node fails and its training subtask is switched to a second worker node, the training dataset address ensures that the second worker node can continue to obtain the required training data. For example, if the first worker node had already processed a portion of the training dataset before the failure, the second worker node needs to use the training dataset address to find the remaining unprocessed data to continue training.

[0205] The storage address of the last checkpoint of a training task is the record of the training state at a certain point in the training process. Checkpoints are set to allow training to be resumed from the last recorded state if training is interrupted. This storage address can be a folder on a local disk or a target location in a distributed storage system. For example, in a deep learning training task, a checkpoint is created after each training round. The storage address of the last checkpoint records the storage location of information such as model parameters and optimizer status at that time.

[0206] For the second working node, the storage address of the last checkpoint is the starting point for its state recovery. Based on this address, the second working node obtains the training state of the first working node before the failure, including information such as the amount of data processed and the current parameters of the model. This enables the second working node to continue executing training subtasks on the same basis, maintaining the consistency and efficiency of training. For example, in a distributed training task for speech recognition, if the first working node fails after 50 rounds of training, the second working node can obtain the model state after these 50 rounds from the storage address of the last checkpoint and continue training for subsequent rounds.

[0207] The model configuration of the AI ​​model trained by the training task includes a description of the model structure, including information such as the number of layers, the number of neurons in each layer, the type of activation function, etc. For example, in a convolutional neural network model, the model configuration describes the number of convolutional layers, the size of the convolution kernel, the location and type of the pooling layer, etc.

[0208] If the first worker node fails and its training subtask is switched to a second worker node, the model configuration is the basis for the second worker node to rebuild the model and continue training. The second worker node must build the same model according to the model structure described in the model configuration and continue training according to the training parameters set in the model configuration.

[0209] Re-executing the training subtask means that after receiving the context information of the training subtask from the first worker node, the second worker node starts the training operation from the initial state of the training subtask, ignoring some of the results of the operations that may have been performed by the first worker node.

[0210] The situations in which the training subtask is re-executed include but are not limited to: there is an integrity problem with the data related to the training subtask of the first working node; or the type of failure occurring in the first working node makes it difficult to accurately restore the status of its training subtask.

[0211] Continuing to execute the training subtask means that after a failure occurs in the first working node, the second working node continues the training operation from the state when the failure occurs based on the partial training results that have been completed by the first working node.

[0212] The situations in which the training subtask can be continued include but are not limited to: the first working node completely saves the status of its training subtask before the failure occurs; or, for certain recoverable failure types, such as temporary errors in the software or brief network interruptions, as long as the training subtask status of the first working node is not severely damaged.

[0213] As an optional implementation, the second working node also includes: a training process control unit; a training process control unit, used to start the training process corresponding to the training program; and re-execute or continue to execute the training subtask corresponding to the first working node based on context information through the training process.

[0214] The training process corresponding to the training program can be quickly started, improving effective training time. The training process control unit can re-execute or continue the training subtask corresponding to the first working node based on context information. This feature enables training tasks to be quickly resumed after a failed node is replaced, avoiding the need to start training from scratch. This continuity is particularly important for distributed training, ensuring training accuracy and effectiveness.

[0215] The training process control unit is a functional unit used to manage and operate the training process. It is responsible for scheduling, starting, pausing, and resuming processes related to the training task. The training process control unit contains multiple sub-modules that work together to effectively control the training process. These include, but are not limited to: a process startup module, which is responsible for initializing the training process; a process scheduling module, which reasonably arranges the execution order of the training process based on factors such as system resource conditions (such as CPU and GPU usage) and the priority of the training task; and a process monitoring module, which monitors the running status of the training process in real time, such as whether there are memory leaks and other problems.

[0216] The training process control unit interacts closely with the training program. The training program is the code that actually performs the training tasks, while the training process control unit manages the training program's execution. The training process control unit is responsible for loading the training program into memory and starting the corresponding training process.

[0217] When the first working node fails, in order to ensure the consistency of the entire distributed training system and the accuracy of the data, all working nodes are suspended. The n working nodes 103 (especially the second working node) begin to wait for receiving a signal to restart or continue execution. At the same time, the controller 101 will prepare the context information of the training subtask corresponding to the first working node. When the second working node receives the signal to restart or continue execution, the training process control unit inside it first starts the training process corresponding to the training program. Once the training process is started, the training process control unit will re-execute or continue to execute the training subtask corresponding to the first working node based on the received context information.

[0218] While the second worker node re-executes or continues to execute the training subtask corresponding to the first worker node based on the context information through the training process, the other worker nodes remain in a suspended waiting state. Once the second worker node completes the initialization or recovery operation for the first worker node's task and reaches a state where it can resume working in coordination with other worker nodes, it sends a ready signal to the controller 101.

[0219] After receiving the signal from the second working node, the controller 101 sends a signal to all working nodes to restart the collaborative training. At this point, all working nodes (including the second working node) will continue to perform large-scale distributed training tasks in the same collaborative manner as before, thus achieving efficient recovery in the event of a failure, reducing the time required to replace the failed node and increasing the effective training time.

[0220] As an optional implementation, the controller 101 is also used to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training subtask. The restart training task message is used to trigger each working node to re-coordinate and execute the various training subtasks of the training task.

[0221] The Restart Training Task message is a directive message sent by the controller 101 to coordinate the n worker nodes 103 in the distributed training system, causing them to resume collaborative execution of the training subtasks. This message prompts each worker node to stop its current waiting or temporary state and restart the training task according to predefined rules, ensuring state consistency and training continuity across the entire distributed training system.

[0222] The restart training task message contains multiple information elements. The first is the identification information of the training task, which is used to clarify which training task needs to be restarted. It can be a unique task identifier or task name; the second is the global configuration information related to the training task, such as the overall goal of the training task (for example, the accuracy and recall rate to be achieved), the basic description of the data set used by the training task (such as the size of the data set, the data type, etc.); in addition, it may also include some timestamp information to record the time when the restart training task message was sent, so that the n working nodes 103 can judge the timeliness of the message and coordinate their own operation sequence.

[0223] When the second worker node begins executing the training subtask from the first worker node, the state of the entire distributed training system changes. To enable other worker nodes to adapt to this change and continue to work together effectively, the restart training task message acts as a synchronization signal, prompting all worker nodes to readjust their state and operational processes.

[0224] The following describes how to send a restart training task message to each working node corresponding to the training task.

[0225] When sending a restart training task message to each working node based on network communication, it is first necessary to select an appropriate network communication protocol; then determine the specific communication method. One method is for the controller 101 to directly establish a one-to-one communication connection with each working node, and then send the restart training task message one by one. Another method is to use a broadcast or multicast communication method. Broadcast is to send the message to all nodes in the network, while multicast is to send the message to a predefined group of nodes.

[0226] Before sending the restart training task message, the controller 101 needs to detect the current status of the n worker nodes 103. The worker nodes may be in different states, such as processing data, idle waiting, or performing fault recovery. Depending on the different states of the nodes, the controller 101 can adopt different sending strategies. To ensure that the restart training task message can be sent to the n worker nodes 103 accurately and orderly, the controller 101 can adopt a message queue management method.

[0227] When the worker node receives the message, it sends a confirmation message to the controller 101. The confirmation message contains the worker node's identification information so that the controller 101 can identify which worker node sent the confirmation. After receiving the confirmation message, the controller 101 knows that the worker node has received the restart training task message and can continue with subsequent operations.

[0228] The hot standby node mechanism in this application greatly reduces the time it takes to replace a faulty node. Since the hot standby node has the same resources, software and hardware environment as the working node and is in a ready-to-wait state for the training program, once the monitor detects a fault in the working node (such as the first working node), the controller can quickly switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution, avoiding long waits for other nodes, thereby effectively improving the fault handling speed and reducing the impact of the fault on the training task; in addition, the hot standby node is in a ready-to-wait state in the system. Although the training process is not started, it can take over the task of the faulty node at any time, making more efficient use of system resources; since the hot standby node can quickly take over the task of the faulty node, the working node can focus more on the training task during normal operation without reserving too many resources or processing power for possible faults; providing a more stable environment for distributed training.

[0229] The present invention provides a distributed training method for Figure 1The distributed training system in the distributed training system includes a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers. Figure 2 As shown, the distributed training method provided in the embodiment of the present application may include the following steps 201 to 204.

[0230] In step 201, the controller deploys the training program corresponding to the training task to n working nodes and m hot standby nodes.

[0231] In step 202, n working nodes collaborate to perform a training task.

[0232] In step 203, the monitor monitors the health status of n working nodes.

[0233] In step 204, when a first working node among the n working nodes fails, the controller switches the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution.

[0234] The hot standby node in this application has the same resources and environment as the working node and is in a ready waiting state. Once the working node (the first working node) fails, the controller can quickly switch its training subtask to the hot standby node (the first hot standby node) for execution, which greatly reduces the time to replace the failed node. Other nodes do not need to wait for a long time, so that the training task can be quickly restored, effectively improving the speed of fault recovery.

[0235] See also Figure 3 The figure shows an overall flow chart of a distributed training method, which is applied to a distributed training system. The distributed training system includes a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers.

[0236] Step 1: The controller deploys the training program corresponding to the training task to multiple computing nodes; and divides the multiple computing nodes into n working nodes and m hot standby nodes.

[0237] Step 2: n worker nodes collaborate to perform training tasks.

[0238] Step 3: n worker nodes report their resource usage and training status information to the monitor.

[0239] Step 4: The monitor determines the first working node to be switched due to a fault based on the resource usage and training status information of the n working nodes.

[0240] Step 5: When a first working node among the n working nodes fails, the monitor selects a first hot standby node from the m hot standby nodes based on a selection factor.

[0241] Optionally, the selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes; the communication bandwidth between the first working node and the m hot standby nodes; the communication speed between the third working node and the m hot standby nodes; the communication speed between the third working node and the m hot standby nodes; wherein the third working node is a working node that has a collaboration requirement with the first working node.

[0242] Step 6: When the first working node among the n working nodes fails, the controller sets the working role of the first hot standby node to the second working node, and switches the training subtask corresponding to the first working node to the second working node for execution.

[0243] Step 7: The controller sends the context information of the training subtask corresponding to the first working node to the second working node.

[0244] Optionally, the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0245] Step 8: The controller sends a message to restart the training task to each working node corresponding to the training task.

[0246] The restart training task message is used to trigger each working node to re-coordinate and execute each training subtask of the training task.

[0247] Step 9: The training process control unit starts the training process corresponding to the training program; and re-executes or continues to execute the training subtask corresponding to the first working node based on the context information through the training process.

[0248] The second working node also includes: a training process control unit.

[0249] Step 10: The controller adjusts the number of m hot standby nodes based on the number of failure switches in the most recent time period.

[0250] Optionally, the number of fault swaps is the number of swaps between the working node and the hot standby node caused by the fault.

[0251] See also Figure 4 An embodiment of the present application provides a controller for use in a distributed training system. The distributed training system includes a controller, a monitor, n working nodes, and m hot standby nodes, where n and m are positive integers. The controller includes:

[0252] Deployment module 401, used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes;

[0253] The switching module 402 is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

[0254] In one possible implementation, the switching module 402 is used to set the working role of the first hot standby node to the second working node when a first working node among n working nodes fails, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0255] In one possible implementation, the deployment module 401 is also used to send the context information of the training subtask corresponding to the first working node to the second working node, and the second working node is used to re-execute or continue to execute the training subtask corresponding to the first working node through the training program based on the context information.

[0256] In one possible implementation, the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0257] In one possible implementation, the switching module 402 is also used to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training subtask. The restart training task message is used to trigger each working node to re-coordinate and execute the various training subtasks of the training task.

[0258] In one possible implementation, the deployment module 401 is used to create a training task and deploy the training program corresponding to the training task to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, and the m hot standby nodes are in a ready waiting state; set the startup command and startup parameters of the training task; and coordinate the n working nodes to start training based on the startup command and startup parameters.

[0259] In a possible implementation, the deployment module 401 is further configured to adjust the number of m hot standby nodes based on the number of fault swaps in a recent time period; wherein the number of fault swaps is the number of swaps between the working node and the hot standby node due to a fault.

[0260] In one possible implementation, the n working nodes and the m hot standby nodes have the same software environment, hardware environment, and running configuration files of the training program.

[0261] It should be understood that the controller provided above is merely an example of the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the controller provided in the above embodiment and the method embodiment are based on the same concept. The specific implementation process and the beneficial effects brought about are detailed in the method embodiment and will not be repeated here.

[0262] See also Figure 5 , an embodiment of the present application provides a distributed training system, the system comprising: a controller 501 and a monitor 502;

[0263] The controller 501 is configured to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes, where n and m are positive integers;

[0264] Monitor 502, used to monitor the health status of n working nodes;

[0265] The controller 501 is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

[0266] In a possible implementation, the monitor 502 is configured to select a first hot standby node from among the m hot standby nodes when a first working node among the n working nodes fails;

[0267] The controller 501 is used to set the working role of the first hot standby node to the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

[0268] In a possible implementation, the controller 501 is configured to send context information of the training subtask corresponding to the first working node to the second working node.

[0269] In one possible implementation, the context information includes at least one of the following: the task role of the first working node in the training task; the identifier of the first working node in the training task; the Internet Protocol IP address of the master node of the training task; the port address of the master node of the training task; the address of the training data set of the training task; the storage address of the last checkpoint of the training task; and the model configuration of the AI ​​model trained by the training task.

[0270] In one possible implementation, the controller 501 is also used to send a restart training task message to each working node corresponding to the training task before the second working node starts executing the training subtask. The restart training task message is used to trigger each working node to re-coordinate and execute the various training subtasks of the training task.

[0271] In one possible implementation, the monitor 502 is used to select a first hot standby node from m hot standby nodes based on selection factors when a first working node among n working nodes fails; the selection factors include at least one of the following: the communication speed between the first working node and the m hot standby nodes; the communication bandwidth between the first working node and the m hot standby nodes; the communication speed between the third working node and the m hot standby nodes; wherein the third working node is a working node that has a collaboration requirement with the first working node.

[0272] In a possible implementation, the monitor 502 is configured to determine a first working node to be switched due to a fault based on resource usage and training status information of the n working nodes.

[0273] In one possible implementation, the controller 501 is used to create a training task and deploy the training program corresponding to the training task to multiple computing nodes; divide the multiple computing nodes into n working nodes and m hot standby nodes, and the m hot standby nodes are in a ready waiting state; set the start command and start parameters of the training task; and coordinate the n working nodes to start training based on the start command and start parameters.

[0274] In a possible implementation, the controller 501 is configured to adjust the number of m hot standby nodes based on the number of fault swaps in a recent time period; wherein the number of fault swaps is the number of swaps between the working node and the hot standby node due to a fault.

[0275] In one possible implementation, the n working nodes and the m hot standby nodes have the same software environment, hardware environment, and running configuration files of the training program.

[0276] It should be understood that the distributed training system provided above is merely an example of the division of the aforementioned functional modules when implementing its functions. In actual applications, the aforementioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the distributed training system provided in the above embodiment and the method embodiment are based on the same concept. The specific implementation process and the beneficial effects brought about are detailed in the method embodiment and will not be repeated here.

[0277] In an exemplary embodiment, a computer device is also provided. The computer device includes a processor and a memory, wherein the memory stores at least one computer program. The at least one computer program is loaded and executed by one or more processors, so that the computer device implements any of the above-described distributed training methods. The computer device can be a server or a terminal device. The following describes the structures of the server and terminal devices respectively.

[0278] Figure 6 This is a structural diagram of a server provided in an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 601 and one or more memories 602, wherein at least one computer program is stored in the one or more memories 602, and the at least one computer program is loaded and executed by the one or more processors 601 to enable the server to implement the distributed training method provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described here.

[0279] Figure 7 This is a schematic diagram of the structure of a computing node virtualization system provided by an embodiment of the present application; the computing node virtualization system can be used in various networks, such as a data center network, an operator network or a local area network. Figure 7 As shown, the computing node virtualization system includes: a computing node infrastructure layer 701, n computing nodes 702, and a distributed training system 703.

[0280] Computing node infrastructure layer 701 includes a hardware resource layer 704, a virtualization layer 705, and a virtual resource layer 706. Computing node infrastructure layer 701 utilizes hardware resources or software resources to implement a virtualized environment deployment. Specifically, hardware resource layer 704 and virtual resource layer 706 are used to provide virtualized resources, such as virtual machines and other forms of virtual containers, for computing node virtualization. Hardware resource layer 704 includes computing hardware, storage hardware, and network hardware. Computing hardware provides processing and computing resources. Storage hardware can be storage capacity provided within the network or residing within the storage hardware itself. Network hardware can be switches, routers, or any network device configured with switching capabilities.

[0281] The virtualization layer 705 abstracts hardware resources from the physical layer and decouples compute nodes, providing virtualized resources to the compute nodes. The virtual resource layer 706 includes virtual computing, virtual storage, and virtual networking. Virtual computing and virtual storage can be provided to compute nodes in the form of virtual machines or other virtual containers. For example, multiple compute nodes can be deployed on a single virtual machine. The virtualization layer 705 abstracts network hardware to form a virtual network. The virtual network can include a virtual switch to provide connectivity between multiple virtual machines.

[0282] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable a computer to implement any of the above-mentioned distributed training methods.

[0283] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0284] In an exemplary embodiment, a computer program product or a computer program is also provided. The computer program product or the computer program includes computer instructions, which are loaded and executed by a processor to enable a computer to implement any of the above-mentioned distributed training methods.

[0285] Optionally, the information involved in this application (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the resource usage and training status information of the working nodes involved in this application are obtained with full authorization.

[0286] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0287] The above description is merely an exemplary embodiment of the present application and does not limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A distributed training system for an artificial intelligence model, characterized in that: The system comprises: a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers; The controller is used to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes; The n working nodes are used to collaboratively execute the training task; The monitor is used to monitor the health status of the n working nodes; The controller is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

2. The system according to claim 1, characterized in that The monitor is used for selecting the first hot standby node from the m hot standby nodes when a first working node among the n working nodes fails; The controller is used to set the working role of the first hot standby node to the second working node, and switch the training subtask corresponding to the first working node to the second working node for execution.

3. The system according to claim 2, characterized in that The controller is used to send the context information of the training subtask corresponding to the first working node to the second working node; The second working node is used to re-execute or continue to execute the training subtask corresponding to the first working node through the training program based on the context information.

4. The system according to claim 3, characterized in that The context information includes at least one of the following: The task role of the first working node in the training task; An identifier of the first working node in the training task; The Internet Protocol IP address of the master node of the training task; The port address of the master node of the training task; The address of the training data set of the training task; The storage address of the last checkpoint of the training task; The model configuration of the AI ​​model trained by the training task.

5. The system according to claim 3, characterized in that The second working node also includes: a training process control unit; The training process control unit is used to start the training process corresponding to the training program; and re-execute or continue to execute the training subtask corresponding to the first working node based on the context information through the training process.

6. The system according to any one of claims 2 to 5, characterized in that: The controller is also used to send a restart training task message to each working node corresponding to the training task before the second working node starts to execute the training subtask, and the restart training task message is used to trigger the each working node to re-coordinate and execute the each training subtask of the training task.

7. The system according to any one of claims 2 to 5, characterized in that: The monitor is used to select the first hot standby node from the m hot standby nodes based on a selection factor when a first working node among the n working nodes fails; the selection factor includes at least one of the following: The communication speed between the first working node and the m hot standby nodes; The communication bandwidth between the first working node and the m hot standby nodes; The communication speed between the third working node and the m hot standby nodes; The third working node is a working node that has a collaboration requirement with the first working node.

8. The system according to any one of claims 1 to 7, characterized in that: The n working nodes are used to report their own resource usage and training status information to the monitor; The monitor is used to determine the first working node that needs to be switched due to a fault based on the resource usage and training status information of the n working nodes.

9. The system according to any one of claims 1 to 8, characterized in that: The controller is used to create a training task and deploy the training program corresponding to the training task to multiple computing nodes; Dividing the multiple computing nodes into the n working nodes and the m hot standby nodes, wherein the m hot standby nodes are in a ready waiting state; Setting the start command and start parameters of the training task; Coordinate the n working nodes to start training based on the startup command and the startup parameters.

10. The system according to claim 9, characterized in that The controller is used to adjust the number of the m hot standby nodes based on the number of failure exchanges in a recent time period; The number of fault exchanges is the number of exchanges between the working node and the hot standby node caused by the fault.

11. The system according to any one of claims 1 to 10, characterized in that: The n working nodes and the m hot standby nodes are identical in software environment, hardware environment and running configuration files of the training program.

12. A distributed training method, characterized in that: Applied to a distributed training system, the distributed training system includes a controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers, and the method includes: The controller deploys the training program corresponding to the training task to the n working nodes and the m hot standby nodes; The n working nodes collaboratively execute the training task; The monitor monitors the health status of the n working nodes; When a first working node among the n working nodes fails, the controller switches the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution.

13. A controller, characterized in that: Applied to a distributed training system, the distributed training system includes the controller, a monitor, n working nodes and m hot standby nodes, where n and m are positive integers, and the controller includes: A deployment module, used to deploy the training program corresponding to the training task to the n working nodes and the m hot standby nodes; A switching module is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

14. A distributed training system, characterized in that: The system includes: a controller and a monitor; The controller is used to deploy the training program corresponding to the training task to n working nodes and m hot standby nodes, where n and m are positive integers; The monitor is used to monitor the health status of the n working nodes; The controller is used to switch the training subtask corresponding to the first working node to the first hot standby node among the m hot standby nodes for execution when a first working node among the n working nodes fails.

Images