Model training system, distributed model training method and electronic equipment

Abstract

The invention provides a model training system, a distributed training method of a model and electronic equipment, the system comprises a control node and a plurality of working nodes, the control node is used for selecting at least one first execution node from the plurality of working nodes and instructing to start to execute a training task; the first execution node is used for executing a training task and storing training process data of the first execution node in a memory of the storage server and the first execution node in a training process; the control node is also used for controlling the at least one first execution node to stop training when the first execution node is abnormal in training, selecting at least one second execution node from the plurality of working nodes, and instructing to continue to execute the training task; and the second execution node is used for acquiring the training process data of the second execution node from the storage server or the memory of the second execution node, and continuing to execute the training task based on the training process data. By applying the scheme, the model training efficiency can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and in particular, to a model training system, a distributed training method of a model and an electronic device. BACKGROUND

[0002] In recent years, with the rapid development of the field of artificial intelligence, artificial intelligence (AI) models have attracted much attention. Due to the increasing size of training data sets and model parameters for training artificial intelligence models or the complexity of artificial intelligence models themselves, the training efficiency of artificial intelligence models is relatively low. Therefore, using a computing cluster with multiple nodes to perform distributed training of artificial intelligence models can effectively meet the computing needs in the training process of artificial intelligence models.

[0003] However, when training an artificial intelligence model, it takes time for a node to obtain a training data set, and the communication overhead between multiple nodes in a cluster is also relatively large. In particular, when a node training is abnormal, training tasks and data scheduling need to be performed between multiple nodes, resulting in consumption of more network and storage bandwidth resources. SUMMARY

[0004] Embodiments of the present application provide a model training system, a distributed training method of a model and an electronic device, which can reduce the communication overhead in the model training process and improve the model training efficiency.

[0005] To achieve the above-mentioned purpose, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the embodiments of the present application provide a model training system, which comprises a control node and a plurality of working nodes, wherein: the control node is configured to select at least one first execution node from the plurality of working nodes and instruct the at least one first execution node to start executing a training task; the first execution node is configured to execute the training task according to the instruction of the control node to start executing the training task, and store training process data of the first execution node in a storage server and a memory of the first execution node during the training process; the control node is further configured to control the at least one first execution node to abort training when there is a training abnormality of the first execution node, and select at least one second execution node from the plurality of working nodes and instruct the at least one second execution node to continue executing the training task; the second execution node is configured to obtain training process data of the second execution node in the storage server or a memory of the second execution node according to the instruction of the control node to continue executing the training task, and continue executing the training task based on the training process data.

[0007] Based on the scheme, when the first execution node appears an exception, the second execution node can continue to execute the training task based on the training process data, so that the model training process does not need to start over, but only needs to continue training from the interruption position, so as to improve the training efficiency of the artificial intelligence model; at the same time, since the training process data is stored in the memory of the node and the storage server, the time for the node to read the training process data will also be reduced accordingly, so the overall training efficiency of the model can be further improved.

[0008] In a possible implementation, the control node is configured to: select a candidate node group meeting the resource condition of the training task from the plurality of working nodes, the candidate node group including at least one working node; if the local storage of any working node in the candidate node group stores the training data set corresponding to the training task, the any working node is added to the at least one first execution node; if the local storage of all working nodes in the candidate node group does not store the training data set corresponding to the training task, any working node in the candidate node group is added to the at least one first execution node.

[0009] Based on the scheme, by judging whether the working node meets the resource condition and whether the local storage of the working node stores the training data set corresponding to the training task, the first execution node meeting the training task requirement can be screened out, so as to improve the success rate of model training and the reading efficiency of the training data set, and further improve the execution efficiency of the training task.

[0010] In another possible implementation, each first execution node is configured to: store the training process data of the first execution node in the storage server and the memory of the first execution node at a preset time interval during the training process of the first execution node; or store the training process data of the first execution node in the storage server and the memory of the first execution node when the first execution node completes a training round.

[0011] Based on the scheme, by storing the training process data, the recoverability of the training process and the traceability of the data can be ensured, and subsequent analysis and debugging are facilitated.

[0012] In a further possible implementation, the control node is configured to: detect whether the first execution node fails and whether the first execution node meets the resource condition of the training task; if the first execution node does not fail and the first execution node meets the resource condition of the training task, add the first execution node to the at least one second execution node; and if the first execution node fails or the first execution node does not meet the resource condition of the training task, add the work node in the plurality of work nodes that meets the resource condition of the training task and stores the training data set corresponding to the training task to the at least one second execution node.

[0013] Based on the scheme, the first execution node that can continue to execute the training task can be preferentially selected as the second execution node, so as to facilitate the rapid recovery of the training process and improve the training efficiency.

[0014] In a further possible implementation, each second execution node is configured to: read the training process data of the second execution node in the memory of the second execution node according to the instruction of the control node to continue to execute the training task; and if the training process data of the second execution node is not stored in the memory of the second execution node, acquire the training process data of the second execution node through the storage server.

[0015] Based on the scheme, the second execution node can read the recently stored training process data from the memory, and acquire the training process data from the storage server when the training process data of the second execution node is not stored in the memory, so as to quickly acquire the training process data and facilitate the rapid recovery of the training process.

[0016] In a further possible implementation, the at least one first execution node includes a master execution node and slave execution nodes, the master execution node is any first execution node in the at least one first execution node, and the slave execution nodes include other first execution nodes in the at least one first execution node except the master execution node; the training process data includes model parameters; the slave execution nodes are configured to execute the training task by using the initial model parameters sent by the master execution node and synchronize the gradient parameters generated by training to the master execution node; the master execution node is configured to process the gradient parameters generated by training of the slave execution nodes, obtain the update parameters of the slave execution nodes, and synchronize the update parameters to the slave execution nodes; and the slave execution nodes are configured to continue training by using the update parameters.

[0017] Based on the scheme, the parameter scheduling and updating can be uniformly performed by the master execution node, so that the model parameters of the multiple slave execution nodes can be updated synchronously, and the training process can be orderly and efficiently performed.

[0018] In another possible implementation, the first execution nodes are multiple, and the multiple first execution nodes form a unidirectional communication loop; the training process data includes model parameters; each first execution node is configured to: when performing the training task, sequentially send, based on the unidirectional communication loop, a gradient parameter corresponding to a model parameter generated by training to a next first execution node in a block manner; and receive and accumulate a corresponding gradient parameter block of a previous first execution node, to obtain an aggregation result of the gradient parameters of all the first execution nodes after multiple rounds of transmission; determine a corresponding update parameter based on the aggregation result, and continue training by using the update parameter.

[0019] Based on the scheme, the gradient parameter transmission and aggregation can be completed by using the communication link between the nodes without setting master and slave nodes, and the communication efficiency and scalability are high.

[0020] In another possible implementation, whether each working node in the candidate node group locally stores a training data set corresponding to the training task is determined according to a data label corresponding to the training task, after any working node in the candidate node group is added to the first execution node group, the control node is further configured to: control a working node in the any working node to obtain the training data set corresponding to the training task by using the storage service end, and store the training data set corresponding to the training task locally; and set the data label corresponding to the training task for the any working node after the working node in the any working node stores the training data set corresponding to the training task locally.

[0021] Based on the scheme, the first execution node can be screened according to whether each working node in the candidate node group locally stores a training data set corresponding to the training task before the training task is performed, and the training data set can be obtained from the storage service end and stored in the local storage of the working node in a case where the training data set is not stored locally in all the working nodes, so that the working node can directly read the training data set from the local storage in the subsequent training process, instead of obtaining the training data set from the storage service end again, and the training data set acquisition efficiency can be improved.

[0022] In a second aspect, the embodiments of the present application also provide a distributed training method of a model, applied to a control node, the method comprising: selecting at least one first execution node from a plurality of working nodes, and instructing the at least one first execution node to start executing a training task; monitoring a training state of the at least one first execution node during execution of the training task by the at least one first execution node; when a training abnormality occurs in a first execution node in the at least one first execution node, controlling the at least one first execution node to abort training, and selecting at least one second execution node from the plurality of working nodes, and instructing the at least one second execution node to continue executing the training task, so that the second execution node acquires training process data of the second execution node in a storage server or a memory of the second execution node according to an instruction of the control node to continue executing the training task, and continues executing the training task based on the training process data.

[0023] Based on the present solution, the control node can control the first execution node to execute the training task, and when the first execution node appears abnormal, the control node controls the first execution node to abort training, and reselects the second execution node, and controls the second execution node to read the training process data to quickly continue executing the training task, thereby improving the training efficiency of the model.

[0024] In a possible implementation, the selecting at least one second execution node from the plurality of working nodes comprises: selecting a candidate node group meeting a resource condition of the training task from the plurality of working nodes, the candidate node group comprising at least one working node; if local storage of any working node in the candidate node group stores a training data set corresponding to the training task, the any working node is added to the at least one first execution node; and if local storage of all working nodes in the candidate node group does not store the training data set corresponding to the training task, any working node in the candidate node group is added to the at least one first execution node.

[0025] Based on the present solution, the control node can determine whether the working node meets the resource condition, and whether the local storage of the working node stores the training data set corresponding to the training task, so that the first execution node meeting the requirement of the training task is screened out, thereby improving the success rate of model training, the reading efficiency of the training data set, and further ensuring the execution efficiency of the training task.

[0026] In another possible implementation, the selecting at least one second execution node from the plurality of worker nodes comprises: detecting whether the first execution node fails and whether the first execution node meets the resource condition of the training task; if the first execution node does not fail and the first execution node meets the resource condition of the training task, adding the first execution node to the at least one second execution node; if the first execution node fails or the first execution node does not meet the resource condition of the training task, adding a worker node that meets the resource condition of the training task and stores the training data set corresponding to the training task to the at least one second execution node.

[0027] Based on the scheme, the control node can preferentially select the first execution node that can continue to execute the training task as the second execution node, so as to facilitate the rapid recovery of the training process and improve the training efficiency.

[0028] In a third aspect, the embodiments of the present application further provide a distributed training method of a model, applied to an execution node, the execution node being any one of a plurality of worker nodes, the method comprising: executing a training task according to an instruction of a control node to start the training task, and storing training process data of the execution node in a storage server and a memory of the execution node in a training process; or, according to an instruction of the control node to continue executing the training task, acquiring the training process data of the execution node from the storage server or the memory of the execution node, and continuing to execute the training task based on the training process data.

[0029] Based on the scheme, the execution node can execute the training task or continue to execute the training task after the training task stops according to the instruction of the control node, so as to improve the training efficiency of the model.

[0030] In a possible implementation, the storing the training process data of the execution node in the storage server and the memory of the execution node in a training process comprises: storing the training process data of the execution node in the storage server and the memory of the execution node at a preset time interval in a training process of the execution node; or, storing the training process data of the execution node in the storage server and the memory of the execution node when the execution node completes one training round.

[0031] Based on the scheme, the execution node can store the training process data in the training process, so as to guarantee the recoverability of the training process and the traceability of the data, and facilitate subsequent analysis and debugging.

[0032] In another possible implementation, the obtaining, by the at least one first execution node, the training process data of the at least one first execution node from the memory of the at least one first execution node or the storage service end comprises: reading, by the at least one first execution node, the training process data of the at least one first execution node from the memory of the at least one first execution node according to the instruction of the control node to continue executing the training task; and obtaining, by the at least one first execution node, the training process data of the at least one first execution node from the storage service end if the training process data of the at least one first execution node is not stored in the memory of the at least one first execution node.

[0033] Based on the scheme, the control node can start to execute the training task or quickly continue to execute the training task under the control of the control node, and the training efficiency of the model can be improved.

[0034] In a fourth aspect, the embodiments of the present application further provide an electronic device, which comprises a memory and a processor; the memory and the processor are coupled; the memory is used to store program instructions; and the processor is used to call the program instructions, so that the electronic device executes the distributed training method of the model according to any one of the above.

[0035] According to the model training system, the distributed training method of the model and the electronic device provided in the present application, the control node selects at least one first execution node from a plurality of working nodes, and instructs the at least one first execution node to execute a training task; the first execution node executes the training task according to the instruction of the control node to start to execute the training task, and stores training process data of the first execution node in the memory of the first execution node and a storage service end during the training process; meanwhile, the control node is further configured to control the at least one first execution node to abort the training when the first execution node has a training exception, and select at least one second execution node from the plurality of working nodes and instruct the at least one second execution node to continue to execute the training task; the second execution node obtains the training process data of the second execution node from the storage service end or the memory of the second execution node according to the instruction of the control node to continue to execute the training task, and continues to execute the training task based on the training process data. By obtaining the training process data, the second execution node can continue to execute the training task based on the training process data, so that the model training process does not need to be restarted, but only needs to continue to train from the interrupted position, and therefore the training efficiency of the artificial intelligence model can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A structural schematic diagram of a model training system provided by the embodiments of the present application;

[0037] Figure 2 A flow schematic diagram of a model training system provided by the embodiments of the present application;

[0038] Figure 3 A storage schematic diagram of training process data provided for an embodiment of the present application;

[0039] Figure 4 A flowchart of a node selection method provided for an embodiment of the present application;

[0040] Figure 5 A flowchart of another node selection method provided for an embodiment of the present application;

[0041] Figure 6 A schematic diagram of an execution node provided for an embodiment of the present application;

[0042] Figure 7 A flowchart of a training method provided for an embodiment of the present application;

[0043] Figure 8 A topology schematic diagram of a node provided for an embodiment of the present application;

[0044] Figure 9 A flowchart of another training method provided for an embodiment of the present application;

[0045] Figure 10 A flowchart of a node screening method provided for an embodiment of the present application;

[0046] Figure 11 A flowchart of a training recovery method provided for an embodiment of the present application;

[0047] Figure 12 A flowchart of a distributed training method of a model provided for an embodiment of the present application;

[0048] Figure 13 A flowchart of another distributed training method of a model provided for an embodiment of the present application;

[0049] Figure 14 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In order to clearly describe the technical solutions in the embodiments of the present application, the first, second, etc. descriptions in the embodiments of the present application are only used for indicating and distinguishing the description objects, and do not have the order, and do not represent the special limitation of the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0051] Distributed training of a model refers to distributing the training process of an artificial intelligence model to multiple nodes or devices to complete collaboratively, accelerating the training through parallel computing and resource collaboration, so that computing, storage, communication and other tasks are split to different nodes or devices, and the overall training goal is achieved through an efficient coordination mechanism.

[0052] However, due to the large size of the training data set of the model, the loading and processing time will be long, and at the same time, a large amount of intermediate data will be generated during the training process, which needs to be frequently accessed and updated, so the data overhead during the actual training of the model is large.

[0053] In view of one or more of the foregoing problems, the embodiments of the present application first provide a model training system, which comprises a control node and a plurality of worker nodes. Referring to Figure 1 As shown in the figure, the model training system 100 can include a control node 110 and a plurality of worker nodes 120, such as Figure 1 worker nodes 121, worker nodes 122, …, worker nodes 12n as shown in the figure.

[0054] The control node 110 is the coordination center of the model training system, responsible for global task scheduling, resource allocation, state monitoring, and parameter synchronization management, such as running and managing containerized applications. For example, the control node 110 can allocate training tasks to worker nodes, and multiple containers can be deployed and run in the worker nodes. At the same time, the control node 110 is responsible for ensuring the load balancing of each worker node, and can also track the training progress, resource utilization of each worker node, detect node failures and trigger recovery mechanisms, and in the event of a worker node failure, reassign tasks or resume training.

[0055] Each of the plurality of worker nodes 120 is a computing unit that actually performs model training, responsible for processing data shards, forward / backward propagation, gradient calculation, and other specific tasks. For example, the worker node can read data shards in a local or distributed storage system, perform data augmentation, normalization, and other preprocessing, exchange parameters with other worker nodes, and maximize the use of local computing resources to complete the training task.

[0056] To illustrate the working process of the model training system provided by the present embodiment, Figure 2 a flowchart of the model training system provided by the present embodiment is shown, as Figure 2 As shown in the figure, the model training system can perform the following steps S210-S240:

[0057] Step S210: The control node selects at least one first execution node in the plurality of worker nodes, and instructs the at least one first execution node to perform a training task.

[0058] The at least one first execution node constitutes a first execution node group, and includes a node for executing the training task, wherein the first execution node can be any one of the plurality of worker nodes.

[0059] Before the training starts, the control node can select the worker nodes capable of executing the training task from the plurality of worker nodes. For example, the control node can select one or more worker nodes that meet the requirements of resources, data, load, affinity and anti-affinity, and the like, from the plurality of worker nodes according to a configuration file written by a user in advance, to obtain the at least one first execution node. The configuration file is a file for defining key parameters of the training task by the user, and can be written by the user in a program file such as YAML (a human-readable data serialization language) or JSON (a lightweight data exchange format), and submitted to the control node through a command line tool, a web interface or an API (application programming interface).

[0060] When the training task is started, the control node can issue a task execution command to each of the at least one first execution node, so that each of the first execution nodes starts the training. In some cases, the control node can also check the communication state between the first execution nodes to ensure that the communication between the first execution nodes is normal, thereby ensuring the successful execution of the training task.

[0061] Step S220: The first execution node executes the training task according to the instruction of the control node to start the training task, and stores the training process data of the first execution node in the memory of the first execution node and the storage server during the training process.

[0062] The storage server refers to a storage system for persistently storing the training process data. For example, the storage system can be a distributed storage system, which is globally shared, supports centralized management and long-term storage of multi-node data, and can ensure data consistency and traceability.

[0063] The training process data of the first execution node refers to the intermediate data generated by the first execution node during the training process. For example, the training process data can include intermediate parameters (model checkpoints) generated by the first execution node during the training, such as the weight, configuration and state of the optimizer of the model, and can also include intermediate data generated by the first execution node during the preprocessing of the corresponding training data set.

[0064] When there is only one first execution node in the at least one first execution node, the first execution node performs the training task alone, that is, the first execution node independently completes the entire training process. For example, the first execution node independently loads and preprocesses the training data set, initializes the artificial intelligence model, defines the optimizer and loss function, and the like, and then performs the training loop, such as inputting the training data set into the artificial intelligence model, calculating the model output, and evaluating the difference between the predicted value and the true value through the loss function, calculating the gradient and updating the model parameters, for iteration until the stopping condition is reached.

[0065] When there are multiple first execution nodes in the at least one first execution node, the first execution nodes can perform the training task in parallel, that is, the multiple first execution nodes work cooperatively to accelerate the training through a distributed strategy.

[0066] In some embodiments, the multiple first execution nodes can perform training in a data parallel mode, that is, each first execution node saves a complete model copy, processes different data subsets in the training data set, and realizes parallel training through synchronized gradients.

[0067] In some embodiments, the multiple first execution nodes can also perform training in a model parallel mode, that is, the artificial intelligence model is split into multiple parts and distributed to different first execution nodes, and the first execution nodes pass intermediate calculation results.

[0068] Since some intermediate data generated during the training process need to be frequently accessed and updated, and the data volume of the intermediate data is usually small, in order to facilitate the acquisition and access of the intermediate data and improve the training speed, for the first execution node, the training process data of the first execution node can be stored in the memory of the first execution node and the storage server during the training process.

[0069] In order to illustrate the storage mode of the training process data, Figure 2 a storage diagram of training process data provided by the embodiment is shown, as shown in Figure 3 Node-0, Node-1, …, Node-N are N first execution nodes, each of which is equipped with a GPU for accelerating the training of the artificial intelligence model. For example, in a data parallel mode, Node-0, Node-1, …, Node-N are used to train different data subsets in the training data set, respectively.

[0070] During the training process, the training process data generated by each first execution node is first stored in the memory corresponding to the first execution node, and then is synchronously stored in the storage server, such as Figure 3 in the distributed storage system, so as to persistently store the training process data.

[0071] Step S230: The control node controls the at least one first execution node to suspend training when a training exception occurs in the first execution node, and selects at least one second execution node from the plurality of worker nodes and instructs the at least one second execution node to continue performing the training task.

[0072] The at least one second execution node constitutes a second execution node group, and includes a node for resuming the training task.

[0073] During training, the control node can monitor the training state of each of the at least one first execution node, and when it is determined that a training exception occurs in one or more of the at least one first execution node, the control node controls the at least one first execution node, i.e., all first execution nodes for performing the training task, to suspend training, and triggers the operation of selecting at least one second execution node from the plurality of worker nodes.

[0074] The training exception of the first execution node can include a training exception caused by a hardware failure, resource shortage, software error, etc. of the first execution node itself, a training exception caused by damage to a training data set corresponding to the first execution node, a training exception caused by an interruption of training, or a training exception caused by a communication failure between the first execution node and other nodes.

[0075] When monitoring the training state of each of the at least one first execution node, the control node can determine whether a training exception occurs in the first execution node by monitoring resource usage of the first execution node, analyzing logs and indicators of the first execution node, and automatically checking a health state of the first execution node.

[0076] After the control node controls the at least one first execution node to suspend training, in order to restart training, the control node can reselect nodes from the plurality of worker nodes, i.e., select at least one second execution node, and instruct the at least one second execution node to continue performing the training task. The second execution node can be any one of the plurality of worker nodes.

[0077] In this embodiment, the execution nodes in the at least one first execution node and the at least one second execution node can be overlapped or completely different. That is, after the at least one second execution node is reselected, all second execution nodes in the at least one second execution node can be different from all first execution nodes in the at least one first execution node, or can be overlapped. In the case that the first execution node where the training exception occurs is restored to normal, the at least one first execution node can be the same as the at least one second execution node, for example, when the at least one first execution node includes node 1, node 2 and node 3, the at least one second execution node also includes node 1, node 2 and node 3.

[0078] Step S240: According to the instruction of the control node to continue executing the training task, the second execution node obtains the training process data of the second execution node in the storage server or the memory of the second execution node, and continues to execute the training task based on the training process data.

[0079] The instruction of the control node to continue executing the training task can be sent by the control node to the second execution node through a task continuation instruction or a signal completion.

[0080] After all the second execution nodes are determined, the second execution node can obtain the training process data of the second execution node in the storage server or the memory of the second execution node according to the instruction of the control node to continue executing the training task, so that the second execution node continues to execute the training task based on the obtained training process data.

[0081] When the first execution node where the training exception occurs exists, by obtaining the training process data, the second execution node can continue to execute the training task based on the training process data, so that the model training process does not need to start again, but only needs to continue training from the interrupted position, so the training efficiency of the model can be improved; at the same time, since the training process data is stored in the memory of the node and the storage server, the node can directly read the training process data from the memory, so the data reading time is also reduced, so the overall training efficiency of the model can be further improved.

[0082] Figure 4 A flowchart of a node selection method provided by the embodiment is shown, and as shown in Figure 4 The control node can select at least one first execution node from a plurality of working nodes by the following method:

[0083] Step S410: In a plurality of working nodes, a candidate node group satisfying the resource condition of the training task is selected. The candidate node group can include at least one working node.

[0084] The resource condition refers to a computing resource condition required for the node to perform the training task, which can include a hardware resource condition, a software environment condition, a node state condition, and a user-defined condition, etc.

[0085] The hardware resource condition refers to a condition required for a GPU, a CPU, a memory, a network bandwidth, etc. of the node; the software environment condition refers to a condition required for an operating system, a dependent framework, a container environment, etc. of the node; the node state condition refers to a condition required for whether the node is online, whether there is a hardware failure, a current utilization rate, a remaining memory, etc. of the node; and the user-defined condition can include a label of the node, etc.

[0086] To facilitate the execution of the training task, the control node can filter the working nodes that meet the above resource conditions from the plurality of working nodes to obtain a candidate node group including at least one working node.

[0087] Step S420: If the local storage of any working node in the candidate node group stores a training data set corresponding to the training task, the any working node is added to the at least one first execution node.

[0088] In some cases, the training data set has a large amount of data, and the memory of the working node can not support storage, which can be stored locally. It should be understood that in the case that the memory of the working node can support the storage of the training data set, the training data set can also be stored in the memory of the working node.

[0089] For the candidate node group, the control node can determine whether the local storage of each working node stores a training data set corresponding to the training task, and if so, the working node is added to the candidate node group.

[0090] In the data parallel training mode, the training data set stored locally by each working node belongs to the same training task, but the training data set itself is not the same. In the model parallel training mode, the training data set stored in the local storage of each working node can be the same.

[0091] Step S430: If the local storage of all working nodes in the candidate node group does not store a training data set corresponding to the training task, any working node in the candidate node group is added to the at least one first execution node.

[0092] When the local storage of all working nodes in the candidate node group does not store a training data set corresponding to the training task, it is not possible to select a node with a training data set, so the control node adds any number of working nodes in the candidate node group to the at least one first execution node.

[0093] By judging whether the worker nodes meet the resource condition and whether the local of the worker nodes stores the training data set corresponding to the training task, the first execution node meeting the requirement of the training task can be screened out, so as to improve the success rate of model training and the reading efficiency of the training data set, and further guarantee the execution efficiency of the training task.

[0094] In some embodiments, whether the local of each worker node in the candidate node group stores the training data set corresponding to the training task is determined according to a data label corresponding to the training task.

[0095] The data label is used to identify the training data set corresponding to the training task.

[0096] For example, when determining whether the local of each worker node in the candidate node group stores the training data set corresponding to the training task, the control node can search the database storing the node information according to the data label corresponding to the training task to determine whether the data label exists in each worker node. If the data label exists in the worker node, the worker node is added to the at least one first execution node, i.e., the worker node is one of the first execution nodes. If no data label corresponding to the training task exists in all worker nodes in the candidate node group, i.e., the local of all worker nodes does not store the training data set corresponding to the training task, it indicates that the training data set corresponding to the training task needs to be obtained from a remote node or other storage server. At this time, any worker node in the candidate node group can be added to the at least one first execution node.

[0097] After adding any worker node in the candidate node group to the first execution node group, the control node controls the worker node in the any worker node to obtain the training data set corresponding to the training task through the storage server and store the training data set corresponding to the training task in the local of the node.

[0098] Then, after the worker node in the any worker node stores the training data set corresponding to the training task in the local of the node, the control node sets the data label corresponding to the training task for the any worker node.

[0099] By this method, before the training task is executed, the first execution node can be screened according to whether the local of each worker node in the candidate node group stores the training data set corresponding to the training task, and the training data set can be obtained from the storage server and stored in the local of the worker node when the local of all worker nodes does not store the training data set. In this way, in the subsequent training process, the worker node can directly read the training data set in the local, without the need to obtain the training data set from the storage server again, which can improve the efficiency of obtaining the training data set.

[0100] Figure 5 A flowchart of another node selection method provided by the embodiment is shown in FIG. 3. Figure 5As shown, the method can comprise the following steps:

[0101] Step S501: The control node initiates execution of a training task.

[0102] Step S502: The control node selects a candidate node group meeting resource conditions of the training task from the plurality of worker nodes.

[0103] Step S503: The control node determines whether the local of the worker node in the candidate node group stores a training data set corresponding to the training task. If yes, step S504 is performed, and if the local of each worker node in the candidate node group does not store the training data set, step S506 is performed.

[0104] Step S504: The worker node storing the training data set locally is added to a first execution node group.

[0105] The first execution node group comprises the worker node for executing the training task.

[0106] For the worker node storing the training data set locally, the efficiency of reading the training data set is higher, so adding the worker node to the first execution node group can facilitate subsequent execution of the training task by the worker node.

[0107] Step S505: A training subtask is created on the first execution node and the training data set stored locally is read for training.

[0108] After the first execution node group is determined, all first execution nodes in the first execution node group start to execute the training task. Specifically, the first execution node can create a training subtask and read the training data set stored locally for training.

[0109] Step S506: Any node in the candidate node group is added to the first execution node group.

[0110] When the local of each worker node in the candidate node group does not store the training data set, any number of worker nodes in the candidate node group can be added to the first execution node group.

[0111] Step S507: The training data set is loaded from a storage server and stored to the local of the first execution node, and the control node sets a data tag for the first execution node.

[0112] Since the local of the first execution node in the first execution node group does not store the training data set, the training data set needs to be obtained through the storage server, and after the training data set is obtained, the control node sets a data tag corresponding to the training task for the first execution node, to facilitate reading of the training data set and training next time through the tag to select the node having the training data set.

[0113] After step S507, the first execution node can perform step S505 to create a training subtask and read the locally stored training data set for training.

[0114] By the method shown in the figure, the node suitable for executing the training task can be selected through the resource condition and the data condition of the worker node, and the training is performed. Figure 5

[0115] Further, after determining all the first execution nodes, the control node can control all the first execution nodes to start training. In some embodiments, the at least one first execution node can include a master execution node and a slave execution node.

[0116] The master execution node can be any one of the at least one first execution node, and the slave execution node can be any one of the at least one first execution node other than the master execution node. The training process data can include model parameters.

[0117] In order to illustrate the relationship between the master execution node and the slave execution node and the training process, Figure 6 A schematic diagram of an execution node provided by the present embodiment is shown in the figure, which shows a first execution node group composed of the at least one first execution node, including a master execution node, i.e. Worker1, and slave execution nodes, i.e. Worker2, Worker3 and Worker4. The master execution node is responsible for the storage and update of model parameters, and the slave execution nodes are responsible for training. The master execution node and the slave execution nodes can communicate through a communication client or the like. Figure 6

[0118] Specifically, in step S220, as shown in the figure, the at least one first execution node performing the training task can be achieved by the following steps: Figure 7

[0119] Step S710: The slave execution node performs the training task using the initial parameters sent by the master execution node, and synchronizes the gradient parameters generated by the training to the master execution node.

[0120] As shown in the figure, the slave execution nodes, i.e. Worker2, Worker3 and Worker4, perform the training task in parallel, for example, Worker2, Worker3 and Worker4 read different training data sets corresponding to the training task respectively, pull the initial parameters, i.e. w, from the master execution node, i.e. Worker1, use w and the training data set to obtain gradient data, i.e. △w, and then upload the gradient parameters to the master execution node. Figure 6

[0121] ​​​​Step S720: The master execution node processes the gradient parameters generated by the slave execution nodes to obtain updated parameters, and synchronizes the updated parameters to the slave execution nodes.

[0122] The updated parameters can be model parameters updated based on the initial parameters.

[0123] Step S730: The slave execution nodes continue training using the updated parameters.

[0124] For example, after receiving the model parameters sent by each execution node in the slave execution nodes, the master execution node calculates the updated parameters based on the corresponding optimization algorithm, and then synchronizes the updated parameters to the corresponding execution node in the slave execution nodes, so that the execution node continues training using the updated parameters.

[0125] In the above structure, through the master execution node and the slave execution nodes, the master execution node can uniformly schedule and update parameters, so that the slave execution nodes can synchronize the updated model parameters, and ensure that the training process is orderly and efficient.

[0126] In some embodiments, the at least one first execution node is multiple, and the multiple first execution nodes form a unidirectional communication loop. Referring to Figure 8 As shown in the figure, node 1, node 2, node 3, node 4 and node 5 form a unidirectional communication loop, for example, node 1 can only transmit information to node 2, and node 2 can only transmit information to node 3.

[0127] Therefore, in step S120, referring to Figure 9 As shown in the figure, each first execution node can execute the following method:

[0128] Step S910: When performing the training task, the gradient parameters corresponding to the model parameters generated by the training are sent to the next first execution node in blocks based on the unidirectional communication loop.

[0129] For example, in the multiple first execution nodes, each first execution node performs the training task in parallel, and in the training process, each first execution node first divides the gradient parameters generated by the training into blocks based on the unidirectional communication loop, and sends one gradient parameter block currently held to the next node, while receiving the gradient parameter block transmitted by the previous node.

[0130] Step S920: Receive and accumulate the corresponding gradient parameter block of the previous first execution node, and obtain the aggregation result of the gradient parameters of all first execution nodes after multiple rounds of transmission.

[0131] After each communication, the first execution node aggregates the received gradient parameter blocks with its own gradient parameter blocks. After N-1 communications, the kth gradient parameter block of each first execution node will contain the sum of the kth gradient parameter blocks of all nodes. Then, the first execution node continues to pass the aggregated gradient parameter blocks along the one-way communication loop, passing one block each time, and after N-1 communications, all first execution nodes will obtain the complete aggregation result.

[0132] Step S930: Determine the corresponding update parameters based on the aggregation result, and continue training using the update parameters.

[0133] For example, each first execution node calculates the update parameters based on the aggregation result and the corresponding optimization algorithm, and then synchronizes the update parameters to the corresponding first execution node, so that each first execution node continues to train using the corresponding update parameters.

[0134] Through the above method, the master-slave nodes do not need to be set, and the communication link between the nodes can be used to complete the transmission and aggregation of the gradient parameters, and has high communication efficiency and scalability.

[0135] In some embodiments, in step S220, the first execution node can store the training process data in the memory of the first execution node and the storage service end by the following method:

[0136] During the training process of the first execution node, the training process data of the first execution node is stored in the memory of the first execution node and the storage service end at a preset time interval; or, when the first execution node completes a training round, the training process data of the first execution node is stored in the memory of the first execution node and the storage service end.

[0137] The preset time interval can be set according to the complexity of the artificial intelligence model, the data amount of the training data set, etc.

[0138] For example, when the first execution node starts training, it will start timing at the same time, and when it reaches the preset time interval, it will trigger the storage operation to store the training process data of the first execution node in the memory of the first execution node and the storage pool. This way can guarantee the data integrity within a certain time range.

[0139] An epoch refers to a process of completing one traversal of all training data sets, for example, when each first execution node completes one traversal of the respective corresponding training data set, it is called completing an epoch. At the end of each training epoch, the first execution node saves the generated training process data to the storage service and the memory of the first execution node at the same time. In this way, the progress of training is taken as a trigger condition, which can ensure that data backup is performed after each complete training cycle.

[0140] By storing the training process data, the recoverability of the training process and the traceability of the data can be ensured, and subsequent analysis and debugging are facilitated.

[0141] Figure 10 A flowchart of a node screening method provided by the embodiment is shown, as shown in Figure 10 may include the following steps:

[0142] Step S1010: detecting whether the first execution node has a failure and whether the first execution node meets the resource condition of the training task.

[0143] During the running of the nodes, the resources, software environment, etc. of the nodes may dynamically change due to the running conditions. Therefore, in order to determine whether the first execution node can continue to execute the training task, the control node can determine whether the first execution node has a failure and whether the first execution node meets the resource condition of the training task according to the monitoring conditions of the first execution nodes.

[0144] The failure of the first execution node can include failure problems of the first execution node caused by hardware failure, software environment, etc. Generally, such failure cannot be automatically recovered.

[0145] Step S1020: if the first execution node has no failure and meets the resource condition of the training task, the first execution node is added to at least one second execution node.

[0146] In the case that the first execution node has no failure and meets the resource condition, the first execution node can recover the training and continue to execute the training task, so the first execution node can be added to the at least one second execution node as a second execution node.

[0147] Step S1030: if the first execution node has a failure or does not meet the resource condition of the training task, the working nodes that meet the resource condition of the training task and store the training data set corresponding to the training task are added to at least one second execution node.

[0148] When the first execution node fails, the first execution node cannot continue training, and the training process data in the memory can be lost due to node failure. Therefore, the control node needs to reselect the resource conditions of the multiple working nodes that meet the training task and the working nodes that store the training data set corresponding to the training task, and add at least one second execution node. Alternatively, in the case that the first execution node does not meet the resource conditions, the control node can also reselect the working nodes that meet the resource conditions and store the training data set corresponding to the training task in the multiple working nodes as the second execution node, and add at least one second execution node. That is, in the case that the first execution node cannot continue to execute the training task, the working node that meets the resource conditions and has the training data set can be preferentially selected as the second execution node.

[0149] In fact, by this method, the first execution node that can continue to execute the training task can be preferentially selected as the second execution node, which facilitates the rapid recovery of the training process and improves the training efficiency.

[0150] In some embodiments, in step S1030, if the first execution node fails, or the first execution node does not meet the resource conditions of the training task, and none of the multiple working nodes that meet the resource conditions stores the training data set, any one or more of the multiple working nodes that meet the resource conditions can be selected as at least one second execution node. For example, the working nodes that meet the resource conditions and have a number comparable to the number of failed nodes can be selected from the multiple working nodes and added to at least one second execution node.

[0151] In some embodiments, in step S240, the second execution node can obtain the training process data by the following method:

[0152] According to the instruction of the control node to continue executing the training task, the training process data of the second execution node is read from the memory of the second execution node.

[0153] If the training process data of the second execution node is not stored in the memory of the second execution node, the training process data of the second execution node is obtained from the storage server.

[0154] In the above method, according to the instruction of the control node to continue executing the training task, the second execution node can first read the recently stored training process data from the memory, and then obtain the training process data from the storage server when the training process data of the second execution node is not stored in the memory, which can improve the efficiency of obtaining the training process data, so as to quickly recover the training process.

[0155] Figure 11 A flowchart of a training recovery method provided by the embodiment is shown in FIG. 1. Figure 11As shown, the method can comprise the following steps:

[0156] Step S1101: During the training process, the control node monitors the training state of each first execution node in the first execution node group.

[0157] For example, the control node can determine whether each first execution node is running normally according to the resource usage of each first execution node and the health state of the automated inspection.

[0158] Step S1102: When there is a training exception in a first execution node in the first execution node group, the control node controls the first execution node group to stop training and determines whether the first execution node in the first execution node group is down. When the first execution node is down, step S1104 is performed, otherwise, when the first execution node is not down, step S1103 is performed.

[0159] When there is an exception in a first execution node, since each node needs to be synchronized for training, the control node needs to control the first execution node group to stop training, and at the same time, check whether each first execution node is down. When a certain first execution node is not down, step S1103 is performed. When all first execution nodes are down, step S1104 is performed.

[0160] Step S1103: The control node determines whether the first execution node meets the resource condition of the training task. If yes, step S1109 is performed, if not, step S1104 is performed.

[0161] Since the resources, storage state, etc. of the first execution node dynamically change, in order to determine the second execution node group, the control node needs to determine again whether the first execution node meets the resource condition. If it meets, step S1109 is performed, if it does not meet, step S1104 is performed.

[0162] Step S1104: Prioritize the work nodes that locally store the training data set in the plurality of work nodes and add them to the second execution node group.

[0163] In the plurality of work nodes, the work nodes that locally store the training data set are prioritized as the second execution node. If the number of nodes is not enough or there is no work node that locally stores the training data set, other work nodes in the plurality of work nodes can be further selected.

[0164] Step S1105: Obtain the training process data of the second execution node through the storage service end and store it in the memory of the second execution node.

[0165] Since the first execution node is down, the memory does not store the training process data when a new work node is selected as the second execution node, so the corresponding training process data needs to be obtained from the storage server and stored in the memory.

[0166] Step S1106: When the second execution node locally stores the training data set, the second execution node reads the training data set locally and continues training using the training process data.

[0167] Step S1107: When the second execution node does not locally store the training data set, the second execution node remotely loads the training data set and stores it locally, and sets the data label.

[0168] By setting the data label, the work node that locally stores the training data set can be preferentially selected next time the training is abnormal.

[0169] Step S1108: The second execution node reads the training data set and the training process data to continue training.

[0170] After obtaining the training data set, the second execution node can read the obtained training data set locally and continue training using the training process data.

[0171] Step S1109: The first execution node continues the training job, reads the training process data in the memory and the locally stored training data set, and continues training.

[0172] Through the above method, the training task execution of the node can be monitored, and when the node is abnormal, the node group is reselected, and the training process is quickly recovered according to the condition of each node in the node group.

[0173] The embodiment also provides a distributed training method of a model, which can be applied to the control node in the model training system in the above embodiments.

[0174] Figure 12 A flowchart of a distributed training method of a model provided by an embodiment of the application is shown, as shown in Figure 12 may include the following steps:

[0175] Step S1210: Select at least one first execution node from the plurality of work nodes, and instruct the at least one first execution node to start executing the training task.

[0176] The first execution node can be any one of the plurality of work nodes, such as a node that meets the resource conditions of the training task and stores the training data set corresponding to the training task, or can also be a node that meets the resource conditions of the training task in the plurality of work nodes.

[0177] Before the training starts, the control node can select a work node capable of performing the training task from the plurality of work nodes to obtain at least one first execution node, and then can issue a task execution instruction or a signal to the at least one first execution node to enable the first execution node to start performing the training task.

[0178] Step S1220: During the execution of the training task by the at least one first execution node, the training state of the at least one first execution node is monitored.

[0179] During the training, the control node can monitor the training state of each of the at least one first execution node, such as whether a training abnormality occurs due to a hardware failure, a resource shortage, a software error, a training data set damage, a training interruption, a node communication failure, etc. of the first execution node.

[0180] Step S1230: When a training abnormality occurs in the at least one first execution node, the control node controls the at least one first execution node to stop the training, selects at least one second execution node from the plurality of work nodes, and instructs the at least one second execution node to continue performing the training task, so that the second execution node obtains training process data of the second execution node from the storage server or the memory of the second execution node according to the instruction of the control node to continue performing the training task, and continues performing the training task based on the training process data.

[0181] When it is determined that one or more first execution nodes have a training abnormality, the control node controls the at least one first execution node, i.e., all the first execution nodes for performing the training task, to stop the training, and triggers the operation of selecting at least one second execution node from the plurality of work nodes to instruct the second execution node to continue performing the training task.

[0182] Specifically, when the second execution node performs the training task, the second execution node can obtain training process data of the second execution node from the storage server or the memory of the second execution node according to the instruction of the control node to continue performing the training task, and continues performing the training task based on the training process data.

[0183] According to the distributed training method of the model provided in the embodiment, the control node can control the first execution node to perform the training task, and when an abnormality occurs in the first execution node, the control node controls the first execution node to stop the training, reselects a second execution node, and controls the second execution node to read training process data to quickly continue performing the training task, thereby improving the training efficiency of the model.

[0184] In a possible implementation, the at least one second execution node can be selected from the plurality of work nodes by the following method:

[0185] In the plurality of worker nodes, a candidate node group satisfying a resource condition of the training task is selected, and the candidate node group includes at least one worker node. The resource condition refers to a computing resource condition required by the node to execute the training task, and can include a hardware resource condition, a software environment condition, a node state condition, and a user-defined condition.

[0186] If the local storage of any worker node in the candidate node group stores the training data set corresponding to the training task, any worker node is added to the at least one first execution node. If the local storage of all worker nodes in the candidate node group does not store the training data set corresponding to the training task, any worker node in the candidate node group is added to the at least one first execution node.

[0187] In the above candidate node group, the node storing the training data set locally is preferentially selected, so that during training, the node can directly read the training data set from the local storage, without the need to obtain the training data set from the storage server, thereby improving the data reading efficiency and the model training efficiency.

[0188] In another possible implementation, at least one second execution node can be selected from the plurality of worker nodes by the following method:

[0189] It is detected whether the first execution node fails and whether the first execution node satisfies the resource condition of the training task. The failure of the first execution node can include a failure problem of the first execution node due to hardware failure, software environment, and the like.

[0190] If the first execution node does not fail and the first execution node satisfies the resource condition of the training task, the first execution node is added to the at least one second execution node. That is, in the case where the first execution node does not fail and satisfies the resource condition, the first execution node can resume the training and continue to execute the training task, so the first execution node can be selected as a second execution node and added to the at least one second execution node.

[0191] If the first execution node fails or the first execution node does not satisfy the resource condition of the training task, a worker node satisfying the resource condition of the training task and storing the training data set corresponding to the training task is added to the at least one second execution node.

[0192] In this process, the control node preferentially selects a worker node satisfying the resource condition and having the training data set as a second execution node in the case where the first execution node cannot continue to execute the training task.

[0193] By this method, the control node can preferentially select the first execution node capable of continuing to execute the training task as a second execution node, so as to facilitate the rapid recovery of the training process and improve the training efficiency.

[0194] It should be noted that the control method performed by the control node has been described in detail in the foregoing embodiment of the model training system, and the related details will not be repeated to avoid redundancy.

[0195] The embodiment also provides a distributed training method of a model, which can be applied to the execution node. The execution node can be any one of the plurality of worker nodes, such as the first execution node and / or the second execution node in the foregoing embodiment.

[0196] Figure 13 A flowchart of another distributed training method of a model provided by the embodiment is shown, which can include the following steps as shown in Figure 13

[0197] Step S1310: According to the instruction of the control node to start the training task, the training task is executed, and the training process data of the execution node is stored in the memory of the storage server and the execution node during the training process.

[0198] For example, after receiving the instruction of the control node to start the training task, the execution node can execute the training task. When there are a plurality of execution nodes, the plurality of execution nodes execute the training task in parallel, and can communicate with each other to interact and update the parameters.

[0199] In some embodiments, the execution node can be a master execution node, which can receive the gradient parameters sent by the slave execution node, determine the updated parameters based on the corresponding optimization algorithm, and feed back the updated parameters to the slave execution node to make it iterate the training process according to the updated parameters.

[0200] Step S1320: According to the instruction of the control node to continue the training task, the training process data of the execution node is obtained from the storage server or the memory of the execution node, and the training task is continued based on the training process data.

[0201] For example, when the execution node receives the instruction of the control node to continue the training task, the training can be continued with the training parameters in the training process data as the starting parameters, such as the model parameters.

[0202] According to the distributed training method of the model provided by the embodiment, the execution node can execute the training task or continue the training task after the training task stops according to the instruction of the control node, which can improve the training efficiency of the model.

[0203] ​In a possible implementation, the execution node can store the training process data of the execution node in the memory of the storage server and the execution node during the training process by the following method:

[0204] During the training process of the execution node, the training process data of the execution node is stored in the memory of the storage server and the execution node according to a preset time interval; or, when the execution node completes a training round, the training process data of the execution node is stored in the memory of the storage server and the execution node.

[0205] By this method, the execution node can store the training process data during the training process, guarantee the recoverability of the training process and the traceability of the data, and facilitate subsequent analysis and debugging.

[0206] In another possible implementation, the execution node can read the training process data of the execution node in the memory of the execution node according to the instruction of the control node to continue the training task; if the training process data of the execution node is not stored in the memory of the execution node, the training process data of the execution node is obtained through the storage server.

[0207] By this method, the recently stored training process data is first read from the memory, and when the training process data of the second execution node is not stored in the memory, the training process data is obtained from the storage server, which can improve the efficiency of obtaining the training process data, so as to quickly recover the training process.

[0208] It should be noted that the method executed by the execution node has been described in detail in the foregoing embodiment of the model training system, and related details will not be repeated to avoid redundancy.

[0209] Figure 14 A structure schematic diagram of an electronic device provided by an embodiment of the present application is shown, as shown in Figure 14 The electronic device 1400 can include a processor 1402, a communications interface 1404, a memory 1406, and a communications bus 1408.

[0210] The processor 1402, the communications interface 1404, and the memory 1406 can complete mutual communication through the communications bus 1408. The communications interface 1404 is configured to communicate with network elements such as clients or other servers. The processor 1402 is configured to execute a program 1410, and specifically can execute related steps in the foregoing distributed training method for a model.

[0211] Specifically, the program 1410 can include program code including computer executable instructions.

[0212] The processor 1402 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The electronic device can include one or more processors of the same type or different types, such as one or more CPUs and one or more ASICs.

[0213] The memory 1406 is configured to store a program 1410. The memory 1406 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory.

[0214] The program 1410 can be specifically invoked by the processor 1402 to cause the electronic device to perform the operation steps of the distributed training method of the model described above.

[0215] It should be noted that the above-described embodiments are illustrative of the present application rather than limiting the present application, and alternative embodiments can be designed by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, the several means can be embodied by one and the same item of hardware. The use of the words first, second and third, etc. does not imply any order. These words can be understood as names. The steps of the above-described embodiments, unless otherwise specified, are not to be understood as being necessarily limited in their sequence. The above-described embodiments are merely illustrative of the present application and are not used to limit the scope of protection of the present application, and any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present application shall be included in the scope of protection of the present application.

[0216] The above-described embodiments are merely specific embodiments of the present application and are not used to limit the scope of protection of the present application, and any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present application shall be included in the scope of protection of the present application.

Claims

1. A model training system comprising a control node and a plurality of worker nodes, wherein: the control node is configured to select at least one first execution node from the plurality of worker nodes and instruct the at least one first execution node to start executing a training task; the first execution node is configured to execute the training task according to the instruction of the control node to start executing the training task, and store training process data of the first execution node in a memory of the first execution node and a storage server during the training process; the control node is further configured to control the at least one first execution node to abort the training when a training exception occurs in the first execution node, and select at least one second execution node from the plurality of worker nodes and instruct the at least one second execution node to continue executing the training task; the second execution node is configured to obtain training process data of the second execution node in a memory of the second execution node or the storage server according to the instruction of the control node to continue executing the training task, and continue executing the training task based on the training process data.

2. The system of claim 1, wherein, the control node is configured to: select a candidate node group from the plurality of worker nodes, the candidate node group comprising at least one worker node, the at least one worker node satisfying a resource condition of the training task; if any worker node in the candidate node group has stored a training data set corresponding to the training task locally, add the any worker node to the at least one first execution node; if none of the worker nodes in the candidate node group has stored the training data set corresponding to the training task locally, add any worker node in the candidate node group to the at least one first execution node.

3. The system of claim 1, wherein, each first execution node is configured to: store training process data of the first execution node in a memory of the first execution node and a storage server at a preset time interval during the training process of the first execution node; or store the training process data of the first execution node in the memory of the first execution node and the storage server when the first execution node completes a training round. the control node is configured to:

4. The system of claim 1, wherein, detect whether the first execution node has failed and whether the first execution node satisfies the resource condition of the training task; if the first execution node has not failed and the first execution node satisfies the resource condition of the training task, add the first execution node to the at least one second execution node; if the first execution node has failed or the first execution node does not satisfy the resource condition of the training task, add a worker node satisfying the resource condition of the training task and storing the training data set corresponding to the training task from the plurality of worker nodes to the at least one second execution node. each second execution node is configured to:

5. The system of claim 1 or 3, wherein, read training process data of the second execution node in a memory of the second execution node according to the instruction of the control node to continue executing the training task. ​ If the training process data of the second execution node is not stored in the memory of the second execution node, the training process data of the second execution node is obtained through the storage server.

6. The system of claim 1, wherein, The at least one first execution node includes a master execution node and a slave execution node, the master execution node is any first execution node in the at least one first execution node, and the slave execution node includes other first execution nodes in the at least one first execution node except the master execution node; the training process data includes model parameters; The slave execution node is configured to execute the training task by using the initial parameters sent by the master execution node and synchronize the gradient parameters generated by training to the master execution node; The master execution node is configured to process the gradient parameters generated by training of the slave execution node to obtain the update parameters of the slave execution node and synchronize the update parameters to the slave execution node; The slave execution node is configured to continue training by using the update parameters.

7. The system of claim 1, wherein, The at least one first execution node is multiple, and multiple first execution nodes form a unidirectional communication loop; the training process data includes model parameters; each first execution node is configured to: When executing the training task, the gradient parameter blocks corresponding to the gradient parameters generated by training are sent to the next first execution node in turn based on the unidirectional communication loop; and After multiple rounds of transmission, the aggregation result of the gradient parameters of all first execution nodes is obtained by receiving and accumulating the corresponding gradient parameter blocks of the previous first execution node; Based on the aggregation result, the corresponding update parameters are determined, and the update parameters are used for continuous training.

8. The system of claim 2, wherein, Whether the local of each working node in the candidate node group stores the training data set corresponding to the training task is determined according to the data label corresponding to the training task; after any working node in the candidate node group is added to the first execution node group, the control node is further configured to: Control the working node in the any working node to obtain the training data set corresponding to the training task through the storage server and store the training data set corresponding to the training task locally; After the working node in the any working node stores the training data set corresponding to the training task locally, the data label corresponding to the training task is set for the any working node.

9. A method for distributed training of a model, the method comprising: The method applied to the control node includes: Selecting at least one first execution node from multiple working nodes and instructing the at least one first execution node to start executing a training task; Monitoring the training state of the at least one first execution node during the execution of the training task by the at least one first execution node; When a first execution node in the at least one first execution node has a training exception, the at least one first execution node is controlled to abort training, at least one second execution node is selected from the plurality of worker nodes, and the at least one second execution node is instructed to continue executing the training task, so that the second execution node acquires training process data of the second execution node in a storage server or a memory of the second execution node according to the instruction of the control node to continue executing the training task, and continues to execute the training task based on the training process data.

10. A method for distributed training of a model, the method comprising: The method is applied to an execution node, and the execution node is any one of a plurality of worker nodes. The method comprises: According to the instruction of the control node to start executing a training task, the training task is executed, and training process data of the execution node is stored in a storage server and a memory of the execution node during training; or According to the instruction of the control node to continue executing the training task, training process data of the execution node is acquired from the storage server or the memory of the execution node, and the training task is continued based on the training process data.

11. An electronic device, comprising: The electronic device comprises a memory and a processor; The memory and the processor are coupled; The memory is used to store program instructions; The processor is used to call the program instructions, so that the electronic device executes the distributed training method of the model according to claim 9 or 10. The electronic device comprises a memory and a processor; The memory and the processor are coupled; The memory is used to store program instructions; The processor is used to call the program instructions, so that the electronic device executes the distributed training method of the model according to claim 9 or 10.

Images