Model training method, model training system, computing device, computer readable storage medium, and computer program product

Abstract

Embodiments of the present disclosure provide a model training method, a model training system, a computing device, a computer readable storage medium, and a computer program product. The model training method is applied to a control node in the model training system. The model training system comprises the control node and a plurality of training nodes, wherein the plurality of training nodes are used for training a target model. The model training method comprises: in a training process of the target model, separately obtaining node states of the plurality of training nodes; when it is determined, on the basis of the node state, that a training node is an abnormal training node, on the basis of the node type and the node state of the abnormal training node, determining a training node to be restarted among the plurality of training nodes; and triggering a restart operation for the training node to be restarted, wherein the restart operation is used for restarting training of the training node to be restarted for the target model. Separate restart of a training node is implemented, thereby avoiding overhead caused by restarting all training nodes in a restart cluster and improving the node restart efficiency and the model training efficiency.

Description

[0001]The present disclosure claims priority to Chinese Patent Application No. 202410471937.0, filed on April 17, 2024, with the Chinese Patent Office, entitled "Model Training Method, Model Training System, Computing Device, Computer-Readable Storage Medium, and Computer Program Product", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD Embodiments of the present disclosure relate to the technical field of computer technology, and particularly to a model training method, a model training system, a computing device, a computer-readable storage medium, and a computer program product. BACKGROUND With the development of computer technology, a deep learning framework (TensorFlow) has gradually become one of the frameworks that are popular and widely used in the field of artificial intelligence. TensorFlow provides powerful and flexible tools and libraries, which enable researchers and developers to conveniently design, build, train, validate, and deploy various types of machine learning models, especially deep neural network models. At present, in the distributed training task of TensorFlow, the training progress is usually saved by using a checkpoint (CKPT, Checkpoint) mechanism to save model parameters to shared storage, so as to avoid the loss of the training progress caused by the failure of the training nodes in the training cluster. However, after the failure of the distributed training task, all the training nodes involved in the training task need to be restarted, which results in a large training overhead and extremely low training efficiency. Therefore, there is an urgent need for an efficient model training scheme. SUMMARY In view of the above, embodiments of the present disclosure provide a model training method. One or more embodiments of the present disclosure also relate to a model training apparatus, a model training system, a computing device, a computer-readable storage medium, and a computer program product, to solve the technical defects in the prior art. According to a first aspect of embodiments of the present disclosure, a model training method is provided, which is applied to a control node in a model training system. The model training system includes the control node and a plurality of training nodes. The plurality of training nodes are used to train a target model. The model training method includes: obtaining node states of the plurality of training nodes respectively in a training process of the target model; in a case where it is determined according to the node states that a training node is an abnormal training node, determining a to-be-restarted training node in the plurality of training nodes according to a node type and a node state of the abnormal training node; and triggering a restart operation on the to-be-restarted training node, wherein the restart operation is used to restart the to-be-restarted training node to train the target model.According to a second aspect of the embodiments of the present disclosure, a model training apparatus is provided, which is applied to a control node in a model training system, the model training system comprising the control node and a plurality of training nodes, the plurality of training nodes being configured to train a target model, the model training apparatus comprising: an obtaining module configured to obtain node states of the plurality of training nodes respectively during a training process of the target model; a determining module configured to, in a case where it is determined according to the node states that a training node is an abnormal training node, determine a to-be-restarted training node in the plurality of training nodes according to a node type and the node state of the abnormal training node; and a triggering module configured to trigger a restart operation on the to-be-restarted training node, wherein the restart operation is configured to restart the training of the target model by the to-be-restarted training node. According to a third aspect of the embodiments of the present disclosure, a model training system is provided, comprising a control node and a plurality of training nodes, the plurality of training nodes being configured to train a target model; the control node is configured to obtain node states of the plurality of training nodes respectively during a training process of the target model; in a case where it is determined according to the node states that a training node is an abnormal training node, a to-be-restarted training node in the plurality of training nodes is determined according to a node type and the node state of the abnormal training node; and a restart operation on the to-be-restarted training node is triggered, wherein the restart operation is configured to restart the training of the target model by the to-be-restarted training node. According to a fourth aspect of the embodiments of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, so as to implement the steps of the model training method provided in the first aspect. According to a fifth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the model training method provided in the first aspect. According to a sixth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the model training method provided in the first aspect.The model training method provided by one embodiment of the present disclosure is applied to a control node in a model training system, the model training system includes the control node and a plurality of training nodes, the plurality of training nodes are used to train a target model, and the model training method includes the following steps: obtaining node states of the plurality of training nodes respectively in a training process of the target model; in the case that it is determined according to the node states that a training node is an abnormal training node, determining a to-be-restarted training node in the plurality of training nodes according to a node type and a node state of the abnormal training node; and triggering a restart operation on the to-be-restarted training node, wherein the restart operation is used to restart the training of the to-be-restarted training node on the target model. By determining the to-be-restarted training node according to the node type and the node state of the abnormal training node, the individual restart of the training node is realized, the overhead caused by restarting all training nodes in a cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is an architecture diagram of a model training system according to one embodiment of the present disclosure; FIG. 2 is a redundant backup diagram of a model checkpoint in a model training method according to one embodiment of the present disclosure; FIG. 3 is a recovery diagram of a model checkpoint in a model training method according to one embodiment of the present disclosure; FIG. 4 is an architecture diagram of another model training system according to one embodiment of the present disclosure; FIG. 5 is a flowchart of a model training method according to one embodiment of the present disclosure; FIG. 6 is a process flowchart of a model training method according to one embodiment of the present disclosure; FIG. 7 is a work flowchart of a management-type training node in a model training method according to one embodiment of the present disclosure; FIG. 8 is a communication process diagram of checkpoint generation in a model training method according to one embodiment of the present disclosure; FIG. 9 is a work flowchart of a training-type training node in a model training method according to one embodiment of the present disclosure; FIG. 10 is a structural diagram of a model training apparatus according to one embodiment of the present disclosure; and FIG. 11 is a structural block diagram of a computing device according to one embodiment of the present disclosure. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure can be practiced without the specific details that are set forth in the following description, and it is understood that persons having ordinary skill in the art can make and use modifications to the present disclosure without departing from the scope of the present disclosure. The terminology used in one or more embodiments of the present disclosure is merely for the purpose of describing particular embodiments and is not intended to limit one or more embodiments of the present disclosure. The singular forms "a", "an" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.It should also be understood that the term “and / or” used in one or more embodiments of the present disclosure means and includes any or all of the possible combinations of one or more associated listed items. It should be understood that although the terms first, second, etc. can be used in one or more embodiments of the present disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, first can also be referred to as second, and similarly, second can also be referred to as first. Depending on the context, the word “if” as used herein can be interpreted as “when” or “upon” or “in response to determining”. In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation interface for the user to choose authorization or refusal. In one or more embodiments of the present disclosure, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, billions, tens of billions, hundreds of billions or even tens of billions of model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model), which is pre-trained through a large amount of unlabeled corpus, and outputs a pre-training model with hundreds of millions of parameters. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability, for example, large-scale language model (LLM, Large Language Model), multi-modal pre-training model (multi-modal pre-training model), etc.In practical applications, a large model can be applied to different tasks by fine-tuning a pre-trained model with a small amount of samples. The large model can be widely applied in natural language processing (NLP, Natural Language Processing) and computer vision, and can be applied to computer vision tasks such as visual question answering (VQA, Visual Question Answering), image captioning (IC, Image Captioning), image generation, and natural language processing tasks such as text-based sentiment classification, text summarization generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. First, the nomenclature related to one or more embodiments of the present disclosure is explained. Computational graph: a graph structure in computer science, which is a directed acyclic graph, used in deep learning to represent model computations. The nodes in the graph represent the types of computations, and the directed edges represent the dependencies of the computations. Checkpoint: used in deep learning to save task states, including model parameters, etc. In the embodiments of the present disclosure, the checkpoint can also save the training progress information of the model training data. Fault tolerance: refers to the process of automatically creating a new training node and loading model parameters according to training progress information to recover the training task when a training node fails in a training task. In a typical distributed training task using a parameter server in TensorFlow, the checkpoint mechanism is usually used to save the training progress by saving the model parameters to shared storage to avoid the loss of the entire training progress caused by the failure of the state node in the training cluster. After the training task fails, the model parameters need to be loaded from the latest checkpoint to recover the model training. However, after the distributed training task fails, all training nodes involved in the training task need to be restarted, resulting in a large training overhead and extremely low training efficiency. To solve the above problems, the embodiments of the present disclosure propose a deep learning training system fault tolerance method based on a distributed model training fault tolerance system, the core of which is: an automatic fault tolerance mechanism for distributed training is proposed, which supports the individual restart of failed computing nodes, avoids the overhead caused by restarting the entire training task, and realizes automatic fault tolerance for large-scale distributed deep learning training tasks.Specifically, the embodiment of the present disclosure provides a model training method, which is applied to a control node in a model training system, the model training system comprising the control node and a plurality of training nodes, the plurality of training nodes being configured to train a target model, the model training method comprising: obtaining node states of the plurality of training nodes respectively in a training process of the target model; in a case where it is determined according to the node states that a training node is an abnormal training node, determining a to-be-restarted training node in the plurality of training nodes according to a node type and the node state of the abnormal training node; triggering a restart operation on the to-be-restarted training node, wherein the restart operation is configured to restart the training of the to-be-restarted training node on the target model. By determining the to-be-restarted training node according to the node type and the node state of the abnormal training node, the individual restart of the training node is realized, the overhead caused by restarting all training nodes in the cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved. In the present disclosure, a model training method is provided, and the present disclosure also relates to a model training device, a model training system, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the embodiments below. Referring to FIG. 1, FIG. 1 shows an architecture diagram of a model training system provided by an embodiment of the present disclosure, the model training system can comprise a control node 102 and a plurality of training nodes 104, the plurality of training nodes 104 being configured to train a target model; the control node 102 is configured to obtain node states of the plurality of training nodes 104 respectively in a training process of the target model; in a case where it is determined according to the node states that a training node 104 is an abnormal training node, determining a to-be-restarted training node in the plurality of training nodes 104 according to a node type and the node state of the abnormal training node; triggering a restart operation on the to-be-restarted training node, wherein the restart operation is configured to restart the training of the to-be-restarted training node on the target model. It should be noted that, in a typical distributed training task of TensorFlow using a parameter server, due to the limited access bandwidth of shared storage, loading model parameters from the checkpoint in the shared storage will consume a long time, causing idle computing resources and wasting the computing power of the training cluster. In order to solve this problem, the embodiment of the present disclosure provides a redundant backup and efficient recovery mechanism for model parameters in a training cluster based on a computation graph, which realizes the rapid recovery of the training task through the training data checkpoint mechanism and the redundant backup of the model parameter checkpoint in the cluster.In an optional embodiment of the present disclosure, the model training system further comprises a shared storage and a parameter storage subgraph, the training nodes comprise a first service-type training node and a second service-type training node, the second service-type training node is a backup node of the first service-type training node; the shared storage is configured to store a model checkpoint of the target model; and the parameter storage subgraph is configured to copy the model checkpoint from the shared storage to the first service-type training node, and copy the model checkpoint stored in the first service-type training node to the second service-type training node. Specifically, the model checkpoint can be referred to as a model parameter checkpoint. The parameter storage subgraph is a kind of calculation graph, and the parameter storage subgraph comprises a local backup checkpoint subgraph (Backup Local CKPT), a local checkpoint sending subgraph (Send Local CKPT) and a remote backup checkpoint subgraph (Backup Remote CKPT). It should be noted that in the embodiment of the present disclosure, the model checkpoint redundancy backup and loading between the service-type training nodes are realized by modifying the storage (Save) subgraph and the recovery (Restore) subgraph of the target model. The asynchronous cache model checkpoint running hook (Asynchronous Cache CKPT Runner Hook) interface is implemented to realize the asynchronization of the model parameter checkpoint redundancy backup, thereby avoiding the influence of the backup process on the training. Referring to FIG. 2, FIG. 2 shows a redundancy backup schematic diagram of a model checkpoint in a model training method according to an embodiment of the present disclosure. As shown in FIG. 2, FIG. 2 shows a process in which a model parameter 1 stored in the first service-type training node in the latest model checkpoint is redundantly backed up to the second service-type training node in the task cluster. This process is asynchronous with the model training process. Specifically, the local backup checkpoint subgraph on the first service-type training node is responsible for copying the model parameter of the latest model checkpoint in the shared storage to the first service-type training node, thereby realizing the local redundancy backup of the model parameter of the first service-type training node. Then, the local checkpoint sending subgraph on the first service-type training node copies a copy of the local redundancy backup of the model checkpoint 1 of the first service-type training node to the remote backup checkpoint subgraph on the second service-type training node. The remote backup checkpoint subgraph on the second service-type training node receives the redundancy backup of the model checkpoint 1 of the first service-type training node by using the high-speed network interconnection within the task cluster, and saves it locally (on the second service-type training node), thereby realizing the backup of the model checkpoint 1 on the first service-type training node on other service-type training nodes.In this way, each service type training node has its own model checkpoint redundancy backup (such as the second service type training node has its own model parameter 2), and the model checkpoint redundancy backup of other service type training nodes is also saved, thereby realizing the redundant storage of the model checkpoint within the task cluster. Through the redundant storage of the model parameter within the task cluster, the model checkpoint can be quickly obtained through the high-speed interconnection within the task cluster according to the scheme of the embodiment of the present disclosure, even if the redundant backup of the own model parameter does not exist in the task cluster, the model checkpoint can also be obtained through the shared storage, and it is ensured that the task can be recovered. In an optional embodiment of the present disclosure, the model training system further includes a shared storage and a parameter recovery subgraph, the training node includes a first service type training node and a second service type training node, the second service type training node is a backup node of the first service type training node; the shared storage is used to store the model checkpoint of the target model; the parameter recovery subgraph is used to copy the model checkpoint stored in the second service type training node to the first service type training node in the case that the first service type training node fails to restart, or copy the model checkpoint stored in the shared storage to the first service type training node. Specifically, the parameter recovery subgraph is a kind of calculation graph, and the parameter recovery subgraph includes a local receive checkpoint subgraph (Receive Local CKPT), a local checkpoint subgraph (Get Local CKPT From Shared Storage) obtained from the shared memory, and a remote send checkpoint subgraph (Send Remote CKPT). Referring to FIG. 3, FIG. 3 shows a recovery diagram of a model checkpoint in a model training method according to an embodiment of the present disclosure, as shown in FIG. 3, FIG. 3 shows the recovery process of the model checkpoint when the first service type training node fails.Since the first service type training node fails to restart, the first service type training node has no model checkpoint redundancy backup locally, and the second service type training node is a redundant backup node of the model checkpoint of the first service type training node. First, the second service type training node checks whether the redundant backup of the model parameter 1 of the first service type training node saved by itself is the latest. If yes, the model parameter 1 is sent to the local receiving checkpoint subgraph of the first service type training node through a remote sending checkpoint subgraph. After the first service type training node receives the model parameter 1, the model parameter 1 of the first service type training node is initialized using the model checkpoint 1. If there is no model parameter on the second service type training node, the first service type training node will obtain the latest checkpoint of the model parameter from the shared memory by obtaining the local checkpoint subgraph from the shared memory, and perform initialization operation O. In this way, if there is a redundant backup of the model checkpoint of the first service type training node in the task cluster, the first service type training node can obtain the model checkpoint through the high-speed interconnection in the task cluster. Even if there is no redundant backup of the model checkpoint of the first service type training node in the task cluster, the model checkpoint can be obtained through the shared storage, so as to ensure that the task can be completed. As for the second service type training node without failure, the model parameter 2 can be initialized and recovered directly from the local backup model checkpoint 2, without the need to obtain it through the slow shared storage, thereby improving the recovery efficiency of the model checkpoint. Referring to FIG. 4, FIG. 4 shows an architecture diagram of another model training system provided by an embodiment of the present disclosure. The model training system can include a task cluster 100 (Job Cluster), a cluster management unit 200 (Compute Cluster Controller) and shared storage 300 (Shared Storage). The task cluster 100 includes a control node 102 (Job Controller Node) and a training node 104 (Job Compute Node). The control node 102 includes a training data management unit 1022 (Data Manager) and a training node management unit 1024 (Compute Nodes Manager). The training node 104 includes a management type training node 1042 (Chief), a service type training node 1044 (PS, Parameter Server) and a training type training node 1046 (Worker). The cluster management unit can be referred to as a compute cluster management unit, which is used to manage the nodes in the entire task cluster, and is responsible for the creation, destruction, running state sensing and the like of the control node and the training node.The shared storage is storage that can be linked to the control node and the training node in a manner such as a network, and is responsible for storing model training data and checkpoints containing model parameters and training progress information, where the linking manner is, for example, remote procedure call (RPC). The training data management unit is used for functions such as model training data sharding, model training data distribution, and model training data consumption progress recording. The training node management unit is responsible for automatic fault tolerance of model training tasks, is used to obtain the node state of the training node in the current task cluster from the cluster management unit, and communicates with the cluster management unit according to the fault tolerance strategy, to implement different fault tolerance actions (such as training node individual restart, task restart, and task termination) according to the node state and the node type. Therefore, the model training method provided in the embodiments of the present disclosure can be specifically applied to the training node management unit in the control node. The training node can be referred to as a computing node, and is a node participating in model parameter updating. The management-type training node is responsible for management tasks such as loading and saving of model parameters, and also participates in model parameter training. The service-type training node is a parameter server responsible for storing model parameters. The training-type training node is only responsible for model training parameters. It should be noted that the model training system provided in the embodiments of the present disclosure is generally deployed on a server, but in other embodiments of the present disclosure, the model training system can also be deployed on a client. In other embodiments, the model training system provided in the embodiments of the present disclosure can also be distributed and deployed on the client and the server. The client can be a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Languages, Hyper Text Markup Languages, version 5) application, or a light application (also known as a small program, a lightweight application program), or a cloud application, etc. The client can be based on the software development kit (SDK) of the corresponding service provided by the server, such as based on the real-time communication (RTC) SDK development, etc. The client can be deployed in an electronic device and needs to depend on the device or some APP in the device to run, etc. The electronic device may, for example, have a display screen and support information browsing, etc., and may be a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc.In the electronic device, various other types of applications can also be configured, such as human-computer interaction type applications, model training type applications, text processing type applications, web browser applications, shopping type applications, search type applications, instant messaging tools, email clients, social platform software, etc. The service end can include servers that provide various services, such as servers that provide communication services for multiple clients, servers that provide support for models used on clients for background training, servers that process data sent by clients, etc. It should be noted that the service end can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a block chain. The server can also be a cloud server of a cloud service, a cloud database, a cloud computing, a cloud function, a cloud storage, a network service, a cloud communication, a middleware service, a domain name service, a security service, a content distribution network (CDN, Content Delivery Network), and a big data and artificial intelligence platform, etc. Basic cloud computing services, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. Next, the model training system deployed on the service end is taken as an example for description. The model training system can establish a communication connection with multiple clients, and in the model training scenario, the model training system is used to provide model training services between multiple clients, and the multiple clients can be used as sending ends or receiving ends, and communication is realized through the model training system. The user can interact with the model training system through the client to receive data sent by other clients, or send data to other clients, etc. In the model training scenario, the user can publish a data stream to the model training system through the client, the model training system trains the model according to the data stream, generates model parameters, and pushes the model parameters to other clients that establish communication. The connection between the client and the model training system is established through the network. The network provides a medium for the communication link between the client and the model training system. The network can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc. The data transmitted by the client can need to be processed through encoding, transcoding, compression, etc. before being published to the model training system. Referring to FIG. 5, FIG. 5 shows a flowchart of a model training method according to an embodiment of the present disclosure. The model training method is applied to a control node in a model training system, and the model training system includes the control node and multiple training nodes. The multiple training nodes are used to train a target model, and specifically include the following steps:In one or more embodiments of the present disclosure, the model training system trains a target model according to model training data in response to a training request for the target model. During the training of the target model, the control node can obtain the node states of the plurality of training nodes respectively, so as to determine whether there is an abnormal training node based on the node states. Specifically, the target model refers to a deep learning model, including but not limited to a convolutional neural network (CNN), a recurrent neural network (RNN), and a long short-term memory network (LSTM). The target model can be a large model. There are various training methods for the target model, including but not limited to supervised fine-tuning (SFT), transfer learning, and adversarial training. The actual situation is selected, and the embodiments of the present disclosure do not make any limitation. The node state of the training node refers to the working state of each training node participating in the training process in a distributed training environment. The node state can be divided into a normal state and an abnormal state, and the abnormal state can include a restart state and a restart failure state. When the training node is in the restart state, it means that the training node is in the restart process due to maintenance, upgrade, failure, etc. When the training node is in the restart failure state, it means that the training node fails to successfully recover normal operation and service after attempting to restart. In actual applications, there are various ways to obtain the node states of the plurality of training nodes by the control node during the training of the target model, and the actual situation is selected, and the embodiments of the present disclosure do not make any limitation. In one possible implementation of the present disclosure, when the control node itself has the ability to monitor the training nodes, the control node can monitor the training nodes in real time, so as to obtain the node states of the training nodes. In another possible implementation of the present disclosure, when the model training system includes a cluster management unit responsible for the creation, destruction, running state sensing, etc. of the control node and the training node, the control node can obtain the node states of the plurality of training nodes from the cluster management unit during the training of the target model. Step 504: In the case where the training node is determined to be an abnormal training node according to the node state, the to-be-restarted training node in the plurality of training nodes is determined according to the node type and the node state of the abnormal training node.In one or more embodiments of the present disclosure, during the training process of the target model, after the control node obtains the node states of the plurality of training nodes respectively, further, it can be determined whether the training node is an abnormal training node according to the node state, and in the case that the training node is determined to be an abnormal training node according to the node state, the to-be-restarted training node in the plurality of training nodes is determined according to the node type and the node state of the abnormal training node. Specifically, the abnormal training node refers to a training node that cannot work normally, and the reasons for the abnormal training node include but are not limited to insufficient resources and data errors. The node type of the training node includes a management type, a service type and a training type. If the node type of the training node is the management type, it means that the training node is a management-type training node. If the node type of the training node is the service type, it means that the training node is a service-type training node. If the node type of the training node is the training type, it means that the training node is a training-type training node. In an optional embodiment of the present disclosure, the node state includes a restart state, the restart state indicates that the training node is an abnormal training node, and the node type includes the management type and the service type. The above-mentioned determination of the to-be-restarted training node in the plurality of training nodes according to the node type and the node state of the abnormal training node can include the following steps: in the case that the node state is the restart state and the node type is the management type or the service type, obtaining node restart information and task restart information of the abnormal training node; and determining the to-be-restarted training node in the plurality of training nodes according to the node restart information and the task restart information. Specifically, the node restart information includes but is not limited to the number of restarts and the restart duration of the training node, and the task restart information includes but is not limited to the number of restarts and the restart duration of the target model training task. The node restart information and the task restart information are specifically selected according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this. It should be noted that in the case that the node state of the training node is the restart state and the node type is the training type, it is determined whether there is a failed node set in the current restart failure state, and the to-be-restarted training node in the plurality of training nodes is determined according to the determination result and the node type. In the case that the node state of the training node is the restart state and the node type is the management type or the service type, it means that there is a training node being restarted in the model training system, and the training node being restarted is a management-type training node or a service-type training node. At this time, the node restart information and the task restart information of the abnormal training node can be obtained.Further, after obtaining the node restart information and the task restart information of the abnormal training node, the to-be-restarted training node of the plurality of training nodes can be determined according to the node restart information and the task restart information. The determination process can be divided into the following cases: in the case where the node restart information satisfies the node restart condition, it is indicated that the training node can be restarted alone at present, then it is determined whether there is a failed node set in a restart failure state, and the to-be-restarted training node in the plurality of training nodes is determined according to the determination result and the node type; in the case where the node restart information does not satisfy the node restart condition, it is determined whether the task restart information satisfies the task restart condition: in the case where the node restart information does not satisfy the node restart condition and the task restart information does not satisfy the task restart condition, it is indicated that the node restart is not supported by the current model training system, then the training task of the target model is marked as failed; in the case where the node restart information does not satisfy the node restart condition and the task restart information satisfies the task restart condition, the plurality of training nodes are determined as the to-be-restarted training node, at this time, the control unit triggers the restart of the entire model training task. By determining the to-be-restarted training node according to the node type and the node state of the abnormal training node, the training node is restarted alone, the overhead of restarting all training nodes in the cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved. In an optional embodiment of the present disclosure, the above determination of the to-be-restarted training node in the plurality of training nodes according to the node restart information and the task restart information can include the following steps: in the case where the node restart information does not satisfy the node restart condition and the task restart information satisfies the task restart condition, the plurality of training nodes are determined as the to-be-restarted training node. Specifically, the node restart condition can be that the restart time length of the training node does not exceed a preset time length, or the restart number of the training node does not exceed a preset number threshold. The task restart condition can be that the restart number of the model training task does not exceed a preset number threshold. It should be noted that in the case where the node restart information does not satisfy the node restart condition and the task restart information satisfies the task restart condition, it is indicated that the abnormal training node is timed out, but the entire model training task does not exceed the task restart condition, at this time, the entire model training task can be restarted. Further, after the plurality of training nodes are determined as the to-be-restarted training node, the training progress information of the model checkpoint and the model training data can be obtained, so that the to-be-restarted training node is restarted based on the model checkpoint and the training progress information. It should be noted that the version of the obtained model checkpoint and the training progress information of the model training data is consistent, which means that the model checkpoint and the model training data are synchronized, so that the problem of overfitting of the model can be avoided.According to the scheme of the embodiment of the present disclosure, in the case that the node restart information does not satisfy the node restart condition and the task restart information satisfies the task restart condition, the plurality of training nodes are determined as the to-be-restarted training nodes, that is, even if the training nodes cannot be restarted individually, the abnormal recovery of the model training task is realized by restarting the entire model training task. In an optional embodiment of the present disclosure, the node state includes a restart failure state, and the restart failure state represents that the training node is an abnormal training node; the above determining the to-be-restarted training nodes from the plurality of training nodes according to the node type and the node state of the abnormal training node can include the following steps: in the case that the node state is the restart failure state, obtaining the node restart information of the abnormal training node; in the case that the node restart information satisfies the node restart condition, determining the to-be-restarted training nodes from the plurality of training nodes according to the node type. Exemplarily, taking the node restart information as the number of restarts and the node restart condition as the number of restarts of the training node being not more than a preset number threshold as an example, in the case that the node state of the abnormal training node is the restart failure state, the number of restarts of the abnormal training node can be obtained, and in the case that the number of restarts is not more than the preset number threshold, the to-be-restarted training nodes from the plurality of training nodes are determined according to the node type. In the case that the number of restarts is more than the preset number threshold, it is indicated that the current model training system does not support node restart, and then the training task of the target model is marked as failed. In actual application, before obtaining the node restart information of the abnormal training node in the case that the node state is the restart failure state, it can be judged whether there is a failed node set, if not, it is returned to judge whether the model training task is completed; if yes, the node restart information of the abnormal training node is obtained. Further, it can be judged whether there is the node restart information of the abnormal training node in the failed node set that does not satisfy the node restart condition, if yes, the training task of the target model is marked as failed; if not, an abnormal training node is selected from the failed node set, the to-be-restarted training nodes are determined according to the node type of the abnormal training node, and until the failed node set is empty, it is returned to judge whether the model training task is completed. According to the scheme of the embodiment of the present disclosure, in the case that the node state is the restart failure state, the node restart information of the abnormal training node is obtained; in the case that the node restart information satisfies the node restart condition, the to-be-restarted training nodes from the plurality of training nodes are determined according to the node type. The individual restart of the training node is realized, the overhead caused by restarting all training nodes in the cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved.In an optional embodiment of the present disclosure, the node type includes a management type, a service type and a training type; and the determining the to-be-restarted training node from the plurality of training nodes according to the node type can include the following steps: in a case where the node type is the service type, determining the abnormal training node as the to-be-restarted training node; in a case where the node type is the training type, determining the abnormal training node as the to-be-restarted training node; and in a case where the node type is the management type, determining the abnormal training node and the training node of the training type as the to-be-restarted training node. It should be noted that, in a case where the node type is the management type, it indicates that the abnormal training node is a management-type training node, if only the management-type training node is restarted and the training-type training node is not restarted, then the restarted management-type training node will repeatedly send part of the model training data to the training-type training node, resulting in redundant training of the training-type training node on part of the model training data, and further, the problem of model overfitting can occur. Therefore, in a case where the node type is the management type, not only the abnormal training node is determined as the to-be-restarted training node, but also the training node of the training type is determined as the to-be-restarted training node. By applying the scheme of the embodiment of the present disclosure, in a case where the node type is the service type, the abnormal training node is determined as the to-be-restarted training node; in a case where the node type is the training type, the abnormal training node is determined as the to-be-restarted training node; and in a case where the node type is the management type, the abnormal training node and the training node of the training type are determined as the to-be-restarted training node. The separate restart of the training node is realized, and at the same time, the model overfitting is avoided, and the training efficiency and accuracy of the model are improved. Step 506: triggering a restart operation on the to-be-restarted training node, wherein the restart operation is used to restart the training of the target model by the to-be-restarted training node. In one or more embodiments of the present disclosure, during the training process of the target model, the control node respectively acquires the node states of the plurality of training nodes; in a case where the training node is determined as the abnormal training node according to the node state, after the to-be-restarted training node from the plurality of training nodes is determined according to the node type and the node state of the abnormal training node, further, the control node can trigger the restart operation on the to-be-restarted training node. In actual application, the restart operation on the to-be-restarted training node can be performed by the control node, that is, the control node can directly restart the training of the target model by the to-be-restarted training node, or can be performed by a cluster management unit in the model training system, that is, a restart request for the to-be-restarted training node is sent to the cluster management unit, so that the cluster management unit restarts the training of the target model by the to-be-restarted training node.According to the scheme of the embodiment of the present disclosure, the training node to be restarted is determined according to the node type and the node state of the abnormal training node, the separate restart of the training node is realized, the overhead caused by restarting all training nodes in the cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved. In an optional embodiment of the present disclosure, the model training system further comprises a cluster management unit; the above-mentioned obtaining the node state of each training node during the training process of the target model can comprise the following steps: obtaining the node state of each training node from the cluster management unit during the training process of the target model; the triggering of the restart operation of the training node to be restarted can comprise the following steps: sending a restart request for the training node to be restarted to the cluster management unit, so that the cluster management unit restarts the training of the target model by the training node to be restarted. It should be noted that when the control node itself has the capability of monitoring the training node, the control node can monitor the training node in real time during the training process of the target model, and directly obtain the node state of the training node. When the control node itself does not have the capability of monitoring the training node, the control node can obtain the node state of each training node from the cluster management unit during the training process of the target model. Similarly, when the control node has the capability of restarting the training node, after determining the training node to be restarted in the plurality of training nodes, the control node can directly restart the training of the target model by the training node to be restarted. When the control node itself does not have the capability of restarting the training node, after determining the training node to be restarted in the plurality of training nodes, the control node can send a restart request for the training node to be restarted to the cluster management unit, so that the cluster management unit restarts the training of the target model by the training node to be restarted, thereby realizing the flexible restart of the training node. Referring to FIG. 6, FIG. 6 shows a processing process flowchart of a model training method provided by an embodiment of the present disclosure. As shown in FIG. 6, after the model training system starts running, a user submits a distributed training task. The cluster management unit in the model training system creates a control node and training nodes according to the task configuration file (the number of training nodes is configured) sent by the user, and starts model training. The training node management unit in the model training system judges whether the task is completed: if yes, the task is marked as successful, and the task is ended; if not, the training node management unit obtains the node state of each training node from the cluster management unit; the training node management unit judges whether there is a node being restarted (that is, the node state of the training node is a restart state): if not, the training node management unit judges whether there is a failed node set at present; if yes, the training node management unit judges whether the restarted training node comprises a service type training node (PS) / a management type training node (Chief).Next, the above two cases are described in detail: determine whether there is a set of failed nodes currently: if not, the training node management unit returns to determine whether the task is completed; if yes, the training node management unit determines whether there is a node in the set that exceeds the number of restarts: if yes, the training node management unit marks the task as failed and ends the task; if not, the training node management unit determines whether the failed node set is empty: if yes, the training node management unit returns to determine whether the task is completed; if not, the training node management unit selects a node from the failed node set, determines whether the node is a training worker (Worker) : if yes, the training node management unit determines the node as a training node to be restarted, and requests the cluster management unit to restart the node, returns to the step of determining whether the failed node set is empty; if not, the training node management unit determines whether the node is a PS: if yes, the training node management unit determines the node as a training node to be restarted, and requests the cluster management unit to restart the node, returns to the step of determining whether the failed node set is empty; if not, the training node management unit determines the node and all workers as training nodes to be restarted, requests the cluster management unit to restart all workers, and requests the cluster management unit to restart the node, returns to the step of determining whether the failed node set is empty; determine whether the restarted training node contains PS / Chief: if not, the training node management unit determines that the failed node set is empty; if yes, the training node management unit determines whether the PS / Chief node restarts timeout: if not, the training node management unit determines that the failed node set is empty; if yes, the training node management unit determines whether the current number of task restarts exceeds the maximum number of task restarts: if yes, the training node management unit marks the task as failed and ends the task; if not, the training node management unit requests the cluster management unit to restart the entire distributed training task. It should be noted that in a typical TensorFlow distributed training task using a parameter server, the checkpoint mechanism only saves the model parameters, and lacks training data consumption progress records, resulting in that part of the training data participates in training again after the training task is restored, which is easy to cause the model to overfit. In order to solve this problem, the present embodiment proposes a training data consumption progress and model parameter synchronization checkpoint generation mechanism to ensure that the training progress information and the version of the model parameters are consistent when the training task is restored, and to avoid causing the model to overfit.In an optional embodiment of the present disclosure, the control node comprises a training data management unit, the training data management unit is configured to store training progress information of model training data, the training node comprises a management type training node of a node type of a management type and a training type training node of a node type of a training type; the model training method can further comprise the following steps: in a training process of a target model, in response to a data stop distribution request sent by the management type training node, stopping sending model training data to the training type training node, and sending training progress information to the management type training node, so that the management type training node stores the training progress information. Specifically, the training progress information of the model training data can be referred to as training data consumption progress information, or can be referred to as a model training data checkpoint. The training progress information can be a data identifier of model training data that has been used, or can be a data identifier of model training data that has not been used, which is specifically selected according to actual conditions, and the embodiment of the present disclosure does not make any limitation thereon. It should be noted that the model training method proposed in the embodiment of the present disclosure adopts a centralized model training data distribution mechanism, and the training data management unit in the control node saves data information of the model training data, such as data slice size, training progress information and the like. The management type training node and the training type training node comprise a data slice request unit (Global Slice Queue), and the data slice request unit is configured to request a data slice of the model training data from the training data management unit in the model training process. Meanwhile, the management type training node further comprises a data slice monitoring unit (Global Slice Queue Listener), and the data slice monitoring unit is configured to communicate with the training data management unit, and synchronize the model checkpoint and the training progress information in the training data management unit. The functions of the data slice monitoring unit include sending a data stop distribution request, obtaining training progress information of current model training data, sending a data distribution request and the like. In actual application, in the training process of the target model, the management type training node triggers a checkpoint storage hook (Checkpoint Saver Hook) interface to detect whether the model checkpoint needs to be saved at present, if not, the model checkpoint is not saved; if yes, subsequent steps are executed. In the case that the model checkpoint needs to be saved at present, the checkpoint storage hook interface can call a before_save method of the data slice monitoring unit to send a data stop distribution request to the training data management unit. After the training data management unit receives the data stop distribution request, the training data management unit stops sending model training data to the training type training node, and returns the training progress information of the training data management unit at present to the management type training node.With the scheme of the embodiments of the present disclosure, in the training process of the target model, in response to the data stop distribution request sent by the management type training node, the sending of the model training data to the training type training node is stopped, and the training progress information is sent to the management type training node, so that the management type training node stores the training progress information, the consumption progress of the model training data is recorded, the consistency of the training progress information and the version of the model parameters is ensured when the training task is resumed, the problem of overfitting of the model is avoided, and the precision of the model training is improved. Referring to FIG. 7, FIG. 7 shows a working flowchart of the management type training node in a model training method according to an embodiment of the present disclosure. As shown in FIG. 7, after the model training task of the target model starts, the management type training node receives the model training data sent by the training data management unit, and trains the target model by using the model training data. The management type training node judges whether the model training is completed. If yes, the model training task is ended. If no, the target model is continuously trained. In the training process of the target model, the management type training node judges whether the current checkpoint needs to be saved. If no, the model checkpoint saving is not performed, and the step of judging whether the model training is completed is returned. If yes, the management type training node sends a data stop distribution request to the training data management unit in the control node, and requests the training progress information of the model training data. After the training data management unit receives the data stop distribution request, the sending of the model training data to the training type training node is stopped, and the current training progress information of the training data management unit is returned to the management type training node. After the management type training node receives the training progress information, the training progress information is saved, and a checkpoint obtaining request is sent to the service type training node. In response to the checkpoint obtaining request sent by the management type training node, the service type training node sends the model checkpoint to the management type training node. After the management type training node receives the model checkpoint, the model parameters are saved. After the training progress information and the model checkpoint are saved, the management type training node continues to send a data distribution request to the training data management unit, requests to continue to distribute the model training data, and returns to the step of judging whether the model training is completed.In an optional embodiment of the present disclosure, the training node comprises a service-type training node of the node type of service type; after the control node stops sending the model training data to the training-type training node and sends the training progress information to the management-type training node in response to the data stop distribution request sent by the management-type training node, the method can further comprise the following steps: in response to the data distribution request sent by the management-type training node, the model training data is sent to the training-type training node and the management-type training node according to the training progress information, wherein the data distribution request is sent by the management-type training node after storing the training progress information and the model checkpoint, the model checkpoint is sent to the management-type training node by the service-type training node in response to the checkpoint acquisition request sent by the management-type training node, and the training progress information and the model checkpoint version stored by the management-type training node are consistent. It should be noted that after the training data management unit in the control node stops sending the model training data to the training-type training node and sends the training progress information to the management-type training node, the management-type training node receives the training progress information, first saves the training progress information, and then saves the model checkpoint acquired from the service-type training node. After the training progress information and the model checkpoint are saved, the checkpoint storage hook calls the after_save method of the data shard monitoring unit, sends the data distribution request of continuing to distribute data to the training data management unit, and the training data management unit receives the data distribution request, sends the model training data to the training-type training node and the management-type training node according to the training progress information. Referring to FIG. 8, FIG. 8 shows a communication process schematic diagram of checkpoint generation in a model training method according to an embodiment of the present disclosure. As shown in FIG. 8, the checkpoint generation process can be divided into a model checkpoint generation process and a training data checkpoint generation process. The model checkpoint generation process: the training-type training node trains the target model to obtain the model parameters of the target model, and the service-type training node saves the model parameters. The management-type training node sends a checkpoint acquisition request to the service-type training node. The service-type training node sends the model checkpoint to the management-type training node in response to the checkpoint acquisition request sent by the management-type training node. After receiving the model checkpoint, the management-type training node saves the model parameters; the training data checkpoint generation process: the management-type training node sends a data stop distribution request to the training data management unit in the cluster management unit and requests the training progress information of the model training data. The training data management unit stops sending the model training data to the training-type training node and returns the current training progress information of the training data management unit to the management-type training node in response to the data stop distribution request; after receiving the training progress information, the management-type training node saves the training progress information.After the management training node saves the training progress information and the model checkpoint, the management training node continues to send a data distribution request to the training data management unit to request to continue distributing the model training data. According to the scheme of the embodiment of the present disclosure, in response to the data distribution request sent by the management training node, the model training data is sent to the training training node and the management training node according to the training progress information, the consumption progress of the model training data is recorded, the consistency of the training progress information and the version of the model parameters is ensured when the training task is recovered, the problem of overfitting of the model is avoided, and the accuracy of the model training is improved. In an optional embodiment of the present disclosure, after the above-mentioned response to the data stop distribution request sent by the management training node, the sending of the model training data to the training training node is stopped, and the training progress information is sent to the management training node, the following steps can be further included: in response to the data distribution request sent by the training training node, the retry request information is sent to the training training node until the data distribution request sent by the management training node is received, and the model training data is sent to the training training node according to the training progress information. It should be noted that, in response to the data stop distribution request sent by the management training node, the control node stops sending the model training data to the training training node, and if the data distribution request sent by the management training node is not received, when the training training node requests the data slice from the training data management unit after consuming the model training data slice received before, the control node does not send the data slice to the training training node, but sends the retry request information, so that the training training node waits for a period of time or repeatedly sends the data distribution request until the management training node sends the data distribution request to the control node, and the data distribution to the management training node and the training training node is resumed according to the training progress information. According to the scheme of the embodiment of the present disclosure, in response to the data distribution request sent by the training training node, the retry request information is sent to the training training node until the data distribution request sent by the management training node is received, and the model training data is sent to the training training node according to the training progress information. The consistency of the training progress information and the version of the model parameters is ensured when the training task is recovered, the problem of overfitting of the model is avoided, and the accuracy of the model training is improved.Referring to FIG. 9, FIG. 9 shows a workflow diagram of a training type training node in a model training method according to an embodiment of the present disclosure. As shown in FIG. 9, after the model training task of the target model is started, the training type training node receives the model training data sent by the training data management unit, trains the target model by using the model training data, judges whether the model training is completed, if yes, ends the model training task; if not, waits for a period of time, and sends a data distribution request to the training data management unit. Then, the training data management unit judges whether the training data management unit returns the request retry information, if yes, waits for a period of time; if not, receives the model training data sent by the training data management unit, trains the target model, and returns to the step of judging whether the model training is completed. Corresponding to the method embodiment, the present disclosure further provides a model training device embodiment. FIG. 10 shows a structural schematic diagram of a model training device according to an embodiment of the present disclosure. As shown in FIG. 10, the device is applied to a control node in a model training system, the model training system includes the control node and a plurality of training nodes, the plurality of training nodes are used for training a target model, and the model training device includes: an obtaining module 1002 configured to obtain node states of the plurality of training nodes respectively in a training process of the target model; a determining module 1004 configured to, in a case where the training nodes are determined to be abnormal training nodes according to the node states, determine a to-be-restarted training node in the plurality of training nodes according to a node type and the node state of the abnormal training node; and a triggering module 1006 configured to trigger a restart operation on the to-be-restarted training node, wherein the restart operation is used to restart the to-be-restarted training node for training the target model. Optionally, the model training system further includes a cluster management unit; the obtaining module 1002 is further configured to obtain the node states of the plurality of training nodes respectively from the cluster management unit in the training process of the target model; and the triggering module 1006 is further configured to send a restart request for the to-be-restarted training node to the cluster management unit, so that the cluster management unit restarts the to-be-restarted training node for training the target model. Optionally, the node state includes a restart state, the restart state represents that the training node is an abnormal training node, and the node type includes a management type and a service type; the determining module 1004 is further configured to, in a case where the node state is the restart state and the node type is the management type or the service type, obtain node restart information and task restart information of the abnormal training node; and determine the to-be-restarted training node in the plurality of training nodes according to the node restart information and the task restart information.Optionally, the determining module 1004 is further configured to determine the plurality of training nodes as the to-be-restarted training nodes in a case that the node restart information does not satisfy the node restart condition and the task restart information satisfies the task restart condition. Optionally, the node state comprises a restart failure state, and the restart failure state represents that the training node is an abnormal training node; the determining module 1004 is further configured to, in a case that the node state is the restart failure state, obtain the node restart information of the abnormal training node; and in a case that the node restart information satisfies the node restart condition, determine the to-be-restarted training nodes in the plurality of training nodes according to the node type. Optionally, the node type comprises a management type, a service type and a training type; the determining module 1004 is further configured to, in a case that the node type is the service type, determine the abnormal training node as the to-be-restarted training node; in a case that the node type is the training type, determine the abnormal training node as the to-be-restarted training node; and in a case that the node type is the management type, determine the abnormal training node and the training node of which the node type is the training type as the to-be-restarted training nodes. Optionally, the control node comprises a training data management unit, the training data management unit is configured to store training progress information of model training data, and the training node comprises a management-type training node of which the node type is the management type and a training-type training node of which the node type is the training type; the method further comprises: a first sending module configured to, in a training process of a target model, stop sending the model training data to the training-type training node and send the training progress information to the management-type training node in response to a data stop distribution request sent by the management-type training node, so that the management-type training node stores the training progress information. Optionally, the training node comprises a service-type training node of which the node type is the service type; the method further comprises: a second sending module configured to send the model training data to the training-type training node and the management-type training node according to the training progress information in response to a data distribution request sent by the management-type training node, wherein the data distribution request is sent by the management-type training node after storing the training progress information and a model checkpoint, the model checkpoint is sent to the management-type training node by the service-type training node in response to a checkpoint acquisition request sent by the management-type training node, and the training progress information stored by the management-type training node is consistent with a version of the model checkpoint. Optionally, the method further comprises: a third sending module configured to send retry request information to the training-type training node in response to a data distribution request sent by the training-type training node, and send the model training data to the training-type training node according to the training progress information until a data distribution request sent by the management-type training node is received.According to the scheme of the embodiment of the present disclosure, the training node to be restarted is determined according to the node type and the node state of the abnormal training node, the individual restart of the training node is realized, the overhead caused by restarting all training nodes in the cluster is avoided, the node restart efficiency is improved, and the model training efficiency is further improved. Through the checkpoint generation mechanism that the training data consumption progress is consistent with the model parameters, it is ensured that the data consumption progress and the version of the model parameters are consistent when the training task is recovered, and the problem that the model is over-fitted is avoided. Through the redundant backup and efficient recovery mechanism of the model parameters in the training cluster based on the computation graph, the efficiency of obtaining the model checkpoint is improved without affecting the model training process, and it is ensured that the model training task can be efficiently recovered. The above is a schematic scheme of a model training device of the embodiment. It should be noted that the technical scheme of the model training device belongs to the same concept as the technical scheme of the model training method described above, and the details of the technical scheme of the model training device that are not described in detail can be referred to the description of the technical scheme of the model training method. FIG. 11 shows a structural block diagram of a computing device according to an embodiment of the present disclosure. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130, and a database 1150 is used to save data. The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.The access device 1140 can include one or more of any type of network interface (e.g., network interface card (NIC)) such as an IEEE 802.11 wireless local area networks (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, etc. In one embodiment of the present disclosure, the above-described components of the computing device 1100, as well as other components not shown in FIG. 11, can be connected to each other by a bus. It should be understood that the computing device structure diagram shown in FIG. 11 is merely for the purpose of example, and is not a limitation on the scope of the present disclosure. Those skilled in the art can add or replace other components as needed. The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1100 can also be a mobile or stationary server. The processor 1120 is configured to execute computer programs / instructions that implement the steps of the above-described model training method when executed by the processor. The above is a schematic scheme of a computing device according to the present embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-described model training method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the model training method. An embodiment of the present disclosure also provides a computer readable storage medium storing computer programs / instructions that implement the steps of the above-described model training method when executed by the processor. The above is a schematic scheme of a computer readable storage medium according to the present embodiment.It should be noted that the technical scheme of the storage medium and the technical scheme of the model training method described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be seen from the description of the technical scheme of the model training method. An embodiment of the present disclosure also provides a computer program product including computer programs / instructions, which, when executed by a processor, implement the steps of the model training method described above. The above is an illustrative scheme of a computer program product of the present embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the model training method described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be seen from the description of the technical scheme of the model training method. The above describes a specific embodiment of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order described in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous. The computer instructions include computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer storage, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice, for example, in some regions, according to patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present disclosure are not limited by the order of the actions described, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the present disclosure are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be seen from the related description of other embodiments.The preferred embodiments of the disclosure disclosed above are only used to illustrate the disclosure. The optional embodiments do not describe all the details and limit the disclosure to the specific embodiments described. Obviously, according to the content of the embodiments of the disclosure, many modifications and changes can be made. The disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the disclosure, so that those skilled in the art can well understand and utilize the disclosure. The disclosure is limited only by the claims and their full scope and equivalents.

Claims

CLAIM 1. A model training method, applied to a control node in a model training system, the model training system comprising the control node and a plurality of training nodes, the plurality of training nodes being configured to train a target model, the method comprising: obtaining node states of the plurality of training nodes respectively during a training process of the target model; in a case where it is determined according to the node states that a training node is an abnormal training node, determining a to-be-restarted training node in the plurality of training nodes according to a node type of the abnormal training node and the node states; and triggering a restart operation on the to-be-restarted training node. The restart operation is used to restart the training of the target model by the to-be-restarted training node.

2. The method of claim 1, wherein the model training system further comprises a cluster management unit; wherein the obtaining the node status of each of the plurality of training nodes during the training process of the target model comprises: obtaining the node status of each of the plurality of training nodes from the cluster management unit during the training process of the target model; and wherein the triggering the restart operation of the training node to be restarted comprises: A restart request for the to-be-restarted training node is sent to the cluster management unit, so that the cluster management unit restarts the training of the target model by the to-be-restarted training node.

3. The method of claim 1 or 2, wherein the node state comprises a restart state, the restart state indicates that the training node is an abnormal training node, the node type comprises a management type and a service type; and determining, according to the node type of the abnormal training node and the node state, a training node to be restarted from the plurality of training nodes comprises: In a case where the node state is the restart state and the node type is the management type or the service type, node restart information and task restart information of the abnormal training node are obtained; and the to-be-restarted training node in the plurality of training nodes is determined according to the node restart information and the task restart information. ​ 4. The method of claim 3, wherein determining, according to the node restart information and the task restart information, the training node to be restarted in the plurality of training nodes comprises: In a case where the node restart information does not satisfy a node restart condition and the task restart information satisfies a task restart condition, the plurality of training nodes are determined as the to-be-restarted training node.

5. The method of claim 1 or 2, wherein the node status comprises a restart failure status, and wherein the restart failure status indicates that the training node is an abnormal training node. The to-be-restarted training node in the plurality of training nodes is determined according to the node type of the abnormal training node and the node state, and includes: In a case where the node state is the restart failure state, node restart information of the abnormal training node is obtained. In a case where the node restart information satisfies a node restart condition, the to-be-restarted training node in the plurality of training nodes is determined according to the node type.

6. The method of claim 5, wherein the node types include a management type, a service type, and a training type. The to-be-restarted training node in the plurality of training nodes is determined according to the node type, and includes: in a case where the node type is the service type, the abnormal training node is determined as the to-be-restarted training node; and in a case where the node type is the training type, the abnormal training node is determined as the to-be-restarted training node. In a case where the node type is the management type, the abnormal training node and a training node of which the node type is the training type are determined as the to-be-restarted training node.

7. The method of any one of claims 1-6, the control node comprises a training data management unit, the training data management unit is configured to store training progress information of model training data, the training node comprises a management type training node of which the node type is a management type and a training type training node of which the node type is a training type; and the method further comprises: during the training of the target model, in response to a data stop distribution request sent by the management type training node, stopping sending model training data to the training type training node, and sending the training progress information to the management type training node, so that the management type training node stores the training progress information.

8. The method of claim 7, wherein the training node comprises a service-type training node with a node type of a service type; and after the service-type training node stops sending model training data to the training-type training node and sends the training progress information to the management-type training node in response to the data stop distribution request sent by the management-type training node, the method further comprises: In response to a data distribution request sent by the management training node, the model training data is sent to the training training node and the management training node according to the training progress information, wherein the data distribution request is sent by the management training node after storing the training progress information and the model checkpoint, the model checkpoint is sent to the management training node by the service training node in response to a checkpoint acquisition request sent by the management training node, and the training progress information stored by the management training node is consistent with the version of the model checkpoint.

9. The method of claim 7 or 8, after the stopping distributing, in response to the data stop distributing request sent by the management-type training node, sending model training data to the training-type training node, and sending the training progress information to the management-type training node, further comprising: In response to a data distribution request sent by the training training node, retry information is sent to the training training node until a data distribution request sent by the management training node is received, and the model training data is sent to the training training node according to the training progress information.

10. A model training system, comprising a control node and a plurality of training nodes, the plurality of training nodes being configured to train a target model; the control node being configured to acquire node states of the plurality of training nodes respectively during a training process of the target model; in a case where it is determined according to the node states that the training nodes are abnormal training nodes, determine, according to a node type of the abnormal training nodes and the node states, a to-be-restarted training node in the plurality of training nodes; and trigger a restart operation on the to-be-restarted training node, wherein, The restart operation is used to restart the training of the target model by the training node to be restarted.

11. The system of claim 10, further comprising a shared storage and parameter storage subgraph, the training nodes comprising a first service training node and a second service training node, the second service training node being a backup node of the first service training node; the shared storage is configured to store a model checkpoint of the target model; and the parameter storage subgraph is configured to copy the model checkpoint from the shared storage to the first service training node, and copy the model checkpoint stored in the first service training node to the second service training node.

12. The system of claim 10, further comprising a shared storage and parameter recovery subgraph, the training nodes comprising a first service training node and a second service training node, the second service training node being a backup node of the first service training node; the shared storage is configured to store a model checkpoint of the target model; and the parameter recovery subgraph is configured to copy the model checkpoint stored in the second service training node to the first service training node in case of failure of the first service training node, or copy the model checkpoint stored in the shared storage to the first service training node. a memory and a processor; 13. A computing device comprising: the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, so as to implement the steps of the method in any one of claims 1 to 9.

14. A computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the method in any one of claims 1 to 9. ​ 15. A computer program product, comprising computer program instructions / a computer program, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9. 18

Images