Model training method, storage medium, electronic apparatus, and computer program product

Abstract

Provided in the embodiments of the present disclosure are a model training method, a storage medium, an electronic apparatus, and a computer program product. The method comprises: when it is determined that a first node in a cluster executing a training task has a fault, acquiring a target second node in the cluster, and scheduling to the target second node a training process in the first node and running the training process; and after the first node recovers from the fault, writing a breakpoint into the second node, and rescheduling to the first node the training process in the target second node.

Description

Model training method, storage medium, electronic device, and computer program product

[0001] Cross-reference to related applications

[0002] [Corresponding to Rule 91, 04.11.2025] The present disclosure is based on Chinese patent application CN202411335258.7 filed on September 23, 2024, and claims priority to the patent application, the disclosure of which is incorporated herein in its entirety by reference. TECHNICAL FIELD

[0003] Embodiments of the present disclosure relate to the field of communications, in particular to a model training method, a storage medium, an electronic device, and a computer program product. BACKGROUND

[0004] Large model training, especially super large scale training of hundreds of billions or trillions of parameters, requires a large scale of graphic processing unit (GPU) resources, and these GPU resources generally adopt a specific networking topology, such as orbital networking. A group of Leaf switches are used to realize parallel training and complete communication within the maximum one-hop switch in the same group. If the resources in the same Leaf switch group are insufficient and need to communicate across Leaf groups, the maximum number of switches will introduce more than 3-hop delay, which will affect the training communication efficiency. Model training is a parallel job mode, and any software or hardware failure during the training process will cause the training to be interrupted. In a super large scale cluster such as a thousand or ten thousand, the problem of reduced effective resource utilization due to failure is more prominent.

[0005] The traditional solution is to use a hot standby strategy within a Leaf switch group. A certain number of servers are reserved as hot standby resources within the same Leaf switch group. When a server fails, the training process running on the servers in the group is rescheduled to the hot standby resource group. However, when the model parameter scale increases, more server resources are needed for 3D or 4D parallel training technology. The resources in the Leaf group may be completely occupied. Once a server fails, and there is no additional server available in the Leaf switch group, the training cannot be restored while maintaining the original topology affinity and communication efficiency. Therefore, the method of reserving hot standby resources only in the Leaf switch group cannot meet the 3D / 4D parallel topology requirements of model training. Moreover, the simultaneous migration of all servers in the group due to a single server failure amplifies the failure and increases the cost of hot standby resources.

[0006] In view of the problem that the related art cannot meet the topology requirements of model training when handling failures in the model training process, and the high cost of hot standby resources, there is currently no effective solution. SUMMARY

[0007] The model training method, the storage medium, the electronic device and the computer program product provided in the embodiments of the present disclosure can at least solve the problem that the topology requirement of model training cannot be met and the cost of hot standby resources is high when a fault in a model training process is processed in the related art.

[0008] According to an embodiment of the present disclosure, a model training method is provided, comprising:

[0009] determining that a first node in a cluster performing a training task fails, obtaining a target second node in the cluster, and scheduling a training process of the first node to the target second node and running; after the first node recovers from the failure, writing a breakpoint at the second node, and rescheduling the training process in the target second node to the first node.

[0010] According to another embodiment of the present disclosure, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is set to execute the steps in any of the above method embodiments when running.

[0011] According to another embodiment of the present disclosure, an electronic device is also provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is set to run the computer program to execute the steps in any of the above method embodiments.

[0012] According to another embodiment of the present disclosure, a computer program product is also provided, comprising a computer program, and the computer program is executed by a processor to implement the steps in any of the above method embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 is a hardware structure block diagram of a mobile terminal running the method embodiment of the present disclosure;

[0014] FIG. 2 is a track-based networking architecture diagram of a cluster server according to an embodiment of the present disclosure;

[0015] FIG. 3 is a flowchart of a model training method according to an embodiment of the present disclosure;

[0016] FIG. 4 is a model training flowchart according to an embodiment of the present disclosure;

[0017] FIG. 5 is a structure block diagram of a model training device according to an embodiment of the present disclosure;

[0018] FIG. 6 is a structure diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the following, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and in conjunction with embodiments.

[0020] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0021] With the rapid development of large language models (LLM, Large Language Model), the model parameter size and training data set are rapidly increasing, thereby increasing the demand for LLM training resources. The training cluster used by a hundred billion to a trillion parameter model reaches or even exceeds ten thousand GPU cards. Based on the huge training cluster, 3D or 4D (Tensor Parallel (TP), Pipeline Parallel (PP), Data Parallel (DP), Expert Parallel (EP)) parallel training techniques are generally used. In the entire training process, any single point software or hardware failure of the GPU infrastructure or training software failure will cause the entire training business to shut down.

[0022] Currently, in order to solve the fault problem in the model training process, so that the training business can automatically recover and run when a fault occurs, some fault recovery mechanisms and fault tolerance means are generally used, including the following:

[0023] 1. A periodic breakpoint Checkpoint writing mechanism is provided in the training framework tool to save data such as weight parameters, intermediate variables and training hyperparameters in the training process, so that the training can be rolled back to the Checkpoint to continue recovery when a fault occurs. The Checkpoint is the training state saved at a certain time point, including weight parameters, intermediate variables and training hyperparameters, so as to continue training from the saved state when a fault occurs.

[0024] 2. Based on the Checkpoint mechanism, backup resources are planned or reserved in the cluster. In order to ensure the communication bandwidth, latency and other key indicators of parallel training, there are affinity requirements for the networking topology of the GPU server, and there are generally two schemes:

[0025] 1) In order to ensure the optimal latency and bandwidth, resources are reserved within the same group of Leaf switches, as shown in FIG. 1. When the GPU server running in the Leaf switch group is abnormal, according to the topology affinity strategy, the idle resources (i.e. redundant GPU servers or hot standby resources) in the group can be selected, or the fault server can be scaled down based on the dynamic elastic training scheme, or the fault server can be rescheduled after self-healing.

[0026] However, the number of servers in the Leaf switch group is limited, and when the model parameter scale increases, the 3D / 4D parallel group may completely occupy all servers in the Leaf switch group, resulting in no additional topologically identical GPU servers (i.e., no hot standby resources) available in the group. In addition, when the model scale expands to require parallel training across multiple Leaf switch groups, this hot standby method cannot effectively provide cross-group hot standby resources, and therefore cannot meet the needs of large-scale parallel training for topology affinity and resource redundancy.

[0027] 2) Reserve a hot standby Leaf switch group and corresponding hot standby GPU servers in the cluster, and when a GPU server fails during training, the training process run by the group of GPU servers is rescheduled to the hot standby resource group (i.e., whole-group hot standby switching), which can provide performance guarantees through resuming training rescheduling when a failure occurs.

[0028] However, to ensure that training performance is not affected, whole-group hot standby switching requires a complete Leaf switch group of resources to be reserved as hot standby in the cluster, and for a large-scale model training cluster, a large number of GPU servers need to be reserved, significantly increasing the cost of hot standby resources. Moreover, in the whole-group hot standby switching scheme, the failure of a single server will cause the server resources of the entire Leaf group to be migrated to the hot standby group, which not only wastes resources but also may extend the training interruption time due to large-scale rescheduling, thereby amplifying the impact of the failure on the training job.

[0029] To address the above technical problems, the embodiments of the present disclosure propose a model training method, the technical concept of which is to schedule the training process of the failed node to the hot standby node after identifying the failed node in the cluster, and after the failure of the failed node is recovered, write the breakpoint to the hot standby node and reschedule the current training process back to the recovered original failed node, thereby weakening the function of the hot standby node in the cluster, so that the function of the hot standby node is only to temporarily replace the failed node to perform the training process before the failed node recovers, thereby solving the problem in the related art that when handling the failure problem in the model training process, the topology requirements of the model training cannot be met, and the cost of hot standby resources is high, simplifying cluster planning, reducing the network topology requirements of the cluster, reducing the number of hot standby nodes required, improving the convenience of cluster operation and maintenance, and reducing the cost of hot standby resources of the cluster.

[0030] The method embodiments provided in the embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking an example of running on a mobile terminal, FIG. 1 is a hardware structure block diagram of a mobile terminal running the method embodiments of the present disclosure. As shown in FIG. 1, the mobile terminal can include one or more (only one is shown in FIG. 1) processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal can further include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that the structure shown in FIG. 1 is only schematic, which does not limit the structure of the mobile terminal. For example, the mobile terminal can further include more or less components than those shown in FIG. 1, or have a different configuration from that shown in FIG. 1.

[0031] The memory 104 can be used to store computer programs, for example, software programs of application software and modules, such as the computer program corresponding to the model training method in the embodiments of the present disclosure, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the mobile terminal through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0032] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module which is used to communicate with the Internet in a wireless manner.

[0033] FIG. 2 is a track networking architecture diagram of a cluster server according to an embodiment of the present disclosure. In a model training business scenario, a GPU server generally adopts an asymmetric track networking or a symmetric Spine / Leaf+TOR switch networking. The embodiment of the present disclosure can run on the network architecture shown in FIG. 2, taking the track networking as an example. As shown in FIG. 2, the networking architecture includes a center / spine Spine switch, a leaf Leaf switch, a GPU server, and the like. In the cluster planning, the Spine switch can be multiple. Each Spine switch can be connected to at least one Leaf switch. Each Leaf switch can be connected to at least one GPU server.

[0034] Part of the GPU servers can be pre-allocated to the LLM training resource group according to the cluster planning needs, and another part of the GPU servers can be allocated to the cluster-level hot standby resource group.

[0035] In the track networking, the model 3D / 4D parallel training selects the GPU servers under the Leaf switch group to construct a 3D / 4D parallel training communication group (i.e., the LLM training resource group) according to the business topology requirements. The GPU server resources in the Leaf switch group do not need to be reserved. The hot standby resource group can be flexibly set according to the cluster size and the networking topology, and does not need to be aligned with the Leaf switch group topology where the business training resources are located.

[0036] Each GPU server in the LLM training resource group can be used as a training node for performing a model training computing task, and is organized together according to the 3D or 4D parallel training requirements, and efficiently communicates through the Leaf switch. Each GPU server in the cluster-level hot standby resource group can be used as a hot standby node, which can quickly take over the training task when a server in the LLM training resource group fails, thereby reducing the training interruption time and improving the training efficiency.

[0037] The hot standby node can not undertake the training task in normal circumstances. The hot standby node is used to cope with possible failures and provides the ability of fault recovery. When a server in the LLM training resource group fails, the hot standby server in the cluster-level hot standby resource group can be scheduled in time to recover the training with minimal performance loss.

[0038] As shown in FIG. 2, in the normal training process, the data flow and the instruction flow are mainly between the parallel group members of the model. These parallel group members are usually located in the same Leaf switch group to ensure the optimal communication bandwidth and the lowest delay. The GPU server nodes in each Leaf switch group perform high-speed data exchange through the RDMA network technology (such as RoCE), realize the training of tensor parallel, pipeline parallel, data parallel, or sequence parallel, and the communication flow direction follows the topology structure of the parallel training.

[0039] When a GPU server (such as GPU Server 3 in FIG. 2) fails (that is, a training node fails and becomes a failed node / faulty server), whether a replacement hot-standby server needs to be found in the cluster can be determined according to the fault type; if it is a business fault (that is, a software fault) or a GPU card fault, the training process can be restarted on GPU 3; if it is a reset recoverable fault (that is, a server-level hardware fault), a hot-standby server in the hot-standby resource group can be called to replace it.

[0040] If there is no available hot-standby server in the Leaf group where the faulty server is located, a hot-standby server in another Leaf group in the cluster can be selected according to the topology affinity policy to be rescheduled, and the training communication crosses the Spine switch to reach the hot-standby resource in another Leaf group to perform cross-group hot-standby node switching after the training node fails.

[0041] When the original faulty server is self-healed and passes the health diagnosis, the topology affinity between the original faulty server and the current hot-standby server can be determined. If the affinity of the original faulty server is better (that is, the communication performance is better), a specific fault code can be actively injected on the current hot-standby server to trigger an immediate breakpoint Checkpoint write, and the training process on the hot-standby server can be rescheduled back to the original faulty server to restore its optimal training state.

[0042] After the Checkpoint write of the hot-standby server is completed, the training state thereof will be saved, that is, the training process can be transferred to the original faulty server. After rescheduling, the original faulty server resumes its training process, and the training communication is restored to the original Leaf group, ensuring the training conditions of communication bandwidth and low latency.

[0043] The model training method of the embodiments of the present disclosure can reduce the influence of the network topology of the server. The network topology of the hot-standby server can be different from that of the faulty server. After the faulty server is self-healed, if the topology affinity of the hot-standby server is not as good as that of the original faulty server, the training process can be rescheduled back to the original server, so as to maintain the high-performance state of the training as much as possible.

[0044] In the present embodiment, a model training method running on the above mobile terminal or networking architecture is provided. FIG. 3 is a flowchart of the model training method according to an embodiment of the present disclosure. As shown in FIG. 3, the flow includes the following steps:

[0045] In step S301, it is determined that a first node in a cluster performing a training task fails, a target second node in the cluster is obtained, and a training process of the first node is scheduled to the target second node and run.

[0046] As an example, the cluster performing the training task can be a network of multiple computing devices (e.g., GPU servers) connected to each other through a specific network architecture (e.g., asymmetric orbiting or symmetric Spine / Leaf+TOR networking) to jointly participate in the training process of a large-scale machine learning model. Each device in the cluster can undertake part of the computational task of model training, accelerating the convergence of the overall model through parallel training.

[0047] As an example, the first node can be a computing device (i.e., a training node in FIG. 2) in the cluster for performing a specific training task, such as a GPU server. During the training process, the first node can fail due to software errors, hardware failures, or network problems, causing it to be unable to continue performing the training task.

[0048] As an example, the state of the software and hardware resources of the nodes performing the training task can be monitored in real time or periodically, and in the case of detecting a failure of a node in the cluster, the failure event can be responded to in a timely manner to trigger subsequent scheduling processes.

[0049] As an example, during the model training process, due to the involvement of a large amount of computing resources and complex software environments, node failures can be caused by various factors, and the types of failures can include but are not limited to hardware failures (e.g., GPUs, CPUs, memories, etc.), software failures (e.g., training framework exceptions, operating system errors, etc.), or network failures (e.g., connection interruptions, increased delays, etc.).

[0050] As an example, the target second node can be a computing device selected from the hot standby servers of the cluster to replace the first node to continue performing the training task after the first node fails, such as a hot standby server.

[0051] As an example, the selection of the target second node can be based on the matching of its network affinity and computing resources with other nodes participating in the training in the cluster to ensure that the training efficiency is not or less affected.

[0052] As an example, when the first node in the cluster fails, the training process can be transferred to the target second node through a scheduling mechanism, and the execution state of the training process can be restored, including but not limited to model parameters, optimizer states, processing progress of training data, etc., so that starting from the beginning of training or recovering from a historical breakpoint at a relatively long time can be avoided, which can cause time waste and reduce training efficiency.

[0053] In an exemplary embodiment, the target second node in the cluster is obtained, including:

[0054] According to the preset identification of each of the plurality of second nodes, a plurality of second nodes in the cluster are obtained.

[0055] obtain network topology information of the cluster, and calculate a topology affinity of each of the second nodes in the cluster according to the network topology information;

[0056] determine the second node with the highest topology affinity as the target second node.

[0057] As an example, when the cluster is constructed, the nodes that can be used as hot standby resources can be planned in advance, and specific identifiers can be assigned to these nodes. These identifiers can be hardware IDs, IP addresses, MAC addresses, or labels defined by the cluster management system, etc., to ensure that each second node has a unique identification code.

[0058] For example, the second node can be labeled with specific labels such as "hot_standby", "recovery_node", etc. by an AI cloud platform or a cluster management system, which can be used to quickly identify the second node in subsequent scheduling.

[0059] As an example, the identifiers of all second nodes, as well as information related thereto such as topology location, hardware state, image warm-up state, etc. can be recorded in the configuration file or database of the cluster or the relevant node identifier list. When the state of the second node changes (such as after being used for fault recovery), its identification state can be updated in time to ensure the validity and accuracy of the identification.

[0060] As an example, when a fault of the first node is detected, the identifiers of the second nodes in the configuration file or database of the cluster or the relevant node identifier list can be queried to obtain each corresponding second node, and based on the obtained network topology information, the topology affinity between each second node and the original training group of the fault node can be calculated. Affinity can be determined by multiple factors such as the length of the network path, delay, bandwidth, etc.

[0061] As an example, the second node with the highest topology affinity can be selected from multiple second nodes as the target second node, which can minimize communication delay, improve training efficiency, and minimize performance loss without changing the original training group topology structure.

[0062] In an exemplary embodiment, before determining the second node with the highest topology affinity as the target second node, it further comprises:

[0063] According to the identifier of the second node, the training process of the first node is downloaded to the second node.

[0064] As an example, image download can quickly deploy a complete and consistent running environment to the local.

[0065] As an example, a pre-warming step can be performed on the second node in advance, i.e., a training process on the first node can be pre-mirrored downloaded to the second node. The training mirror contains all the environment configurations required for model training, including but not limited to the environment of the operating system, framework, library, model code, and pre-processed data, etc. Through pre-download, the second node can be prepared in advance with an environment identical to that of the first node, so that when the first node fails, the training process can be seamlessly switched to the second node for continuous execution.

[0066] Pre-downloading the mirror to the hot standby server can significantly reduce the waiting time when the training process starts. Through mirror downloading, resources can be more effectively allocated according to the resource status and topology location of the hot standby server when the training job is scheduled, avoiding resource waste and performance bottlenecks.

[0067] Through the above pre-warming step, the embodiments of the present disclosure can effectively deal with possible software and hardware failures in a large-scale cluster without sacrificing training performance, reduce the time of training interruption, improve the utilization of resources, and thus improve the overall model training efficiency. At the same time, this method also avoids the increase of network delay caused by scheduling the hot standby server across the switch group, thereby ensuring the continuity and high performance of the training.

[0068] In an exemplary embodiment, the first node in the cluster that performs the training task is determined to have a failure, and a target second node in the cluster is obtained, comprising:

[0069] identifying the type of the failure;

[0070] determining that the type of the failure is a first node reset recoverable failure, and obtaining the target second node in the cluster.

[0071] As an example, all nodes in the training cluster can be periodically monitored to detect whether a software and hardware failure occurs. In the case of detecting a node failure, further analysis can be performed to identify the specific type of the failure.

[0072] As an example, the failure type can include but is not limited to: service failure, GPU card failure, first node reset recoverable failure, etc.

[0073] When the failure type is a first node reset recoverable failure, it can be solved by resetting the first node, rather than by software-level repair or replacing hardware. The first node reset recoverable failure can include, for example, network connection interruption, temporary hardware failure, etc., which can be solved by resetting the node (such as restarting the server).

[0074] As an example, when the fault type is determined to be the first node reset recoverable fault, a target second node can be acquired to continue the model training task seamlessly during the self-recovery of the first node or when the first node cannot be immediately recovered, thereby avoiding the loss caused by training interruption.

[0075] In an exemplary embodiment, the type of the fault further includes a service fault of the first node and a GPU card fault of the image processing GPU of the first node; and the method further includes:

[0076] determining that the type of the fault is the service fault of the first node, reconstructing the training process of the first node, and performing fault recovery on the first node; or

[0077] determining that the type of the fault is the GPU card fault of the first node, resetting the GPU, reconstructing the training process of the first node, and performing fault recovery on the first node.

[0078] As an example, the fault type of the node can include at least one of the following: a first node reset recoverable fault, a service fault of the first node, and a GPU card fault of the first node. Different types of faults can occur / exist simultaneously, for example, the first node can simultaneously exist the first node reset recoverable fault, the service fault, and the GPU card fault, the first node can simultaneously exist the first node reset recoverable fault and the service fault, the first node can simultaneously exist the service fault and the GPU card fault, and the first node can only exist the first node reset recoverable fault or the service fault or the GPU card fault, etc.

[0079] As an example, when the service fault of the first node (i.e., the GPU server that has occurred a fault) is detected, i.e., the training process itself has a problem, which can be caused by software errors, process crashes, or memory leaks, etc. For this type of fault, the training process can be directly reconstructed on the first node. This process can include reloading the Checkpoint data, restoring the training state, and then restarting the training process. The recovery of the service fault is usually simpler and faster than the recovery of the hardware fault, because there is no need for hardware-level operations.

[0080] As an example, if the fault type is a GPU card fault, such as GPU hardware overheating, driver abnormalities, or hardware damage, etc., the reset operation of the GPU card can be performed to clear or repair the hardware state causing the fault. After the reset, the training process can be tried again to be reconstructed. The GPU card reset can reset the GPU to the initial state, clean up the existing error state, and enable the GPU to work normally again. After the GPU card reset, the training process can be resumed on the first node after the fault is successfully solved.

[0081] Step S302, after the failure recovery of the first node, write a breakpoint in the second node and reschedule the training process in the target second node to the first node.

[0082] In the embodiments of the present disclosure, after the self-recovery of the first node (i.e., after the failure recovery of the first node), through the active failure injection and instant checkpoint writing mechanism, the training process can be quickly switched back to the first node with better topology affinity without losing the training results, so as to maximize the training efficiency and resource utilization while ensuring the continuity of model training.

[0083] In an exemplary embodiment, the writing of the breakpoint in the target second node and the rescheduling of the training process in the target second node to the first node comprises:

[0084] The topology affinity of the first node and the target second node in the cluster is respectively acquired;

[0085] In the case that the topology affinity of the first node is greater than that of the target second node, a breakpoint is written in the target second node, and the training process in the target second node is rescheduled to the first node.

[0086] In the embodiments of the present disclosure, when the failed server (i.e., the first node) is self-recovered, the topology affinity between the first node and the hot standby server (i.e., the target second node) currently carrying the training task of the failed server can be evaluated to determine whether to reschedule the training task from the target second node back to the first node. This process can ensure that the model training is carried out under the optimized network topology, thereby minimizing the training time and improving the training efficiency.

[0087] As an example, the topology affinity refers to the network communication efficiency of the current node with other related nodes in the network topology structure of the cluster, including bandwidth, delay and other key performance indicators. This information can be obtained in real time by a network monitoring system or predicted based on historical data. Generally, nodes with higher topology affinity can exchange data with other nodes more efficiently when performing large-scale parallel computing, reducing communication delay and thus improving the overall training speed.

[0088] As an example, if the topology affinity of the first node is higher than that of the target second node, a failure injection mechanism can be triggered to write a breakpoint (i.e., trigger a checkpoint save) in the target second node. By writing a breakpoint, the current training state of the target second node can be saved, including but not limited to weight parameters, intermediate variables and training hyperparameters, etc., to ensure that the training will not lose important information due to switching nodes.

[0089] In an example embodiment, the writing of the breakpoint at the target second node and the rescheduling of the training process in the target second node to the first node comprises:

[0090] In the target second node, a preset fault code is injected to trigger the writing of the breakpoint and the rescheduling of the training process in the target second node to the first node.

[0091] As an example, the injection of the fault code can enable the training framework or platform to identify a specific error condition, thereby triggering the writing of the checkpoint. When the training process on the hot standby server (i.e., the target second node) receives this preset fault code, the checkpoint writing operation can be immediately performed. This operation saves the current model state (which can include weight parameters, intermediate variables, training hyperparameters, etc.) so that when the training process is rescheduled, the training can continue based on this saved state.

[0092] As an example, the checkpoint is successfully written, and the training process can be rescheduled from the hot standby server (the target second node) to the original faulty server (the first node), thereby ensuring that the training can continue on a node with better affinity.

[0093] In an example embodiment, the method further comprises:

[0094] In the case where the topology affinity of the first node is less than or equal to the topology affinity of the target second node, the training process continues to run on the target second node.

[0095] As an example, based on the optimal choice of performance considerations and economic analysis, when the topology affinity of the original faulty server is less than or equal to that of the hot standby server, the training process can continue to run on the hot standby server, thereby avoiding unnecessary resource switching and performance loss, while reducing the cost of hot standby resources and improving the flexibility and economic efficiency of the overall training system.

[0096] In the embodiments of the present disclosure, by determining that a first node in a cluster performing a training task fails, obtaining a target second node in the cluster, and scheduling a training process of the first node to the target second node and running, after the first node recovers from the failure, writing a breakpoint at the second node and rescheduling the training process in the target second node to the first node. In related technologies, when handling fault problems in the model training process, there is a problem that the topology requirements of the model training cannot be met, and the cost of hot standby resources is high. The cluster planning is simplified, the network topology requirements of the cluster are reduced, the number of hot standby nodes is reduced, the cluster operation convenience is improved, and the cost of hot standby resources of the cluster is reduced.

[0097] The model training method of the embodiment of the present disclosure is further described below through an example.

[0098] Example 1

[0099] FIG. 4 is a schematic diagram of a model training process according to an embodiment of the present disclosure, which can include training process scheduling, preparation process S1, training process running process S2, breakpoint resuming training process S3, self-healing resuming training process S4, etc. As shown in FIG. 4, each process is described below respectively:

[0100] S1 training process scheduling and preparation process

[0101] Specifically, the following steps can be included:

[0102] S11, the training task can be submitted through the AI platform to prepare for training;

[0103] S12, the server cluster topology affinity calculation can be completed according to the training process resource requirement, and the training process can be scheduled to the affinity node;

[0104] It should be noted that in the field of computing and network, the affinity node usually refers to a computing node that is more suitable for a specific workload or task in terms of physical location, network topology or resource characteristics. The affinity node in the present example 1 is the training node used to execute the training task, that is, the first node of the embodiment of the present disclosure.

[0105] S13, according to the server cluster hot standby node identifier, the training process image can be downloaded to the hot standby node;

[0106] As an example, after preheating, the subsequent breakpoint resuming training image loading and starting time can be shortened, which can specifically include the following steps:

[0107] 1) according to the resource planning of the training cluster, a proper number of hot standby nodes are selected, and the hot standby label is added on the AI cloud platform;

[0108] 2) according to the training process scheduled in step S12, the training process image can be loaded to the node with hot standby label through the task scheduler of the AI cloud platform;

[0109] Exemplarily, the hot standby node image can be cleaned up according to different strategies (such as the number of uses, time, etc.).

[0110] 3) after the training process is completed and the hot standby node is preheated, the next process can be entered.

[0111] S2 training process running process

[0112] Specifically, the following steps can be included:

[0113] S21, after the training process completes scheduling, start training;

[0114] S22, write Checkpoint once every preset training number;

[0115] It should be noted that the training process can be regarded as training once after each training sample is executed. The preset training number can be configured according to the actual situation of the training framework, module, and intelligent computing hardware infrastructure and software infrastructure.

[0116] The training is ended when the training accuracy meets the standard or all training progress is completed, otherwise S21 is executed cyclically.

[0117] S3 breakpoint training continuation process

[0118] Specifically, the following steps can be included:

[0119] During the running of the training process in S2, the fault state of the software and hardware resources related to the training job can be monitored throughout the cycle.

[0120] S31, training process, infrastructure software and hardware fault awareness, and periodic detection;

[0121] S32, if there is no fault event, wait for the next detection period or wait for the fault event report;

[0122] S33, if a fault event (i.e. affinity node fault) is detected, trigger the breakpoint training continuation process;

[0123] S34, according to the detected fault type, select the corresponding fault recovery action;

[0124] For example, for the business fault type, the training process restart operation can be selected for recovery; for the GPU card fault type, the GPU card reset operation can be selected for recovery; for the node reset recoverable fault type, the server reset operation can be selected for recovery.

[0125] S35, if it is a GPU card reset recoverable fault, the GPU card reset operation can be performed, and if it is a business fault, the training process can be reconstructed;

[0126] S36, if it is a node reset recoverable fault, the fault node (i.e. the affinity node after the fault) can be isolated, the optimal target hot standby server is selected from the hot standby servers in the cluster based on the topology affinity, and the training is recovered through rescheduling;

[0127] S37, enter the self-healing training continuation process (S4) of the fault server;

[0128] S4 self-healing training continuation process

[0129] Specifically, the following steps can be included:

[0130] S41, initiating a reset fault recovery action on the fault node through the AI platform;

[0131] S42, after the reset start is completed, initiating health detection to determine whether the original fault server is recovered;

[0132] S43, if the fault node is not recovered, the reset fault recovery action of step S41 can be repeatedly executed within a preset repetition number threshold range;

[0133] S44, if the fault of the fault node has been recovered and its topology affinity is better than that of the active hot standby node, step S45 can be executed; if the topology of the active hot standby node is better, the current state can be maintained to continue running;

[0134] S45, injecting a preset fault code into the active hot standby node to trigger immediate checkpoint writing;

[0135] For example, after the AI platform detects the preset fault code, the immediate checkpoint writing action is started; after the AI platform completes the checkpoint writing, the subsequent rescheduling action can be executed.

[0136] S46, rescheduling the training process in the active hot standby node to the original fault node that has been recovered;

[0137] S47, completing the rescheduling of the hot standby node to the original fault node, and recovering the training to the topology optimal resource combination.

[0138] The above process of the present example 1 reduces the demand for the number of hot standby resources of the AI training cluster, reduces the cost of the hot standby resources of the cluster, and also reduces the network topology demand of the hot standby resources, thereby improving the convenience of cluster operation; through the affinity comparison between the hot standby node and the fault node, the training process can be rescheduled back to the original server as needed to improve the training performance; through the hot standby resource label planning at the cluster level and the task level, the training image can be downloaded and preheated during the process scheduling stage, compared with the traditional global scheduling to find a replacement node, the image startup time during the continuation training is greatly shortened, and the training performance is improved.

[0139] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform as necessary, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product in essence or in the part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc) and includes a number of instructions for causing an end device (which can be a mobile phone, a computer, a server, or a network device) to perform the methods described in the various embodiments of the present disclosure.

[0140] In the present embodiment, a model training device is also provided, which is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and contemplated.

[0141] FIG. 5 is a structural block diagram of a model training device according to an embodiment of the present disclosure, as shown in FIG. 5, the device includes an AI platform C1, a task scheduler C2, an AI hardware device C3, and a breakpoint resume training module C4.

[0142] The AI platform C1 can be used to receive requests for AI training tasks, and to pass the training tasks and the requirements for related resources to the C2; and is connected to the breakpoint resume training module C4, for quickly responding and recovering the training state when a fault occurs.

[0143] The AI platform C1 can manage resources such as the code of the training model, the data set, and the training parameters, and provide a running environment such as a container or a virtual machine for the training task; and can also be used for interaction with the training framework and the user interface, and for monitoring and logging the training process.

[0144] The task scheduler C2 is in communication connection with the AI platform C1, and can be used to optimally allocate resources according to the training task requirements submitted by the AI platform C1, and the resource state and topology information of the AI hardware infrastructure C3; and can select a GPU server for task scheduling according to the requirements of 3D or 4D parallel training, and can also complete the image preheating of the hot standby server.

[0145] The AI hardware device C3 can be used to send its own resource state and topology information to the task scheduler C2 and the AI platform C1, so as to facilitate the scheduling and execution of tasks; to receive the training tasks allocated by the task scheduler C2 and perform computing and communication operations; and to report fault events to the C4 when a fault occurs, so as to recover from the fault.

[0146] The AI hardware device C3 can include, but is not limited to, a GPU server, a RoCE switch, a storage server, and the like.

[0147] The breakpoint resume training module C4 is configured to process fault recovery in the model training process, and includes a fault perception submodule C41, a breakpoint resume training submodule C42, and a fault self-recovery submodule C43.

[0148] The fault perception submodule C41 is in communication connection with the AI hardware device C3 to perceive hardware faults, and is in communication connection with the task scheduler C2 to perceive training task states. The fault perception submodule C41 is configured to send the perceived fault types to the breakpoint training submodule C42.

[0149] The breakpoint training submodule C42 is configured to determine whether to perform local scheduling or hot standby server scheduling on the training process according to the fault types sent by the fault perception module C41. If it is hot standby server scheduling, the fault self-recovery submodule C43 is triggered to complete fault server self-recovery, health diagnosis, and related affinity discrimination.

[0150] The fault self-recovery submodule C43 is configured to inject a fault into the on-duty hot standby server, trigger immediate checkpoint writing, and re-schedule the training process of the on-duty hot standby server to the original server to resume training if the original fault server has better affinity. If the on-duty hot standby server has better affinity, the current state is maintained to continue running.

[0151] It should be noted that the above modules can be implemented by software or hardware. For the latter, the following implementation manners can be used, but are not limited thereto: the above modules are located in the same processor; or the above modules are located in different processors in any combination.

[0152] Embodiments of the present disclosure also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0153] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic or optical disk, and various media that can store computer programs.

[0154] FIG. 6 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. Embodiments of the present disclosure also provide an electronic device 600, which includes a memory 601 and a processor 602, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the method embodiments described above.

[0155] In an example embodiment, the electronic device described above can further include a transmission device connected to the processor, and an input / output device connected to the processor.

[0156] The specific examples in the present embodiment can refer to the examples described in the above embodiments and example implementations, which will not be repeated here.

[0157] Embodiments of the present disclosure also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the method embodiments described above.

[0158] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be realized by general computing devices, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0159] The above only describes preferred embodiments of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art can make various modifications and changes to the present disclosure. Any modification, equivalent replacement, improvement, etc. within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A model training method, comprising: determining that a first node in a cluster performing a training task fails, obtaining a target second node in the cluster, and scheduling a training process of the first node to the target second node and running; after the first node recovers from the failure, writing a breakpoint at the second node and rescheduling the training process in the target second node to the first node.

2. The method of claim 1, wherein, The writing of the breakpoint at the target second node and the rescheduling of the training process in the target second node to the first node comprises: obtaining the topology affinity of the first node and the target second node in the cluster respectively; in the case where the topology affinity of the first node is greater than that of the target second node, writing a breakpoint at the target second node and rescheduling the training process in the target second node to the first node.

3. The method of claim 1, wherein, The writing of the breakpoint at the target second node and the rescheduling of the training process in the target second node to the first node comprises: injecting a preset fault code at the target second node to trigger writing of the breakpoint and rescheduling of the training process in the target second node to the first node.

4. The method of claim 3, wherein, Further comprising: in the case where the topology affinity of the first node is less than or equal to that of the target second node, continuing to run the training process at the target second node.

5. The method of claim 1, wherein, The obtaining of the target second node in the cluster comprises: obtaining a plurality of second nodes in the cluster according to the identification of each of the plurality of second nodes; obtaining network topology information of the cluster and calculating the topology affinity of each of the second nodes in the cluster according to the network topology information; determining the second node with the highest topology affinity as the target second node.

6. The method of claim 5, wherein, Before determining the second node with the highest topology affinity as the target second node, further comprising: mirroring the training process of the first node to the second node according to the identification of the second node.

7. The method of claim 1, wherein, The determination of the failure of the first node in the cluster performing the training task and the obtaining of the target second node in the cluster comprises: identifying the type of the failure; determining that the type of the failure is a first node reset recoverable failure, and obtaining the target second node in the cluster.

8. The method of claim 7, wherein, The type of the failure further comprises a service failure of the first node and a GPU card failure of the first node; the method further comprises: determining that the type of the failure is a service failure of the first node, reconstructing the training process of the first node, and performing failure recovery at the first node; or determining that the type of the failure is a GPU card failure of the first node, resetting the GPU, reconstructing the training process of the first node, and performing failure recovery at the first node.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images