Fault-tolerant recovery system and method applied to heterogeneous training cluster

Abstract

The invention relates to a fault-tolerant recovery system and method applied to a heterogeneous training cluster, the system comprises a scheduling layer and a functional layer, the functional layer is provided with a predefined uniform interface and is used for accessing different cloud manufacturers and different chips, and the functional layer is used for executing log monitoring, state monitoring, node detection and alarm notification processes; and the scheduling layer is used for executing training task monitoring, training task inspection, fault analysis and training recovery processes. The method comprises the following steps: accessing a plurality of cloud manufacturers and a plurality of chips based on a predefined uniform interface; executing a fault-tolerant recovery process: polling the state of each training task, carrying out bad node detection, restarting the training task after removing the bad node, and simultaneously carrying out corresponding alarm prompt; and checking and exception handling are carried out on the parameter service. Compared with the prior art, the method can support the training fault tolerance of cross-cloud manufacturers of heterogeneous training clusters, facilitates the expansion of various heterogeneous chips, and improves the training efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and in particular, to a fault tolerance and recovery system and method applied to a heterogeneous training cluster. Background Art

[0002] When training large models in large-scale distributed scenarios, a Fault Tolerance System plays a crucial role. Since the training process usually involves a large amount of computing resources, long running times (days or even weeks), and complex data and model distributions, any hardware failure, software error, or network problem may cause the training to be interrupted or fail. The role of the fault tolerance system is to ensure that the training process can continue to run in the face of these failures, thereby minimizing resource waste and time loss.

[0003] Generally, the functions that a fault tolerance system needs to implement are as follows: ① Checkpointing mechanism: Regularly save the training state (implemented by the training framework). ② Task Retry: Automatically retry failed tasks. ③ Failover: Transfer tasks to backup resources. ④ Logging & Monitoring: Real-time tracking of the system state.

[0004] The role of the fault tolerance system in large-scale distributed training of large models is to ensure the stability, reliability, and efficiency of the training. It minimizes the impact of faults on the training through technologies such as fault detection, checkpoints, and resource reallocation, thereby reducing resource waste and time costs and supporting large-scale and long-term training tasks.

[0005] As the number of model parameters increases, large training clusters require more computing power, and multiple cloud providers often need to provide multiple types of chips to meet the computing power requirements. Due to the large differences in chips in heterogeneous clusters, an asynchronous parallel scheme of a parameter server (3DPS) is generally adopted in large-scale heterogeneous cluster distributed training. As Figure 1 shown, a training task is split into several subtasks, so that multiple types of chips from multiple cloud providers can participate in the training, and multiple training subtasks can be deployed on different chips of different cloud providers.

[0006] However, existing fault tolerance solutions are all for training tasks running on homogeneous chips of a single cloud provider and cannot solve the problem of large-scale heterogeneous training tasks across cloud providers. Since they can only support training tasks of a single cloud provider and a single type of chip, in the face of large-scale heterogeneous training scenarios across cloud providers, multiple sets of fault tolerance services often need to be deployed in multiple cloud providers, and even one set needs to be deployed for each type of chip, which results in high costs and each set of fault tolerance services cannot perceive the global information of the training tasks. Summary of the Invention

[0007] The object of the present invention is to overcome the defects existing in the above-mentioned prior art, and provide a fault tolerance and recovery system and method applied to heterogeneous training clusters, which can support the training fault tolerance of heterogeneous training clusters across cloud providers, facilitate the expansion of various heterogeneous chips, and improve the training efficiency.

[0008] The object of the present invention can be achieved by the following technical solutions: A fault tolerance and recovery system applied to heterogeneous training clusters, including a scheduling layer and a functional layer. The functional layer is provided with predefined unified interfaces for accessing different cloud providers and different chips. The functional layer is used to execute log monitoring, status monitoring, node detection, and alarm notification processes;

[0009] The scheduling layer is used to execute training task monitoring, training task inspection, fault analysis, and training recovery processes.

[0010] Further, the unified interface includes a data structure, a unified task interface, and a unified cloud provider interface.

[0011] Further, the data structure includes an abstract task data structure, an abstract cloud provider data structure, a bad node data structure, and an inspection information briefing data structure.

[0012] Further, the unified task interface includes:

[0013] Start a task;

[0014] Obtain the task status through the cloud provider;

[0015] Delete a bad node;

[0016] Stop a task;

[0017] Obtain the most recent loss (i.e., loss value) of the task from the log;

[0018] Judge whether the task is stuck according to the log;

[0019] Obtain the training task performance value from the log;

[0020] Obtain other custom monitoring metrics;

[0021] The cloud provider identifier used.

[0022] Further, the unified cloud provider interface includes:

[0023] Start a task according to the task ID;

[0024] Obtain the task status according to the task ID;

[0025] Delete a bad node;

[0026] Stop the task according to the task ID.

[0027] A fault tolerance and recovery method applied to a heterogeneous training cluster, comprising the following steps:

[0028] S1. Based on a predefined unified interface, access multiple cloud providers and various chips;

[0029] S2. Execute the fault tolerance and recovery process: Poll the status of each training task, perform bad node detection, remove the bad node and then restart the training task, and at the same time perform corresponding alarm prompts;

[0030] And check and handle exceptions for the parameter service.

[0031] Further, the specific process of step S1 is as follows:

[0032] S11. Based on a predefined unified task interface, access each training task and the parameter service respectively;

[0033] Based on a predefined unified cloud provider interface, access multiple cloud providers;

[0034] S12. For each heterogeneous chip, adapt the corresponding training framework.

[0035] Further, in step S11, accessing multiple cloud providers specifically uses the unified cloud provider interface to respectively start the training task, obtain the task status, delete the bad node, stop the task, and obtain the log by calling the cloud provider service interface.

[0036] Further, in step S2, polling the status of each training task specifically performs a task inspection process once according to a preset time interval to collect the task status; the average tgs (performance metric), the minimum value of loss, the average value of loss, and the maximum value of loss for the last M training iterations (iter); the loss (loss value) curve graph.

[0037] Further, the specific process of step S2 is as follows:

[0038] S21. Check each training subtask in turn and handle exceptions:

[0039] If the task status is abnormal, trigger a task status abnormal alarm, stop the training subtask, perform bad node detection, remove the bad node and then restart the subtask;

[0040] Analyze the loss (loss value) abnormality from the log and trigger a loss (loss value) abnormal alarm;

[0041] The training task gets stuck, triggering a task stuck warning. Stop the training subtask, perform bad node detection, remove the bad nodes, and then restart the subtask.

[0042] The performance of the training task is lower than the preset normal value, triggering a low performance warning. Stop the training subtask, perform bad node detection, remove the bad nodes, and then restart the subtask.

[0043] Other preset custom metrics are abnormal, triggering an alarm.

[0044] S22. Check the parameter service and handle abnormal situations:

[0045] The task status is abnormal, triggering a task status abnormal warning. Stop the task and then restart it.

[0046] The preset custom monitoring metrics are abnormal, triggering an alarm.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] The present invention designs a fault tolerance and recovery system applied to heterogeneous training clusters, including a scheduling layer and a functional layer. Among them, the functional layer is provided with predefined unified interfaces, including a unified task interface and a unified cloud provider interface, for accessing different cloud providers and different chips. The functional layer is used to execute log monitoring, status monitoring, node detection, and alarm notification processes; the scheduling layer is used to execute training task monitoring, training task inspection, fault analysis, and training recovery processes. This enables the fault tolerance service to support training tasks of multiple cloud providers and multiple heterogeneous chips; by defining unified interfaces, the structural differences of different cloud providers and different chips can be shielded. After multiple cloud provider heterogeneous chips are adapted and accessed based on the unified interface, the fault tolerance service can achieve unified implementation without relying on specific cloud providers and specific chips.

[0049] The present invention defines the access processes of training tasks and parameter services, enabling multiple cross-cloud-provider heterogeneous training tasks to quickly access the fault tolerance service based on the unified task interface and the unified cloud provider interface.

[0050] The present invention designs a fault tolerance and recovery process, including task inspection, abnormal alarm, and automatic recovery processes. Thus, for various abnormal situations occurring in the distributed training process of large-scale heterogeneous clusters, it can support cross-cloud-provider training fault tolerance for a ten-thousand-card heterogeneous training cluster across cloud providers, reducing costs and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of a large-scale heterogeneous cluster distributed training framework in the prior art;

[0052] Figure 2 It is a schematic diagram of the method flow of the present invention;

[0053] Figure 3 Schematic diagram of the application architecture of the embodiment;

[0054] Figure 4 Schematic diagram of the process of accessing cloud providers and chips;

[0055] Figure 5 Schematic diagram of the fault tolerance recovery process of the training task. Detailed implementation manners

[0056] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Embodiment

[0058] A fault tolerance recovery system applied to a heterogeneous training cluster includes a scheduling layer and a functional layer. Among them, the functional layer is provided with a predefined unified interface for accessing different cloud providers and different chips, and the functional layer is used to execute the processes of log monitoring, status monitoring, node detection, and alarm notification;

[0059] The scheduling layer is used to execute the processes of training task monitoring, training task inspection, fault analysis, and training recovery.

[0060] Based on the above system, a fault tolerance recovery method applied to a heterogeneous training cluster is implemented, as Figure 2 shown, including the following steps:

[0061] S1. Based on the predefined unified interface, access multiple cloud providers and multiple types of chips;

[0062] S2. Execute the fault tolerance recovery process: poll the status of each training task, perform bad node detection, remove the bad node and then restart the training task, and at the same time perform corresponding alarm prompts;

[0063] And check and handle exceptions for the parameter service.

[0064] This embodiment applies the above solution to build an application framework as Figure 3 shown, and the main contents are:

[0065] I. Unified interface

[0066] 1.1. Definition of data structure

[0067] Abstract task data structure, abstract cloud provider data structure, bad node data structure, inspection information briefing data structure

[0068] 1.2. Definition of unified interface

[0069] Unified task interface definition, including: ① Starting a task; ② Obtaining the task status through a cloud provider; ③ Deleting bad nodes; ④ Stopping a task; ⑤ Obtaining the most recent loss value of the task from the log; ⑥ Judging whether the task is stuck according to the log; ⑦ Obtaining the training task performance value from the log; ⑧ Obtaining other custom monitoring metrics; ⑨ Cloud provider identifier used.

[0070] Unified cloud provider interface, including: ① Starting a task according to the task ID; ② Obtaining the task status according to the task ID; ③ Deleting bad nodes; ④ Stopping a task according to the task ID.

[0071] II. Accessing Multiple Cloud Providers and Multiple Chips Based on the Unified Interface

[0072] As Figure 4 shown, each training task and parameter service are respectively accessed through the unified task interface. Among them, the parameter service realizes: ① Starting a task; ② Obtaining the task status through a cloud provider; ③ Stopping a task; ④ Cloud provider used; ⑤ Obtaining custom monitoring metrics.

[0073] After multiple cloud providers are accessed through the unified cloud provider interface, a function implementation for starting a task is provided. By calling the cloud provider service interface, the function of starting a training task is realized;

[0074] A function implementation for obtaining the task status is provided. By calling the cloud provider service interface, the function of obtaining the current status of the training task is realized;

[0075] A function implementation for deleting bad nodes is provided. By calling the cloud provider service interface, the function of deleting bad nodes is realized;

[0076] A function implementation for stopping a task is provided. By calling the cloud provider service interface, the function of stopping a task is realized;

[0077] A function for obtaining the log is provided.

[0078] In addition, for heterogeneous chip adaptation of the training framework, each chip adapts the training framework and outputs its own training log as a training subtask. In practical applications, only the function of analyzing the training log needs to be developed to obtain the following metrics:

[0079] Obtaining the most recent loss value of the task;

[0080] Obtaining the tgs (performance value) of the training task;

[0081] Judging whether the training task is stuck.

[0082] III. Fault Tolerance and Recovery Process

[0083] Design an alarm module and a task inspection module. Among them, the alarm module is used to achieve: ① Task restart alarm; ② Training task stuck alarm; ③ Alarm for specific strings in the log; ④ Detection of bad node alarm; ⑤ Abnormal task status; ⑥ Loss (loss value) abnormal alarm; ⑦ Training task performance lower than normal value alarm.

[0084] The task inspection module is used to collect the following: ① Task status; ② Average tgs (performance metric), minimum loss value, average loss value, maximum loss value of the last 50 training iterations (iter); ③ Loss (loss value) curve graph.

[0085] During the execution process, the fault-tolerant scheduling collects training task information and sends the information to the alarm service, and the alarm service determines whether to trigger an alarm;

[0086] The inspection scheduling collects training task inspection information and sends the task inspection information to the alarm tool.

[0087] The specific fault-tolerant recovery process includes the following three sub-processes:

[0088] 1. As Figure 5 shown, check each training sub-task in turn, and the abnormal situation handling process:

[0089] ① Abnormal task status, trigger the task status abnormal alarm. Stop the training sub-task, perform bad node detection, and restart the sub-task after removing the bad node;

[0090] ② Analyze the loss (loss value) abnormality from the log and trigger the loss (loss value) abnormal alarm.

[0091] ③ The training task is stuck, trigger the task stuck alarm. Stop the training sub-task, perform bad node detection, and restart the sub-task after removing the bad node;

[0092] ④ The training task performance is lower than the normal value, trigger the low performance alarm. Stop the training sub-task, perform bad node detection, and restart the sub-task after removing the bad node.

[0093] ⑤ Other custom metric abnormalities, trigger an alarm.

[0094] 2. Check the parameter service, and the abnormal situation handling process:

[0095] ① Abnormal task status, trigger the task status abnormal alarm, and restart the task after stopping the task.

[0096] ② Custom monitoring metric abnormality, trigger an alarm.

[0097] 3. Task inspection process

[0098] Execute once at a fixed time interval to collect: ① task status; ② the average tgs (performance metric), the minimum value of loss, the average value of loss, and the maximum value of loss for the last 50 training iterations (iters); ③ the loss (value) curve graph, and combine them into a briefing and send it to the communication tool used by the user.

[0099] In summary, on the one hand, this solution defines a unified interface for training clusters of heterogeneous chips from multiple cloud providers (including defining multiple unified abstract data structures, defining a unified task interface, and a unified cloud provider interface), and adapts the unified interface for heterogeneous chips from multiple cloud providers (multiple cloud providers implement the unified cloud provider interface, and tasks running on different chips adapt to implement the unified task interface, so as to achieve the decoupling of the fault tolerance process and the differences of cloud providers and chips). As a result, it is more convenient to expand various heterogeneous chips for fault tolerance, and the chip access cost is lower.

[0100] On the other hand, it defines a fault tolerance process, including a fault recovery method based on multiple cloud providers and multiple heterogeneous chips, and defines the processing processes for various faults, which can support the training fault tolerance of a heterogeneous training cluster with tens of thousands of cards across multiple cloud providers, greatly reducing costs and improving training efficiency.

[0101] Apply this solution to practice. In an actual training of a training task using multiple domestic chips provided by two cloud providers with a scale of tens of thousands of cards for several months, it has been verified that this solution can achieve the above technical effects.

Claims

1. A fault tolerance and recovery system applied to a heterogeneous training cluster, characterized in that It includes a scheduling layer and a function layer. The function layer is provided with a predefined unified interface for accessing different cloud providers and different chips, and is used to execute log monitoring, status monitoring, node detection, and alarm notification processes; The scheduling layer is used to execute training task monitoring, training task inspection, fault analysis, and training recovery processes.

2. The fault tolerance and recovery system for heterogeneous training clusters according to claim 1, wherein The unified interface includes a data structure, a unified task interface, and a unified cloud provider interface.

3. The fault tolerance and recovery system for heterogeneous training clusters according to claim 2, wherein, The data structure includes an abstract task data structure, an abstract cloud provider data structure, a bad node data structure, and an inspection information briefing data structure.

4. A fault tolerance and recovery system applied to a heterogeneous training cluster according to claim 2, wherein, The unified task interface includes: Start a task; Obtain the task status through the cloud provider; Delete a bad node; Stop a task; Obtain the most recent loss value of the task from the log; Judge whether the task is stuck according to the log; Obtain the training task performance value from the log; Obtain other custom monitoring metrics; The cloud provider identifier used.

5. A fault tolerance and recovery system applied to a heterogeneous training cluster according to claim 4, wherein, The unified cloud provider interface includes: Start a task according to the task ID; Obtain the task status according to the task ID; Delete a bad node; Stop a task according to the task ID.

6. A fault tolerance and recovery method applied to a heterogeneous training cluster, implemented based on a fault tolerance and recovery system applied to a heterogeneous training cluster as described in claim 5, characterized in that, It includes the following steps: S1. Based on the predefined unified interface, access multiple cloud providers and multiple types of chips; S2. Execute the fault tolerance and recovery process: Poll the status of each training task, perform bad node detection, remove the bad node and then restart the training task, and at the same time give corresponding alarm prompts; And check and handle exceptions for the parameter service.

7. A fault tolerance and recovery method applied to a heterogeneous training cluster according to claim 6, characterized in that The specific process of step S1 is: S11. Based on the predefined unified task interface, access each training task and the parameter service respectively; Based on the predefined unified cloud provider interface, access multiple cloud providers; S12. For each heterogeneous chip, adapt the corresponding training framework.

8. A fault tolerance and recovery method applied to a heterogeneous training cluster according to claim 7, characterized in that, In step S11, accessing multiple cloud providers specifically utilizes the unified cloud provider interface. By calling the cloud provider service interface, starting a training task, obtaining the task status, deleting a bad node, and stopping a task are respectively implemented. In addition, the unified cloud provider interface is used to obtain logs.

9. A fault tolerance and recovery method applied to a heterogeneous training cluster according to claim 6, characterized in that In step S2, polling the status of each training task specifically means performing a task inspection process once according to a preset time interval to collect the task status; the average tgs of the most recent M training iterations, the minimum loss value, the average loss value, the maximum loss value; the loss curve graph.

10. A fault tolerance and recovery method applied to a heterogeneous training cluster according to claim 9, characterized in that, The specific process of step S2 is: S21. Check each training subtask in turn and handle abnormal situations: The task status is abnormal, trigger an alarm for abnormal task status, stop the training subtask, perform bad node detection, remove the bad node and then restart the subtask; Analyze the loss anomaly from the log and trigger an alarm for loss anomaly; The training task is stuck, trigger an alarm for task stuck, stop the training subtask, perform bad node detection, remove the bad node and then restart the subtask; The performance of the training task is lower than the preset normal value, trigger an alarm for low performance, stop the training subtask, perform bad node detection, remove the bad node and then restart the subtask; Other preset custom metric anomalies, trigger an alarm; S22. Check the parameter service and handle abnormal situations: The task status is abnormal, triggering an alarm for abnormal task status. Stop the task and then restart it. The preset custom monitoring metric is abnormal, triggering an alarm.

Images