Monitoring Method for Large Model Training in Heterogeneous Clusters and Electronic Device

Abstract

The present application provides a monitoring method and electronic device for large-model training of a heterogeneous cluster. The monitoring method for large-model training of a heterogeneous cluster includes: obtaining the current training data of the large model on each device in the heterogeneous cluster that is in the startup state; determining the current performance data used to characterize the training process based on the current training data; if the current performance data is not within the normal range, determining the fault information based on the historical performance data and the current performance data during the last training process; determining whether the large-model training is abnormal based on the fault information, so as to monitor the training process of the large model. The present application can monitor the training process of the large-model training and promptly discover abnormal situations such as faults.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and machine learning technology, and in particular to a monitoring method and electronic device for large-scale model training of heterogeneous clusters. Background Art

[0002] With the rapid development of artificial intelligence (AI), large-scale deep learning models have been widely used in various fields. Training large models typically consumes a significant amount of computing resources and time, especially in heterogeneous cluster environments. Differences in hardware configurations across devices can easily lead to various anomalies during training, such as hardware and network failures.

[0003] Currently, there is a lack of effective monitoring methods for the large-scale model training process, making it difficult to detect abnormalities in the training process in a timely manner, which can easily lead to interruption or failure of the training task. Summary of the Invention

[0004] The present application provides a monitoring method and electronic device for large-scale model training of heterogeneous clusters, which can improve training efficiency.

[0005] This application provides a monitoring method for large model training of heterogeneous clusters, including:

[0006] Obtain the current training data of the large model on each device in the heterogeneous cluster that is in the startup state;

[0007] Determining, based on the current training data, current performance data for characterizing the training process;

[0008] If the current performance data is not within the normal range, determine fault information based on the historical performance data from the previous training process and the current performance data;

[0009] Based on the fault information, it is determined whether an abnormality occurs in the large model training to monitor the training process of the large model.

[0010] Optionally, obtaining the current training data of the large model on each device in the heterogeneous cluster that is in an activated state includes:

[0011] receiving an operation mode and a scheduling strategy for the large model training, where the operation mode and the scheduling strategy are set according to the training task of the large model training and parameter information of the heterogeneous cluster;

[0012] Initiate the large model training on the heterogeneous cluster according to the operation mode and the scheduling strategy;

[0013] During the large model training process, collecting operation data;

[0014] Simulate expansion of the heterogeneous cluster and collect post-expansion operation data;

[0015] The current training data is obtained according to the operating data and the expanded operating data.

[0016] Optionally, determining the current performance data for characterizing the training process based on the current training data includes:

[0017] In the training data of the current session, performance-related data is collected; the performance-related data includes at least one of a task start time, a task completion time, a training start time, a training end time, a start time of each iteration, an end time of each iteration, a computing power value of each iteration, a loading breakpoint start time, a loading breakpoint completion time, a saving breakpoint start time, a saving breakpoint completion time, a fault occurrence time, and a fault recovery time;

[0018] The current performance data is determined based on the performance-related data.

[0019] Optionally, the performance-related data include the start time and the end time of each round of iteration, and the current performance data includes the effective training duration; determining the current performance data based on the performance-related data includes: taking the difference between the end time of each round of iteration and the start time of each round of iteration as the effective training duration.

[0020] Optionally, the performance-related data collected in the current training data includes:

[0021] According to the preset sampling target and sampling interval, performance-related data is collected from the training data at that time.

[0022] Optionally, the performance-related data includes application layer performance-related data and micro-architecture layer performance-related data.

[0023] Optionally, the fault information includes a fault discovery time and a fault recovery time; and determining the fault information based on the historical performance data from the previous training process and the current performance data includes:

[0024] Determine the fault discovery time based on the difference between the start time of this training and the end time of the previous training;

[0025] The fault recovery time is determined according to the difference between the start time of this training and the completion time of the last save breakpoint in the previous training.

[0026] Optionally, the determining of the fault information includes: when the heterogeneous cluster is elastically expanded, respectively determining the fault information on each device in the startup state in the heterogeneous cluster.

[0027] Optionally, the monitoring method further includes: issuing an alarm when an abnormality occurs in the large model training.

[0028] Optionally, the monitoring method further includes: sending the fault information to the training task, so that the training task dynamically allocates the large model training in the heterogeneous cluster according to the fault information.

[0029] The present application provides an electronic device comprising one or more processors for implementing a monitoring method for large model training of a heterogeneous cluster as described in any one of the above items.

[0030] The present application also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the monitoring method for large-scale model training of heterogeneous clusters as described in any of the above items is implemented.

[0031] In some embodiments, the current performance data used to characterize the training process is determined based on the current training data of the large model on each device in the heterogeneous cluster that is in the started state; if the current performance data is not within the normal range, the fault information is determined based on the historical performance data and the current performance data during the last training process; based on the fault information, it is determined whether an abnormality has occurred in the large model training. In this way, the training process of the large model can be monitored and abnormal situations such as faults can be discovered in a timely manner.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 Shown is a flowchart of an embodiment of a method for monitoring large model training of a heterogeneous cluster of the present application.

[0035] Figure 2 Shown is a partial schematic diagram of an embodiment of the training process of the large model training of the present application.

[0036] Figure 3 Shown is a partial schematic diagram of another embodiment of the training process of large model training of the present application.

[0037] Figure 4 Shown is a structural block diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0038] The present application provides a method and electronic device for monitoring large-scale model training of heterogeneous clusters. The method and electronic device for monitoring large-scale model training of heterogeneous clusters of the present application are described in detail below with reference to the accompanying drawings. The features of the following embodiments and implementations may be combined with each other unless they conflict.

[0039] Figure 1 FIG. 1 is a flow chart of an embodiment of a method 10 for monitoring large model training of heterogeneous clusters of the present application. Figure 1 As shown, the monitoring method 10 for large model training of heterogeneous clusters includes: steps 11 to 15.

[0040] Step 11: Obtain the current training data of the large model on each device in the heterogeneous cluster that is in the startup state.

[0041] Heterogeneous clusters include different types of computing devices (such as CPUs, GPUs, and TPUs). When training large models, these devices work together. At the same time, some of these devices may be powered on, while others may be powered off.

[0042] When training a large model, a training task may need to be restarted multiple times. For example, a large model training task requires 100 epochs. During the first training session, starting from epoch 0, a failure occurs during epoch 10, interrupting the training and ending the training session. During the second training session, training is resumed from epoch 11.

[0043] When a device is powered on and participating in training, the current training data for each device is obtained. This data includes input sample data, model parameter data, and device operating status data (such as computing resource usage and memory usage). This data allows you to understand the actual status of the current training.

[0044] Step 12: Determine the current performance data for characterizing the training process based on the current training data.

[0045] The current performance data indicates whether the training process is normal. The current training data contains a large amount of information, and it is necessary to extract the current performance data that can effectively represent the training process, such as memory usage, video memory usage, power consumption fluctuations, and the time taken for different training phases.

[0046] Step 13: Determine whether the current performance data is within a normal range.

[0047] You can compare the current performance data with a preset performance threshold to determine whether it is within the normal range. The performance threshold can be determined based on factors such as previous model training experience and the expected performance of the model. If the current performance data is within the normal range, the current training process is generally normal; if it is outside the normal range, there may be potential problems.

[0048] If the performance data is within the normal range, the monitoring method 10 is terminated. When training is performed again, the monitoring method 10 is restarted.

[0049] Step 14: If the current performance data is not within the normal range, determine the fault information based on the historical performance data during the last training process and the current performance data.

[0050] If the current performance data is not within the normal range, an anomaly may have occurred during training. Relying solely on the current performance data may not accurately identify the fault. Therefore, it is necessary to refer to historical performance data from previous training sessions, such as the last training start and end time. By combining historical performance data with the current performance data, fault information can be determined, such as the duration of the rollback training, the time it took to discover the fault, and the time it took to recover from the fault.

[0051] Step 15: Determine whether an abnormality occurs in the large model training based on the fault information to monitor the training process of the large model.

[0052] Based on the fault information, determine whether a large model training anomaly has occurred, such as a program error, memory overflow, network anomaly, or device anomaly. For example, determine whether a large model training anomaly has occurred by checking whether the rollback training duration is within the rollback training threshold. You can also use multiple indicators to comprehensively determine whether an anomaly exists.

[0053] In some embodiments, performance data representing the training process is determined based on the current training data of the large model on each device in the heterogeneous cluster that is in an activated state. If the current performance data is not within a normal range, fault information is determined based on historical performance data from the previous training process and the current performance data. Based on the fault information, it is determined whether an abnormality has occurred in the large model training. In this way, the large model training process can be monitored and abnormalities such as faults can be discovered in a timely manner. If an abnormality occurs, training can be restarted in a timely manner to improve training efficiency.

[0054] In some embodiments, step 11 includes: receiving the operating mode and scheduling strategy of the large model training, the operating mode and scheduling strategy are set according to the training tasks of the large model training and the parameter information of the heterogeneous cluster; starting the large model training on the heterogeneous cluster according to the operating mode and scheduling strategy; collecting operating data during the large model training process; performing simulation expansion on the heterogeneous cluster and collecting the operating data after expansion; obtaining the current training data based on the operating data and the operating data after expansion.

[0055] The training tasks for large models are known. Parameter information for heterogeneous clusters can also be acquired in advance. This information includes data such as the CPU (Central Processing Unit) model, GPU (Graphics Processing Unit) architecture, FPGA (Field-Programmable Gate Array), and TPU (Tensor Processing Unit). CPU models include the AtomicSimple model, TimingSimple model, InOrder model, and O3 model. GPU architectures include Tesla, Fermi, Kepler, Volta, and Turing.

[0056] Large model training can be run in serial or parallel modes. The scheduling strategy includes parameter configuration files for different hardware combinations that can execute the scheduling strategy.

[0057] Based on the scheduling strategy, the large model is run according to the set operation mode. During the training process, operation data is collected. The operation data is the operation behavior characteristics of the large model.

[0058] Kubernetes-based virtualization technology allows for the simulation and expansion of cluster nodes within a large, running model. For example, by deploying a cluster using Kubernetes and simulating the characteristics of physical machines using the Kubernetes container orchestration tool, a small number of nodes can be used to simulate a large-scale cluster. This allows for the simulation of large-scale heterogeneous cluster operations and the generation of expanded operational data.

[0059] In some embodiments, step 12 includes: collecting performance-related data in the current training data; the performance-related data includes at least one of the task start time, task completion time, training start time, training end time, each round of iteration start time, each round of iteration end time, computing power value of each round of iteration, loading breakpoint start time, loading breakpoint completion time, saving breakpoint start time, saving breakpoint completion time, fault occurrence time, and fault recovery time; determining the current performance data based on the performance-related data.

[0060] The training data contains a large amount of information, and it is necessary to filter out performance-related data related to the current performance data. By analyzing and processing this performance-related data, the current performance data can be obtained. Performance-related data includes application-layer performance data and microarchitecture-layer performance data. Application-layer performance data includes CPU utilization, memory utilization, disk I / O size, network bandwidth, etc. Microarchitecture-layer performance indicators include IPC, branch prediction, and cache misses.

[0061] In some embodiments, collecting performance-related data in the current training data includes: collecting performance-related data in the current training data according to a preset sampling target and sampling interval.

[0062] Specifically, a script is configured to collect performance data related to the application layer and microarchitecture layer of the large model, generating a data collection configuration file that allows users to freely select performance data and sampling intervals. Based on the sampling intervals in the data collection configuration file, data is collected from the large model during simulation, generating performance data such as the time cost to complete each training session, average job turnaround time, and changes in overall cluster resource utilization. The required data is then selected from the obtained performance data to determine the performance data for that session.

[0063] In some embodiments, the sampling interval is 10 seconds or 30 seconds.

[0064] Figure 2 Shown is a partial schematic diagram of an embodiment of the training process of the large model training of the present application.

[0065] In some embodiments, the performance-related data includes the start time and the end time of each round of iteration, and the performance data includes the effective training duration; determining the performance data based on the performance-related data includes: taking the difference between the end time of each round of iteration and the start time of each round of iteration as the effective training duration.

[0066] The performance data also includes: model initialization time, training time, effective training time, effective training time ratio, loading breakpoint time, and saving breakpoint time.

[0067] Determining the current performance data based on the performance-related data further includes:

[0068] The difference between the start time of this training and the start time of the task is used as the model initialization duration;

[0069] The difference between the end time and the start time of training is taken as the training duration;

[0070] The difference between the completion time of each round of iteration and the start time of each round of iteration in this training is used as the duration of each round of iteration; the duration of each round of iteration in this training is accumulated as the effective training duration;

[0071] The ratio of effective training time to training time is taken as the effective training time ratio;

[0072] The difference between the end time of the loading breakpoint and the start time of the loading breakpoint is used as the loading breakpoint duration;

[0073] The difference between the save breakpoint completion time and the save breakpoint start time is used as the save breakpoint duration.

[0074] Combining historical performance data with current performance data yields cumulative performance data. This data includes total training time, total effective training time, percentage of total effective training time, and total saved breakpoint duration. This cumulative performance data can be used to identify any anomalies, such as failures, that occurred throughout the training process of a large model.

[0075] The method for determining the cumulative performance data is as follows: add up the training duration of multiple training sessions during the training process to obtain the total training duration; add up the effective training duration of multiple training sessions and reduce the rollback training duration to obtain the total effective training duration; take the ratio of the total effective training duration to the total training duration as the proportion of the total effective training duration; add up the duration of multiple saved breakpoints to obtain the total saved breakpoint duration.

[0076] In some embodiments, determining the current performance data based on the performance-related data also includes: determining whether the performance-related data meets the performance threshold; if the performance threshold is met, determining the current performance data based on the performance-related data; if the performance threshold is not met, re-collecting the performance-related data.

[0077] The performance threshold is a pre-set threshold used to mark whether performance-related data is normal data or abnormal data.

[0078] Figure 3 Shown is a partial schematic diagram of another embodiment of the training process of large model training of the present application.

[0079] In some embodiments, the fault information includes the fault discovery time and the fault recovery time; and determining the fault information based on the historical performance data and the current performance data during the previous training process includes:

[0080] Determine the fault discovery time based on the difference between the start time of this training and the end time of the previous training;

[0081] The fault recovery time is determined based on the difference between the start time of this training and the completion time of the last save breakpoint in the previous training.

[0082] Fault information also includes: rollback training duration, fault discovery time, fault recovery time, number of faults, and number of restarts.

[0083] The determining of fault information based on the historical performance data and the current performance data during the previous training process further includes:

[0084] The rollback training duration is calculated by accumulating the duration of each iteration (i.e., the effective training duration) after the last save breakpoint is completed.

[0085] The difference between the start time of this training and the end time of the previous training is taken as the fault discovery time;

[0086] The difference between the start time of this training and the completion time of the last breakpoint save in the previous training is used as the fault recovery time.

[0087] Determine the number of failures and restarts based on the number of interruptions that occurred during the entire training task.

[0088] In some embodiments, step 14 includes: elastically expanding the amount of resources for the training task; adopting an opportunity retention mechanism to perform the training task; determining the historical performance data when the last interruption occurred; and determining fault information based on the historical performance data and the current performance data.

[0089] Elastically scaling training resources includes dynamically changing the number of GPUs and switching GPU types. For example, increasing the number of GPUs from 4 to 8, or changing the GPU type from NVIDIA Tesla V100 to NVIDIA A100.

[0090] In a heterogeneous cluster, when a device is idle, it is automatically expanded, and training data is collected on the expanded device to determine the fault information.

[0091] In some embodiments, the “determining fault information” described in step 14 includes: when the heterogeneous cluster is elastically expanded, determining fault information on each device in the started state in the heterogeneous cluster.

[0092] On each device in the heterogeneous cluster that is in the startup state, its training data is collected and the fault information is further determined.

[0093] In some embodiments, the monitoring method 10 further includes: recording the current performance data into a training log and saving it into a database; and recording event information during the training process into an event information table and saving it into a database.

[0094] Event information can be directly collected from training data or derived from collected performance-related data. Event information includes: task start time, task start command, training start time, log output at the start of training, fault occurrence time, fault error log, training end time, training end log, task completion time, and training completion log.

[0095] Recording the performance data and event information of the time is conducive to subsequent overall analysis of the training task and improving the effectiveness of training.

[0096] In some embodiments, the monitoring method 10 further includes: issuing an alarm when an abnormality occurs in the large model training.

[0097] When a large model training anomaly is detected, an alert can be issued in various ways, such as through sound or icons on the user terminal, or through email or instant messaging tools, to inform the user of the anomaly. The user can then promptly review the fault, troubleshoot the problem, and restart the task.

[0098] In some embodiments, the monitoring method 10 further includes: sending fault information to the training task, so that the training task dynamically allocates large model training in the heterogeneous cluster according to the fault information.

[0099] After collecting the fault information, the electronic device of the present application will send it to the training task. The communication method between the electronic device and the management module of the training task can be message passing based on the network protocol, such as using the TCP / IP protocol to encapsulate and transmit the fault information in a specific data format (such as JSON or XML). After receiving the fault information, the management module of the training task analyzes and evaluates it. According to the severity and scope of the fault, the large model training tasks are redistributed in the heterogeneous cluster to ensure the continuity and efficiency of the training. The Gossip algorithm can be used to dynamically allocate computing tasks according to the situation of the devices in the heterogeneous cluster. For example, for devices with low memory usage and power, more computing tasks can be allocated to balance the training time of different devices and speed up the training progress.

[0100] Figure 4 Shown is a structural block diagram of an embodiment of the electronic device of the present application.

[0101] like Figure 4 As shown, the electronic device includes one or more processors 21 for implementing the monitoring method 10 for large model training of heterogeneous clusters as described above.

[0102] In some embodiments, the electronic device may include a computer-readable storage medium 22, which may store a program that can be called by the processor 21 and may include a non-volatile storage medium. In some embodiments, the electronic device may include a memory 23 and an interface 24. In some embodiments, the electronic device may also include other hardware depending on the actual application.

[0103] The computer-readable storage medium 22 of the embodiment of the present application stores a program thereon, which, when executed by the processor 21, is used to implement the monitoring method 10 for large-scale model training of heterogeneous clusters as described above.

[0104] The present application may take the form of a computer program product implemented on one or more computer-readable storage media 22 (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media 22 include both permanent and non-permanent, removable and non-removable media, and may implement information storage using any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media 22 include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

Claims

1. A monitoring method for large model training of heterogeneous clusters, characterized in that: include: Obtain the current training data of the large model on each device in the heterogeneous cluster that is in the startup state; Determining, based on the current training data, current performance data for characterizing the training process; If the performance data for the current session is not within the normal range, fault information is determined based on the historical performance data from the previous training session and the performance data for the current session; wherein the fault information includes the fault discovery time and the fault recovery time; the fault discovery time is determined based on the difference between the start time of the current training session and the end time of the previous training session; and the fault recovery time is determined based on the difference between the start time of the current training session and the completion time of the last saved breakpoint in the previous training session. Based on the fault information, it is determined whether an abnormality occurs in the large model training to monitor the training process of the large model.

2. The method for monitoring large model training of heterogeneous clusters according to claim 1, characterized in that: The step of obtaining the current training data of the large model on each device in the heterogeneous cluster that is in an activated state includes: receiving an operation mode and a scheduling strategy for the large model training, where the operation mode and the scheduling strategy are set according to the training task of the large model training and parameter information of the heterogeneous cluster; Initiate the large model training on the heterogeneous cluster according to the operation mode and the scheduling strategy; During the large model training process, collecting operation data; Simulate expansion of the heterogeneous cluster and collect post-expansion operation data; The current training data is obtained according to the operating data and the expanded operating data.

3. The method for monitoring large model training of heterogeneous clusters according to claim 1, characterized in that: The determining, based on the current training data, the current performance data for characterizing the training process includes: In the training data of the current session, performance-related data is collected; the performance-related data includes at least one of a task start time, a task completion time, a training start time, a training end time, a start time of each iteration, an end time of each iteration, a computing power value of each iteration, a loading breakpoint start time, a loading breakpoint completion time, a saving breakpoint start time, a saving breakpoint completion time, a fault occurrence time, and a fault recovery time; The current performance data is determined based on the performance-related data.

4. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: The performance-related data includes the start time and end time of each iteration, and the performance data includes the effective training time. Determining the current performance data based on the performance-related data includes: taking the difference between the end time of each round of iteration and the start time of each round of iteration as the effective training duration.

5. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: In the training data, the performance-related data collected includes: According to the preset sampling target and sampling interval, performance-related data is collected from the training data at that time.

6. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: The performance-related data includes application layer performance-related data and micro-architecture layer performance-related data.

7. The method for monitoring large model training of heterogeneous clusters according to claim 1, characterized in that: The determining of the fault information includes: when the heterogeneous cluster is elastically expanded, determining the fault information on each device in the startup state in the heterogeneous cluster; and / or, The monitoring method further includes: issuing an alarm when an abnormality occurs in the large model training; and / or, The monitoring method further includes: sending the fault information to a training task, so that the training task dynamically allocates the large model training in the heterogeneous cluster according to the fault information.

8. An electronic device, characterized in that: The method comprises one or more processors for implementing the monitoring method for large model training of a heterogeneous cluster as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the monitoring method for large model training of a heterogeneous cluster as described in any one of claims 1 to 7 is implemented.

Images