Heterogeneous cluster large model training monitoring method and electronic equipment

Abstract

The invention provides a monitoring method for large model training of a heterogeneous cluster and electronic equipment. The monitoring method for large model training of the heterogeneous cluster comprises the steps of obtaining current training data of a large model on each device in a starting state in the heterogeneous cluster; according to the current training data, determining current performance data used for representing the training process; if the current performance data is not in the normal range, determining fault information according to the historical performance data in the last training process and the current performance data; and determining whether large model training is abnormal or not according to the fault information so as to monitor the training process of the large model. According to the method, the training process of large model training can be monitored, and abnormal conditions such as faults can be found in time.

Description

Technical Field

[0001] This application relates to the technical fields of artificial intelligence and machine learning, and particularly relates to a monitoring method for large model training in a heterogeneous cluster and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence technology, large-scale deep learning models have been widely used in various fields. Training large models usually requires a large amount of computing resources and time. Especially in a heterogeneous cluster environment, due to differences in the hardware configurations of different devices, various abnormal situations are likely to occur during the training process, such as hardware failures and network failures.

[0003] Currently, there is a lack of an effective monitoring method for the process of large model training, making it difficult to detect abnormal situations during the training process in a timely manner, which easily leads to the interruption or failure of the training task. Summary of the Invention

[0004] This application provides a monitoring method for large model training in a heterogeneous cluster and an electronic device, which can improve the training efficiency.

[0005] This application provides a monitoring method for large model training in a heterogeneous cluster, including: Obtaining the current training data on each device in the heterogeneous cluster where the large model is in the startup state; Determining the current performance data used to characterize the training process according to the current training data; If the current performance data is not within the normal range, determining fault information according to the historical performance data and the current performance data during the previous training process; Determining whether an abnormality occurs in the large model training according to the fault information to monitor the training process of the large model.

[0006] Optionally, the obtaining the current training data on each device in the heterogeneous cluster where the large model is in the startup state includes: Receiving the running mode and scheduling strategy of the large model training, where the running mode and the scheduling strategy are set according to the training task of the large model training and the parameter information of the heterogeneous cluster; Starting the large model training on the heterogeneous cluster according to the running mode and the scheduling strategy; Collecting running data during the large model training process; Performing a simulation expansion on the heterogeneous cluster and collecting the running data after expansion; Obtaining the current training data according to the running data and the running data after expansion.

[0007] Optionally, determining the performance data for the current training process based on the current training data includes: Collecting performance-related data from the current training data; the performance-related data includes at least one of the task start time, task completion time, training start time, training end time, start time of each iteration, end time of each iteration, computing power value of each iteration, start time of loading breakpoint, completion time of loading breakpoint, start time of saving breakpoint, completion time of saving breakpoint, fault occurrence time, and fault recovery time. Determining the current performance data based on the performance-related data.

[0008] Optionally, the performance-related data includes the start time of each iteration and the end time of each iteration, and the current performance data includes the effective training duration; determining the current performance data based on the performance-related data includes: taking the difference between the end time of each iteration and the start time of each iteration as the effective training duration.

[0009] Optionally, collecting performance-related data from the current training data includes: Collecting performance-related data from the current training data according to a preset sampling target and sampling interval.

[0010] Optionally, the performance-related data includes application layer performance-related data and microarchitecture layer performance-related data.

[0011] Optionally, the fault information includes the fault discovery duration and the fault recovery duration; determining the fault information based on the historical performance data in the previous training process and the current performance data includes: Determining the fault discovery duration based on the difference between the start time of this training and the end time of the previous training; Determining the fault recovery duration based on the difference between the start time of this training and the completion time of the last saved breakpoint in the previous training.

[0012] Optionally, determining the fault information includes: when the heterogeneous cluster is elastically expanded, determining the fault information on each device in the start state in the heterogeneous cluster respectively.

[0013] Optionally, the monitoring method further includes: sending an alarm prompt when an abnormality occurs in the large model training.

[0014] Optionally, the monitoring method further includes: sending the fault information to the training task so that the training task dynamically allocates the large model training in the heterogeneous cluster according to the fault information.

[0015] The present application provides an electronic device, including one or more processors, for implementing the monitoring method of large model training of the heterogeneous cluster as described in any one of the above.

[0016] The present application also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the monitoring method of large model training of the heterogeneous cluster as described in any one of the above is implemented.

[0017] In some embodiments, according to the current training data on each device in the heterogeneous cluster where the large model is in the startup state, the current performance data for characterizing the training process is determined; if the current performance data is not within the normal range, according to the historical performance data and the current performance data in the previous training process, the fault information is determined; according to the fault information, it is determined whether an abnormality occurs in the large model training. In this way, the training process of the large model can be monitored, and abnormal situations such as faults can be detected in a timely manner.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0020] Figure 1 Shown is a flowchart of an embodiment of the monitoring method for large model training of the heterogeneous cluster of the present application.

[0021] Figure 2 Shown is a partial schematic diagram of an embodiment of the training process of the large model training of the present application.

[0022] Figure 3 Shown is a partial schematic diagram of another embodiment of the training process of the large model training of the present application.

[0023] Figure 4 Shown is a structural block diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The present application provides a monitoring method and an electronic device for large model training of a heterogeneous cluster. The following will describe in detail the monitoring method and the electronic device for large model training of the heterogeneous cluster of the present application with reference to the accompanying drawings. Without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0025] Figure 1 Shown is a flowchart of an embodiment of the monitoring method 10 for large model training of the heterogeneous cluster of the present application. As Figure 1As shown, the monitoring method 10 for large model training in a heterogeneous cluster includes: steps 11 to 15.

[0026] Step 11, obtain the current training data on each device in the heterogeneous cluster that is in the startup state during large model training.

[0027] The heterogeneous cluster includes different types of computing devices (such as CPUs, GPUs, TPUs, etc.). During large model training, these devices work together. At the same moment, among these devices, some may be in the startup state and some may be in the shutdown state.

[0028] During large model training, a training task may need to be restarted multiple times. For example, the large model training task needs to be trained for 100 rounds. In the first training, starting from round 0, a failure occurs during training from round 0 to round 10, and the training is interrupted, ending this training. In the second training, starting from round 11, the training continues.

[0029] When the device is in the startup state and participates in training, obtain the current training data on each device. The current training data includes input sample data, parameter data of the model, operating state data of the device (such as the usage of computing resources, memory occupancy, etc.), etc. Through the current training data, the actual situation of the current training can be understood.

[0030] Step 12, determine the current performance data used to characterize the training process based on the current training data.

[0031] The current performance data is data used to characterize whether the training process is normal. The current training data contains a large amount of information, and it is necessary to extract the current performance data that can effectively characterize the training process from it, such as memory occupancy, video memory occupancy, power consumption fluctuations, and the time consumption in different training stages, etc.

[0032] Step 13, determine whether the current performance data is within the normal range.

[0033] It can be determined whether the current performance data is within the normal range by comparing the current performance data with a preset performance threshold. The performance threshold can be determined based on factors such as previous model training experience and the expected performance of the model. If the current performance data is within the normal range, it indicates that the current training process is basically normal; if it exceeds the normal range, there may be potential problems.

[0034] If the current performance data is within the normal range, end the monitoring method 10. When training is carried out again, restart the monitoring method 10.

[0035] Step 14, if the current performance data is not within the normal range, determine the fault information based on the historical performance data and the current performance data in the previous training process.

[0036] If the current performance data is not within the normal range, it indicates that an abnormality may have occurred during the training process. Relying solely on the current performance data may not accurately determine the fault information. Therefore, it is necessary to refer to the historical performance data during the previous training process, such as the start time and end time of the previous training. By combining the historical performance data and the current performance data, fault information can be determined, such as the rollback training duration, fault discovery duration, fault recovery duration, etc.

[0037] Step 15, determine whether an abnormality has occurred in the large model training according to the fault information to monitor the training process of the large model.

[0038] Judge whether an abnormality has actually occurred in the large model training according to the fault information, such as program errors, memory overflows, network abnormalities, device abnormalities, etc. For example, judge whether an abnormality has occurred in the large model training according to whether the rollback training duration is within the rollback training threshold range. Multiple indicators can also be used to comprehensively judge whether there are abnormal situations.

[0039] In some embodiments, according to the current training data on each device in the heterogeneous cluster where the large model is in the startup state, determine the current performance data used to characterize the training process; if the current performance data is not within the normal range, determine the fault information according to the historical performance data and the current performance data during the previous training process; determine whether an abnormality has occurred in the large model training according to the fault information. In this way, the training process of the large model can be monitored, and abnormal situations such as faults can be detected in a timely manner. After an abnormal situation occurs, the training can be restarted in a timely manner to improve the training efficiency.

[0040] In some embodiments, step 11 includes: receiving the running mode and scheduling strategy of the large model training, where the running mode and scheduling strategy are set according to the training task of the large model training and the parameter information of the heterogeneous cluster; starting the large model training on the heterogeneous cluster according to the running mode and scheduling strategy; collecting running data during the large model training process; performing simulation expansion on the heterogeneous cluster and collecting the running data after expansion; obtaining the current training data according to the running data and the running data after expansion.

[0041] The training tasks for large model training are known information. The parameter information of the heterogeneous cluster can also be obtained in advance. The parameter information of the heterogeneous cluster includes data such as the CPU (Central Processing Unit) model, GPU (Graphics Processing Unit) architecture, and FPGA (Field - Programmable Gate Array), TPU (Tensor Processing Unit), etc. The CPU models include AtomicSimple model, TimingSimple model, InOrder model, O3 model, etc. The GPU architectures include Tesla architecture, Fermi architecture, Kepler architecture, Volta architecture, Turing architecture, etc.

[0042] The running modes of large model training include serial running mode, parallel running mode, etc. The scheduling strategy includes parameter configuration files for different hardware combinations capable of executing the scheduling strategy.

[0043] Based on the scheduling strategy, the large model runs in the set running mode, and during the training process, running data is collected. The running data is the running behavior characteristics of the large model.

[0044] Through the Kubernetes virtualization technology, the cluster nodes of the running large model can be simulated and expanded. For example, by adopting a Kubernetes deployment cluster and based on the Kubernetes container orchestration tool, the characteristics of physical machines can be simulated to realize a cluster with a small number of nodes simulated as a large - scale node cluster. In this way, the operation of a large - scale heterogeneous cluster is simulated to obtain the expanded running data.

[0045] In some embodiments, step 12 includes: collecting performance - related data in the current training data; the performance - related data includes at least one of the task start time, task completion time, training start time, training end time, start time of each iteration, end time of each iteration, computing power value of each iteration, start time of loading breakpoint, completion time of loading breakpoint, start time of saving breakpoint, completion time of saving breakpoint, fault occurrence time, and fault recovery time; determining the current performance data according to the performance - related data.

[0046] The current training data contains a large amount of information, and it is necessary to screen out the performance-related data related to the current performance data. By analyzing and processing the performance-related data, the current performance data can be obtained. The performance-related data includes application layer performance-related data and microarchitecture layer performance-related data. The application layer performance-related data includes: CPU utilization rate, memory utilization rate, disk IO size, network bandwidth, etc. The microarchitecture layer performance metrics include IPC, branch prediction, and cache misses, etc.

[0047] In some embodiments, collecting performance-related data in the current training data includes: collecting performance-related data in the current training data according to a preset sampling target and sampling interval.

[0048] Specifically, script settings are performed on the application layer performance-related data and microarchitecture layer performance-related data of the large model to obtain a data collection configuration file containing the function of freely selecting performance-related data and sampling interval. Based on the sampling interval in the data collection configuration file, data collection is performed on the large model during simulation operation to obtain performance-related data, such as the time cost of each training completion, the average job turnaround time, and the change in the overall resource utilization rate of the cluster, etc. Among the obtained performance-related data, the required data is selected, and then the current performance data is determined.

[0049] In some embodiments, the sampling interval is 10 seconds or 30 seconds.

[0050] Figure 2 The figure shows a partial schematic diagram of an embodiment of the training process of the large model training of the present application.

[0051] In some embodiments, the performance-related data includes the start time and end time of each iteration, and the current performance data includes the effective training duration; determining the current performance data according to the performance-related data includes: taking the difference between the end time and start time of each iteration as the effective training duration.

[0052] The current performance data also includes: model initialization duration, training duration, effective training duration, proportion of effective training duration, duration of loading breakpoint, duration of saving breakpoint.

[0053] Determining the current performance data according to the performance-related data further includes: Taking the difference between the start time of this training and the task startup time as the model initialization duration; Taking the difference between the training end time and the training start time as the training duration; Taking the difference between the completion time and start time of each iteration in this training as the duration of each iteration; accumulating the duration of each iteration in this training as the effective training duration; Take the ratio of the effective training duration to the training duration as the proportion of the effective training duration. Take the difference between the end time of loading the breakpoint and the start time of loading the breakpoint as the loading breakpoint duration. Take the difference between the completion time of saving the breakpoint and the start time of saving the breakpoint as the saving breakpoint duration.

[0054] Combining historical performance data and current performance data can obtain cumulative performance data. The cumulative performance data includes: total training duration, total effective training duration, proportion of total effective training duration, total saving breakpoint duration. Through the cumulative performance data, abnormal situations such as faults during the entire training task of large model training can be obtained.

[0055] The method for determining cumulative performance data is as follows: Add up the training durations of multiple trainings during the training process to obtain the total training duration; Add up the effective training durations of multiple trainings and subtract the rollback training duration to obtain the total effective training duration; Take the ratio of the total effective training duration to the total training duration as the proportion of the total effective training duration; Add up the multiple saving breakpoint durations to obtain the total saving breakpoint duration.

[0056] In some embodiments, the determining the current performance data according to the performance-related data further includes: determining whether the performance-related data meets the performance threshold. If it meets the performance threshold, determine the current performance data according to the performance-related data. If it does not meet the performance threshold, re-collect the performance-related data.

[0057] The performance threshold is a pre-set threshold used to mark whether the performance-related data is normal data or abnormal data.

[0058] Figure 3 The figure shows a partial schematic diagram of another embodiment of the training process of large model training of the present application.

[0059] In some embodiments, the fault information includes the fault discovery duration and the fault recovery duration; the determining the fault information according to the historical performance data and the current performance data in the previous training process includes: Determine the fault discovery duration according to the difference between the start time of the current training and the end time of the previous training; Determine the fault recovery duration according to the difference between the start time of the current training and the completion time of the last saved breakpoint in the previous training.

[0060] The fault information further includes: rollback training duration, fault discovery duration, fault recovery duration, number of faults, number of restarts.

[0061] The determining the fault information according to the historical performance data and the current performance data in the previous training process further includes: Accumulate the duration of each iteration after the completion time of the last saved breakpoint (i.e., the effective training duration) as the rollback training duration; Take the difference between the start time of this training and the end time of the last training as the fault detection duration; Take the difference between the start time of this training and the completion time of the last saved breakpoint in the last training as the fault recovery duration; Determine the number of faults and restarts according to the number of interruptions that occur during the entire training task.

[0062] In some embodiments, step 14 includes: elastically expanding the resource amount of the training task; adopting an opportunity retention mechanism to perform the training task; determining the historical performance data at the time of the last interruption; and determining the fault information according to the historical performance data and the current performance data.

[0063] Elastically expanding the resource amount of the training task includes dynamically changing the number of GPUs and switching the GPU type. For example, increasing the number of GPUs from 4 to 8; or changing the GPU type, switching from NVIDIA Tesla V100 to NVIDIA A100.

[0064] In a heterogeneous cluster, when there are idle devices, automatically expand, and collect training data on the expanded devices, and then determine the fault information.

[0065] In some embodiments, "determining the fault information" in step 14 includes: when the heterogeneous cluster is elastically expanded, determining the fault information on each device in the starting state in the heterogeneous cluster respectively.

[0066] Collect the training data of each device in the starting state in the heterogeneous cluster respectively, and further determine the fault information.

[0067] In some embodiments, the monitoring method 10 further includes: recording the current performance data in the training log and saving it to the database; recording the event information during the training process in the event information table and saving it to the database.

[0068] The event information can be directly collected from the training data or obtained according to the collected performance-related data. The event information includes: task start time, task start command, training start time, log output at the start of training, fault occurrence time, error log at the time of fault, training end time, log at the end of training, task completion time, log at the completion of training.

[0069] Recording the current performance data and event information is beneficial for subsequent overall analysis of the training task and improving the training effectiveness.

[0070] In some embodiments, the monitoring method 10 further includes: when an abnormality occurs in the large model training, sending an alarm prompt.

[0071] When it is detected that an abnormality occurs in the large model training, the alarm prompt can be sent in various ways. For example, on the user terminal, it can be sent by means of sound or illustration, or by means of email, instant messaging tool, etc., to inform the user of the abnormal situation. The user can check the fault situation in time, troubleshoot the problem, and restart the task.

[0072] In some embodiments, the monitoring method 10 further includes: sending fault information to the training task so that the training task dynamically allocates the large model training in the heterogeneous cluster according to the fault information.

[0073] After the electronic device of the present application collects the fault information, it will send it to the training task. The communication method between the electronic device and the management module of the training task can be message passing based on a network protocol. For example, using the TCP / IP protocol, the fault information is encapsulated and transmitted in a specific data format (such as JSON or XML). After the management module of the training task receives the fault information, it analyzes and evaluates it. According to the severity and impact range of the fault, the large model training task is reallocated in the heterogeneous cluster to ensure the continuity and efficiency of the training. The Gossip algorithm can be used to dynamically allocate computing tasks according to the situation of the devices in the heterogeneous cluster. For example, for devices with a low memory occupancy rate and low power, more computing tasks can be allocated to balance the training time of different devices and speed up the training progress.

[0074] Figure 4 The structural block diagram of an embodiment of the electronic device of the present application is shown.

[0075] As Figure 4 shown, the electronic device includes one or more processors 21 for implementing the monitoring method 10 of the large model training in the heterogeneous cluster as described above.

[0076] In some embodiments, the electronic device may include a computer-readable storage medium 22. The computer-readable storage medium 22 may store a program that can be called by the processor 21, and may include a non-volatile storage medium. In some embodiments, the electronic device may include a memory 23 and an interface 24. In some embodiments, the electronic device may also include other hardware according to actual applications.

[0077] The computer-readable storage medium 22 of the embodiment of the present application stores a program thereon. When the program is executed by the processor 21, it is used to implement the monitoring method 10 of the large model training in the heterogeneous cluster as described above.

[0078] The present application may take the form of a computer program product implemented on one or more computer-readable storage media 22 (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain program code. The computer-readable storage media 22 include both permanent and non-permanent, removable and non-removable media, and may implement the storage of information by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media 22 include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

Claims

1. A monitoring method for large model training of heterogeneous clusters, characterized in that: include: Obtain the current training data of the large model on each device in the heterogeneous cluster that is in the startup state; Determining the current performance data for characterizing the training process according to the current training data; If the current performance data is not within the normal range, determine the fault information based on the historical performance data in the last training process and the current performance data; Based on the fault information, determine whether an abnormality occurs in the large model training to monitor the training process of the large model.

2. The monitoring method for large model training of heterogeneous clusters according to claim 1 is characterized in that: The step of obtaining the current training data of the large model on each device in the heterogeneous cluster that is in the startup state includes: Receiving an operation mode and a scheduling strategy of the large model training, where the operation mode and the scheduling strategy are set according to the training task of the large model training and parameter information of the heterogeneous cluster; Initiate the large model training on the heterogeneous cluster according to the operation mode and the scheduling strategy; During the large model training process, collecting operation data; Simulate expansion of the heterogeneous cluster and collect operation data after expansion; The current training data is obtained according to the operating data and the expanded operating data.

3. The monitoring method for large model training of heterogeneous clusters according to claim 1 is characterized in that: The determining, based on the current training data, the current performance data used to characterize the training process includes: In the training data of the current time, performance-related data is collected; the performance-related data includes at least one of the task start time, task completion time, training start time, training end time, each round of iteration start time, each round of iteration end time, computing power value of each round of iteration, loading breakpoint start time, loading breakpoint completion time, saving breakpoint start time, saving breakpoint completion time, fault occurrence time, and fault recovery time; The current performance data is determined according to the performance-related data.

4. The method for monitoring large model training of heterogeneous clusters according to claim 3 is characterized in that: The performance-related data includes the start time of each iteration and the end time of each iteration, and the current performance data includes the effective training duration; Determining the current performance data based on the performance-related data includes: taking the difference between the end time of each round of iteration and the start time of each round of iteration as the effective training duration.

5. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: In the training data of the current time, the performance-related data collected includes: According to the preset sampling target and sampling interval, performance related data is collected in the current training data.

6. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: The performance-related data includes application layer performance-related data and micro-architecture layer performance-related data.

7. The method for monitoring large model training of heterogeneous clusters according to claim 3, characterized in that: The fault information includes the fault discovery time and the fault recovery time; The determining of fault information according to the historical performance data in the last training process and the current performance data includes: Determine the fault discovery time according to the difference between the start time of this training and the end time of the previous training; The fault recovery duration is determined according to the difference between the start time of this training and the completion time of the last save breakpoint in the previous training.

8. The method for monitoring large model training of heterogeneous clusters according to claim 1, characterized in that: The determining of the fault information includes: when the heterogeneous cluster is elastically expanded, respectively determining the fault information on each device in the startup state in the heterogeneous cluster; and / or, The monitoring method further includes: issuing an alarm when an abnormality occurs in the large model training; and / or, The monitoring method further includes: sending the fault information to a training task, so that the training task dynamically allocates the large model training in the heterogeneous cluster according to the fault information.

9. An electronic device, characterized in that: It comprises one or more processors for implementing the monitoring method for large model training of a heterogeneous cluster as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the monitoring method for large model training of a heterogeneous cluster as described in any one of claims 1 to 8 is implemented.

Images