A fault handling method for large model distributed training based on dynamic checkpoint strategy

Abstract

The present invention belongs to the field of large-scale model distributed training, and particularly relates to a large-scale model distributed training fault handling method based on a dynamic checkpoint strategy. It includes: (1) a checkpoint distributed access strategy based on cluster topology and dynamic environmental awareness; (2) a large-scale model iteration time prediction method based on change trend awareness; and (3) a checkpoint frequency decision method based on model training iteration time and trend awareness. The present invention solves the speed and loss problems in the traditional checkpoint recovery process, and also optimizes resource utilization and reduces redundant overhead during training by dynamically adjusting the checkpoint update frequency. These improvements enable large-scale deep learning models to respond to faults more flexibly and efficiently when facing complex training environments, ensuring the continuity and stability of training, thereby providing new ideas and directions for the development of large-scale model training technology.

Description

Technical Field

[0001] The present invention belongs to the field of large-model distributed training, and in particular relates to a large-model distributed training fault handling method based on a dynamic checkpoint strategy. Background Art

[0002] Large language models (LLMs) have garnered widespread attention from both academia and industry. However, the number of parameters in LLMs continues to increase. While GPT-1 in 2018 had only 117 million parameters, GPT-3.5, released in 2023, boasts 175 billion parameters. This increase in model size significantly increases the cost of training a model. Because LLMs have a large number of parameters and require massive amounts of data for training, training them must be performed on a large-scale GPU cluster, requiring thousands of GPUs and months of training time. This process inevitably leads to frequent failures. For example, during OPT model training, failures averaged two per day, while Llama 3.1 experienced 466 failures over 54 days of training. Training failures have become a bottleneck affecting training efficiency.

[0003] In the context of existing large-scale distributed training technology, existing model training frameworks such as Pytorch and DeepSpeed ​​typically recover training progress from model checkpoints in large-scale GPU clusters to address failures. Specifically, checkpoint information, such as parameters and optimizers during deep learning model training, is saved according to specific rules in a location accessible to training nodes. After a failure, the training process is restarted and the checkpoint is loaded. However, for LLM training failures with large data volumes and long single iteration times, traditional checkpoint solutions suffer from slow recovery and high failure losses.

[0004] In this context, the current prior art solutions closest to the present invention are mainly reflected in the following aspects:

[0005] 1. Placement of checkpoints for large models

[0006] Most existing methods store training states externally. Check-N-Run uses model compression technology to reduce the time and storage overhead of saving terabyte-sized checkpoints to external storage. DeepFreeze studies the implementation of asynchronous checkpointing for large models in the context of external checkpoint storage. Chronicles, a 176-byte multilingual LLM, uses external storage for checkpoints during training. However, the large size of LLM checkpoints leads to high communication latency between external storage and the GPU training cluster, resulting in significant time overhead for resuming training in the event of training failures. A few methods use CPU memory on training nodes to store checkpoints. Unicorn stores a copy of checkpoints in both local memory and external storage on large model training nodes. However, local memory is highly unreliable and prone to copy loss, requiring training states to be restored from external storage. GEMINI stores checkpoints in distributed CPU memory. By grouping the CPUs of large model training nodes and storing them in a ring-shaped manner within the same group, it achieves multi-copy fault tolerance in distributed storage. However, the lack of analysis of the physical topology between training nodes leads to redundant copies and high resource overhead.

[0007] 2. Large model training time analysis method:

[0008] The effectiveness of methods for determining checkpoint frequency depends on the time it takes to iterate model parameter updates. Zeng et al. currently model iteration time by analyzing the computation and communication steps in training iterations and summarizing the overlap and interference between multiple working nodes. Gao et al. modeled model information, parameter size, and node communication bandwidth in the model topology to construct a graph neural network to predict the execution time and performance of deep learning models. Zancato et al., without performing any training, approximated the training loss and accuracy at any point in the training process by solving low-dimensional stochastic differential equations (SDEs) in function space. However, existing methods only consider performance and time predictions for a single training batch at the model level and do not support time series predictions and trend analysis of model iterations over a period of time in large-model distributed training scenarios. This results in poor timeliness in methods for determining checkpoint frequency.

[0009] 3. Update frequency of large model checkpoints

[0010] The update frequency determines how often checkpoints are performed. Most existing methods use a static update frequency. Methods such as ByteCheck, PCcheck, and DeepFreeze focus on optimizing the efficiency of checkpoint storage and recovery processes at the operating system level, supporting statically specified checkpoint frequencies during training. These methods typically adopt a relatively low frequency. GEMINI and Unicorn both provide fault tolerance by performing a checkpoint after each iteration. However, a low frequency can cause significant loss of training state in the event of a failure, while a high frequency results in a large number of redundant training state save operations. Very few methods consider dynamic checkpoint frequencies. CheckFreq considers the significant time overhead of storing a checkpoint to external storage, which can span multiple iterations. It dynamically adjusts the checkpoint frequency to approximately wait for the next checkpoint after each storage. However, this is not suitable for efficient local memory distributed scenarios and still does not address performance issues at high frequencies.

[0011] During the distributed training of existing large models, proper checkpointing is used to prevent operations and optimize the failure recovery process. However, random placement strategies result in unreliable checkpoint placement or high latency. Most existing methods store checkpoints in external storage, which results in significant latency in the checkpoint acquisition process and failure recovery process. A few methods randomly select multiple nodes locally and store a copy of the checkpoint in the memory of each node, reducing the latency of failure recovery. However, these methods treat different storage nodes equally, ignoring the differences in reliability and available resources between local distributed storage locations, resulting in unreliable placement schemes or high latency in the checkpoint transmission process.

[0012] In addition to optimizing the checkpoint storage process, the frequency of checkpoint updates is also a critical issue. Checkpoint update frequency decisions, which only consider average training performance, are difficult to adapt to dynamic environmental changes, leading to significant redundancy or failure losses. Most existing methods use a static checkpoint update frequency. However, a low frequency can lead to significant loss of training state due to failures, while a high frequency results in significant redundant operations, resulting in significant resource waste, both of which affect training efficiency. A few methods dynamically adjust the update frequency during training. These methods calculate and predict the model's single iteration time, combine it with the lifetime cost of a checkpoint update, and then use this time to select an acceptable number of iterations between checkpoint updates. However, in reality, due to the dynamic changes in computing, storage, and communication resources, the word iteration time of DL training tasks also changes dynamically as training progresses. This variability makes analysis strategies based solely on a single average value difficult to adapt to the updated iteration situation. For example, when network conditions deteriorate and training slows down, the same number of update intervals may result in greater failure losses.

[0013] Existing methods for evaluating training time only analyze the performance of a single iteration based on the average of historical iteration times or calculations of basic information such as the model's network architecture, distributed topology, and hyperparameters. These methods fail to accurately capture the patterns of iteration time changes in a dynamically changing environment. Errors in performance estimation exacerbate inappropriate checkpoint frequency settings, making it difficult to adapt to future conditions, leading to redundant operations or significant losses from failures. Summary of the Invention

[0014] To address the challenges of existing technologies, this paper proposes a large-scale distributed training fault handling method based on a dynamic checkpointing strategy. This paper designs a four-layer collaborative checkpoint placement architecture driven by heterogeneous priorities and constructs a cluster-level checkpoint location decision model agent based on deep reinforcement learning. Through a multi-dimensional reliability-aware mechanism, it integrates real-time resource load dynamics, failure mode characteristics, and distributed storage reliability information to ensure efficient and reliable checkpoint transmission during fault recovery. A multi-step deep learning training time prediction method based on joint spatiotemporal modeling is proposed. Using graph neural networks and long short-term memory networks, it captures the topological spatial characteristics and temporal dynamics of distributed training, respectively, to predict future training times for multiple steps. The future load variation trend and magnitude are then extracted as key indicators to drive the adaptive optimization of the checkpoint update frequency. A dual-frequency checkpoint update method for a multi-layer placement architecture is proposed. By establishing a continuity representation model for failure loss, the method cost function at different update frequencies is quantified. This method, combining the identification of future training time variation trends and the quantification of magnitude, enables online dynamic adjustment of the checkpoint update frequency. While ensuring fault tolerance, it reduces checkpoint operation overhead and improves training efficiency.

[0015] Technical Solution

[0016] A method for handling large model distributed training faults based on a dynamic checkpoint strategy, including:

[0017] (1) Checkpoint distributed access strategy based on dynamic perception of cluster topology and environment: Design a four-layer access topology checkpoint distributed access method, dynamically perceive the topology structure and resource status of the GPU cluster, analyze the access locations of multiple copies of checkpoints during training, and determine the optimal access strategy.

[0018] (2) Trend-aware large-scale model iteration time prediction method: Implement large-scale model training iteration time series prediction and analyze future iteration time trends through historical data. Combined with trend analysis, the timeliness of checkpoint update frequency is improved to cope with dynamic changes in the training environment.

[0019] (3) Model training iteration time and trend-aware checkpoint frequency decision method: Based on loss analysis, the checkpoint update frequency is dynamically adjusted to ensure that the overhead of checkpoint operations is reduced while minimizing failure losses. By calibrating the tolerable loss and adjusting the frequency, the checkpoint update strategy is optimized to improve the efficiency and reliability of the training process.

[0020] Beneficial effects

[0021] In existing large-model distributed training technologies, traditional checkpoint recovery solutions face slow recovery speeds and significant failure losses, particularly when processing large amounts of data and long iteration times. To address this, the present invention proposes a four-layer, heterogeneous priority-driven collaborative checkpoint placement architecture. This architecture, powered by a deep reinforcement learning model agent, dynamically determines checkpoint placement, taking into account multiple factors such as resource load, failure modes, and storage reliability. This approach not only improves the efficiency and reliability of checkpoint transmission but also effectively reduces training time lost during the recovery process.

[0022] Existing training time prediction methods primarily focus on performance analysis of single training batches and are unable to adapt to the dynamic changes in large-scale distributed training scenarios. To address this, this paper proposes a multi-step deep learning training time prediction method based on joint spatiotemporal modeling, leveraging graph neural networks and long-short-term memory networks to capture the topological spatial characteristics and temporal dynamics of the training process. This innovation enables the early identification of future load trends, driving adaptive optimization of the checkpoint update frequency, ensuring efficient training while reducing the overhead associated with frequent checkpointing.

[0023] Most existing methods use a static checkpoint update frequency, which may result in the loss of a large amount of training state when a fault occurs, and an excessively high frequency will increase operational redundancy. The dual-frequency checkpoint update method of the present invention establishes a continuity characterization model, quantifies the cost function under different update frequencies, and combines it with the identification of future training time change trends to achieve online dynamic adjustment of the checkpoint update frequency. This method not only enhances fault tolerance, but also significantly improves training efficiency and solves the problem of inefficiency caused by improper frequency selection in the prior art. Through these innovations, the present invention effectively improves the reliability and efficiency of large-scale model training and has important practical application value.

[0024] In summary, the present invention provides a more efficient and reliable solution to the shortcomings of existing large-model distributed training by introducing a four-layer collaborative checkpoint placement architecture driven by heterogeneous priorities, a multi-step training time prediction method based on joint spatiotemporal modeling, and a checkpoint dual-frequency update strategy. These innovations not only solve the speed and loss issues in the traditional checkpoint recovery process, but also optimize resource utilization and reduce redundant overhead during training by dynamically adjusting the checkpoint update frequency. These improvements enable large-scale deep learning models to respond to failures more flexibly and efficiently when faced with complex training environments, ensuring the continuity and stability of training, thereby providing new ideas and directions for the development of large-model training technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the algorithm flow of the large-model distributed training fault handling method based on the dynamic checkpoint strategy of the present invention;

[0026] Figure 2 A schematic diagram of the hierarchical storage architecture proposed by the present invention;

[0027] Figure 3 The neural network architecture design of the iterative time series prediction method proposed in this invention;

[0028] Figure 4 This is an algorithm flow chart of the iterative time change trend analysis method proposed by the present invention;

[0029] Figure 5 A schematic diagram of the fault loss modeling method proposed in the present invention;

[0030] Figure 6 This is an algorithm flow chart of the dynamic checkpoint frequency decision method proposed by the present invention;

[0031] Figure 7 A comparison chart of the total model training time of the present invention and other existing methods;

[0032] Figure 8 A comparison chart of the average loss and average recovery time of a single failure between the present invention and other existing methods;

[0033] Figure 9 A comparison chart of the additional overhead of checkpoint operations between the present invention and other existing methods. DETAILED DESCRIPTION

[0034] The technical solution provided by this application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of this application will become more apparent with reference to the following description.

[0035] A large model distributed training fault handling method based on dynamic checkpoint strategy, such as Figure 1 ,include:

[0036] (1) Checkpoint distributed access strategy based on dynamic perception of cluster topology and environment: Design a four-layer access topology checkpoint distributed access method, dynamically perceive the topology structure and resource status of the GPU cluster, analyze the access locations of multiple copies of checkpoints during training, and determine the optimal access strategy.

[0037] (2) Trend-aware large-scale model iteration time prediction method: Implement large-scale model training iteration time series prediction and analyze future iteration time trends through historical data. Combined with trend analysis, the timeliness of checkpoint update frequency is improved to cope with dynamic changes in the training environment.

[0038] (3) Model training iteration time and trend-aware checkpoint frequency decision method: Based on loss analysis, the checkpoint update frequency is dynamically adjusted to ensure that the overhead of checkpoint operations is reduced while minimizing failure losses. By calibrating the tolerable loss and adjusting the frequency, the checkpoint update strategy is optimized to improve the efficiency and reliability of the training process.

[0039] Furthermore, the cluster topology and environment dynamic perception checkpoint distributed access strategy is based on a four-layer access topology checkpoint distributed access method. Through dynamic perception of cluster topology and resources, a reinforcement learning decision model based on the Deep Q Network (DQN) is constructed to make decisions on the access locations of multiple replicas. This includes:

[0040] (1) Cluster topology information collection: Collect the topology information of the four layers of storage media in the GPU cluster, including local memory, physical machine memory, memory of other machines in the cluster, and external storage. Figure 2 Schematic diagram of the hierarchical storage architecture of the present invention.

[0041] (2) Dynamic environmental perception: Dynamically perceive the status of cluster resources and analyze the access conditions in the current environment.

[0042] (3) Access location decision: Based on the collected topology information and resource status, the access location of multiple copies is determined to achieve rapid fault recovery.

[0043] Furthermore, the large model iteration time prediction method of the change trend perception is as follows Figure 3 、 4 , by predicting the future iteration time series and analyzing the changing trend, the timeliness of the checkpoint frequency update method is improved, including:

[0044] (1) Historical data analysis: Collect and analyze historical training iteration time series data to identify the change patterns of the time series.

[0045] (2) Trend prediction: Use the GNN-LSTM network to predict future iteration time and output future change trends.

[0046] (3) Classification and response: Classify the predicted change trends (such as fluctuations and regularities), promptly identify the instability of the training environment, and provide support for checkpoint strategies.

[0047] Furthermore, the model training iteration time and trend-aware checkpoint frequency decision method is as follows Figure 5 、 6 The dynamic hierarchical decision method for checkpoint update frequency based on loss analysis reduces the overhead of checkpoint operations while minimizing failure losses. This includes:

[0048] (1) Loss tolerance setting: Determine the tolerable training loss L base , and combined with the coefficient of variation , calculate the future loss tolerance L1.

[0049] (2) Frequency adjustment strategy: On the basis of ensuring that the loss does not exceed L1, gradually increase the checkpoint acquisition interval Δ to find the maximum interval that meets the conditions.

[0050] (3) Overhead optimization: Combined with the overhead of the access strategy, the checkpoint update frequency is dynamically adjusted to reduce unnecessary checkpoint operations and ensure the efficiency of the training process.

[0051] Example 1

[0052] This embodiment provides a large model distributed training fault handling method based on dynamic checkpoint strategy, such as Figure 1 As shown, including:

[0053] Based on the four-layer access topology checkpoint distributed access method, the access location of multiple copies is decided through dynamic perception of cluster topology and resources.

[0054] The large model training iterative time series prediction and change trend analysis method improves the timeliness of the checkpoint frequency update method by predicting the future iterative time series and analyzing the change trend.

[0055] A dynamic hierarchical decision method for checkpoint update frequency based on loss analysis reduces the overhead of checkpoint operations while minimizing failure losses.

[0056] The following will describe in detail the fault handling method provided by this embodiment with reference to the diagram. Figure 1 The following is a flow chart of the algorithm for handling large-scale model distributed training faults based on a dynamic checkpoint strategy, which includes the following steps:

[0057] S1, based on a four-layer access topology checkpoint distributed access method, makes decisions on the access locations of multiple copies through dynamic perception of cluster topology and environmental resources.

[0058] In this embodiment, for the distributed access method, by perceiving the information of the four-level storage media topology and environmental resource information in the GPU cluster, the access locations of multiple copies of the checkpoints during the training process are analyzed, and the access locations of each copy are output.

[0059] S11, the infrastructure used to store checkpoints such as Figure 2 shown.

[0060] The four-layer access topology of the checkpoint is as follows: the storage media are local memory, physical machine memory, memory of other machines in the cluster, and external storage in order of transmission speed between the training node and the storage medium.

[0061] The first level is local memory. For training processes, data in local CPU memory is the fastest to access. Therefore, storing a checkpoint in local memory allows for direct loading of the checkpoint in the event of a training process failure, enabling the fastest recovery.

[0062] When a training virtual machine fails, the checkpoints in the local memory will be corrupted. Therefore, a memory hierarchy of other virtual machines in the same physical machine is introduced to solve the problem of virtual machine failure.

[0063] The second level consists of the memory of other virtual machines on the same physical machine as the training VM. The advantage of this level is that if a single VM fails or the checkpoint in local memory becomes corrupted, the checkpoint can be quickly transferred to the restarted VM via shared memory, the physical machine's internal network, and other methods. Regarding the number of replicas, if two VMs on the same physical machine crash simultaneously, the failure is likely caused by a failure of the entire machine. This makes storing two or more replicas at this level meaningless and incurs additional resource overhead. Therefore, a single VM on the same physical machine with the lowest memory utilization should be selected to store a single checkpoint copy.

[0064] When a physical machine fails, the copies of the first two levels will be lost. Therefore, the memory level of other virtual machines in the cluster is introduced to solve the problem of physical machine failure.

[0065] The third layer is the memory of other virtual machines in the cluster. This layer uses the data center's internal network to transfer checkpoint copies to training nodes when copies of the first two layers are lost. At the same time, due to the high overhead of external storage networks, the existence of available copies at this layer should be ensured as much as possible. By establishing a reinforcement learning model to perceive changes in environmental resources, the number and access locations of checkpoint copies at this layer in the cluster can be dynamically adjusted to ensure the effectiveness of this layer without excessively impacting the training process.

[0066] The fourth level is external storage. While using checkpoints stored in external storage, such as cloud storage or external databases, introduces significant latency and communication overhead, it ensures the availability of checkpoint copies. If all copies stored within the cluster are lost, checkpoints can be transferred to training nodes via the data center's external network.

[0067] S12, in order to solve the distributed storage problem of checkpoints in the memory of other virtual machines in the cluster in S11, a reinforcement learning decision model based on deep Q network (DQN) was constructed to optimize the storage location of multiple replicas.

[0068] The reinforcement learning decision model: The cluster includes N virtual machines, and the checkpoint of the training process on a single training node is the minimum operation unit targeted by the DQN model.

[0069] State space: Model the information related to checkpoints and virtual machine resources in the cluster separately to describe the state space that the decision model relies on. It is as follows:

[0070] For N virtual machine resource information in the cluster, its bandwidth utilization , memory utilization Number of failures As resource indicators, describe the The communication resource load, memory resource load, and number of failures of each virtual machine in the past time window are analyzed.

[0071] For checkpoint information, its replica size s , basic number of copies n As indicators, they describe the storage space occupied by its copies and the number of copies required. Each time the layer fails without the failure of the entire cluster, the value increases by 1 until the maximum threshold is reached.

[0072] Action space: consists of an N-dimensional bool vector Composition, of which Indicates in Virtual Machine VM i A copy of the current decision object (checkpoint) is stored in the memory of Not in Virtual Machine VMi A copy of the current decision object (checkpoint) is stored in memory.

[0073] When selecting an action, the following three conditions must be met: first, at most one copy must be stored on the same physical machine to avoid the failure of both copies due to a physical machine failure; second, the amount of available memory in the selected virtual machine must be less than the copy size to ensure the operability of the solution storage; third, the total number of copies must be , when no feasible solution is found, the size of the basic replica number n is reduced one by one.

[0074] When executing an action, the changes in the state space are described by storing the communication and memory resources occupied by the operation and calculating the occupied values ​​to support the calculation of the reward function.

[0075] Reward mechanism: The decision model scores actions based on a reward function R to select a distributed storage solution:

[0076] R

[0077]

[0078]

[0079]

[0080] in, is the coefficient and satisfies The reward function R consists of three parts: The ratio of historical failure information of virtual machines with checkpoints in the entire cluster. This ratio is used to describe the reliability of storage solutions based on historical failure information. The lower the ratio, the better. and It is used to describe the impact of the checkpoint storage operation on the performance of the training process itself. is the average memory utilization of each storage virtual machine, is the average network bandwidth utilization of each storage virtual machine, which is used to measure the impact of storage checkpoint operations on the virtual machine's memory performance and communication capabilities. The lower the better.

[0081] In terms of network architecture, the model uses a feedforward neural network to evaluate the Q value of the state, and the dimension of the input parameter is ,in, For the sake of N The network consists of two hidden layers, each using a ReLU activation function. The output layer outputs the Q-value, which is the expected future reward after executing an action, given the current state.

[0082] S2, a method for predicting and analyzing the iterative time series of large model training, predicts and analyzes the changing trends of future iterative time series, and provides the results to S3 to improve the timeliness of the checkpoint frequency update method.

[0083] In this embodiment, the iteration time analysis method consists of two parts: a time series prediction module and a change trend analysis module. The time series module uses a GNN-LSTM network to predict future iteration time series based on historical iteration time series and the distributed architecture information of model training. The change trend analysis module outputs a prediction of future change trends based on the iteration time series over a period of time.

[0084] S21, Time Series Prediction: To predict the future iterative time series of distributed training of large models, feature modeling is performed based on the training topology and historical iterative time series, and a network model prediction result is constructed. The specific design of the neural network prediction model is as follows: Figure 3 shown.

[0085] For feature modeling, feature modeling is divided into two parts, including training topology and historical iteration time series.

[0086] Modeling of LLM training topology: Nodes in the topology graph Corresponding to the worker in the large model distributed training, nodes , the node characteristics are described using the node's computational overhead and the node's resource load, that is, the number of floating-point operations performed by the node in one iteration and the CPU utilization of the node , memory utilization and GPU utilization The edges in the topology graph represent the data transfer relationship between workers. The edge features include the parallel training relationship between nodes (including data parallelism, pipeline parallelism, and tensor parallelism), the size of the tensor transmitted between nodes, and the communication bandwidth between nodes.

[0087] For modeling of historical iteration time series: intercept the specified number of iterations W as the window, select W number of recently executed training iteration times to form the historical iteration time series.

[0088] In terms of network architecture, the prediction model consists of a GCN network and an LSTM network. The GCN network uses the training topology as input. Its purpose is to analyze the data transmission relationships and worker characteristics between workers during the LLM distributed training process to improve the accuracy of the LSTM model's prediction results. The GCN model results are combined with the historical event sequence as input to the LSTM model. A rolling forecast is used to obtain the prediction results for the future iterative time series of a specified window size. The GCN network consists of three graph convolutional layers and one linear layer, while the LSTM network consists of one unidirectional LSTM layer and one linear layer.

[0089] The prediction results obtained by the time series prediction module are provided to S22 for trend analysis and to S3 for analysis and decision-making on the checkpoint update frequency.

[0090] S22, Trend Analysis: For an iterative time series with a given window size of M Analyze the iterative time change trend. This sequence is the output result of the time series prediction module. The specific analysis method is as follows: Figure 4 shown.

[0091] First, based on the stability of the sequence, we categorize the change trends into two main categories: fluctuations and regularities. Fluctuating trends indicate strong unpredictability and instability within the time window, indicating that the training environment has experienced or is about to fail, necessitating a checkpoint acquisition strategy to address this situation. Regularity trends indicate that the model training iteration time is relatively stable over time, and trends over a long period of time, such as increases, decreases, or no change, are relatively reliable. Analyzing the degree of change in regularity provides quantitative data support for improving time effectiveness.

[0092] The coefficient of variation (CV) refers to the ratio of the standard deviation to the mean of a set of data. It is usually used in mathematics to reflect the degree of dispersion of a set of data and can measure the stability of the data. Based on the analysis of the coefficient of variation, the sequence T is divided into fluctuations and regularities. Take a threshold C and set the coefficient of variation of T to be ,when When , the change trend of T is considered to be fluctuation, and the fluctuation should be directly returned as the analysis result. Otherwise, it is a regularity. As an example, experiments show that when the threshold C is set to 0.2, it can effectively distinguish different time series change patterns and determine a better checkpoint update frequency.

[0093] When the trend of the sequence T changes regularly, it is necessary to measure the direction and degree of change of T to quantitatively describe the change of training iteration time in the future. The analysis is performed by linear fitting. First, the slope k and intercept b of the linear relationship between the iteration batch number and the iteration time are calculated based on the least squares method; then the future iteration batch number is calculated. The ratio of the iteration time of the first batch to the iteration time of the first batch ( ), called the coefficient of variation ,Right now , which is used to measure the degree of change in iteration time within the future window. The closer the value is to , the analysis results of checkpoint acquisition frequency based on the coefficient of variation will be more effective in the distant future, but will also sacrifice the reliability of recent checkpoints.

[0094] S3, a dynamic hierarchical decision method for checkpoint update frequency based on loss analysis, reduces the overhead of checkpoint operations while minimizing failure losses.

[0095] In this embodiment, the checkpoint frequency decision method first calibrates the tolerable failure loss based on the prediction of future iteration time series and the analysis of change trends. Then, based on the calibration results and the access policy overhead, a frequency decision is made and the checkpoint acquisition frequency is output.

[0096] S31, based on the hierarchical storage architecture, proposes a hierarchical checkpoint frequency decision method. For storage locations with higher bandwidth (such as local memory and other virtual machine memories in the same physical machine), a checkpoint is updated once for each training batch to ensure effective recovery when small-scale failures occur; for storage locations with lower bandwidth (other virtual machine memories in the data center and external storage), a dynamically adjusted, lower update frequency is adopted to reduce the time and communication overhead of the checkpoint process while ensuring that the loss of training progress is minimized.

[0097] S32. The basic idea behind the dynamic checkpoint frequency decision method based on loss analysis is to model and analyze the loss of training progress when a possible failure occurs, and then select the maximum checkpoint interval that can be taken while maintaining a tolerable training loss. The goal is to minimize the overhead of checkpoint operations while ensuring acceptable training loss.

[0098] Figure 5 This is a diagram showing the loss analysis at a specific update frequency and iteration time series. The further away from the checkpoint, the greater the loss when training loss occurs.

[0099] First, the loss of training progress when a failure occurs under a given checkpoint acquisition scheme is modeled. For the model iteration time series T, the checkpoint frequency is one checkpoint every p iterations, and a total of Checkpoint operations. Assume that a checkpoint operation costs , first calculate the start time of each checkpoint operation and end time :

[0100]

[0101]

[0102]

[0103] in, Indicates that any failure that occurs before the first checkpoint operation requires retraining from the initial model, and Indicates the time when training ends.

[0104] Next, define the training progress loss at any time t and the average training progress loss as follows:

[0105]

[0106]

[0107] in, It represents the average value of the failure loss caused when a failure may occur at any time within the predicted future time window under a given checkpoint operation interval p.

[0108] Figure 5 A diagram showing the calculation of the average loss over failures with a training batch size of 7 and a checkpoint frequency of every 2 batches is shown. The average loss is the quotient of the sum of the areas of all triangles divided by the total training time.

[0109] S33, based on the analysis of the average loss of training progress at different frequencies, the checkpoint frequency decision results can be improved to make a more reasonable evaluation. The specific algorithm is as follows Figure 6 shown.

[0110] The checkpoint acquisition interval Δ of the benchmark is defined as 1, and different decision strategies are adopted for different change trends.

[0111] If the change trend is fluctuating, it means that the state of the training cluster is very unstable and the training process may fail at any time. Therefore, the algorithm returns the fastest checkpoint frequency Δ=1.

[0112] If the change trend is regular, it means that the change of iteration time is stable and regular. At this time, the training loss can be tolerated. Based on the coefficient of variation Multiply them together so that the decision basis can match the training situation at a longer time, that is:

[0113]

[0114] Next, unless the fastest checkpoint frequency still cannot meet the requirements, the checkpoint acquisition interval Δ is gradually increased until a frequency that satisfies the training progress average loss is no more than The maximum interval is returned as the result of the algorithm, so the maximum checkpoint interval that makes the total overhead of the checkpoint operation as small as possible is found.

[0115] Example 2

[0116] This embodiment provides a verification test of a large-model distributed training fault handling method based on a dynamic checkpoint strategy (DesCheck, where Des represents Dynamic methods and Check represents checkpoint) of the present invention, and verifies and illustrates the technical effects adopted in this method.

[0117] Traditional technical methods include the GEMINI method, the Unicron method, and the CheckFreq method. Most traditional technical methods only consider static checkpoint strategies, which usually results in their lack of fault handling capabilities in large model training scenarios.

[0118] In order to verify that the DesCheck method of the present invention can obtain the fault handling capability adapted to the distributed training process of large models compared with the traditional method, in this embodiment, the traditional GEMINI method, Unicron method, CheckFreq method and the present method will be used to respectively perform checkpoint operations in the distributed training process of large models and handle faults in the training process.

[0119] Test environment: We used the open source project DeepSpeedTutorial and related public fault trajectory data as test samples. We used different methods to perform checkpoint operations during the training process. We ran the solution on a real distributed training cluster for 50 batches of training and obtained test results.

[0120] Using the different methods, we obtained data based on the experimental results and analyzed the total training time, resource cost, and fault overhead of each method.

[0121] The results of model training time are as follows Figure 7As shown in the figure, a shorter total time overhead indicates a faster model training process and higher performance. It can be seen that this method has the highest performance among the four methods, being 25.79%, 11.86%, and 44.41% faster than the GEMINI method, the Unicron method, and the CheckFreq method, respectively.

[0122] The DesCheck method of the present invention is compared with the traditional GEMINI method, Unicron method, and CheckFreq method in terms of resource indicators and failure costs per unit time during the training process. Figure 8 Indicates resource indicators, including memory and CPU resource usage per unit time; Figure 9 This represents the failure cost metric, including the average training progress loss when a failure occurs and the average recovery time after a failure. These two factors need to be considered comprehensively. For example, the CheckFreq method, while offering very low resource overhead, incurs a significant cost for failure recovery, resulting in the worst overall training efficiency. This method achieves the highest overall performance among the five methods, demonstrating superior fault handling capabilities.

[0123] The above description is only a description of the preferred embodiments of the present application and does not limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical content should be regarded as equivalent valid embodiments and fall within the scope of protection of the technical solution of the present application.

Claims

1. A large model distributed training fault handling method based on dynamic checkpoint strategy, characterized in that: include: (1) Checkpoint distributed access strategy based on dynamic perception of cluster topology and environment: Design a four-layer access topology checkpoint distributed access method, dynamically perceive the topology and resource status of the GPU cluster, analyze the access locations of multiple copies of checkpoints during training, and determine the optimal access strategy; (2) Trend-aware large-scale model iteration time prediction method: Implement large-scale model training iteration time series prediction, analyze future iteration time trends through historical data; combine trend analysis to improve the timeliness of checkpoint update frequency to cope with dynamic changes in the training environment; (3) Model training iteration time and trend-aware checkpoint frequency decision method: Based on loss analysis, the checkpoint update frequency is dynamically adjusted to ensure that the overhead of checkpoint operations is reduced while minimizing failure losses. By calibrating the tolerable loss and adjusting the frequency, the checkpoint update strategy is optimized to improve the efficiency and reliability of the training process. The four-layer access topology is specifically as follows: Storage media are divided into four levels according to the transmission speed between the training nodes: local memory, physical machine memory, memory of other machines in the cluster, and external storage; The first level is local memory; The second level is the memory of other virtual machines on the same physical machine as the training virtual machine. When a single virtual machine fails or the checkpoint in local memory is damaged, the checkpoint is transferred to the restarted failed virtual machine through shared memory and the physical machine's internal network. In terms of the number of replicas, the virtual machine with the lowest memory utilization on the same physical machine is selected to store a copy of the checkpoint. The third layer is the memory of other virtual machines in the cluster. When copies of the first two layers are lost, this layer uses the data center's internal network to transmit checkpoint copies to the training nodes. By establishing a reinforcement learning model to perceive changes in environmental resources, the number and access location of checkpoint copies at this layer in the cluster are dynamically adjusted to ensure the effectiveness of this layer without excessively affecting the training process. The fourth level is external storage; The large-scale model iteration time prediction method based on change trend awareness improves the timeliness of the checkpoint frequency update method by predicting future iteration time series and analyzing change trends, including: (1) Historical data analysis: Collect and analyze historical training iteration time series data to identify the change pattern of the time series; (2) Trend prediction: Use the GNN-LSTM network to predict future iteration time and output future change trends; (3) Classification and response: Classify the predicted change trends, promptly identify the instability of the training environment, and provide support for checkpoint strategies; The model training iteration time and trend-aware checkpoint frequency decision method and the dynamic hierarchical decision method for checkpoint update frequency based on loss analysis reduce the overhead of checkpoint operations while minimizing failure losses, including: (1) Loss tolerance setting: Determine the tolerable training loss L base , and combined with the coefficient of variation , calculate the future loss tolerance L1; (2) Frequency adjustment strategy: On the basis of ensuring that the loss does not exceed L1, gradually increase the checkpoint acquisition interval Δ to find the maximum interval that meets the conditions; (3) Overhead optimization: Combined with the overhead of the access strategy, the checkpoint update frequency is dynamically adjusted to reduce unnecessary checkpoint operations and ensure the efficiency of the training process.

2. A large model distributed training fault handling method based on a dynamic checkpoint strategy as described in claim 1, characterized in that: The checkpoint distributed access strategy based on cluster topology and dynamic environment perception is based on a four-layer access topology checkpoint distributed access method. Through dynamic perception of cluster topology and resources, a reinforcement learning decision model based on a deep Q network is constructed to make decisions on the access locations of multiple replicas, including: (1) Cluster topology information collection: Collect the topology information of the four layers of storage media in the GPU cluster, including local memory, physical machine memory, memory of other machines in the cluster, and external storage; (2) Dynamic environment perception: Dynamically perceive the status of cluster resources and analyze the access conditions in the current environment; (3) Access location decision: Based on the collected topology information and resource status, the access location of multiple copies is determined to achieve rapid fault recovery.

3. A large model distributed training fault handling method based on a dynamic checkpoint strategy as described in claim 2, characterized in that: The reinforcement learning decision model based on the deep Q network is used to optimize the storage location of multiple replicas. Specifically, the cluster includes N virtual machines, and the checkpoint of the training process on a single training node is the minimum operation unit targeted by the DQN model. State space: Models information about checkpoints and virtual machine resources in the cluster separately to describe the state space on which the decision model depends. as follows: For N virtual machine resource information in the cluster, its bandwidth utilization , memory utilization Number of failures As resource indicators, they describe the communication resource load, memory resource load, and the number of failures in the past time window respectively; For checkpoint information, the replica size s and the number of basic replicas n are used as indicators to describe the storage space occupied by the replica and the number of required replicas, respectively. Each time the layer fails without a complete cluster failure, the value increases by 1 until the maximum threshold is reached. Action space: consists of an N-dimensional bool vector Composition, of which Indicates in A copy of the current decision object, i.e., the checkpoint, is stored in the memory of each virtual machine; When selecting an action, the following three conditions must be met: first, at most one copy must be stored on the same physical machine to avoid the failure of both copies due to a physical machine failure; second, the amount of available memory in the selected virtual machine must be less than the copy size to ensure the operability of the solution storage; third, the total number of copies must be greater than the total number of copies. , when no feasible solution is found, the size of the number of basic copies n is reduced one by one; When executing an action, the changes in the state space are described by storing the communication and memory resources occupied by the operation and calculating the occupied values ​​to support the calculation of the reward function; Reward Mechanism: The decision model scores actions based on a reward function to select a distributed storage solution: R in, is the coefficient and satisfies ;The reward function consists of three parts: The percentage of historical failure information of virtual machines with checkpoints in the entire cluster, used to describe the reliability of storage solutions based on historical failure information. and It is used to describe the impact of the checkpoint storage operation on the performance of the training process itself. is the average memory utilization of each storage virtual machine, is the average network bandwidth utilization of each storage virtual machine, which is used to measure the impact of storage checkpoint operations on the virtual machine's memory performance and communication capabilities; In terms of network architecture, the model uses a feedforward neural network to evaluate the Q value of the state, and the dimension of the input parameter is ,in, is the historical failure, memory and communication resource status of N virtual machines respectively, and uses one dimension to describe the size of the checkpoint copy; the network has two hidden layers and uses the ReLU activation function; the output layer is used to output the Q value given the current state, that is, the expectation of future rewards after the action is executed.

4. A large model distributed training fault handling method based on a dynamic checkpoint strategy as claimed in claim 1, characterized in that: The time series prediction is to predict the future iterative time series of large model distributed training, perform feature modeling on the training topology and historical iterative time series, and construct the network model prediction results; For feature modeling, feature modeling is divided into two parts, including training topology and historical iteration time series; Modeling of LLM training topology: Nodes in the topology graph Corresponding to the worker in the large model distributed training, nodes , the node characteristics are described using the node's computational overhead and the node's resource load, that is, the number of floating-point operations performed by the node in one iteration and the CPU utilization of the node , memory utilization and GPU utilization The edges in the topology graph represent the data transfer relationship between workers. The edge features include the parallel training relationship between nodes, the size of the tensor transmitted between nodes, and the communication bandwidth between nodes. For modeling of historical iteration time series: intercept the specified number of iterations W as a window, select W number of recently executed training iteration times to form the historical iteration time series; In terms of network architecture, the network in the prediction model includes a GCN network and an LSTM network. The GCN network uses the training topology as input, with the aim of analyzing the data transmission relationship and worker characteristics between each worker during the LLM distributed training process to improve the accuracy of the LSTM model's prediction results. The results of the GCN model and the historical event sequence are combined as the input of the LSTM model, and the prediction results of the future iterative time series of the specified window size are obtained through rolling prediction. Among them, the GCN network consists of three graph convolutional layers and one linear layer, while the LSTM network consists of a unidirectional LSTM layer and a linear layer.

5. A large model distributed training fault handling method based on a dynamic checkpoint strategy as claimed in claim 1, characterized in that: The change trend analysis: for the iterative time series with a given window size of M Perform analysis on the iterative time change trend. This sequence is the output result of the time series prediction module. First, according to the stability of the sequence, the change trend is divided into two categories: fluctuation and regularity; Based on the analysis of the coefficient of variation CV, the sequence T is divided into fluctuations and regularities; a threshold C is taken, and the coefficient of variation of T is set to ,when When , the changing trend of T is considered as fluctuation, and the fluctuation should be directly returned as the analysis result; The opposite is a rule; When the trend of sequence T changes regularly, it is necessary to measure the direction and degree of change of T to quantitatively describe the change of training iteration time in the future. The analysis is performed by linear fitting. First, the slope k and intercept b of the linear relationship between the iteration batch number and the iteration time are calculated based on the least squares method. Then, the future iteration batch number is calculated. The ratio of the iteration time of the first batch to the iteration time of the first batch ( ), called the coefficient of variation ,Right now , used to measure the degree of change in iteration time within the future window; The closer the value is to , the analysis results of checkpoint acquisition frequency based on the coefficient of variation will be more effective in the distant future, but will also sacrifice the reliability of recent checkpoints.

6. A large model distributed training fault handling method based on a dynamic checkpoint strategy as claimed in claim 1, characterized in that: The model training iteration time and trend-aware checkpoint frequency decision method first calibrates the tolerable failure loss based on the prediction of future iteration time series and the change trend analysis results; Then, based on the calibration results and the cost of the access strategy, a frequency decision is made and the checkpoint acquisition frequency is output; specifically, the following steps are performed: S31 proposes a hierarchical checkpoint frequency decision method based on a hierarchical storage architecture. For storage locations with higher bandwidth, a checkpoint is updated once per training batch to ensure effective recovery from small-scale failures. For storage locations with lower bandwidth, a dynamically adjusted, lower update frequency is adopted. S32, by modeling and analyzing the amount of training progress loss when a possible failure occurs, selects the maximum checkpoint interval that can be taken while ensuring that the training loss is tolerable. The purpose is to reduce the overhead of checkpoint operations while ensuring that the training loss is acceptable. First, the loss of training progress when a failure occurs under a given checkpoint acquisition scheme is modeled. For the model iteration time series T, the checkpoint frequency is one checkpoint every p iterations, and a total of checkpoint operation; set a checkpoint operation to cost , first calculate the start time of each checkpoint operation and end time : in, Indicates that any failure that occurs before the first checkpoint operation requires retraining from the initial model, and Indicates the time when training ends; Next, define the training progress loss at any time t and the average training progress loss as follows: in, It represents the average value of the failure loss caused when a failure may occur at any time within the predicted future time window under a given checkpoint operation interval p; S33, based on the analysis of the average loss of training progress at different frequencies, improves the checkpoint frequency decision results for more reasonable evaluation; the specific algorithm is as follows: Define the baseline checkpoint acquisition interval Δ as 1, and adopt different decision strategies for different change trends; If the change trend is fluctuating, it means that the state of the training cluster is very unstable and the training process may fail at any time. Therefore, the algorithm returns to the fastest checkpoint frequency Δ=1; If the change trend is regular, it means that the change of iteration time is stable and regular. At this time, the training loss is tolerable. Based on the coefficient of variation Multiply them together to make the decision basis match the training situation at a more distant time, that is: Next, unless the fastest checkpoint frequency still cannot meet the requirements, the checkpoint acquisition interval Δ is gradually increased until a frequency that satisfies the training progress average loss is no more than The maximum interval is returned as the result of the algorithm, thus finding the maximum checkpoint interval that makes the total overhead of the checkpoint operation small.

Images