Distributed fault-tolerant and stable guarantee method for large model training

Abstract

The application relates to the technical field of large model training strategies, in particular to a distributed fault tolerance and stability guarantee method for large model training, which comprises the following steps: acquiring model training related data in a large model training process; analyzing the abnormality degree of the model training state in a single training cycle; analyzing the stability change degree of the training state of the model in a continuous training process; and evaluating whether the model parameters of a single training cycle are saved by a checkpoint. The application aims to adjust the saving strategy of the checkpoint by analyzing the model training state, so as to improve the efficiency of breakpoint resuming training of the model.

Description

Technical Field

[0001] This application relates to the field of large model training strategy technology, specifically to a distributed fault-tolerant and stability guarantee method for large model training. Background Technology

[0002] In recent years, with the rapid growth in the parameter scale of large models, the demand for computing resources has increased. Training large models requires a large amount of computing resources and time. The memory requirements of large models mean that a single GPU is far from sufficient to handle the calculations of the entire model's parameters, necessitating the distribution of the model across multiple GPUs for distributed parallel training. In real-world scenarios of distributed training of large models, thousands of GPUs may need to work in parallel for weeks or even months to complete the training. In such scenarios, the frequency of hardware and software failures is very high, resulting in significant waste of computing resources. Therefore, how to achieve efficient fault tolerance in large model training, and how to perform fault checking and checkpoint saving under limited computing resources to ensure the efficiency and reliability of resuming training from breakpoints, has become an urgent problem to be solved.

[0003] Traditional checkpoint saving methods typically perform full checkpoint saving of the model at fixed intervals. This approach has significant shortcomings in large-scale data-parallel model training scenarios with thousands of calories per second. First, the model parameters are enormous, and the total data volume of the full model save can reach terabytes, causing severe read / write (I / O) congestion in centralized storage systems, blocking backpropagation and communication operations, and reducing GPU utilization. In addition, fixed-period checkpoint saving cannot capture the state changes during the training process, which may result in checkpoint saving not being triggered during stages with drastic gradient fluctuations or strong node anomalies. This leads to excessively large rollback spans when resuming training from a breakpoint, resulting in significant loss of model training progress. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a distributed fault-tolerance and stability guarantee method for large model training, thereby resolving the existing problems.

[0005] The distributed fault tolerance and stability guarantee method for large model training proposed in this application adopts the following technical solution:

[0006] One embodiment of this application provides a distributed fault tolerance and stability guarantee method for training large models, the method comprising the following steps:

[0007] Acquire model training-related data during the large model training process, including resource usage data and model training data for each node;

[0008] The first index is constructed by analyzing the abnormal dispersion of node resource usage data and model training data in different optimization steps within a single training cycle, as well as the degree of difference between different nodes, to characterize the degree of abnormality in the model training state in a single training cycle.

[0009] A second metric is constructed by analyzing the changes in node resource usage data and model training data across multiple consecutive training cycles. This metric is used to characterize the degree of stability change in the model's training state during continuous training.

[0010] A comprehensive index for a single training cycle is constructed by combining the first and second indicators, which is used to evaluate whether the model parameters for a single training cycle are saved at checkpoints.

[0011] Preferably, constructing the first indicator includes:

[0012] Detrending analysis is performed on the same type of data in the model training data of a single node in a single training cycle to obtain a data sequence with trend components eliminated;

[0013] The first parameter and the second parameter are calculated for all different types of data in the resource usage data of a single node in a single training cycle and in the detrended model training data, respectively. The first parameter represents the degree of abnormal dispersion of the overall distribution of the same type of data in a single node, and the second parameter represents the degree of difference of the same type of data between different nodes.

[0014] The mean of the first parameter of all nodes of different types of data in a single training cycle is used as the first parameter of the single training cycle.

[0015] The mean of the second parameter of all different types of data in a single training cycle is used as the second parameter of the single training cycle.

[0016] The first metric for a single training cycle is calculated based on the first and second parameters of that single training cycle.

[0017] Preferably, the method for calculating the first parameter includes:

[0018] Normalize the data of the same type for a single node in a single training cycle across all optimization steps;

[0019] The absolute value of the difference between the data value of each optimization step and the mean of the data values ​​of all optimization steps is calculated as the deviation value;

[0020] The mean of the deviation values ​​of all optimization steps is used as the first parameter for data of the same type in a single node within a single training cycle.

[0021] Preferably, the method for calculating the second parameter includes:

[0022] Calculate the cosine similarity between data sequences of the same type between different nodes in a single training cycle;

[0023] The difference distance between different nodes is determined based on cosine similarity, and the mean of the difference distances between all different nodes is used as the second parameter for the same type of data between different nodes in a single training cycle.

[0024] Preferably, the first indicator of a single training cycle is positively correlated with the first parameter and the second parameter of the single training cycle, respectively.

[0025] Preferably, constructing the second indicator includes:

[0026] Calculate the coefficient of variation of all optimization steps at all nodes in a single training cycle for the same type of data;

[0027] The mean of the absolute values ​​of the differences between the coefficient of variation of a single training period under the same type of data and the coefficients of variation of all training periods under the same type of data in the period window of the single training period is used as the third parameter of the single training period under the same type of data.

[0028] The mean of the third parameter of all different types of data in a single training cycle is used as the second metric for that single training cycle.

[0029] Preferably, the comprehensive index of a single training cycle is positively correlated with the first indicator and the second indicator, respectively.

[0030] Preferably, the evaluation method for whether the model parameters of a single training cycle are saved by checkpoint includes:

[0031] The checkpoint retention threshold is calculated using the three-standard-deviation rule for the comprehensive index of all training cycles within a single training cycle's cycle window.

[0032] When the composite index of a single training cycle exceeds the checkpoint saving threshold, the model parameters of that single training cycle are checked and saved.

[0033] Preferably, the period window of a single training cycle consists of a preset number of training cycles preceding the single training cycle.

[0034] Preferably, the resource usage data for each node includes GPU utilization, temperature, and communication latency; the model training data for each node includes parameter gradient norm and loss value.

[0035] This application has at least the following beneficial effects:

[0036] This application addresses the problem that traditional checkpoint saving strategies, which save checkpoints at fixed intervals, can lead to communication congestion and reduced GPU utilization in large-scale model training scenarios with thousands of calories per second. Conversely, saving checkpoints too frequently can result in significant losses during breakpoint training if the save frequency is too low, preventing timely saving before the system node training state becomes unstable. This application constructs a first index by analyzing the node resource usage data and model training data changes at different optimization steps within a single training cycle. This index characterizes the changes in the model training state within a single training cycle. A second index is constructed by analyzing the node resource usage data and model training data changes across multiple consecutive training cycles. This index characterizes the changes in the model's training state during continuous training. A comprehensive index is built based on the first and second indices to evaluate whether checkpoint saving is necessary for the current training cycle. By analyzing the model's adaptive checkpoint saving strategy based on its training state, the efficiency of breakpoint training is improved. Attached Figure Description

[0037] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart of the distributed fault tolerance and stability guarantee method for large model training provided in this application. Detailed Implementation

[0039] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the distributed fault-tolerant and stability-assured method for large model training proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0041] The following section, in conjunction with the accompanying drawings, details the specific scheme of the distributed fault tolerance and stability guarantee method for large model training provided in this application.

[0042] One embodiment of this application provides a distributed fault-tolerant and stability-guaranteeing method for training large models.

[0043] Specifically, the following distributed fault tolerance and stability guarantee methods for training large models are provided; please refer to [link / reference]. Figure 1 The method includes the following steps:

[0044] Step 1: Obtain model training-related data during the large model training process. The resource manager for the large model training cluster is used to collect relevant data from each node in real time. The resource manager can use exporters such as NVIDIA ADCGM or Node Exporter to read resource usage data and model training data from nodes during training; no specific restrictions are imposed. The relevant data includes resource usage data and model training data from each node.

[0045] The resource usage data of each node includes, but is not limited to, GPU utilization, temperature, data throughput, CPU utilization, memory, and network communication latency.

[0046] The model training data for each node includes, but is not limited to, parameter gradient norms and loss values.

[0047] Step 2: Construct an adaptive checkpoint saving strategy based on model training data.

[0048] In checkpoint saving strategies for large model training, traditional fixed-interval checkpoint saving strategies cannot adjust the frequency of checkpoint saving according to the real-time state of the current model training. This leads to problems such as information redundancy in checkpoint saving or severe training loss due to failures. If the checkpoint saving frequency is too high, saving model parameters for multiple consecutive and stable training cycles will cause GPU read / write congestion, repeatedly reading and writing a large amount of redundant parameter data, affecting the execution efficiency of the training process. If the checkpoint saving frequency is too low, when a failure occurs during training and training is interrupted, resuming training with checkpoints at greater intervals will result in the loss of a large amount of training state, and the cost of resuming training from a breakpoint is high. Traditional checkpoint saving strategies cannot balance the changes in training state and the occupation of GPU read / write resources, resulting in low overall training efficiency of the model.

[0049] To address the aforementioned issues, this application constructs a first index by analyzing the node resource usage data and model training data change characteristics of different optimization steps within a single training cycle. This index characterizes the degree of abnormality in the model training state within a single training cycle. A second index is constructed by analyzing the node resource usage data and model training data change characteristics across multiple consecutive training cycles. This second index characterizes the changes in the model's training state during continuous training. A comprehensive index is built based on the first and second indices to assess whether checkpoints have been saved for the current training cycle. By analyzing the adaptive checkpoint saving strategies of the model's training state, the efficiency of resuming training from breakpoints is improved.

[0050] Furthermore, in the data acquired by the resource manager during the training process of the large model, the first... In the training cycle, the first The node of the first The updated resource usage data and model training data after each optimization step are represented as follows: , , , , ,in This represents the total number of training cycles for the model. The total number of nodes. To determine the total number of optimization steps in a given training cycle. The calculation can be expressed as the ratio of the total number of training data to the batch size, rounded up.

[0051] (1) In different optimization steps of a single training cycle, the end of each optimization step indicates that the optimizer has completed a forward propagation-loss calculation-backpropagation-parameter update for the model.

[0052] In a single training cycle, a node needs to perform a series of calculations on the data between different optimization steps. Ideally, with a consistent batch size, node resource usage should remain within a relatively stable range. Regarding model training data, since model parameters are continuously updated as optimization steps progress, the model training data (such as gradient norm and loss value) should exhibit a gradual convergence between different optimization steps, showing a smooth, decreasing trend. In a large model training cluster, different nodes operate in parallel, model parameter updates across all nodes remain synchronized, and communication latency after data resource calculations maintains a certain consistency. Therefore, within the same training cycle, the resource usage data and model training data of different nodes maintain a certain similarity.

[0053] Based on the above description, this application constructs a first index by analyzing the abnormal dispersion of node resource usage data and model training data in different optimization steps within a single training cycle within the same node and the degree of difference between different nodes, which is used to characterize the abnormality of the model training state in a single training cycle.

[0054] Preferably, in this application, the method for constructing the first index includes: performing detrending analysis on the same type of data in the model training data of a single node in a single training cycle to obtain a data sequence with trend components eliminated; calculating a first parameter and a second parameter for all different types of data in the resource usage data of a single node in a single training cycle and the detrended model training data, wherein the first parameter characterizes the abnormal dispersion of the overall distribution of the same type of data in a single node, and the second parameter characterizes the degree of difference of the same type of data between different nodes; taking the mean of the first parameters of different types of data in all nodes in a single training cycle as the first parameter of the single training cycle; taking the mean of the second parameters of all different types of data in a single training cycle as the second parameter of the single training cycle; and calculating the first index of the single training cycle based on the first parameter and the second parameter of the single training cycle. The first index of the single training cycle is positively correlated with the first parameter and the second parameter of the single training cycle, respectively.

[0055] It is understandable that a positive correlation means that the dependent variable increases as the independent variable increases, and the dependent variable decreases as the independent variable decreases. This is determined by the actual application and is not subject to any special restrictions in this application.

[0056] In one preferred embodiment, the calculation method of the first parameter includes: normalizing the data of the same type for a single node in all optimization steps in a single training cycle; calculating the absolute value of the difference between the data value of each optimization step and the mean of the data of all optimization steps as the deviation value; and using the mean of the deviation values ​​of all optimization steps as the first parameter of the data of the same type for a single node in a single training cycle.

[0057] Specifically, in this embodiment, the model training data... ( For the first In the training cycle, the first Each node contains In the model training data sequence of each optimization step, according to different data types, detrending analysis is performed on the same type of model training data to obtain the data sequence without trend components, which is represented as follows: Detrending analysis can be performed using methods such as STL decomposition, without any specific restrictions. When using STL decomposition for detrending analysis, the period frequency parameter is the total number of optimization steps within a single training period. For model training data The purpose of detrending analysis is to eliminate the convergence and decline trend of the model training data caused by the iteration of different optimization steps, so as to facilitate statistical analysis of the model training data on the baseline.

[0058] Furthermore, regarding the first In the training cycle, the first Resource usage data of each node and detrended model training data Statistical analysis was performed on all different types of data. The specific steps for statistical analysis are as follows:

[0059] by Taking GPU utilization data as an example, for The GPU utilization of the optimization step is normalized to its maximum and minimum values, and then the calculation of the _th step is performed. The absolute value of the difference between the GPU utilization of each optimization step and the mean of all GPU utilization is denoted as the GPU utilization deviation. Then, the first parameter of GPU utilization can be expressed as: The mean of the GPU utilization deviation values. The normalization of the maximum and minimum values ​​is a well-known technique and will not be elaborated here. In other embodiments, the normalization method can also use Z-score standardization to normalize the data. The specific method can be set by the implementer according to the actual situation.

[0060] It should be noted that in data statistical analysis, calculating the difference between each data point and the mean of all data points in the data sequence can reflect the degree of abnormal dispersion of the data sequence. In this scheme, calculating the difference between the GPU utilization of each optimization step in a single node and the overall mean GPU utilization can reflect the degree of abnormal dispersion of the overall distribution of GPU utilization in the data sequence. That is, the first parameter is used to characterize the degree of abnormal dispersion of the overall distribution of the same type of data in a single node. A larger value indicates that the GPU utilization of a local optimization step is highly dispersed relative to the distribution of the entire data sequence, indicating the existence of local anomalies. Conversely, a smaller value indicates that the system runs more stably during training.

[0061] Similarly, calculate separately and The first parameters of the other different types of data and the first parameters of different types of data in different nodes. Further, let the first... The first parameter in each training cycle For the first The mean of the first parameter of different types of data for all nodes in each training cycle.

[0062] Furthermore, different nodes should maintain parameter synchronization during training. Therefore, the degree of difference in resource utilization and gradient synchronization between different nodes can reflect the consistency and communication efficiency of the entire system in the current training cycle. When the resource and model training data curves of each node are highly similar, it indicates that the training process is in a cooperative and stable state. Conversely, if some nodes deviate significantly, it may indicate that there is a network communication anomaly or a local computational anomaly in the current training process.

[0063] In one preferred embodiment, the calculation method of the second parameter includes: calculating the cosine similarity of data sequences of the same type between different nodes in a single training cycle; determining the difference distance between different nodes based on the cosine similarity; and using the mean of the difference distances between all different nodes as the second parameter of data of the same type between different nodes in a single training cycle.

[0064] Specifically, in this embodiment, for the first Statistical analysis was performed on resource usage data at different nodes during each training cycle and on different types of data in the model training data. The specific steps for data statistical analysis are as follows:

[0065] Taking GPU utilization data as an example, the first The second parameter of the GPU utilization data for each training cycle can be expressed as: ,in , They represent the first In the training cycle, the first , A sequence of GPU utilization data for each node, containing a total of [number] GPU utilization data sequences. GPU utilization in each optimization step For cosine similarity, Indicates the first , The difference distance between GPU utilization data sequences of each node is used, since the cosine similarity value ranges from [-1, 1]. The range of cosine similarity values ​​is guaranteed to be within the range of [0,1].

[0066] It should be noted that in data analysis, cosine similarity can represent the similarity between two sequences. In this approach, by calculating the similarity of GPU utilization sequences between different nodes and performing a negative mapping, the difference distance between the two sequences can be represented. A larger value indicates a greater data difference between the two sequences, and vice versa. The final result... This reflects the differences in GPU utilization among different nodes during the training period. A larger value indicates that some nodes may have abnormal deviations due to network communication anomalies or node computation anomalies. Conversely, a smaller value indicates that the model training process of each node is relatively stable during the training period and the degree of system anomaly is low.

[0067] Similarly, calculate the second parameter for the remaining data types. Further, let the... The second parameter for each training cycle For the first The mean of the second parameter of all different types of data in each training cycle.

[0068] Furthermore, based on the above analysis, in this embodiment, the first... The first indicator for each training cycle One way to calculate this can be expressed as:

[0069]

[0070] In the formula, 、 The preset weighting coefficients for the first and second parameters are respectively, satisfying... In this embodiment , Take values ​​of 0.5 and 0.5 respectively.

[0071] It should be noted that when the first If the first and second parameters are large in the first training cycle, it indicates that there are localized issues with abnormally discrete resource usage data and model training data distribution within a single node and between different nodes within this training cycle. This disrupts the expected stability during model training and the consistent stability between different nodes. The first indicator for each training cycle A large value indicates that abnormal changes have occurred during the model training process.

[0072] The specific calculation method for the first indicator is not subject to any special restrictions, as long as it can show the positive correlation between the first and second parameters and the first indicator. The implementer can set the specific method according to the actual situation.

[0073] (2) Furthermore, in distributed training of large models, the fluctuations within a single training cycle only reflect the transient changes of the model, while the evolution trend and stage stability of the model in multiple consecutive training cycles are also of great significance for evaluating the degree of abnormality in a single training cycle.

[0074] When the Resource usage data and model parameter data for each node in the training cycle relative to the first training cycle. If the data before the training epoch shows a smooth and stable change, and the change is within the normal convergence range, it indicates that the model training is in a relatively stable stage. Conversely, if the data before the training epoch shows a smooth and stable change, and the change is within the normal convergence range, it indicates that the model training is in a relatively stable stage. If significant distribution anomalies appear in the data prior to a training cycle, it indicates that there is abnormal network communication congestion or abnormal computing power of local nodes, which disrupts the stability of the training state.

[0075] Based on the above analysis, this application constructs a second index by analyzing the changing characteristics of node resource usage data and model training data between multiple consecutive training cycles, which is used to characterize the degree of stability change of the model's training state during continuous training.

[0076] Preferably, in this application, the method for constructing the second index includes: calculating the coefficient of variation of all optimization steps under all nodes in a single training cycle under the same type of data; calculating the mean of the absolute values ​​of the differences between the coefficient of variation of a single training cycle under the same type of data and the coefficients of variation of all training cycles under the same type of data in the cycle window of the single training cycle, as the third parameter of the single training cycle under the same type of data; and using the mean of the third parameter of all different types of data in the single training cycle as the second index of the single training cycle.

[0077] Specifically, in this embodiment, the first step is to... One training cycle before The training cycle consists of the first... Period window of training cycles ,in For the preset quantity, The size can be set by the implementer according to the implementation scenario, without special restrictions. In this embodiment, the size is 7. Specifically, when At this stage, which is the initial phase of model training, no statistical analysis is performed on the construction period window of the training cycle, and its second indicator is 0.

[0078] Furthermore, regarding the first Statistical analysis was performed on the resource usage data of all nodes and different types of data in the model training data during each training cycle. The specific steps of the data statistical analysis are as follows:

[0079] Taking GPU utilization data as an example, calculate the first... Coefficient of variation of GPU utilization for all optimization steps at all nodes in a training cycle Specifically, when calculating the coefficient of variation, a very small positive number is added to the denominator. To prevent calculation errors caused by a denominator of 0, Let's take 0.01. In data analysis, the coefficient of variation represents the dispersion of data; the larger the coefficient of variation, the higher the dispersion of the data distribution.

[0080] Then calculate Coefficient of variation of GPU utilization for all optimization steps at all nodes in each training cycle .

[0081] Further, calculate the first Coefficient of variation of GPU utilization per training cycle With the Coefficient of variation of GPU utilization for all training cycles within a cycle window of a training cycle. The mean of the absolute values ​​of the differences between them is used as the first... The third parameter is the GPU utilization rate for each training cycle.

[0082] Similarly, calculate the first one respectively. The third parameter of the other different types of data in the training cycle will be the third parameter of the training cycle. The mean of the third parameter of all different types of data in each training epoch is used as the first parameter. The second indicator for each training cycle .

[0083] By analyzing the first The second metric is constructed by comparing the changes in resource usage data and model training data over a given training period with those data over several consecutive training periods. This metric reflects the changes in resource usage data and model training data over the previous training periods. The degree of abnormal changes in the stage-by-stage training evolution of each training cycle.

[0084] It should be noted that the first The third parameter of GPU utilization in the training cycle represents the... Each training cycle relative to the cycle window The GPU utilization distribution in other training cycles is discrete. When the system is in the... During stable operation in each training cycle, at this time If it is relatively small, then When smaller, it means in Several training cycles to the first If the system operates stably during each training cycle, then... A smaller value indicates that the system is in a stable operating state without any abnormal changes. In this case, checkpoint saving is not performed to avoid redundant model parameter saving; conversely, if... When both are relatively large, it indicates that in The system's operating state was unstable during several training cycles. A larger value indicates that the system's operating state has transitioned from an unstable to a stable state. In this case, a checkpoint will be saved to promptly save the model parameter file. Similarly, when... The calculation principle remains the same when the value is large: if the system's operating state changes during model training, checkpoints are saved in a timely manner.

[0085] (3) Based on the above analysis, this application constructs a comprehensive index by combining the first and second indicators to evaluate whether the model parameters of a single training cycle are saved at checkpoints. The comprehensive index of a single training cycle is positively correlated with both the first and second indicators.

[0086] Specifically, in this embodiment, the first The comprehensive index for each training cycle can be expressed as: .

[0087] Furthermore, according to The comprehensive index of each training cycle and The checkpoint retention threshold was calculated using the three-standard-deviation rule. ,like Exceeding the threshold Then for the first The model parameter data for each training cycle is saved as checkpoints.

[0088] It should be noted that when the first The abnormal changes of each node in each training cycle and the correlation differences between different nodes are significant, and the first... When the resource usage data and model training data of a training cycle differ significantly from those of the preceding training cycles, a checkpoint for that cycle is saved. This ensures that updated model parameter files are saved promptly when the system begins to become unstable, avoiding significant losses from rolling back training after a system error interruption.

[0089] Step 3: Construct a distributed fault-tolerance and stability guarantee method for large model training.

[0090] Based on the adaptive checkpoint saving method in step two, it is determined whether to save the checkpoint during the current training cycle in the training process of the large model.

[0091] When a system detection node malfunctions and causes model training to be interrupted, the training task can be quickly restarted on backup resources using saved checkpoints, achieving breakpoint resumption of training. This minimizes the impact of node failures on the training task and improves the efficiency of breakpoint resumption of training after a failure. The breakpoint resumption of training for large models is an existing technology, and the specific process will not be described in detail.

[0092] The above technical features constitute the preferred embodiment of this application, which has strong adaptability and the best implementation effect. Unnecessary technical features can be added or removed according to actual needs to meet the needs of different situations.

Claims

1. A distributed fault-tolerance and stability guarantee method for large model training, characterized in that, The method includes the following steps: Acquire model training-related data during the large model training process, including resource usage data and model training data for each node; The first index is constructed by analyzing the abnormal dispersion of node resource usage data and model training data in different optimization steps within a single training cycle, as well as the degree of difference between different nodes, to characterize the degree of abnormality in the model training state in a single training cycle. A second metric is constructed by analyzing the changes in node resource usage data and model training data across multiple consecutive training cycles. This metric is used to characterize the degree of stability change in the model's training state during continuous training. A comprehensive index for a single training cycle is constructed by combining the first and second indicators, which is used to evaluate whether the model parameters for a single training cycle are saved at checkpoints. The construction of the first indicator includes: Detrending analysis is performed on the same type of data in the model training data of a single node in a single training cycle to obtain a data sequence with trend components eliminated; The first parameter and the second parameter are calculated for all different types of data in the resource usage data of a single node in a single training cycle and in the detrended model training data, respectively. The first parameter represents the degree of abnormal dispersion of the overall distribution of the same type of data in a single node, and the second parameter represents the degree of difference of the same type of data between different nodes. The mean of the first parameter of all nodes of different types of data in a single training cycle is used as the first parameter of the single training cycle. The mean of the second parameter of all different types of data in a single training cycle is used as the second parameter of the single training cycle. The first metric for a single training cycle is calculated based on the first and second parameters of that single training cycle.

2. The distributed fault tolerance and stability guarantee method for large model training as described in claim 1, characterized in that, The calculation method for the first parameter includes: Normalize the data of the same type for a single node in a single training cycle across all optimization steps; The absolute value of the difference between the data value of each optimization step and the mean of the data values ​​of all optimization steps is calculated as the deviation value; The mean of the deviation values ​​of all optimization steps is used as the first parameter for data of the same type in a single node within a single training cycle.

3. The distributed fault tolerance and stability guarantee method for large model training as described in claim 1, characterized in that, The calculation method for the second parameter includes: Calculate the cosine similarity between data sequences of the same type between different nodes in a single training cycle; The difference distance between different nodes is determined based on cosine similarity, and the mean of the difference distances between all different nodes is used as the second parameter for the same type of data between different nodes in a single training cycle.

4. The distributed fault tolerance and stability guarantee method for large model training as described in claim 1, characterized in that, The first indicator of a single training cycle is positively correlated with the first parameter and the second parameter of the single training cycle, respectively.

5. The distributed fault-tolerance and stability guarantee method for large model training as described in claim 1, characterized in that, The construction of the second metric includes: Calculate the coefficient of variation of all optimization steps at all nodes in a single training cycle for the same type of data; The mean of the absolute values ​​of the differences between the coefficient of variation of a single training period under the same type of data and the coefficients of variation of all training periods under the same type of data in the period window of the single training period is used as the third parameter of the single training period under the same type of data. The mean of the third parameter of all different types of data in a single training cycle is used as the second metric for that single training cycle.

6. The distributed fault tolerance and stability guarantee method for large model training as described in claim 1, characterized in that, The comprehensive index of a single training cycle is positively correlated with the first indicator and the second indicator, respectively.

7. The distributed fault-tolerance and stability guarantee method for large model training as described in claim 6, characterized in that, The evaluation methods for whether the model parameters for a single training cycle are checked and saved include: The checkpoint retention threshold is calculated using the three-standard-deviation rule for the comprehensive index of all training cycles within a single training cycle's cycle window. When the composite index of a single training cycle exceeds the checkpoint saving threshold, the model parameters of that single training cycle are checked and saved.

8. The distributed fault-tolerance and stability guarantee method for large model training as described in claim 5 or 7, characterized in that, The period window of a single training cycle consists of a preset number of training cycles preceding the single training cycle.

9. The distributed fault-tolerance and stability guarantee method for large model training as described in any one of claims 1-7, characterized in that, Resource usage data for each node includes GPU utilization, temperature, and communication latency; model training data for each node includes parameter gradient norm and loss value.

Images