Fault-tolerant analysis system and method for whole process of large model training
By quantifying cross-modal consistency entropy and scene disturbance propagation intensity, a multimodal coupled state space is constructed, fault tolerance boundaries are determined, and real-time early warning is provided. This solves the stability and efficiency problems caused by environmental disturbances in the training of large multimodal models, and achieves stable training in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING LIANCHENG TECH DEV
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-16
AI Technical Summary
Existing multimodal large model training methods are easily affected by factors such as heterogeneous devices, asynchronous data transmission, and environmental noise in complex environments, leading to cross-modal feature alignment errors, inconsistent gradient updates, decreased model training efficiency, and degraded generalization ability. There is a lack of effective fault tolerance analysis methods.
By quantifying the cross-modal consistency entropy and scene disturbance propagation intensity during model training, a multimodal coupled state space is constructed, the training state trajectory is determined, and fault tolerance boundaries are set to achieve real-time fault tolerance early warning and avoid error cascading propagation.
It improves the training stability and robustness of multimodal large models in complex environments, provides proactive warnings to avoid model instability, reduces resource waste, and improves training efficiency.
Smart Images
Figure CN121860093B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large model training technology, and in particular to a fault-tolerant analysis system and method for the entire training process of large models. Background Technology
[0002] Multimodal learning is an important branch of machine learning, and large models based on multimodal data fusion are widely used in fields such as intelligent manufacturing, intelligent transportation, and smart cities. By jointly modeling multi-source information such as images, text, speech, and sensor signals, multimodal learning enables models to acquire more comprehensive environmental representation capabilities, thereby significantly improving recognition accuracy and decision-making ability in complex tasks. During the training of large models, different modalities are typically coupled through feature alignment mechanisms or joint optimization objectives, enabling the model to achieve cross-modal information fusion in a unified representation space.
[0003] However, most existing multimodal large model training methods are based on controlled training environments, assuming that each modality's data has stable acquisition conditions, synchronized time bases, and consistent data quality. In real-world applications, due to the complexity of equipment sources, the diversity of sensor types, and the dynamic changes in system deployment environments, the training process is inevitably affected by factors such as heterogeneous equipment, asynchronous data transmission, network fluctuations, and environmental noise. This leads to shifts in the statistical characteristics of data between different modalities, resulting in problems such as cross-modal feature alignment errors, inconsistent gradient updates, and abnormal model convergence paths.
[0004] Meanwhile, the training of large multimodal models relies on cross-modal alignment and gradient joint optimization mechanisms. When environmental perturbations occur during training, small errors generated in local modalities will rapidly propagate, accumulate, and be exponentially amplified between different modalities through cross-modal interaction links, forming an error cascade effect. Under continuous accumulation of perturbations, the model will enter an unrecoverable unstable state, leading to decreased training efficiency, degraded generalization ability, a significant waste of training resources, and a substantial extension of the training cycle.
[0005] Existing technologies typically address these issues passively through methods such as post-event repair, fixed threshold pruning, or static data cleaning. They lack quantitative analysis of the perturbation propagation intensity and model tolerance boundaries, and thus cannot meet the stable training requirements of multimodal large models in real-world complex scenarios. Summary of the Invention
[0006] In view of this, embodiments of the present invention provide a fault tolerance analysis system and method for the entire training process of large models. By quantifying the fault tolerance boundary of model training, the system effectively improves the stability and robustness of multimodal large models in complex environments.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] This invention provides a fault-tolerant analysis method for the entire training process of large models, including:
[0009] During the training of a large model, model training state data, training scene perturbation data, and forward propagation feature data are collected synchronously according to the training cycle.
[0010] By performing probability mapping on the feature representations of each modality during the forward propagation process using forward propagation feature data, and calculating cross-modal consistency entropy based on the probability distribution of different modalities, it is used to characterize the degree of information coupling and alignment consistency between multimodalities.
[0011] The cross-modal consistency change and cross-modal gradient direction change during the perturbation propagation process are analyzed using model training state data and training scene perturbation data. The scene perturbation propagation intensity is calculated by the cross-modal consistency entropy change rate and the gradient direction coupling change amount.
[0012] A multimodal coupled state space is constructed by using cross-modal consistency entropy and scene disturbance propagation intensity as state dimensions;
[0013] The training state trajectory is constructed in the multimodal coupled state space, and the model training fault tolerance boundary is determined according to the distribution of the training state trajectory under the perturbation conditions of the training scenario.
[0014] Model training fault tolerance early warning is generated based on the distance relationship between the training state trajectory of the current training phase and the model training fault tolerance boundary.
[0015] Preferably, the step of calculating the cross-modal consistency entropy based on the probability distributions of different modalities includes:
[0016] Forward propagation feature matrices for different modalities in the current training period are extracted from the forward propagation feature data. The row vectors in the forward propagation feature matrix of each modality correspond to the feature representation of a single training sample, and the column vectors correspond to the feature dimensions.
[0017] The average difference value of the cross-modal probability distribution corresponding to a single training sample is calculated by analyzing the forward propagation feature matrix.
[0018] The cross-modal consistency entropy is calculated by normalizing the average difference value of the cross-modal probability distribution of all training samples in the current training period and substituting the normalized proportion into the entropy calculation formula.
[0019] Preferably, the step of calculating the average difference value of the cross-modal probability distribution corresponding to a single training sample by calculating the forward propagation feature matrix includes:
[0020] By normalizing the forward propagation feature matrix, the normalized feature matrix corresponding to each mode is obtained. Then, the probability distribution matrix corresponding to each mode is obtained by performing probability mapping on the normalized feature matrix of each mode through the Softmax function. Each element in the probability distribution matrix represents the probability value of the corresponding training sample in the corresponding feature dimension, and the sum of the probability values of a single training sample in all feature dimensions is 1.
[0021] For each training sample, the probability distribution vector corresponding to each mode is extracted according to the probability distribution matrix. The KL divergence between the probability distribution vectors of any two modes is calculated to characterize the difference in probability distribution of different modes on the same training sample.
[0022] The average difference in the cross-modal probability distribution of a single training sample is obtained by averaging the KL divergence among all different modalities corresponding to the single training sample.
[0023] Preferably, the step of calculating the scene disturbance propagation intensity by coupling the cross-modal uniformity entropy change rate with the gradient direction change includes:
[0024] The cross-modal consistency entropy of the current training cycle and the previous adjacent training cycle is extracted, and the rate of change of cross-modal consistency entropy of the current training cycle is calculated by the first-order difference method.
[0025] Gradient data of each modality in the current training cycle is extracted from the model training state data, and gradient vectors corresponding to each modality are constructed. The cosine of the angle between the gradient vectors of any two modalities is calculated by vector dot product, and the cosine of the angle is used as the gradient direction similarity between the two modalities.
[0026] The average gradient direction similarity of the current training period is obtained by averaging the gradient direction similarity of all different modes, and the difference between the average gradient direction similarity of the current training period and the previous training period is used as the gradient direction coupling change.
[0027] The scene perturbation propagation intensity of the current training cycle is obtained by weighted summation of the cross-modal consistency entropy change rate and the gradient direction coupling change amount. The weights of the cross-modal consistency entropy change rate and the gradient direction coupling change amount are obtained by calculating the Pearson correlation coefficients between the training scene perturbation data and the cross-modal consistency entropy change rate and the gradient direction coupling change amount, respectively, and then normalizing the absolute value of the Pearson correlation coefficients.
[0028] Preferably, the construction of the multimodal coupled state space includes: concatenating the cross-modal consistency entropy value corresponding to each training cycle with the scene perturbation propagation intensity to form a joint state vector, wherein the cross-modal consistency entropy in the joint state vector is the first state dimension and the scene perturbation propagation intensity is the second state dimension; standardizing the joint state vector and constructing a two-dimensional coordinate system with the standardized cross-modal consistency entropy as the horizontal axis and the standardized scene perturbation propagation intensity as the vertical axis; mapping the joint state vector to the two-dimensional coordinate system to obtain the multimodal coupled state space.
[0029] Preferably, determining the model training fault tolerance boundary based on the training state trajectory distribution under training scenario perturbation conditions includes:
[0030] Using the training period as a time series, the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training period are combined to form state points, and the state points are connected in the multimodal coupled state space in chronological order to construct the training state trajectory.
[0031] Calculate the rate of change of trajectory direction between state points in adjacent training cycles to characterize the local evolution trend of the training state trajectory;
[0032] State points whose trajectory direction change rate is greater than a preset trajectory direction change threshold are used as critical state points to characterize the critical transition of the training state trajectory from stable convergence to divergent evolution, and all critical state points are counted to construct a critical state point set.
[0033] The critical state range is obtained by performing region fitting on the set of critical state points, and the critical state range is used as the fault tolerance boundary for model training.
[0034] Preferably, the step of providing early warning of model training fault tolerance based on the distance relationship between the training state trajectory corresponding to the current training stage and the model training fault tolerance boundary includes:
[0035] By calculating the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training cycle in the current training phase, the training state trajectory of the current training phase is constructed in the multimodal coupled state space, and the shortest Euclidean distance between the latest state point corresponding to the training state trajectory and the model training fault tolerance boundary is used as the model training fault tolerance margin.
[0036] When the model training fault tolerance margin is greater than the preset model training fault tolerance threshold, no model training fault tolerance warning is issued and model training continues. When the model training fault tolerance margin is less than or equal to the preset model training fault tolerance threshold, a model training fault tolerance warning is issued and model training is interrupted, rolling back to the model parameter checkpoint where the model training fault tolerance margin was greater than the preset model training fault tolerance threshold.
[0037] This invention provides a fault-tolerant analysis system for the entire training process of large models, including:
[0038] The data acquisition module is used to synchronously collect model training status data, training scene perturbation data, and forward propagation feature data according to the training cycle during the training of large models.
[0039] The cross-modal consistency entropy calculation module is used to perform probability mapping on the feature representation of each modality during the forward propagation process using forward propagation feature data, and to calculate the cross-modal consistency entropy based on the probability distribution of different modalities. This is used to characterize the degree of information coupling and alignment consistency between multimodalities.
[0040] The scene disturbance propagation intensity calculation module is used to analyze the cross-modal consistency change and cross-modal gradient direction change during the disturbance propagation process through model training state data and training scene disturbance data, and calculate the scene disturbance propagation intensity by the cross-modal consistency entropy change rate and the gradient direction coupling change amount.
[0041] The multimodal coupled state space construction module is used to construct a multimodal coupled state space by taking cross-modal consistency entropy and scene disturbance propagation intensity as state dimensions.
[0042] The model training fault tolerance boundary determination module is used to construct training state trajectories in the multimodal coupled state space and determine the model training fault tolerance boundary based on the distribution of training state trajectories under training scenario perturbation conditions.
[0043] The model training fault tolerance early warning module is used to provide early warning of model training fault tolerance based on the distance relationship between the training state trajectory of the current training stage and the model training fault tolerance boundary.
[0044] This invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the large model training full-process fault tolerance analysis methods described above, and completes the large model training full-process fault tolerance analysis.
[0045] This invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements any of the large model training full-process fault-tolerant analysis methods described above, and completes the large model training full-process fault-tolerant analysis.
[0046] Compared with the prior art, the significant advantages of this invention are:
[0047] 1. This invention, by synchronously collecting model training status, scene perturbation and forward propagation feature data throughout the entire large model training process, and introducing cross-modal consistency entropy and scene perturbation propagation intensity as core quantitative indicators, can accurately characterize the degree of modal information coupling, alignment consistency and the propagation and diffusion law of scene perturbation between modes during the training process of multimodal large models.
[0048] 2. This invention constructs a multimodal coupled state space and training state trajectory, determines the model training fault tolerance boundary based on the critical state point of convergence-forward-divergence, and realizes model training fault tolerance early warning based on the distance between the real-time state point and the fault tolerance boundary. It can proactively warn before the model training trajectory deviates, effectively avoid the cascading propagation of errors in the cross-modal alignment mechanism, and effectively improve the training stability and reliability of multimodal large models in complex environments. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0050] Figure 1 This is a flowchart of the fault-tolerant analysis method for the entire training process of large models in this invention;
[0051] Figure 2 This is a flowchart of the process for constructing the training state trajectory in this invention. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0053] like Figure 1 As shown, Figure 1 This is a flowchart of the fault-tolerant analysis method for the entire training process of large models in this invention, including:
[0054] Step S101: During the large model training process, model training state data, training scene perturbation data, and forward propagation feature data are collected synchronously according to the training cycle. Specifically, model training state data is mainly used to characterize the parameter update and optimization process of the model during the training phase, including gradient data, parameter update magnitude, loss function value, gradient norm, and gradient variance of the network layers corresponding to each modality. This can be obtained by logging the backpropagation process in the training framework or by reading the gradient tensor and optimizer state parameters in real time through the deep learning training framework. Training scene perturbation data is used to characterize external perturbation factors in the training environment and data sources that affect the stability of model training, including device heterogeneity information, data transmission delay information, differences in modal data acquisition timestamps, and the proportion of missing modal data. This can be obtained through the logs of the distributed training system monitoring system and the data acquisition system. Forward propagation feature data is used to reflect the feature representation formed by each modal input during the forward propagation process of the model, including the feature vectors of each modal feature extraction network. This can be obtained by extracting features from the output of each modal coding layer during the forward propagation phase of the model to obtain the feature representation of the corresponding training samples under different modalities.
[0055] Step S102: Probabilistically map the feature representations of each modality during the forward propagation process using the forward propagation feature data, and calculate the cross-modal consistency entropy based on the probability distribution of different modalities, which is used to characterize the degree of information coupling and alignment consistency between multimodalities.
[0056] Cross-modal consistency entropy is used to globally characterize the degree of information coupling and alignment consistency between multimodalities. Based on the average difference in the cross-modal probability distributions of all samples within the current training period, a quantitative indicator is calculated using entropy values. Cross-modal consistency entropy captures changes in the alignment state of multimodalities during training, identifying alignment deviations caused by disturbances such as device heterogeneity and asynchronous transmission. This provides crucial decision support for subsequent disturbance propagation analysis and fault tolerance control, improving the stability and robustness of model training. Cross-modal consistency entropy is calculated based on the probability distributions of different modalities, including:
[0057] Forward propagation feature matrices for different modalities in the current training cycle are extracted from the forward propagation feature data. In each modality's forward propagation feature matrix, the row vectors correspond to the feature representation of a single training sample, and the column vectors correspond to the feature dimensions. Specifically, during the forward propagation process of the large model in the current training cycle, forward propagation feature data output from the model feature extraction layer for each modality is captured synchronously. The forward propagation feature data is then denoised and dimension aligned. Denoising can be achieved using a 5×5 kernel Gaussian filter to remove random noise, and dimension alignment can be achieved by unifying the feature dimensions of different modalities to a preset dimension using linear interpolation, for example, a preset dimension of 512. The features are then categorized and organized according to modality, and a forward propagation feature matrix is constructed for each modality. Constructing the forward propagation feature matrix provides a standardized feature foundation for subsequent probability mapping and cross-modal difference analysis, thereby eliminating invalid interference in the original features and achieving a structured presentation of features from different modalities, avoiding analytical biases caused by messy feature data and inconsistent dimensions.
[0058] The average difference value of the cross-modal probability distribution corresponding to a single training sample is calculated by calculating the cross-modal probability distribution average difference value of the feature matrix of the forward propagation. The average difference value of the cross-modal probability distribution is used to measure the consistency of the feature representation of different modalities at the level of a single training sample. By calculating the average difference value of the cross-modal probability distribution, the distribution difference between different modalities is transformed into a quantifiable value, which provides the basic input for the subsequent calculation of cross-modal consistency entropy.
[0059] The cross-modal consistency entropy is calculated by normalizing the average difference in the cross-modal probability distribution of all training samples in the current training period and then substituting the normalized proportion into the entropy calculation formula. Specifically, the total number of training samples in the current training period is counted. Collect all The average difference values of the cross-modal probability distributions corresponding to each training sample constitute the difference value set. ,in, This represents the average difference value of the cross-modal probability distribution of the first training sample. This represents the average difference value of the cross-modal probability distribution of the second training sample. Indicates the first The average difference value of the cross-modal probability distribution of each training sample; the set of difference values is normalized, and the normalized proportion of the difference value of each training sample is calculated. The sum of the normalized proportions of all training samples is 1; substituting the normalized proportions into the entropy calculation formula... The cross-modal consistency entropy is obtained. The larger the cross-modal consistency entropy, the lower the information coupling and the worse the alignment consistency between the multimodalities. Conversely, the smaller the value, the higher the information coupling and the better the alignment consistency between the multimodalities.
[0060] The average difference value of the cross-modal probability distribution corresponding to a single training sample is calculated by analyzing the forward propagation feature matrix, including:
[0061] By normalizing the forward propagation feature matrix, normalized feature matrices for each modality are obtained. Then, the Softmax function is used to perform probability mapping on the normalized feature matrices of each modality to obtain the probability distribution matrix for each modality. Each element in the probability distribution matrix represents the probability value of the corresponding training sample in the corresponding feature dimension, and the sum of the probability values of a single training sample in all feature dimensions is 1. Specifically, for the normalized feature matrix corresponding to each modality, the Softmax function is performed row by row on the feature vector to obtain the probability value of a single training sample in each feature dimension. The probability values of all training samples are then output in the row and column order of the original matrix to obtain the probability distribution matrix for each modality. By transforming abstract feature values into interpretable probability distributions, a unified scale is provided for the quantitative analysis of cross-modal distribution differences.
[0062] For each training sample, the probability distribution vector corresponding to each mode is extracted based on the probability distribution matrix. The KL divergence (Kullback-Leibler divergence) between the probability distribution vectors of any two modes is calculated to characterize the difference in probability distributions of different modes on the same training sample. Specifically, for a single training sample, the vector of the corresponding row is extracted from the probability distribution matrix of each mode as the probability distribution vector of that mode, ensuring that the dimension of the probability distribution vectors of all modes is consistent. For any two mode probability distribution vectors... and Using the KL divergence formula In the formula and Represent the probability distribution vectors respectively and The Middle The probability values of each dimension; by traversing all modalities in pairs and calculating the KL divergence value of each modality pair, fine-grained basis for identifying local modal anomalies is provided by locating the modality pairs with alignment deviations in a single training sample.
[0063] By averaging the KL divergence among all different modalities corresponding to a single training sample, the average difference value of the cross-modal probability distribution of a single training sample is obtained, thereby achieving sample-level cross-modal consistency evaluation and improving the accuracy of fault tolerance analysis for model training.
[0064] Step S103: Analyze the cross-modal consistency changes and cross-modal gradient direction changes during the perturbation propagation process using model training state data and training scene perturbation data, and calculate the scene perturbation propagation intensity by using the cross-modal consistency entropy change rate and gradient direction coupling change amount;
[0065] By combining the dynamic changes in cross-modal consistency with the cooperative shift of gradient directions, the propagation intensity of scene perturbations across multiple modalities is quantified, capturing the impact of perturbations on model training. This provides core quantitative basis for subsequent construction of multimodal coupled state spaces, training state trajectories, and fault tolerance boundaries. The propagation intensity of scene perturbations is calculated using the rate of change of cross-modal consistency entropy and the amount of change in gradient direction coupling, including:
[0066] By extracting the cross-modal consistency entropy of the current training period and the previous adjacent training period, and calculating the rate of change of the cross-modal consistency entropy of the current training period using the first-order difference method, specifically, the formula for calculating the rate of change of the cross-modal consistency entropy is as follows: In the formula This represents the cross-modal consistency entropy of the current training cycle. This represents the cross-modal consistency entropy of the previous adjacent training cycle. This indicates the time interval between the current training cycle and the adjacent training cycle;
[0067] Gradient data for each modality within the current training period is extracted from the model training state data. Gradient vectors corresponding to each modality are constructed. The cosine of the angle between the gradient vectors of any two modalities is calculated using the vector dot product, and the cosine of the angle is used as the gradient direction similarity between the two modalities. Specifically, constructing gradient vectors for each modality includes: extracting gradient update data for the feature extraction layer, cross-modal alignment layer, and output layer of each modality within the current training period from the model training state data, and removing invalid data with a gradient of 0; classifying gradient data according to modality, and concatenating the gradient data of different network layers of the same modality in layer order to obtain the gradient vector corresponding to that modality. Each dimension of the gradient vector corresponds to the gradient value of a single training sample in each network layer of that modality. The number of network layers is equal to the number of dimensions of the gradient vector. By quantifying the synergy of gradient directions of each modality, the gradient direction shift between modalities caused by scene perturbations can be captured.
[0068] By averaging the gradient direction similarity among all different modalities, the average cross-modal gradient direction similarity of the current training period is obtained. The difference between the average cross-modal gradient direction similarity of the current training period and the previous training period is used as the gradient direction coupling change. By calculating the gradient direction coupling change, the scattered gradient relationships between modalities are transformed into a unified quantitative index.
[0069] The scene perturbation propagation intensity for the current training period is obtained by weighted summation of the cross-modal consistency entropy change rate and the gradient direction coupling change. The weights of the cross-modal consistency entropy change rate and the gradient direction coupling change are calculated by taking the Pearson correlation coefficients between the training scene perturbation data and each of these coefficients, and then normalizing the absolute values of these Pearson correlation coefficients. Specifically, the formula for calculating the scene perturbation propagation intensity is as follows: In the formula This represents the rate of change of cross-modal consistency entropy. This represents the amount of change in the gradient direction coupling. The weights representing the rate of change of cross-modal consistency entropy. , The weights represent the changes in the gradient direction coupling. ,in, The Pearson correlation coefficient represents the rate of change of cross-modal consistency entropy between perturbation data in the training scenario. The Pearson correlation coefficient represents the amount of change in the coupling between the perturbation data in the training scenario and the gradient direction. Weights are assigned using the Pearson correlation coefficient to reflect the degree of influence of different indicators on disturbance propagation.
[0070] Step S104: Construct a multimodal coupled state space by using cross-modal consistency entropy and scene disturbance propagation intensity as state dimensions;
[0071] By constructing a multimodal coupled state space to characterize the distribution and changing trends of multimodal coupled states under different training cycles, a foundation is provided for subsequent training state trajectory construction and fault tolerance boundary determination. The construction of the multimodal coupled state space includes: concatenating the cross-modal consistency entropy value and the scene disturbance propagation intensity corresponding to each training cycle into a joint state vector, where the cross-modal consistency entropy in the joint state vector is the first state dimension and the scene disturbance propagation intensity is the second state dimension; standardizing the joint state vector and constructing a two-dimensional coordinate system with the standardized cross-modal consistency entropy as the horizontal axis and the standardized scene disturbance propagation intensity as the vertical axis; mapping the joint state vector to the two-dimensional coordinate system to obtain the multimodal coupled state space, where the first state dimension is used to characterize the coupling and alignment consistency of multimodal information, and the second state dimension is used to characterize the degree of propagation influence of scene disturbances among multimodalities.
[0072] Step S105: Construct training state trajectories in the multimodal coupled state space, and determine the model training fault tolerance boundary based on the distribution of training state trajectories under training scenario perturbation conditions;
[0073] By defining the boundary between stable and unstable model training, a basis for subsequent fault tolerance warnings is provided, preventing model training from entering an irreversible unstable state, such as... Figure 2 As shown, Figure 2 This is a flowchart of constructing the training state trajectory in this invention. The model training fault tolerance boundary is determined based on the distribution of the training state trajectory under training scene perturbation conditions, including:
[0074] Using the training period as a time series, the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training period are combined to form state points. The state points are then connected in the multimodal coupled state space in chronological order to construct the training state trajectory. By generating the evolution trajectory of the training state with the period, the abstract training state is transformed into a trajectory form, providing a carrier for trajectory analysis and critical state identification.
[0075] Calculate the rate of change of trajectory direction between state points in adjacent training cycles to characterize the local evolution trend of the training state trajectory;
[0076] State points whose trajectory direction change rate exceeds a preset trajectory direction change threshold are designated as critical state points. These critical state points characterize the critical transition of the training state trajectory from stable convergence to divergent evolution. A set of critical state points is constructed by statistically analyzing all critical state points. The preset trajectory direction change threshold can be obtained through statistical analysis of training state trajectories in historical training tasks. Specifically, in historical large-scale model training tasks with normal convergence, the trajectory direction change rate between adjacent training cycle state points in the multimodal coupled state space is recorded, and their distribution characteristics are statistically analyzed. The principle is to calculate the mean of the rate of change of trajectory direction for training tasks of large models that have historically converged normally. and standard deviation ,Will As a preset threshold for trajectory direction change, it is used to identify the critical state point where the training state evolves from stable convergence to divergence.
[0077] The critical state range is obtained by performing region fitting on the set of critical state points, and the critical state range is used as the fault tolerance boundary for model training to characterize the boundary region where the model training transitions from a stable state to an unstable state.
[0078] Step S106: Provide early warning of model training fault tolerance based on the distance relationship between the training state trajectory of the current training stage and the model training fault tolerance boundary.
[0079] By monitoring the deviation between the current training state and the fault tolerance boundary in real time, early identification and warning of training risks can be achieved, avoiding model training crashes, reducing the consumption of ineffective computing resources, and thus improving the training stability and efficiency of multimodal large models in complex perturbation scenarios. Model training fault tolerance warnings are provided based on the distance relationship between the training state trajectory corresponding to the current training stage and the model training fault tolerance boundary, including:
[0080] By calculating the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training cycle in the current training phase, the training state trajectory of the current training phase is constructed in the multimodal coupled state space, and the shortest Euclidean distance between the latest state point corresponding to the training state trajectory and the model training fault tolerance boundary is used as the model training fault tolerance margin.
[0081] When the model training fault tolerance margin is greater than the preset model training fault tolerance threshold, no model training fault tolerance warning is issued and model training continues. When the model training fault tolerance margin is less than or equal to the preset model training fault tolerance threshold, a model training fault tolerance warning is issued and model training is interrupted, rolling back to the model parameter checkpoint where the model training fault tolerance margin was greater than the preset model training fault tolerance threshold. The warning information must clearly state the current training stage and the model training fault tolerance margin. The preset model training fault tolerance threshold can be determined based on the relationship between the fault tolerance margin distribution and the quality of training results in the historical training process. Specifically, the preset model training fault tolerance threshold can be determined by the quantile method: statistically analyze the fault tolerance margin data when the loss function converges and fluctuates in the historical training task, and take the 90th quantile as the preset fault tolerance threshold, so that the model training fault tolerance warning is triggered when the training state approaches this threshold.
[0082] This invention also proposes a fault-tolerant analysis system for the entire training process of large models, which completes the fault-tolerant analysis of the entire training process of large models based on any of the fault-tolerant analysis methods described in the invention.
[0083] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the large model training full-process fault tolerance analysis methods described above, and completes the large model training full-process fault tolerance analysis.
[0084] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements any of the large model training full-process fault-tolerant analysis methods described above, and completes the large model training full-process fault-tolerant analysis.
[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A fault-tolerant analysis method for the entire training process of large models, characterized in that, include: During the training of the large model, training state data, training scene perturbation data, and forward propagation feature data are collected synchronously according to the training cycle. Among them, the training state data includes gradient data, parameter update magnitude, loss function value, gradient norm, and gradient variance of each modality's corresponding network layer; the training scene perturbation data includes device heterogeneity information, data transmission delay information, differences in modality data collection timestamps, and the proportion of missing modality data; and the forward propagation feature data includes feature vectors of each modality feature extraction network. By performing probability mapping on the feature representations of each modality during the forward propagation process using forward propagation feature data, and calculating cross-modal consistency entropy based on the probability distribution of different modalities, it is used to characterize the degree of information coupling and alignment consistency between multimodalities. The cross-modal consistency change and cross-modal gradient direction change during the perturbation propagation process are analyzed using model training state data and training scene perturbation data. The scene perturbation propagation intensity is calculated by the cross-modal consistency entropy change rate and the gradient direction coupling change amount. A multimodal coupled state space is constructed by using cross-modal consistency entropy and scene disturbance propagation intensity as state dimensions; The training state trajectory is constructed in the multimodal coupled state space, and the model training fault tolerance boundary is determined according to the distribution of the training state trajectory under the perturbation conditions of the training scenario. Model training fault tolerance early warning is generated based on the distance relationship between the training state trajectory of the current training phase and the model training fault tolerance boundary.
2. The fault-tolerant analysis method for the entire training process of a large model according to claim 1, characterized in that, The calculation of cross-modal consistency entropy based on the probability distribution of different modalities includes: Forward propagation feature matrices for different modalities in the current training period are extracted from the forward propagation feature data. The row vectors in the forward propagation feature matrix of each modality correspond to the feature representation of a single training sample, and the column vectors correspond to the feature dimensions. The average difference value of the cross-modal probability distribution corresponding to a single training sample is calculated by analyzing the forward propagation feature matrix. The cross-modal consistency entropy is calculated by normalizing the average difference value of the cross-modal probability distribution of all training samples in the current training period and substituting the normalized proportion into the entropy calculation formula.
3. The fault-tolerant analysis method for the entire training process of a large model according to claim 2, characterized in that, The step of calculating the average difference value of the cross-modal probability distribution corresponding to a single training sample by analyzing the forward propagation feature matrix includes: By normalizing the forward propagation feature matrix, the normalized feature matrix corresponding to each mode is obtained. Then, the probability distribution matrix corresponding to each mode is obtained by performing probability mapping on the normalized feature matrix of each mode through the Softmax function. Each element in the probability distribution matrix represents the probability value of the corresponding training sample in the corresponding feature dimension, and the sum of the probability values of a single training sample in all feature dimensions is 1. For each training sample, the probability distribution vector corresponding to each mode is extracted according to the probability distribution matrix. The KL divergence between the probability distribution vectors of any two modes is calculated to characterize the difference in probability distribution of different modes on the same training sample. The average difference in the cross-modal probability distribution of a single training sample is obtained by averaging the KL divergence among all different modalities corresponding to the single training sample.
4. The fault-tolerant analysis method for the entire training process of a large model according to claim 1, characterized in that, The calculation of scene disturbance propagation intensity by coupling the cross-modal uniformity entropy change rate with the gradient direction change includes: The cross-modal consistency entropy of the current training cycle and the previous adjacent training cycle is extracted, and the rate of change of cross-modal consistency entropy of the current training cycle is calculated by the first-order difference method. Gradient data of each modality in the current training cycle is extracted from the model training state data, and gradient vectors corresponding to each modality are constructed. The cosine of the angle between the gradient vectors of any two modalities is calculated by vector dot product, and the cosine of the angle is used as the gradient direction similarity between the two modalities. The average gradient direction similarity of the current training period is obtained by averaging the gradient direction similarity of all different modes, and the difference between the average gradient direction similarity of the current training period and the previous training period is used as the gradient direction coupling change. The scene perturbation propagation intensity of the current training cycle is obtained by weighted summation of the cross-modal consistency entropy change rate and the gradient direction coupling change amount. The weights of the cross-modal consistency entropy change rate and the gradient direction coupling change amount are obtained by calculating the Pearson correlation coefficients between the training scene perturbation data and the cross-modal consistency entropy change rate and the gradient direction coupling change amount, respectively, and then normalizing the absolute value of the Pearson correlation coefficients.
5. The fault-tolerant analysis method for the entire training process of a large model according to claim 1, characterized in that, The construction of the multimodal coupled state space includes: concatenating the cross-modal consistency entropy value and the scene perturbation propagation intensity corresponding to each training period into a joint state vector, wherein the cross-modal consistency entropy in the joint state vector is the first state dimension and the scene perturbation propagation intensity is the second state dimension; standardizing the joint state vector and constructing a two-dimensional coordinate system with the standardized cross-modal consistency entropy as the horizontal axis and the standardized scene perturbation propagation intensity as the vertical axis; mapping the joint state vector to the two-dimensional coordinate system to obtain the multimodal coupled state space.
6. The fault-tolerant analysis method for the entire training process of a large model according to claim 1, characterized in that, The step of determining the model training fault tolerance boundary based on the training state trajectory distribution under training scenario perturbation conditions includes: Using the training period as a time series, the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training period are combined to form state points, and the state points are connected in the multimodal coupled state space in chronological order to construct the training state trajectory. Calculate the rate of change of trajectory direction between state points in adjacent training cycles to characterize the local evolution trend of the training state trajectory; State points whose trajectory direction change rate is greater than a preset trajectory direction change threshold are used as critical state points to characterize the critical transition of the training state trajectory from stable convergence to divergent evolution, and all critical state points are counted to construct a critical state point set. The critical state range is obtained by performing region fitting on the set of critical state points, and the critical state range is used as the fault tolerance boundary for model training.
7. The fault-tolerant analysis method for the entire training process of a large model according to claim 1, characterized in that, The method of providing early warning of model training fault tolerance based on the distance relationship between the training state trajectory corresponding to the current training stage and the model training fault tolerance boundary includes: By calculating the cross-modal consistency entropy and scene disturbance propagation intensity corresponding to each training cycle in the current training phase, the training state trajectory of the current training phase is constructed in the multimodal coupled state space, and the shortest Euclidean distance between the latest state point corresponding to the training state trajectory and the model training fault tolerance boundary is used as the model training fault tolerance margin. When the model training fault tolerance margin is greater than the preset model training fault tolerance threshold, no model training fault tolerance warning is issued and model training continues. When the model training fault tolerance margin is less than or equal to the preset model training fault tolerance threshold, a model training fault tolerance warning is issued and model training is interrupted, rolling back to the model parameter checkpoint where the model training fault tolerance margin was greater than the preset model training fault tolerance threshold.
8. A fault-tolerant analysis system for the entire training process of a large model, used to implement the fault-tolerant analysis method for the entire training process of a large model as described in any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to synchronously collect model training status data, training scene perturbation data, and forward propagation feature data according to the training cycle during the training of large models. The cross-modal consistency entropy calculation module is used to perform probability mapping on the feature representation of each modality during the forward propagation process using forward propagation feature data, and to calculate the cross-modal consistency entropy based on the probability distribution of different modalities. This is used to characterize the degree of information coupling and alignment consistency between multimodalities. The scene disturbance propagation intensity calculation module is used to analyze the cross-modal consistency change and cross-modal gradient direction change during the disturbance propagation process through model training state data and training scene disturbance data, and calculate the scene disturbance propagation intensity by the cross-modal consistency entropy change rate and the gradient direction coupling change amount. The multimodal coupled state space construction module is used to construct a multimodal coupled state space by taking cross-modal consistency entropy and scene disturbance propagation intensity as state dimensions. The model training fault tolerance boundary determination module is used to construct training state trajectories in the multimodal coupled state space and determine the model training fault tolerance boundary based on the distribution of training state trajectories under training scenario perturbation conditions. The model training fault tolerance early warning module is used to provide early warning of model training fault tolerance based on the distance relationship between the training state trajectory of the current training stage and the model training fault tolerance boundary.
9. A computer device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes the fault-tolerant analysis method for the entire training process of a large model as described in any one of claims 1-7 by calling the computer program stored in the memory.
10. A computer-readable storage medium, characterized in that, The computer program is stored therein, and when the computer program is run on the computer, it causes the computer to perform the fault-tolerant analysis method for the entire training process of a large model as described in any one of claims 1-7.