Industrial fault diagnosis method driven by multi-mode and single-mode information complementary gain

Abstract

The invention relates to a multi-modal and single-modal information complementary gain driven industrial fault diagnosis method, the method uses a multi-modal fusion fault diagnosis model to output a fault diagnosis result, the model comprises: a modal alignment module for performing alignment preprocessing on multi-modal data to obtain modal characterization of each modal; the cross-modal cross feature extraction module is used for realizing information interaction and complementary feature fusion among different modals by taking modal representations of all modals as input, and outputting cross enhancement features corresponding to each modal; the shared time sequence module takes the modal representation of the corresponding modal and the cross enhancement features of all the modalities as input, combines a self-attention mechanism to capture dependency characteristics in the time dimension, and outputs multi-modal feature representation and single-modal feature representation; and the multi-mode classification head and the single-mode classification head respectively output corresponding fault diagnosis results. Compared with the prior art, mutual gain of multi-modal information and single-modal information can be realized, and the fault diagnosis precision of a complex industrial system is improved.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal information fusion and industrial fault diagnosis, and in particular to an industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information. Background Art

[0002] Modern production systems place higher demands on the timeliness and accuracy of fault diagnosis to effectively ensure the safety of industrial production and reduce the economic losses caused by unplanned downtime. Currently, modern industrial systems generate a large amount of multi-source, heterogeneous data during operation, including but not limited to time-series sensor signals, process operation logs, and video surveillance data streams. This multimodal data reflects the operating status of industrial equipment from different perspectives, such as vibration, temperature, control logic status, and visual appearance. Therefore, if this multi-source modal data can be effectively integrated, it will not only achieve the complementary advantages of various modal information, but also significantly improve the detection accuracy and robustness of the fault diagnosis system, enhancing the safety assurance capabilities of the industrial production process.

[0003] Traditional multimodal fusion methods are primarily categorized into feature-level fusion and decision-level fusion. Feature-level fusion manually extracts features from each modality and then splices them into the input model, while decision-level fusion weights or votes on the diagnostic results of each unimodal model. However, these methods often struggle to achieve precise mapping and adaptive adjustment when dealing with structural differences, scale inconsistencies, and dynamic relationships in sensor data, and their ability to learn complex nonlinear features is limited.

[0004] In recent years, the rise of deep learning and large model architectures has provided a more flexible solution for multimodal diagnosis. Fusion technology based on the attention mechanism can dynamically allocate the weights of each modality during the training process, achieve adaptive focusing on important information, and thus take into account both efficiency and generalization capabilities. For example, CN118011994A discloses a fault diagnosis system and method based on a cross-modal industrial large model, including: a data acquisition module: used to collect cross-modal historical data and corresponding industrial system fault types from the industrial large model, and the cross-modal historical data includes: sensor data, image data and sound data; a data classification module: classifies the industrial system fault types corresponding to the collected cross-modal historical data, and adds fault type labels to the cross-modal historical data according to each fault type; helps to discover the relationship between multiple feature data and fault types; extracts multimodal data and links it with multiple fault types, so as to facilitate better diagnosis of industrial equipment faults. However, this method and existing research are all aimed at improving the overall fusion performance, and do not focus on the gain effect of multimodal information in single-modal diagnosis scenarios. In real-world industrial applications, data from certain modalities is often missing or abnormal due to sensor failures, communication interruptions, or harsh environments. If diagnostic models only optimize multimodal fusion while ignoring improvements in single-modal performance, it will be difficult to ensure stable and reliable diagnostic results in scenarios with incomplete data or limited time.

[0005] In summary, the challenges and research gaps in leveraging multimodal information for industrial fault diagnosis are as follows: First, current research focuses on directly integrating multiple modalities, but has not explored how multimodal information can inversely enhance the diagnostic capabilities of a single modality—how to leverage compensatory information from additional modalities to enhance the accuracy and robustness of fault identification under single-modality conditions. Second, industrial sites are often affected by sensor failures, communication interruptions, or environmental interference, resulting in the loss of some modal data. Existing diagnostic models lack the ability to operate stably when critical information is missing.

[0006] Therefore, there is an urgent need for a general adaptive fault diagnosis method that can fully utilize the advantages of multimodal collaboration and obtain enhanced diagnostic effects under single modal conditions, so as to meet the dual needs of complex industrial systems for high availability and high reliability. Summary of the Invention

[0007] The purpose of the present invention is to provide an industrial fault diagnosis method driven by the complementary gain of multimodal and unimodal information, which utilizes the multimodal data collected during the operation of complex industrial systems to achieve accurate diagnosis of various fault conditions of the system.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] An industrial fault diagnosis method driven by complementary gain of multimodal and unimodal information is proposed. The method inputs the acquired multimodal data of the industrial system and the fault state label into a multimodal fusion fault diagnosis model based on the attention mechanism and outputs the fault diagnosis result. The multimodal fusion fault diagnosis model includes a modality alignment module, a cross-modal feature extraction module corresponding to each modality and a shared timing module, as well as a multimodal classification head and a unimodal classification head.

[0010] The modal alignment module performs alignment preprocessing on the multimodal data to obtain the modal representation of each modality; the cross-modal cross-feature extraction module takes the modal representation of all modalities as input, realizes information interaction and complementary feature fusion between different modalities, and outputs the cross-enhanced features corresponding to each modality; the shared timing module takes the modal representation of the corresponding modality and the cross-enhanced features of all modalities as input, combines the self-attention mechanism to capture the dependency characteristics in the time dimension, and outputs multimodal feature representation and unimodal feature representation; the multimodal feature representation is input into the multimodal classification head to perform multimodal fault diagnosis and output the multimodal fault diagnosis result; the unimodal feature representation corresponding to each modality is respectively input into the corresponding unimodal classification head, and the corresponding unimodal fault diagnosis result is output;

[0011] The multimodal fusion fault diagnosis model updates parameters by combining a similarity-aware gradient alignment mechanism with an uncertainty-guided parameter estimation method.

[0012] The modal alignment module performs the following operations:

[0013] The M types of modal data collected from the operation of the industrial system are recorded as X (1) ,…,X (m) ,…,X (M) :in Input data for each mode, l (·) ,d (·) and N represent the sequence length, feature dimension, and number of samples, respectively. For the modal data corresponding to any modality m∈{1,2,…,M}, the modality alignment module extracts features and aligns the modal data through an encoder and a one-dimensional convolutional layer:

[0014]

[0015] in, represents the modal representation obtained after mode m is processed, d c To unify the feature dimension, Encoder (m) (·) represents the modality-specific encoder structure, whose parameters are Used to extract original features within the modality; Conv1D (m) is a one-dimensional convolution operation, and its parameters are Used to map modal features to a unified dimension.

[0016] For any modality m, its corresponding cross-modal feature extraction module uses the modality representation H of the modality (m) As a query, by comparing with other modalities m ′ The attention mechanism calculation between them realizes the active extraction of cross-modal complementary information, where m ′ ∈{1,2,…,M} and m ′ ≠m, specifically:

[0017] The mode m ′ Modal representation H (m′) Converted to query vector Q (m′) , the representation H of mode m (m) Convert to key K (m) Sum V (m) :

[0018]

[0019] in, Represent the corresponding linear transformation weight matrix, and H (m′) and H (m) Convert to d u dimensional space; H (m) Convert to d v Dimensional space;

[0020] Calculate the modality m from modality m through the attention mechanism ′ The cross-enhanced feature H obtained from (m′→m) :

[0021]

[0022] in, Indicates CATT (m′→m) The set of learnable parameters in CATT (m′→m) Indicates that from mode m ′ Cross-modal attention computation to modality m.

[0023] The shared timing module obtains multimodal feature representations through the following steps:

[0024] All cross-enhanced features related to the target modality m are concatenated to form the aggregated cross-modal feature representation of modality m, which is denoted as: in, represents the multimodal aggregation feature of mode m, the superscript ω indicates that the feature belongs to the multimodal representation space, H (i→m)Cross-enhanced features obtained by modality i from modality m;

[0025] Aggregate multimodal features Input to the corresponding mth shared timing module SelfAttn (m) (·), dynamically captures the dependency patterns at different time steps, and the calculation formula is as follows:

[0026]

[0027] Among them, Z (ω,m) represents the multimodal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

[0028] The shared timing module obtains a single-modal feature representation through the following steps:

[0029] Modal representation H for mode m (m) Repeated stacking is performed to form an extended feature matrix for sequence feature modeling, which is expressed as:

[0030] Expand the feature matrix Input to the corresponding mth shared timing module SelfAttn (m) (·) is processed, and the calculation process is as follows:

[0031]

[0032] Among them, Z (m) represents the single-modal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

[0033] The multimodal fusion fault diagnosis model uses cross entropy loss to measure the difference between the model output and the true label. The total loss function of the model is Expressed as:

[0034]

[0035] Among them, M is the number of modalities, and the loss of the multimodal classification head is Defined as:

[0036]

[0037] Among them, C represents the number of classes, N represents the number of samples, Indicates the real fault status label, represents the predicted probability of the i-th class of the j-th sample output by the multimodal classification head;

[0038] The loss of the unimodal classification head corresponding to modality m Defined as:

[0039]

[0040] in, represents the predicted probability of the i-th class of the j-th sample output by the unimodal classification head corresponding to modality m.

[0041] The parameters of the multimodal classification head, unimodal classification head and cross-modal cross feature extraction module are updated according to their corresponding gradients. The parameter gradient in the modal alignment module is calculated according to the total loss function of the multimodal fusion fault diagnosis model. The parameter gradient of the shared timing module is determined according to the losses corresponding to the multimodal classification head and the unimodal classification head, and gradient fusion is performed using an adaptive gradient modulation algorithm.

[0042] The loss of the multimodal classification head and the loss of the unimodal classification head The gradients relative to the shared parameters of the shared timing module are: and in, is the multimodal gradient, is a single-mode gradient, Represents the corresponding loss function for parameter θ (m) Gradient calculation of ;

[0043] The adaptive gradient modulation algorithm performs the following steps:

[0044] In the tth iteration of training, a mini-batch is randomly sampled from the training dataset, denoted as B t , the sample size is b, and the mini-batch gradient is recorded as Defined as Where (.)∈{ω,m} indicates that the gradient corresponds specifically to the multimodal branch or the unimodal branch, and the mini-batch gradient is used as the full gradient According to the central limit theorem, when the batch size b is larger than the preset value, the multimodal gradient and the unimodal gradient and Approximately regarded as obeying multivariate Gaussian distribution:

[0045]

[0046] in, and Respectively represent the covariance matrix of multimodal and unimodal stochastic gradients at the tth iteration, N (.) represents a normal distribution;

[0047] At the t+1th iteration, a fusion gradient is constructed based on the multimodal gradient and the unimodal gradient to update the parameters of the mth shared timing module, which is recorded as

[0048]

[0049] Among them, c ω ,c m is the adaptive weighting parameter, c ω ,c m ≥0 and c ω +c m =1;

[0050] Based on the Gaussian distribution assumption, the fusion gradient Modeled as:

[0051]

[0052] Among them, N (f) is the fusion distribution, is the mean vector of the fused gradient, is the covariance matrix of the fused gradient;

[0053] To determine the optimal fusion gradient direction, minimize the fusion distribution N (f) With source distribution N (ω) 、N (m) The weighted sum of the KL divergence between , the specific optimization objectives are as follows:

[0054]

[0055] Among them, D KL represents the KL distance, λ ω and λ m It is a dynamic adjustment scale factor for similarity perception, which is used to adjust the contribution weight of the multimodal and unimodal loss gradients to the fusion gradient respectively;

[0056] Based on the best fusion gradient direction Update the parameters of the mth shared timing module:

[0057]

[0058] Where η is the learning rate, Represents the parameters of the mth shared timing module at the tth iteration.

[0059] The dynamic adjustment scale factor is determined based on a similarity-aware gradient alignment mechanism, specifically:

[0060] Let γ be the multimodal gradient and unimodal gradient Calculate the gradient similarity for the angle between them:

[0061]

[0062] Let cos(α) be the target similarity between two gradients. If the gradient similarity cos(γ) > cos(α), it means that the two types of gradient directions are sufficiently aligned, and set λ ω = λ m = 1. If the gradient similarity cos(γ) < cos(α), then implement the similarity-aware gradient alignment mechanism. The detailed steps are as follows:

[0063] If the prediction result of the multimodal classification head is correct while the prediction result of the unimodal classification head is incorrect, then correct the unimodal gradient direction. α is the angle between the fused gradient and the multimodal gradient directions, and the target similarity is:

[0064]

[0065] Model the gradient fusion based on geometric relationships and apply the sine law to the triangle formed by three gradient vectors:

[0066] <​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0076] Based on the dynamic adjustment of the scaling factor, the optimization objective is expanded to:

[0077]

[0078] Pick Relative to The gradient of , and setting it to zero gives:

[0079]

[0080] Thus we get The closed-form solution of :

[0081]

[0082] Experience Gradient and is used as an estimate of the true gradient mean, i.e.

[0083] The inverse of the gradient covariance is approximated using the Hessian matrix of the loss function:

[0084]

[0085] Get the fused gradient The calculation formula is:

[0086]

[0087] The adaptive weighting parameters are:

[0088]

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] (1) The present invention combines the similarity-aware gradient alignment mechanism with the uncertainty-guided parameter estimation method to propose a principled optimization strategy for fusing multimodal and unimodal information. Specifically, similarity-aware alignment can ensure the consistency of the multimodal and unimodal gradient directions, thereby promoting consistent and practical parameter updates and avoiding training instability caused by conflicts between modalities. This alignment mechanism achieves two-way optimization: on the one hand, the multimodal representation enhances the expressive power of the unimodal representation by providing rich contextual information; on the other hand, the stability of the unimodality provides guidance for the multimodal fusion process, thereby improving the robustness of the overall system. At the same time, uncertainty-guided parameter estimation, as an adaptive modulation mechanism, can dynamically adjust the contribution ratio of each modal gradient in the model learning process according to the confidence of each modal gradient. This mechanism effectively prevents the model from being dominated by unreliable modal features, promotes balanced learning between modalities, and further improves the generalization ability of the model and the stability of training. In summary, the parameter estimation algorithm based on similarity-awareness and uncertainty-guided proposed in the present invention jointly promotes an efficient multimodal fusion strategy. This strategy effectively captures the rich features of multimodal representation while taking into account the stability of unimodal features, thereby achieving an optimal balance between model expressiveness and robustness, and ultimately achieving efficient feature interaction and stable parameter optimization throughout the entire training process.

[0091] (2) When the industrial site is affected by sensor failure, communication interruption or environmental interference, and some modal data is lost, the diagnostic model of the present invention can stably output diagnostic results through the complementarity and balance of single-modal classification and multi-modal classification, and still has the ability to operate stably. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of the structure of the multimodal fusion fault diagnosis model of the present invention;

[0093] Figure 2 is the adaptive weight coefficient ratio λ in the embodiment of the present invention ω / λ m Schematic diagram of the evolution process;

[0094] Figure 3 is the confusion matrix obtained by performing fault state diagnosis in the embodiment of the present invention, wherein (3a) is the confusion matrix of the multimodal diagnosis result, (3b) is the confusion matrix of the pressure modal diagnosis result, (3c) is the confusion matrix of the process modal diagnosis result, and (3d) is the confusion matrix of the video modal diagnosis result;

[0095] Figure 4The results of ablation experiments on the proposed model structure, similarity-aware gradient alignment mechanism, and uncertainty-based parameter estimation algorithm are presented. DETAILED DESCRIPTION

[0096] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0097] This embodiment provides an industrial fault diagnosis method driven by the complementary gain of multimodal and unimodal information. The method inputs the acquired multimodal data of the industrial system and the fault status label into a multimodal fusion fault diagnosis model based on the attention mechanism and outputs the fault diagnosis result. Figure 1 As shown in FIG, the multimodal fusion fault diagnosis model includes a modal alignment module, a cross-modal feature extraction module and a shared timing module corresponding to each modality, as well as a multimodal classification head and a single-modal classification head.

[0098] (1) Modal alignment module

[0099] The M types of modal data collected from the operation of the industrial system are recorded as X (1) ,…,X (m) ,…,X (M) :in Input data for each mode, l (·) ,d (·) and N represent the sequence length, feature dimension, and number of samples, respectively.

[0100] Since each modal information comes from different sources, the physical mechanisms and perceptual dimensions contained therein are different, so there is natural heterogeneity and statistical distribution inconsistency between modalities. This difference and imbalance poses a major challenge to the fusion of multimodal information. If not handled properly, it may even reduce the overall diagnostic performance of the system. In order to solve the above problems, the present invention designs corresponding feature preprocessing structures for different types of modalities. For example, for industrial video modalities that contain a large amount of redundant and spatiotemporal related information, a specific video preprocessing module is used to improve the efficiency of information extraction. On this basis, in order to achieve consistency in the feature space between modalities, the extracted intra-modal features are mapped to a unified feature dimension d through a one-dimensional convolution (Conv1D) layer. c , so that cross-modal attention mechanism and information fusion operations can be performed later.

[0101] For the modal data corresponding to any modality m∈{1,2,…,M}, the modality alignment module extracts features and aligns the modal data through an encoder and a one-dimensional convolutional layer:

[0102]

[0103] in, represents the modal representation obtained after mode m is processed, d c To unify the feature dimension, Encoder (m) (·) represents the modality-specific encoder structure, whose parameters are Used to extract original features within the modality; Conv1D (m) is a one-dimensional convolution operation, and its parameters are Used to map modal features to a unified dimension.

[0104] (2) Cross-Fusion Channel Attention (CFCA)

[0105] After completing the modal alignment, the present invention further constructs the CFCA module, which aims to realize the information interaction and complementary feature fusion between different modalities, so as to fully explore the potential synergistic relationship between the modal data in the industrial system. Specifically, the modal representation H of all modalities is used. (m) , m∈{1,2,…,M} is taken as input, and a cross-modal attention mechanism is introduced to enhance the feature coupling capability between modalities, thereby improving the multimodal fusion effect and representation capability of the overall model, and outputting the cross-enhanced features corresponding to each modality.

[0106] Given that the system contains M modalities, in order to achieve interactive learning among all modalities, the present invention constructs an independent cross-modal attention mechanism for each modality m, and cumulatively forms M CFCA modules. For any modality m, its corresponding cross-modal cross feature extraction module uses the modality representation H of the modality (m) As a query, by comparing with other modalities m ′ The attention mechanism calculation between them realizes the active extraction of cross-modal complementary information, where m ′ ∈{1,2,…,M} and m ′ ≠m.

[0107] To achieve the above goal, we first transform the modal representation H of mode m into (m′) Converted to query vector Q (m′ ), the representation H of mode m (m) Convert to key K (m) Sum V (m) :

[0108]

[0109] in, Represent the corresponding linear transformation weight matrix, and H(m′) and H (m) Convert to d u dimensional space; H (m) Convert to d v Dimensional space.

[0110] Calculate the modality m from modality m through the attention mechanism ′ The cross-enhanced feature H obtained from (m′→m) :

[0111]

[0112] in, Indicates CATT (m′→m) The set of learnable parameters in CATT (m′→m) Indicates that from mode m ′ Cross-modal attention computation to modality m.

[0113] Since the attention weight is based on the query Q (m′ ) and key K (m) The similarity between them is calculated, so the output of the CFCA module H (m′→m) The sequence length dimension remains consistent with the query modality feature. However, the representation space of the output feature is affected by the value matrix V (m) The influence of the defined feature space means that the output has been projected into a feature representation domain consistent with the value matrix. This cross-modal cross-feature extraction module enables modality m to selectively focus on and integrate features from modality m. ′ The useful information collected effectively improves the collaborative learning ability between different modalities and provides richer and more accurate feature support for subsequent decision modules.

[0114] (3) Shared Temporal Attention (STA)

[0115] After using the cross-modal cross-feature extraction module to complete the intra-modal feature extraction and inter-modal feature interaction of modality m, the present invention takes the modal representation of the corresponding modality and the cross-enhanced features of all modalities as input, further combines the self-attention mechanism to capture the dependency characteristics in the temporal dimension, and outputs multimodal feature representation and unimodal feature representation. The STA module designed by the present invention is not only applied to intra-modal representation, but also acts on cross-modal representation, enabling each modality to share and fuse temporal dependency information, thereby jointly promoting a more comprehensive and robust feature expression.

[0116] Specifically, for temporal dependency modeling between modalities, all cross-enhanced features related to the target modality m are first concatenated to form an aggregated cross-modal feature representation of modality m, denoted as: in, represents the multimodal aggregation feature of mode m, the superscript ω indicates that the feature belongs to the multimodal representation space, H (i→m) The cross-enhanced features obtained by modality i from modality m.

[0117] Then, the multimodal aggregation features Input to the corresponding mth shared timing module SelfAttn (m) (·) dynamically captures dependency patterns at different time steps. The STA module can effectively integrate multimodal time series information by calculating global temporal dependencies, improving the temporal consistency and expressiveness of feature representation. The specific calculation formula is as follows:

[0118]

[0119] in, represents the multimodal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

[0120] At the same time, the STA module is used to capture the temporal dependency within the mode, thereby providing support for single-mode diagnosis tasks. Specifically, the modal representation H of mode m is first (m) Repeated stacking is performed to form an extended feature matrix for sequence feature modeling, which is expressed as:

[0121] Then, the feature matrix is ​​expanded Input to the corresponding mth shared timing module SelfAttn (m) (·) is processed, and the calculation process is as follows:

[0122]

[0123] in, represents the single-modal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

[0124] It is worth noting that in order to achieve the joint modeling of the temporal dynamics within a single modality and the interaction characteristics between different modalities, the present invention introduces a shared parameter θ (m)This design strategy effectively unifies the interactive learning of temporal dependencies within and between modalities, and promotes the collaborative optimization of unimodal and multimodal features. Through the above-mentioned joint optimization mechanism, unimodal features not only contribute to the stabilization and complementation of features in the multimodal fusion process, but also multimodal contextual information can provide effective guidance for unimodal temporal modeling. Ultimately, the generated unimodal and multimodal feature representations are used for unimodal and multimodal classification tasks, respectively, supporting the flexibility and generalization of the proposed method.

[0125] (4) Classification Header

[0126] The present invention proposes a multi-classification head framework, which enables the model to perform fault diagnosis based on both multimodal interaction information and each single modal information simultaneously, thereby flexibly responding to modal missing problems that may occur in industrial scenarios and improving the robustness and practicality of the system.

[0127] Specifically, the multimodal features of M STA modules are represented as Z (ω,m) The final time step of is concatenated in the feature dimension to form a unified multimodal fusion representation For each sample j, record represents z (ω) The jth column of , the feature vector is then input into the multimodal classification head for multimodal fault diagnosis. The calculation process is as follows:

[0128]

[0129] Among them, CLS head (ω) (·) represents the multimodal classification head, Its learnable parameters, V (ω) and b (ω) Represent the weight matrix and bias term respectively. The softmax function is used to output the predicted probability of each fault category. is the output multimodal fault diagnosis result.

[0130] At the same time, for each STA module output single modal feature representation Z (m) The final time step of is concatenated on the feature dimension, and the resulting fusion representation Input to its corresponding unimodal classification head. For each sample j, represents z (m) The j-th column of , the output is calculated as follows:

[0131]

[0132] Among them, CLS head (m) (·) represents the unimodal classification head of modality m, Its learnable parameters, V (m) and b (m) The corresponding weight matrices and bias terms, is the single-mode fault diagnosis result corresponding to mode m output.

[0133] In summary, the model proposed in this invention generates a total of M+1 classification outputs, including multimodal fault diagnosis outputs and M single-mode fault diagnosis results where m∈{1,2,...,M}. This multi-classification head architecture not only improves the model's fault identification capability in multimodal fusion scenarios, but also gives the model the flexibility and generalization to independently diagnose even in scenarios where a single modality is missing or incomplete.

[0134] (5) Model training process

[0135] The multimodal fusion fault diagnosis model uses cross entropy loss to measure the difference between the model output and the true label. The total loss function of the model is Expressed as:

[0136]

[0137] Among them, M is the number of modalities, and the loss of the multimodal classification head is Defined as:

[0138]

[0139] Among them, C represents the number of classes, N represents the number of samples, Indicates the real fault status label, represents the predicted probability of the i-th class of the j-th sample output by the multimodal classification head;

[0140] The loss of the unimodal classification head corresponding to modality m Defined as:

[0141]

[0142] in, represents the predicted probability of the i-th class of the j-th sample output by the unimodal classification head corresponding to modality m.

[0143] The model parameters proposed in this invention are optimized and learned by iterative back propagation algorithm based on loss function, and uncertainty guided parameter estimation algorithm is introduced for parameter estimation. Multimodal classification head parameters And CFCA module parameters are updated only according to their corresponding gradients. At the same time, the parameters in the modality alignment module and They are mainly responsible for capturing low-level feature representations of the modality and are therefore less sensitive to conflicts in supervisory signals for specific tasks. Therefore, their gradients are calculated based on the total loss defined in Equation (8), thereby achieving effective and sufficiently stable gradient accumulation. In contrast, the mth STA module adopts a shared parameter design to take into account the dual tasks of processing single-modality and cross-modal temporal features, thus requiring a special adaptive gradient modulation algorithm to achieve complementary enhancement between the two feature representations.

[0144] Loss of the multimodal classification head and the loss of the unimodal classification head The gradients relative to the shared parameters of the shared timing module are: and in, For multimodal gradients, capture rich global and context-aware features, It is a single-modal gradient, which emphasizes modal specificity and stable intra-modal feature expression. Represents the corresponding loss function for parameter θ (m) Directly merging the two types of gradients may lead to gradient interference, as the conflict between different objectives will hinder the convergence of the training process and even reduce the quality of the final feature representation. In addition, considering the differences in modal quality and integrity of each sample in industrial scenarios, the use of a static gradient fusion strategy is prone to overfitting or invalid learning. Based on the above problems, the present invention proposes an adaptive gradient modulation algorithm that can simultaneously consider the semantic alignment characteristics and their statistical confidence when fusing gradients.

[0145] Specifically, the model proposed in this invention is optimized using stochastic gradient descent (SGD). In the tth iteration of training, a mini-batch is randomly sampled from the training dataset, denoted as B t , the sample size is b, and the mini-batch gradient is recorded as Defined as Where (.)∈{ω,m} indicates that the gradient corresponds specifically to the multimodal branch or the unimodal branch, and the mini-batch gradient is used as the full gradient According to the central limit theorem, when the batch size b is large enough, the multimodal gradient and the unimodal gradient and Approximately regarded as obeying multivariate Gaussian distribution:

[0146]

[0147] in, and Respectively represent the covariance matrix of the multimodal and unimodal stochastic gradients at the tth iteration, which is used to measure the uncertainty of its gradient, N (.) represents a normal distribution. A smaller covariance indicates a more stable gradient, while a larger covariance reflects a higher volatility in the gradient.

[0148] At the t+1th iteration, a fusion gradient is constructed based on the multimodal gradient and the unimodal gradient to update the parameters of the mth shared timing module, which is recorded as

[0149]

[0150] Among them, c ω ,c m is the adaptive weighting parameter, c ω ,c m ≥0 and c ω +c m = 1. Fusion gradient The construction aims to balance the gradient contributions from the multimodal and unimodal branches, and the adaptive weighting parameter c ω and c m The method for determining is based on a similarity-aware gradient alignment mechanism, which will be described in detail later.

[0151] Based on the Gaussian distribution assumption in formula (11), the fusion gradient Modeled as:

[0152]

[0153] Among them, N (f) is the fusion distribution, is the mean vector of the fused gradient, is the covariance matrix of the fused gradient.

[0154] To determine the optimal fusion gradient direction, minimize the fusion distribution N (f) With source distribution N (ω) 、N (m) The weighted sum of the KL divergence between , the specific optimization objectives are as follows:

[0155]

[0156] Among them, D KL represents the Kullback-Leibler (KL) distance, λ ω and λ m This is a similarity-aware dynamic scaling factor that adjusts the weight of the multimodal and unimodal loss gradients on the fusion gradient. This mechanism ensures that the fusion gradient is updated in a consistent direction while maintaining optimization stability.

[0157] Based on the best fusion gradient direction Update the parameters of the mth shared timing module:

[0158]

[0159] Where η is the learning rate, Represents the parameters of the mth shared timing module at the tth iteration.

[0160] In summary, KL divergence provides a principled framework for fusing two source distributions. The introduction of a similarity-aware, dynamically adjusted scaling factor further guides the gradients toward a consistent fusion direction, taking into account both multimodal and unimodal gradient characteristics. The following describes the proposed similarity-aware gradient alignment mechanism and overall parameter estimation process to achieve this.

[0161] 1. Similarity-aware gradient alignment mechanism

[0162] In some industrial scenarios, some modalities may be subject to noise interference or data loss. In this case, the model must maintain reliable single-mode diagnostic capabilities while effectively utilizing the available multimodal information gain diagnostic capabilities. To this end, the STA module proposed in the present invention can effectively jointly model intra-modal features and inter-modal interaction features, so that the network can adaptively support single-modal and multi-modal diagnostic tasks. However, due to factors such as noise interference, signal conflict or modal data imbalance, the gradients derived from single-modal and multi-modal diagnostic targets may be at risk of inconsistent directions. Therefore, in KL-based gradient fusion, if equal weights are simply assigned to each source gradient, it may lead to inconsistent parameter update directions, affecting the performance and convergence stability of the model. Specifically, the change in the loss value at the tth iteration of the training process is Performing Taylor expansion yields:

[0163]

[0164] Where <·,·> represents the inner product. Substituting the weighted gradient fusion of (12) into the above expression, we can obtain:

[0165]

[0166] Among them, S ωm Representing quantized multimodal gradients and unimodal gradient The gradient similarity between the alignments is measured using cosine similarity, which is defined as:

[0167]

[0168] where γ is the multi-modal gradient and the single-modal gradient between the angles.

[0169] Therefore, Equation (17) shows that the change in loss depends not only on the magnitude of each gradient, but also on their alignment. When the multi-modal gradient and the single-modal gradient are perfectly aligned (i.e., S ωm ≈1), their contributions reinforce each other, resulting in greater loss reduction and more efficient gradient updates. On the contrary, when the gradients are poorly aligned or in conflict, they may neutralize each other in the loss change resulting in weaker or even harmful gradient updates.

[0170] To solve the above problems, the present invention proposes a similarity-aware gradient alignment mechanism, which promotes more effective feature learning by dynamically adjusting the contribution weights of the KL divergence terms. Specifically, the present invention does not force the two types of gradients to achieve perfect alignment, because forcing the gradients to be perfectly aligned may suppress the useful specific information contained in different tasks, thereby weakening the discriminative ability of the model. To achieve an effective balance between the contributions of each gradient, the present invention proposes a similarity-aware strategy based on the prediction result orientation. This strategy has result orientation and self-adaptability, and can dynamically adjust the fusion weights of the multi-modal and single-modal gradients according to the current prediction result, while retaining the advantages of the complementary signals of each modality and more precisely controlling the degree of gradient alignment.

[0171] Let cos(α) be the target similarity between the two gradients. If the gradient similarity cos(γ)>cos(α), it means that the two types of gradient directions are fully aligned. In this case, no further correction is required, and λ ω =λ m =1. If the gradient similarity cos(γ)<cos(α), the similarity-aware gradient alignment mechanism is implemented, and the detailed steps are as follows:

[0172] i) If the prediction result of the multi-modal classification head is correct, while the prediction result of the single-modal classification head is incorrect, it means that the single-modal gradient deviates from the global optimal direction. Although increasing the weight of the multi-modal gradient may be a simple solution, implementing it without selection will introduce optimization bias and破坏训练的稳定性。基于此,本发明提出一种更有原则的策略:不是放大多模态梯度,而是校正单模态梯度方向,使其能够从信息量更大的多模态信息中受益。具体来说,α成为融合梯度 and the multi-modal gradient between the angles, and the corresponding target similarity is:

[0173] It should be noted that there is an unclear expression "破坏训练的稳定性" in the original text. You may need to check and correct it according to the actual situation.

[0174] To determine the appropriate fusion weight λ ω and λ m , models gradient fusion based on geometric relationships and applies the law of sine to the triangle formed by the three gradient vectors:

[0175]

[0176] Further deduction yields:

[0177]

[0178] This formula (21) calculates the relative importance of two gradients, not only taking into account their magnitude and similarity, but also encouraging progress towards better diagnostic results.

[0179] ii) If the prediction result of the unimodal classification head is correct, while the prediction result of the multimodal classification head is incorrect, then the unimodal representation provides more reliable information, indicating that the multimodal fusion mechanism may have introduced noise or misalignment. To solve this problem, the present invention adjusts the multimodal gradient to be consistent with the unimodal gradient to ensure that the fusion process does not mask the key unimodal features. Then α becomes the fusion gradient and unimodal gradient The angle between them is:

[0180]

[0181] Similarly, gradient fusion is modeled based on geometric relationships and the sine law is applied to the triangle formed by the three gradient vectors:

[0182]

[0183] Further deduction yields:

[0184]

[0185] iii) If the prediction results of the unimodal classification head and the multimodal classification head are both correct or wrong, it means that the model has no obvious bias in the representation of the two modalities. Therefore, equal weights are assigned to maintain the optimization balance and λ is set. ω =λ m = 1. This ensures equal contributions from both learning objectives, maintains optimization stability and achieves optimal feature fusion.

[0186] 2. Parameter Estimation Method

[0187] In obtaining the similarity perception weight λ ω and λ mFinally, the complete process of the final fusion gradient direction estimation method proposed in the present invention is as follows: Based on the assumption that each gradient obeys Gaussian distribution, the objective function in formula (14) can be further expanded to:

[0188]

[0189] Since the present invention only optimizes the mean And the fusion covariance is fixed (or can be considered a constant during the optimization process), so take Relative to The gradient of , and setting it to zero gives:

[0190]

[0191] Thus we get The closed-form solution of :

[0192]

[0193] Experience Gradient and is used as an estimate of the true gradient mean, i.e.

[0194] The present invention uses the Hessian matrix of the loss function to approximate the inverse of the gradient covariance:

[0195]

[0196] Get the fused gradient The calculation formula is:

[0197]

[0198] Then the adaptive weighting parameters in formula (12) are:

[0199]

[0200] From an intuitive perspective, the magnitude of the Hessian matrix of the loss function along a certain gradient direction reflects the curvature characteristics of that direction. A larger Hessian matrix indicates that the direction has a higher curvature, which means that the learning process in this direction is relatively stable and has a clear optimization path. Therefore, assigning a higher weight to this direction helps promote more reliable and stable model convergence. Conversely, a smaller Hessian matrix means low curvature and greater uncertainty, which means that in the current training stage, the effective learning signal carried by this direction is weak, or the contribution of the corresponding mode is relatively low. In order to prevent noise or unstable gradients from adversely affecting the overall optimization process, it is crucial to reduce the impact of such uncertain modes.

[0201] This paper combines a similarity-aware gradient alignment mechanism with an uncertainty-guided parameter estimation method to propose a principled optimization strategy for fusing multimodal and unimodal information. Specifically, similarity-aware alignment ensures the consistency of multimodal and unimodal gradient directions, thereby promoting consistent and meaningful parameter updates and avoiding training instability caused by intermodal conflicts. This alignment mechanism achieves two-way optimization: on the one hand, the multimodal representation enhances the expressive power of the unimodal representation by providing rich contextual information; on the other hand, the stability of the unimodality guides the multimodal fusion process, thereby improving the robustness of the overall system. Furthermore, uncertainty-guided parameter estimation, as an adaptive modulation mechanism, dynamically adjusts the contribution of each modal gradient in the model learning process based on the confidence level of each modal gradient. This mechanism effectively prevents the model from being dominated by unreliable modal features, promotes balanced learning between modalities, and further improves the model's generalization and training stability. In summary, the similarity-aware and uncertainty-guided parameter estimation algorithms proposed in this paper jointly promote an efficient multimodal fusion strategy. This strategy effectively captures the rich features of multimodal representation while taking into account the stability of unimodal features, thereby achieving an optimal balance between model expressiveness and robustness, and ultimately achieving efficient feature interaction and stable parameter optimization throughout the entire training process.

[0202] By adopting the technical solution provided by the present invention, it is helpful to utilize the multimodal data collected during the operation of complex industrial systems to achieve comprehensive perception and accurate modeling of system status, which can effectively improve the real-time monitoring capability of system operation status and improve the safety and stability of the production process.

[0203] To evaluate the model's performance, this embodiment uses the PRONTO benchmark dataset to conduct a case study of industrial fault diagnosis. Two-phase flow enters a riser through a horizontal pipe. The riser can be bypassed with a manual valve, directing the flow directly into the riser. An S-shaped section is located in the middle of the riser. At the top, a two-phase separator separates the liquid and gas, which are then transported to a three-phase separator. The water returns to a storage tank, while the air is discharged to the atmosphere. In this industrial scenario, three different modalities—process data, pressure data, and video data—are used to comprehensively represent the state of a complex industrial system. Each modality has a different sampling frequency and recorded variables. Process and pressure data are collected synchronously using fixed sensors, with each state recorded for 60 seconds to ensure precise time alignment. Video data is captured using a handheld device, with recording durations ranging from 30 to 55 seconds for each fault state. State-level alignment is performed only with the other modalities. To fully utilize the multimodal data, this embodiment of the present invention divides the dataset into one-second intervals. Each sample contains process data, high-frequency pressure data, and a corresponding one-second video clip. Specifically, long videos are first segmented into 1-second segments to match the temporal granularity of the sensor data. During sample construction, process and pressure data are arranged sequentially, and 1-second segments are randomly selected from video data associated with the same fault state to ensure state-level consistency with the corresponding process and pressure sensor data. This sample construction strategy is consistent with industrial data collection practices and meets the requirements of industrial fault diagnosis.

[0204] This experiment simulated different system conditions by adjusting valve openings, resulting in two primary fault types: 1) air leakage, introduced by gradually opening valve V10, allowing air to escape to the atmosphere; and 2) flow diversion, created by gradually opening bypass valve U39, causing some mixed flow to bypass the riser and enter the horizontal pipe before rejoining the riser. Both fault modes were tested under fixed operating conditions, with each fault type encompassing two severity levels. This resulted in five distinct system fault condition categories, along with the normal state.

[0205] This embodiment uses diagnostic accuracy, average precision of all fault categories, recall rate and F1 score as evaluation indicators to assess the effectiveness of each method in multimodal and single-modal fault diagnosis.

[0206] Figure 2 The adaptive weight coefficient ratio λ in the embodiment of the present invention is shown ω / λ m During the training phase, the STA module corresponding to each mode is analyzed, where the x-axis corresponds to the number of training iterations and the horizontal line in the figure is located at λ ω / λ m= 1, which is used to mark the position where the multimodal and unimodal gradient weights are equal. This figure shows that the algorithm proposed in this invention can achieve dynamic learning signal balance adjustment between multimodal representation and unimodal representation. At the beginning of training, the λ of all three modalities is ω / λ m All of them show significant volatility, reflecting that the algorithm actively adjusts the gradient amplitude of each mode to enhance the synergy between the representation ability of specific modes and the fusion representation. It should be pointed out that this volatility is not a manifestation of system instability, but a beneficial adjustment mechanism that reflects the model's ability to adapt to the inherent heterogeneity between multiple modalities, which is particularly critical for processing complex multimodal industrial environment data. As the training progresses, the λ corresponding to all modes ω / λ m The ratio tends to be stable and eventually converges to a value close to 1. This trend verifies that the similarity-aware gradient modulation mechanism adopted in the present invention has good convergence and robustness in multimodal fusion tasks.

[0207] Figure 3 The confusion matrix obtained by the fault state diagnosis of the embodiment of the present invention is shown, covering the diagnosis results of multimodal and three single-modal inputs (i.e., pressure, process, and video). It can be observed from the single-modal results shown in Figures (3b) to (3d) that different modalities can effectively capture the different dimensional characteristics of the fault mode: the pressure modality shows relatively stable performance in identifying specific types of faults; the process modality maintains consistency in the recognition accuracy of the main categories, reflecting a strong general discrimination ability; although the video modality is more sensitive to subtle state changes, it shows excellent results in identifying categories such as normal, diversion-1, and diversion-2, indicating that it has significant advantages in extracting visual features related to the operating state. The above results show that although single-modal input can provide valuable information related to the system state, it has limitations in information coverage. The multimodal fusion model proposed in the present invention can fully integrate the complementary information between the modalities and show higher accuracy and robustness in distinguishing the system fault state.

[0208] Figure 4This paper presents ablation experiment results for the proposed model structure, similarity-aware gradient alignment mechanism, and uncertainty-based parameter estimation algorithm. The vertical axis represents the fault state diagnosis rate. When only the basic model structure is retained and trained using conventional gradient descent, the overall model performance significantly deteriorates, indicating that the model cannot effectively adapt to the complex modality-specific characteristics when relying solely on the basic structure. Despite this, the accuracy of multimodal diagnosis is still significantly higher than that of each single modality, verifying that the proposed basic model structure has excellent multimodal information fusion capabilities and can integrate complementary information between different modalities to a certain extent. The introduction of the similarity-aware gradient modulation mechanism significantly improves the model's performance in all tasks. When only the uncertainty-guided parameter estimation mechanism is used, the model exhibits superior performance in multimodal and process mode diagnosis tasks, but without the cooperation of other modules, it is still insufficient to fully improve the performance of all modalities. The experimental results show that the proposed method can significantly improve the overall effectiveness of the model in complex multimodal fault diagnosis tasks, verifying its practicality and advanced nature.

[0209] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information, characterized in that: This method inputs the acquired multimodal data and fault status labels of the industrial system into a multimodal fusion fault diagnosis model based on the attention mechanism and outputs the fault diagnosis results. The multimodal fusion fault diagnosis model includes a modal alignment module, a cross-modal feature extraction module corresponding to each modality and a shared timing module, as well as a multimodal classification head and a single-modal classification head. The modal alignment module performs alignment preprocessing on the multimodal data to obtain the modal representation of each modality; the cross-modal cross-feature extraction module takes the modal representation of all modalities as input, realizes information interaction and complementary feature fusion between different modalities, and outputs the cross-enhanced features corresponding to each modality; the shared timing module takes the modal representation of the corresponding modality and the cross-enhanced features of all modalities as input, combines the self-attention mechanism to capture the dependency characteristics in the time dimension, and outputs multimodal feature representation and unimodal feature representation; the multimodal feature representation is input into the multimodal classification head to perform multimodal fault diagnosis and output the multimodal fault diagnosis result; the unimodal feature representation corresponding to each modality is respectively input into the corresponding unimodal classification head, and the corresponding unimodal fault diagnosis result is output; The multimodal fusion fault diagnosis model updates parameters by combining a similarity-aware gradient alignment mechanism with an uncertainty-guided parameter estimation method.

2. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: The modal alignment module performs the following operations: The M types of modal data collected from the operation of the industrial system are recorded as X (1) ,…,X (m) ,…,X (M) :in Input data for each mode, l (·) ,d (·) and N represent the sequence length, feature dimension, and number of samples, respectively. For the modal data corresponding to any modality m∈{1,2,…,M}, the modality alignment module extracts features and aligns the modal data through an encoder and a one-dimensional convolutional layer: in, represents the modal representation obtained after mode m is processed, d c To unify the feature dimension, Encoder (m) (·) represents the modality-specific encoder structure, whose parameters are Used to extract original features within the modality; Conv1D (m) is a one-dimensional convolution operation, and its parameters are Used to map modal features to a unified dimension.

3. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: For any modality m, its corresponding cross-modal feature extraction module uses the modality representation H of the modality (m) As a key value, by combining it with other modal m ′ The attention mechanism calculation between them realizes the active extraction of cross-modal complementary information, where m ′ ∈{1,2,…,M} and m ′ ≠m, specifically: The modal representation H of mode m (m′) Converted to query vector Q (m′ ), the representation H of mode m (m) Convert to key K (m) Sum V (m) : in, Represent the corresponding linear transformation weight matrix, and H (m′) and H ()) Convert to d u dimensional space; H (m) Convert to d v Dimensional space; Calculate the modality m from modality m through the attention mechanism ′ The cross-enhanced feature H obtained from (m′→m) : in, Indicates CATT (m′→m) The set of learnable parameters in CATT (m′→m) Indicates that from mode m ′ Cross-modal attention computation to modality m.

4. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: The shared timing module obtains multimodal feature representations through the following steps: All cross-enhanced features related to the target modality m are concatenated to form the aggregated cross-modal feature representation of modality m, which is denoted as: in, represents the multimodal aggregation feature of mode m, the superscript ω indicates that the feature belongs to the multimodal representation space, H (i→m) Cross-enhanced features obtained by modality i from modality m; Aggregate multimodal features Input to the corresponding mth shared timing module SelfAttn (m) (·), dynamically captures the dependency patterns at different time steps, and the calculation formula is as follows: Among them, Z (ω,m) represents the multimodal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

5. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: The shared timing module obtains a single-modal feature representation through the following steps: Modal representation H for mode m (m) Repeated stacking is performed to form an extended feature matrix for sequence feature modeling, which is expressed as: Expand the feature matrix Input to the corresponding mth shared timing module SelfAttn (m) (·) is processed, and the calculation process is as follows: Among them, Z (m) represents the single-modal feature representation of mode m after being processed by the shared timing module, θ (m) represents the learnable shared parameters of the mth shared timing module.

6. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: The multimodal fusion fault diagnosis model uses cross entropy loss to measure the difference between the model output and the true label. The total loss function of the model is Expressed as: Among them, M is the number of modalities, and the loss of the multimodal classification head is Defined as: Among them, C represents the number of classes, N represents the number of samples, Indicates the real fault status label, represents the predicted probability of the i-th class of the j-th sample output by the multimodal classification head; The loss of the unimodal classification head corresponding to modality m Defined as: in, represents the predicted probability of the i-th class of the j-th sample output by the unimodal classification head corresponding to modality m.

7. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 1, characterized in that: The parameters of the multimodal classification head, unimodal classification head and cross-modal cross feature extraction module are updated according to their corresponding gradients. The parameter gradient in the modal alignment module is calculated according to the total loss function of the multimodal fusion fault diagnosis model. The parameter gradient of the shared timing module is determined according to the losses corresponding to the multimodal classification head and the unimodal classification head, and gradient fusion is performed using an adaptive gradient modulation algorithm.

8. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 7, characterized in that: The loss of the multimodal classification head and the loss of the unimodal classification head The gradients relative to the shared parameters of the shared timing module are: and in, is the multimodal gradient, is a single-mode gradient, Represents the corresponding loss function for parameter θ (m) Gradient calculation of ; The adaptive gradient modulation algorithm performs the following steps: In the tth iteration of training, a mini-batch is randomly sampled from the training dataset, denoted as B t , the sample size is b, and the mini-batch gradient is recorded as Defined as Where (.)∈{ω,m} indicates that the gradient corresponds specifically to the multimodal branch or the unimodal branch, and the mini-batch gradient is used as the full gradient According to the central limit theorem, when the batch size b is larger than the preset value, the multimodal gradient and the unimodal gradient and Approximately regarded as obeying multivariate Gaussian distribution: in, and Respectively represent the covariance matrix of multimodal and unimodal stochastic gradients at the tth iteration, N (.) represents a normal distribution; At the t+1th iteration, a fusion gradient is constructed based on the multimodal gradient and the unimodal gradient to update the parameters of the mth shared timing module, which is recorded as Among them, c ω ,c m is the adaptive weighting parameter, c ω ,c m ≥0 and c ω +c m =1; Based on the Gaussian distribution assumption, the fusion gradient Modeled as: Among them, N (f) is the fusion distribution, is the mean vector of the fused gradient, is the covariance matrix of the fused gradient; To determine the optimal fusion gradient direction, minimize the fusion distribution N (f) With source distribution N (ω) 、N (m) The weighted sum of the KL divergence between , the specific optimization objectives are as follows: Among them, D KL represents the KL distance, λ ω and λ m It is a dynamic adjustment scale factor for similarity perception, which is used to adjust the contribution weight of the multimodal and unimodal loss gradients to the fusion gradient respectively; Based on the best fusion gradient direction Update the parameters of the mth shared timing module: Where η is the learning rate, Represents the parameters of the mth shared timing module at the tth iteration.

9. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 8, characterized in that: The dynamic adjustment scale factor is determined based on a similarity-aware gradient alignment mechanism, specifically: Let γ be the multimodal gradient and unimodal gradient The angle between them is used to calculate the gradient similarity: Let cos(α) be the target similarity between the two gradients. If the gradient similarity cos(γ)>cos(α), it means that the two types of gradient directions are fully aligned. Set λ ω =λ m =1, if the gradient similarity cos(γ)<cos(α), the similarity-aware gradient alignment mechanism is implemented, and the detailed steps are as follows: If the prediction result of the multimodal classification head is correct, but the prediction result of the unimodal classification head is incorrect, the unimodal gradient direction is corrected, and α is the fusion gradient and multimodal gradients The angle between the directions, the target similarity is: Gradient fusion is modeled based on geometric relationships and applies the law of sine to the triangle formed by the three gradient vectors: Further deduction yields: If the prediction result of the unimodal classification head is correct, but the prediction result of the multimodal classification head is incorrect, the multimodal gradient is adjusted to be consistent with the unimodal gradient, and α is the fusion gradient and unimodal gradient The angle between them is: Gradient fusion is modeled based on geometric relationships and applies the law of sine to the triangle formed by the three gradient vectors: Further deduction yields: If the prediction results of the unimodal classification head and the multimodal classification head are both correct or both wrong, set λ ω =λ m =1.

10. The industrial fault diagnosis method driven by complementary gain of multimodal and single-modal information according to claim 9, characterized in that: Based on the dynamic adjustment of the scaling factor, the optimization objective is expanded to: Pick Relative to The gradient of , and setting it to zero gives: Thus we get Closed-form solution of Experience Gradient and is used as an estimate of the true gradient mean, i.e. The inverse of the gradient covariance is approximated using the Hessian matrix of the loss function: Get the fused gradient The calculation formula is: The adaptive weighting parameters are:

Images