Rolling bearing fault diagnosis method and system based on acoustic-vibration multi-modal fusion

Abstract

The application discloses a kind of based on acoustic vibration multimodal fusion rolling bearing fault diagnosis method and system, it is related to bearing fault diagnosis field, including: using the fusion physical information's multimodal rolling bearing fault diagnosis model that is constructed and trained in advance to carry out fault diagnosis;The multimodal rolling bearing fault diagnosis model is used to obtain vibration characteristics, acoustic characteristics and physical consistency score according to sample;Vibration characteristics and acoustic characteristics are preliminarily grouped and fused, and multimodal preliminary fusion feature is obtained;Based on coordinate attention mechanism, multimodal preliminary fusion feature is enhanced in high order coordinate, and multimodal enhanced fusion feature is obtained;According to multimodal enhanced fusion feature and physical consistency score, predicted time series state sequence is obtained;According to predicted time series state sequence, predicted fault diagnosis result is obtained.The application effectively solves the problems of insufficient modal-specific feature extraction, fusion redundancy conflict and lack of physical constraints in time series modeling in multimodal fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of motor bearing fault diagnosis technology, and in particular to a method and system for rolling bearing fault diagnosis based on acoustic and vibration multimodal fusion. Background Technology

[0002] As a critical and fundamental component in rotating machinery, the operating condition of rolling bearings directly affects the safety and reliability of the entire equipment. Statistics show that approximately 30% of rotating machinery failures are related to bearings, and bearing failures can lead to catastrophic accidents and enormous economic losses. Therefore, achieving accurate fault diagnosis of rolling bearings is of significant practical importance for ensuring industrial production safety and improving equipment operating efficiency.

[0003] With the rapid development of Industrial Internet and Internet of Things (IoT) technologies, data-driven intelligent fault diagnosis methods have been widely applied. Traditional fault diagnosis methods mainly rely on single-modal vibration signal analysis, achieving fault classification through time-frequency domain feature extraction and pattern recognition. However, in actual industrial environments, single sensor signals are often susceptible to noise interference and changes in operating conditions, resulting in limited diagnostic accuracy. In recent years, multimodal information fusion technology has provided a new solution for fault diagnosis. In particular, the fusion of acoustic and vibration signals can fully utilize the complementary characteristics of the two modes: vibration signals can reflect the high-frequency impact characteristics inside the equipment, while acoustic signals are more sensitive to low-frequency radiation characteristics. Studies have shown that acoustic-vibration fusion can significantly improve the accuracy and robustness of fault diagnosis.

[0004] Despite some progress in multimodal fault diagnosis methods, current research still faces three key challenges. First, in the feature extraction stage, existing methods often neglect the modal-specific differences between high-frequency vibration signals and low-frequency acoustic radiation, leading to inaccurate feature extraction. For example, subspace learning and Grassmann manifold learning methods do not fully consider the physical differences in modal signals, and are still insufficient in the differentiated extraction of high-frequency vibration and low-frequency acoustic radiation features. Second, in the multimodal fusion stage, traditional fusion methods are prone to feature redundancy and cross-modal conflicts, affecting the fusion effect. While the Convolutional and Cross-Fusion Transformer (CCFT) method enhances intermodal correlations, its parallel computational characteristics result in weak modeling capabilities for temporal dynamic features. Finally, in the temporal modeling stage, while existing deep learning models such as CNN, LSTM, and Transformer can learn data features, they lack physical constraints, making it difficult to accurately characterize the dynamic evolution of faults, thus limiting the model's generalization ability and interpretability. For example, the Physical Information Fusion Constraint Guided (PFCG) method uses a Transformer backbone network, which is weaker than LSTM in capturing local impulse features of time-series signals, and it does not design physical constraint mechanisms for the novel memory structure of xLSTM.

[0005] Therefore, some studies have begun to explore the integration of physical information and deep learning. Physically constrained loss embedding methods, such as Physical Information Neural Networks (PINN) and Deep Physical Information Consistent Embedding Networks (DPICEN), guide model learning by using the residuals of physical equations as loss terms. However, the former is difficult to directly transfer to the acoustic-vibration coupling scenario of rolling bearings, while the latter uses a static physical encoding method, which cannot adapt to the dynamic changes of physical parameters during fault evolution. Physical-data hybrid modeling methods, such as Inverse Physical Information Neural Networks and Physical Information Residual Networks (PIResNet), achieve complementarity between mechanism and data by constructing physical and data branches. However, the former relies on an accurate dynamic model, and the residual network used in the latter has weaker long-range dependency modeling ability for time-series signals than LSTM-like structures. Existing methods in the acoustic-vibration fusion fault diagnosis of rolling bearings generally suffer from insufficient extraction of modal-specific features, prominent redundancy and heterogeneity conflicts in the fusion stage, and limited generalization ability due to the lack of physical constraints in time-series modeling. Summary of the Invention

[0006] To address the technical problems existing in the background art, this invention proposes a rolling bearing fault diagnosis method and system based on acoustic and vibration multimodal fusion.

[0007] In a first aspect, the present invention proposes a rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion, comprising:

[0008] Construct a training set, which includes multiple samples, including vibration signals, acoustic signals, tachometer signals, bearing parameters, and fault labels; construct a multimodal rolling bearing fault diagnosis model that integrates physical information; train the multimodal rolling bearing fault diagnosis model using the training set, and use the trained multimodal rolling bearing fault diagnosis model to diagnose the faults of the rolling bearing under test;

[0009] The multimodal rolling bearing fault diagnosis model includes: a physical constraint feature extraction module, used to extract vibration and acoustic features in parallel to obtain vibration and acoustic features, and obtain a physical consistency score based on tachometer signals and bearing parameters; a two-stage collaborative grouped attention multimodal fusion module, used to obtain preliminary multimodal fusion features based on vibration and acoustic features; a channel attention module, used to generate channel attention based on coordinate attention mechanism and preliminary multimodal fusion features, and obtain enhanced multimodal fusion features based on preliminary multimodal fusion features and channel attention; a physically guided hybrid enhanced LSTM temporal modeling module, used to obtain a predicted temporal state sequence based on enhanced multimodal fusion features and physical consistency score; and a classification module, used to obtain a predicted fault diagnosis result based on the predicted temporal state sequence.

[0010] Preferably, the physical constraint feature extraction module includes parallel vibration signal extraction branches and acoustic signal extraction branches, both of which are lightweight convolutional networks embedded with physical residual constraints based on the second-order discrete wave equation.

[0011] Preferably, the process of generating the preliminary multimodal fusion features includes: concatenating the parallel input vibration features and acoustic features, and grouping the concatenated features to obtain multiple grouped features; the local attention branch extracts the local correlation of each grouped feature through grouped convolution to obtain the first weight of each grouped feature; the global attention branch extracts the global context of each group of features through global average pooling and grouped convolution to obtain the second weight of each group; the first and second weights of each group are activated separately by activation functions and then fused to obtain the third weight of each group; and the preliminary multimodal fusion features are generated by weighting the grouped features and their third weights through a weighted residual method.

[0012] Preferably, the process of generating channel attention weights includes: embedding coordinate information into the primary fusion feature through adaptive max pooling, and then compressing the channel dimension of the primary fusion feature with embedded coordinate information through a convolutional layer, a batch normalization layer, and a nonlinear activation unit based on an adaptive switching mechanism to generate channel attention weights; wherein, the nonlinear activation unit is: In the formula, The output of the nonlinear activation unit, The input features are those of the nonlinear activation unit. and For learnable scaling parameters, For the Sigmoid function, A switching factor to control the degree of nonlinearity of the activation function.

[0013] Preferably, the physics-guided hybrid enhanced LSTM time series modeling module includes a hybrid physics-data modeling layer and a dynamic fusion layer. The hybrid physics-data modeling layer includes a physical mechanism branch and a data-driven branch. The physical mechanism branch is used to obtain an ideal evolution sequence based on multimodal enhanced fusion features and a preset bearing failure mechanism. The data-driven branch is used to obtain a data-driven branch state sequence based on multimodal enhanced fusion features and physical consistency scores. The dynamic fusion layer is used to dynamically fuse the ideal evolution sequence and the data-driven branch state sequence through an adaptive gating factor to obtain a predicted state sequence.

[0014] Preferably, the data-driven branch adopts a frequency-domain decoupled matrix-scalar cascaded interactive topology. The matrix-scalar cascaded interactive topology includes a hierarchical bottom layer and a high layer. The bottom layer includes a matrix-type memory update unit with group aggregation, and the high layer includes a physically guided exponential scalar update unit. A bypass connection is also established between the exponential scalar update unit and the matrix memory update unit based on a memory-gated injection mechanism.

[0015] The matrix-type memory update unit projects multimodal enhanced fusion features onto multiple orthogonal subspaces according to a preset grouping and aggregation strategy. It independently maintains a memory matrix in each subspace and accumulates global context information to update the memory matrix through a covariance update rule. Based on the updated memory matrix, it obtains an intermediate hidden state sequence and extracts the energy state descriptor of the memory matrix through the Frobenius norm. The bypass connection injects the energy state descriptor into the exponential input gate of the exponential scalar update unit. The exponential scalar update unit uses an exponential domain gating mechanism to perform exponential domain gating updates by combining the intermediate hidden state sequence, energy state descriptor, and physical consistency score. It outputs the final intermediate hidden state sequence and uses it as a data-driven branch state sequence.

[0016] Preferably, the physically-guided hybrid enhanced LSTM time series modeling module further includes a correction layer, which is used to calculate the minimum regularized path distance between the predicted state sequence and the ideal evolution sequence using a preset physical-data consistency checker. Furthermore, a regularization constraint function is constructed based on the minimum regularized path distance to inversely constrain the parameter updates of the branches.

[0017] Preferably, the classification module includes a fully connected layer, which is used to map the predicted state sequence to the fault diagnosis results of the rolling bearing under test.

[0018] Preferably, the training process uses a total loss function to optimize the parameters of the multimodal rolling bearing fault diagnosis model; wherein, the total loss function includes a fault diagnosis task loss function and a multi-physical constraint loss function; wherein, the fault diagnosis task loss function includes the classification task loss function of the classification module, the feature extraction task loss function of the physical constraint feature extraction module, the fusion task loss function of the group attention multimodal fusion module based on two-stage collaboration, and the temporal consistency loss function of the physically guided hybrid enhanced LSTM temporal modeling module;

[0019] The multi-physical constraint loss function includes the physical residual constraint function of the physical constraint feature extraction module and the regularization constraint function of the physical-guided hybrid enhanced LSTM time series modeling module.

[0020] Secondly, this invention also proposes a rolling bearing fault diagnosis system based on acoustic and vibration multimodal fusion, comprising: a training set construction module for constructing a training set; a model construction module for constructing a multimodal rolling bearing fault diagnosis model that integrates physical information; a training module for training the multimodal rolling bearing fault diagnosis model using the training set to obtain a trained multimodal rolling bearing fault diagnosis model; and a diagnosis module for using the trained multimodal rolling bearing fault diagnosis model to perform fault diagnosis on the rolling bearing under test.

[0021] The multimodal rolling bearing fault diagnosis model includes: a physical constraint feature extraction module, used to extract vibration and acoustic features in parallel to obtain vibration and acoustic features, and obtain a physical consistency score based on tachometer signals and bearing parameters; a two-stage collaborative grouped attention multimodal fusion module, used to obtain preliminary multimodal fusion features based on vibration and acoustic features; a channel attention module, used to generate channel attention based on coordinate attention mechanism and preliminary multimodal fusion features, and obtain enhanced multimodal fusion features based on preliminary multimodal fusion features and channel attention; a physically guided hybrid enhanced LSTM temporal modeling module, used to obtain a predicted temporal state sequence based on enhanced multimodal fusion features and physical consistency score; and a classification module, used to obtain a predicted fault diagnosis result based on the predicted temporal state sequence.

[0022] The proposed method and system for rolling bearing fault diagnosis based on acoustic-vibration multimodal fusion in this invention constructs a multimodal rolling bearing fault diagnosis model that integrates physical information. A physical constraint feature extraction module performs parallel feature extraction on vibration and acoustic signals to obtain vibration and acoustic features, and obtains a physical consistency score based on tachometer signals and bearing parameters. A two-stage collaborative grouped attention multimodal fusion module obtains preliminary multimodal fusion features based on vibration and acoustic features. Channel attention is generated based on coordinate attention mechanisms and the preliminary multimodal fusion features, and enhanced multimodal fusion features are obtained based on the preliminary multimodal fusion features and channel attention. A physically guided hybrid enhanced LSTM temporal modeling module obtains a predicted temporal state sequence based on the enhanced multimodal fusion features and physical consistency score. A classification module obtains the predicted fault diagnosis result based on the predicted temporal state sequence. This effectively solves the problems of insufficient modality-specific feature extraction, fusion redundancy conflicts, and lack of physical constraints in temporal modeling during multimodal fault diagnosis. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion in one embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the physical constraint feature extraction module in one embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of the preliminary fusion unit in one embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of a physically guided hybrid enhanced LSTM timing modeling module in one embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of a grouping aggregation strategy in one embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of the structure of an exponential scalar update unit in one embodiment of the present invention.

[0029] Figure 7 This is a visual diagram illustrating the results of fault diagnosis using a multimodal rolling bearing fault diagnosis model that integrates physical information, as proposed in one embodiment of the present invention.

[0030] Figure 8 This is a visual representation of the results of fault diagnosis using an existing CNN model.

[0031] Figure 9This is a visualization of the results of fault diagnosis using an existing LSTM model.

[0032] Figure 10 This is a visual representation of the results of fault diagnosis using the existing Transformer model. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Firstly, referring to Figures 1-6 The present invention proposes a rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion, comprising:

[0035] Construct a training set; construct a multimodal rolling bearing fault diagnosis model that integrates physical information; train the multimodal rolling bearing fault diagnosis model using the training set to obtain a trained multimodal rolling bearing fault diagnosis model; use the trained multimodal rolling bearing fault diagnosis model to diagnose the faults of the rolling bearing under test.

[0036] This invention effectively solves the problems of insufficient extraction of modal-specific features, lack of physical constraints in the fusion of redundant conflicts, and lack of physical constraints in the timing modeling of multimodal rolling bearings by constructing a multimodal rolling bearing fault diagnosis model that integrates physical information.

[0037] The training set in this embodiment includes multiple samples, each of which includes vibration signals, acoustic signals, tachometer signals, bearing parameters, and fault labels. The fault labels are categorized as follows: normal state, inner ring fault with a crack size of 0.3 mm, inner ring fault with a crack size of 1.0 mm, outer ring fault with a crack size of 0.3 mm, and outer ring fault with a crack size of 1.0 mm.

[0038] In this embodiment, the multimodal rolling bearing fault diagnosis model includes a Physical Constraint Feature Extraction (PCFE-MS) module, a Grouped Attention Multimodal Fusion (GAMF-TSC) module based on two-stage collaboration, a Physically Guided Hybrid Enhanced LSTM Temporal Modeling (PxLSTM-HM) module, and a classification module connected in sequence.

[0039] The module comprises several sub-modules: a Physically Constrained Feature Extraction (PCFE-MS) module, which extracts vibration and acoustic signals from the rolling bearing under test in parallel to obtain vibration and acoustic features, ensuring physical consistency of the features while preserving high-frequency impact characteristics in the generated vibration features and low-frequency radiation characteristics in the generated acoustic features; a Grouped Attention Multimodal Fusion (GAMF-TSC) module based on two-stage collaborative mechanisms, which generates preliminary multimodal fusion features based on the vibration and acoustic features; a channel attention mechanism based on the coordinate attention mechanism and the preliminary multimodal fusion features; and a multimodal enhanced fusion feature based on the preliminary multimodal fusion features and the channel attention; a Physically Guided Hybrid Enhanced LSTM Temporal Modeling (PxLSTM-HM) module, which obtains a predicted temporal state sequence based on the enhanced multimodal fusion features; and a classification module, which obtains the fault diagnosis results of the rolling bearing under test based on the predicted temporal state sequence.

[0040] This embodiment constructs a multimodal rolling bearing fault diagnosis model that integrates physical information. A physical constraint feature extraction module performs parallel feature extraction on vibration and acoustic signals to obtain vibration and acoustic features. Based on the tachometer signal and bearing parameters, a physical consistency score is obtained. A two-stage collaborative grouped attention multimodal fusion module performs initial grouping and fusion of vibration and acoustic features to obtain preliminary multimodal fusion features. Then, a coordinate attention mechanism is used to enhance the preliminary multimodal fusion features with higher-order coordinates, resulting in enhanced multimodal fusion features. A physically guided hybrid enhanced LSTM temporal modeling module obtains a predicted temporal state sequence based on the enhanced multimodal fusion features and the physical consistency score. A classification module then uses the predicted temporal state sequence to obtain the predicted fault diagnosis result. This effectively solves the problems of insufficient modality-specific feature extraction, fusion redundancy conflicts, and lack of physical constraints in temporal modeling during multimodal fault diagnosis. Furthermore, the fully embedded physical information ensures that the features learned by the multimodal rolling bearing fault diagnosis model are closely related to the physical nature of the fault, significantly improving the model's generalization ability and robustness under noise interference and varying operating conditions.

[0041] In a further embodiment, the physical constraint feature extraction module includes a parallel vibration signal extraction branch and an acoustic signal extraction branch, which are used to perform parallel feature extraction on the vibration signal and acoustic signal to obtain vibration features and acoustic features.

[0042] Specifically, the vibration signal extraction branch and the acoustic signal extraction branch are used to simultaneously receive the original vibration sequence data and acoustic sequence data, and output the vibration features and acoustic features synchronously to the grouped attention multimodal fusion module based on two-stage collaboration.

[0043] The training set in this embodiment also includes tachometer signals and bearing geometric parameters. In addition to extracting vibration and acoustic features, the physical constraint feature extraction module in this embodiment calculates the theoretical fault frequency based on the input rotational speed and bearing geometric parameters, and compares it with the envelope spectrum of the actual signal to calculate a physical consistency score. The lower the physical consistency score, the more the feature conforms to physical laws.

[0044] While outputting vibration and acoustic characteristics, the PCFE-MS module simultaneously outputs the physical consistency score. The physical consistency score As an auxiliary variable, the physical consistency score is passed to the backend physically-guided hybrid enhanced LSTM timing modeling module. As a gating factor: when physical consistency is high, the gating mechanism allows more information to pass through; otherwise, it inhibits it, thereby achieving physical guidance of information flow control.

[0045] In this embodiment, both the vibration signal extraction branch and the acoustic signal extraction branch are lightweight convolutional networks with embedded physical constraints. The lightweight convolutional network model is a lightweight two-layer one-dimensional convolutional network.

[0046] In practical implementation, a lightweight convolutional network model is used to extract multi-channel vibration or acoustic features from single-channel vibration or acoustic signals, and physical constraints are explicitly embedded in the learning process of both vibration and acoustic features. This embodiment guides feature learning to conform to the laws of mechanical vibration and sound wave propagation through these physical constraints, while preserving the high-frequency impact characteristics of the vibration signal and the low-frequency radiation characteristics of the acoustic signal. This forces the extracted feature manifold to approximate the dynamic trajectory of an ideal physical system, achieving differentiated and accurate extraction of dual-modal features, thereby realizing physical interpretability in the data-driven model.

[0047] In one specific embodiment, the physical constraint is a physical residual constraint based on the second-order discrete wave equation, which is applied to the single-mode signal and constrains the feature extraction process to conform to the wave equation.

[0048] like Figure 2 As shown, in this embodiment, in order to guide the lightweight convolutional network to extract features that conform to the propagation law of mechanical waves, a physical feature residual loss function based on the discrete second-order wave equation is introduced during the training process of the physical constraint feature extraction module. Since the feature sequences extracted by the neural network in the initial learning stage often deviate from the ideal physical wave law, this embodiment calculates the difference between the actual feature output of the network and the ideal second-order wave equation, i.e., the physical residual, and minimizes it as a regularization term, thereby forcibly constraining the feature extraction direction of the network. Specifically, for the feature signal output at time step t... Its discrete form of the second-order time derivative The approximate calculation using the central difference scheme is as follows: ;

[0049] Then, the physical residual of the characteristic signal deviating from the ideal fluctuation law at time step t is defined. for: In the formula, These are the learnable wave speed-related physical parameters in lightweight convolutional network models. These are the learnable frequency-related physical parameters in the model. The physical feature residual loss function generated by the physical constraint feature extraction module. Defined as the mean square error of the physical residuals over all time steps: In the formula, N is the total number of time steps for the feature sequence. Finally, the total loss function during the training process of the physical constraint feature extraction module... The weighted sum of the fault diagnosis task loss function and the physical characteristic residual loss function is calculated using the following formula: In the formula, This is the task loss function used to measure the error of the physical constraint feature extraction module; The hyperparameter weighting coefficients are used to balance the strength of physical constraints.

[0050] In this embodiment, the grouped attention multimodal fusion module based on two-stage collaboration includes a primary fusion unit and a higher-order enhancement unit;

[0051] The primary fusion unit is used to initially fuse vibration features and acoustic features based on group attention to obtain preliminary multimodal fusion features;

[0052] Higher-order enhancement units are used to obtain channel attention weights based on coordinate attention mechanism and multimodal preliminary fusion features, and then use the channel attention weights to weight the multimodal preliminary fusion features to obtain multimodal enhanced fusion features.

[0053] like Figure 3 As shown, in a further embodiment, vibration features and acoustic features are initially fused based on grouped attention to obtain multimodal preliminary fused features. Specifically, this includes: concatenating the parallel-input vibration features and acoustic features; grouping the concatenated vibration features and acoustic features to obtain multiple grouped features; and using local attention branches... Group convolution extracts the local correlation of each group feature to obtain the first weight of each group feature; the global attention branch extracts the global context of each group feature through global average pooling and group convolution to obtain the second weight of each group; the first and second weights of each group are activated by activation function and then fused to obtain the third weight of each group; the group features and their third weights are weighted residual to generate a preliminary multimodal fusion feature.

[0054] It is important to note that the primary fusion unit incorporates physical constraints to ensure the physical rationality of the fusion. The physical constraints of the primary fusion unit (GAMF-TSC) act on the multimodal joint feature space. They constrain the synergistic consistency of vibration and acoustic signals during fusion, meaning both should point to the same physical fault source. This is achieved by calculating the cross-correlation or consistency metric between the vibration and acoustic feature distributions and comparing it with the theoretical fault period.

[0055] The third weight is specifically as follows: In the formula, As an activation function, it maps the values ​​within the parentheses to the interval (0, 1), making them a probabilistic gate value; The output of the local attention branch is generated by the branch at the top of the diagram and extracted through Conv1d (one-dimensional convolution), focusing on capturing the local correlation of features within a short distance. The output of the global attention branch is extracted through global average pooling and convolution, focusing on capturing the overall global context of features; + indicates element-wise addition, realizing the dual-stage collaborative fusion of local and global perspectives.

[0056] This embodiment fuses contextual information from both local and global perspectives and transforms it into gating coefficients between 0 and 1 through an activation function.

[0057] The specific multimodal preliminary fusion features in this embodiment are as follows: In the formula, The main input representing the current feature group, such as a vibration signal; Indicates complementary features, such as sound signals; As the third weight, it is used as a gate for control. Information throughput; These are complementary weights used to control... Information throughput ensures overall energy balance; This is the Hadamard product, i.e., element-wise multiplication, which weights each channel of the feature vector individually; the number 2 is the scaling factor. and The average expectation is usually 0.5. In order to prevent the amplitude (energy) of the signal from decaying after weighted fusion, it is multiplied by 2 for rescaling. This represents the initial fusion characteristics of multimodal modes.

[0058] This embodiment first calculates the importance of features (third weight) through local attention branches and global attention branches. Then, this third weight is used to dynamically weight the feature and its complementary parts, thereby generating a fusion feature with clear physical meaning and strong noise resistance.

[0059] In a further embodiment, channel attention weights are obtained based on coordinate attention mechanism and multimodal preliminary fusion features. Specifically, this includes: embedding coordinate information into the primary fusion features through adaptive max pooling, and then compressing the channel dimension of the primary fusion features with embedded coordinate information through a convolutional layer, a batch normalization layer, and a nonlinear activation unit based on an adaptive switching mechanism to generate channel attention weights.

[0060] In this embodiment, the nonlinear activation unit dynamically adjusts the linear and nonlinear ratios of the activation function by learning parameters, specifically: In the formula, The output characteristics of the nonlinear activation unit, The input features are those of the nonlinear activation unit. and For learnable scaling parameters, This is the Sigmoid function, used to map the switching factor to the (0,1) interval to control the activation mode; A switching factor to control the nonlinearity of the activation function.

[0061] For the specific scenario of "acoustic-vibration multimodal fusion", this embodiment has made structural improvements to the nonlinear activation unit, as shown in the expression. as well as , It is not a fixed constant, but rather based on the input characteristics after the acoustic and vibration signals are fused. It performs dynamic adaptive adjustment. This design enables the model to automatically switch between "linear smoothing" and "nonlinear enhancement" during the diagnostic process based on the strength of the input signal, in order to capture subtle early fault characteristics. Therefore, through this nonlinear activation unit based on an adaptive switching mechanism, the model in this embodiment can adaptively amplify the characteristic response of the fault impact region while suppressing gradient propagation in the background noise region.

[0062] In a further embodiment, the channel attention weights are used to weight the initial multimodal fusion features to obtain the enhanced multimodal fusion features. Specifically, the channel attention weights are multiplied by the initial fusion features channel by channel to obtain the enhanced multimodal fusion features, so as to further refine the initial fusion features, enhance the key failure modes and suppress redundant noise, and obtain the enhanced multimodal fusion features.

[0063] It is important to understand that the loss function of the group attention multimodal fusion module based on two-stage collaboration is the fusion physical residual. The fusion physical residual integrates all physical inconsistencies in the primary fusion stage (group attention) and the higher-order enhancement stage (coordinate attention), that is, it simultaneously constrains the alignment error between modes and the deviation of the fusion features from the physical laws.

[0064] In some embodiments, the physics-guided hybrid enhanced LSTM time series modeling module includes a hybrid physics-data modeling layer and a dynamic fusion layer. The hybrid physics-data modeling layer includes a physical mechanism branch and a data-driven branch. The physical mechanism branch is used to obtain an ideal evolution sequence based on multimodal enhanced fusion features and a preset bearing failure mechanism. The data-driven branch is used to obtain a data-driven branch state sequence based on multimodal enhanced fusion features and physical consistency scores. The dynamic fusion layer is used to dynamically fuse the ideal evolution sequence and the data-driven branch state sequence through an adaptive gating factor to obtain a predicted state sequence.

[0065] In this embodiment, a physical mechanism branch and a data-driven branch are designed in the hybrid physical-data modeling layer. The physical mechanism branch generates an ideal evolution sequence in real time, and the information flow of the data-driven branch is directly driven by a gating mechanism. The physical mechanism is directly embedded into the neural network as a reference system, rather than relying solely on an external loss function. The ideal evolution sequence and the state sequence of the data-driven branch are dynamically fused by an adaptive gating factor to obtain an accurate predicted state sequence.

[0066] It's important to understand that in this embodiment, the hybrid physics-data modeling layer runs two independent branches in parallel at each time step. The physics mechanism branch is an auxiliary branch, primarily consisting of a lightweight standard LST unit. Its input is a theoretical feature sequence constructed based on a preset bearing fault mechanism, such as a theoretical fault frequency sine wave calculated from the rotational speed and bearing parameters, or an ideal decaying pulse sequence. Its main function is to simulate the system behavior under ideal noise-free and interference-free conditions, generating an ideal evolution sequence. The data-driven branch, on the other hand, is the main branch. Its input comes from the multimodal enhanced fusion features of sensors, and its main function is to capture the complex nonlinear and time-varying features in the actual signal.

[0067] In a further embodiment, the data-driven branch employs a frequency-domain decoupled matrix-scalar (MS) cascaded interactive topology.

[0068] The matrix-scalar cascaded interactive topology differs from the traditional single sequential stacking; instead, it constructs a deep coupling mechanism between the underlying matrix memory and the high-level scalar gating.

[0069] Among them, the matrix-scalar cascaded interactive topology is connected in a hierarchical manner. The bottom layer includes a matrix memory update unit (mLSTM) with grouped aggregation, and the top layer includes a physically guided exponential scalar update unit (sLSTM). The exponential scalar update unit and the matrix memory update unit are also connected by a bypass connection based on a memory-gated injection mechanism.

[0070] The matrix-type memory update unit projects multimodal enhanced fusion features onto multiple orthogonal subspaces according to a preset grouping and aggregation strategy. It independently maintains a memory matrix within each subspace and updates the memory matrix by accumulating global context information through a covariance update rule. Based on the updated memory matrix, it obtains the intermediate hidden state sequence. ; and extract the energy state descriptor of the memory matrix using the Frobenius norm. ;

[0071] The bypass connection is used to inject the energy state descriptor into the exponential input gate of the exponential scalar update unit;

[0072] Exponential scalar update units are used to employ an exponential field gating mechanism, combined with an intermediate hidden state sequence. Energy state descriptor Physical consistency score Perform exponential domain gating updates and output the final intermediate hidden state sequence. and use it as a data-driven branch state sequence. .

[0073] In practice, the multimodal enhanced fusion features are first input into the bottom-layer mLSTM. A grouping and aggregation strategy is adopted, and the matrix-form memory states are used to capture the complex nonlinear deviations of the actual system caused by wear and other factors in the high-dimensional space. The intermediate hidden state sequence is then output to the top layer, along with the extracted global energy state descriptor. This is then injected into the exponential input gate of the top-level exponential scalar update unit via a bypass connection; the gating mechanism of the top-level sLSTM then begins to operate, not only receiving the intermediate hidden state sequence transmitted from the lower layer but also simultaneously "injecting" the energy state descriptor from the lower layer. And utilize the physical consistency score from the outgoing route. The sLSTM acts as a "regulator" (i.e., a physical gate) to filter information; after a physically constrained exponential domain-gated update, the final intermediate hidden state sequence output by the sLSTM serves as the data-driven branch state sequence. .

[0074] In this embodiment, the underlying layer captures the long-term dependence and low-frequency degradation trend in the acoustic vibration signal through a matrix-type memory update unit, and outputs the intermediate hidden state sequence. The energy state descriptor is transmitted to the top layer and will serve as a quantification indicator of the global background intensity. This method utilizes a memory-gated injection mechanism to transmit information to higher layers, dynamically adjusting the upper layers' sensitivity to transient impacts based on the global background intensity. This overcomes the deficiency of "global information being truncated by local gating" in traditional serial stacking, achieving adaptive guidance of local features by global trends. Higher layers sharpen high-frequency fault impacts within the global background through exponential scalar update units. This embodiment solves the problem of forgetting multimodal features under complex operating conditions through this interlayer energy injection mechanism, achieving cross-layer information enhancement.

[0075] like Figure 4 As shown, in a standard LSTM, In the formula, For time steps, The cell state at time step t. For the Gate of Oblivion The cell state at time step t-1. For input gate, For candidate cell states at time step t, For time steps The hidden state; For time step The output gate; The activation function is the Tanh function (hyperbolic tangent function).

[0076] In this embodiment, In the formula, It is the Frobenius norm.

[0077] In the exponential scalar update unit of this embodiment, In the formula, It is an exponential function; The linear projection value of the input gate; The weights are a learnable parameter matrix used to adjust the underlying energy state descriptor. The extent of the impact on high-rise entrance doors; For time step The normalized stable state, used for the state variables of numerical stability, due to the exponential function It can easily produce extremely large values ​​(gradient explosion), using Store the maximum value of the historical input for subtraction shifting in exponential operations (LogSumExp trick) to ensure numerical stability of the calculation process; The physical consistency score, ranging from 0 to 1, reflects the degree to which the signal at the current moment conforms to the physical fault mechanism (such as wave equation, fault frequency). It is used as a physical gating factor and directly multiplied at the end of the formula. When (in accordance with physical laws), the input gate works normally; when When there is poor physical consistency, such as noise, the input gate is forcibly closed to prevent invalid information from being written into memory.

[0078] With this configuration, this embodiment ensures that high-intensity feature updates can only be activated when they conform to physical laws and deviate from the global background trend, thereby achieving precise identification of weak fault features.

[0079] In this embodiment, the physical-guided exponential scalar update unit simultaneously employs multi-scale physical constraints, applying wave equation residual constraints at both the original scale and the downsampling scale to further enhance feature physical consistency.

[0080] It should be understood that the data-driven branching generates a sequence of data-driven branch states. Ideal evolutionary sequences generated by branches of physical mechanisms Alignment is performed within the same feature space. Instead of being directly used as output, the data is transformed into physically consistent weights or biases through a linear mapping layer. These weights are then connected to the input or forget gates of the data-driven branch. When the features of the data-driven branch align with the trend of the physically ideal sequence, the gate opens, allowing information to pass through; when they deviate (i.e., non-physical noise interference occurs), the gate closes, suppressing the feature.

[0081] The predicted state sequence in this embodiment is as follows:

[0082]

[0083] in, As an adaptive gating factor, The sigmoid function is used to map attention weights to the (0,1) interval to generate gating coefficients; For the gated weight matrix, This indicates a feature concatenation operation. For an ideal evolutionary sequence, For data-driven branch state sequences, This is the gated bias term, which is related to the gated weight matrix. The learnable bias parameters used in conjunction with this function are used to shift the threshold of the activation function.

[0084] The structure of the physically guided hybrid enhanced LSTM temporal modeling module in this embodiment realizes the complementary fusion of the mechanism-based ideal model and the residual compensation model based on high-dimensional matrix memory.

[0085] In a further embodiment, the physics-guided hybrid enhanced LSTM time series modeling module also includes a calibration layer. The outputs of both the data-driven branch and the physical mechanism branch are connected to the calibration layer. The calibration layer is used during training to calculate the minimum regularized path distance between the predicted state sequence and the ideal evolution sequence using a preset physics-data consistency checker. And based on the minimum regularized path distance A regularization constraint function is constructed to inversely constrain the parameter updates of the data-driven branches. When the predicted state sequence deviates from the preset target, the parameters of the physically guided hybrid enhanced LSTM temporal modeling module are adjusted until the target is met, thereby forcing the data-driven feature trajectory to remain synchronized with the physical mechanism in time phase and eliminating the time drift problem that may be caused by multi-layer stacking.

[0086] Specifically, the physical-data consistency checker employs a dynamic time warping mechanism to mitigate timing misalignment issues. Among these, In the formula, To minimize the regular path distance, A normalized path refers to the optimal index-matching path found during DTW computation to align two sequences; it consists of a set of index pairs. The set of represents the set of the predicted sequence. The nth moment should correspond to the nth time in the ideal sequence. At this moment: These are the time step indices in the normalized path, representing the time point indices in the two sequences (predicted sequence and ideal sequence), respectively. Indicates the best alignment path In the predicted sequence, the first The time of the ideal sequence At which point they are determined to be a match; Let i be the predicted state sequence at time i. For the first The ideal evolutionary sequence at any given moment; The square of the Euclidean distance is used to calculate the numerical difference between two eigenvectors after alignment.

[0087] In this embodiment, the DTW distance between the predicted hidden state sequence and the ideal hidden state sequence generated by the physical branch is calculated by the correction layer. This distance is used as part of the regularization constraint function to constrain the feature trajectory learned by the data-driven branch to conform to the physical evolution law, and finally outputs accurate fault diagnosis results.

[0088] What needs to be understood is that Figure 4 The enhanced LSTM blocks in this embodiment refer to the matrix-type memory update unit and the exponential scalar update unit. The "enhancement" is mainly reflected in four aspects: first, upgrading scalar memory to covariance matrix memory to capture global trends; second, employing exponential gating to sharpen local shocks; third, introducing inter-layer energy injection to prevent information forgetting; and fourth, deeply embedding grouping aggregation strategies and physical consistency gating (including backend DTW alignment), thereby achieving strong constraints on physical mechanisms within the data-driven model.

[0089] In this embodiment, the classification module includes a fully connected layer, which is used to map the predicted state sequence to the fault diagnosis results of the rolling bearing under test.

[0090] In this embodiment, the training process of the multimodal rolling bearing fault diagnosis model includes: inputting samples from the training set into the multimodal rolling bearing fault diagnosis model to obtain predicted diagnosis results; calculating the total loss function based on the predicted diagnosis results; optimizing the parameters of the multimodal rolling bearing fault diagnosis model based on the total loss function until the model meets the preset convergence conditions; and using the parameters that meet the preset convergence conditions as the parameters of the multimodal rolling bearing fault diagnosis model to obtain the trained multimodal rolling bearing fault diagnosis model.

[0091] The total loss function includes the fault diagnosis task loss function and the multi-physical constraint loss function. The fault diagnosis task loss function includes the classification task loss function of the classification module, the feature extraction task loss function of the physical constraint feature extraction module, the fusion task loss function of the group attention multimodal fusion module based on two-stage collaboration, and the temporal consistency loss function of the physically guided hybrid enhanced LSTM temporal modeling module.

[0092] The classification loss uses cross-entropy loss.

[0093] The multi-physics constraint loss function includes the physical residual constraint function of the physical constraint feature extraction module and the regularization constraint function of the physical-guided hybrid enhanced LSTM temporal modeling module. The fusion task loss function includes the preliminary fusion physical residual loss and the higher-order enhanced physical residual loss.

[0094] In this embodiment, during the optimization of the parameters of the multimodal rolling bearing fault diagnosis model based on the total loss function, the fault diagnosis task loss function and the multi-physical constraint loss function are simultaneously transmitted to the physical constraint feature extraction module, the grouped attention multimodal fusion module based on two-stage collaboration, the physical-guided hybrid enhanced LSTM temporal modeling module, and the classification module through the backpropagation algorithm, forming an end-to-end physical consistency closed-loop control structure.

[0095] In summary, the modality-specific physical constraint feature extraction module in this invention achieves differentiated extraction of high-frequency vibration impact and low-frequency acoustic radiation features through lightweight 1D convolution and wave equation residual constraints. It utilizes a two-stage collaborative grouped attention multimodal fusion strategy, employing both primary fusion and higher-order enhancement, combined with grouped attention and coordinate attention mechanisms to suppress feature redundancy and modal conflicts. Furthermore, it employs a physically guided hybrid enhanced LSTM temporal modeling module to dynamically fuse the deviation between the ideal dynamic model and the actual system, improving the modeling accuracy of fault dynamic evolution. This invention effectively solves the problems of insufficient modality-specific feature extraction, fusion of redundant conflicts, and lack of physical constraints in temporal modeling during multimodal fault diagnosis.

[0096] Secondly, this invention also proposes a rolling bearing fault diagnosis system based on acoustic and vibration multimodal fusion, comprising: a training set construction module for constructing a training set; a model construction module for constructing a multimodal rolling bearing fault diagnosis model that integrates physical information; a training module for training the multimodal rolling bearing fault diagnosis model using the training set to obtain a trained multimodal rolling bearing fault diagnosis model; and a diagnosis module for using the trained multimodal rolling bearing fault diagnosis model to perform fault diagnosis on the rolling bearing under test.

[0097] The multimodal rolling bearing fault diagnosis model includes: a physical constraint feature extraction module, used to extract vibration and acoustic features in parallel to obtain vibration and acoustic features, and obtain a physical consistency score based on tachometer signals and bearing parameters; a two-stage collaborative grouped attention multimodal fusion module, used to obtain preliminary multimodal fusion features based on vibration and acoustic features; a channel attention module, used to generate channel attention based on coordinate attention mechanism and preliminary multimodal fusion features, and obtain enhanced multimodal fusion features based on preliminary multimodal fusion features and channel attention; a physically guided hybrid enhanced LSTM temporal modeling module, used to obtain a predicted temporal state sequence based on enhanced multimodal fusion features and physical consistency score; and a classification module, used to obtain a predicted fault diagnosis result based on the predicted temporal state sequence.

[0098] The multimodal rolling bearing fault diagnosis model in this embodiment is the same as the multimodal rolling bearing fault diagnosis model described in any one of the first aspects.

[0099] In one specific embodiment, the proposed multimodal rolling bearing fault diagnosis model (PxLSTM-MFDM) that integrates physical information, along with existing CNN, LSTM, and Transformer methods, is used to perform fault diagnosis on the KAIST bearing dataset. The results are shown in Table 1 and... Figures 7-10 As shown.

[0100] Table 1. Comparison of fault diagnosis performance of different models on the KAIST dataset

[0101]

[0102] exist Figures 7-10 In the diagram, "normal" represents data without faults. BPFI_03 indicates that the inner ring of the bearing has a crack size of 0.3mm, BPFI_10 indicates that the inner ring of the bearing has a crack size of 1.0mm, BPFO_03 indicates that the outer ring of the bearing has a crack size of 0.3mm, and BPFO_10 indicates that the outer ring of the bearing has a crack size of 1.0mm.

[0103] From Table 1 and Figures 7-10 As can be seen, the diagnostic accuracy of the PxLSTM-MFDM proposed in this embodiment reaches 100%, which is better than mainstream models or methods such as CNN, LSTM, and Transformer, demonstrating high accuracy and strong robustness in multimodal rolling bearing fault diagnosis.

[0104] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for fault diagnosis of rolling bearings based on acoustic-vibration multimodal fusion, characterized in that, include: A multimodal rolling bearing fault diagnosis model and training set integrating physical information are constructed; the multimodal rolling bearing fault diagnosis model is trained using the training set, and the trained multimodal rolling bearing fault diagnosis model is used to diagnose the faults of the rolling bearing under test; wherein, the training set includes multiple samples, including vibration signals, acoustic signals, tachometer signals, bearing parameters and fault labels; The multimodal rolling bearing fault diagnosis model includes: The physical constraint feature extraction module is used to extract features from vibration and acoustic signals in parallel to obtain vibration and acoustic features, and to obtain physical consistency scores based on tachometer signals and bearing parameters. The two-stage collaborative grouped attention multimodal fusion module is used to obtain preliminary multimodal fusion features based on vibration and acoustic features; generate channel attention based on coordinate attention mechanism and preliminary multimodal fusion features; and obtain enhanced multimodal fusion features based on preliminary multimodal fusion features and channel attention. A physics-guided hybrid enhanced LSTM temporal modeling module is used to obtain the predicted temporal state sequence based on multimodal enhanced fusion features and physical consistency scores; The classification module is used to obtain the predicted fault diagnosis results based on the predicted time-series state sequence.

2. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The physical constraint feature extraction module includes parallel vibration signal extraction branches and acoustic signal extraction branches. Both vibration signal extraction branches and acoustic signal extraction branches are lightweight convolutional networks with embedded physical residual constraints based on the second-order discrete wave equation.

3. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The process of generating preliminary multimodal fusion features includes: concatenating the parallel input vibration features and acoustic features, and grouping the concatenated features to obtain multiple grouped features; the local attention branch extracts the local correlation of each grouped feature through grouped convolution to obtain the first weight of each grouped feature; the global attention branch extracts the global context of each group of features through global average pooling and grouped convolution to obtain the second weight of each group; the first and second weights of each group are activated separately by activation functions and then fused to obtain the third weight of each group; and the preliminary multimodal fusion features are generated by weighting the residuals of each grouped feature and its third weight.

4. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The process of generating channel attention weights includes: embedding coordinate information into the primary fusion feature through adaptive max pooling, and then compressing the channel dimension of the primary fusion feature with embedded coordinate information through a convolutional layer, a batch normalization layer, and a nonlinear activation unit based on an adaptive switching mechanism to generate channel attention weights. The expression for the nonlinear activation unit is as follows: ； In the formula, The output of the nonlinear activation unit, The input features are those of the nonlinear activation unit. and For learnable scaling parameters, For the Sigmoid function, A switching factor to control the degree of nonlinearity of the activation function.

5. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The physics-guided hybrid enhanced LSTM time series modeling module includes a hybrid physics-data modeling layer and a dynamic fusion layer. The hybrid physics-data modeling layer includes a physics mechanism branch and a data-driven branch. The physics mechanism branch is used to obtain the ideal evolution sequence based on the multimodal enhanced fusion features and the preset bearing failure mechanism. The data-driven branch is used to obtain the data-driven branch state sequence based on the multimodal enhanced fusion features and the physical consistency score. The dynamic fusion layer is used to dynamically fuse the ideal evolution sequence and the data-driven branch state sequence through an adaptive gating factor to obtain the predicted state sequence.

6. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 5, characterized in that, The data-driven branch adopts a frequency-domain decoupled matrix-scalar cascaded interactive topology. The matrix-scalar cascaded interactive topology includes a hierarchical bottom layer and a high layer. The bottom layer includes a matrix-type memory update unit with group aggregation, and the high layer includes a physically guided exponential scalar update unit. A bypass connection is also established between the exponential scalar update unit and the matrix memory update unit based on a memory-gated injection mechanism. The matrix-type memory update unit projects multimodal enhanced fusion features onto multiple orthogonal subspaces according to a preset grouping and aggregation strategy. It independently maintains a memory matrix in each subspace and accumulates global context information through a covariance update rule to update the memory matrix. Based on the updated memory matrix, it obtains an intermediate hidden state sequence. It also extracts the energy state descriptor of the memory matrix using the Frobenius norm. A bypass connection injects the energy state descriptor into the exponential input gate of the exponential scalar update unit. The exponential scalar update unit employs an exponential domain gating mechanism, combining the intermediate hidden state sequence, energy state descriptor, and physical consistency score for exponential domain gating updates, and outputs the final intermediate hidden state sequence, which serves as the data-driven branch state sequence.

7. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 6, characterized in that, The physics-guided hybrid enhanced LSTM time series modeling module also includes a calibration layer. During training, the calibration layer is used to calculate the minimum regularized path distance between the predicted state sequence and the ideal evolution sequence using a pre-defined physics-data consistency checker, and to construct a regularization constraint function based on the minimum regularized path distance to inversely constrain the parameter updates of the data-driven branches.

8. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The classification module includes a fully connected layer, which is used to map the predicted state sequence to the fault diagnosis results of the rolling bearing under test.

9. The rolling bearing fault diagnosis method based on acoustic-vibration multimodal fusion according to claim 1, characterized in that, The training process uses a total loss function to optimize the parameters of the multimodal rolling bearing fault diagnosis model. This total loss function includes a fault diagnosis task loss function and a multi-physical constraint loss function. Specifically, the fault diagnosis task loss function includes the classification task loss function of the classification module, the feature extraction task loss function of the physical constraint feature extraction module, the fusion task loss function of the two-stage collaborative grouped attention multimodal fusion module, and the temporal consistency loss function of the physically guided hybrid enhanced LSTM temporal modeling module. The multi-physical constraint loss function includes the physical residual constraint function of the physical constraint feature extraction module and the regularization constraint function of the physically guided hybrid enhanced LSTM temporal modeling module.

10. A rolling bearing fault diagnosis system based on acoustic-vibration multimodal fusion, characterized in that, include: The training set construction module is used to build the training set; the training set includes multiple samples, including vibration signals, acoustic signals, tachometer signals, bearing parameters, and fault labels. The model building module is used to build a multimodal rolling bearing fault diagnosis model that integrates physical information. The training module is used to train the multimodal rolling bearing fault diagnosis model using the training set; The diagnostic module is used to diagnose the faults of the rolling bearing under test using a trained multimodal rolling bearing fault diagnosis model. The multimodal rolling bearing fault diagnosis model includes: The physical constraint feature extraction module is used to extract features from vibration and acoustic signals in parallel to obtain vibration and acoustic features, and to obtain physical consistency scores based on tachometer signals and bearing parameters. The two-stage collaborative grouped attention multimodal fusion module is used to obtain preliminary multimodal fusion features based on vibration and acoustic features; generate channel attention based on coordinate attention mechanism and preliminary multimodal fusion features; and obtain enhanced multimodal fusion features based on preliminary multimodal fusion features and channel attention. A physics-guided hybrid enhanced LSTM temporal modeling module is used to obtain the predicted temporal state sequence based on multimodal enhanced fusion features and physical consistency scores; The classification module is used to obtain the predicted fault diagnosis results based on the predicted time-series state sequence.

Images