Bearing fault diagnosis method based on multi-modal information fusion of self-supervised learning

Abstract

The present application belongs to the technical field of aero-engine state monitoring and intelligent fault diagnosis, and discloses a multi-modal information fusion bearing fault diagnosis method based on self-supervised learning. Firstly, the stable feature representation insensitive to mask disturbance is extracted from the multi-modal signal without label through mask reconstruction self-supervised pre-training, and the exponential moving average algorithm is used to dynamically update the feature benchmark points of each mode; in the downstream fault diagnosis task, a multi-modal joint decision model containing a pre-training encoder, a single-modal classifier and a fusion classifier is constructed, the contribution degree is calculated based on the cooperative game Shapley value, the deviation degree of the mode feature and the benchmark point is combined, and the adaptive weighted fusion of multi-modal decision is realized. The present application effectively reduces the dependence of the deep neural network on the fault labeled data, and improves the accuracy and robustness of the diagnosis system in the multi-modal signal diagnosis scene through the dynamic fusion mechanism, and is suitable for the industrial scene with limited sample label resources.

Description

Technical Field

[0001] This invention belongs to the field of aero-engine condition monitoring and intelligent fault diagnosis technology, and relates to a bearing fault diagnosis method based on self-supervised learning and multimodal information fusion. Background Technology

[0002] As a high-speed rotating core component, the health of aero-engine bearings directly affects the overall operational safety and reliability of the engine. With the increasing prevalence of intelligent operation and maintenance concepts, deep learning-based fault diagnosis methods have become a research hotspot in the industry. These methods can automatically extract fault characteristics from complex monitoring data, achieving intelligent identification and early warning. However, in practical engineering applications, these methods still face two major bottlenecks:

[0003] First, the training of deep neural networks heavily relies on acquired signals with clearly labeled fault information, which contradicts the massive amounts of unlabeled fault information data in practical engineering applications. Most current mainstream deep learning methods follow a supervised learning paradigm, and their powerful feature extraction capabilities depend on a large amount of high-quality fault-labeled data. However, the vast majority of the massive monitoring data generated during the operation of aero-engines lacks clear fault labeling. Accurate fault labeling requires the knowledge and experience of domain experts, resulting in high labeling costs and long cycles, making large-scale fault labeling difficult. This difficulty in data labeling makes it difficult to use large amounts of unlabeled fault information for training neural networks, limiting the generalization performance and application scope of deep learning-based methods in practical fault diagnosis scenarios.

[0004] Secondly, the lack of an effective multimodal information fusion mechanism makes it difficult to fully utilize the information collected by multiple sensors and establish a mapping relationship with fault characteristics under complex operating conditions. With the development of modern aero-engine monitoring systems, the deployment of various sensors such as vibration, acoustic, and temperature sensors can achieve multi-dimensional monitoring of equipment status. Faced with multi-source sensor data, existing fault diagnosis fusion methods mostly adopt static strategies such as shallow feature splicing of neural networks or classifier decision voting, failing to fully consider the dynamic changes in signal-to-noise ratio and reliability of different sensor signals under different operating conditions and fault modes. When a sensor causes severe data distortion due to its own performance drift, transient electromagnetic interference, physical damage, or loose installation, traditional fusion methods lack effective reliability assessment and adaptive adjustment capabilities, and cannot effectively suppress the interference of abnormal sensors in the decision-making process. This causes abnormal data to affect the decision-making process, leading to a significant decline in the overall diagnostic performance of the system or even misjudgment.

[0005] In summary, in addressing the current fault diagnosis challenges, it is necessary to research methods that can effectively utilize massive amounts of fault-free labeled data for deep neural network training, while simultaneously achieving intelligent and adaptive multimodal information fusion. This would enable stable and reliable fault diagnosis of aero-engine bearings in real-world industrial scenarios, providing a solid guarantee for the safe operation of aero-engines. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a bearing fault diagnosis method based on self-supervised learning and multimodal information fusion. This method aims to reduce reliance on labeled fault data through self-supervised pre-training and improve the diagnostic performance of the system when labeled fault information is insufficient through a dynamic weighted fusion mechanism.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A bearing fault diagnosis method based on self-supervised learning and multimodal information fusion includes the following steps:

[0009] Step 1: Single-modal self-supervised pre-training based on data reconstruction;

[0010] Step 1.1, Modal data mask;

[0011] For a given modal time-domain signal sample Two independent binary masks are randomly generated. and In a binary mask, a value of 0 represents a position where the modal time-domain signal is masked; element-wise multiplication is used to sample the modal time-domain signal. Multiplying by the binary mask generates two partially masked signal views:

[0012]

[0013] in, Indicates element-wise multiplication. and These represent the results after binary masking. and Signal view;

[0014] Step 1.2, Feature extraction and feature baseline update;

[0015] Feature encoder Using a convolutional neural network design, from the signal view and Extracting deep feature vectors and :

[0016]

[0017] A convolutional neural network consists of cascaded convolutional modules, expressed as:

[0018]

[0019] in, For the first The output feature tensor of a multilayer convolutional neural network. Indicates the first The input feature tensor of a multilayer convolutional neural network, For convolution operations, For the first Trainable convolutional kernel weights of a multilayer convolutional neural network;

[0020] When self-supervised pre-training reaches a specified iteration Extracting modal time-domain signal samples using convolutional neural networks After obtaining the deep feature vectors, the mean of the feature vectors of the unmasked data in each iteration is calculated. The modal baseline points are updated using an exponential moving average algorithm:

[0021]

[0022] in, For the first Modal reference point after the next iteration For the first Modal reference points before the next iteration For the first The mean eigenvector of the unmasked data in the nth iteration, at the... In the next iteration, the initial value of the modal reference point , This is the update rate hyperparameter;

[0023] Step 1.3, Data Reconstruction;

[0024] Feature Decoder The deep feature vector output by the feature encoder and Reconstruction yields reconstructed signals and :

[0025]

[0026] Feature Decoder Used only during the pre-training phase, the basic unit is a fully connected network:

[0027]

[0028] in, For the first The output vector of a layer neuron. For the first The input vector of a layer neuron. For the first The trainable weight matrix of a layer of neurons, For the first Trainable bias vectors of layer neurons;

[0029] Step 1.4, Loss Calculation;

[0030] Calculate deep feature vectors and The consistency loss between representations is calculated using the cosine consistency loss function:

[0031]

[0032] in, and These represent deep feature vectors. and The L2 norm;

[0033] Feature Decoder The optimization objective is to make the reconstructed signal approximate the modal time-domain signal sample in the mask region. The reconstruction loss is:

[0034]

[0035] in, This represents the mean squared error loss function. and This means that the reconstruction loss is calculated only at the masked locations;

[0036] The total loss function for the pre-training stage is obtained by weighted summation of the reconstruction loss and the cosine consistency loss:

[0037]

[0038] in, These are weight hyperparameters;

[0039] Step 2, multimodal joint decision-making for downstream fault diagnosis;

[0040] For including For fault diagnosis tasks with input modal data, a multimodal joint decision-making fault diagnosis model is constructed from data from various modalities. Pre-trained feature encoder with frozen parameters for each modality ,correspond Single-modal classifier requiring parameter fine-tuning for each modality A feature fusion classifier that requires parameter fine-tuning It consists of a pre-trained encoder and a weight allocation module. The number of pre-trained encoders and single-modal classifiers is determined by the number of modalities.

[0041] Step 2.1, Feature Extraction;

[0042] For mode Pre-trained feature encoder The convolutional neural network, pre-trained using modal data masking and data reconstruction tasks, is expressed as follows:

[0043]

[0044] in, For modality The input signal, For pre-trained feature encoders Extracted modal feature vectors;

[0045] During the fine-tuning phase of a convolutional neural network, the parameters of all pre-trained feature encoders are frozen and do not participate in gradient updates.

[0046] Step 2.2, Feature fusion classification and modality ensemble classification;

[0047] Feature fusion classifier Receives concatenation of multiple modal feature vectors And perform classification prediction:

[0048]

[0049] in, Indicates fusion features The category prediction probability vector;

[0050] For mode Single-modal classifier Employing a fully connected network, based on modal feature vectors Performing classification and prediction on fault states is expressed as follows:

[0051]

[0052] in, Representing modes The vector of predicted probability values ​​for the fault category;

[0053] Shapley value calculation mode based on cooperative game theory Contribution to integrated decision-making:

[0054]

[0055] in, For the set of all modalities, For the modal fusion order, For the modal fusion order set, For modal fusion order In, compared to modes The first set of modal fusion sequences to be added. For modality marginal contribution express The value function of is specifically expressed as:

[0056]

[0057] in, Used to quantify the degree of deviation of each modal signal from the modal reference. For modality The final modal reference points obtained during self-supervised pre-training;

[0058] Normalizing the Shapley values ​​in cooperative game theory yields the final modal contribution weights. :

[0059]

[0060] Using modal contribution weights Calculate the weighted ensemble probability vector :

[0061]

[0062] Step 2.3, calculate the model loss;

[0063] Calculate the cross-entropy loss based on the category prediction values ​​to obtain the single-modality classification loss. , fusion classification loss and ensemble classification loss The specific calculation method is as follows:

[0064]

[0065] in, The actual label vector;

[0066] The total loss function is the sum of the single-modal classification loss, the fusion classification loss, and the ensemble classification loss:

[0067] .

[0068] The beneficial effects of this invention are as follows: By constructing a mask reconstruction self-supervised pre-training task, a deep neural network is trained to adaptively learn fault information-related features from multimodal signals, effectively solving the bottleneck problem of scarce sample labels for aero-engine bearing faults and providing a new technical solution for fault diagnosis tasks. Adaptive decision fusion of multi-source information is achieved through contribution weight calculation based on Shapley values. This mechanism can dynamically sense and suppress modal signal data offsets or failures, significantly improving the accuracy of fault diagnosis and its diagnostic capabilities under complex operating conditions. A lightweight encoder is used for feature extraction, and the classifier is fine-tuned in downstream tasks, ensuring the fault diagnosis capabilities of the deep neural network while reducing the number of model parameters and improving computational efficiency, making it suitable for deployment on edge devices with limited sample label resources.

[0069] This invention proposes a fault diagnosis method for aero-engine bearings based on self-supervised learning and multimodal decision fusion. The method collects multimodal signals such as vibration, temperature, and acoustic data, constructs a masked reconstruction self-supervised pre-training task, learns fault feature representations from unlabeled data, and dynamically updates the health status feature benchmarks for each modality. In the fault diagnosis stage, multimodal features are extracted using a parameter-frozen pre-trained encoder, and single-modal and fusion classifiers are trained in parallel. Simultaneously, a Shapley value weighting mechanism based on cooperative game theory is introduced, combined with confidence assessment provided by feature center points, to achieve adaptive weighted fusion of multimodal decisions. This method effectively reduces the neural network's dependence on labeled data and improves fault diagnosis accuracy and tolerance to modal failures through an adaptive fusion strategy. Attached Figure Description

[0070] Figure 1 A flowchart of a method for diagnosing aero-engine bearing faults based on self-supervised learning and multimodal information decision fusion.

[0071] Figure 2 This is a structural diagram of a single-modal pre-trained encoder.

[0072] Figure 3 This is a structural diagram of the fault diagnosis model.

[0073] Figure 4 The curves represent the total loss during the pre-training process of the current mode encoder and the validation loss on the test set.

[0074] Figure 5 The curves represent the consistency loss and reconstruction loss during the pre-training process of the current mode encoder.

[0075] Figure 6 This is the total loss curve during the training process of the fault diagnosis model.

[0076] Figure 7This refers to the loss curves of each loss component during the training process of the fault diagnosis model.

[0077] Figure 8 This is the test accuracy curve during the training process of the fault diagnosis model. Detailed Implementation

[0078] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0079] The flowchart of the bearing fault diagnosis method based on self-supervised learning and multimodal information fusion is as follows: Figure 1 As shown, the structure of a single-modal self-supervised pre-trained feature encoder is as follows: Figure 2 As shown, the structure of the fault diagnosis network is as follows: Figure 3 As shown below, the specific implementation of each step is described in detail.

[0080] Step 1, Multimodal signal data processing;

[0081] Vibration, acoustic, temperature, and drive current signals of bearings from the same engine under different fault conditions are collected. These sensor data comprehensively describe the characteristics of the engine bearings under different operating conditions. The bearing fault types are inner ring fault (IF) and outer ring fault (OF), and fault-free bearings are recorded as healthy (N).

[0082] Dataset middle, This indicates that the data used in the pre-training phase without fault information annotations totals [amount missing]. Group data, containing , Acceleration data in the axial direction , Acoustic data Temperature data Drive current data ; This indicates the amount of data labeled with fault information used in the downstream task phase, totaling [amount missing]. Group data, compared to Added fault type labels , The values ​​0, 1, and 2 correspond to the three bearing states: N, IF, and OF, respectively.

[0083] right The data in the dataset is processed independently for each modality, with each modality's data segment treated as an independent sample unit. The data in the dataset is used to construct multimodal samples in an aligned manner, combining five modal data segments within the same time period. Bind it to a data group, and then associate it with the fault label corresponding to that time window. This correlation ensures consistency of each sample in terms of both time sequence and category.

[0084] Step 2: Construct the feature encoder and feature decoder;

[0085] The feature encoder employs a convolutional neural network to extract deep feature vectors from randomly masked modal time-domain signal samples. Its core structure consists of cascaded convolutional modules, expressed as follows:

[0086]

[0087] in, For the first The output feature tensor of a multilayer convolutional neural network. Indicates the first The input feature tensor of a multilayer convolutional neural network, For convolution operations, For the first Trainable convolutional kernel weights of a multilayer convolutional neural network;

[0088] Depthwise separable convolutional layers decompose standard convolution into two independent operations: depthwise convolution and pointwise convolution, reducing the number of model parameters and computational complexity. Depthwise separable convolutions extract spatial features independently for each input channel.

[0089]

[0090] in, For the first The first layer of the output feature tensor Each channel component Indicates the first The first layer of input feature tensor Each channel component For the first The first layer of a convolutional neural network Trainable convolutional kernel weights for each channel, This represents the number of channels for the input feature;

[0091] Pointwise convolution achieves information fusion between channels through 1×1 convolution:

[0092]

[0093] in, For the first Position of the layer's output feature tensor The amount, Indicates the first Position of the input feature tensor of the layer The amount, For the first Position of layer-by-layer pointwise convolutional neural network Trainable convolutional kernel weights, Indicates the length of the input feature;

[0094] The feature encoder of the present invention consists of two deep convolutional neural networks and two deep separable convolutional neural networks. Batch normalization and activation function GELU are added after each network layer to extract features and reduce dimensionality of the input signal. The dimensionality-reduced data is then processed by a pooling module consisting of global average pooling and linear layers to obtain a feature vector that matches the input dimension of the feature decoder.

[0095] The feature decoder receives the feature vectors output by the feature encoder and reconstructs the modal time-domain signal samples using these feature vectors. The feature decoder is only used during the pre-training phase, and its basic unit is a fully connected network, expressed as:

[0096]

[0097] in, For the first The output vector of a layer neuron. For the first The input vector of a layer neuron. For the first The trainable weight matrix of a layer of neurons, For the first Trainable bias vectors of layer neurons;

[0098] Table 1 Parameters of each part of the pre-trained model

[0099]

[0100] Step 3: Construct a self-supervised learning task for single-mode signals;

[0101] For a given modal time-domain signal sample Two independent binary masks are randomly generated. and Modal time-domain signal samples are obtained through element-wise multiplication. Multiplying by the binary mask generates two partially masked signal views:

[0102]

[0103] in Indicates element-wise multiplication. and They represent the process. and The mask signal;

[0104] Signal View and Input the feature encoder to obtain the depth feature vector after dimensionality reduction by the feature encoder. and :

[0105]

[0106] Deep feature vectors and The consistency loss between representations is calculated using the cosine consistency loss function:

[0107]

[0108] in, and These represent deep feature vectors. and The L2 norm;

[0109] When self-supervised pre-training reaches a specified iteration Extracting modal time-domain signal samples using convolutional neural networks After obtaining the deep feature vectors, the mean of the feature vectors in each iteration is calculated. The modal baseline points are updated using an exponential moving average algorithm:

[0110]

[0111] in For the first Modal reference point after the next iteration For the first Modal reference points before the next iteration For the first The mean eigenvector of the unmasked data in the nth iteration, at the... In the next iteration, the initial value of the reference point , This is the update rate hyperparameter;

[0112] Feature Decoder The deep feature vector output by the feature encoder and Reconstruction yields reconstructed signals and :

[0113]

[0114] Feature Decoder The optimization objective is to make the reconstructed signal approximate the modal time-domain signal sample in the mask region. The reconstruction loss is:

[0115]

[0116] in, This represents the mean squared error loss function. and This means that the reconstruction loss is calculated only at the masked locations;

[0117] The total loss function for the pre-training stage is obtained by weighted summation of the reconstruction loss and the cosine consistency loss:

[0118]

[0119] in, For weight hyperparameters.

[0120] Step 4: Construct a fault diagnosis model;

[0121] The constructed multimodal information joint decision-making fault diagnosis model consists of five feature encoders that have been pre-trained with data from different modalities, a trainable feature fusion classifier, and a Shapley value calculation module based on cooperative game theory.

[0122] Feature encoders are used to extract deep features from multimodal data, and are expressed as follows:

[0123]

[0124] in, For the first Input signals of one modality, For the feature encoder of the corresponding mode, For pre-trained feature encoders Extracted modal feature vectors, Extracting respectively Features ;

[0125] During the fine-tuning phase of a convolutional neural network, the parameters of all pre-trained feature encoders are frozen and do not participate in gradient updates.

[0126] The feature fusion classifier is located at the back end of the multimodal information joint decision-making fault diagnosis model. It outputs the probability distribution of fault categories based on the features extracted by the feature encoder. The structure of the feature fusion classifier is a fully connected network, which is the same as the fully connected layer expression in step 3. It learns the mapping relationship between input features and fault categories.

[0127] Single-modal classifier Employing a fully connected network, based on modal feature vectors The classification and prediction of fault states is expressed as follows:

[0128]

[0129] in, Representing modes The vector of predicted probability values ​​for the fault category. For features respectively Classify;

[0130] Feature fusion classifier Receives concatenation of multiple modal feature vectors And perform classification prediction:

[0131]

[0132] in, Indicates fusion features The category prediction probability vector;

[0133] Table 2. Parameters of each part of the classifier

[0134]

[0135] Step 5: Construct a fault diagnosis task based on decision fusion and feature fusion;

[0136] Calculate the cross-entropy loss based on the category prediction values ​​to obtain the single-modality classification loss. , fusion classification loss and ensemble classification loss The specific calculation method is as follows:

[0137]

[0138] Shapley value calculation mode based on cooperative game theory Contribution to integrated decision-making:

[0139]

[0140] in, For the set of all modalities, For the modal fusion order, For the modal fusion order set, For modal fusion order In, compared to modes The first set of modal fusion sequences to be added. For modality marginal contribution express The value function of is specifically expressed as:

[0141]

[0142] in, Used to quantify the degree of deviation of each modal signal from the modal reference. For modality The final modal reference points obtained during self-supervised pre-training;

[0143] Calculate the weighted ensemble probability vector based on the contribution of the Shapley value and the single-modal category prediction probability vector. :

[0144]

[0145] The total loss function is the sum of the single-modal classification loss, the fusion classification loss, and the ensemble classification loss:

[0146] .

[0147] The final prediction probability vector for multimodal data is: The final predicted label is:

[0148]

[0149] in, Indicates that the category belongs to The probability, This represents the predicted category value of the modal data corresponding to the feature.

[0150] Step 6, Model Training;

[0151] The training process of this invention includes a pre-training part of self-supervised learning of the feature encoder and a learning part of the multimodal information joint decision-making fault diagnosis model, so as to ensure that the network can efficiently and stably learn feature representations and classification capabilities suitable for fault diagnosis.

[0152] In the pre-training part of the feature encoder, the feature encoder and feature decoder are trained for each modal data in turn. The following operations are performed in each training iteration: (1) Generate a binary mask to mask the modal time domain signal sample; (2) The feature encoder extracts features from the masked signal view; (3) Determine whether a specific period has been reached. If it has, extract features from the unmasked modal time domain signal sample and update the modal center point; (4) Calculate the feature extraction consistency loss; (5) Reconstruct the data of the masked part; (6) Calculate the data reconstruction loss; (7) Calculate the total loss of the pre-training stage and use the Adam optimizer to update the parameters of each network according to the calculated gradient.

[0153] In the training of the fault diagnosis network, a corresponding feature fusion classifier is trained for each modal data feature encoder for subsequent decision fusion. At the same time, a feature fusion classifier with the fusion features of the feature encoder as input is trained. The following operations are performed in each training iteration: (1) Read the pre-trained feature encoders of each modality, (2) Use the feature encoder to extract features from the training data, (3) Calculate the similarity of single modal features and perform feature fusion, (4) Calculate the cross-entropy loss for the fused features, calculate the single modal classification loss for the single modal features, and calculate the ensemble classification loss based on the result of decision fusion, (5) Calculate the total loss and use the Adam optimizer to update the parameters of each network module based on the calculated gradient.

[0154] Dataset composition:

[0155] The training data came from the publicly available multi-sensor bearing failure dataset from the Korea Advanced Institute of Science and Technology (KAIST), including vibration, acoustic, temperature, and current data under different load conditions. Bearing failures, including inner ring failures and outer ring failures, were simulated in different load condition tests based on crack sizes (0.3 mm, 1.0 mm, and 3.0 mm).

[0156] The sampling frequency for vibration, temperature, and drive current data is 25.6 kHz, with 120 seconds of data acquisition under normal conditions and 60 seconds under fault conditions. The sampling frequency for acoustic data is 51.2 kHz.

[0157] To ensure the balance and comparability of the dataset, data collected for various fault modes and modalities were standardized. Data under normal conditions was truncated to the first 60 seconds to control the consistency of time span and sample size. Acoustic sensor signals were downsampled to reduce the data rate while maintaining consistency in temporal characteristics. To unify the fault diagnosis benchmark for multimodal data, the analysis scope was limited to five no-load fault states covered by the acoustic sensors: healthy state, inner ring fault (0.3 mm and 1.0 mm damage), and outer ring fault (0.3 mm and 1.0 mm damage). Data from all modalities were analyzed based on these five fault states.

[0158] The signal was segmented into segments of 2048 bytes each. There were 480 samples of the fault-free state, 480 samples each of the two inner ring faults (0.3 mm and 1.0 mm), and 480 samples each of the two outer ring faults (0.3 mm and 1.0 mm), for a total of 2400 samples. Of these, 1400 samples were used for the unlabeled pre-training task, 600 samples for the labeled fine-tuning task, and the remaining 400 samples were used as test samples.

[0159] Model training and testing results analysis

[0160] parameter Set to 10000, mask ratio to 40%, iteration period The value is set to 2000. The total number of cycles for both the pre-training of the encoder self-supervised learning and the supervised learning of the fault diagnosis model is 5000, and the learning rate is 0.0001.

[0161] Figure 4 This shows the current mode model training process. The loss curve and the reconstruction loss of the validation set, Figure 5 This shows the model training process. Two components , The loss curve. Due to The value is relatively small, so a large hyperparameter is set for it. Therefore, in the early stages of training (before 2000 iterations), the decrease in model loss was mainly due to consistency loss. The decrease in density means that during this training process, the encoder is able to extract stable deep features without being affected by the mask. After stabilization, The data begins to decline, improving the data reconstruction effect and thus further revealing the deeper features of the data.

[0162] Figure 6 The total loss curve during model training is shown. Figure 7 The curves of each loss component during model training are shown. Figure 8 The accuracy curves during model training are shown. The results indicate that although the model's total loss only stabilizes after 3000 iterations, the fault diagnosis accuracy achieves good results after only 1000 iterations, with the fusion loss... The early convergence allows for a rapid improvement in the accuracy of the fusion classification. Subsequently, the convergence of the ensemble loss and the loss of each individual modality classifier increases the robustness of the model in the face of increasing differences in the distribution of data from different modalities.

[0163] This invention proposes a multimodal information decision fusion method for aero-engine bearing fault diagnosis based on self-supervised learning. Results show that the constructed single-modal encoder can effectively extract deep features from aero-engine multimodal data even when labels are insufficient. After fine-tuning the fault diagnosis model with a small amount of labeled data, it can effectively identify faults in aero-engine multimodal data with high accuracy. In summary, the model proposed in this invention achieves good application results.

Claims

1. A bearing fault diagnosis method based on self-supervised learning and multimodal information fusion, characterized in that, Includes the following steps: Vibration, acoustic, temperature, and drive current signals of bearings from the same engine under different faults were collected. The bearing fault types were inner ring fault (IF) and outer ring fault (OF). Fault-free bearings were recorded as healthy (N). Step 1: Single-modal self-supervised pre-training based on data reconstruction; Step 2, multimodal joint decision-making for downstream fault diagnosis; For including For fault diagnosis tasks with input modal data, a multimodal joint decision-making fault diagnosis model is constructed from data from various modalities. Pre-trained feature encoder with frozen parameters for each modality ,correspond Single-modal classifier requiring parameter fine-tuning for each modality A feature fusion classifier that requires parameter fine-tuning It consists of a pre-trained encoder and a weight allocation module. The number of pre-trained encoders and single-modal classifiers is determined by the number of modalities. The specific implementation process of step 2 is as follows: Step 2.1, Feature Extraction; For mode Pre-trained feature encoder The convolutional neural network, pre-trained using modal data masking and data reconstruction tasks, is expressed as follows: in, For modality The input signal, For pre-trained feature encoders Extracted modal feature vectors; During the fine-tuning phase of a convolutional neural network, the parameters of all pre-trained feature encoders are frozen and do not participate in gradient updates. Step 2.2, Feature fusion classification and modality ensemble classification; Feature fusion classifier Receives concatenation of multiple modal feature vectors And perform classification prediction: in, Indicates fusion features The category prediction probability vector; For mode Single-modal classifier Employing a fully connected network, based on modal feature vectors Performing classification and prediction on fault states is expressed as follows: in, Representing modes The vector of predicted probability values ​​for the fault category; Shapley value calculation mode based on cooperative game theory Contribution to integrated decision-making: in, For the set of all modalities, For the modal fusion order, For the modal fusion order set, For modal fusion order In, compared to modes The first set of modal fusion sequences to be added. For modality marginal contribution express The value function of is specifically expressed as: in, Used to quantify the degree of deviation of each modal signal from the modal reference. For modality The final modal reference points obtained during self-supervised pre-training; Normalizing the Shapley values ​​in cooperative game theory yields the final modal contribution weights. : Using modal contribution weights Calculate the weighted ensemble probability vector : Step 2.3, calculate the model loss; Calculate the cross-entropy loss based on the category prediction values ​​to obtain the single-modality classification loss. , fusion classification loss and ensemble classification loss The specific calculation method is as follows: in, The actual label vector; The total loss function is the sum of the single-modal classification loss, the fusion classification loss, and the ensemble classification loss: 。 2. The bearing fault diagnosis method based on self-supervised learning and multimodal information fusion according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: Step 1.1, Modal data mask; For a given modal time-domain signal sample Two independent binary masks are randomly generated. and In a binary mask, a value of 0 represents a position where the modal time-domain signal is masked; element-wise multiplication is used to sample the modal time-domain signal. Multiplying by the binary mask generates two partially masked signal views: in, Indicates element-wise multiplication. and These represent the results after binary masking. and Signal view; Step 1.2, Feature extraction and feature baseline update; Feature encoder Using a convolutional neural network design, from the signal view and Extracting deep feature vectors and : A convolutional neural network consists of cascaded convolutional modules, expressed as: in, For the first The output feature tensor of a multilayer convolutional neural network. Indicates the first The input feature tensor of a multilayer convolutional neural network, For convolution operations, For the first Trainable convolutional kernel weights of a multilayer convolutional neural network; When self-supervised pre-training reaches a specified iteration Extracting modal time-domain signal samples using convolutional neural networks After obtaining the deep feature vectors, the mean of the feature vectors of the unmasked data in each iteration is calculated. The modal baseline points are updated using an exponential moving average algorithm: in, For the first Modal reference point after the next iteration For the first Modal reference points before the next iteration For the first The mean eigenvector of the unmasked data in the nth iteration, at the... In the next iteration, the initial value of the modal reference point , This is the update rate hyperparameter; Step 1.3, Data Reconstruction; Feature Decoder The deep feature vector output by the feature encoder and Reconstruction yields reconstructed signals and : Feature Decoder Used only during the pre-training phase, the basic unit is a fully connected network: in, For the first The output vector of a layer neuron. For the first The input vector of a layer neuron. For the first The trainable weight matrix of a layer of neurons, For the first Trainable bias vectors of layer neurons; Step 1.4, Loss Calculation; Calculate deep feature vectors and The consistency loss between representations is calculated using the cosine consistency loss function: in, and These represent deep feature vectors. and The L2 norm; Feature Decoder The optimization objective is to make the reconstructed signal approximate the modal time-domain signal sample in the masked region. The reconstruction loss is: in, This represents the mean squared error loss function. and This means that the reconstruction loss is calculated only at the masked locations; The total loss function for the pre-training stage is obtained by weighted summation of the reconstruction loss and the cosine consistency loss: in, For weight hyperparameters.

Images