Fault diagnosis method based on cross-modal bridging graph convolutional neural network

Abstract

This invention discloses a fault diagnosis method using a cross-modal bridging graph convolutional neural network. The specific implementation process is as follows: for the collected raw multimodal data, each sample represents a node, and the first k faulty nodes are used to construct an adjacency matrix. Graph convolution is used to extract the specific features of multiple modalities, and a cross-modal bridging module is introduced to complement and enhance the feature information of different modalities. Considering the common fault features hidden in multiple modalities, a dynamic collaborative sharing mechanism is introduced to capture similar features. Based on this process, multi-channel fusion is used to aggregate the information from different channels, and then the fused features are input into a Transformer to perform fault classification. Experimental results show that the proposed method is an effective fault diagnosis method.

Description

Technical Field

[0001] The present invention relates to technical fields such as graph convolutional neural networks and multimodal fault diagnosis, and specifically to a fault diagnosis method of a cross-modal bridging graph convolutional neural network, which can be applied to the field of fault diagnosis. Background Art

[0002] With the acceleration of industrialization, modern industrial systems are becoming increasingly complex, making efficient fault diagnosis particularly important. Traditional fault diagnosis methods rely on signal processing techniques (such as wavelet analysis, fast Fourier transform, and empirical mode decomposition) and machine learning models (such as principal component analysis and support vector machines), requiring extensive manual intervention and difficult to fully automate. Deep learning, as an efficient data-driven method, has demonstrated excellent performance in fault diagnosis, automatically extracting fault features from large amounts of data, overcoming the shortcomings of traditional methods. However, traditional methods perform poorly when processing multimodal data, often encountering difficulties in feature extraction and poor generalization capabilities, resulting in unstable diagnostic results. To address these issues, the present invention proposes a cross-modal bridging graph convolutional neural network. This method generates multimodal graph structure data by constructing a KNN graph, extracts key features using graph convolution, and achieves feature fusion through a cross-modal bridging mechanism to enhance information complementarity. The introduced dynamic collaborative mechanism improves the comprehensive analysis capabilities of multimodal data, and combined with the multi-channel fusion Transformer module, comprehensive information integration and fault identification are achieved. Summary of the Invention

[0003] To address the difficulties of traditional fault diagnosis methods in extracting comprehensive fault features and their relatively weak generalization capabilities, this paper builds on the foundation of graph convolutional neural networks by incorporating cross-modal bridging mechanisms and dynamic collaboration mechanisms. It then uses a multi-channel fusion Transformer to process multimodal information and achieve accurate classification. The specific implementation steps of this paper are as follows:

[0004] The steps to construct the adjacency matrix using KNN are as follows:

[0005] Calculate Euclidean distance for multimodal fault data:

[0006] The multimodal data X i =[x1,…,x n ] T ∈R n×d For each sample x i As a node, it represents the feature in a certain state, and the distance between any two samples is measured using the Euclidean distance. i and x j Its Euclidean distance is calculated as

[0007]

[0008] Select k nearest neighbor nodes to construct the adjacency matrix:

[0009] and are samples x i and x j The value of the mth feature dimension, M is the total number of feature dimensions. D is the Euclidean distance matrix obtained after calculation; select sample x i Establish connection edges between the samples of its k nearest neighbors, and then construct the adjacency matrix A of the graph structure under fault mode i i ;

[0010] 1. Use the cross-modal bridging module to complement and enhance multi-modal fault characteristics. The steps are as follows:

[0011] (1a) Utilize graph convolution operations to extract specific features of multimodal faults:

[0012] The adjacency matrix A constructed i and feature X i As input, the graph convolution output of layer l under fault mode i is processed using OP i (l) Indicates that OP i (l) It can be calculated as:

[0013]

[0014] Where: σ is a nonlinear activation function, and OP i (0) =X i ;DCW i (l-1) is the learnable weight matrix of the l-1 layer under the i modality; in addition I is the identity matrix; for degree matrix of ; For Adjacency matrix normalization operation; through the above graph convolution operation, the specific fault features of each mode can be effectively extracted;

[0015] (1b) Using cross-modal bridging mechanism to bridge and enhance multimodal features:

[0016] The present invention uses bimodal data, and its cross-modal bridging enhancement operation can be expressed as:

[0017]

[0018] Where: OP1 and OP2 are the output results calculated in step (1a), which are the output of mode 1 and mode 2 respectively; X1 and X2 are the characteristic matrices of mode 1 and mode 2 respectively; and They are the output results after cross-modal bridging enhancement operations;

[0019] 2. Use the dynamic collaborative sharing mechanism to learn multimodal hidden similar features. The steps are as follows:

[0020] Leveraging dynamic collaborative sharing mechanisms to learn multimodal hidden similarities:

[0021] This mechanism is used to dynamically and adaptively adjust the weights between the modes, ensuring that the unique characteristics of each mode are retained while capturing the common characteristics between multiple modes. This mechanism not only enhances the model's ability to express fault characteristics, but also improves the reliability and generalization of the diagnosis results. Under fault mode i, the output of the lth layer is It can be expressed as:

[0022]

[0023] Where: σ is a nonlinear activation function, and is the learnable weight matrix of layer l-1 in mode i, and The graph convolution operations of each modality are the same; in addition I is the identity matrix; for degree matrix of ; For Adjacency matrix normalization operation; use M comi To represent the output of each modality; finally, the average of the results of each modality is taken as the output of the dynamic collaboration module, which is calculated as:

[0024]

[0025] Where N represents the total number of modes, M com To calculate the final result of the common features of multiple modes;

[0026] 3. Use the multi-channel fusion Transformer module to process multimodal information. The steps are as follows:

[0027] (3a) After the above cross-feature enhanced graph convolution and multimodal association graph convolution operations, N+1 multimodal channels will be generated, where N is the total number of fault modes.

[0028] (3b) For the generated N+1 channels, the attention mechanism is used to dynamically adjust the information weight of each channel and calculate the attention score of each channel. The calculation of each channel is

[0029]

[0030] in: is the result obtained by cross-feature enhanced graph convolution operation on N modal data; M com is the common feature result calculated by multimodal association graph convolution operation on N modal data; α1,…,α N ,α com is the attention score of each N+1 channel;

[0031] (3c) Using the above calculation of the attention score of each channel, the features of each channel are weighted summed and the OP final To express the final result, OP final Calculated as:

[0032]

[0033] (3d) OP after fusion final Input into Transformer for final fault classification. The method of the present invention has the following advantages:

[0034] (1) The present invention can efficiently extract the specific features of multiple fault modes and enhance the complementary features through a cross-modal bridging mechanism. This mechanism can effectively integrate the information of each mode, significantly improve the shortcomings of feature representation, and enhance the robustness of the method and the stability of fault diagnosis.

[0035] (2) This invention introduces a dynamic collaborative sharing mechanism that can deeply learn similar features in multimodal faults and reveal common features between faults. This mechanism can capture more comprehensive and accurate fault information, thereby effectively improving the accuracy and reliability of fault diagnosis.

[0036] (3) This paper proposes a multi-channel fusion Transformer module to aggregate information from different modalities. This module not only effectively integrates various types of information but also alleviates oversmoothing, thereby more accurately capturing the connectivity characteristics of global nodes. In this way, the richness of feature representation is enhanced, the accuracy of classification is improved, and more precise fault diagnosis is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall framework diagram of the present invention

[0038] Figure 2This is a scatter plot of the dimensionality reduction effect of a random experiment on the Paderborn dataset

[0039] Figure 3 This is the confusion matrix classification effect diagram of random experiments on the Paderborn dataset

[0040] Figure 4 This is the confusion matrix classification effect diagram of the random experiment on the AUST dataset DETAILED DESCRIPTION

[0041] The specific implementation steps of the present invention are as follows:

[0042] 1. Transform multimodal data X i =[x1,…,x n ] T ∈R n×d For each sample x i Treated as a node, it represents the feature in a certain state, and the distance between any two samples is measured using the Euclidean distance. Select k nearest neighbor nodes to construct the adjacency matrix.

[0043] 2. Based on the construction of the adjacency matrix, the graph convolution operation is used to extract the unique features of multimodal faults. i and feature X i As input, the graph convolution output of layer l under fault mode i is processed using OP i (l) Representation, calculation Enhance actions using cross-modal bridging

[0044] and They are the output results after cross-modal bridging enhancement operations.

[0045] 3. Use dynamic coordination mechanism to adaptively adjust the weights between each mode to ensure that the unique characteristics of each mode are preserved while capturing the common characteristics between multiple modes. Calculated as Using M comi To represent the output of each modality. Finally, the average of the results of each modality is taken as the output of the dynamic collaboration module, which is calculated as

[0046] 4. Use the multi-channel fusion Transformer module to process multimodal information, use the attention mechanism to dynamically adjust the information weight of each channel, and calculate the attention score of each channel. The calculation of each channel is By calculating the attention scores of each channel, the features of each channel are weighted summed and the OP is used.final To express the final result, OP final Calculated as OP after fusion final Input into Transformer for final fault classification.

[0047] The effect of the present invention is further verified by the following experiments:

[0048] The experiment was verified on the Paderborn and AUST datasets. Two modes, vibration signals and torque mechanical parameters, were selected from the Paderborn dataset for analysis and verification. The sampling frequency of the vibration signal in the experiment was 64kHz, while the sampling frequency of the torque was 4kHz. The fault types contained in the dataset are divided into four categories, namely RF1, RF2, RF3 and RF4. 27 consecutive data points were selected to form a sample. 250 samples were collected for each fault type. These samples were distributed as follows: 70% for training, 10% for verification, and 20% for testing. A total of 1,000 samples were used for the experiment. The average accuracy of the present invention in five random experiments on the Paderborn dataset reached 0.993, showing good stability under variable working conditions.

[0049] The AUST experimental platform is equipped with a variety of high-precision sensors: displacement sensors, noise sensors, and vibration sensors. A manual hydraulic cylinder applies a radial force of 100N to the bearing to simulate the load in an actual production environment. In the experiment, the sampling frequency of the vibration and noise signals is 12KHz, and the sampling frequency of the displacement signal is 1KHz. By artificially simulating damage to deep groove ball bearings, four types of faults are obtained: inner ring fault, rolling element fault, combined fault of the rolling element and inner ring, and no fault. Each sample in the experiment consists of 27 consecutive sampling points, and a total of 280 samples are collected for each fault type. 70% of them are selected as the training set, 10% as the validation set, and 20% as the test set. A total of 1,120 samples are used for the experiment. The average accuracy of the present invention in five random experiments on the AUST dataset reached 0.984, confirming the effectiveness of this invention.

Claims

1. A fault diagnosis method for a cross-modal bridging graph convolutional neural network includes the following steps: (1) Using the cross-modal bridging module to complement and enhance multimodal fault features, specifically including using graph convolution operations to extract the specific features of multimodal faults and using the cross-modal bridging mechanism to bridge and enhance multimodal features; (2) A dynamic collaborative sharing mechanism is introduced to learn the similar features hidden in multiple modalities. This mechanism is used to dynamically and adaptively adjust the weights between the modalities to ensure that the unique features of each modality are preserved while capturing the common features between multiple modalities. (3) Use the multi-channel fusion Transformer module to fuse multimodal information and fault classification. The specific steps are as follows: (3a) After the cross-feature enhanced graph convolution and multimodal correlation graph convolution operations, N+1 multimodal channels are generated, where N is the total number of fault modes; (3b) For the generated N+1 channels, the attention mechanism is used to dynamically adjust the information weight of each channel and calculate the attention score of each channel. The calculation of each channel is: in: is the result obtained by cross-feature enhanced graph convolution operation on N modal data; M com It is the common feature result calculated by multimodal association graph convolution operation on N modal data; α1,…,α N ,α com is the attention score of each N+1 channel; (3c) Using the above calculation of the attention score of each channel, the features of each channel are weighted summed and the OP final To express the final result, OP final Calculated as: (3d) OP after fusion final Input into Transformer for final fault classification.

2. The cross-modal bridged graph convolutional neural network fault diagnosis method according to claim 1 is characterized in that: The method of utilizing the cross-modal bridging module to complementarily enhance multi-modal fault characteristics in step (1) includes the following steps: (1a) Utilize graph convolution operations to extract specific features of multimodal faults: The adjacency matrix A constructed i and feature X i As input, the graph convolution output of layer l under fault mode i is processed using OP i (l) Indicates that OP i (l) It can be calculated as: Where: σ is a nonlinear activation function, and OP i (0) =X i ;DCW i (l-1) is the learnable weight matrix of the l-1 layer under the i modality; in addition I is the identity matrix; for degree matrix of ; For Adjacency matrix normalization operation; through the above graph convolution operation, the specific fault features of each mode can be effectively extracted; (1b) Using cross-modal bridging mechanisms to bridge and enhance multimodal features: The present invention uses bimodal data, and its cross-modal bridging enhancement operation can be expressed as: Where: OP1 and OP2 are the output results calculated in step (1a), which are the output of mode 1 and mode 2 respectively; X1 and X2 are the characteristic matrices of mode 1 and mode 2 respectively; and They are the output results after cross-modal bridging enhancement operations.

3. The fault diagnosis method of the cross-modal bridged graph convolutional neural network according to claim 1 is characterized in that: The introduction of a dynamic collaborative sharing mechanism to learn multimodal hidden similar features in step (2) includes the following steps: The output of layer l under fault mode i It can be expressed as: Where: σ is a nonlinear activation function, and is the learnable weight matrix of layer l-1 in mode i, and The graph convolution operations of each modality are the same; in addition I is the identity matrix; for degree matrix of ; For Adjacency matrix normalization operation; use M comi To represent the output of each modality; finally, the average of the results of each modality is taken as the output of the dynamic collaboration module, which is calculated as: Where N represents the total number of modes, M com This is the final result of calculating the common features of multiple modes.

Images