End-to-end fault diagnosis and identification method based on multi-modal fusion

Abstract

The invention discloses an end-to-end fault diagnosis and identification method based on multi-modal fusion, and the method comprises the steps: 1), collecting a vibration signal and an acoustic signal, carrying out the preprocessing, and constructing a training sample set; 2) performing feature extraction to obtain a high-dimensional modal feature vector; 3) generating a sparse adjacency matrix through an end-to-end deep learning graph generation module, and establishing a graph generation structure relation; 4) constructing a multi-receptive field Chebyshev graph convolutional network, and extracting node-level features in a graph generation structure; 5) inputting the structure sensing features into a full-connection layer for mapping, and completing prediction and discrimination of a fault category to which an input sample belongs; performing model supervision training, and optimizing model parameters in an end-to-end mode; and 6) carrying out prediction output on the fault identification model on the test set, and carrying out quantitative evaluation on the fault identification result to obtain the fault identification device.The method belongs to the technical field of equipment operation state monitoring and fault diagnosis, and realizes accurate fault diagnosis of the rotating equipment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of equipment operation state monitoring and fault diagnosis, and relates to an end-to-end fault diagnosis identification method based on multi-modal fusion. BACKGROUND

[0002] In modern industrial production, rotating machinery (such as bearings, gears, motors, oil pumps, etc.) is widely used in energy, power, aerospace, transportation and other industries. After long-term operation, due to reasons such as friction, fatigue damage, insufficient lubrication, material aging, assembly error or load impact, wear, cracks, peeling and increased clearance may occur at key parts (such as bearing rolling elements and raceways), and the typical manifestations are physical signal fluctuations such as increased vibration and intensified noise. The fault signal often presents characteristics such as non-stationarity, periodic modulation and repeatability, and contains rich equipment operation state information. How to efficiently and timely capture these early weak signals and identify key fault features has become a key link to improve equipment health management capability.

[0003] The physical signal-based condition monitoring method is the mainstream technology path for rotating equipment fault diagnosis, mainly relying on vibration signals and acoustic emission signals. For the non-stationary vibration signals generated in the running process of rotating equipment, the traditional signal processing method mainly adopts the combination of frequency domain, time domain and time-frequency domain analysis to extract fault features. The common methods include short-time Fourier transform, wavelet transform, empirical mode decomposition and its derivative technology, which have the ability to mine key fault information in non-stationary signals to a certain extent, and are widely used in the condition monitoring and fault identification of rotating machinery. Although these methods can obtain good diagnostic performance under the experimental conditions of high signal-to-noise ratio and stable working condition, and provide effective feature support for the construction of subsequent fault classification model, there are obvious deficiencies: the fault in the early stage is weak, the noise is serious, the manually designed features are difficult to accurately extract the discriminant pattern, which easily leads to the decrease of diagnostic accuracy and the influence of diagnostic performance. Due to the limitation of information source, it is difficult to fully reflect the influence of the coupling effect of multi-class faults on the dynamic response of the system, which limits the anti-interference ability and generalization performance of the model. Due to the heterogeneity and correlation of multi-source data in time, space and frequency domain, the traditional feature fusion strategy lacks a unified modeling mechanism, and cannot effectively integrate the dynamic correlation of multi-source information, resulting in serious information redundancy or poor fusion effect, which limits the robustness and generalization ability of the diagnostic system. At present, multi-modal signal fusion can realize the complementary advantages under different working conditions, and improve the robustness and accuracy of the fault diagnosis system. The end-to-end learning framework based on deep neural network provides an advanced solution for rotating equipment fault diagnosis. Through the multi-layer nonlinear transformation structure, the unified modeling of feature extraction, feature expression and decision output is realized, which has the ability to directly learn complex distribution and high-level semantic features from original time series signals, significantly reduces the dependence on manual feature engineering, can efficiently process high-dimensional and nonlinear data, and is suitable for complex industrial application scenarios.

[0004] Therefore, it is a technical problem to be solved to develop a new multi-modal signal fusion technology, overcome the deficiencies of the existing methods, and realize accurate fault diagnosis of rotating equipment. SUMMARY

[0005] The purpose of the present application is to provide an end-to-end fault diagnosis and identification method based on multi-modal fusion, which solves the problems of insufficient multi-source information fusion modeling ability, insufficient mining of complex structural relationships, and insufficient accuracy and robustness of fault identification in the prior art.

[0006] The technical scheme adopted by the present application is an end-to-end fault diagnosis and identification method based on multi-modal fusion, which is implemented according to the following steps: Step 1, collect vibration signals and acoustic signals, pre-process, and construct a training sample set; Step 2, feature extraction, obtain high-dimensional modal feature vector; Step 3, generate a sparse adjacency matrix through the graph generation module of end-to-end deep learning, and establish a graph generation structure relationship; Step 4, construct a multi-receptive field Chebyshev graph convolution network to extract node-level features in the graph generation structure; Step 5, input the structure perception features into a fully connected layer for mapping, complete the prediction and discrimination of the fault category to which the input sample belongs, and then perform model supervised training to optimize the model parameters in an end-to-end manner; Step 6, perform prediction output on the test set to quantitatively evaluate the fault recognition result.

[0007] The beneficial effects of the present application include the following aspects: 1) The vibration signal and the acoustic signal are used as input channels, and various data enhancement strategies such as sliding window slicing, normalization, random cropping and disturbance are adopted to improve the adaptability of the model to high noise and complex working conditions; in view of the modal heterogeneity problem, a convolutional subnetwork with the same structure but independent weights is introduced to extract and fuse two types of modal features, thereby breaking through the limitations of traditional methods using single signal modeling.

[0008] 2) The modal attention mechanism and SEBlock channel attention mechanism are introduced to model the feature importance at the modal level and the channel level respectively, automatically learn the weight coefficients of various types of information, suppress redundancy and enhance the key feature expression ability, and improve the perception ability and discrimination ability of the model to fault features.

[0009] 3) An end-to-end trainable graph generation module is proposed, which automatically constructs a graph generation structure according to the similarity between sample fusion features, avoiding the drawbacks of traditional graph neural networks relying on manually set adjacency matrices; combined with the multi-receptive field Chebyshev graph convolution module, it can simultaneously capture local disturbance and global structure features, achieving stronger structure perception ability; this mechanism effectively enhances the robustness, generalization and engineering deployment flexibility of the model while maintaining the end-to-end trainability of the overall process, and is particularly suitable for industrial equipment fault identification in dynamic working conditions and unstable environments. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The structure block diagram of the verification process of the method of the present application is shown in the figure; Figure 2 The overall flowchart of the method of the present application is shown in the figure; Figure 3a The t-SNE visualization graph of SVM at the 24th iteration is shown in the figure; Figure 3b The t-SNE visualization graph of MLP at the 30th iteration is shown in the figure; Figure 3c The t-SNE visualization graph of RF is shown in the figure; Figure 3dt-SNE visualization plot of GRU at the 45th iteration; Figure 3e t-SNE visualization plot of Transformer at the 40th iteration; Figure 3f t-SNE visualization plot of CNN at the 56th iteration; Figure 3g t-SNE visualization plot of version one at the 36th iteration; Figure 3h t-SNE visualization plot of version two at the 40th iteration; Figure 3i t-SNE visualization plot of the model of the method at the 52nd iteration; Figure 4a Confusion matrix of SVM at the 24th iteration; Figure 4b Confusion matrix of MLP at the 30th iteration; Figure 4c Confusion matrix of RF; Figure 4d Confusion matrix of GRU at the 45th iteration; Figure 4e Confusion matrix of Transformer at the 40th iteration; Figure 4f Confusion matrix of CNN at the 56th iteration; Figure 4g Confusion matrix of version one at the 36th iteration; Figure 4h Confusion matrix of version two at the 40th iteration; Figure 4i Confusion matrix of the model of the method at the 52nd iteration. DETAILED DESCRIPTION

[0011] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0012] Reference Figure 2 , the present application is based on a multi-modal fusion end-to-end fault diagnosis recognition method, which is implemented according to the following steps: Step 1, collect the vibration signal and acoustic signal of the target device, pre-process, and construct a training sample set, First, collect the vibration signal and acoustic signal during the operation of the target device to form two-dimensional multi-modal original data, wherein the original data needs to be processed by sliding window slicing, and data enhancement methods such as normalization, random cropping and disturbance are used to improve data quality and model robustness.

[0013] Then, the two-dimensional multi-modal original data is pre-processed, including data segmentation, data enhancement, normalization processing and adding labels, specifically including: 1.1) Data segmentation, that is, the process of dividing a long sequence into fixed-length segments, the expression is: , Wherein, is the segmented segment sequence, For the original signal sequence, For the starting position of each segment in the original signal, For the sliding interval of each segmentation, For the fixed length of each output segment; 1.2) Data augmentation is performed on the segmented data, including adding random Gaussian noise, random cropping and random stretching, which are described as follows: Add random Gaussian noise, the expression is: , Where, is the value of the signal after adding random Gaussian noise at time t ; is the value of the original signal at time t ; is independent and identically distributed Gaussian noise with mean 0 and variance ; Determine the starting index of random cropping, the expression is: , Where, is the starting index of random cropping, which is an integer; Uniform distribution on the interval , that is, ; is the length of the cropping window; Implement random cropping, the expression is: , Where, is the value of the cropped signal at time t ; is the cropping interval; Set the coefficient of random stretching, the expression is: , Where, is the stretching / compression coefficient, is the stretching control parameter of the signal; Calculate the target length, the expression is: ,

[0014] Where, is the target length after stretching / compression, is the floor function, is the signal sequence after data augmentation, is the signal Resample to a new signal with length , padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal padding for the end of the signal 1.3) Set label: The data set collected by the bearing under different faults is sampled several times for different data, and the corresponding label is added to the sample according to the corresponding fault. For example, the sample label of the bearing in the healthy state is 0, the sample label of the bearing in the inner ring fault is 1, the sample label of the bearing in the outer ring fault is 2, the sample label of the bearing in the ball fault is 3, and the sample label of the bearing in the cage fault is 4.

[0015] From this, the training sample set is constructed.

[0016] Step 2, feature extraction, obtain high-dimensional modal feature vector, In view of the heterogeneous feature distribution of different modalities, the convolutional neural network (CNN) module with the same structure but independent weights is used to extract features from vibration signals and acoustic signals respectively, and two groups of high-dimensional modal feature vectors are obtained; Lightweight modal attention mechanism is used to fuse the two types of features, calculate the modal weight coefficient, realize weighted fusion based on weight, input the fused modal features into the SEBlock channel attention module, extract channel importance information and enhance key channel expression through global average pooling and channel-by-channel activation mechanism, and suppress redundant features; The specific process is, 2.1) Extract the features of acceleration and acoustic signals through two independent CNN branches, the expression is: , where, is the element of the output feature map, is the element of the output feature map, is the element of the output feature map, is the convolution kernel weight, is the convolution kernel size, is the input signal, is the bias, is the number of input channels; 2.2) Fusion and weighting of acceleration and acoustic features through modal attention and channel attention mechanisms, generating fused features and highlighting important channels, expressions are as follows: Modal attention operation is performed, and the expression is: , , , wherein, , and are the feature matrices of the vibration signal and acoustic emission signal respectively, is the feature matrix obtained by splicing operation, , is the modal attention weight matrix, , is the modal attention bias vector, is the rectified linear unit activation function, is used to determine the weight proportion of different modal features, , and are the attention weight vectors of the vibration signal and acoustic emission signal respectively, is the fused feature matrix; Channel attention operation is performed, and the expression is: , , , wherein, is the global average pooling channel, is the number of channels, is the feature of different channels in the input feature map, is the channel attention weight vector, , is the channel attention weight matrix, , is the channel attention bias vector, is the Sigmoid function, is the rectified linear unit activation function, is the feature matrix weighted by the channel attention mechanism.

[0017] Step 3, generate a sparse adjacency matrix through an end-to-end deep learning graph generation module, establish a graph generation structure relationship, the specific process is: The sparse adjacency matrix is automatically generated according to the similarity between the fused features by a graph generation module (GGL) of end-to-end deep learning, avoiding manual graphing errors, establishing the graph generation structure relationship between samples for subsequent graph convolution, and the expression is as follows: 3.1) Implement node representation, the expression is:

[0018] wherein, is the graph embedding output matrix, , , is the full connection layer weight matrix, , , is the full connection layer bias vector; 3.2) Set the similarity matrix, the expression is: , wherein, is the similarity matrix, is the transpose matrix of ; 3.3) L2 normalization, the expression is: , wherein, is the matrix obtained after normalization, is the similarity value between nodes, is the summation index, is the minimum value, ; 3.4) Implement Top-K edge selection, the expression is: , , wherein, is the similarity matrix, is the specified dimension in , the maximum k ( batch _ size ) values are selected; is the vector of the maximum k values selected; is the index vector of the maximum value in the corresponding row of ; is the edge index matrix constructed by the stacking operation; is the stacking operation, traversing 0 to , the node index and the corresponding Stacked in order, forming an index matrix representing edge connection relationships.

[0019] Step 4, construct a multi-receptive field Chebyshev graph convolution network, extract node-level features in the graph generation structure, Construct a multi-receptive field Chebyshev graph convolution network (Multi-Receptive Field GCN), use different order ChebConv graph convolution operators to extract node-level features in the graph generation structure, realize the graph domain enhancement and interaction of features, the specific process is: 4.1) According to the constructed sparse adjacency matrix, the degree matrix of the graph is calculated, the expression is: , Where, is the diagonal element of the degree matrix . 4.2) Construct a non-normalized graph Laplacian matrix, the expression is: , Where, is the non-normalized graph Laplacian matrix, is the degree matrix; 4.3) Spectral scaling, the expression is:

[0020] Where, is the matrix after spectral scaling, is the maximum eigenvalue of the non-normalized graph Laplacian matrix L , is the unit matrix; 4.4) Construct Chebyshev polynomial kernel, the expression is: , Where, , , , is the Chebyshev polynomial kernel; 4.5) The input node feature acts on the multi-order graph convolution kernel to obtain each order output, the expression is: , Where, is the first layer of the first K order output of Chebyshev graph convolution, is the Chebyshev polynomial kernel of different orders, is the feature matrix, is the first layer of the firstK Order weight matrix, F It is the input feature dimension. This is the dimension of the output features of this layer; 4.6) Concatenate the outputs of the multi-level receptive fields as the output of graph convolutional layer 1, the expression is: , in, This is the output of the first layer of the Chebyshev graph convolution. This refers to the batch size. For splicing operations; 4.7) Output from the first layer As input, repeat the graph convolution process described above, with the expression: , , in, The second layer of the Chebyshev graph convolution. K First-order output, The second layer of the Chebyshev graph convolution. K Order weight matrix, This is the output of the second layer of the Chebyshev graph convolution.

[0021] Step 5: Input the structure-aware features output by GCN into the fully connected layer for mapping, completing the prediction and discrimination of the fault category of the input sample; then use the cross-entropy loss function for supervised model training, and optimize the model parameters in an end-to-end manner. The specific process is as follows: 5.1) The classifier layer maps features to the class space and outputs the classification result; the features are then mapped from the model output dimension to the number of bottleneck layers, and then to the number of classes. The bottleneck layer is determined by the following expression: , in, This is the output matrix of the bottleneck layer. This is the weight matrix of the bottleneck layer. This is the bottleneck layer bias vector; The classification layer is determined by the following expression: , in, The output vector of the classification layer. The weight matrix of the classifier. The bias vector of the classification layer; select the highest value in the confidence score vector, as the final assessment result of this diagnosis, realize the accurate determination of the running state of rotating machinery equipment. Here, the classifier layer is used to classify faults, cross-entropy, optimizer and ReduceLROnPlateau are used to optimize the training process, and the subsequent confusion matrix, t-SNE graph, accuracy, recall rate and F1 score are used to judge the training effect of the model.

[0022] The cross-entropy loss is determined, and the expression is: , Wherein, the cross-entropy loss value is =5 is the number of categories, is the true label, is the probability of the th category predicted by the model, The output of the classification layer is converted into a probability distribution, and the prediction probability of each category is in the interval (0, 1), and the sum of all category probabilities is 1. 5.2) In the training process of step 5, the Adam optimizer is used, the learning rate scheduler is the learning rate decay ReduceLROnPlateau based on the plateau phenomenon, and the early stopping mechanism, The Adam optimizer is set, and the expression is: , , , , Among them, is the first moment estimation of the gradient at moment, is the exponential decay rate of the first moment estimation, is the first moment estimation of the gradient at moment, is the gradient vector of the model parameters at moment, which is obtained by taking the derivative of the loss function with respect to the model parameters , is the second moment estimation of the gradient at moment, is the exponential decay rate of the second moment estimation, is the second moment estimation of the gradient at moment, is the first moment estimation of after bias correction, is the first moment estimation of after bias correction, , respectively and t to the power of for The parameter vector of the time-matter model. The initial learning rate, for The updated model parameter vector at each time step. It is the minimum value; The ReduceLROnPlateau operation is performed using the following expression: , in, Let be the learning rate at time t. for Learning rate at any given moment The performance did not improve over five consecutive training rounds.

[0023] At this point, the trained fault identification model is obtained.

[0024] Furthermore, step 6 may be included after step 5.

[0025] Step 6: Perform prediction output on the test set using the fault identification model trained in Step 5. Quantitatively evaluate the fault identification results using metrics such as accuracy, confusion matrix, and t-SNE plot to obtain the fault identification results.

[0026] The fault identification model obtained in step 5 is compared with other mainstream classification models (including convolutional neural networks (CNNs) based on single-modal input, multi-channel CNN models using multimodal input and attention mechanisms, traditional machine learning methods based on manually designed statistical features—support vector machines (SVMs), and multilayer perceptrons (MLPs)) under the same dataset and training strategy. The effectiveness and superiority of the fault identification model of this invention are evaluated from dimensions such as classification performance, robustness, and generalization ability. The fault identification results are quantitatively evaluated by generating a confusion matrix, visualizing t-SNE, and using accuracy curves to assess model performance. The expression is as follows: 6.1) Determine the confusion matrix, expressed as: , in, The real label is Predicted as The number of samples, where 1 is the indicator function. It is the number of samples. For the first The true label of each sample For the first Predicted labels for each sample; 6.2) Determine the t-SNE objective function, expressed as: , , , where, is the similarity probability of samples and in high-dimensional space, is the similarity probability of samples and in low-dimensional space, is the feature vector of samples and in high-dimensional space, is the feature vector of samples and in low-dimensional space, is the Euclidean distance, is the local variance parameter of the sample, related to the perplexity, is the kernel function of t-distribution; 6.3) Calculate the accuracy, expressed as: , where, is true, the number of positive classes predicted as positive classes; is true negative, the number of negative classes predicted as negative classes; is false positive, the number of negative classes predicted as positive classes; is false negative, the number of positive classes predicted as negative classes; 6.4) Calculate the recall rate, expressed as: , 6.5) F1 score: , .

[0027] Conclusion: When the signal data of rotating machinery equipment is sent as input instances into the completed trained model, the system will perform a series of complex feature extraction and pattern recognition processes. The final product of this process is a set of probability distribution or confidence score vectors generated for a series of predefined operating states or abnormal working conditions (e.g. normal state, imbalance, misalignment, bearing failure, gear failure, etc.). These values calculated by the model, which quantitatively reflect the possibility of each state existing, constitute the quantitative indicators of the method of the invention. In order to derive clear fault diagnosis conclusions from these quantitative indicators, the Maximum A Posteriori (MAP) principle is used inside the model: the class with the highest value in the confidence score vector is selected as the final evaluation result of this diagnosis. This mechanism ensures that the model can achieve automated and accurate determination of the operating condition of rotating machinery equipment based on its deep analysis of the input data.

[0028] Experimental verification: The rolling bearing fault data set published by the University of Ottawa was used as the data source, which covers five typical states: normal state, inner ring fault, outer ring fault, rolling element (ball) fault, and cage fault. Each state contains multiple independent working condition samples, totaling 60 experimental data. All data sampling frequencies are 42000 Hz, each group of signal recording time is 10 seconds, and it contains two channels of synchronous signals, i.e. vibration signal and acoustic emission signal, which constitute a typical multi-modal original signal structure.

[0029] The original dual-channel time-domain signal is used as the direct input of the model, and the entire modeling process does not rely on any manual feature extraction operation, fully embodying the characteristics of the end-to-end learning framework. In the sample construction process, each original 10-second signal is divided into multiple sample segments by sliding window division with a fixed window length of 1024 and a step size of 1024. Each sample consists of two channel signals with a data dimension of [2, 1024], where the first channel is the vibration signal and the second channel is the acoustic emission signal. All samples are labeled with the corresponding state category to build a complete data set for subsequent training and testing.

[0030] To improve the robustness of the model under complex working conditions and the identification ability of small perturbations, the method introduces a variety of specially designed data enhancement strategies in the training stage. It should be particularly pointed out that the method always takes the original signal as the input main body, and the data enhancement only provides the disturbance amount in the training process, without changing the original authenticity of the model input source. The enhancement strategy mainly includes: introducing additive Gaussian noise to simulate environmental interference; using random cropping to simulate local information loss scenarios; enhancing the model's adaptability to speed fluctuations by randomly stretching signals; for acoustic emission signals, whether to introduce a small disturbance is controlled according to the configuration options to avoid excessive damage to the expression ability of weak acoustic features. All enhanced data are uniformly normalized before entering the model to maintain the consistency of the numerical scale between channels.

[0031] In terms of data division, the complete data set is divided into training set and test set by sample-level stratified random sampling method, and the division ratio is 70% training set and 30% test set, ensuring that the distribution of all classes in the two subsets remains consistent. The training set is mainly used for end-to-end learning of model parameters, and the test set is used to evaluate the generalization ability and discrimination performance of the model on unseen samples. In the training set, further enhancement strategies are applied to improve the fault tolerance of the model to data perturbations and structural changes, while the test set always remains original to truly reflect the performance of the model in actual application.

[0032] As Figure 1 The overall framework of the model of the method is an end-to-end multi-modal graph convolutional neural network, which uses two parallel convolutional neural networks (CNNs) to process the acceleration channel and the acoustic emission channel respectively, and independently extracts the time series features in each channel. By introducing a channel attention mechanism, the features of the two channels are weighted and adjusted to improve the representation ability of the fault-sensitive channel. On this basis, a modal attention mechanism is further introduced to weight and fuse the global information between the two modalities, automatically learn the contribution weight of different modalities to the classification task, and generate a fused feature representation. The fused multi-modal feature vector enters the graph generation structure modeling module, and the graph generation layer dynamically constructs the adjacency relationship graph between samples based on feature similarity to mine the potential class relationship and structure topology between samples. The multi-receptive field graph convolutional network extracts local and global features on the constructed graph generation structure, and realizes fault state discrimination through a fully connected classifier.

[0033] During the model training process, the cross-entropy loss function is used as the optimization objective, the Adam optimizer is used for iterative updating, and the learning rate scheduling mechanism and early stopping method are introduced to improve the training stability and prevent overfitting. The accuracy of the training set and the test set is recorded after each training, and the accuracy curve is used for dynamic analysis of training. To further verify the feature learning effect of the model, the intermediate layer features are extracted during training and the t-SNE algorithm is used for dimensionality reduction visualization to observe the clustering trend of different class samples in the embedding space.

[0034] During the experiment verification, in order to comprehensively evaluate the effectiveness and advantages of the method, several existing deep learning and traditional machine learning models are selected as comparative baselines, including support vector machine (SVM), multilayer perceptron (MLP), convolutional neural network (CNN), gated recurrent unit network (GRU), and Transformer structure, etc. The same data input structure and preprocessing process are used for all existing technology comparison models, and training and testing are carried out under the same training and testing division to ensure the effectiveness and comparability of the experimental results.

[0035] The above models are all trained on the same training set and evaluated on the same test set. In order to comprehensively measure the performance of the model, multi-class indicators are used for comparative analysis. The accuracy reflects the correctness of the overall classification of the model; the precision measures the credibility of the model in predicting a certain class, and evaluates its ability to avoid false positives; the recall rate represents the recognition coverage of the model for certain samples, and is used to measure the false negative situation; the F1 score is the harmonic mean of precision and recall, which comprehensively reflects the accuracy and integrity of classification. All experimental results are shown in Tables 1, 2 and 3, the method is referred to as MAGNet, and the comparison unit is %.

[0036] Table 1, classification accuracy of the method and other models under each fault state

[0037] Table 2, recall rate of the method and other models under each fault state

[0038] Table 3, F1 score of the method and other models under each fault state

[0039] The t-SNE dimension reduction visualization map maps the high-dimensional features extracted by the model to a two-dimensional space, which can intuitively show the aggregation degree (intra-class compactness) of the same samples and the separation degree (inter-class separability) between different class samples, thereby reflecting the ability of the model to depict the structure of the feature space. The confusion matrix details the correspondence between the true label and the predicted label of each class of samples, facilitating the analysis of the misjudgment of the model in specific categories, identifying the category pair that is easy to confuse, providing a direction for subsequent model optimization, and the experimental results are shown in Figures 3a-3i and Figures 4a-4i . Figures 4a-4i For showing the discrimination ability of different models in multi-class recognition tasks, the horizontal axis is the real label and the vertical axis is the predicted label. It can be seen that the recognition accuracy of the method of the application is higher in most categories, and the misclassification rate is significantly lower than that of the comparative models, which reflects the recognition robustness and generalization ability of the method of the application in complex working conditions. The specific explanations are as follows: 1) Compared with traditional machine learning methods: three existing technologies, support vector machine (SVM), multilayer perceptron (MLP), and random forest (RF), were selected, and the experimental results corresponded to Figure 3a , Figure 3b , Figure 3c and the corresponding Figure 4a , Figure 4b , Figure 4c . Among them, Figure 3a is the t-SNE visualization result map of the 24th iteration of SVM in the feature space of the test sample; Figure 3b is the t-SNE visualization result map of the 30th iteration of MLP in the feature space of the test sample; Figure 3c is the t-SNE visualization result map of RF in the feature space of the test sample; Figure 4a is the confusion matrix of SVM in the 24th iteration in the fault classification task; Figure 4b is the confusion matrix of MLP in the 30th iteration in the fault classification task; Figure 4c is the confusion matrix of RF in the fault classification task.

[0040] 2) Compared with the deep learning method based on time series modeling: three existing technologies, gated recurrent unit (GRU), Transformer structure, and convolutional neural network (CNN), were selected, and the experimental results corresponded to Figure 3d , Figure 3e , Figure 3f and Figure 4d , Figure 4e , Figure 4f , wherein Figure 3d is the t-SNE visualization result map of the 45th iteration of GRU in the feature space of the test sample; Figure 3eFig. 6 is a t-SNE visualization result diagram of the CNN at the 56th iteration in the feature space of the test sample; Figure 3f Fig. 6 is a t-SNE visualization result diagram of the CNN at the 56th iteration in the feature space of the test sample; Figure 4d Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task; Figure 4e Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task; Figure 4f Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task.

[0041] 3) The model of the method is deformed to obtain two versions: version one is a single-branch convolutional neural network structure based on single-modal input; version two is a double-parallel convolutional channel for multi-modal feature extraction, but does not introduce an attention mechanism, and the experimental results correspond to Figure 3g , Figure 3h and Figure 4g , Figure 4h wherein, Figure 3g Fig. 6 is a t-SNE visualization result diagram of the CNN at the 56th iteration in the feature space of the test sample; Figure 3h Fig. 6 is a t-SNE visualization result diagram of the CNN at the 56th iteration in the feature space of the test sample; Figure 4g Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task; Figure 4h Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task. The complete model of the method is called MAGNet, and the experimental results correspond to Figure 3i and Figure 4i wherein, Figure 3i Fig. 6 is a t-SNE visualization result diagram of the CNN at the 56th iteration in the feature space of the test sample; Figure 4i Fig. 7 is a confusion matrix of the GRU at the 45th iteration in the fault classification task.

[0042] Through the above indexes and visualization means, the model performance is quantitatively and qualitatively evaluated from multiple angles, not only revealing the differences between the models in overall accuracy, but also showing the advantages and disadvantages in class discrimination accuracy, feature expression ability and fine-grained classification. The experimental results show that the model constructed by the method is superior to the traditional method in multi-modal information integration, feature extraction and complex working condition adaptability, and exhibits high recognition accuracy and generalization ability, and is suitable for the field of industrial equipment health monitoring and fault diagnosis.

[0043] Embodiment 1 The experimental object of embodiment 1 is a bearing health state vibration signal and acoustic emission signal data set, and the data set samples correspond to the normal operation state of the bearing.

[0044] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample quantity is 410. The determination time is 1.9580 seconds, 392 samples are determined as the healthy state, and the accuracy is 95.60%.

[0045] Example 2 The experimental object of example 2 is the vibration signal and acoustic emission signal data set under the bearing inner ring fault (developing) state, and the data set sample corresponds to the bearing normal operation state.

[0046] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample quantity is 410. The determination time is 1.9352 seconds, 408 samples are determined as the inner ring fault state, and the accuracy is 99.51%.

[0047] Example 3 The experimental object of example 3 is the vibration signal and acoustic emission signal data set under the bearing inner ring fault state, and the data set sample corresponds to the bearing normal operation state.

[0048] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample quantity is 410. The determination time is 2.3696 seconds, 404 samples are determined as the inner ring fault state, and the accuracy is 98.53%.

[0049] Example 4 The experimental object of example 4 is the vibration signal and acoustic emission signal data set under the bearing outer ring fault (developing) state, and the data set sample corresponds to the bearing normal operation state.

[0050] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample quantity is 410. The determination time is 2.2260 seconds, 409 samples are determined as the inner ring fault state, and the accuracy is 99.75%.

[0051] Example 5 The experimental object of embodiment 5 is a vibration signal and acoustic emission signal data set under the bearing outer ring fault state, and each sample of the data set corresponds to the normal operation state of the bearing.

[0052] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000 Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample number is 410. The determination time is 2.0802 seconds, 408 samples are determined as the inner ring fault state, and the accuracy is 99.51%.

[0053] Embodiment 6 The experimental object of embodiment 6 is a vibration signal and acoustic emission signal data set under the bearing rolling body fault state, and each sample of the data set corresponds to the normal operation state of the bearing.

[0054] The data set is classified and verified by using the method of the application, and the specific parameter settings are as follows: the sampling frequency is 42,000 Hz, the feature extraction window size is 1024 sampling points, the model inference batch size is 128, the feature dimension is 256, and the sample number is 410. The determination time is 2.0802 seconds, 408 samples are determined as the inner ring fault state, and the accuracy is 99.51%.

[0055] The experimental results of the above six embodiments show that when the vibration signal and acoustic emission signal data set under different health states is classified and verified by using the method of the application, under the parameter settings of a sampling frequency of 42,000 Hz, a feature extraction window size of 1024 sampling points, a model inference batch size of 128, and a feature dimension of 256, the determination time of 410 different sample sets is in the interval of 1.9352 seconds-2.3696 seconds, and the classification accuracy is distributed in the range of 95.60%-99.75%. The experimental results show that the method of the application can effectively identify the health state and various typical fault states (including the fault development state), has high classification accuracy and efficiency, and has effectiveness and reliability.

Claims

1. An end-to-end fault diagnosis and identification method based on multi-modal fusion, characterized in that, The following steps are implemented: Step 1, collect vibration signals and acoustic signals, preprocess, and construct a training sample set; Step 2, feature extraction, obtain high-dimensional modal feature vector; Step 3, through the graph generation module of end-to-end deep learning, generate sparse adjacency matrix and establish graph generation structure relationship; Step 4, construct multi-receptive field Chebyshev graph convolution network, extract node-level features in graph generation structure; Step 5, input structure perception features into fully connected layer for mapping, complete prediction and discrimination of input sample belonging to fault category; then perform model supervised training, optimize model parameters through end-to-end method; Step 6, predict the output of the fault recognition model on the test set, and quantitatively evaluate the fault recognition results to obtain the fault recognition results.

2. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, characterized in that, In step 1, the specific process is: First, collect the vibration signals and acoustic signals during the operation of the target device to form two-dimensional multi-modal original data, wherein the original data is processed by sliding window slicing, and data enhancement methods such as normalization, random cropping and perturbation are used; Then, preprocess the two-dimensional multi-modal original data, including data segmentation, data enhancement, normalization processing and adding labels, which specifically includes: 1.1) Data segmentation, that is, the process of dividing long sequences into fixed length segments, the expression is: ， wherein, is the sequence of the segmented segments, is the original signal sequence, is the starting position of each segment in the original signal, is the sliding interval of each segmentation, is the fixed length of each output segment; 1.2) Data enhancement is added to the segmented data, random Gaussian noise, random cropping and random stretching, which are described as follows: Add random Gaussian noise, the expression is: ， wherein, is the value of the signal after adding random Gaussian noise at time t ; is the value of the original signal at time t ; is independent and identically distributed Gaussian noise with mean 0 and variance ; Determine the starting index of random cropping, the expression is: ， wherein is a starting index for the random crop, is an integer; a uniform distribution over the interval , i.e. ; is the length of the crop window; Implement random cropping, the expression is: ， wherein, is the value of the clipped signal at time t ; is the clipping interval; Set the random stretching coefficient, the expression is: ， wherein, is the stretch / compression coefficient, is the stretch control parameter of the signal; Calculate the target length, the expression is: ， where, is the target length after stretching / compression, is the floor function, is the signal sequence after data augmentation, is the signal resample to a new signal of length , pad the end of the signal with elements of value 0 for zero padding when the length is insufficient; is the array slicing operation, take the first elements for truncation when the length is too long; Normalize the mean and standard deviation of all data after data enhancement, the expression is: ， in, For the first The signal sequence after channel normalization For the first A signal sequence of each channel, The mean, Standard deviation; 1.3) Set the label: the data set collected by the bearing under different faults, sample different data multiple times, and add corresponding labels to the samples according to the corresponding fault, to obtain the training sample set.

3. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 2, characterized in that, In step 1.3), the specific way of setting the label is: the bearing sample label under healthy state is 0, the bearing sample label under inner ring fault is 1, the bearing sample label under outer ring fault is 2, the bearing sample label under ball fault is 3, and the bearing sample label under cage fault is 4.

4. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, characterized in that, In step 2, the specific process is: 2.1) Extract the features of acceleration and acoustic signals through two independent CNN branches, the expression is: ， wherein, is the output feature map, is the first element of the output feature map, is the first element of the output feature map, is the convolution kernel weight, is the convolution kernel size, is the input signal, is the bias, is the number of input channels; 2.2) Perform modal attention and channel attention mechanisms on acceleration and acoustic features to generate fusion features and highlight important channels, which are described as follows: Perform modal attention operation, the expression is: ， ， ， wherein, , are the feature matrices of the vibration signal and the acoustic emission signal respectively, is the feature matrix obtained by splicing operation, , is the modal attention weight matrix, , is the modal attention bias vector, is the rectified linear unit activation function, is used to determine the weight proportion of different modal features, , are the attention weight vectors of the vibration signal and the acoustic emission signal respectively, is the fused feature matrix; Perform channel attention operation, the expression is: ， ， ， wherein, is the global average pooling channel, is the number of channels, is the feature of different channels in the input feature map, is the channel attention weight vector, , is the channel attention weight matrix, , is the channel attention bias vector, is the Sigmoid function, is the rectified linear unit activation function, is the feature matrix weighted by the channel attention mechanism.

5. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, characterized in that, In step 3, the specific process is: Through the graph generation module of end-to-end deep learning, generate sparse adjacency matrix according to the similarity between fusion features, and establish the graph generation structure relationship between samples, the expression is as follows: 3.1) Implement node representation, the expression is: wherein, is the graph embedding output matrix, , , is the fully connected layer weight matrix, , , is the fully connected layer bias vector; 3.2) Set the similarity matrix, the expression is: ， wherein, is a similarity matrix, is the transpose matrix of 3.3) L2 normalization, the expression is: ， wherein, is the normalized matrix, is the similarity value between nodes and nodes, is the summation index, is the minimum value, ; 3.4) Implement Top-K edge selection, the expression is: ， ， wherein, is the similarity matrix, is the specified dimension of , and the largest k values are selected; is the vector of the largest k values selected; is the index vector of the largest values in corresponding row; is the edge index matrix constructed by the stacking operation; is the stacking operation, iterating from 0 to , the node index and the corresponding are stacked in order to form the index matrix representing the edge connection relationship.

6. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, characterized in that, In step 4, the specific process is: In step 4, the specific process is: 4.1) According to the constructed sparse adjacency matrix, the degree matrix of the graph is calculated, and the expression is: ， wherein are the diagonal elements of the degree matrix are the diagonal elements of the degree matrix 4.2) Construct a non-normalized graph Laplacian matrix, and the expression is: ， wherein, is a non-normalized graph Laplacian matrix, is a degree matrix; 4.3) Implement spectral scaling, and the expression is: wherein, is the spectrum scaled matrix, is the non-normalized graph Laplacian matrix L is the largest eigenvalue of the matrix is the identity matrix; 4.4) Construct the Chebyshev polynomial kernel, and the expression is: ， wherein , , , is a Chebyshev polynomial kernel; 4.5) Input node features are combined with the multi-order graph convolution kernel to obtain each order output, and the expression is as follows: with the multi-order graph convolution kernel to obtain each order output, and the expression is as follows: ， wherein, is the 1st layer of Chebyshev graph convolution, K output of order, are Chebyshev polynomial kernels of different orders, is the feature matrix, is the 1st layer of Chebyshev graph convolution, K weight matrix of order, F is the input feature dimension, is the output feature dimension of the layer; 4.6) Concatenate the multi-order receptive field output as the output of the graph convolution layer 1, and the expression is: ， wherein, is the Chebyshev graph convolution layer 1 output, is the batch size, is the concatenation operation; 4.7) output in the first layer As input, repeat the above graph convolution process, the expression is: ， ， wherein, is the 2nd layer output of the Chebyshev graph convolution, K is the 2nd layer output of the Chebyshev graph convolution, is the 2nd layer output of the Chebyshev graph convolution, K is the 2nd layer output of the Chebyshev graph convolution, is the 2nd layer output of the Chebyshev graph convolution.

7. The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, characterized in that, In step 5, the structure-aware features output by the GCN are input into the fully connected layer for mapping, and the prediction and discrimination of the fault class to which the input sample belongs are completed; then the cross-entropy loss function is used for model supervision training, and the model parameters are optimized through an end-to-end manner, and the specific process is: 5.1) Map the features to the class space through the classifier layer to output the classification results; map the features from the model output dimension to the number of bottleneck layers, and then to the number of classes, as follows: Determine the bottleneck layer, and the expression is: ， wherein, is an output matrix of the bottleneck layer, is a weight matrix of the bottleneck layer, is a bias vector of the bottleneck layer; Determine the classification layer, and the expression is: ， wherein, is an output vector of the classification layer, is a weight matrix of the classifier, is a bias vector of the classification layer; the class with the highest value in the confidence score vector is selected as the final evaluation result of this diagnosis, so as to realize accurate determination of the running state of the rotating machinery equipment. Determine the cross-entropy loss, and the expression is: ， where Lceis the cross-entropy loss value, = 5 is the number of classes, is the true label, is the probability of the model predicting the th class, converts the classification layer output into a probability distribution, with the predicted probability of each class in the interval (0, 1) and the sum of all class probabilities being 1. 5.2) In the training process, use the Adam optimizer, the learning rate scheduler is the learning rate decay ReduceLROnPlateau based on the plateau phenomenon, and the early stopping mechanism, as follows: Set the Adam optimizer, and the expression is: ， ， ， ， in, for First-moment estimation of the gradient at time intervals. The exponential decay rate is estimated by the first moment. for First-moment estimation of the gradient at time intervals. for The gradient vector of the model parameters at each time step is obtained by applying the loss function. Regarding model parameters Differentiation yields, for Second-moment estimation of the gradient at time intervals. The exponential decay rate is estimated by the second moment. for Second-moment estimation of the gradient at time intervals. right First-order moment estimation after bias correction right First-order moment estimation after bias correction , They are respectively and t to the power of for The parameter vector of the time-matter model. The initial learning rate, for The updated model parameter vector at each time step. It is the minimum value; Implement the ReduceLROnPlateau operation, and the expression is: ， wherein, is the learning rate at time t, is is the learning rate at time t, is the trained fault identification model when the performance does not improve in the last 5 training rounds. 8.The end-to-end fault diagnosis and identification method based on multi-modal fusion according to claim 1, wherein, In step 6, the specific process is: Use accuracy, confusion matrix, t-SNE graph, and other indicators to quantitatively evaluate the fault recognition results, and obtain the fault recognition results, 6.1) Determine the confusion matrix, and the expression is: ， wherein is the number of samples for which the true label is predicted to be is the number of samples for which the true label is is the indicator function, is the number of samples, is the true label for the is the predicted label for the sample. 6.2) Determine the t-SNE objective function, and the expression is: ， ， ， wherein, is the similarity probability of the sample in the high-dimensional space, is the similarity probability of the sample in the low-dimensional space, is the similarity probability of the sample in the low-dimensional space, is the feature vector of the sample in the high-dimensional space, is the feature vector of the sample in the low-dimensional space, is the feature vector of the sample in the low-dimensional space, is the Euclidean distance, is the local variance parameter of the sample, which is related to the perplexity, is the kernel function of t-distribution; 6.3) Calculate the accuracy, and the expression is: ， wherein, true positives, the number of positive classes predicted as positive classes; true negatives, the number of negative classes predicted as negative classes; false positives, the number of negative classes falsely predicted as positive classes; false negatives, the number of positive classes predicted as negative classes; 6.4) Calculate the recall rate, and the expression is: ， 6.5) F1 score: ， ， Combine the classification details presented by the confusion matrix, the feature distribution reflected by the t-SNE graph, and the class discrimination degree, and then quantitatively evaluate the model performance from different dimensions through accuracy, recall rate, and F1 score.

Images