Fault diagnosis method based on multi-modal multi-scale feature fusion

Abstract

The invention provides a fault diagnosis method based on multi-modal multi-scale feature fusion, and the method comprises the steps: collecting signal sources when faults occur at different positions of an aero-engine bearing through a sensor, the signal sources including but not limited to vibration signals, sound signals, temperature, humidity and current; performing unified de-noising processing on the vibration signal to obtain an initial one-dimensional signal X0; performing one-dimensional feature extraction on the initial one-dimensional signal X0 through an MCNN module; performing image conversion on the initial one-dimensional signal X0 to obtain an initial two-dimensional image T1, and performing two-dimensional feature extraction on the initial two-dimensional image T1; fusing the extracted one-dimensional features with the extracted two-dimensional features to generate a multi-modal feature H1; and inputting the fused multi-modal feature H1 into an LSTM network for fault diagnosis and classification. According to the method, the fused multi-modal features are input into the LSTM classification module for fault diagnosis and classification, so that fault detection of the aero-engine bearing is realized.

Description

Technical Field

[0001] The present invention relates to the field of fault diagnosis, and in particular to a fault diagnosis method based on multi-modal and multi-scale feature fusion. Background Art

[0002] With the continuous development of artificial intelligence, deep learning technology has been widely used in the field of fault diagnosis, such as using convolutional neural networks (CNN) to process image data to effectively extract feature information; using recurrent neural networks (LSTM) to process time series data to analyze and predict fault data.

[0003] The application of deep learning technology means that fault diagnosis no longer relies on a large amount of domain knowledge and experience. It can diagnose faults under different equipment models or parameters and has strong adaptability and intelligence.

[0004] In the existing technology, when performing fault diagnosis, one-dimensional signals are often converted into images and input into neural networks for fault diagnosis. However, this processing method may lose some important information of the signal, affecting the accuracy of subsequent fault diagnosis. The noise in the one-dimensional signal will become more obvious and complex after being converted into an image, resulting in reduced accuracy of the model. Extracting features directly from one dimension cannot extract the deep features of the signal, resulting in low accuracy of the model. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a fault diagnosis method based on multimodal and multi-scale feature fusion, by fusing one-dimensional multi-scale features and two-dimensional features into multimodal features, a multimodal image fusion module, and inputting the fused multimodal features into an LSTM classification module for fault diagnosis and classification, thereby improving the accuracy of aircraft engine bearing fault diagnosis tasks.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a fault diagnosis method based on multi-modal and multi-scale feature fusion, the method comprising:

[0007] The sensors are used to collect signal sources when faults occur at different positions of the aircraft engine bearing, including but not limited to: vibration signals, acoustic signals, temperature, humidity, and current;

[0008] Performing denoising on the signal source to obtain an initial one-dimensional signal X0;

[0009] For the initial one-dimensional signal X0, one-dimensional feature extraction is performed through the MCNN module;

[0010] Perform image conversion on the initial one-dimensional signal X0 to obtain an initial two-dimensional image T1, and perform two-dimensional feature extraction on the initial two-dimensional image T1;

[0011] The extracted one-dimensional features are fused with the two-dimensional features to generate multimodal features H1;

[0012] The fused multimodal feature H1 is input into the LSTM network for fault diagnosis and classification.

[0013] Preferably, the initial one-dimensional signal X0 is subjected to one-dimensional feature extraction by the MCNN module, comprising:

[0014] The one-dimensional feature extraction is divided into two branches. In the first branch, the initial one-dimensional signal X0 is convolved with 20*1, 10*1, and 1*1 convolution kernels, and then processed with 2*2 global pooling to obtain a one-dimensional feature F1:

[0015] F1=GMP(C 1*1 (C 10*1 (C 20*1 (X0))));

[0016] Among them, GMP represents the global maximum pooling, C 20*1 , C 10*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively;

[0017] In the second branch, the initial one-dimensional signal X0 is convolved by 9*1 and 7*1 convolution kernels, then by 2*2 global pooling, then by 5*1, 5*1, and 1*1 convolution kernels, and finally by 2*2 global pooling to obtain a one-dimensional feature F2:

[0018] F2=GMP(C 1*1 (C 5*1 (C 5*1 (GMP(C 7*1 (C 9*1 (X0)))))));

[0019] Among them, GMP represents the global maximum pooling, C 9*1 , C 7*1 , C 5*1 , C 5*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively.

[0020] Preferably, performing image conversion on the initial one-dimensional signal X0 to obtain the initial two-dimensional image T1 includes:

[0021] The image processing module uses GASF, GADF, and RP to convert the initial one-dimensional signal X0 into three corresponding different images X1, X2, and X3. After fusion, the initial two-dimensional image T1 is obtained, which is expressed as:

[0022] X1,X2,X3,=GASF(X0),GADF(X0),RP(X0);

[0023] T1 = ImageFusion(X1,X2,X3);

[0024] Among them, ImageFusion stands for image fusion, GASF stands for Gram angle sum field processing, GADF stands for Gram angle difference field processing, RP stands for recursive graph processing, and T1 is the initial two-dimensional image after fusion.

[0025] Preferably, the extracting two-dimensional features from the initial two-dimensional image T1 includes:

[0026] The initial two-dimensional image T1 first enters the 3*3 convolution kernel for convolution calculation, then enters the 3*3 global pooling for global pooling processing, then enters the residual block with a convolution kernel size of 3*3 for calculation, then enters the 3*3 global pooling for a global pooling processing, then enters the ECA attention mechanism module, and then enters a residual block with a convolution kernel size of 3*3 for calculation, and finally performs a global pooling processing through the 3*3 global pooling to obtain the two-dimensional feature F3:

[0027] F3=GMP(Rebl(ECA(GMP(Rebl(GMP(C 3*3 (T1)))))));

[0028] Among them, GMP represents the global maximum pooling, C 3*3 is a 3*3 convolution kernel, Rebl is a residual block with a convolution kernel size of 3*3, ECA is the attention mechanism, and T1 is the initial two-dimensional image.

[0029] Preferably, the step of fusing the extracted one-dimensional features with the two-dimensional features to generate the multimodal feature H1 includes:

[0030] The extracted one-dimensional features F1 and F2 are multiplied element by element with the two-dimensional feature F3 to generate the multimodal feature H1:

[0031] H1=mul(F1,F2,F3);

[0032] Among them, mu l represents the element-by-element product operation of F1, F2, and F3, and the elements of F1, F2, and F3 at the same position are multiplied, and H1 represents the fused multimodal features.

[0033] Preferably, the step of inputting the fused multimodal feature H1 into the LSTM network for fault diagnosis and classification includes:

[0034] R=Softmax(Dense(Dropout(LSTM(LSTM(H1)))));

[0035] Among them, H1 represents the fused multimodal features, LSTM represents the LSTM network, the Dropout layer receives the output of the LSTM layer as input, Dense represents a fully connected layer, which receives the output from the Dropout layer as input, Softmax represents converting the output of the Dense layer into a probability distribution, ensuring that all output values ​​range between 0 and 1 and their sum is 1, and R represents the probability value of the fault.

[0036] Preferably, after the model is built, the mean square error loss is used to build the loss function, and the calculation formula is:

[0037]

[0038] Where: n samples Represents the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample, represents the square of the difference between the true value and the predicted value of the i-th sample, and the right side of the equation represents the average of the sum of the differences between the true value and the predicted value of all samples.

[0039] Preferably, the method further comprises: after the model is constructed, calculating the accuracy and recall of the model, and obtaining an F1-Score value according to the accuracy and recall;

[0040]

[0041] Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample;

[0042]

[0043] Among them, Recall represents the recall rate of the model, TP represents the true positive examples in the sample, and FN represents the false negative examples in the sample;

[0044]

[0045] Among them, F1Score represents the F1-Score value of the model, Acc represents the accuracy of the model, and Recall represents the recall rate of the model.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a fault diagnosis method based on multimodal and multiscale feature fusion, which fuses one-dimensional features and two-dimensional features into multimodal features, inputs the fused multimodal features into an LSTM classification module to perform fault diagnosis and classification, and realizes the detection of aircraft engine bearing faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only preferred embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 is a HMLC model diagram for bearing fault diagnosis provided by an embodiment of the present invention;

[0050] Figure 2 is a flow chart of the operation of the HMLC model for bearing fault diagnosis provided by an embodiment of the present invention;

[0051] Figure 3 It is a one-dimensional feature extraction model diagram of MCNN provided by an embodiment of the present invention;

[0052] Figure 4 is an image conversion model diagram provided by an embodiment of the present invention;

[0053] Figure 5 is a Rebl residual model diagram provided by an embodiment of the present invention;

[0054] Figure 6 It is a model diagram of the ECA attention mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The enumerated embodiments are only used to explain the present invention and are not used to limit the scope of the present invention.

[0056] Embodiment 1

[0057] Figure 1 The HMLC model diagram for bearing fault diagnosis provided in the first embodiment of the present invention is as follows: Figure 1As shown in the figure, the bearing fault diagnosis model consists of two parts: the GCNN multimodal feature extraction module and the LSTM classification module. The GCNN multimodal feature extraction module includes the MCNN module, the image conversion module, the Conv convolution kernel, the Pooling global maximum pooling module, the ECA attention mechanism, the Rebl residual block, and the Feature Fusion image fusion module, which are used to extract one-dimensional and two-dimensional features; the LSTM classification module is used for fault classification.

[0058] Figure 2 The operation flow chart of the HMLC model for bearing fault diagnosis provided in the first embodiment of the present invention is as follows: Figure 2 As shown in the figure, after preprocessing, the signal source enters the MCNN module and the image conversion module respectively. The signal source enters the MCNN module to obtain one-dimensional features; the signal source entering the image conversion module enters the image fusion module and the two-dimensional feature extraction module in turn to obtain two-dimensional features. The last one-dimensional feature is combined with the two-dimensional feature and enters the multimodal feature fusion module for fusion, and finally passes through the LSTM classification module for fault diagnosis and classification.

[0059] Embodiment 1 of the present invention provides a fault diagnosis method based on multi-modal and multi-scale feature fusion, including:

[0060] S1: The sensor collects the signal source when a fault occurs at different positions of the aircraft engine bearing.

[0061] Specifically, aircraft engine bearing failures mainly include: pitting damage to the inner and outer rings or rolling elements, improper bearing installation, or other failures caused by poor lubrication and maintenance. Both types of failures are accompanied by changes in vibration signals.

[0062] Pitting damage to the inner and outer rings or rolling elements is usually caused by repeated exposure to large and changing stresses. When a bearing is damaged by pitting, it usually generates strong vibration signals, acoustic signals and temperature changes.

[0063] Other failure modes caused by improper bearing installation or poor lubrication and maintenance. For example, when the lubricating oil is insufficient, the bearing burns, poor lubrication causes direct contact between the bearing surfaces or excessive contact caused by foreign objects falling, the bearing is damaged due to improper assembly, the inner ring is broken, and the inner and outer rings and cages are crushed.

[0064] Signal sources include but are not limited to: vibration signals, sound signals, temperature, humidity, current, etc.

[0065] S2: De-noise the signal source to obtain the initial one-dimensional signal X0.

[0066] Taking vibration signals as an example, the sensor may mix other signals during the acquisition process, causing interference to the waveform. Different denoising methods can be used according to the characteristics of the noise components, such as signal denoising methods based on improved median filters and wavelet packet technology. The specific steps include:

[0067] Firstly, the window width of the median filter is calculated according to the sampling frequency, so that the impulse noise and part of the white noise can be effectively filtered out. Secondly, the improved adaptive wavelet packet denoising technology is used to remove the residual white noise. Finally, a useful vibration signal is obtained.

[0068] S3: For the initial one-dimensional signal X0, one-dimensional feature extraction is performed through the MCNN module.

[0069] Figure 3 The MCNN one-dimensional feature extraction model diagram provided in the first embodiment of the present invention is as follows: Figure 3 As shown in the figure, the one-dimensional feature extraction is divided into two branches. In the first branch, the initial one-dimensional signal X0 is convolved with 20*1, 10*1, and 1*1 convolution kernels, and then processed by 2*2 global pooling to obtain the one-dimensional feature F1:

[0070] F1=GMP(C 1*1 (C 10*1 (C 20*1 (X0))));

[0071] Among them, GMP represents the global maximum pooling, C 20*1 , C 10*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively.

[0072] Convolution is a mathematical operation, which is essentially the same as subtraction, addition, multiplication and division. When processing an image, the convolution kernel is used to process a small area of ​​the input image. The weighted average of the pixels in the input image becomes each corresponding pixel in the output image, where the weight is defined by a function called the convolution kernel. The role of the convolution kernel is filtering and feature extraction. The original image can extract certain specified features of the image through mathematical operations with the convolution kernel. For the extraction of a certain feature, if it passes through different convolution kernels, the effect after convolution will be different.

[0073] Convolutional neural networks can solve regression and classification problems, but common convolutional neural networks must finally achieve classification through a fully connected layer, which often results in a surge in the number of neurons and the amount of computation in the fully connected layer, which often brings some side effects, especially for networks with high regression requirements. Therefore, after the convolution kernel is processed, the convolution result is changed to global maximum pooling.

[0074] The global maximum pooling (GMP) operation extracts the maximum value of each channel in the feature map, thereby obtaining a one-dimensional vector with the same number of channels. Each element of this vector represents the maximum activation value on the corresponding channel, so it can be regarded as a global feature representation of the channel.

[0075] The advantage of global maximum pooling (GMP) is that it can highlight the most significant information in the feature map, thereby alleviating the overfitting phenomenon to a certain extent. In addition, since the global maximum pooling operation is performed independently on each channel, it has a certain spatial invariance, that is, it has a certain robustness to the spatial transformation of the input.

[0076] In the second branch, the initial one-dimensional signal X0 is convolved by 9*1 and 7*1 convolution kernels, then by 2*2 global pooling, then by 5*1, 5*1, and 1*1 convolution kernels, and finally by 2*2 global pooling to obtain a one-dimensional feature F2:

[0077] F2=GMP(C 1*1 (C 5*1 (C 5*1 (GMP(C 7*1 (C 9*1 (X0)))))));

[0078] Among them, GMP represents the global maximum pooling, C 9*1 , C 7*1 , C 5*1 , C 5*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively.

[0079] S4: Perform image conversion on the initial one-dimensional signal X0 to obtain an initial two-dimensional image T1, and perform two-dimensional feature extraction on the initial two-dimensional image T1.

[0080] Figure 4 The image conversion model diagram provided by the first embodiment of the present invention is as follows: Figure 4 As shown, the image processing module uses GASF, GADF, and RP to convert the initial one-dimensional signal X0 into three corresponding different images X1, X2, and X3, and the initial two-dimensional image T1 is obtained after fusion;

[0081] X1,X2,X3,=GASF(X0),GADF(X0),RP(X0);

[0082] T1 = ImageFusion(X1,X2,X3);

[0083] Among them, ImageFusion stands for image fusion, GASF stands for Gram angle sum field processing, GADF stands for Gram angle difference field processing, RP stands for recursive graph processing, and T1 is the initial two-dimensional image after fusion.

[0084] It should be noted that GASF and GADF are both analysis methods for time series, which are used to convert time series into two-dimensional images.

[0085] GASF represents a pixel as the angle sum between the values ​​of adjacent time points to form a two-dimensional image. GADF is similar to GASF, but GADF uses the angle difference. These two methods can convert time series into two-dimensional images, so as to use image processing technology for feature extraction and classification.

[0086] GASF is a time-frequency analysis method based on wavelet analysis. It decomposes the signal into multiple sub-bands by applying wavelet transform and performs spectral analysis on each sub-band to obtain analysis results in the time-frequency domain. GASF has a better effect on the time-frequency analysis of stationary signals.

[0087] GADF is a time-frequency analysis method based on the autoregressive model. By decomposing the signal into multiple sub-bands and fitting the autoregressive model on the sub-bands, the analysis results in the time-frequency domain are obtained. GADF has a good effect on the time-frequency analysis of non-stationary signals.

[0088] RP is an image representing the distance between trajectories extracted from the original time series. It is an important method for analyzing the periodicity, chaos and non-stationarity of time series. It can reveal the internal structure of time series and provide prior knowledge about similarity, information content and predictability. RP is suitable for short time series data and can test the stationarity and internal similarity of time series.

[0089] Furthermore, the initial two-dimensional image T1 first enters the 3*3 convolution kernel for convolution calculation, then enters the 3*3 global pooling for global pooling processing, then enters the residual block with a convolution kernel size of 3*3 for calculation, and then enters the 3*3 global pooling for a global pooling processing, then enters the ECA attention mechanism module, and then enters a residual block with a convolution kernel size of 3*3 for calculation, and finally performs a global pooling processing through the 3*3 global pooling to obtain the two-dimensional feature F3:

[0090] F3=GMP(Rebl(ECA(GMP(Rebl(GMP(C 3*3 (T1)))))));

[0091] Among them, GMP represents the global maximum pooling, C 3*3is a 3*3 convolution kernel, Rebl is a residual block with a convolution kernel size of 3*3, ECA is the attention mechanism, and T1 is the initial two-dimensional image.

[0092] Figure 5 The Rebl residual model diagram provided in the first embodiment of the present invention is as follows: Figure 5 As shown in Figure 1, there are two main paths in this residual block: the main path and the short-circuit connection. The main path contains two 3*3 convolution kernels. The output signal is obtained by adding the input signal and the convolution signal.

[0093] Residual learning proposes to train the model by learning the residual between features, that is, the difference between the actual output and the expected output. Compared with directly learning the original mapping, residual learning is easier to optimize and can effectively solve the gradient problem.

[0094] The residual block implements residual learning by introducing skip connections. Its basic structure is that the input data passes through two branches, one is the identity mapping branch and the other is the nonlinear mapping branch, and the two are added together as the output of the residual block.

[0095] The design of the residual block enables the network to maintain better performance when it is deeper, which is beneficial to the training and generalization ability of the model.

[0096] Figure 6 The ECA attention mechanism model diagram provided in the first embodiment of the present invention is as follows: Figure 6 As shown in Figure 2, the ECA attention mechanism consists of a global average pooling layer (GAP) and a 3*1 convolution kernel with a fixed convolution kernel, as well as a sigmoid activation layer. The inputs are the feature maps GMP (C 3*3 (T1)), with shape (B,H,W,C), where B is the batch size, H and W are the height and width of the feature map respectively, and C is the number of channels.

[0097] First, GAP compresses the spatial dimension of the input feature map into the channel dimension, and obtains an output with a shape of (B, C). Then, by expanding the dimension of this output twice, its shape becomes (B, 1, 1, C) to meet the input requirements of the 1D convolution layer. Next, the convolution kernel performs a convolution operation on the expanded feature map to capture local cross-channel interaction information. Then, the sigmoid activation layer maps the output of the convolution layer to between 0 and 1 to obtain the weight of each channel. Finally, the weight Y of each channel is multiplied element by element with the input X.

[0098] The global average pooling (GAP) operation calculates the average value of each channel of the feature map to obtain a one-dimensional vector with the same number of channels. Each element of this vector represents the average activation value on the corresponding channel, which can be regarded as a global statistical feature of the channel. The channel identifier g generated by global average pooling is also a vector, and each element is the average value on the corresponding channel.

[0099] The advantage of global average pooling (GAP) is that it can extract the overall information of the feature map and average the information within the channel, which helps to reduce the model's sensitivity to specific locations. In addition, the global average pooling operation can also reduce the number of model parameters and reduce model complexity, thereby improving the generalization ability of the model.

[0100] The core idea of ​​the ECA module is to capture the dependencies between channels through one-dimensional convolution. Compared with the traditional attention mechanism, the ECA module avoids the complex dimensionality reduction and dimensionality increase process, thus achieving high efficiency and lightweight characteristics.

[0101] S5: Fuse the extracted one-dimensional features with the two-dimensional features to generate a multimodal feature H1.

[0102] The extracted one-dimensional features F1 and F2 are multiplied element by element with the two-dimensional feature F3 to generate the multimodal feature H1:

[0103] H1=mul(F1,F2,F3);

[0104] Among them, mul means performing element-by-element multiplication operation on F1, F2, and F3, and the elements at the same position of F1, F2, and F3 are multiplied, and H1 represents the fused multimodal features.

[0105] S6: Input the fused multimodal feature H1 into the LSTM network for fault diagnosis and classification, including:

[0106] R=Softmax(Dense(Dropout(LSTM(LSTM(H1)))));

[0107] Among them, H1 represents the fused multimodal features, LSTM represents the LSTM network, the Dropout layer receives the output of the LSTM layer as input, Dense represents a fully connected layer, which receives the output from the Dropout layer as input, Softmax represents converting the output of the Dense layer into a probability distribution, ensuring that all output values ​​range between 0 and 1 and their sum is 1, and R represents the probability value of the fault.

[0108] After the model is built, the mean square error loss is used to construct the loss function, and the calculation formula is:

[0109]

[0110] Where: n samples Represents the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample, represents the square of the difference between the true value and the predicted value of the i-th sample, and the right side of the equation represents the average of the sum of the differences between the true value and the predicted value of all samples.

[0111] The purpose of constructing the loss function is to supervise the training of the HMLC model for bearing fault diagnosis.

[0112] After the model is built, it also includes:

[0113] Calculate the accuracy and recall of the model, and get the F1-Score value based on the accuracy and recall;

[0114]

[0115] Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample;

[0116] TP refers to true positive examples, that is, the number of samples that are actually positive and predicted as positive. FP refers to false positive examples, that is, the number of samples that are actually negative but predicted as positive, also known as false positives. FN refers to false negative examples, that is, the number of samples that are actually positive but predicted as negative, also known as false negatives. TN refers to true negative examples, that is, the number of samples that are actually negative and predicted as negative.

[0117]

[0118] Among them, Recall represents the recall rate of the model, TP represents the true positive examples in the sample, and FN represents the false negative examples in the sample;

[0119]

[0120] Among them, F1Score represents the F1-Score value of the model, Acc represents the accuracy of the model, and Recall represents the recall rate of the model.

[0121] Accuracy is a basic indicator used to evaluate the performance of classification models in machine learning and data analysis. If the accuracy value is close to 1, it means that the model can make correct predictions in most cases. If the accuracy value is close to 0, it means that the model rarely makes correct predictions and the performance is poor. The value of accuracy is between 0 and 1, and the higher the value, the better the model performance.

[0122] Recall is a commonly used evaluation metric in machine learning and classification problems. It measures the proportion of samples that are successfully predicted as positive examples among all samples that are actually positive examples. The range of recall or true positive rate (TPR) is [0,1]. When the recall is close to 1, it means that the model has found almost all positive examples, that is, the recall rate of the model is very high. When the recall is close to 0, it means that the model has found almost no positive examples, that is, the recall rate of the model is very low.

[0123] F1-Score is an indicator used to comprehensively evaluate the performance of classification models. It combines the advantages of accuracy and recall, and can simultaneously reflect the classification effect and recall rate of the model. The value range of F1-Score is 0 to 1, where 1 represents the best performance and 0 represents the worst performance.

[0124] The embodiment of the present invention uses MCNN to extract one-dimensional features, and uses GASF, GADF, and RP to convert the signal into three images, and then fuses the three images into a three-channel image similar to RGB for two-dimensional feature extraction, and finally fuses the one-dimensional features and the two-dimensional features into multimodal features, and inputs the fused multimodal features into the LSTM classification module for fault diagnosis and classification. The model of the present invention is applied to the aircraft engine bearing data set for fault diagnosis, and combined with the accuracy, recall rate and F1-Score value, the model has superiority in fault diagnosis, and improves the accuracy of fault diagnosis of aircraft engine bearings.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fault diagnosis method based on multi-modal and multi-scale feature fusion, characterized in that: The method comprises: The sensors are used to collect signal sources when faults occur at different positions of the aircraft engine bearing, including but not limited to: vibration signals, acoustic signals, temperature, humidity, and current; Performing denoising on the signal source to obtain an initial one-dimensional signal X0; For the initial one-dimensional signal X0, one-dimensional feature extraction is performed through the MCNN module; Perform image conversion on the initial one-dimensional signal X0 to obtain an initial two-dimensional image T1, and perform two-dimensional feature extraction on the initial two-dimensional image T1; The extracted one-dimensional features are fused with the two-dimensional features to generate multimodal features H1; The fused multimodal feature H1 is input into the LSTM network for fault diagnosis and classification.

2. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 1, characterized in that: The initial one-dimensional signal X0 is subjected to one-dimensional feature extraction by the MCNN module, including: The one-dimensional feature extraction is divided into two branches. In the first branch, the initial one-dimensional signal X0 is convolved with 20*1, 10*1, and 1*1 convolution kernels, and then processed with 2*2 global pooling to obtain a one-dimensional feature F1: F1=GMP(C 1*1 (C 10*1 (C 20*1 (X0)))); Among them, GMP represents the global maximum pooling, C 20*1 , C 10*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively; In the second branch, the initial one-dimensional signal X0 is convolved by 9*1 and 7*1 convolution kernels, then by 2*2 global pooling, then by 5*1, 5*1, and 1*1 convolution kernels, and finally by 2*2 global pooling to obtain a one-dimensional feature F2: F2=GMP(C 1*1 (C 5*1 (C 5*1 (GMP(C 7*1 (C 9*1 (X0))))))); Among them, GMP represents the global maximum pooling, C 9*1 , C 7*1 , C 5*1 , C 5*1 , C 1*1 The convolution kernels are 20*1, 10*1, and 1*1 respectively.

3. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 1, characterized in that: The image conversion of the initial one-dimensional signal X0 to obtain the initial two-dimensional image T1 includes: The image processing module uses GASF, GADF, and RP to convert the initial one-dimensional signal X0 into three corresponding different images X1, X2, and X3. After fusion, the initial two-dimensional image T1 is obtained, which is expressed as: X1,X2,X3,=GASF(X0),GADF(X0),RP(X0); T1 = ImageFusion(X1,X2,X3); Among them, ImageFusion stands for image fusion, GASF stands for Gram angle sum field processing, GADF stands for Gram angle difference field processing, RP stands for recursive graph processing, and T1 is the initial two-dimensional image after fusion.

4. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 3, characterized in that: The two-dimensional feature extraction of the initial two-dimensional image T1 includes: The initial two-dimensional image T1 first enters the 3*3 convolution kernel for convolution calculation, then enters the 3*3 global pooling for global pooling processing, then enters the residual block with a convolution kernel size of 3*3 for calculation, then enters the 3*3 global pooling for a global pooling processing, then enters the ECA attention mechanism module, and then enters a residual block with a convolution kernel size of 3*3 for calculation, and finally performs a global pooling processing through the 3*3 global pooling to obtain the two-dimensional feature F3: F3=GMP(Rebl(ECA(GMP(Rebl(GMP(C 3*3 (T1))))))); Among them, GMP represents the global maximum pooling, C 3*3 is a 3*3 convolution kernel, Rebl is a residual block with a convolution kernel size of 3*3, ECA is the attention mechanism, and T1 is the initial two-dimensional image.

5. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 2 or 4, characterized in that: The step of fusing the extracted one-dimensional features with the two-dimensional features to generate a multimodal feature H1 includes: The extracted one-dimensional features F1 and F2 are multiplied element by element with the two-dimensional feature F3 to generate the multimodal feature H1: H1=mul(F1,F2,F3); Among them, mul means performing element-by-element multiplication operation on F1, F2, and F3, and the elements at the same position of F1, F2, and F3 are multiplied, and H1 represents the fused multimodal features.

6. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 1, characterized in that: The step of inputting the fused multimodal feature H1 into the LSTM network for fault diagnosis and classification includes: R=Softmax(Dense(Dropout(LSTM(LSTM(H1))))); Among them, H1 represents the fused multimodal features, LSTM represents the LSTM network, the Dropout layer receives the output of the LSTM layer as input, Dense represents a fully connected layer, which receives the output from the Dropout layer as input, Softmax represents converting the output of the Dense layer into a probability distribution, ensuring that all output values ​​range between 0 and 1 and their sum is 1, and R represents the probability value of the fault.

7. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 1, characterized in that: The method comprises: After the model is built, the mean square error loss is used to construct the loss function, and the calculation formula is: Where: n samples represents the total number of samples, y i is the true value of the i-th sample, is the predicted value of the ith sample, represents the square of the difference between the true value and the predicted value of the i-th sample, and the right side of the equation represents the sum of the differences between the true value and the predicted value of all samples and the average value.

8. A fault diagnosis method based on multi-modal and multi-scale feature fusion according to claim 7, characterized in that: The method further comprises: After the model is built, calculate the accuracy and recall of the model, and get the F1-Score value based on the accuracy and recall; Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample; Among them, Recal l represents the recall rate of the model, TP represents the true positive examples in the sample, and FN represents the false negative examples in the sample; Among them, F1 Score represents the F1-Score value of the model, Acc represents the accuracy of the model, and Recal l represents the recall rate of the model.

Images