Engine fault diagnosis method based on audio extraction and cross-modal migration

Abstract

The invention provides an engine fault diagnosis method based on audio extraction and cross-modal migration. The method comprises the following steps: (1) collecting vibration and audio data when an engine operates; (2) constructing an audio extraction module, and extracting an audio signal of a vulnerable part of the engine; (3) establishing a training set including vibration and extracted audio data, and a test set only including extracted audio data; (4) establishing a vibration fault diagnosis network to realize vibration feature extraction and fault classification; (5) establishing an audio fault diagnosis network to realize audio feature extraction and fault classification; (6) establishing a cross-modal information decoupling module to realize fine-grained alignment and unified representation of vibration and audio features; (7) establishing a multi-modal index moving module to realize aggregation of cross-modal information; (8) designing six loss function updating model parameters; and (9) in a test stage, obtaining audio data of the target vulnerable part through an audio extraction module, and inputting the audio data into an audio fault diagnosis network to realize fault diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of engine fault detection, and in particular to an engine fault diagnosis method based on audio extraction and cross-modal migration. Background Art

[0002] In today's rapidly developing industrial society, various mechanical equipment has penetrated into all areas of production and life, greatly promoting the development of productivity and the improvement of human living standards. However, with the widespread use of these equipment, they will inevitably suffer from wear and aging during long-term operation, which will lead to failures. These failures will not only affect the normal operation of the equipment and reduce production efficiency, but may also cause safety accidents, resulting in casualties and property losses.

[0003] In order to ensure the stable operation of mechanical equipment, it is particularly important to detect and diagnose equipment failures in a timely and accurate manner. At present, technicians generally use vibration sensors to monitor the operating status of equipment. Vibration sensors can collect vibration signals from equipment, which contain rich information about equipment operation. By analyzing vibration signals, technicians can determine whether the equipment has a fault and the type and location of the fault. In particular, triaxial vibration data can provide a more comprehensive and accurate diagnostic basis, thereby improving the accuracy of fault detection.

[0004] However, the deployment and maintenance of vibration sensors require high costs and technical requirements, which to some extent limits their application in a wider range of scenarios. In addition, the installation and debugging process of vibration sensors is relatively complicated and requires professional personnel to operate. In order to overcome these limitations, technicians began to explore the possibility of using audio signals for fault diagnosis. Audio signals have the advantages of easy acquisition and low cost, and can achieve non-contact monitoring, which provides a new way for fault diagnosis.

[0005] Audio signals also contain information about the operating status of the device. By analyzing the characteristics of the audio signal, it is possible to identify whether the device is abnormal. For example, when a device fails, its operating sound will change, and the sound signal collected by the audio sensor will reflect these changes. However, audio signals are easily interfered by environmental noise, and the collected data may contain a large amount of irrelevant information, which brings challenges to the analysis and processing of audio signals.

[0006] In order to improve the fault diagnosis accuracy of audio signals, researchers began to try to combine vibration signals and audio signals for multimodal fault diagnosis. By fusing the features of the two signals, their respective advantages can be fully utilized to improve the accuracy and robustness of fault detection. However, there are relatively few studies on the fusion of vibration signals and audio signals. How to effectively combine the two signals and extract useful information is still a problem worthy of in-depth exploration.

[0007] The present invention is proposed in this context, and aims to achieve high-precision fault diagnosis of vulnerable engine parts through audio extraction technology and cross-modal knowledge distillation methods. By synchronously collecting the vibration data and audio data of the engine, the audio signal of the target vulnerable part is separated from the mixed signal using the audio extraction module, and then combined with the vibration signal for comprehensive analysis, the accurate identification of the engine fault can be achieved. This method not only improves the accuracy of fault diagnosis, but also has the advantages of non-contact and low cost. It is particularly suitable for engine fault diagnosis under special working conditions such as high temperature and high pressure, and has important practical application value. Summary of the invention

[0008] This paper proposes an engine fault diagnosis method based on audio extraction and cross-modal migration. The core of this method is to use vibration data to enhance the fault diagnosis capability of audio data, aiming to solve the problem that fault diagnosis using only audio models is easily interfered by environmental noise, while overcoming the high cost and deployment difficulty of relying solely on vibration data for diagnosis.

[0009] The key technical steps of the present invention include:

[0010] (1) The vibration data of the engine when it is running is collected by an acceleration sensor, and the audio data of the engine when it is running and the audio data of the target vulnerable parts when they are running alone are collected by an audio collection device.

[0011] (2) Data preprocessing: The synchronously collected audio data contains the audio of multiple vulnerable parts in operation, which affects the fault diagnosis results. An audio extraction module is established, including a multi-scale adaptive encoder, a main refinement stage, and a feature recombination unit. The multi-scale adaptive encoder is responsible for extracting the multi-scale features of the mixed audio and the target audio; the main refinement stage uses the mutual attention mechanism and multiple iterations to strengthen the features of the target parts and integrate information of different dimensions; the feature recombination unit enhances the high-frequency components during the refinement process to retain the details of the audio signal; the audio extraction module can effectively process and reconstruct the audio of the target parts to obtain the audio data of the target vulnerable parts in operation.

[0012] (3) Divide the data set into a training set and a test set in a ratio of 7:3, where each sample in the training set contains vibration data collected in the same time period and audio data after audio extraction, and the test set only contains audio data.

[0013] (4) A vibration signal fault diagnosis network is established as the teacher network. The CNN network is first used to extract multi-scale feature information, which is then input into the BiLSTM layer to capture the temporal correlation and bidirectional features of the time series data. Finally, the features are sent to the fully connected layer for classification and identification of fault types.

[0014] (5) An audio signal fault diagnosis network is established as a student network. First, the time-frequency information of the audio signal is extracted in the frequency domain through the CNN network. CNN can capture local features through multi-layer convolution operations and build more complex and abstract feature representations layer by layer. Then, the LSTM network is used to further capture the contextual information of the audio signal. Finally, the features are sent to the fully connected layer for classification and identification of fault types.

[0015] (6) A cross-modal information decoupling module is established between the teacher model and the student model. The cross-modal information decoupling module achieves fine-grained alignment and unified representation across modalities by minimizing the mutual information between modality-specific information and semantic information and maximizing the mutual information of semantic information between different modalities, thereby promoting the effective transfer and generalization of knowledge between different modalities.

[0016] (7) A multimodal exponential shift module is established between the teacher model and the student model. The multimodal exponential shift module performs iterative learning and vector quantization through the teacher-student mechanism, promotes the aggregation and alignment of cross-modal information, and realizes a unified representation in a shared semantic space, thereby enhancing the generalization and migration capabilities of the model on multimodal data.

[0017] (8) The above model is trained by the training set, and the reconstruction loss, vector quantization loss, commitment loss, mutual information minimization loss, mutual information maximization loss and cross-modal contrast loss are combined to obtain the total pre-training loss; and the pre-training loss and downstream task loss are balanced through hyperparameters, and the model parameters are updated by back-propagation and optimization algorithms.

[0018] (9) For the audio data to be identified, the audio data of the target vulnerable parts is first obtained through the audio extraction module, and then input into the audio fault diagnosis network to output the fault type of the engine vulnerable parts.

[0019] Furthermore, in step (2), an audio extraction module is established to input a mixed audio signal, where y(t) is the target vulnerable part audio s(t) and I interference audio or background noise b i (t) Mixture:

[0020]

[0021] Input the reference audio x(t) into the BLSTM network to obtain the target vulnerable part embedding vector g(x); the BLSTM network is used to capture the timing characteristics of the reference audio, and the formula is as follows:

[0022] g(x)=BLSTM(x(t))

[0023] The output of the BLSTM network is passed through an average pooling layer to obtain a vector of fixed dimension. The mixed signal y(t) is input into the convolutional neural network to obtain the embedding coefficient. The convolutional network converts the time domain signal into the embedding coefficient through a series of convolutional layers and activation functions. The formula is as follows:

[0024] MC=ConvNet(y(t))

[0025] The specific convolution operation is defined as:

[0026] Conv(y(t))=y(t)*w+b

[0027] Where w and b are the convolution kernel and bias parameters respectively, and * represents the convolution operation; the embedding coefficient and the target vulnerable part embedding vector are input into the target vulnerable part extractor to generate the mask M(t); the target vulnerable part extractor can adopt a gated convolutional neural network, and its mask generation formula is as follows:

[0028] M(t)=GCNN(MC,g(x))

[0029] The mask M(t) usually ranges from [0,1] and is used to selectively pass the audio components of the target vulnerable parts. The mask M(t) is applied to the embedding coefficients to obtain the masked coefficients:

[0030] MA=MC×M(t)

[0031] The masked coefficients are input into the audio decoder to reconstruct the time domain audio signal of the target vulnerable part The audio decoder can use a deconvolutional neural network, and its reconstruction formula is as follows:

[0032]

[0033] The deconvolution operation is defined as:

[0034] Conv(MA)=MA*w'+b'

[0035] Where w' and b' are the deconvolution kernel and bias parameters respectively.

[0036] In the further step (6), the cross-modal information decoupling module minimizes the mutual information between the modality-specific information and the semantic information as follows:

[0037] I CLUB =E p(x,y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)]

[0038] And the formula for maximizing the mutual information of semantic information between different modalities is as follows:

[0039]

[0040] (1) In the further step (8), the reconstruction loss is as follows:

[0041]

[0042] The vector quantization loss is as follows:

[0043]

[0044] The promised losses are as follows:

[0045]

[0046] The mutual information minimization loss is as follows:

[0047] L CLUB =F p(x,y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)]

[0048] The mutual information maximization loss is as follows:

[0049]

[0050] The cross-modal contrast loss is as follows:

[0051]

[0052] The total pre-training loss is as follows:

[0053] L total =α recon L recon +α vq L vq +α commit L commit +α CLUB ICLUB +α Cross-CPC L Cross-CPC +α contrastive L contrastive

[0054] Beneficial effects: The engine vulnerable parts fault diagnosis method based on audio extraction and cross-modal knowledge distillation proposed in the present invention realizes high-precision diagnosis of engine faults by combining audio and vibration data. This method not only inherits the non-contact diagnosis characteristics of a single audio mode, but also accurately extracts the audio signal of the target vulnerable parts through specific audio extraction technology to realize targeted fault diagnosis. In addition, this method makes full use of the rich information in the vibration data and significantly improves the accuracy and reliability of diagnosis. As an innovative technical means, it provides a new solution for engine maintenance and fault prediction, and is particularly suitable for fault diagnosis under special working conditions such as high temperature and high pressure, and has important practical application value and promotion prospects. Experimental results show that this method shows high accuracy and robustness in fault identification, and can maintain high recognition accuracy even when the audio data is interfered by environmental noise or the data acquisition conditions are limited. At the same time, this method has the advantages of non-contact and low cost, is easy to deploy, reduces technical requirements and costs, and provides a strong guarantee for the stable operation of mechanical equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a principle block diagram of the fault diagnosis method according to an embodiment of the present invention;

[0056] Figure 2 This is the teacher model described in the embodiment of the present invention.

[0057] Figure 3 This is the student model described in the embodiment of the present invention.

[0058] Figure 4 This is the training framework described in the embodiment of the present invention.

[0059] Figure 5 This is the experimental result of the present invention. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] like Figure 1 As shown, this embodiment discloses an engine fault diagnosis method based on audio extraction and cross-modal migration, including:

[0062] (1) The vibration data of the engine when it is running is collected by an acceleration sensor, and the audio data of the engine when it is running and the audio data of the target vulnerable parts when they are running alone are collected by an audio collection device.

[0063] (2) Performing data preprocessing: The synchronously collected audio data contains the audio of multiple vulnerable parts in operation, which affects the fault diagnosis results. An audio extraction module is established, including a multi-scale adaptive encoder, a main refinement stage, and a feature recombination unit. The multi-scale adaptive encoder is responsible for extracting multi-scale features of mixed audio and target audio; the main refinement stage uses a mutual attention mechanism and multiple iterations to strengthen the features of the target parts and integrate information of different dimensions; the feature recombination unit enhances high-frequency components during the refinement process to retain the details of the audio signal; the audio extraction module can effectively process and reconstruct the audio of the target parts to obtain the audio data of the target vulnerable parts in operation.

[0064] (3) Divide the data set into a training set and a test set in a ratio of 7:3, where each sample in the training set contains vibration data collected in the same time period and audio data after audio extraction, and the test set only contains audio data.

[0065] (4) A vibration signal fault diagnosis network is established as the teacher network. The CNN network is first used to extract multi-scale feature information, which is then input into the BiLSTM layer to capture the temporal correlation and bidirectional features of time series data. The improved particle swarm optimization (PSO) algorithm is used to optimize the training hyperparameters of the CNN-BiLSTM network.

[0066] (5) Establish an audio signal fault diagnosis network as a student network. First, read the audio data as time series data and convert the audio signal.

[0067] Divide into short time frames for short-time Fourier transform. The formula of Fourier transform is as follows:

[0068]

[0069] After the edge drawing is completed, the power spectrum of each frame is calculated. The formula is as follows:

[0070] P[k]=|X[k]| 2

[0071] The power spectrum is passed through the Mel filter bank, the formula is as follows:

[0072]

[0073] Taking the logarithm of the result after passing through the filter, the formula is as follows:

[0074] logS[m]=log(S[m])

[0075] Perform discrete cosine transform (DCT) on the logarithmic Mel spectrum to obtain the MFCC coefficients:

[0076]

[0077] Arrange the MFCC coefficients of all frames to form a feature matrix. Then use the CNN network to perform convolution operations on the MFCC features in the frequency domain. CNN can capture local features through multi-layer convolution operations and build more complex and abstract feature representations layer by layer. Then use the LSTM network to further capture the contextual information of the audio signal. Finally, send the features to the fully connected layer for fault type classification and identification.

[0078] (6) A cross-modal information decoupling module is established between the teacher model and the student model. The cross-modal information decoupling module minimizes the mutual information between modality-specific information and semantic information as follows:

[0079] I CLUB =E p(x,y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)]

[0080] And the formula for maximizing the mutual information of semantic information between different modalities is as follows:

[0081]

[0082] Achieve fine-grained alignment and unified representation across modalities, thereby facilitating the effective transfer and generalization of models across different modalities.

[0083] (7) A multimodal exponential shift module is established between the teacher model and the student model. The multimodal exponential shift module performs iterative learning and vector quantization through the teacher-student mechanism, promotes the aggregation and alignment of cross-modal information, and realizes a unified representation in a shared semantic space, thereby enhancing the generalization and migration capabilities of the model on multimodal data.

[0084] (8) The above model is trained using the training set, and the reconstruction loss is as follows:

[0085]

[0086] The vector quantization loss is as follows:

[0087]

[0088] The promised losses are as follows:

[0089]

[0090] The mutual information minimization loss is as follows:

[0091] L CLUB =E p(x,y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)]

[0092] The mutual information maximization loss is as follows:

[0093]

[0094] The cross-modal contrast loss is as follows:

[0095]

[0096] The total pre-training loss is as follows:

[0097] L total =α recon L recon +α vq L vq +α commit L commit +α CLUB I CLUB +α Cross-CPC L Cross-CPC +α contrastive L contrastive

[0098] (9) The pre-training loss and downstream task loss are balanced through hyperparameters, and the model parameters are updated through back-propagation and optimization algorithms.

[0099] (10) For the audio data to be identified, the audio data of the target vulnerable parts is first obtained through the audio extraction module, and then input into the audio fault diagnosis network to output the fault type of the engine vulnerable parts. This not only retains the unique advantages of non-contact diagnosis, but also significantly improves the accuracy of fault diagnosis.

[0100] This embodiment describes in detail an application scenario of an engine fault diagnosis method based on audio extraction and cross-modal migration. The method includes a data acquisition module, a signal preprocessing module, a deep learning module and a fault identification module, and each module works together to achieve high-precision diagnosis of target vulnerable parts faults.

[0101] The following is a verification of 14 types of faults in the collected data. The confusion matrix of the model is as follows: Figure 5 described.

[0102] The experimental results show that the model has high accuracy and robustness in fault identification. Figure 5 The results (as shown in the figure) clearly demonstrate the classification effect of the model on different fault types, among which the recognition accuracy of most fault types exceeds 95%, proving the potential and reliability of the model in practical applications.

[0103] In addition, it can be observed that the audio data of the vulnerable parts obtained by the audio extraction module plays a key role in fault diagnosis. This finding further confirms the important value of audio signals in engine fault diagnosis, especially when the audio data is interfered by environmental noise or the data acquisition conditions are limited.

[0104] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An engine fault diagnosis method based on audio extraction and cross-modal migration, characterized in that: The following steps are involved: (1) Collecting vibration data of the engine when it is running through an acceleration sensor, and collecting audio data of the engine when it is running and audio data of the target vulnerable parts when they are running alone through an audio collection device; (2) Perform data preprocessing. The above synchronously collected audio data contains the audio of multiple vulnerable parts in operation, which affects the fault diagnosis results. With the help of audio extraction technology, the audio signals of the vulnerable parts of the engine are accurately captured; an audio extraction module is established, including a multi-scale adaptive encoder, a main refinement stage, and a feature recombination unit. The multi-scale adaptive encoder is responsible for extracting multi-scale features of the mixed audio and the target audio; The main refinement stage uses the mutual attention mechanism and multiple iterations to strengthen the features of the target parts and integrate information from different dimensions. The feature recombination unit enhances the high-frequency components during the refinement process to retain the details of the audio signal. The audio extraction module can effectively process and reconstruct the audio of the target parts to obtain the audio data of the target vulnerable parts during operation. (3) Dividing the data set into a training set and a test set in a ratio of 7:3, where each sample in the training set contains vibration data collected in the same time period and audio data after audio extraction, and the test set contains only audio data; (4) A vibration signal fault diagnosis network is established as the teacher network. First, a CNN network is used to extract multi-scale feature information, which is then input into the BiLSTM layer to capture the temporal correlation and bidirectional features of the time series data. Finally, the features are sent to the fully connected layer for classification and identification of fault types. (5) An audio signal fault diagnosis network is established as a student network. First, the time-frequency information of the audio signal is extracted in the frequency domain through the CNN network. CNN can capture local features through multi-layer convolution operations and build more complex and abstract feature representations layer by layer. Then, the LSTM network is used to further capture the contextual information of the audio signal. Finally, the features are sent to the fully connected layer for classification and identification of fault types. (6) A cross-modal information decoupling module is established between the teacher model and the student model. The cross-modal information decoupling module achieves fine-grained alignment and unified representation across modalities by minimizing the mutual information between modality-specific information and semantic information and maximizing the mutual information between semantic information of different modalities. (7) A multimodal exponential shift module is established between the teacher model and the student model. The multimodal exponential shift module performs iterative learning and vector quantization through the teacher-student mechanism, promotes the aggregation and alignment of cross-modal information, and realizes a unified representation in a shared semantic space, thereby enhancing the generalization and migration capabilities of the model on multimodal data. (8) Train the above model through the training set, combine the reconstruction loss, vector quantization loss, commitment loss, mutual information minimization loss, mutual information maximization loss and cross-modal contrast loss to obtain the total pre-training loss; and balance the pre-training loss and downstream task loss through hyperparameters, and update the model parameters through back propagation and optimization algorithms; (9) For the audio data to be identified, the audio data of the target vulnerable parts is first obtained through the audio extraction module, and then input into the audio fault diagnosis network. Since the student model has learned the rich knowledge of vibration modes in the teacher model, high-precision fault diagnosis can still be achieved even in the absence of vibration data input.

2. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1 is characterized in that: The audio extraction module in step (2) includes: (2-1) Multi-scale adaptive encoder, based on asymmetric convolution design, captures features at different scales, extracts the contextual relationship between mixed audio and target audio, and provides rich feature representation for subsequent fusion and refinement; (2-2) The main stage includes multiple iterations, each of which includes components such as bidirectional long short-term memory network (BLSTM), multi-scale large kernel attention, mobile anti-bottleneck convolution and full-band self-attention; (2-3) The refinement stage is similar to the main stage, but each iteration includes an additional feature reorganization unit (FRU) to enhance high-frequency components; (2-4) Feature recombination unit, generates a DC component through a global average pooling layer, then subtracts the DC component from the input feature to obtain the high-frequency component, and reweights these components through learnable parameters.

3. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1 is characterized in that: The teacher network in step (4) includes: the teacher network uses a convolutional neural network (CNN) to extract high-dimensional features of the original vibration signal, and captures the time-related features of the vibration signal through a bidirectional long short-term memory network (BiLSTM), and uses an improved particle swarm optimization (PSO) algorithm to optimize the training hyperparameters of the CNN-BiLSTM network.

4. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1, characterized in that: In step (5), the student model includes: the student network first converts the extracted audio data of the target vulnerable parts into Mel-frequency cepstral coefficient (MFCC) features, and then uses a CNN network to perform a convolution operation on the MFCC features in the frequency domain to extract time-frequency information; then uses a (LSTM) network to receive the output of the CNN network and capture the context information of the audio signal; and finally inputs it into a classifier to diagnose the fault type.

5. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1 is characterized in that: The cross-modal information decoupling (CID) module in step (6) includes: (6-1) The contrast logarithmic ratio upper bound (CLUB) unit is used to optimize the upper bound of the mutual information between the semantic information that is agnostic to the vibration mode and the specific information of the audio mode. Given two variables x and y, the objective function of CLUB is defined as: I vCLUB (x;y):=E p(x,y) [logq θ (y|x)]-E p(x) E p(y) [logq θ (y|x)] where q θ is the variational approximation of the true posterior of y given x; (6-2) The cross-modal contrastive predictive coding (Cross-CPC) unit uses a single-layer bidirectional long short-term memory network (LSTM) to summarize the information of all time steps in the audio modality; through the optimization of the InfoNCE loss function, the contextual representation of the audio modality is used to predict the information of future steps in the vibration modality; the InfoNCE loss function first randomly selects N-1 random negative samples and 1 random positive sample from the vibration modality to obtain a set Use later Predict the kth step into the future in the vibration mode So the loss function of InfoNCE loss is as follows:

6. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1 is characterized in that: The multimodal exponential movement (MM-EM) module in step (7) comprises: (7-1) Cross-attention mechanism, which is used to extract relevant information from vibration modalities and strengthen the semantic association between modalities; using audio features as queries and vibration features as keys, information is extracted by calculating the association between query and key; (7-2) an exponential moving average algorithm, used to dynamically update the encoding vector of each modality based on information from different modalities; (7-3) Teacher-student learning framework allows knowledge transfer and learning between different modalities.

7. The engine fault diagnosis method based on audio extraction and cross-modal migration according to claim 1, wherein the loss function in step (8) comprises: (8-1) Reconstruction loss (L recon ), which is used to ensure that the discrete latent codes output by the encoder can reconstruct the original modality features; ( 8-2) Commitment loss (L commit ), used to force the encoder output to be close to a predefined discrete potential code; (8-3) Cross-modal contrast prediction coding loss (L cpc ), used to maximize the mutual information between different modalities through cross-modal prediction; (8-4) Multimodal exponential moving average loss (L mm-ema ), used to align and aggregate information between different modalities; (8-5) Mutual information maximization loss (L mi ), which is used to further optimize the decoupling and alignment of semantic features between modalities.

Images