Water turbine fault classification diagnosis method based on multi-modal fusion and meta learning

Abstract

The invention discloses a water turbine fault classification diagnosis method based on multi-modal fusion and meta-learning. The method comprises the following steps: step 1, respectively extracting time domain features and Mel-language spectrogram features of monitoring noise signals of a water turbine through a feature extraction module; step 2, performing cross-modal attention mechanism fusion on the time domain features and the voiceprint features through a multi-modal fusion module, and adjusting a fusion weight based on a dynamic weight distribution mechanism; and step 3, performing small sample training optimization on the fused features through a meta-learning module, and improving the classification capability of the model for new fault types in combination with a Triplet Loss-KNN algorithm and twin network pre-training. The fault diagnosis method based on the time domain-voiceprint fusion network and meta learning has the advantages of being rapid in diagnosis, accurate in classification, high in generalization ability and the like, and the fault diagnosis precision of the water turbine based on noise signals can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical fault diagnosis and intelligent monitoring, and particularly relates to a method for classifying and diagnosing water turbine faults based on multi-modal fusion and meta-learning. Background Technique

[0002] With the rapid development of deep learning technology, breakthroughs have been made in the field of intelligent fault diagnosis. Deep learning has shown significant advantages in fault diagnosis with its powerful feature extraction and correlation capabilities. However, in actual industrial scenarios, especially in the fault diagnosis of key equipment such as water turbine units, the effective application of deep learning methods faces severe challenges. This is mainly reflected in two aspects: First, the failure rate of the water turbine flow path is relatively low, and the available fault samples are extremely limited, which makes it difficult for unsupervised learning methods that rely only on normal samples for fault identification; Second, mechanical faults often show progressive development characteristics, and it is difficult to collect process data, further exacerbating the problem of scarce fault samples.

[0003] Therefore, a method for classifying and diagnosing water turbine faults based on multi-modal fusion and meta-learning is proposed to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above deficiencies and provide a method for classifying and diagnosing water turbine faults based on multi-modal fusion and meta-learning, and solve the following problems existing in the traditional methods: Single-modal features are difficult to capture the dynamic characteristics of complex noise; The ability of the model to classify faults is weak in the scenario of lacking a large amount of fault data.

[0005] For the above problems, the technical solution adopted by the present invention is: A method for classifying and diagnosing water turbine faults based on multi-modal fusion and meta-learning, including the following steps: Step 1: Respectively extract the time-domain features and Mel spectrogram features of the water turbine monitoring noise signal through the feature extraction module; Step 2: Perform cross-modal attention mechanism fusion on the time-domain features and voiceprint features through the multi-modal fusion module, and adjust the fusion weight based on the dynamic weight allocation mechanism; Step 3: Optimize the fused features through small-sample training by the meta-learning module, and combine the TripletLoss-KNN algorithm and the twin network pre-training to improve the classification ability of the model for new fault types.

[0006] Preferably, the feature extraction module in the first step includes: Perform multi-level residual convolution compression on the time-domain noise signal, extract time-domain statistical features, including variance, skewness, kurtosis, and clearance factor; perform masking enhancement processing on the Mel spectrogram through the residual two-dimensional convolution module, and extract robust voiceprint features.

[0007] Preferably, the specific structure of the multi-level residual convolution compression is as follows: The first layer: the input channel is 1, the output channel is 64, the convolution kernel size is 7, the stride is 2, and the output length is compressed to 60000; The second layer: the input channel is 64, the output channel is 128, the convolution kernel size is 9, the stride is 3, and the output length is compressed to 15000; The third layer: the input channel is 128, the output channel is 256, the convolution kernel size is 9, the stride is 5, and the output length is compressed to 3750; The fourth layer: through the multi-scale fusion layer and the pooling layer, finally output a time-domain feature vector with a length of 256.

[0008] Preferably, perform STFT transformation on the noise signal to generate a power spectrum, filter it through the Mel filter bank and perform logarithmic compression, and then process it through the residual two-dimensional convolution module; The residual two-dimensional convolution module includes a main path and a shortcut path. The main path is sequentially a channel adjustment convolution layer, a batch normalization layer, a GELU activation layer, and a convolution layer that maintains the number of channels, and finally adds and activates with the output of the shortcut path.

[0009] Preferably, the cross-modal attention mechanism adopted by the multi-modal fusion module in the second step specifically includes: Use the time-domain feature as the query vector, that is, Q, and the voiceprint feature as the key-value vector, that is, K, V; Dynamically adjust the weights of the time-domain feature and the voiceprint feature through learnable parameters, where: Time-domain weight formula:

[0010] Voiceprint weight formula:

[0011] Where: is the weight matrix of the time-domain feature, which is a learnable parameter; is the weight matrix of the acoustic feature, which is a learnable parameter; is the extracted feature vector of the time-domain signal; is the extracted feature vector of the acoustic image; is the bias term of the time-domain feature, which is a learnable parameter; is the bias term of the acoustic feature and is a learnable parameter; is the Sigmoid function; Further enhance the features through element-wise multiplication, and its expression is as follows: ; ; where is the enhanced acoustic feature vector after applying the weight; is the enhanced time-domain feature vector after applying the weight; represents element-wise multiplication; And concatenate along the channel dimension to obtain the fused feature, and its expression is as follows: Bimodal feature fusion: ; where: represents concatenation along the channel dimension.

[0012] Preferably, the Triplet Loss-KNN pre-training in step three includes: Construct a triplet of anchor, positive sample and negative sample, extract the feature vector through the Siamese network, and calculate the Euclidean distance between the anchor and the positive sample and the Euclidean distance between the anchor and the negative sample , taking the Triplet Loss function as the optimization model, and its expression is: ; where: : represents the distance between the anchor and the positive sample; : represents the distance between the anchor and the negative sample; : hyperparameter, defining the minimum interval between the positive sample and the negative sample; And construct a feature vector library based on the KNN algorithm to achieve classification through cosine similarity matching.

[0013] Preferably, the step three also includes meta-learning optimization: In the N-way-1-shot task, perform L2 normalization on the support set and query set features, calculate the normalized cosine similarity, and combine the cross-entropy loss and the average entropy regularization term to obtain the optimization model, and its expression is: ; where: is a hyperparameter, is the mean entropy of the predicted probability distribution of the query samples.

[0014] Preferably, the multimodal fusion module further includes a dynamic position encoding and a multi-head self-attention mechanism for capturing the long-range dependencies of the time-domain signals and the frequency-domain context information.

[0015] Preferably, a masking enhancement operation is introduced during the generation process of the Mel spectrogram to simulate a noisy environment and improve the robustness of the model.

[0016] Preferably, the method further includes joint training of industrial-end noise signals and the public dataset DCASE to enhance the generalization ability of the model.

[0017] The present invention has the following beneficial effects: 1. Aiming at the problem of insufficient utilization of the transient characteristics of time-domain signals and the spectral characteristics of voiceprint signals in the existing methods, the present invention designs a novel multimodal feature fusion mechanism. By deeply mining the complementary characteristics between different modal signals, the mechanism significantly improves the information utilization efficiency and provides richer feature support for subsequent fault diagnosis. 2. The present invention proposes a meta-learning optimization framework for mechanical signal characteristics, which significantly improves the generalization ability of the model in the case of small samples and ensures accurate classification of fault types even in the absence of a large amount of fault data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is the logical structure diagram of the present invention; Figure 2 is the schematic diagram of the time-domain voiceprint network and the shared attention module of the present invention; Figure 3 is the schematic diagram of the dynamic weight feature fusion module of the present invention; Figure 4 is the meta-learning flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The present invention will be further described below with reference to the drawings and embodiments: Referring to Figures 1 to 4 as shown, a hydroturbine fault classification and diagnosis method based on multimodal fusion and meta-learning is provided in this specific embodiment, including the following steps: Step 1: Extract the time-domain features and Mel spectrogram features of the hydroturbine monitoring noise signals through the feature extraction module respectively; Step 2: Perform cross-modal attention mechanism fusion on the time-domain features and voiceprint features through the multimodal fusion module, and adjust the fusion weights based on the dynamic weight allocation mechanism; Step 3: Optimize the fused features through few-shot training of the meta-learning module, and combine the TripletLoss-KNN algorithm and the pre-training of the Siamese network to improve the classification ability of the model for new fault types.

[0020] Preferably, Example 1 is for multi-modal data acquisition, specifically including: Industrial data acquisition: Obtain real-time operating audio signals from the noise monitoring system of the water turbine unit; Dataset introduction: Introduce the DCASE industrial noise dataset to enhance the generalization ability of the model; Joint training ratio: It can be mixed at 4:1, that is, in each batch of training data, industrial data accounts for 80% and DCASE data accounts for 20% to balance domain differences and enhance generalization.

[0021] Preferably, Example 2 provides the implementation details of the time-domain network: Receive the original audio signal with a shape of (B, 1, 240000) and input it into the time-domain network; Among them, the network structure includes: The first part: Residual convolution module: Adopt cascaded residual convolution blocks, and each layer contains a convolutional layer, batch normalization, and GELU activation function.

[0022] The specific structure is: The first layer: Input channels 1 → 64, kernel size 7, stride 2, and the output length is compressed to 60000; The second layer: 64 → 128, kernel size 9, stride 3, and the output length is compressed to 15000; The third layer: 128 → 256, kernel size 9, stride 5, and the output length is compressed to 3750; The fourth layer: Multi-scale fusion layer and pooling layer, and the final output length is 256.

[0023] The second part: Statistical feature enhancement module: Calculate 4 types of statistical features of the signal features after convolutional compression: variance, skewness, kurtosis, and clearance factor. The formulas are as follows: Variance: ; Skewness: ; Kurtosis: ; Clearance factor: ; Concatenate the statistical features with the output of the residual convolution module along the channel dimension, adjust the number of channels to 256 through 1x1 convolution, and finally pass through the shared attention module.

[0024] Preferably, Example 3 provides the implementation details of the voiceprint network: Mel-spectrum generation: Perform STFT on the input signal to obtain a complex spectrogram, whose expression is as follows: ; where, is the input signal, is the time-frequency representation.

[0025] Calculate the power spectrum at each time point, whose expression is as follows: ; Construct a Mel filter bank and filter the power spectrum. The filter function on the Mel frequency scale is designed according to the distribution of Mel frequencies: ; Logarithmic compression: Perform logarithmic compression on the Mel spectrum to reduce the dynamic range and obtain the final Mel spectrum: ; where, is a small constant used to avoid the case of logarithm being zero.

[0026] Spectrum enhancement: Simulate signal noise through Mel-spectrum masking to enhance robustness.

[0027] Similarly, cascaded residual convolutional blocks are adopted, and each layer contains a convolutional layer, batch normalization, and GELU activation function.

[0028] Preferably, Embodiment 4 describes the implementation details of the shared attention module: Cross-modal self-attention: By taking the input time-domain feature vector and the voiceprint feature vector , dynamically learn the time weight and the voiceprint weight .

[0029] For cross-modal self-attention, the attention calculation process is: ; where: , ; The final cross-modal feature is: ; Through the parameter , adjust the importance of the feature, and its expression is as follows: Time weight formula: ; Voiceprint weight formula: ; Among them is the Sigmoid function.

[0030] Feature enhancement process: ; ; Among them represents element-wise multiplication, is the feature vector after applying attention.

[0031] Bimodal feature fusion: ; Among them represents concatenation along the channel dimension.

[0032] Concatenate the enhanced time-domain and voiceprint features to obtain the final fusion result: Preferably, Example 5 discloses training a Siamese network with labeled normal samples: The first part: Pretraining stage: Train the time-domain-voiceprint fusion network model, and use Triplet Loss-KNN to train the Siamese network for classifying labeled noise signals; The specific training process is as follows: Training dataset: The dataset contains labeled noise signals, and each signal is a specific class.

[0033] Point selection: In each training, select a noise signal as the anchor point, denoted as ; Select the positive sample : Randomly select a noise signal from the same class as the anchor point as the positive sample . The positive sample should be close to the anchor point in the feature space; Select the negative sample : Randomly select a noise signal from a different class as the negative sample to ensure that the anchor point and the negative sample are far apart in the feature space; The Siamese network will process these three samples - the anchor point, the positive sample, and the negative sample, and generate their feature vectors, denoted as: is the positive sample, is the anchor point, is the negative sample; Distance calculation: Use the Euclidean distance, i.e., the L2 norm, to calculate the distances between these feature vectors: , ; Among them: represents the distance between the anchor point and the positive sample. Represents the distance between the anchor point and the negative sample.

[0034] Loss function: The objective of the Triplet Loss function is to minimize the distance between the anchor point and the positive sample while maximizing the distance between the anchor point and the negative sample. The loss function is defined as: ; where: margin is a hyperparameter used to define the minimum interval between positive and negative samples; should be as small as possible, indicating similar samples, i.e., the anchor point and the positive sample are close in the feature space; should be as large as possible, indicating different samples, i.e., the anchor point and the negative sample are far apart in the feature space. If then the loss is zero, indicating that the negative sample is far enough away from the anchor point. If this condition is not met, the loss function will increase, indicating that the distance between the anchor point and the negative sample is too close. By minimizing this loss function, gradient descent can be used to train the siamese network. The parameters of the network will be adjusted to ensure that the feature vectors of samples of the same class are close in the embedding space, while the feature vectors of samples of different classes are far apart.

[0035] Second part: KNN classification. During the training process, after feature extraction by the siamese network, the labeled noisy signals of all the datasets used for training will obtain a feature vector. By performing inference of the siamese network on each training sample, we get the embedding vectors of each sample, and these embedding vectors form the feature library of the training data. The labels of the training data will also be saved, and these labels together with the corresponding embedding features constitute the basis of the training data. In the test phase, for each new test sample, its embedding feature vector is obtained by performing feature extraction on it through the already trained siamese network. Then, calculate the similarity between this test sample and all samples in the training data feature library, i.e., cosine similarity; after calculating the distances between all class prototypes and the test sample, select the class corresponding to the prototype with the minimum distance.

[0036] Preferably, Example 6 describes the process of meta-learning classification and optimization: After pre-training is completed, the model enters the meta-learning stage; at this time, the siamese network structure is used to implement the N-way-1-shot meta-learning task, that is, learning how to classify from a small number of samples; in the meta-learning task, the model receives N classes, and there is only 1 sample in each class. It is necessary to judge the similarity between different input pairs through metric learning and learn how to perform class judgment from these few samples. At the same time, adjust the model hyperparameters to improve the model's prediction ability and achieve abnormal sound diagnosis of the operation of the water turbine. The pre-trained Siamese network can be used for one-shot learning tasks. In one-shot learning, it is necessary to classify query samples based on a single support sample: Query samples, that is, the samples to be classified, whose categories are unknown; Support samples, that is, one sample in each category, usually not in the training set.

[0037] The training process is as follows: Data preprocessing: Prepare the training dataset, including support samples and query samples. Each support sample corresponds to one category, and each query sample needs to predict its category.

[0038] Feature extraction: Extract the feature vectors of support samples and query samples through the trained Siamese network.

[0039] Feature vectors of support samples: For each category, extract the feature vectors from the support samples Feature vectors of query samples: For query samples, extract the feature vectors

[0040] Initialize the weight matrix , and calculate the mean of the feature vectors of all support samples in each category to initialize the weight matrix Form the weight matrix .

[0041] Calculate the cosine similarity between the weight matrix and each query sample The cosine similarity formula for each category is: ; Use the softmax calculation of cosine_similarity, and take the calculated cosine similarity value as the input of softmax. Specifically, the input expression of softmax is: ; where is the bias term, and finally the class probability of the query sample obtained through softmax calculation is: ; where N is the total number of categories.

[0042] Calculate the entropy of the predicted probability distribution for each query sample , calculate the entropy values of all query samples, and its calculation expression is as follows: ; Calculate the Entropy Regularization: Average the entropy of all query samples to obtain the average entropy, and its calculation expression is as follows: ; Where: M is the total number of query samples.

[0043] Optimize the training, use the backpropagation algorithm to optimize the parameters of the network, and minimize the loss function: ; Where, is the traditional cross-entropy loss.

[0044] Preferably, Example 7 describes the dynamic position encoding and the multi-head self-attention mechanism, and its specific steps are: The first part: The specific implementation of the dynamic position encoding: Let the feature vector of the input time-domain signal be , and the dynamic position encoding matrix is generated by the following formula: ; Where: T is the time step, and d is the feature dimension; α is a learnable parameter used to dynamically adjust the contribution of the input features to the position encoding; i is the feature dimension index; Initially, α is set to 0.1 and optimized through backpropagation; The encoded feature vector is .

[0045] The second part: The specific implementation of the multi-head self-attention mechanism The multi-head self-attention module is used to fuse the time-domain and frequency-domain features and enhance the global context modeling ability; Number of attention heads: Set to 8 heads, and the dimension of each head ; Attention calculation: ; Where, Q, K, and V are respectively generated by mapping the time-domain features and the voiceprint features; Parameter initialization: The weight matrix is initialized with the He normal distribution, and the bias term is initialized to zero.

[0046] The third part: The combination with long-range dependence modeling The dynamic position encoding enables the model to capture the correlations across time steps in the signal by introducing temporal dynamic information; the multi-head self-attention mechanism enhances the fusion of frequency-domain context information through multi-perspective feature interactions. For example, after the residual convolution module, the dynamic position encoding is embedded into the feature vector and then input into the multi-head self-attention layer. The formula is: ; is the encoded feature vector.

[0047] The above embodiments are only the preferred technical solutions of the present invention and should not be regarded as limitations on the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including the equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A method for fault classification and diagnosis of hydraulic turbines based on multimodal fusion and meta-learning, characterized in that It includes the following steps: Step 1: Respectively extract the time-domain features and Mel spectrogram features of the hydroturbine monitoring noise signal through the feature extraction module; Step 2: Through the multimodal fusion module, perform cross-modal attention mechanism fusion on the time-domain features and voiceprint features, and adjust the fusion weights based on the dynamic weight allocation mechanism; Step 3: Through the meta-learning module, perform few-shot training optimization on the fused features, and combine the Triplet Loss-KNN algorithm and siamese network pre-training to improve the classification ability of the model for new fault types.

2. The turbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 1, characterized in that In the feature extraction module in Step 1, it includes: Perform multi-level residual convolution compression on the time-domain noise signal to extract time-domain statistical features, including variance, skewness, kurtosis, and clearance factor; perform masking enhancement processing on the Mel spectrogram through the residual two-dimensional convolution module to extract robust voiceprint features.

3. The turbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 2, wherein The specific structure of the multi-level residual convolution compression is: The first layer: The input channel is 1, the output channel is 64, the convolution kernel size is 7, the stride is 2, and the output length is compressed to 60000; The second layer: The input channel is 64, the output channel is 128, the convolution kernel size is 9, the stride is 3, and the output length is compressed to 15000; The third layer: The input channel is 128, the output channel is 256, the convolution kernel size is 9, the stride is 5, and the output length is compressed to 3750; The fourth layer: Through the multi-scale fusion layer and the pooling layer, finally output a time-domain feature vector with a length of 256.

4. A hydroturbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 1, characterized in that Perform STFT transformation on the noise signal to generate a power spectrum, filter it through the Mel filter bank and perform logarithmic compression, and then process it through the residual two-dimensional convolution module; The residual two-dimensional convolution module includes a main path and a shortcut path. The main path is successively a channel adjustment convolution layer, a batch normalization layer, a GELU activation layer, and a convolution layer that maintains the number of channels, and finally adds and activates with the output of the shortcut path.

5. A method for turbine fault classification and diagnosis based on multi-modal fusion and meta-learning according to claim 1, characterized in that, The cross-modal attention mechanism adopted by the multimodal fusion module in Step 2 specifically includes: Use the time-domain features as the query vector, that is, Q, and the voiceprint features as the key-value vectors, that is, K and V; Dynamically adjust the weights of the time-domain features and voiceprint features through learnable parameters, where: Time-domain weight formula: ； Voiceprint weight formula: ； Wherein: is the weight matrix of the time-domain feature, which is a learnable parameter; is the weight matrix for acoustic features and is a learnable parameter; is the extracted feature vector of the time-domain signal; It is the extracted feature vector for the acoustic image; is the bias term of the time-domain feature and is a learnable parameter; is a bias term for acoustic features and is a learnable parameter; is the Sigmoid function; Further enhance the features through element-wise multiplication, and its expression is as follows: ； ； Among them is the enhanced acoustic feature vector after applying weights; The enhanced time-domain feature vector after applying weights; represents element-wise multiplication; And splice along the channel dimension to obtain the fused features, and its expression is as follows: Bimodal Feature Fusion: ; Wherein: Indicates splicing along the channel dimension.

6. A method for classifying and diagnosing faults of a hydraulic turbine based on multimodal fusion and meta-learning according to claim 1, characterized in that The Triplet Loss-KNN pre-training in Step 3 includes: Construct triplets of anchors, positive samples, and negative samples, extract feature vectors through a siamese network, and calculate the Euclidean distance between the anchor and the positive sample and the Euclidean distance between the anchor and the negative sample , using the Triplet Loss function as the optimization model, and its expression is: ； Wherein: : represents the distance between the anchor point and the positive sample; : represents the distance between the anchor point and the negative sample; : Hyperparameter that defines the minimum margin between positive and negative samples; And construct a feature vector library based on the KNN algorithm, and achieve classification through cosine similarity matching.

7. A hydroturbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 6, characterized in that Meta-learning optimization is also included in Step 3: In the N-way-1-shot task, perform L2 normalization on the support set and query set features, calculate the normalized cosine similarity, and combine the cross-entropy loss and the average entropy regularization term to obtain the optimized model, and its expression is: ； Wherein: is a hyperparameter, is the entropy mean of the predicted probability distribution of the query samples.

8. A hydroturbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 1, characterized in that The multimodal fusion module also includes dynamic positional encoding and multi-head self-attention mechanism, which are used to capture the long-range dependence relationship of the time-domain signal and the frequency-domain context information.

9. A turbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 1, characterized in that, A masking enhancement operation is introduced in the Mel spectrogram generation process to simulate the noise environment to improve the robustness of the model.

10. A turbine fault classification and diagnosis method based on multimodal fusion and meta-learning according to claim 1, characterized in that, This method further includes the joint training of industrial-end noise signals and the public dataset DCASE to enhance the generalization ability of the model.

Images