Converter valve fault identification method based on voiceprint small sample learning

Abstract

The invention provides a converter valve fault identification method based on voiceprint small sample learning, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a to-be-identified target audio comprising converter valve environment noise, and carrying out the noise reduction processing based on FRCRN, and obtaining a noise-reduced audio signal; carrying out feature extraction based on the Mel frequency cepstrum coefficient and the Gammatone filter bank cepstrum coefficient, and carrying out dimension reduction on features based on a principal component analysis method to obtain feature vectors; inputting the feature vector into a voiceprint recognition model which is completely trained based on a voiceprint small sample for recognition, and obtaining a recognition result; according to the method, the to-be-recognized audio is denoised based on the FRCRN, most noise in the converter valve environment can be filtered out, and the technical effect of accurately recognizing the to-be-recognized audio is achieved; by adjusting the structure of the voiceprint recognition network, useful information of the features can be highlighted, redundancy of repeated information is reduced, and therefore the recognition accuracy of the voiceprint recognition model in the converter valve noisy environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of converter valve fault identification, and in particular to a converter valve fault identification method based on voiceprint small sample learning. Background Art

[0002] The converter valve is the core equipment of the UHV DC transmission system, and its status directly affects the safety of the power grid and the energy transmission and consumption capacity. The cost of converter valve equipment usually accounts for more than 20% of the total construction cost of DC transmission projects. In addition, since the capacity of large-scale new energy projects is often at the GW level, once the equipment fails and damages and causes the transmission system to shut down, it will have a strong impact on the stability of the power grid system, resulting in huge direct and indirect economic losses and social impacts. When the converter valve has different faults, the different sound signals it emits contain very rich status information, which can largely reflect the operating status of the converter valve. With the rapid development of artificial intelligence, the fault diagnosis of the mechanical status of converter valve equipment using voiceprint recognition technology has become a new research hotspot.

[0003] Voiceprint recognition technology, a type of biometric information recognition technology, is a method of determining the speaker by extracting the features of human voice signals, including speaker identification and speaker confirmation. It extracts feature vectors of voice signals of different speakers to form a feature vector library, and then compares the similarity of feature vectors to determine the speaker's identity. Compared with other biometric recognition technologies, voiceprint recognition technology not only has very high security and convenience, but also has the characteristics of low equipment cost, flexible application, and high user acceptance. It has been applied to many fields, such as national defense, public security, and finance. Since the sound signal of the operating converter valve contains rich equipment information and is closely related to the structure and operating status of the converter valve, the existing voiceprint recognition technology can be applied to the converter valve fault identification, effectively remove the complex noise mechanism and identify different fault states, providing an important basis for the early warning and diagnosis of converter valve faults.

[0004] When the voiceprint recognition model based on deep learning is trained with a large amount of voice data, it can automatically learn rich acoustic features (spectrum, pitch, resonance peak, etc.), which improves the voiceprint recognition rate. However, it is not ideal for voiceprint recognition with small samples. This is because training a deep neural network requires a large number of training samples and learning millions of parameters; and for some actual task scenarios, it is impossible to obtain a large amount of voiceprint data. If only a small sample is used to train the neural network, the network cannot reach a good convergence state, resulting in a low voiceprint recognition rate.

[0005] The mainstream noise reduction algorithms currently used in the front end are traditional algorithms such as spectral subtraction, Wiener filtering, and minimum mean square error filtering. These algorithms cannot completely filter out noise and will cause certain losses to the sound information of the converter valve.

[0006] The existing sound signal feature vectors mainly include Linear Prediction Cestrum Coefficient LPCC (Linear Prediction Cestrum Coefficient) and Mel Frequency Cestrum Coefficient MFCC (Mel Frequency Cestrum Coefficient). Among them, MFCC is the mainstream feature vector for voiceprint recognition. However, the feature parameter design of MFCC is based on the human perception of sound. There is a big difference between the sound of the converter valve and the human voice. If MFCC is directly applied to the voiceprint recognition of the converter valve, its effect is limited and further research is needed.

[0007] Most of the end-to-end voiceprint recognition models currently used are based on recurrent neural networks (RNN) or convolutional neural networks (CNN). Considering that the temperature rise feature is essentially a time series data, some methods only use CNN to extract its spatial features during the feature extraction process, which has certain defects. At the same time, when using RNN to directly extract features from longer time series data, due to the high complexity of the model, the convergence speed during training is usually slow. Summary of the invention

[0008] In view of this, it is necessary to provide a converter valve fault identification method based on voiceprint small sample learning to solve the technical problems of low voiceprint recognition accuracy and low recognition efficiency in noisy scenes in the existing converter valve voiceprint recognition technology.

[0009] A method for identifying converter valve faults based on voiceprint small sample learning includes the following steps:

[0010] Step S1: Acquire audio data to be identified of the converter valve including ambient noise around the converter valve;

[0011] Step S2: performing noise reduction processing on the audio data to be recognized based on the frequency recurrent convolutional recurrent network FRCRN model to obtain noise reduction enhanced audio;

[0012] Step S3: extracting features of the noise reduction enhanced audio based on the Mel frequency cepstral coefficients MFCC and the gammatone filter bank cepstral coefficients GFCC, concatenating the two features in series, and then reducing the dimension of the concatenated features based on the principal component analysis method PCA to obtain a feature vector;

[0013] Step S4: Input the feature vector into a voiceprint recognition model that is fully trained based on a small voiceprint sample to obtain a recognition result.

[0014] Beneficial effects: Compared with the prior art, the voiceprint recognition method for the converter valve has simple operation steps and does not require special equipment. It only requires conventional microphone array sensors and computers to complete data collection, data processing, training of neural network models and recognition and classification of audio to be recognized. The voiceprint recognition method provided by the present invention for the converter valve environment can filter out most of the noise in the environment and retain the sound related to the converter valve by performing noise reduction processing on the audio to be recognized based on the frequency cycle convolution cycle network, so as to achieve the technical effect of accurately recognizing the audio to be recognized even if the audio to be recognized includes a large amount of environmental noise. Secondly, the present invention overcomes the deficiency of designing feature parameters of a single MFCC feature based on the human perception characteristics of sound by combining the two features of MFCC and GFCC, and can better fit the characteristics of the converter valve sound. Thirdly, the present invention adopts a training method of pre-training on a large-scale data set and fine-tuning on a small sample of voiceprints, so that the recognition accuracy of the voiceprint recognition model can be improved without collecting a large amount of voiceprint data. Furthermore, the present invention, by setting a voiceprint recognition model including selecting an encoder-decoder, and adjusting the depth of encoding and decoding to adjust the receptive fields of different sizes, enhances the representation capability of features, helps to extract richer and more accurate temporal and spatial feature information, and improves the recognition accuracy of the voiceprint recognition model for the sound of the converter valve to be identified with environmental noise; and, by setting a compression-expansion attention module in the encoder-decoder structure, the correlation between features can be enhanced, thereby improving the expressiveness and accuracy of the voiceprint recognition model, and further improving the recognition accuracy of the voiceprint recognition model for the sound of the converter valve to be identified with environmental noise, while improving the voiceprint recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic flow chart of an embodiment of the voiceprint recognition method provided by the present invention.

[0016] Figure 2 This is a schematic diagram of the process of performing noise reduction processing on audio data to be recognized based on the frequency recurrent convolutional recurrent network model of the present invention.

[0017] Figure 3 For the present invention Figure 2 A flowchart of an embodiment of step S202 in FIG.

[0018] Figure 4 For the present invention Figure 1 A flowchart of an embodiment of step S103 in FIG.

[0019] Figure 5 For the present invention Figure 1 A schematic flow chart of an embodiment of the pre-training stage in the voiceprint recognition model training process provided by the present invention in step S104.

[0020] Figure 6 A schematic diagram of the structure of an encoder-decoder according to an embodiment of the present invention.

[0021] Figure 7 For the present invention Figure 1 FIG. 1 is a flow chart of an embodiment of a voiceprint recognition model provided by the present invention in step S104. DETAILED DESCRIPTION

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Please see Figure 1 The converter valve fault identification method based on voiceprint small sample learning includes the following steps:

[0024] Step S1: Acquire audio data to be identified of the converter valve including ambient noise around the converter valve;

[0025] In a preferred embodiment, the specific method of obtaining the audio data to be identified of the converter valve including the ambient noise around the converter valve is: a voiceprint acquisition device composed of a microphone array sensor acquires the audio to be identified in real time, or the audio data to be identified is acquired by calling from a storage device storing the audio to be identified.

[0026] Step S2: performing noise reduction processing on the audio data to be recognized based on a frequency recurrent convolutional recurrent network (FRCRN) model to obtain noise reduction enhanced audio;

[0027] Among them, please see Figure 2 The frequency recurrent convolutional recurrent network model includes an encoder module, an attention module, a recurrent module, and a decoder module.

[0028] Based on the encoder module, encoding the Fourier transform result to improve the feature expression capability along the frequency axis and obtain an encoder output with reduced dimension;

[0029] Based on the cyclic module, the output of the encoder is processed to enhance the temporal dynamic characteristics of the features, and the output of the cyclic module is obtained; the cyclic module is composed of two time domain FSMNs, and the output of the cyclic module is obtained.

[0030] Based on the compression-expansion function of the attention module, each small attention unit performs compression-expansion operations on the outputs of the encoder and the decoder respectively to obtain the attention output of the encoder and the attention output of the decoder;

[0031] Based on the decoder module, for the first decoder, the output of the loop module is obtained, and then passed through the attention block of the decoder to obtain the attention output of the first decoder; the second decoder concatenates the attention output of the first decoder and the attention output of the encoder corresponding to the penultimate position as input to obtain the output of the second decoder, and so on to obtain the dimensionally restored output of the decoder module;

[0032] Performing an inverse short-time Fourier transform (ISTFT) on the output of the decoder module to obtain enhanced audio frames, and splicing the enhanced audio frames to obtain the noise-reduced and enhanced audio data.

[0033] For details, please see Figure 3 In a preferred embodiment, the steps of performing noise reduction processing on the audio data to be recognized based on the FRCRN model to obtain noise reduction enhanced audio are as follows:

[0034] Step S21: performing a frame operation on the audio data to be recognized, and performing a short-time Fourier transform (STFT) frame by frame to obtain an STFT result of each audio frame;

[0035] Step S22: input each frame of STFT result into the fully trained FRCRN model for denoising to obtain a denoised and enhanced STFT;

[0036] Step S23: Performing an inverse short-time Fourier transform (ISTFT) on the result to obtain an audio frame after noise reduction and enhancement;

[0037] Step S24: rejoin the noise reduction enhanced audio frames to obtain noise reduction enhanced audio.

[0038] In a preferred embodiment, the encoder module includes a plurality of encoders connected in series, the encoder includes a two-dimensional convolution kernel and a frequency domain feedforward sequential memory network FSMN, and the STFT result is encoded in the encoder to become a feature vector. In the encoder module, except for the terminal encoder, the feature vectors of other encoders each pass through a compression-and-excitation (SE) attention unit. Specifically, the two-dimensional convolution kernel in the encoder performs convolution encoding on the feature vector in turn, and each encoder reduces the dimension of the feature vector to half of that at the input, while the number of channels continues to increase to twice that of the input, to obtain an encoder output with reduced dimension. Based on the compression-expansion attention module, except for the last encoder that does not require an attention block, the other encoders correspond to an attention module, and the attention module first compresses the channel dimension of the feature vector output by the encoder, and then expands the channel dimension.

[0039] Similarly, the decoder module includes multiple decoders connected in series, and the number of decoders is the same as the number of encoders. The decoder includes a two-dimensional deconvolution kernel and a frequency domain FSMN. The two-dimensional deconvolution kernel performs deconvolution encoding on the feature vector and outputs it to the frequency domain FSMN. The feature vector is then output to the attention module for channel dimension compression and then channel dimension expansion. The feature vector processed by the two-dimensional convolution kernel and the two-dimensional deconvolution kernel is concatenated and input to the two-dimensional deconvolution kernel of the next decoder, and then output to the frequency domain FSMN. This process repeats until the decoder at the end outputs the denoised feature vector to obtain denoised enhanced audio.

[0040] Then, step S3 is performed to extract features of the noise reduction enhanced audio based on the Mel frequency cepstral coefficient MFCC and the gammatone filter group cepstral coefficient GFCC, and the two features are concatenated in series, and then the concatenated features are reduced in dimension based on the principal component analysis method PCA to obtain a feature vector;

[0041] In a preferred embodiment, see Figure 4 , the feature extraction method includes the following steps:

[0042] S31: extracting MFCC features from the above-mentioned noise reduction and enhanced audio;

[0043] S32: extracting GFCC features from the above-mentioned noise reduction and enhanced audio;

[0044] S33: performing feature splicing on the extracted MFCC and GFCC features to obtain a spliced ​​feature vector;

[0045] S34: Based on the principal component analysis method PCA, the concatenated feature vector is reduced in dimension to obtain a reduced-dimensional feature vector.

[0046] Wherein, in step S31, the MFCC features of the noise reduction and enhancement audio are extracted, including pre-processing of pre-emphasis, framing, and windowing operations to obtain an audio frame to be recognized;

[0047] The purpose of pre-emphasis is to increase the high-frequency components of the sound signal and make the entire signal spectrum flatter. It is generally achieved by using a high-pass filter, which is defined as:

[0048] H(z)=1-μz -1

[0049] Wherein, μ is the pre-emphasis coefficient, and its value is between 0.9375 and 0.97, and is taken as 0.95 in the present invention; if the signal before pre-emphasis is x[n], and the signal after pre-emphasis is y[n], then:

[0050] y[n]=x[n]-μx[n-1]

[0051] The purpose of framing is to divide a sound signal into several short-time signals with short-time stability. Each short-time signal constitutes a frame. The formula is:

[0052]

[0053] Where m is the number of frames, n is the total length of the audio to be recognized, l is the frame length, and s is the frame shift;

[0054] The purpose of windowing is to increase the continuity of the left and right ends of the frame. The commonly used window function is the Hamming window, which is defined as:

[0055]

[0056] After preprocessing, the time domain frame signal x(n) is obtained, and then x(n) is transformed by fast Fourier transform FFT to obtain the linear spectrum X(k). The transformation formula is:

[0057]

[0058] The linear spectrum X(k) is squared to obtain the amplitude spectrum, and then the Mel filter is performed through an M-dimensional Mel filter bank to calculate its logarithmic energy:

[0059] There is a nonlinear correspondence between the frequency of the sound signal heard by the human ear and its actual frequency, namely the Mel frequency. The relationship between the Mel frequency and the actual frequency can be expressed as:

[0060]

[0061] Mel filtering is implemented by a filter bank consisting of several triangular bandpass filters. The transfer function of each triangular filter is defined as:

[0062]

[0063] Wherein, M is the order of the Mel filter, which is determined by the cutoff frequency of the sound signal. In the present invention, the value is 24, and f m is the center frequency of the Mel filter;

[0064] Calculate the logarithmic energy me(m), the formula is:

[0065]

[0066] The above logarithmic energy spectrum me(m) is subjected to discrete cosine transform DCT to obtain the MFCC coefficient M(n), whose formula is:

[0067]

[0068] In S32, the GFCC features extracted from the above-mentioned noise reduction enhanced audio are specifically:

[0069] Performing pre-processing including pre-emphasis, framing and windowing operations on the audio data to be recognized to obtain an audio frame to be recognized;

[0070] After preprocessing, the time domain frame signal x(n) is obtained, and then x(n) is transformed by fast Fourier transform FFT to obtain the linear spectrum X(k);

[0071] The linear spectrum X(k) is squared to obtain the amplitude spectrum, which is filtered by the Gammatone filter bank and its logarithmic energy is calculated:

[0072] The expression of the jth filter of the Gammatone filter bank is:

[0073]

[0074] In the formula, A, f 0 are the gain and center frequency of the Gammatone filter, is the phase, a is the order of the filter, N is the number of filters, b j is the attenuation factor, which is related to the bandwidth of the jth filter and its formula is:

[0075] b j =1.019ERB(f j )

[0076]

[0077] In the formula, ERB(f j ) is the equivalent rectangular bandwidth, and the center frequency of each filter is equally divided on the ERB scale and then mapped to the linear scale;

[0078] Calculate the logarithmic energy ge(m) using the formula:

[0079]

[0080] In the formula, G m (k) is the Gammatone filter g j (t) transfer function;

[0081] The above logarithmic energy spectrum ge(m) is subjected to discrete cosine transform DCT to obtain the GFCC coefficient G(n), whose formula is:

[0082]

[0083] In step S33, the extracted MFCC and GFCC features are spliced ​​to obtain a spliced ​​feature vector, specifically:

[0084] Assume that the MFCC coefficient obtained by S31 is a={a 1 ,a 2 ,…,a n}, the GFCC coefficient obtained by S32 is b = {b 1 ,b 2 ,…,b m}, where n and m are the dimensions of MFCC and GFCC features, a n and b m is a column vector, and the two are connected in series to obtain the fused feature vector c = {a 1 ,a 2 ,…,a n ,b 1 ,b 2 ,…,b m}.

[0085] In step S34, based on the principal component analysis method PCA, the concatenated feature vector is reduced in dimension to obtain the reduced-dimensional feature vector:

[0086] The concatenated fusion feature vector c obtained in step S33 is reduced in dimension based on the principal component analysis method PCA to obtain a reduced-dimensional feature vector:

[0087] The basic idea of ​​PCA is to generate a new eigenvector by linearly combining the dimensional components of the original eigenvector, so that the dimensional components of the new eigenvector are independent of each other and the variance is as large as possible;

[0088] Suppose there are e eigenvectors forming a matrix G, and the dimension of each eigenvector is h, then G can be expressed as:

[0089]

[0090] Calculate the correlation matrix R of G as:

[0091] R=G T G / (e-1)

[0092] Based on this, the eigenvalue λ of the correlation matrix R is calculated 1 ,λ 2 ,…,λ h and the corresponding eigenvector u 1 ,u 2 ,…,u h ;

[0093] Calculate the variance contribution rate η i And the cumulative variance contribution rate η(l) is:

[0094]

[0095] The number of principal components is generally determined based on the cumulative variance contribution rate. Usually, l that makes the cumulative variance contribution rate greater than 75% is selected as the number of principal components, that is, the dimension of the reduced eigenvector;

[0096] Calculate the new eigenvector. The l eigenvectors corresponding to the l principal components are U = [u 1 ,u 2 ,…,u l ], then the new eigenvector matrix Z can be expressed as:

[0097] Z=GU

[0098] The final eigenvector can be obtained from the eigenvector matrix Z.

[0099] Finally, according to step S4: the feature vector is input into a voiceprint recognition model that is fully trained based on a small voiceprint sample for recognition to obtain a recognition result.

[0100] The voiceprint recognition model includes a convolutional neural network (CNN) and a long short-term memory (LSTM) network.

[0101] The training of the voiceprint recognition model is divided into two stages: large-scale data pre-training and voiceprint small sample fine-tuning:

[0102] Please check Figure 5 , is a flow chart of an embodiment of the pre-training stage in the voiceprint recognition model training process provided by the present invention. The pre-training stage is divided into two steps: constructing a training set and training. Specifically,

[0103] For the step of constructing the dataset, we use the noise set in the DNS-Challenge dataset uploaded by Microsoft on Github as a large-scale dataset with a size of 20GB, and perform pre-emphasis on each noise in it;

[0104] Extract the MFCC and GFCC features of the noise and concatenate the two features in series. Perform dimension reduction on the concatenated features based on principal component analysis to obtain the feature vector corresponding to each noise. All feature vectors form the training set, validation set, and test set in a ratio of 7:2:1.

[0105] For the training step, a batch is selected from the training set, which contains all the feature vectors of several noise types; from this batch, a feature vector of a noise type is selected as an anchor each time, denoted by x a , select another feature vector in the same class as the anchor point as a positive sample (positive), denoted by x+ , select a feature vector in a different class from the anchor point as a negative sample (negative), denoted by x — ;

[0106] The feature vector x a 、x + and x — The input is sent to the convolutional neural network CNN and the long short-term memory network LSTM respectively. The two networks are denoted as f, and the output feature f(x a )、f(x + ) and f(x — );

[0107] Calculate the output feature f(x a ) and f(x + ) and f(x a ) and f(x — ), respectively denoted as d + and d — , the formula is:

[0108]

[0109]

[0110] Utilize d + and d — Calculate the triplet loss (TripletLoss), the loss function formula is:

[0111] L=max{0,d + +α-d -},α>0

[0112] In the formula, α is the offset, and the optimization goal of pre-training is to minimize the triple loss function so that the distance d between the positive sample and the anchor point is + As small as possible, the distance d between the negative sample and the anchor point — As large as possible, that is, the similarity between the positive sample and the anchor point is as large as possible, and the similarity between the negative sample and the anchor point is as small as possible;

[0113] The fine-tuning phase includes two steps: constructing a fine-tuning dataset and fine-tuning. Specifically:

[0114] For the step of constructing the fine-tuning dataset, the sample data in the fine-tuning dataset is the collected and labeled audio data of the converter valve, which contains different fault types that occur in the converter valve during operation, and the number is recorded as n. Each fault type contains 10 audio data, and the number is recorded as j. Each audio has a one-hot code representing its type, recorded as y. j,The fine-tuning dataset is divided into a support set (Support Set) and a query set (Query Set) in a ratio of 7:3, that is, for each fault type, the support set has 7 data and the query set has 3 data. The support set and query set can be regarded as the training set and test set of the fine-tuning step;

[0115] After preprocessing the audio data in each type of fault, extracting and concatenating MFCC and GFCC features, and reducing the dimension of the feature vector based on principal component analysis, the voiceprint recognition model f is used to extract features, denoted as f(x j ), and perform averaging operation on the features of the same type of faults to obtain the mean vectors corresponding to different faults, denoted as μ 1 ,μ 2 ,…,μ n , μ 1 ,μ 2 ,…,μ n Splice into matrix M, M is:

[0116]

[0117] For the fine-tuning step, f(x j ) is recorded as p j , the formula is:

[0118]

[0119] In the formula, W and b are the learnable feature matrix and bias in fine-tuning, respectively. Initialize W = M, b = 0, and replace the inner product term in the above formula with the calculated cosine similarity, then p j The formula becomes:

[0120]

[0121] In the formula, the calculation formula of the cosine similarity sim(w,f) function is:

[0122]

[0123] Calculate y j and p j The cross entropy of W and b is learned by minimizing the cross entropy loss function, as follows:

[0124] min∑ j CrossEntropyLoss(y j ,p j )

[0125] In the formula, ∑ jThe cross entropy loss is summed over all features in the support set to learn the classifier Softmax to get the expected prediction value p j As close as possible to the true label y j ;

[0126] In order to prevent overfitting, an entropy regularization term is added to the above loss function, specifically:

[0127] Let x be a data in the query set. i , calculate x i The predicted probability is:

[0128] p i =Softmax(sim(W,f(x i ))+b)

[0129] P i All the probabilities p in ij Calculate entropy using the formula:

[0130] H(p i )=-Σ j p ij log(p ij )

[0131] Calculate the entropy of all data in the query set and take the average to get the entropy regularization term. The formula is:

[0132]

[0133] In the formula, N is the total number of query set data. During the fine-tuning process, the entropy regularization term is minimized to achieve the regularization effect.

[0134] In a preferred embodiment, see Figure 6 The recognition steps of the voiceprint recognition model of the present invention include:

[0135] Step S41: extracting spatial information from the feature vector obtained in step S34 through a convolutional neural network (CNN);

[0136] S42, extracting temporal information from the acquired spatial information through a long short-term memory network LSTM;

[0137] S43, calculating the cosine similarity between the obtained space-time information and the feature vector of the support set to obtain the final recognition result.

[0138] The difference between convolutional neural network and recurrent convolutional recurrent network model is that convolutional neural network has no recurrent module, and the working process of the two is similar.

[0139] For Convolutional Neural Networks:

[0140] The feature vector output by the encoder is first subjected to an average pooling operation to average the spatial domain dimensions, that is, the spatial domain dimensions disappear;

[0141] The average feature vector passes through a Linear layer and a ReLU activation function layer to compress the channel dimension and obtain the feature vector compressed in the channel dimension;

[0142] The dimensionally compressed feature vector passes through a Linear layer and a Sigmoid activation function layer, and the channel dimension is expanded to obtain the feature vector restored in the channel dimension.

[0143] Expand the above feature vector by a dimension in the spatial domain so that the output and the input keep the same number of dimensions, and obtain the input of the decoder module;

[0144] Please see Figure 7 , based on the decoder module, which consists of multiple decoders. Each decoder is a two-dimensional deconvolution kernel. The feature vectors are deconvoluted and decoded in turn to obtain the decoder output with dimension restoration:

[0145] For the first decoder, get the output of the last encoder for decoding, and then pass it through the attention block of the first decoder to get the attention output of the first decoder;

[0146] The attention output of the first decoder is concatenated with the attention output of the encoder corresponding to the penultimate position in the channel dimension to obtain the input of the second decoder, and the output of the second decoder is obtained after decoding;

[0147] And so on, the dimensionally restored output of the decoder module is obtained;

[0148] Step 43 is specifically as follows:

[0149] Based on the long short-term memory network LSTM, the time information between frames is extracted from the output of the convolutional neural network CNN, and the cosine similarity between the feature vectors of the support set is calculated. The category with the largest similarity is the recognition result of the voiceprint recognition model.

[0150] The above is a detailed introduction to the voiceprint recognition method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. For technicians in this technical field, they can still modify the technical solutions recorded in the aforementioned specific implementation methods, or make equivalent transformations or replacements for some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of the present invention, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A method for identifying converter valve faults based on voiceprint small sample learning, characterized in that: The following steps are involved: Step S1: Acquire audio data to be identified of the converter valve including ambient noise around the converter valve; Step S2: performing noise reduction processing on the audio data to be recognized based on the frequency recurrent convolutional recurrent network FRCRN model to obtain noise reduction enhanced audio; Step S3: extracting features of the noise reduction enhanced audio based on the Mel frequency cepstral coefficients MFCC and the gammatone filter bank cepstral coefficients GFCC, concatenating the two features in series, and then reducing the dimension of the concatenated features based on the principal component analysis method PCA to obtain a feature vector; Step S4: Input the feature vector into a voiceprint recognition model that is fully trained based on a small voiceprint sample to obtain a recognition result.

2. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 1, characterized in that: In step S1, the specific method of obtaining the audio data to be identified of the converter valve including the ambient noise around the converter valve is: a voiceprint collection device composed of a microphone array sensor collects the audio to be identified in real time.

3. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 1, characterized in that: In step S2, the audio data to be recognized is subjected to denoising based on the FRCRN model, and the steps for obtaining denoising enhanced audio are as follows: Step S21: performing a frame operation on the audio data to be recognized, and performing a short-time Fourier transform (STFT) frame by frame to obtain an STFT result of each audio frame; Step S22: input each frame of STFT result into the fully trained FRCRN model for denoising to obtain a denoised and enhanced STFT; Step S23: Performing an inverse short-time Fourier transform (ISTFT) on the result to obtain an audio frame after noise reduction and enhancement; Step S24: rejoin the noise reduction enhanced audio frames to obtain noise reduction enhanced audio.

4. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 1, characterized in that: In step S3, feature extraction of the noise reduction enhanced audio based on the Mel frequency cepstral coefficients MFCC and the Gammatone filter bank cepstral coefficients GFCC includes the following steps: S31: extracting MFCC features from the above-mentioned noise reduction and enhanced audio; S32: extracting GFCC features from the above-mentioned noise reduction and enhanced audio; S33: performing feature splicing on the extracted MFCC and GFCC features to obtain a spliced ​​feature vector; S34: Based on the principal component analysis method PCA, the concatenated feature vector is reduced in dimension to obtain a reduced-dimensional feature vector.

5. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 4, characterized in that: In S32, extracting GFCC features from the above-mentioned noise reduction enhanced audio includes the following steps: Step S321: performing pre-processing including pre-emphasis, framing, and windowing operations on the audio data to be recognized, to obtain an audio frame to be recognized; Step S322: After preprocessing, a time domain frame signal x(n) is obtained, and then x(n) is transformed by fast Fourier transform FFT to obtain a linear spectrum X(k); Step S323: square the modulus of the linear spectrum X(k) to obtain an amplitude spectrum, filter it through a Gammatone filter bank, and calculate its logarithmic energy.

6. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 6, characterized in that: Filtering through the Gammatone filter bank and finding its logarithmic energy includes the following steps: Step S3231: The expression of the jth filter of the Gammatone filter bank is: In the formula, A and f0 are the gain and center frequency of the Gammatone filter respectively. is the phase, a is the order of the filter, N is the number of filters, b j is the attenuation factor, which is related to the bandwidth of the jth filter and its formula is: b j =1.019ERB(f j ) In the formula, ERB(f j ) is the equivalent rectangular bandwidth, and the center frequency of each filter is equally divided on the ERB scale and then mapped to the linear scale; Step S3232: Calculate the logarithmic energy ge(m), the formula is: In the formula, G m (k) is the Gammatone filter g j (t) transfer function; Step S3233: The logarithmic energy spectrum ge(m) is subjected to discrete cosine transform DCT to obtain the GFCC coefficient G(n), the formula of which is:

7. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 4, characterized in that: In step S34, based on the principal component analysis method PCA, the concatenated feature vector is reduced in dimension, and obtaining the reduced-dimensional feature vector includes the following steps: Step S341: Based on the principal component analysis method PCA, the concatenated fusion feature vector c obtained in step S33 is reduced in dimension to obtain a reduced-dimensional feature vector: Step S342: Assume that there are e eigenvectors forming a matrix G, the dimension of each eigenvector is h, and G is expressed as: Step S343: Calculate the correlation matrix R of G, which is: R=G T G / (e-1); Step S344: Calculate the eigenvalues ​​λ1,λ2,…,λ of the correlation matrix R h and the corresponding eigenvectors u1,u2,…,uh; Step S345: Calculate variance contribution rate η i And the cumulative variance contribution rate η(l) is: Step S346: Calculate new eigenvectors; the l eigenvectors corresponding to the l principal components are U = [u1, u2, ..., u l ], the new eigenvector matrix Z is: Z = GU; The final reduced-dimensional eigenvector is obtained from the eigenvector matrix Z.

8. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 1, characterized in that: The training of the voiceprint recognition model includes the pre-training stage and the voiceprint small sample fine-tuning stage. The pre-training stage includes the following steps: Step S411: using the noise set in the DNS-Challenge dataset uploaded by Microsoft on Github as a large-scale dataset, and performing pre-emphasis processing on each noise therein; Step S412: extract the MFCC and GFCC features of the noise and concatenate the two features in series, perform dimension reduction on the concatenated features based on principal component analysis, and obtain the feature vector corresponding to each noise. All feature vectors form a training set (Training Set), a validation set, and a test set in a ratio of 7:2:1; Step S413: Select a batch from the training set, the batch containing all feature vectors of several noise types; from the batch, select a feature vector of one noise type at a time as an anchor, denoted by x a , select another feature vector in the same class as the anchor point as a positive sample (positive), denoted by x + , select a feature vector in a different class from the anchor point as a negative sample (negative), denoted by x — ; Step S414: Transform the feature vector x a 、x + and x — The input is sent to the convolutional neural network CNN and the long short-term memory network LSTM respectively. The two networks are denoted as f, and the output feature f(x a )、f(x + ) and f(x — ); Step S415: Calculate the output feature f(x a ) and f(x + ) and f(x a ) and f(x — ), respectively denoted as d + and d — , the formula is: Step S416: Using d + and d — Calculate the triplet loss (TripletLoss), the loss function formula is: L=max{0,d + +a-d - },α>0; Where α is the offset.

9. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 8, characterized in that: The fine-tuning phase includes the following steps: Step S421: The sample data in the fine-tuning data set is the collected and labeled converter valve audio data, which contains different fault types that occur in the converter valve during operation, and the number is recorded as n. Each fault type contains 10 audio data, and the number is recorded as j. Each audio has a one-hot code representing its type, recorded as y. j ,The fine-tuning dataset is divided into a support set (SupportSet) and a query set (QuerySet) in a ratio of 7:3, that is, for each fault type, the support set has 7 data and the query set has 3 data. The support set and query set can be regarded as the training set and test set of the fine-tuning step; Step S422: After preprocessing the audio data in each type of fault, extracting and concatenating MFCC and GFCC features, and reducing the dimension of the feature vector based on principal component analysis, the voiceprint recognition model f is used to extract features, denoted as f(x j ), and perform averaging operation on the features of the same type of faults to obtain the mean vectors corresponding to different faults, denoted as μ1,μ2,…,μ n , μ1,μ2,…,μ n Splice into matrix M, M is: Step S423: f(x j ) is recorded as p j , the formula is: Where W and b are the learnable feature matrix and bias in fine-tuning, respectively. Initialize W = M, b = 0. Step S423: Replace the inner product term in the above formula with the calculated cosine similarity, then p j The formula becomes: In the formula, the calculation formula of the cosine similarity sim(w,f) function is: Step S423: Calculate y j and p j The cross entropy of W and b is learned by minimizing the cross entropy loss function, as follows: min∑ j CrossEntropyLoss(y j ,p j ); In the formula, ∑ j The cross entropy loss is summed over all features in the support set to learn the classifier Softmax to get the expected prediction value p j As close as possible to the true label y j。 10. The method for identifying converter valve faults based on voiceprint small sample learning according to claim 9, characterized in that: The recognition steps of the voiceprint recognition model of the present invention include: Step S41: extracting spatial information from the feature vector obtained in step S34 through a convolutional neural network (CNN); S42, extracting temporal information from the acquired spatial information through a long short-term memory network LSTM; S43, calculating the cosine similarity between the obtained space-time information and the feature vector of the support set to obtain a final recognition result.

Images