Convolutional recurrent neural network voice noise reduction method based on self-attention mechanism

Abstract

The invention belongs to the technical field of artificial intelligence and neural networks, and relates to a convolutional recurrent neural network voice noise reduction method based on a self-attention mechanism, which comprises a training stage and an enhancement stage. In the training stage, firstly, noise and pure voice are superposed to synthesize noisy voice, then feature extraction is performed on the noisy voice, then the pure voice and the noisy voice after feature extraction are sent into a convolutional recurrent neural network model to learn a mapping relation between the noisy voice and the pure voice, and after the training stage is finished, the pure voice and the noisy voice are subjected to feature extraction to learn the mapping relation between the noisy voice and the pure voice. Storing the trained model; and the enhancement stage comprises the steps of firstly performing feature extraction on noisy voice, then sending the noisy voice into the trained model for voice noise reduction, and finally outputting the voice after noise reduction. And effective separation of voice and noise is realized. And the separated signal is restored to a clear voice dimension, so that the whole voice noise reduction process is completed.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence and neural network technology, relates to audio processing technology, and provides a convolutional recurrent neural network speech denoising method based on a self-attention mechanism. Background Art

[0002] With the rapid development of digital communication and audio processing technology, speech noise reduction has become increasingly important in various applications. Digital audio evidence in criminal cases is often accompanied by complex environmental noise, and traditional speech noise reduction methods, such as spectral subtraction and Wiener filtering, are not effective in reducing the noise of such noisy speech. These traditional methods often rely on assumptions about noise characteristics and are difficult to adapt to dynamically changing environments and complex noise types.

[0003] With the development of artificial intelligence technology, especially deep learning methods, speech noise reduction methods based on neural networks have begun to attract attention. This method can more accurately identify and process various noises by learning a large amount of data. However, existing speech noise reduction technologies based on neural networks often have problems such as poor generalization, speech distortion, low signal-to-noise ratio, and poor noise reduction effects. In-depth analysis and research have found that different noises have different performance characteristics in the time-frequency domain, but most deep learning-based algorithms pay no attention to noise features of different dimensions, which leads to incomplete noise reduction or distortion of the speech spectrum.

[0004] Therefore, there is an urgent need for a speech noise reduction method that can effectively adapt to various noise environments and can focus on processing important features in speech signals. In order to meet the growing communication and audio processing needs, it should be introduced into the public security and judicial fields, which will promote the construction of modern intelligent policing. Summary of the invention

[0005] In view of the deficiencies in the prior art, in particular to improve the denoising effect of speech containing non-stationary noise, the present invention proposes a convolutional recurrent neural network speech denoising method based on a self-attention mechanism, which can not only increase the temporal relationship between different speech frames to reduce the impact of the time-varying characteristics of non-stationary noise, but also expand the dimensional information of a single speech frame, so that the model can accurately identify speech and noise, and enhance the model's ability to reconstruct pure speech, thereby achieving the best effect of model denoising.

[0006] The specific technical solution adopted by the present invention is as follows:

[0007] A convolutional recurrent neural network speech denoising method based on a self-attention mechanism, including a training phase and an enhancement phase;

[0008] The training phase: firstly, the noise and the clean speech are superimposed to synthesize the noisy speech, then the feature of the noisy speech is extracted, and then the clean speech and the noisy speech after the feature extraction are sent to the convolutional recurrent neural network model to learn the mapping relationship between the noisy speech and the clean speech, and after the training phase is completed, the trained model is saved;

[0009] The enhancement stage: firstly, the noisy speech is feature extracted, then sent to the trained model for speech noise reduction, and finally the noise-reduced speech is output.

[0010] It also includes a preferred stage: the convolutional recurrent neural network model uses an encoder-decoder structure as a deep learning architecture, the encoder obtains compressed data by removing the noise part in the noisy speech, and then inputs the data into the decoder to reconstruct the data to restore the estimated form of the pure speech.

[0011] The convolutional recurrent neural network model consists of five parts: position embedding layer, feature extraction layer, multi-head self-attention mechanism layer and dimension reduction layer;

[0012] The position embedding layer embeds the speech frame after the local long short-term memory network module and then passes through the encoder position encoding; the feature extraction layer increases the information contained in a single frame of speech through the decoder, and uses the multi-head attention mechanism to assign different weights to the noisy speech, thereby improving the model's attention to the difference between noise and speech; the transposed convolution module in the decoder is used as a dimensionality restoration layer to sample and restore the speech blocks of different dimensions after the self-attention mechanism weights, ensuring the original dimensional relationship of the speech, thereby improving the auditory quality of the denoised speech, thereby completing speech denoising and obtaining the predicted speech frame.

[0013] The position embedding layer changes the original Transformer position embedding to a local LSTM module that is more suitable for speech denoising, which not only achieves the effect of efficient data processing, but also takes into account the characteristic that the noise information of non-stationary noise is only related to the speech information of the previous and next frames, and can better capture the local information between speech frames;

[0014] The feature extraction layer includes a convolution layer, a normalization layer and a linear rectification function ReLU activation function layer;

[0015] The multi-head self-attention mechanism layer introduces an attention mechanism, which enables the model to perform nonlinear weight distribution on each frame of noisy speech according to different noise conditions; the model can thus pay more attention to each frequency point in the noisy speech and give different attention levels to noise and speech; in this way, the model can more effectively distinguish between noise and speech components in noisy speech, laying the foundation for further comprehensive and accurate removal of the noise part;

[0016] The dimension reduction layer includes a transposed convolution layer, a normalization layer and a Relu activation function layer; in order to avoid the potential gradient vanishing problem caused by the codec module, this model adds a residual link in each corresponding convolution module and transposition module; it can not only effectively prevent the occurrence of gradient vanishing, but also help prevent the loss of speech information in the deep network transmission.

[0017] The convolutional recurrent neural network speech denoising method based on the self-attention mechanism includes the following calculation steps:

[0018] Step 1: Use a local LSTM module that is more suitable for speech noise reduction to positionally embed the noisy speech frames. In terms of the number of frames, the average pronunciation duration of each phoneme is about 30 milliseconds, and the pronunciation of a syllable is between 120 and 150 milliseconds. Therefore, every 7 frames are selected as a speech frame segment, and the frame shift is set to 1 frame. Then, the position of each speech frame segment is embedded through the local LSTM with shared weights.

[0019] Step 2: The position-encoded speech frame is passed through the CRN network decoder to increase the information contained in a single frame of speech; in the convolution module, each convolution module contains a convolution layer, a batch normalization layer, and a relu activation function; the noisy speech features are superimposed so that the spliced ​​speech frame incorporates rich multi-dimensional information;

[0020] Step 3: Self-attention mechanism structure, the input vector is scaled dot product through h K, Q and V matrices to obtain the similarity between two vectors; where h is the number of K, Q and V, the input vector is mapped to different spaces through multiple weight matrices; then the h output results are concatenated using the concat function and converted back to the input shape through linear mapping;

[0021] Step 4: Use the transposed convolution module as a dimensionality reduction layer, which consists of a transposed convolution layer, a batch normalization layer, and an activation function. Upsample and restore the speech blocks of different dimensions after the self-attention mechanism weighting to ensure the original dimensional relationship of the speech.

[0022] Step 5: Adding residual links to each corresponding convolution module and transposition module can not only avoid gradient vanishing, but also prevent information loss caused by speech information flowing in the deep network; the processing process of the neural network model is now completed, and the denoised speech is output.

[0023] The step 3 specifically includes the following steps:

[0024] Step 3.1: First, the Q, K and V matrices are obtained by formula 1:

[0025]

[0026] Among them, x is the encoder output result, W Q , W K , W V are weight matrices respectively;

[0027] Step 3.2: After using the reshape function to transform the dimension, we can get from formula 2 that Q and K T Do the scaling point product to obtain the similarity between each frequency point, and use the softmax function for normalization to form the similarity probability W between different noisy speech frames. i ;

[0028]

[0029] Among them, d k The size of the input vector, divided by It can prevent the gradient from disappearing during calculation;

[0030] Step 3.3: Multiply the obtained similarity probability W by V to obtain the weighted vector matrix of the attention mechanism, as shown in Formula 3;

[0031] Attention(Q,K,V)=W i ·V Formula 3

[0032] Step 3.4: After obtaining the h attention weighted matrices, use the concat function to concatenate and transform the dimensions, and transform them to the same size as the input through linear mapping, as shown in Formula 4;

[0033] MultiHead(Q,K,V)=Linear(concat(W i ,...,W h )) Formula 4

[0034] The convolutional recurrent neural network speech denoising method based on the self-attention mechanism also includes calculation step 6:

[0035]

[0036] Where n is the number of convolution modules and transposed convolution modules; x noise is the input noisy speech frame, x is the clean speech frame after noise reduction, and the output is the noise-reduced speech.

[0037] The advantages of the present invention are:

[0038] 1. The present invention proposes a convolutional recurrent network speech denoising model based on a multi-head self-attention mechanism. The model uses local LSTM to positionally encode the input speech frame and increases the dimensional information of a single speech frame through a convolution module. By using the self-attention mechanism in the Transformer, weights are assigned to each frequency point of the noisy speech frame, thereby achieving effective separation of speech and noise. The separated signal is restored to a clear speech dimension through a transposed convolution module, thereby completing the entire speech denoising process.

[0039] 2. The present invention combines the advantages of attention and convolution. While breaking through the limitation that convolutional neural networks cannot perform parallel calculations, it uses the attention mechanism to give it the ability to process longer speech sequences, achieving the effect of local and global modeling of speech information.

[0040] 3. The present invention compares different models of noisy speech under various signal-to-noise ratios and finds that the convolutional recurrent network based on the self-attention mechanism has better noise reduction effects than other models under various noise types. At the same time, the PESQ and STOI scores are significantly improved. By comparing the spectrograms, it is found that the improved model can more effectively remove noise and improve the auditory quality of speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings of the embodiment are briefly introduced below.

[0042] Figure 1 is a flow chart of the speech noise reduction method of the present invention;

[0043] Figure 2 It is a neural network model diagram of the present invention;

[0044] Figure 3 It is a schematic diagram of the local LSTM position encoding steps of the present invention;

[0045] Figure 4 It is a structural diagram of the self-attention mechanism of the present invention;

[0046] Figure 5 is a flow chart of the self-attention mechanism of the present invention;

[0047] Figure 6 The software architecture of the speech intelligent analysis visualization system of the present invention is shown in the figure;

[0048] Figure 7 It is the functional module design diagram of the whole speech intelligent analysis visualization system software of the present invention;

[0049] Figure 8 is the pure speech spectrogram used in the present invention;

[0050] Fig. 9 The present invention is used in Figure 8 Spectrogram with stationary noise and narrowband noise added to the speech shown;

[0051] Fig.10 It is a spectrogram of speech after noise reduction using the LSTM method of the present invention;

[0052] Fig.11 It is a spectrogram of speech after noise reduction using the CRN method of the present invention;

[0053] Fig.12 is a spectrogram of speech after noise reduction using the A-CRN method of the present invention;

[0054] Fig.13 It is a spectrogram of speech after noise reduction using the Tra-CRN method of the present invention. Specific implementation methods

[0056] The following is combined with Figure 1-Figure 13 The specific embodiments of the present invention are described in detail, and the principles of the present application are explained through examples. Other aspects, features and advantages of the present application will become clear through the detailed description. In the drawings referred to, the same or similar parts in different figures are represented by the same reference numerals.

[0057] The present application provides a convolutional recurrent neural network speech denoising method based on a self-attention mechanism, including a training phase and an enhancement phase:

[0058] Training phase: First, the noise and clean speech are superimposed to synthesize noisy speech, then the features of the noisy speech are extracted, and then the clean speech and the noisy speech after feature extraction are sent to the neural network model to learn the mapping relationship between noisy speech and clean speech. After the training phase is completed, the trained model is saved;

[0059] Enhancement stage: First, the noisy speech is feature extracted, then sent to the trained model for speech noise reduction, and finally the noise-reduced speech is output.

[0060] Optimization stage: The neural network model uses an encoder-decoder structure as a deep learning architecture. Generally, the encoder obtains compressed data by removing the noise part of the noisy speech, and then inputs the data into the decoder to reconstruct it to restore the estimated form of the pure speech.

[0061] Further, the convolutional recurrent neural network (Tra-CRN) network model based on the self-attention mechanism used in this application is composed of five parts, namely, a position embedding layer, a feature extraction layer, a multi-head self-attention mechanism layer and a dimension reduction layer. After the speech frame embedded in the local long short-term memory network (Long Short-Term Memory, LSTM) module is encoded by the encoder position, the information contained in the single-frame speech is increased by the convolution recurrent network (Convolution Recurrent Network, CRN) decoder, and the multi-head attention mechanism in the Transformer is used to assign different weights to the noisy speech to improve the model's attention to noise and speech. The transposed convolution module in the decoder is used as a dimension reduction layer, and the speech blocks of different dimensions after the self-attention mechanism are weighted are sampled and restored to ensure the original dimensional relationship of the speech, thereby improving the auditory quality of the denoised speech, thereby completing speech denoising and obtaining a predicted speech frame.

[0062] Furthermore, for the position embedding layer, the present application changes the original Transformer position embedding to a local LSTM (local-LSTM) module that is more suitable for speech noise reduction for position embedding. This not only achieves the effect of efficient data processing, but also takes into account the characteristic that the noise information of non-stationary noise is only related to the speech information of the previous and next frames, and can better capture the local information between speech frames.

[0063] The feature extraction layer includes convolutional layer, normalization layer and linear rectification function (Rectified Linear Unit, ReLU) activation function layer.

[0064] For the multi-head self-attention mechanism layer, the introduction of the attention mechanism enables the model to perform nonlinear weight distribution on each frame of noisy speech according to different noise conditions. The model can thus pay more attention to each frequency point in the noisy speech and give different attention levels to noise and speech. In this way, the model can more effectively distinguish between noise and speech components in noisy speech, laying the foundation for further comprehensive and accurate removal of the noise part.

[0065] The dimension reduction layer includes the transposed convolution layer, the normalization layer, and the Relu activation function layer. In order to avoid the potential gradient vanishing problem caused by the encoder-decoder module, this model adds residual links in each corresponding convolution module and transposition module. This can not only effectively prevent the occurrence of gradient vanishing, but also help prevent the loss of speech information in the deep network transmission.

[0066] Furthermore, the convolutional recurrent neural network speech denoising method based on the self-attention mechanism of the present application has at least the following beneficial effects:

[0067] In order to verify the effectiveness of the convolutional neural network based on the self-attention mechanism, three groups of control models are set up, namely: LSTM network, consisting of three layers of LSTM network and one fully connected layer, with parameter settings of 512, 512, 512 and 161 respectively; convolutional recurrent network (CRN), an encoding and decoding structure composed of two-dimensional convolution and transposed convolution; convolutional recurrent network (A-CRN) based on channel attention mechanism, which adds channel attention mechanism to the middle layer on the basis of convolutional recurrent network.

[0068] 1. Speech quality and speech intelligibility: The perceptual evaluation of speech quality (PESQ) and short time objective intelligibility (STOI) values ​​are used as indicators. By inputting various types of noise into the four models, it can be found that compared with the comparison model, the model based on the self-attention mechanism has achieved a significant improvement in various types of noise, which is better than the other three models. At the same time, the A-CRN model is better than the CRN model, indicating that the introduction of the attention mechanism can improve the speech quality of the model. A-CRN and CRN are better than LSTM in some non-stationary noises (such as f16, m109 and other types of noise), indicating that the convolutional network is better than the LSTM network in extracting speech frequency features. Under different signal-to-noise ratios, Tra-CRN has achieved higher scores, and when the signal-to-noise ratio is 7, the PESQ score is improved by about 59.2% compared with the noise. At the same time, in the case of low signal-to-noise ratios that are more difficult to handle, Tra-CRN still achieves a higher score. In the case of 0 SNR, the LSTM model scores higher than the CRN, which indicates that in the case of low SNR, the extraction of temporal features of noisy speech has a greater impact on the restoration of speech quality.

[0069] 2. Spectrogram analysis: Input various types of noise into four models respectively. Analyzing the denoised spectrograms, we can find that: the three comparison models have different degrees of residual noise after denoising, among which the residual non-stationary noise of the LSTM model is the most obvious; the two models based on convolutional recurrent networks can effectively eliminate non-stationary noise, but there is a large degree of residual narrowband noise. The spectrogram after denoising by the Tra-CRN network clearly shows that it has basically completely eliminated non-stationary noise and narrowband noise.

[0070] 3. VoiceBank-DEMAND dataset scoring: VoiceBank-DEMAND dataset is one of the main datasets for comparing the effects of speech denoising networks. From the comparison results, we can see that the Tra-CRN network has a greater advantage in scoring on different indicators than the current mainstream speech denoising networks. This shows that the Tra-CRN network not only has a good denoising performance for noisy speech under different signal-to-noise ratios, but also has a good generalization for untrained noise.

[0071] This application aims to make full use of the advantages of deep learning methods with the help of the CRN model and the multi-head self-attention mechanism in the Transformer to improve the noise and speech information contained in a single noisy speech frame, so that the model can more effectively distinguish between noise and speech parts, and improve the model's noise reduction effect and noise generalization ability.

[0072] The specific execution steps of the method of the present application are as follows: The speech noise reduction of the present application is carried out in two stages, including a training stage and an enhancement stage. Figure 1 shown.

[0073] In the training phase, noise and clean speech are first synthesized into noisy speech, and then the noisy speech is feature extracted. Then the clean speech and the noisy speech after feature extraction are sent to the neural network model to learn the relationship between noisy speech and clean speech. After the training phase is completed, the trained model is saved.

[0074] In the enhancement stage, the noisy speech is firstly extracted for features, then sent to the trained model for speech enhancement, and finally the enhanced speech is output.

[0075] The neural network model proposed in this application can be roughly composed of three parts, namely, an encoding module, a decoding module and a denoising module.

[0076] Different convolution kernels in the encoder can map noisy speech to different dimensional spaces, such as high-frequency information, low-frequency information, and noise capture, thereby increasing the information contained in a single speech frame and solving the problem that a single noisy speech frame cannot effectively complete attention weighting. The feature extraction layer consists of multiple convolution modules, each of which contains a convolution layer, a batch normalization layer, and a relu activation function. After the noisy speech passes through the N superimposed convolution modules of the feature extraction layer, feature information in multiple different dimensions can be obtained. The results of different convolution kernels are spliced ​​through the dimensional splicing function, so that each frame of the spliced ​​speech incorporates information in different dimensions.

[0077] The decoder in the convolutional recurrent network consists of a transposed convolution module and an LSTM module. The transposed convolution module consists of a transposed convolution layer, a batch normalization layer, and an activation function. To build the causal relationship of the model, the transposed convolution setting is the same as the convolution module. The LSTM module consists of an LSTM network and a fully connected network, which is used to restore the temporal relationship of speech and improve the auditory quality of the denoised speech.

[0078] Transposed convolution is the reverse process of convolution, which can reconstruct the dimensional information encoded by the encoder to complete the speech noise reduction process. Each feature map is mapped in sequence according to the step size through the convolution kernel of the transposed convolution.

[0079] Furthermore, the convolutional recurrent neural network based on the self-attention mechanism proposed in this application includes five parts: position embedding layer, feature extraction layer, multi-head self-attention mechanism layer and dimension reduction layer. The network structure diagram is shown in Figure 2 shown.

[0080] Step 1: Use the local LSTM module, which is more suitable for speech noise reduction, to embed the position of the noisy speech frame. In terms of the number of frames, the average duration of human pronunciation for each phoneme is about 30 milliseconds, and the pronunciation of a syllable is between 120 and 150 milliseconds. Therefore, every 7 frames are selected as a speech frame segment, and the frame shift is set to 1 frame. Then, the position of each speech frame segment is embedded through the local LSTM with shared weights, as shown in the following figure. Figure 3 shown.

[0081] Step 2: The position-encoded speech frame is passed through the CRN network decoder to increase the information contained in a single frame of speech. In the convolution module, each convolution module contains a convolution layer, a batch normalization layer, and a relu activation function. The noisy speech features are superimposed, so that the spliced ​​speech frame incorporates rich multi-dimensional information.

[0082] Step 3: The structure of the self-attention mechanism is as follows Figure 4 As shown in the figure, the input vector is scaled dot product through h K, Q and V matrices to obtain the similarity between the two vectors. Among them, h is the number of K, Q and V. The input vector is mapped to different spaces through multiple weight matrices. Then the h output results are spliced ​​using the concat function and converted back to the input shape through linear mapping.

[0083] The data flow of the self-attention mechanism is as follows Figure 5 shown.

[0084] Step 3.1: First, the Q, K and V matrices are obtained by formula 1:

[0085]

[0086] Among them, x is the encoder output result, WQ , W K , W V are weight matrices respectively.

[0087] Step 3.2: After using the reshape function to transform the dimension, we can get from formula 2 that Q and K T Do the scaling point product to obtain the similarity between each frequency point, and use the softmax function for normalization to form the similarity probability W between different noisy speech frames. i .

[0088]

[0089] Among them, d k The size of the input vector, divided by This prevents gradients from disappearing during calculations.

[0090] Step 3.3: Multiply the obtained similarity probability W by V to obtain the weighted vector matrix of the attention mechanism, as shown in Formula 3.

[0091] Attention(Q,K,V)=W i ·V Formula 3

[0092] Step 3.4: After obtaining the h attention weighted matrices, use the concat function to concatenate and convert the dimensions, and transform them to the same size as the input through linear mapping, as shown in Formula 4.

[0093] MultiHead(Q,K,V)=Linear(concat(W i ,...,W h )) Formula 4

[0094] Step 4: Use the transposed convolution module as a dimensionality restoration layer, which consists of a transposed convolution layer, a batch normalization layer, and an activation function. Upsample and restore the speech blocks of different dimensions after weighting by the self-attention mechanism to ensure the original dimensional relationship of the speech.

[0095] Step 5: Adding residual links in each corresponding convolution module and transposition module can not only avoid gradient vanishing, but also prevent information loss caused by speech information flowing in the deep network.

[0096] Furthermore, the calculation steps of the convolutional recurrent network based on the self-attention mechanism are as follows:

[0097]

[0098] Where n is the number of convolution modules and transposed convolution modules. noiseis the input noisy speech frame, and x is the clean speech frame after denoising.

[0099] At this point, the processing of the neural network model is completed, and the noise-reduced speech is output.

[0100] The experimental voice uses the acoustic-phoneme continuous speech corpus (TIMIT library) constructed by the Massachusetts Institute of Technology (MIT), which contains a total of 6,300 sentences; it consists of 10 given sentences spoken by 630 people from multiple major dialect areas in the United States; the noise data used are 15 types of noise from the Ohio State University laboratory (respectively: white noise, car noise, military vehicle noise, tank internal noise, restaurant noise, high-frequency channel noise, pink noise, machine gun noise, factory workshop noise 1, factory workshop noise 2, F16 cockpit noise, destroyer cabin noise, destroyer operations room background noise, Buccaneer cockpit noise 1, Buccaneer cockpit noise 2). The noise sampling frequency in the speech and noise library of the TIMIT dataset was converted to 16 kHz, and 4500 clean speech (4-7s) were selected from them. Segments of the same length as the clean speech were randomly generated from 15 types of noise (about 3 minutes each) and superimposed at different signal-to-noise ratios (-10, -5, 0, 5, 10). A training set of 4500 noisy speech (a total of 22500) was generated at each signal-to-noise ratio. Randomly selecting the noise segments required for superposition from each type of noise can enhance the generalization ability of the model and improve the model's denoising ability for unknown noise compared to using the same noise segments. At the same time, 218 clean speech different from the training set were selected and 654 noisy speech were generated at three signal-to-noise ratios (-7, 0, 7) using the same method to form the test set of the model.

[0101] All models use the Adam gradient optimizer, the loss function is MSE, the initial learning rate is 0.01, and it is multiplied by 0.1 every ten rounds. In the training input, the amplitude spectrum containing the most speech feature information is selected as the training feature. According to the human ear's insensitivity to the phase spectrum, the speech is converted into a time domain waveform by inverse Fourier transform of the predicted clean amplitude spectrum and the noisy speech phase spectrum.

[0102] In order to verify the effectiveness of the method proposed in this application, the proposed method is compared with other speech denoising methods. The network models used in the comparison method include LSTM network, which consists of three layers of LSTM network and one fully connected layer, with parameters of 512, 512, 512 and 161 respectively; convolutional recurrent network (CRN), a codec structure consisting of two-dimensional convolution and transposed convolution; convolutional recurrent network based on channel attention mechanism (A-CRN), which adds channel attention mechanism in the middle layer based on convolutional recurrent network.

[0103] The three indicators are compared and analyzed separately:

[0104] 1. The results of perceptual evaluation of speech quality (PESQ) and short time objective intelligibility (STOI) are shown in the following table:

[0105] Table 1 PESQ scores of each model at different signal-to-noise ratios;

[0106]

[0107] Table 2 STOI scores of each model at different signal-to-noise ratios

[0108]

[0109] 2. The comparison of VoiceBank-DEMAND dataset scores is shown in Table 3:

[0110] Table 3:

[0111]

[0112] It can be seen from the above three indicators that the method proposed in this application is higher than the comparison algorithm in terms of indicators, indicating that the method proposed in this application has certain advantages.

[0113] This application proposes a convolutional recurrent network speech denoising model based on a multi-head self-attention mechanism. In this model, we use local LSTM to positionally encode the input speech frame, and expand the dimensional information of a single speech frame through a convolution module. Using the self-attention mechanism in the Transformer, weights are assigned to each frequency point of the noisy speech frame to effectively separate speech and noise. Finally, the separated speech is restored to a clear speech dimension through a transposed convolution module, thereby achieving efficient speech denoising processing.

[0114] By comparing different models with noisy speech at various signal-to-noise ratios, it is found that the convolutional recurrent network based on the self-attention mechanism has better noise reduction effects than other models under various types of noise, and the PESQ and STOI scores have been greatly improved. By comparing the spectrograms, it can be seen that the improved model can better remove noise and improve the auditory quality of speech.

[0115] In order to realize the visualization system of speech intelligent analysis, MATLAB can be used for design. This system uses AppDesigner design tool to interactively develop the framework for the development of the entire system. The overall software architecture of the entire system is as follows: Figure 6 shown.

[0116] The system includes functions such as file management, preprocessing, audio verification, voice identity recognition and voice analysis report. According to the functional division, file management is divided into: file opening, file saving, file attributes and exit software modules; voice preprocessing is divided into: voice noise reduction, voice separation, endpoint detection and noise analysis modules; audio verification is divided into: homology detection, voice extraction, base frequency extraction and formant extraction modules; voice identity recognition is divided into: voiceprint recognition and voiceprint database modules; and voice analysis report module. Figure 7 shown.

[0117] The above is only the preferred implementation mode of the present application, which certainly cannot be used to limit the scope of rights of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and changes can be made without departing from the principles of the present application, and these improvements and changes are also regarded as the scope of protection of the present application.

Claims

1. A convolutional recurrent neural network speech denoising method based on self-attention mechanism, characterized in that: It includes a training phase and a reinforcement phase; The training phase: firstly, the noise and the clean speech are superimposed to synthesize the noisy speech, then the feature of the noisy speech is extracted, and then the clean speech and the noisy speech after the feature extraction are sent to the convolutional recurrent neural network model to learn the mapping relationship between the noisy speech and the clean speech, and after the training phase is completed, the trained model is saved; The enhancement stage: firstly, the noisy speech is feature extracted, then sent to the trained model for speech noise reduction, and finally the noise-reduced speech is output.

2. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 1, characterized in that: It also includes a preferred stage: the convolutional recurrent neural network model uses an encoder-decoder structure as a deep learning architecture, the encoder obtains compressed data by removing the noise part in the noisy speech, and then inputs the data into the decoder to reconstruct the data to restore the estimated form of the pure speech.

3. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 1, characterized in that: The convolutional recurrent neural network model consists of five parts: position embedding layer, feature extraction layer, multi-head self-attention mechanism layer and dimension reduction layer; The position embedding layer embeds the speech frame after the local long short-term memory network module and then passes through the encoder position encoding; the feature extraction layer increases the information contained in a single frame of speech through the decoder, and uses the multi-head attention mechanism to assign different weights to the noisy speech, thereby improving the model's attention to the difference between noise and speech; the transposed convolution module in the decoder is used as a dimensionality restoration layer to sample and restore the speech blocks of different dimensions after the self-attention mechanism weights, ensuring the original dimensional relationship of the speech, thereby improving the auditory quality of the denoised speech, thereby completing speech denoising and obtaining the predicted speech frame.

4. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 3, characterized in that: The position embedding layer changes the original Transformer position embedding to a local LSTM module that is more suitable for speech denoising, which not only achieves the effect of efficient data processing, but also takes into account the characteristic that the noise information of non-stationary noise is only related to the speech information of the previous and next frames, and can better capture the local information between speech frames; The feature extraction layer includes a convolution layer, a normalization layer and a linear rectification function ReLU activation function layer; The multi-head self-attention mechanism layer introduces an attention mechanism, which enables the model to perform nonlinear weight distribution on each frame of noisy speech according to different noise conditions; the model can thus pay more attention to each frequency point in the noisy speech and give different attention levels to noise and speech; in this way, the model can more effectively distinguish between noise and speech components in noisy speech, laying the foundation for further comprehensive and accurate removal of the noise part; The dimension reduction layer includes a transposed convolution layer, a normalization layer and a Relu activation function layer; in order to avoid the potential gradient vanishing problem caused by the codec module, this model adds a residual link in each corresponding convolution module and transposition module; it can not only effectively prevent the occurrence of gradient vanishing, but also help prevent the loss of speech information in the deep network transmission.

5. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 1, characterized in that: The calculation steps include: Step 1: Use a local LSTM module that is more suitable for speech noise reduction to positionally embed the noisy speech frames. In terms of the number of frames, the average pronunciation duration of each phoneme is about 30 milliseconds, and the pronunciation of a syllable is between 120 and 150 milliseconds. Therefore, every 7 frames are selected as a speech frame segment, and the frame shift is set to 1 frame. Then, the position of each speech frame segment is embedded through the local LSTM with shared weights. Step 2: The position-encoded speech frame is passed through the CRN network decoder to increase the information contained in a single frame of speech; in the convolution module, each convolution module contains a convolution layer, a batch normalization layer, and a relu activation function; the noisy speech features are superimposed so that the spliced ​​speech frame incorporates rich multi-dimensional information; Step 3: Self-attention mechanism structure, the input vector is scaled dot product through h K, Q and V matrices to obtain the similarity between two vectors; where h is the number of K, Q and V, the input vector is mapped to different spaces through multiple weight matrices; then the h output results are concatenated using the concat function and converted back to the input shape through linear mapping; Step 4: Use the transposed convolution module as a dimensionality reduction layer, which consists of a transposed convolution layer, a batch normalization layer, and an activation function. Upsample and restore the speech blocks of different dimensions after the self-attention mechanism weighting to ensure the original dimensional relationship of the speech. Step 5: Adding residual links to each corresponding convolution module and transposition module can not only avoid gradient vanishing, but also prevent information loss caused by speech information flowing in the deep network; the processing process of the neural network model is now completed, and the denoised speech is output.

6. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 5, characterized in that: The step 3 specifically includes the following steps: Step 3.1: First, the Q, K and V matrices are obtained by formula 1: Among them, x is the encoder output result, W Q , W K , W V are weight matrices respectively; Step 3.2: After using the reshape function to transform the dimension, we can get from formula 2 that Q and K T Do the scaling point product to obtain the similarity between each frequency point, and use the softmax function for normalization to form the similarity probability W between different noisy speech frames. i ; Among them, d k The size of the input vector, divided by It can prevent the gradient from disappearing during calculation; Step 3.3: Multiply the obtained similarity probability W by V to obtain the weighted vector matrix of the attention mechanism, as shown in Formula 3; Attention(Q,K,V)=W i ·V Formula 3 Step 3.4: After obtaining the h attention weighted matrices, use the concat function to concatenate and transform the dimensions, and transform them to the same size as the input through linear mapping, as shown in Formula 4; MultiHead(Q,K,V)=Linear(concat(Wi,...,Wh)) Formula 4.

7. The method for speech denoising based on a convolutional recurrent neural network with a self-attention mechanism according to claim 1, characterized in that: Also includes calculation step 6: Where n is the number of convolution modules and transposed convolution modules; x noise is the input noisy speech frame, x is the clean speech frame after noise reduction, and the output is the noise-reduced speech.

Images