Complex de-reverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM

Abstract

The invention discloses a multi-head attention mechanism and Bi-LSTM (Bidirectional Long Short Term Memory)-based complex de-reverberation speech enhancement method, and belongs to the technical field of speech enhancement. According to the method, de-reverberation processing is carried out in a complex field, firstly, STFT is carried out on a signal to obtain a complex spectrum, so that amplitude and phase information is fully utilized; the complex spectral features are then sent to an advanced channel attention mechanism (HCHA) to fuse the features. Secondly, in order to capture correlation among different positions in an input sequence and capture voice key components by using time and frequency information, a time-frequency dual-path multiple attention mechanism TF-CMHA is designed, voice components are recognized more accurately, and reverberation components are suppressed. And finally, introducing a complex multiplication operation and a joint constraint loss function, simulating the correlation between the amplitude and the phase, and obtaining a more accurate description of the long-time voice correlation. According to the invention, the performance of the de-reverberation speech enhancement system is improved, various measurement indexes of the de-reverberation speech of the system are improved, and the method is suitable for speech front-end processing.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and in particular relates to a complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM. Background Art

[0002] Speech enhancement technology has been applied to many scenarios in life. In actual use scenarios, in addition to noise, reverberation is also one of the important factors affecting speech quality. In particular, the sound signal received by the microphone in a reverberant environment contains not only the original speech signal, but also a fusion of multiple delayed and attenuated reflected sounds, which may mask the key information in the speech, thereby affecting the quality and intelligibility of the speech signal. Therefore, in the presence of reverberation noise, dereverberation speech enhancement technology has important practical significance.

[0003] Deep learning techniques are becoming increasingly prominent and widely implemented in public applications, including but not limited to speech processing, image analysis, and natural language processing. Due to their powerful fitting capabilities and effective use of big data, deep learning-based methods can more accurately model the complex relationship between target and interfering signals. In addition, deep learning methods have stronger generalization capabilities for various signal types and are not limited by assumptions such as signal independence and Gaussian noise, thus having superior performance in speech enhancement.

[0004] Due to the complexity and diversity of reverberation signals, it is difficult for the model to accurately suppress the reverberation components. In addition, most existing deep learning models are based on amplitude spectrum processing, ignoring the complex characteristics of speech signals, which may result in the loss of some useful information when dealing with reverberation problems.

[0005] In recent years, a lot of research has been done on speech enhancement methods in the dereverberation direction. De-dereverberation methods based on deep learning mainly use two principles: spectrum mapping and time-frequency masking. The model based on multi-hidden layer neural network aims to restore the amplitude spectrum of the source speech by mapping the amplitude spectrum of the reverberated speech, thereby enhancing the dereverberation performance of the model. However, due to the limitations of the shallow network architecture and the design of the loss function, this mapping method still faces challenges in achieving accurate recovery of the amplitude spectrum of the source speech. Wang, Williamson and others used TF masking technology, using DNN training to obtain the complex ideal ratio mask of the amplitude spectrum of the reverberated speech, and then extracted the source speech by multiplying this mask with the reverberated speech spectrum, which can improve the clarity of the speech after dereverberation. However, this method still faces the challenge of not being able to effectively suppress the highly correlated part of the reverberation. Liu Bin and others used a recursive neural network combined with a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, Bi-LSTM) to map the logarithmic power spectrum features of the reverberated speech, effectively capturing the correlation features of the context. Zhao, Wang and others introduced the attention mechanism into the dereverberation model. By finely modeling the correlation between frequency band information, the model's performance in mapping the source speech amplitude spectrum can be significantly improved. Without relying on the prior information of the target speech and acoustic environment, Jie Yuan and others proposed a speech enhancement method based on a time-frequency Wiener masked inverse filter with blind signal extraction using the primal-dual splitting algorithm, which has certain advantages in noisy and reverberant scenes. The present invention fully studies and improves the performance of the speech enhancement system with attention mechanism in reverberant noise environment, and designs a complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM. Summary of the invention

[0006] In view of the problem that the above-mentioned prior art is insufficient in dereverberation, the present invention provides a complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM, introduces a time-frequency dual-path complex multi-head attention mechanism TF-CMHA, captures the correlation between different positions within the input sequence and uses time and frequency information to capture the key component information of speech; introduces a complex bidirectional long short-term memory network Bi-LSTM, uses complex multiplication operations to obtain a more accurate description of long-term speech correlation, and achieves an improvement in the performance of the dereverberation speech enhancement system.

[0007] A complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM comprises the following steps:

[0008] Step S1: Input the speech signal containing reverberation and perform STFT transformation to convert the speech signal into the frequency domain, extract the amplitude spectrum features and the complex spectrum features, and obtain the frequency domain representation F;

[0009] Step S2: Input the feature F into the high-level channel attention mechanism HCA, F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and the average pooling feature and the maximum pooling feature are fused to obtain the new feature F a+m , F avg and F max The features are processed by convolution blocks and then combined with F a+m Fusion, get the feature representation new feature F sig , F sig Multiply and fuse element by element with F to get the output feature F HCA ;

[0010] Step S3: The output feature F of HCA HCA The input is fed into the time-frequency dual-path complex multi-head attention mechanism TF-CMHA, which uses the time and frequency information to capture the key components of the speech signal based on the correlation between different positions within the input sequence, and identifies and suppresses the reverberation components.

[0011] Step S4: The output features of TF-CMHA are divided into two channels, real and imaginary, and enter the complex bidirectional long short-term memory network CBi-LSTM to capture the context of the deep features of the speech signal. The correlation between amplitude and phase is simulated by the introduced complex multiplication operation to obtain a more accurate description of the long-term speech correlation;

[0012] Step S5: a joint constraint loss function is designed to train the network model, in which the real and imaginary parts of the complex spectrum feature are regarded as independent features and trained synchronously to reduce the error between the estimated value and the true value, and finally restored to the estimated time domain signal through ISTFT transformation;

[0013] Step S6: Perform a performance evaluation on the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM.

[0014] As a further improvement of the present invention, the specific steps of step S2 are:

[0015] Step S21: F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and add each dimension of the average pooling feature and the maximum pooling feature element by element as a new feature F a+m ;

[0016] Step S22: The average pooling feature and the maximum pooling feature are processed by convolution blocks respectively and combined with the new feature F a+mOnce again, the elements are added and fused, and activated by the sigmoid function to obtain the new feature F sig .

[0017] Step S23: Output feature F sig Multiply element-wise with the input feature F to get F HCA , emphasizing the important parts of the input features and suppressing unimportant information, and also fusing the information between different features to obtain a richer feature representation.

[0018] As a further improvement of the present invention, the specific steps of step S3 are:

[0019] Step S31: Input feature F with dimension T×F×C HCA Reshape into FC vector of dimension 1×T and TC vector of dimension 1×F respectively, one along the time axis and the other along the frequency axis.

[0020] Step S32: Apply the multi-head attention mechanism MHA in parallel along each axis to capture the intrinsic connections between different positions of the input data.

[0021] Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher-level representation, and then passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model.

[0022] Step S34: reshape and splice the output features of the real part and the imaginary part as the final output features of the single time axis and the single frequency axis. Then combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result X of TF-CMHA A .

[0023] As a further improvement of the present invention, in step S32, the specific steps are:

[0024] Step S321: Input the multi-head attention mechanism MHA with 8 attention heads.

[0025] Step S322: The MHA attention calculation is performed in parallel on the real part and the imaginary part of the feature respectively.

[0026] As a further improvement of the present invention, the specific steps of step S4 are:

[0027] Step S41: Divide into two channels of real part and imaginary part and enter into CBi-LSTM, and apply complex multiplication operation.

[0028] Step S42: Each channel consists of two stacked Bi-LSTM layers, and the output dimension of each time step is 1446, resulting in a feature matrix of 426×1446.

[0029] Step S43: After passing through the linear layer (1446, 723), the final 426×723 feature matrix is ​​obtained.

[0030] As a further improvement of the present invention, the step S42 is as follows:

[0031] Step S421: define the complex-valued convolution filter W as W=W r +jW i , where the real-valued matrix W r and W i represent the real and imaginary parts of the complex-valued convolution kernel respectively, and the input complex spectrum can be defined as X = X r +jX i Therefore, the complex spectrum output Y can be obtained through complex convolution operation, which can be expressed as

[0032] Y=(X r *W r -X i *W i )+j(X r *W i +X i *W r ).

[0033] Step S422: The output feature after TF-CMHA is X A , its real and imaginary parts are represented by X Ar , X Ai , as two input channels, are input into Bi-LSTM respectively. The specific operation process is as follows. The final output of CBi-LSTM is F out .

[0034] F rr =BiLSTM r (X Ar )

[0035] F ri =BiLSTM r (X Ai )

[0036] F ir =BiLSTM i (X Ar )

[0037] F ii =BiLSTM i (X Ai )

[0038] F out =(F rr -F ii )+j(F ri +F ir )

[0039] Among them, BiLSTM r and BiLSTM i Two traditional Bi-LSTMs, F, represent the real part and the imaginary part respectively. rr Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ri Indicates that the real feature X will be input Ai The output feature after being fed into the real Bi-LSTM, F ir Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ii Indicates that the real feature X will be input Ai Output features after being fed into the real Bi-LSTM.

[0040] As a further improvement of the present invention, the specific steps of step S5 are:

[0041] Step S51: define the time domain MSE loss function:

[0042]

[0043] x represents the clean speech signal, Represents the estimated speech signal.

[0044] Step S52: a joint loss function is designed, including the amplitude spectrum of the speech signal, the real part of the complex spectrum and the imaginary part. After calculating their MSE losses respectively, they are combined as the final loss function of the model:

[0045]

[0046] Among them, Y mag represents the amplitude spectrum of the clean speech signal, Represents the amplitude spectrum of the estimated signal, Y r and Y i Represent the real and imaginary parts of the clean speech signal respectively, and They represent the real and imaginary parts of the estimated signal respectively, and α, β, γ∈[0,1] are weight coefficients.

[0047] Step S53: Finally, output the predicted time domain speech signal after ISTFT transformation.

[0048] As a further improvement of the present invention, the specific method for implementing step S6 is:

[0049] Step S61: Discuss the specific value of the weight of the weight loss in the joint constraint loss and explore the optimal weight value.

[0050] Step S62: Conduct an ablation experiment on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part.

[0051] Step S63: A comparative experiment is conducted between the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM and the LSTM model and the CRN model to verify the effectiveness of the entire model.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. When solving the problem of reverberant speech enhancement, most of the existing deep learning models are based on amplitude spectrum processing, ignoring the complex characteristics of speech signals, which may lead to the loss of some useful information when dealing with reverberation problems. The present invention takes advantage of the complex domain processing and represents the speech signal in complex form to make full use of the amplitude and phase information of the speech signal.

[0054] 2. A time-frequency dual-path complex multi-head attention mechanism TF-CMHA is designed, which can capture the correlation between different positions within the input sequence and consider both time and frequency information at the same time, so that the model can focus on multiple key parts of the speech signal, more accurately identify and suppress reverberation components, and further improve the computational efficiency and representation ability of the model.

[0055] 3. In order to capture the context of deep features of speech signals, the present invention introduces complex multiplication operations to better simulate the correlation between amplitude and phase, and obtain a more accurate description of long-term speech correlation. At the same time, a joint constraint loss function is designed to train the network model to further improve the performance of the dereverberation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the operation flow of the present invention.

[0057] Figure 2 This is a specific structural diagram of the high-level channel attention mechanism HDC in the present invention.

[0058] Figure 3 This is a flow chart of the implementation of the TF-CMHA module of the present invention.

[0059] Figure 4 This is a model architecture diagram of the CBi-LSTM module of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0061] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0062] like Figures 1 to 4 As shown, the present invention provides a complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM, applies a deep neural network to speech enhancement in a reverberation environment, is used to extract deep features, has a certain inhibitory effect on reverberation noise, and can build a higher performance dereverberation speech enhancement system, including the following steps:

[0063] Step S1: The database used in the present invention is the VoiceBank corpus. 3597 speech sounds of the speaker are randomly selected for mixing and used as a training set, and 285 speech sounds are selected from the remaining speech sounds for mixing as a test set. The mirror method is used to simulate the impulse response in the room, and a virtual room is customized with a size of 7.72m×8.10m×7.60m (length, width and height). The speaker is fixed at the position [3.0, 4.0, 1.5], the microphone is fixed at the position [5.0, 7.0, 2.5], and the reverberation time is set to 0.5s. Then, it is convolved with the selected clean speech signal to form a training set and a test set. The sampling rate of the data is 16kHz. The input speech signal containing reverberation is subjected to an STFT transform with a point number of 512 points and a frame shift of 128. The window function used in the STFT transform is a Hanning window with a window size of 2048. The speech signal is converted to the frequency domain, and the amplitude spectrum features and complex spectrum features are extracted to obtain the representation F in the frequency domain.

[0064] Step S2: The output feature F is input into the high-level channel attention mechanism HCA. The HCA structure is shown in the figure Figure 2 As shown, F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and the average pooling feature and the maximum pooling feature are fused to obtain the new feature F a+m , F avg and F maxThe features are processed by convolution blocks and then combined with F a+m Fusion, get the feature representation new feature F sig , F sig Multiply and fuse element by element with F to get the output feature F HCA .

[0065] Step S2 specifically includes:

[0066] Step S21: F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and add each dimension of the average pooling feature and the maximum pooling feature element by element as a new feature F a+m ;

[0067] Step S22: The average pooling feature and the maximum pooling feature are processed by convolution blocks respectively and combined with the new feature F a+m Once again, the elements are added and fused, and activated by the sigmoid function to obtain the new feature F sig .

[0068] Step S23: Output feature F sig Multiply element-wise with the input feature F to get F HCA , emphasizing the important parts of the input features and suppressing unimportant information, and also fusing the information between different features to obtain a richer feature representation.

[0069] Step S22 specifically includes:

[0070] Step S221: The first two-dimensional convolutional layer reduces the number of channels of the input feature map to half, thereby reducing the complexity and number of parameters of the model.

[0071] Step S222: ReLU activation function transformation to enhance the expressive power of the model.

[0072] Step S223: The last two-dimensional convolutional layer restores the number of channels to the number of channels when HCA is input, so as to retain and integrate more feature information.

[0073] Step S224: The Sigmoid function can map the input value to a range between 0 and 1. The closer to 1, the more important the channel is, and the closer to 0, the less important the channel is.

[0074] Step S3: The output feature F of HCA HCA Input into TF-CMHA, the flow chart is as follows Figure 3As shown in the figure, TF-CMHA can capture the correlation between different positions within the input sequence and consider both time and frequency information, so that the model can focus on multiple key parts of the speech signal and more accurately identify and suppress reverberation components.

[0075] Step S3 specifically includes:

[0076] Step S31: Input feature F with dimension T×F×C HCA Reshaped into FC vectors of dimension 1×T and TC vectors of dimension 1×F, respectively, one along the time axis and the other along the frequency axis.

[0077] Step S32: Apply the multi-head attention mechanism MHA in parallel along each axis to capture the intrinsic connections between different positions of the input data.

[0078] Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher-level representation, and then passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model.

[0079] Step S34: reshape and splice the output features of the real part and the imaginary part as the final output features of the single time axis and the single frequency axis. Then combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result X of TF-CMHA A .

[0080] Step S32 specifically includes:

[0081] Step S321: The number of attention heads of the multi-head attention mechanism MHA is 8.

[0082] Step S322: The MHA attention calculation is performed in parallel on the real part and the imaginary part of the feature respectively.

[0083] Step S33 specifically includes:

[0084] Step S331: The dimension of the linear layer is (512, 723).

[0085] Step S332: The dropout probability of the dropout layer is 0.1.

[0086] Step S4: The feature X output by TF-CMHA A The signal is divided into two channels, the real part and the imaginary part, and enters CBi-LSTM to capture the causal relationship of the deep features of the speech signal. The complex multiplication operation introduced can better simulate the correlation between amplitude and phase, and obtain a more accurate description of the long-term speech correlation.

[0087] Step S4 specifically includes:

[0088] Step S41: Divide into two channels, the real part and the imaginary part, and enter the CBi-LSTM. Use complex multiplication operation. The model architecture is as follows Figure 4 shown.

[0089] Step S42: Each channel consists of two stacked Bi-LSTM layers, and the output dimension of each time step is 1446, resulting in a feature matrix of 426×1446.

[0090] Step S43: After passing through the linear layer (1446, 723), the final 426×723 feature matrix is ​​obtained.

[0091] Step S42 specifically includes:

[0092] Step S421: define the complex-valued convolution filter W as W=W r +jW i , where the real-valued matrix W r and W i represent the real and imaginary parts of the complex-valued convolution kernel respectively, and the input complex spectrum can be defined as X = X r +jX i Therefore, the complex spectrum output Y can be obtained through complex convolution operation, which can be expressed as

[0093] Y=(X r *W r -X i *W i )+j(X r *W i +X i *W r ).

[0094] Step S422: The output feature after TF-CMHA is X A , its real and imaginary parts are represented by X Ar , X Ai , as two input channels, are input into Bi-LSTM respectively. The specific operation process is as follows. The final output of CBi-LSTM is F out .

[0095] F rr =BiLSTM r (X Ar )

[0096] F ri =BiLSTM r (X Ai )

[0097] F ir =BiLSTM i (XAr )

[0098] F ii =BiLSTM i (X Ai )

[0099] F out =(F rr -F ii )+j(F ri +F ir )

[0100] Among them, BiLSTM r and BiLSTM i Two traditional Bi-LSTMs, F, represent the real part and the imaginary part respectively. rr Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ri Indicates that the real feature X will be input Ai The output feature after being fed into the real Bi-LSTM, F ir Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ii Indicates that the real feature X will be input Ai Output features after being fed into the real Bi-LSTM.

[0101] Step S5: Using the real and imaginary parts of the complex spectrum and the amplitude characteristics, a joint constraint loss function is designed to train the network model to reduce the error between the estimated value and the true value.

[0102] Step S5 specifically includes:

[0103] Step S51: define the time domain MSE loss function:

[0104]

[0105] x represents the clean speech signal, Represents the estimated speech signal.

[0106] Step S52: Because the invention is studied in the complex domain, in order to fully consider the independence and importance difference between the real part and the imaginary part, as well as the relationship between amplitude and phase, a joint loss function is designed, including the amplitude spectrum of the speech signal, the real part and the imaginary part of the complex spectrum, a total of three parts, after calculating their MSE losses respectively, they are combined as the final loss function of the model:

[0107]

[0108] Among them, Y magrepresents the amplitude spectrum of the clean speech signal, Represents the amplitude spectrum of the estimated signal, Y r and Y i Represent the real and imaginary parts of the clean speech signal respectively, and They represent the real and imaginary parts of the estimated signal respectively, and α, β, γ∈[0,1] are weight coefficients.

[0109] Step S53: Finally, output the predicted time domain speech signal after ISTFT transformation.

[0110] Step S6: Perform a performance evaluation on the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM.

[0111] Step S6 specifically includes:

[0112] Step S61: Discuss the specific value of the weight of the weight loss in the joint constraint loss and explore the optimal weight value.

[0113] Step S62: Conduct an ablation experiment on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part.

[0114] Step S63: A comparative experiment is conducted between the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM and the LSTM model and the CRN model to verify the effectiveness of the entire model.

[0115] In this embodiment, the sampling rate of the data used is 16kHz, the speech signal is converted to the frequency domain through a 512-point STFT, and the frame shift is 128. The iteration cycle of the entire training is 30, the batch size is 2, the optimizer is Adam, and the initial learning rate is set to 0.0005. This embodiment uses two speech evaluation indicators, PESQ and STOI, to evaluate the network dereverberation performance.

[0116] An experiment was conducted to verify the effectiveness of the joint constraint loss function. Three sets of comparative experiments with different weight coefficients were set up to evaluate the impact of the real and imaginary parts of the amplitude spectrum features and the complex spectrum features on the performance of the dereverberation system. The experimental results are shown in Table 1:

[0117] Table 1

[0118]

[0119] It can be observed from Table 1 that the overall performance of the model using the joint constraint loss function is greater than that of the model using the traditional MSE loss function, and the network model pays more attention to the imaginary part of the complex spectrum information. When its proportion in the joint loss is high, the dereverberation performance of the model is the best. From the experimental results, it can be seen that compared with the traditional MSE loss function, the joint constraint loss function with a large amplitude spectrum weight improves PESQ by 0.03 and STOI by 0.01. Compared with the amplitude spectrum feature, the high proportion of the complex spectrum feature weight enables the model to pay more attention to the details of the real or imaginary part while maintaining the overall structure of the complex spectrum, which is more conducive to improving the dereverberation performance of the model. When the imaginary part weight is larger, the PESQ and STOI indicators are better, which means that the model will pay more attention to the recovery of phase information during training, thereby improving the quality of speech enhancement. In summary, the proposed joint constraint loss function is effective.

[0120] The effectiveness of each part is explored, where the mark "√" indicates that the module is used in the network. The experimental results are shown in Table 2:

[0121] Table 2

[0122] Bi-LSTM Complex multiplication TF-CMHA HCA PESQ STOI √ 2.37 0.81 √ √ 2.48 0.87 √ √ √ 2.66 0.91 √ √ √ √ 2.70 0.93

[0123] As shown in Table 2, when complex multiplication is introduced into Bi-LSTM, PESQ is improved by 0.11 and STOI is improved by 0.06, indicating that the model pays more attention to the importance of amplitude and phase. When TF-CMHA is added, PESQ is further improved by 0.18 and STOI is further improved by 0.03, indicating that capturing contextual information from the dimensions of time and frequency and capturing the feature correlation between different positions within the input sequence is crucial for the dereverberation speech enhancement task. Finally, after adding HCA, PESQ is improved by 0.04 and STOI is improved by 0.02, indicating that the multi-dimensional attention mechanism can effectively improve the expressiveness of input and output features to improve the performance of the speech enhancement model.

[0124] The effectiveness of the complex dereverberation speech enhancement method based on multi-head attention and Bi-LSTM is evaluated, and it is compared with the LSTM model, CRN model and the model in Chapter 3. The experimental results are shown in Table 3:

[0125] Table 3

[0126] Model PESQ STOI Model PESQ STOI LSTM 2.28 0.74 LSTM 2.28 0.74 CRN 2.42 0.82 CRN 2.42 0.82 MHA-Bi-LSTM 2.70 0.93 MHA-Bi-LSTM 2.70 0.93

[0127] As shown in Table 3, compared with LSTM, PESQ is improved by 0.42 and STOI is improved by 0.2. This is because the complex Bi-LSTM has a strong ability to consider the sequence dependencies in amplitude and phase at the same time. Compared with CRN, PESQ is improved by 0.28 and STOI is improved by 0.12, indicating that the multi-dimensional attention mechanism effectively improves the expressiveness of input and output features, and can focus on key features more accurately. Complex spectral features can also enhance the use of amplitude and phase information, thereby improving network performance. In summary, the proposed method has better dereverberation performance.

[0128] In summary, the present embodiment proposes a method for complex dereverberation speech enhancement based on multi-head attention and Bi-LSTM, and proposes three methods that can improve the performance of the speech enhancement system from the three aspects of weight loss, attention mechanism, and complex operation. The experimental results verify the rationality and effectiveness of these three methods. Compared with other methods, the method of the present embodiment is improved in multiple measurement indicators, which can overcome the shortcomings of ordinary methods for speech enhancement and dereverberation to a certain extent, and has a good reference significance in practical applications.

[0129] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM, characterized in that: The steps include: Step S1: Input the speech signal containing reverberation and perform STFT transformation to convert the speech signal into the frequency domain, extract the amplitude spectrum features and the complex spectrum features, and obtain the frequency domain representation F; Step S2: Input the feature F into the high-level channel attention mechanism HCA, F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and the average pooling feature and the maximum pooling feature are fused to obtain the new feature F a+m , F avg and F max The features are processed by convolution blocks and then combined with F a+m Fusion, get the feature representation new feature F sig , F sig Multiply and fuse element by element with F to get the output feature F HCA ; Step S3: The output feature F of HCA HCA The input is fed into the time-frequency dual-path complex multi-head attention mechanism TF-CMHA, which uses the time and frequency information to capture the key components of the speech signal based on the correlation between different positions within the input sequence, and identifies and suppresses the reverberation components. Step S4: The output features of TF-CMHA are divided into two channels, the real part and the imaginary part, and enter the complex bidirectional long short-term memory network CBi-LSTM to capture the context of the deep features of the speech signal. The correlation between amplitude and phase is simulated by the introduced complex multiplication operation to obtain a more accurate description of the long-term speech correlation. Step S5: a joint constraint loss function is designed to train the network model, in which the real and imaginary parts of the complex spectrum feature are regarded as independent features and trained synchronously to reduce the error between the estimated value and the true value, and finally restored to the estimated time domain signal through ISTFT transformation; Step S6: Perform a performance evaluation on the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM.

2. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that: The specific steps of step S2 are: Step S21: F avg and F max Represent the average pooling feature and the maximum pooling feature respectively, and add each dimension of the average pooling feature and the maximum pooling feature element by element as a new feature F a+m ; Step S22: The average pooling feature and the maximum pooling feature are processed by convolution blocks respectively and combined with the new feature F a+m Once again, the elements are added and fused, and activated by the sigmoid function to obtain the new feature F sig ; Step S23: Output feature F sig Multiply element-wise with the input feature F to get F HCA , emphasizing the important parts of the input features and suppressing unimportant information.

3. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that: The specific steps of step S3 are: Step S31: Input feature F with dimension T×F×C HCA Reshape into FC vector of dimension 1×T and TC vector of dimension 1×F respectively, one along the time axis and the other along the frequency axis; Step S32: Apply the multi-head attention mechanism MHA in parallel along each axis to capture the intrinsic connection between different positions of the input data; Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher-level representation, and then passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model; Step S34: reshape and splice the output features of the real part and the imaginary part as the final output features of the single time axis and the single frequency axis, and then combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result X of TF-CMHA A .

4. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 3, characterized in that: In step S32, the specific steps are: Step S321: input the multi-head attention mechanism MHA with 8 attention heads; Step S322: The MHA attention calculation is performed in parallel on the real part and the imaginary part of the feature respectively.

5. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that: The specific steps of step S4 are: Step S41: Divide into two channels, the real part and the imaginary part, and enter the CBi-LSTM, and use complex multiplication operation; Step S42: Each channel consists of two stacked Bi-LSTM layers, and the output dimension of each time step is 1446, resulting in a feature matrix of 426×1446; Step S43: After passing through the linear layer (1446, 723), the final 426×723 feature matrix is ​​obtained.

6. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 5, characterized in that: The specific steps of step S42 are: Step S421: define the complex-valued convolution filter W as W=W r +jW i , where the real-valued matrix W r and W i represent the real and imaginary parts of the complex-valued convolution kernel respectively, and the input complex spectrum can be defined as X = X r +jX i Therefore, the complex spectrum output Y can be obtained through complex convolution operation, which can be expressed as Y=(X r *W r -X i *W i )+j(X r *W i +X i *W r ); Step S422: The output feature after TF-CMHA is X A , its real and imaginary parts are represented by X Ar , X Ai , as two input channels, are input into Bi-LSTM respectively. The specific operation process is as follows. The final output of CBi-LSTM is F out , F rr =BiLSTM r (X Ar ) F ri =BiLSTM r (X Ai ) F ir =BiLSTM i (X Ar ) F ii =BiLSTM i (X Ai ) F out =(F rr -F ii )+j(F ri +F ir ) Among them, BiLSTM r and BiLSTM i Two traditional Bi-LSTMs, F, represent the real part and the imaginary part respectively. rr Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ri Indicates that the real feature X will be input Ai The output feature after being fed into the real Bi-LSTM, F ir Indicates that the real feature X will be input Ar The output feature after being fed into the real Bi-LSTM, F ii Indicates that the real feature X will be input Ai Output features after being fed into the real Bi-LSTM.

7. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that: The specific steps of step S5 are: Step S51: define the time domain MSE loss function: x represents the clean speech signal, represents the estimated speech signal; Step S52: Because the invention is studied in the complex domain, in order to fully consider the independence and importance difference between the real part and the imaginary part, as well as the relationship between amplitude and phase, a joint loss function is designed, including the amplitude spectrum of the speech signal, the real part and the imaginary part of the complex spectrum, a total of three parts, after calculating their MSE losses respectively, they are combined as the final loss function of the model: Among them, Y mag represents the amplitude spectrum of the clean speech signal, Represents the amplitude spectrum of the estimated signal, Y r and Y i Represent the real and imaginary parts of the clean speech signal respectively, and Represent the real and imaginary parts of the estimated signal respectively, α, β, γ∈[0,1] are weight coefficients; Step S53: Finally, output the predicted time domain speech signal after ISTFT transformation.

8. The method for complex dereverberation speech enhancement based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that: The specific method for implementing step S6 is: Step S61: Discuss the specific value of the weight loss in the joint constraint loss and explore the optimal weight value; Step S62: performing an ablation experiment on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part; Step S63: A comparative experiment is conducted between the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM and the LSTM model and the CRN model to verify the effectiveness of the entire model.

Images