A complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM

Abstract

The application discloses a complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM, and belongs to the technical field of speech enhancement. The application carries out dereverberation processing in a complex domain, firstly carries out STFT transformation on signals to obtain a complex spectrum, so as to fully utilize amplitude and phase information. Then, the complex spectrum feature is sent into an advanced channel attention mechanism HCA to fuse the feature. Secondly, in order to be able to capture the correlation between different positions in the input sequence and utilize time and frequency information to capture speech key components, a time-frequency dual-path complex multi-head attention mechanism TF-CMHA is designed, speech components are more accurately identified, and reverberation components are suppressed. Finally, complex multiplication operation and a joint constraint loss function are introduced, the correlation between amplitude and phase is simulated, and a more accurate long-time speech correlation description is obtained. The application improves the performance of a dereverberation speech enhancement system, speech after dereverberation of the system is improved in various measurement indexes, and the application is suitable for speech front-end processing.

Description

Technical Field

[0001] This invention belongs to the field of speech enhancement technology, specifically relating to a complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM. Background Technology

[0002] Speech enhancement technology has been applied in many aspects of life. In practical applications, besides noise, reverberation is also a significant factor affecting speech quality. In particular, the sound signal received by a microphone in a reverberant environment not only contains the original speech signal but also incorporates multiple delayed and attenuated reflections. This can mask crucial information in the speech, thus affecting the quality and intelligibility of the speech signal. Therefore, dereverberation speech enhancement technology has significant practical implications in the presence of reverberant noise.

[0003] Deep learning technology is increasingly prominent and widely implemented in public applications, including but not limited to speech processing, image analysis, and natural language processing. Due to its powerful fitting capabilities and effective utilization of large datasets, deep learning-based methods can more accurately model the complex relationships between target and interference signals. Furthermore, deep learning methods have stronger generalization capabilities across various signal types and are not limited by assumptions such as signal independence and Gaussian noise, thus exhibiting superior performance in speech enhancement.

[0004] The complexity and diversity of reverberation signals make it difficult for models to accurately suppress reverberation components. Furthermore, most existing deep learning models are based on amplitude spectrum processing, which ignores the complex characteristics of speech signals, potentially leading to the loss of some useful information when dealing with reverberation problems.

[0005] In recent years, extensive research has been conducted on directional speech enhancement methods for reverberation. Deep learning-based dereverberation methods mainly utilize two principles: spectral mapping and time-frequency masking. Models based on multi-hidden-layer neural networks aim to recover the amplitude spectrum of the source speech by mapping the amplitude spectrum of the reverberant speech, thereby enhancing the model's dereverberation performance. However, due to limitations in shallow network architecture and loss function design, this mapping method still faces challenges in achieving accurate recovery of the source speech amplitude spectrum. Wang, Williamson, and others employed TF masking technology, using DNN training to obtain a complex ideal ratio mask of the reverberant speech amplitude spectrum. They then extracted the source speech by multiplying this mask by the reverberant speech spectrum, which improved the clarity of the speech after reverberation removal. However, this method still faces the challenge of not being able to effectively suppress the highly correlated parts of the reverberation. Liu Bin et al. used a recurrent neural network combined with a bidirectional long short-term memory (Bi-LSTM) network to map the logarithmic power spectrum features of the reverberant speech, effectively capturing the contextual correlation features. Zhao, Wang, and others introduced an attention mechanism into their dereverberation models, significantly improving the model's performance in mapping the amplitude spectrum of source speech through refined modeling of the correlations between frequency band information. Jieyuan et al., without relying on prior information about the target speech and acoustic environment, proposed a speech enhancement method based on a time-frequency Wiener masking inverse filter using a primal-dual splitting algorithm, demonstrating superior performance in noisy and reverberant scenarios. This invention thoroughly studies and improves the performance of attention-based speech enhancement systems in reverberant noise environments, designing a complex dereverberation speech enhancement method based on a multi-head attention mechanism and Bi-LSTM. Summary of the Invention

[0006] To address the problem of insufficient dereverberation in existing technologies, this invention provides a complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM. It introduces a time-frequency dual-path complex multi-head attention mechanism (TF-CMHA) to capture the correlation between different positions within the input sequence and utilizes time and frequency information to capture key speech components. Furthermore, it introduces a complex bidirectional long short-term memory network (Bi-LSTM) to obtain a more accurate description of long-term speech correlation using complex multiplication operations, thereby improving the performance of the dereverberation speech enhancement system.

[0007] A complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM includes the following steps:

[0008] Step S1: Input a reverberant speech signal and perform STFT transformation to convert the speech signal to the frequency domain, extract amplitude spectrum features and complex spectrum features, and obtain the frequency domain representation F;

[0009] Step S2: Input the feature F into the high-level channel attention mechanism HCA, F avg and F max Let F represent the average pooling feature and the max pooling feature, respectively. The average pooling feature and the max pooling feature are then fused to obtain a new feature F. a+m F avg and F max Feature processing is performed separately using convolutional blocks and then compared with F. a+m The features are fused to obtain a new feature F. sig F sig The output feature F is obtained by performing element-wise multiplication and fusion with F. HCA ;

[0010] Step S3: Convert the output feature F of HCA HCA The input is fed into the time-frequency dual-path complex multi-head attention mechanism TF-CMHA, which uses time and frequency information to capture key components in the speech signal based on the correlation between different positions within the input sequence, and identifies and suppresses reverberation components.

[0011] Step S4: The output features of TF-CMHA are divided into real and imaginary channels and fed into the complex bidirectional long short-term memory network CBi-LSTM to capture the deep relationship between speech signal features. By introducing complex multiplication operations to simulate the correlation between amplitude and phase, a more accurate description of long-term speech correlation is obtained.

[0012] Step S5: Design a joint constraint loss function to train the network model, where the real and imaginary parts of the complex spectral features are treated as independent features and trained synchronously to reduce the error between the predicted value and the true value. Finally, the estimated time domain signal is recovered by ISTFT transformation.

[0013] Step S6: Evaluate the performance of the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM.

[0014] As a further improvement of the present invention, the specific steps of step S2 are as follows:

[0015] Step S21: F avg and F max Let F represent the average pooling feature and the max pooling feature, respectively. Each dimension of the average pooling feature and the max pooling feature is summed element-wise to form a new feature F. a+m ;

[0016] Step S22: Average pooling features and max pooling features are processed through convolutional blocks, and then combined with the new feature F. a+mThe elements are added together again and then activated using the sigmoid function to obtain the new feature representation F. sig .

[0017] Step S23: Output feature F sig F is obtained by element-wise multiplication with the input feature F. HCA It emphasizes the important parts of the input features and suppresses unimportant information, and can also fuse information between different features to obtain richer feature representations.

[0018] As a further improvement of the present invention, the specific steps of step S3 are as follows:

[0019] Step S31: Input features F with dimension T×F×C HCA They are reshaped into FC vectors of dimension 1×T and TC vectors of dimension 1×F, one along the time axis and the other along the frequency axis.

[0020] Step S32: Apply the multi-head attention mechanism (MHA) parallel along each axis to capture the intrinsic relationships between different locations in the input data.

[0021] Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher level of representation. Then, they are passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model.

[0022] Step S34: Reconstruct and concatenate the output features of the real and imaginary parts to obtain the final output features for the single time axis and single frequency axis. Then, combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result X of TF-CMHA. A .

[0023] As a further improvement of the present invention, step S32 specifically includes the following steps:

[0024] Step S321: Input the multi-head attention mechanism MHA with 8 attention heads.

[0025] Step S322: The attention calculation for MHA is performed in parallel on the real and imaginary parts of the feature.

[0026] As a further improvement of the present invention, the specific steps of step S4 are as follows:

[0027] Step S41: The real and imaginary parts are split into two channels and fed into the CBi-LSTM, where complex multiplication is performed.

[0028] Step S42: Each channel consists of two stacked Bi-LSTM layers, with an output dimension of 1446 at each time step, resulting in a 426×1446 feature matrix.

[0029] Step S43: After passing through the linear layer (1446, 723), the final feature matrix of 426×723 is obtained.

[0030] As a further improvement of the present invention, step S42 is as follows:

[0031] Step S421: Define the complex-valued convolution filter W as W = W r +jW i The real-valued matrix W here r and W i Let X represent the real and imaginary parts of the complex convolution kernel, respectively. The input complex spectrum can be defined as X = X r +jX i Therefore, the output Y of the complex spectrum can be obtained through complex convolution operations, and can be expressed as:

[0032] Y = (X r *W r -X i *W i )+j(X r *W i +X i *W r ).

[0033] Step S422: The output feature after TF-CMHA is X A The real part and the imaginary part are represented as X. Ar X Ai As two input channels, they are respectively input into the Bi-LSTM. The specific operation process is as follows, and the final output of the CBi-LSTM is F. out .

[0034] F rr =BiLSTM r (X Ar )

[0035] F ri =BiLSTM r (X Ai )

[0036] F ir =BiLSTM i (X Ar )

[0037] F ii =BiLSTM i (X Ai )

[0038] F out =(F rr -F ii )+j(F ri +F ir )

[0039] Among them, BiLSTM r and BiLSTM i F represents two traditional Bi-LSTMs, one for the real part and one for the imaginary part. rr This indicates that the input real feature X will be used. Ar The output characteristics after being fed into a real Bi-LSTM, F ri This indicates that the input real feature X will be used. Ai The output characteristics after being fed into a real Bi-LSTM, F ir This indicates that the input real feature X will be used. Ar The output characteristics after being fed into a real Bi-LSTM, F ii This indicates that the input real feature X will be used. Ai Output characteristics after being fed into a real Bi-LSTM.

[0040] As a further improvement of the present invention, the specific steps of step S5 are as follows:

[0041] Step S51: Define the time-domain MSE loss function:

[0042]

[0043] x represents a clean speech signal. This represents the estimated speech signal.

[0044] Step S52: A joint loss function is designed, comprising three parts: the amplitude spectrum of the speech signal, the real part of the complex spectrum, and the imaginary part. Their MSE losses are calculated separately and then combined as the final loss function of the model.

[0045]

[0046] Among them, Y mag This represents the amplitude spectrum of a clean speech signal. Y represents the amplitude spectrum of the predicted signal. r and Y i Let represent the real and imaginary parts of a clean speech signal, respectively. and Let α, β, and γ represent the real and imaginary parts of the predicted signal, respectively, and let α, β, and γ ∈ [0,1] be the weighting coefficients.

[0047] Step S53: Finally, output the predicted time-domain speech signal after ISTFT transformation.

[0048] As a further improvement of the present invention, the specific method for implementing step S6 is as follows:

[0049] Step S61: Discuss the specific values ​​of the weights in the weight loss of the joint constraint loss, and explore the optimal weight values.

[0050] Step S62: Conduct ablation experiments on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part.

[0051] Step S63: Compare the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM with the LSTM model and CRN model to verify the effectiveness of the entire model.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] 1. When solving the problem of reverberant speech enhancement, most existing deep learning models are based on amplitude spectrum processing, ignoring the complex characteristics of speech signals. This may result in the loss of some useful information when processing reverberation. This invention utilizes the advantages of complex domain processing to represent speech signals in complex form, so as to make full use of the amplitude and phase information of speech signals.

[0054] 2. A time-frequency dual-path complex multi-head attention mechanism (TF-CMHA) was designed, which can capture the correlation between different positions within the input sequence and simultaneously consider time and frequency information. This enables the model to focus on multiple key parts of the speech signal, more accurately identify and suppress reverberation components, and further improve the computational efficiency and representational ability of the model.

[0055] 3. To capture the deep features and relationships of speech signals, this invention introduces complex multiplication, which can better simulate the correlation between amplitude and phase, obtaining a more accurate description of long-term speech correlation. Simultaneously, a joint constraint loss function is designed to train the network model, further improving the performance of the dereverberation system. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the operation process of the present invention.

[0057] Figure 2 This is a structural diagram of the High-Level Channel Attention (HDC) mechanism in this invention.

[0058] Figure 3 This is a flowchart illustrating the implementation of the TF-CMHA module of the present invention.

[0059] Figure 4 This is a model architecture diagram of the CBi-LSTM module of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the solutions of this invention are shown in the drawings, while other details not closely related to this invention are omitted.

[0061] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0062] like Figures 1 to 4 As shown, this invention provides a complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM. It applies deep neural networks to speech enhancement in reverberant environments to extract deep features, which has a certain suppression effect on reverberant noise. This allows for the construction of a higher-performance dereverberation speech enhancement system, and includes the following steps:

[0063] Step S1: The database used in this invention is the VoiceBank corpus. 3597 randomly selected speech samples from the speaker were mixed and used as the training set. 285 samples from the remaining speech samples were mixed and used as the test set. The impact response in a room was simulated using the mirror method. A virtual room with dimensions of 7.72m × 8.10m × 7.60m (length × width × height) was defined. The speaker was fixed at position [3.0, 4.0, 1.5], and the microphone was fixed at position [5.0, 7.0, 2.5]. The reverberation time was set to 0.5s. This was then convolved with the selected clean speech signal to form the training and test sets. The data sampling rate was 16kHz. The input reverberant speech signal underwent a 512-point STFT transformation with a frame shift of 128. The STFT transformation used a Hanning window with a window size of 2048. The speech signal was converted to the frequency domain, and amplitude spectrum features and complex spectrum features were extracted to obtain the frequency domain representation F.

[0064] Step S2: The output feature F is input into the high-level channel attention mechanism HCA. The structure diagram of HCA is shown below. Figure 2 As shown, F avg and F max Let F represent the average pooling feature and the max pooling feature, respectively. The average pooling feature and the max pooling feature are then fused to obtain a new feature F. a+m F avg and F maxFeature processing is performed separately using convolutional blocks and then compared with F. a+m The features are fused to obtain a new feature F. sig F sig The output feature F is obtained by performing element-wise multiplication and fusion with F. HCA .

[0065] Step S2 specifically includes:

[0066] Step S21: F avg and F max Let F represent the average pooling feature and the max pooling feature, respectively. Each dimension of the average pooling feature and the max pooling feature is summed element-wise to form a new feature F. a+m ;

[0067] Step S22: Average pooling features and max pooling features are processed through convolutional blocks, and then combined with the new feature F. a+m The elements are added together again and then activated using the sigmoid function to obtain the new feature representation F. sig .

[0068] Step S23: Output feature F sig F is obtained by element-wise multiplication with the input feature F. HCA It emphasizes the important parts of the input features and suppresses unimportant information, and can also fuse information between different features to obtain richer feature representations.

[0069] Step S22 specifically includes:

[0070] Step S221: The first two-dimensional convolutional layer reduces the number of channels in the input feature map by half, thereby reducing the complexity and number of parameters of the model.

[0071] Step S222: ReLU activation function transformation to enhance the expressive power of the model.

[0072] Step S223: The last two-dimensional convolutional layer restores the number of channels to the number of channels when inputting HCA, in order to retain and integrate more feature information.

[0073] Step S224: The Sigmoid function can map the input value to a range between 0 and 1. The closer to 1, the more important the channel is, and the closer to 0, the less important the channel is.

[0074] Step S3: Convert the output feature F of HCA HCA Input into TF-CMHA, flowchart as follows Figure 3As shown, TF-CMHA can capture the correlation between different positions within the input sequence and simultaneously consider time and frequency information, enabling the model to focus on multiple key parts of the speech signal and more accurately identify and suppress reverberation components.

[0075] Step S3 specifically includes:

[0076] Step S31: Input features F with dimension T×F×C HCA They are reshaped into FC vectors of dimension 1×T and TC vectors of dimension 1×F, one along the time axis and the other along the frequency axis.

[0077] Step S32: Apply the multi-head attention mechanism (MHA) parallel along each axis to capture the intrinsic relationships between different locations in the input data.

[0078] Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher level of representation. Then, they are passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model.

[0079] Step S34: Reconstruct and concatenate the output features of the real and imaginary parts to obtain the final output features for a single time axis and a single frequency axis. Then, combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result X of TF-CMHA. A .

[0080] Step S32 specifically includes:

[0081] Step S321: The number of attention heads in the multi-head attention mechanism (MHA) is 8.

[0082] Step S322: The attention calculation for MHA is performed in parallel on the real and imaginary parts of the feature.

[0083] Step S33 specifically includes:

[0084] Step S331: The dimension of the linear layer is (512, 723).

[0085] Step S332: The dropout probability of the dropout layer is 0.1.

[0086] Step S4: Convert the feature X output by TF-CMHA A The signal is divided into two channels, real and imaginary, which are fed into the CBi-LSTM to capture the deep features of the speech signal and their relationships. The complex multiplication operation introduced can better simulate the correlation between amplitude and phase, and obtain a more accurate description of long-term speech correlation.

[0087] Step S4 specifically includes:

[0088] Step S41: The data is split into two channels, real and imaginary, and fed into the CBi-LSTM. Complex multiplication is then applied. The model architecture is as follows: Figure 4 As shown.

[0089] Step S42: Each channel consists of two stacked Bi-LSTM layers, with an output dimension of 1446 at each time step, resulting in a 426×1446 feature matrix.

[0090] Step S43: After passing through the linear layer (1446, 723), the final feature matrix of 426×723 is obtained.

[0091] Step S42 specifically includes:

[0092] Step S421: Define the complex-valued convolution filter W as W = W r +jW i The real-valued matrix W here r and W i Let X represent the real and imaginary parts of the complex convolution kernel, respectively. The input complex spectrum can be defined as X = X r +jX i Therefore, the output Y of the complex spectrum can be obtained through complex convolution operations, and can be expressed as:

[0093] Y = (X r *W r -X i *W i )+j(X r *W i +X i *W r ).

[0094] Step S422: The output feature after TF-CMHA is X A The real part and the imaginary part are represented as X. Ar X Ai As two input channels, they are respectively input into the Bi-LSTM. The specific operation process is as follows, and the final output of the CBi-LSTM is F. out .

[0095] F rr =BiLSTM r (X Ar )

[0096] F ri =BiLSTM r (X Ai )

[0097] F ir =BiLSTM i (XAr )

[0098] F ii =BiLSTM i (X Ai )

[0099] F out =(F rr -F ii )+j(F ri +F ir )

[0100] Among them, BiLSTM r and BiLSTM i F represents two traditional Bi-LSTMs, one for the real part and one for the imaginary part. rr This indicates that the input real feature X will be used. Ar The output characteristics after being fed into a real Bi-LSTM, F ri This indicates that the input real feature X will be used. Ai The output characteristics after being fed into a real Bi-LSTM, F ir This indicates that the input real feature X will be used. Ar The output characteristics after being fed into a real Bi-LSTM, F ii This indicates that the input real feature X will be used. Ai Output characteristics after being fed into a real Bi-LSTM.

[0101] Step S5: Using the real and imaginary parts of the complex spectrum and amplitude characteristics, a joint constraint loss function is designed to train the network model, thereby reducing the error between the predicted value and the true value.

[0102] Step S5 specifically includes:

[0103] Step S51: Define the time-domain MSE loss function:

[0104]

[0105] x represents a clean speech signal. This represents the estimated speech signal.

[0106] Step S52: Because the invention is conducted in the complex domain, to fully consider the differences in independence and importance between the real and imaginary parts, as well as the relationship between amplitude and phase, a joint loss function is designed, comprising three parts: the amplitude spectrum of the speech signal, the real part of the complex spectrum, and the imaginary part. Their MSE losses are calculated separately and then combined as the final loss function of the model.

[0107]

[0108] Among them, Y magThis represents the amplitude spectrum of a clean speech signal. Y represents the amplitude spectrum of the predicted signal. r and Y i Let represent the real and imaginary parts of a clean speech signal, respectively. and Let α, β, and γ represent the real and imaginary parts of the predicted signal, respectively, and let α, β, and γ ∈ [0,1] be the weighting coefficients.

[0109] Step S53: Finally, output the predicted time-domain speech signal after ISTFT transformation.

[0110] Step S6: Evaluate the performance of the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM.

[0111] Step S6 specifically includes:

[0112] Step S61: Discuss the specific values ​​of the weights in the weight loss of the joint constraint loss, and explore the optimal weight values.

[0113] Step S62: Conduct ablation experiments on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part.

[0114] Step S63: Compare the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM with the LSTM model and CRN model to verify the effectiveness of the entire model.

[0115] In this embodiment, the sampling rate of the data used is 16kHz, and the speech signal is converted to the frequency domain through a 512-point STFT with a frame shift of 128. The entire training iteration cycle is 30, the batch size is 2, the optimizer is Adam, and the initial learning rate is set to 0.0005. This embodiment uses two speech evaluation metrics, PESQ and STOI, to evaluate the network's dereverberation performance.

[0116] Experiments were conducted to verify the effectiveness of the joint constraint loss function. Three sets of comparative experiments with different weighting coefficients were set up to evaluate the impact of the real and imaginary parts of the amplitude spectrum feature and the complex spectrum feature on the performance of the dereverberation system. The experimental results are shown in Table 1.

[0117] Table 1

[0118]

[0119] Table 1 shows that the model using the joint constraint loss function generally outperforms the model using the traditional MSE loss function. Furthermore, the network model pays more attention to the imaginary part of the complex spectrum, and the model achieves the best dereverberation performance when this part has a higher weight in the joint loss. Experimental results show that compared to the traditional MSE loss function, the joint constraint loss function with a larger amplitude spectrum weight improves PESQ by 0.03 and STOI by 0.01. Compared to amplitude spectrum features, the higher weight of complex spectrum features allows the model to focus more on the details of the real or imaginary parts while maintaining the overall structure of the complex spectrum, thus improving the dereverberation performance. The better PESQ and STOI indices when the imaginary part weight is larger indicate that the model pays more attention to the recovery of phase information during training, thereby improving the quality of speech enhancement. In conclusion, the proposed joint constraint loss function is effective.

[0120] The effectiveness of each component was investigated, with a "√" indicating that the module was used in the network. The experimental results are shown in Table 2:

[0121] Table 2

[0122] Bi-LSTM Complex multiplication TF-CMHA HCA PESQ STOI √ 2.37 0.81 √ √ 2.48 0.87 √ √ √ 2.66 0.91 √ √ √ √ 2.70 0.93

[0123] Table 2 shows that introducing complex multiplication into Bi-LSTM improves PESQ by 0.11 and STOI by 0.06, indicating that the model pays more attention to the importance of amplitude and phase. Adding TF-CMHA further improves PESQ by 0.18 and STOI by 0.03, demonstrating that capturing contextual information from both temporal and frequency dimensions, and capturing the feature correlations between different positions within the input sequence, is crucial for the dereverberation speech enhancement task. Finally, adding HCA improves PESQ by 0.04 and STOI by 0.02, indicating that the multi-dimensional attention mechanism can effectively improve the expressive power of input and output features, thereby enhancing the performance of the speech enhancement model.

[0124] The effectiveness of the complex dereverberation speech enhancement method based on multi-head attention and Bi-LSTM is evaluated. It is compared with the LSTM model, CRN model, and the model from Chapter 3. The experimental results are shown in Table 3.

[0125] Table 3

[0126] Model PESQ STOI Model PESQ STOI LSTM 2.28 0.74 LSTM 2.28 0.74 CRN 2.42 0.82 CRN 2.42 0.82 MHA-Bi-LSTM 2.70 0.93 MHA-Bi-LSTM 2.70 0.93

[0127] Table 3 shows that compared to LSTM, PESQ improves performance by 0.42 and STOI by 0.2. This is because complex Bi-LSTM has a stronger ability to simultaneously consider the sequential dependencies in amplitude and phase. Compared to CRN, PESQ improves performance by 0.28 and STOI by 0.12, indicating that the multi-dimensional attention mechanism effectively improves the expressive power of input and output features, allowing for more accurate focus on key features. Complex spectral features also enhance the utilization of amplitude and phase information, thereby improving network performance. In summary, the proposed method demonstrates better dereverberation performance.

[0128] In summary, this embodiment proposes a complex dereverberation speech enhancement method based on multi-head attention and Bi-LSTM. It offers three methods to improve the performance of speech enhancement systems from three aspects: weight loss, attention mechanism, and complex number operations. Experimental results verify the rationality and effectiveness of these three methods. Compared with other methods, the method in this embodiment achieves improvements in multiple measurement indicators and can overcome the shortcomings of ordinary methods in speech enhancement and dereverberation to a certain extent. It has significant reference value in practical applications.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM, characterized in that, Includes the following steps: Step S1: Input a reverberant speech signal and perform STFT transformation to convert the speech signal to the frequency domain. Extract amplitude spectrum features and complex spectrum features to obtain the frequency domain representation. ; Step S2: Add features Input to the advanced channel attention mechanism HCA, and These represent the average pooling feature and the max pooling feature, respectively. The average pooling feature and the max pooling feature are then fused to obtain a new feature. , and Feature processing is performed separately using convolutional blocks and then combined again. Fusion yields new features representing the features. , and Element-wise multiplication and fusion are performed to obtain the output features. ; Step S3: Extract the output features of HCA The input is fed into the time-frequency dual-path complex multi-head attention mechanism TF-CMHA, which uses time and frequency information to capture key components in the speech signal based on the correlation between different positions within the input sequence, and identifies and suppresses reverberation components. Step S4: The output features of TF-CMHA are divided into real and imaginary channels and fed into the complex bidirectional long short-term memory network CBi-LSTM to capture the relationship between deep features of the speech signal. The correlation between amplitude and phase is simulated by the introduced complex multiplication operation to obtain a more accurate description of long-term speech correlation. Step S5: Design a joint constraint loss function to train the network model, where the real and imaginary parts of the complex spectral features are treated as independent features and trained synchronously to reduce the error between the predicted value and the true value. Finally, the estimated time domain signal is recovered by ISTFT transformation. Step S6: Evaluate the performance of the proposed complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM; The specific steps of step S4 are as follows: Step S41: The real and imaginary parts are split into two channels and fed into the CBi-LSTM, where complex multiplication is performed. Step S42: Each channel consists of two stacked Bi-LSTM layers, with an output dimension of 1446 at each time step, resulting in a 426×1446 feature matrix; Step S43: After passing through a linear layer (1446, 723), the final feature matrix of 426×723 is obtained; The specific steps of step S42 are as follows: Step S421: Define the complex-valued convolution filter for The real-valued matrix here and Representing the real and imaginary parts of the complex convolution kernel, respectively, and with the input complex spectrum, it can be defined as follows: Therefore, the complex spectrum can be obtained through complex convolution operations. , can be represented as ； Step S422: The output characteristics after TF-CMHA are The real part and the imaginary part are represented as , As two input channels, they are respectively input into the Bi-LSTM. The specific operation process is as follows, and the final output of the CBi-LSTM is... , in, and Two traditional Bi-LSTMs, representing the real and imaginary parts respectively, This indicates that the input real features will be used. The output characteristics after being fed into a real Bi-LSTM This indicates that the input real features will be used. The output characteristics after being fed into a real Bi-LSTM This indicates that the input real features will be used. The output characteristics after being fed into a real Bi-LSTM This indicates that the input real features will be used. Output characteristics after being fed into a real Bi-LSTM; The specific steps of step S5 are as follows: Step S51: Define the time-domain MSE loss function: Indicates a clean speech signal. This represents the estimated speech signal; Step S52: Because the invention is conducted in the complex domain, to fully consider the differences in independence and importance between the real and imaginary parts, as well as the relationship between amplitude and phase, a joint loss function is designed, comprising three parts: the amplitude spectrum of the speech signal, the real part of the complex spectrum, and the imaginary part. Their MSE losses are calculated separately and then combined as the final loss function of the model. in, This represents the amplitude spectrum of a clean speech signal. This represents the amplitude spectrum of the predicted signal. and Let represent the real and imaginary parts of a clean speech signal, respectively. and These represent the real and imaginary parts of the predicted signal, respectively. These are the weighting coefficients; Step S53: Finally, output the predicted time-domain speech signal after ISTFT transformation.

2. The complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S21: and Let these represent the average pooling feature and the max pooling feature, respectively. Each dimension of the average pooling feature and the max pooling feature is summed element-wise to form a new feature. ; Step S22: Average pooling features and max pooling features are processed through convolutional blocks, and then combined with the new features. The elements are added together again and then activated using the sigmoid function to obtain new feature representations. ; Step S23: Output features Input features Element-wise multiplication yields It emphasizes the important parts of the input features and suppresses unimportant information.

3. The complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S31: Input features of dimension T×F×C Reshape them into FC vectors of dimension 1×T and TC vectors of dimension 1×F, one along the time axis and the other along the frequency axis, respectively. Step S32: Apply the multi-head attention mechanism (MHA) parallel along each axis to capture the intrinsic relationships between different locations in the input data; Step S33: The output results are then passed through a fully connected layer to further integrate and refine the features to obtain a higher level of representation. Then, they are passed through a dropout layer and a layer normalization layer to prevent overfitting and improve the accuracy and stability of the model. Step S34: Reconstruct and concatenate the output features of the real and imaginary parts to obtain the final output features for the single time axis and single frequency axis. Then, combine the final output features of the time-frequency dual branch with the original input features to obtain the final output result of TF-CMHA. .

4. The complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM according to claim 3, characterized in that, In step S32, the specific steps are as follows: Step S321: Input the multi-head attention mechanism MHA with 8 attention heads; Step S322: The attention calculation for MHA is performed in parallel on the real and imaginary parts of the feature.

5. The complex dereverberation speech enhancement method based on multi-head attention mechanism and Bi-LSTM according to claim 1, characterized in that, The specific method for implementing step S6 is as follows: Step S61: Discuss the specific values ​​of the weights in the weight loss of the joint constraint loss, and explore the optimal weight values; Step S62: Conduct ablation experiments on the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM to verify the effectiveness of each part; Step S63: Compare the complex dereverberation speech enhancement algorithm based on multi-head attention and Bi-LSTM with the LSTM model and CRN model to verify the effectiveness of the entire model.

Images