Primary-secondary network speech enhancement system with fusion attention mechanism

Abstract

This invention belongs to the field of signal and information processing technology, specifically relating to a master-slave network speech enhancement system incorporating an attention mechanism. It includes a master network, a slave network, and a speech reconstruction module. The master network comprises a first feature extraction module and a second feature extraction module. The first feature extraction module performs convolution and feature mapping on the speech input signal to obtain a first feature. The second feature extraction module processes the input signal through a three-layer bidirectional gated recurrent unit (BiGRU) and a multi-head attention mechanism to obtain a second feature. The first and second features are concatenated to form the master network output feature. The slave network concatenates the master network output feature with the input signal to obtain a third feature, which is then sent to a first CNN network to calculate the slave network output feature. The speech reconstruction module adds the master network output feature and the slave network output feature and reconstructs the enhanced speech. This invention can improve speech enhancement capabilities.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of signal and information processing, and particularly relates to a main and auxiliary network speech enhancement system fusing an attention mechanism. BACKGROUND

[0002] In daily hearing environments, speech signals are distorted by background noise, which seriously reduces the speech intelligibility and quality of listeners and makes many speech-related tasks, such as automatic speech recognition and speaker recognition, more difficult. Based on the above problems, speech enhancement technology is applied, which has attracted much attention and research. Speech enhancement algorithm aims to improve speech quality and intelligibility of speech signals degraded by noise, and enhance the understanding of speech by humans or machines, which is an important step for many speech-related tasks. Typical applications include hearing aids, automatic speech recognition and audio-video communication in noisy environments. Deep learning can well learn the complex mapping relationship between pure speech and noisy speech, so the concept of deep learning is widely used in speech tasks and has rapidly developed into an important branch of machine learning.

[0003] Due to the ability of convolutional neural network (CNN) to model spatial and temporal correlations, it has achieved remarkable success in image recognition and computer vision. The structural design of CNN uses two-dimensional structured input, and focuses on local information by using local connection. Tomas et al. effectively applied CNN to the speech enhancement task, and the CNN structure contains a fully connected layer, which contains more parameters and is not suitable for mobile devices. In 2018, Tan et al. proposed a convolutional recurrent neural network (CRN) and integrated phase information, and the experimental results showed that this network is superior to the traditional CNN. A novel fully convolutional neural network FLGCNN was proposed to solve the end-to-end speech time domain enhancement problem. FLGCNN is mainly based on an encoder and a decoder, and a time convolution module (TCM) is inserted between the two, which can effectively model the long-term dependence of speech signals. At the same time, CNN has certain defects and cannot capture long-distance features. Although convolution operation is good at focusing on local details of input speech signals, its receptive field is very limited, so it needs to stack multiple layers to learn the context dependence of speech signals. However, with the increase of the number of layers, a lot of redundant information will be generated in the network, which is not conducive to the learning of the network after layer-by-layer transmission.

[0004] Speech often contains redundant or irrelevant information. The human brain can quickly prioritize information based on its importance, and a single CNN lacks this learning ability. Some researchers have attempted to teach models how to allocate their attention, i.e., weighting the input signal. They use attention mechanisms to score various dimensions of the input and then weight features according to the scores to highlight the impact of important features on subsequent models. The Google Mind team has also achieved significant results combining attention mechanisms with RNNs for image classification. Hu et al. introduced a Squeeze-and-Excitation Block (SE) to focus on the relationships between channels and applied it to image tasks, improving accuracy, but it cannot fully utilize global context and only focuses on channel information. Bahdanau et al. applied attention mechanisms to natural language processing, greatly improving translation accuracy and leading to the continuous development and application of attention mechanisms in various fields.

[0005] In summary, the existing single-network learning capabilities are insufficient, failing to accurately and completely recognize and enhance speech information, thus limiting the quality of speech recognition. Summary of the Invention

[0006] This invention overcomes the shortcomings of the existing technology, and the technical problem to be solved is: to provide a primary and secondary network speech enhancement system that integrates attention mechanisms to improve the quality and intelligibility of speech.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a master-slave network speech enhancement system that integrates attention mechanisms, comprising:

[0008] Main network: The main network includes a first feature extraction module, a second feature extraction module, and a first concatenation module. The first feature extraction module is used to extract time-domain features and frequency features from the speech input signal, and then perform convolution processing and feature mapping on the time-domain features and frequency features, as well as fuse the CBAM attention mechanism to obtain the first feature.

[0009] The second feature extraction module is used to process the speech input signal through a three-layer bidirectional gated recurrent unit (BiGRU) and a multi-head attention mechanism to obtain the second feature;

[0010] The first splicing unit is used to splice the first feature and the second feature to obtain the output feature of the main network;

[0011] Auxiliary Network: The auxiliary network includes a second splicing module and a first CNN network. The second splicing module is used to splice the output features of the main network and the speech input signal to obtain a third feature, which is then sent to the first CNN network. The first CNN network is used to calculate the output features of the auxiliary network based on the third feature.

[0012] Speech reconstruction module: This module adds the output features of the main network and the output features of the auxiliary network and reconstructs the speech to obtain the enhanced speech.

[0013] The first feature extraction module includes:

[0014] Frequency feature extraction module: used to extract the time-domain features of the speech input signal;

[0015] Temporal feature extraction module: used to extract frequency domain features of the speech input signal;

[0016] The second CNN network is used to perform convolution processing on time-domain features and frequency-domain features respectively;

[0017] Note the CBAM mechanism: used to map the time-domain and frequency-domain features after convolution to obtain the first feature.

[0018] The loss function used in the main network for:

[0019] ;

[0020] Where N is the number of frames in the time-frequency signal, and k is the number of frequency bands in the time-frequency signal. S ( n, k ) represents the clean speech signal characteristics of each frequency band in the nth frame. Ŝ ( n, k ) represents the enhanced features of each frequency band in the nth frame. T ( n, k ) represents the target magnification factor. It is a constant.

[0021] Target magnification factor T ( n, k The formula for calculating ) is:

[0022] ;

[0023] in, θ and α These are all parameters that control the degree of exaggeration. S max ( k )and S min ( k ) indicates the first kThe maximum and minimum values ​​of clean speech features at each frequency point throughout the entire training set.

[0024] The θ The value of is 3.

[0025] The second CNN network includes two convolutional layers and one pooling layer. The convolutional kernels of the two convolutional layers are 3×7 and 7×3, respectively.

[0026] The first CNN network consists of three convolutional layers.

[0027] During training, the auxiliary network uses the difference between the output signal of the main network and the target training signal as the learning target.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. This invention provides a master-slave network speech enhancement system that integrates attention mechanisms, comprising a master network and a slave network. The master network uses a bidirectional gated recurrent unit (BiGRU) to improve the CNN, and the feature information between the two can be complementary to extract global features and details of speech features. The CNN and BiGRU are fused, and different types of attention mechanisms are introduced to improve the network's learning ability.

[0030] 2. Commonly used MSE loss functions often result in overly smoothed speech reconstructions, leading to information loss in high-frequency components. To better suppress noise and improve speech quality and intelligibility, this invention employs the RMSE loss function combined with a target exaggeration method, constructing a novel loss function, RTMSE, to optimize the main network model. Furthermore, an auxiliary network is used to train the residual between the main network output and the learning target. By training the residual between the main network output and the learning target and then using the output to compensate for information loss during the main network's learning process, this invention improves speech reconstruction quality and intelligibility. Attached Figure Description

[0031] Figure 1 The present invention provides a schematic diagram of the structure of a master-slave network speech enhancement system that incorporates an attention mechanism, as provided in this embodiment of the invention.

[0032] Figure 2 This is a schematic diagram of the main network structure in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the auxiliary network structure in an embodiment of the present invention;

[0034] Figure 4 For different White noise θ Comparison of PESQ and STOI values ​​under three signal-to-noise ratios;

[0035] Figure 5 A comparison of waveforms for different algorithms;

[0036] Figure 6 A comparison of spectrograms from different algorithms. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, this embodiment of the invention provides a primary and secondary network speech enhancement system that integrates an attention mechanism, comprising:

[0039] Main Network: The main network includes a first feature extraction module, a second feature extraction module, and a first concatenation module. The first feature extraction module is used to extract time-domain features and frequency features from the speech input signal, and then perform convolution and feature mapping processing on the time-domain features and frequency features, and fuse the CBAM attention mechanism to obtain the first feature. The second feature extraction module is used to process the speech input signal through a three-layer bidirectional gated recurrent unit (BiGRU) and a multi-head attention mechanism to obtain the second feature. The first concatenation unit is used to concatenate the first feature and the second feature to obtain the main network output feature.

[0040] Auxiliary Network: The auxiliary network includes a second splicing module and a first CNN network. The second splicing module is used to splice the output features of the main network and the speech input signal to obtain a third feature, which is then sent to the first CNN network. The first CNN network is used to calculate the output features of the auxiliary network based on the third feature.

[0041] Speech reconstruction module: This module adds the output features of the main network and the output features of the auxiliary network and reconstructs the speech to obtain the enhanced speech.

[0042] like Figure 2 The diagram shown is a schematic representation of the main network structure in this embodiment. The first feature extraction module includes:

[0043] Frequency feature extraction module: used to extract the time-domain features of the speech input signal;

[0044] Temporal feature extraction module: used to extract frequency domain features of the speech input signal;

[0045] The second CNN network is used to perform convolution processing on time-domain features and frequency-domain features respectively;

[0046] Note the CBAM mechanism: used to map the time-domain and frequency-domain features after convolution to obtain the first feature.

[0047] Specifically, such as Figure 2 The second CNN network includes two convolutional layers and one pooling layer, with the convolutional kernels of the two convolutional layers being 3×7 and 7×3, respectively.

[0048] Specifically, in this embodiment, the loss function used in the main network is... for:

[0049] (1)

[0050] Where N is the number of frames in the time-frequency signal, and k is the number of frequency bands in the time-frequency signal. S ( n, k ) represents the clean speech signal characteristics of each frequency band in the nth frame. Ŝ ( n, k ) represents the enhanced features of each frequency band in the nth frame. T ( n, k ) represents the target magnification factor. It is a constant.

[0051] Among them, the target magnification factor T ( n, k The formula for calculating ) is:

[0052] (2)

[0053] in, θ and α These are all parameters that control the degree of exaggeration. S max ( k )and S min ( k ) indicates the first k The maximum and minimum values ​​of clean speech features at each frequency point throughout the entire training set. θ The value of is 3.

[0054] like Figure 3 The diagram shown illustrates the structure of the auxiliary network in this embodiment. The first CNN network includes three convolutional layers. During training, the auxiliary network uses the difference between the output signal of the main network and the target training signal as the training objective.

[0055] Specifically, in the speech enhancement system of this embodiment, the adjustment and training process of the main network and the auxiliary network can be divided into two stages: training and testing. In the training stage, the amplitude spectrum of the noisy speech y(t) and clean speech s(t) in the training set is calculated using STFT transformation. Then, the obtained noisy and clean speech features are input into the main network for training. During the optimization of the network model, the maximum and minimum values ​​of the clean speech features in the entire training set are used to calculate the exponential target exaggeration factor. Then, the output of the main network is concatenated with the clean speech and input into the auxiliary network. The difference between the clean speech and the output prediction is used as the learning objective to train the auxiliary network. In the speech enhancement testing stage, the speech features of the test set speech are extracted and input into the trained optimal network model to estimate the speech features. After the main network and auxiliary network are tested, during speech reconstruction, the output of the auxiliary network is added to the output of the main network, and the enhanced speech features are combined with the phase spectrum of the noisy speech to reconstruct the enhanced speech.

[0056] In this embodiment of the invention, speech signals are processed. CNN networks consider local region information, and can only process one local region at a time, thus all operations are local. General squared convolutional kernels (such as 3×3 kernels) cannot effectively utilize the time-frequency domain features of speech. Therefore, in the second CNN network of the main network in this invention, both sets of convolutional kernels are rectangular (3×7, 7×3), which can capture time-domain and frequency-domain contextual information from the input spectrogram, respectively, obtaining more feature information. The padding mode of the convolutional layers is set to "same"; therefore, the size of the feature maps does not change after convolution. After convolution in the time and frequency domains, the two feature maps have the same size. These two types of features are concatenated, and this method fuses the features extracted by convolutional kernels of different shapes. A CBAM attention mechanism is added after the last convolutional layer of the second CNN network. By focusing on channel and spatial features, its purpose is to assign a weight to each feature map and reduce the weights of feature maps that do not contribute much to separation or are redundant. It also highlights useful features. A GAP layer is added at the last layer, directly achieving dimensionality reduction and greatly reducing the network parameters.

[0057] Furthermore, the second feature extraction unit in the main network uses three layers of BiGRU and one fully connected (FC) layer. BiGRU can better utilize the contextual information of speech, and FC is used to map the BiGRU output to the desired dimension. A multi-head attention mechanism is added to the last BiGRU layer. The outputs of the two parts of the network, the first feature extraction unit and the second feature extraction unit, are concatenated to form the output of the main network.

[0058] The output of the main network is concatenated with the clean speech and input into the auxiliary network. The difference between the clean speech and the output prediction is used as the learning target for training. The auxiliary network consists of three convolutional layers, flattened layers and fully connected layers. The training result is added to the output of the main network to compensate for the information lost by the main network during training, and the final output result is obtained. The waveform is then reconstructed using the final output result to obtain the enhanced speech.

[0059] The experiment used the IEEE speech database, selecting 610 clean speech samples as the training set and 110 samples as the test set. The NOISE-92 noise database was used to select Babble, Buccaneer2, Destroyerops, Destroyerengine, and Factory1 noise samples, as well as F16, Pink, White, and Factory2 noise samples as the test set. These data were mixed at signal-to-noise ratios of 0dB, 5dB, and 10dB. The sampling frequency was 16kHz. The effectiveness of the proposed algorithm was verified using two evaluation metrics: Short-Time Target Proof (STOI) and Perceptual Speech Quality Assessment (PESQ).

[0060] Experimental Results and Analysis

[0061] Experiment 1: Parameters controlling the degree of exaggeration in the loss function θ Selecting factors that affect the experimental results

[0062] White noise was used for comparison at 0dB, 5dB, and 10dB. θ The effect of the selected value on the PESQ and STOI values. The results are as follows: Figure 4 As shown, through Figure 4 The data changes show that θ When the values ​​are taken between [0,3], all evaluation indicators show an upward trend. θ When the values ​​are between [3, 5], all evaluation indicators show a downward trend. Therefore, θ When the value is around 3, the model performance can be presented at a good level.

[0063] Experiment 2: Performance Comparison of Different Algorithms

[0064] Algorithm 1: CNN-based speech enhancement algorithm;

[0065] Algorithm 2: A speech enhancement algorithm using an improved parallel network of CNN and BiGRU;

[0066] Algorithm 3: An improved speech enhancement algorithm based on CNN+CBAM+BiGRU+Multi-Head Attention;

[0067] Algorithm 4: A speech enhancement algorithm based on Algorithm 3 with the TRMSE function;

[0068] Algorithm 5: Speech enhancement algorithm based on Algorithm 4 with fused auxiliary network (i.e., the algorithm in the embodiment of this invention).

[0069] Tables 1 and 2 show the comparison results of PESQ and STOI for various algorithms under different noise levels.

[0070] Table 1. Comparison of PESQ results for various algorithms under different noise levels.

[0071]

[0072] Table 2. Comparison of STOI results for various algorithms under different noise levels.

[0073]

[0074] Analyze the experimental results based on the data in the table:

[0075] (1) The comparison results of Algorithm 1 and Algorithm 2 show that the improved CNN fusion with BiGRU is better than the traditional CNN alone. The average PESQ value is increased by about 0.142 and the average STOI value is increased by about 0.022.

[0076] (2) Compared with Algorithm 3, Algorithm 2 demonstrates that the attention mechanism plays an important role in network optimization. The average PESQ value is improved by about 0.142, and the average STOI value is improved by about 0.022.

[0077] (3) Compared with the results of Algorithm 3, Algorithm 4 improved the PESQ value by an average of about 0.048 and the STOI value by an average of about 0.013, proving that the new loss function can improve speech quality and intelligibility.

[0078] (4) Algorithm 5 proves that the proposed training of residuals plays an important role in improving the performance of speech enhancement models. Compared with Experiment 1, the PESQ value is improved by an average of about 0.42 and the STOI value is improved by an average of about 0.065.

[0079] Under the above experimental conditions, waveforms and spectrograms were plotted with a signal-to-noise ratio of 5dB and Factory2 as the background noise. Figure 5 and Figure 6 The comparison shows that Algorithm 5 achieves the best enhancement effect, with more complete restored speech information, demonstrating the advantages of the speech enhancement system in speech enhancement according to the present invention.

[0080] This invention addresses the insufficient learning capacity of single-network systems by providing a master-slave network speech enhancement system that incorporates an attention mechanism. Speech signals exhibit complex temporal correlations, and speech signals from different time periods may possess semantic correlations. While convolutional operations excel at focusing on local details of the input speech signal, their receptive field is very limited, failing to capture long-range features. Therefore, multiple layers are needed to learn the contextual dependencies of the speech signal. However, as the number of layers increases, redundant information is generated within the network, hindering learning as it is passed layer by layer. This invention's master network employs a gated recurrent unit (GRU) to overcome this shortcoming. It not only considers the input from the previous time step but also endows the network with a memory function for previous content. When predicting information at the current time step, it considers not only the current input but also information output from previous time steps, making it more suitable for processing time-related information. However, its feature learning capability is insufficient; it can only learn temporal features in a single direction within the sample, failing to fully consider the combined influence of preceding and following features on the current state. Therefore, this invention improves the CNN by using a bidirectional gated recurrent unit (BiGRU) in the main network. The feature information between the two can be complementary to extract global features and details of speech features. The CNN and BiGRU are fused, and different types of attention mechanisms are introduced to improve the network's learning ability. The commonly used MSE loss function reconstructs speech that is too smooth, resulting in information loss in the high-frequency range. To better suppress noise and improve speech quality and intelligibility, this invention uses the RMSE loss function combined with a target exaggeration method to construct a new loss function, RTMSE, to optimize the model. An auxiliary network is used to train the residual between the main network output and the learning target. By training the residual between the main network output and the learning target and then using the output to compensate for information loss during the main network's learning process, the quality of speech enhancement is improved.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A primary-secondary network speech enhancement system with fused attention mechanism, characterized in that, Comprise: The main network comprises a first feature extraction module, a second feature extraction module, a first splicing module, the first feature extraction module is used for extracting time domain features and frequency features of a voice input signal, and then performing convolution processing and feature mapping on the time domain features and the frequency features and processing by a CBAM attention mechanism to obtain first features; The second feature extraction module is used for processing a voice input signal by a three-layer bidirectional gate recurrent unit (BiGRU) and a multi-head attention mechanism to obtain second features; The first splicing module is used for splicing the first features and the second features to obtain main network output features; The auxiliary network comprises a second splicing module and a first CNN network, the second splicing module is used for splicing the main network output features and the voice input signal to obtain third features and then sending the third features to the first CNN network, and the first CNN network is used for calculating the third features to obtain auxiliary network output features; The voice reconstruction module is used for adding the main network output features and the auxiliary network output features and reconstructing to obtain enhanced voice; In the main network, the loss function adopted is is: ； where N is the number of frames of the time-frequency signal, k is the number of frequency bands of the time-frequency signal, S n, k represents the clean speech signal feature of the nth frame of the kth frequency band, Ŝ n, k represents the enhanced feature of the nth frame of the kth frequency band, T n, k represents the target amplification factor, is a constant;​​​ Target magnification factor T n, k The formula for calculating the target magnification factor (M) is:​ ； wherein, θ and α are both parameters to control the exaggeration degree, S max ( k ) and S min ( k ) represent the maximum and minimum values of the clean speech features at the first k frequency point in the entire training set.

2. The primary-secondary network speech enhancement system with fusion attention mechanism according to claim 1, characterized in that, The first feature extraction module comprises: A frequency feature extraction module is used for extracting time domain features of a voice input signal; A time domain feature extraction module is used for extracting frequency domain features of a voice input signal; A second CNN network is used for performing convolution processing on the time domain features and the frequency domain features respectively; An attention mechanism CBAM is used for mapping the convolution-processed time domain features and the convolution-processed frequency domain features to obtain first features. 3.The primary-secondary network speech enhancement system with fusion attention mechanism according to claim 1, wherein, the θ value of 3.

4. The primary-secondary network speech enhancement system with fusion attention mechanism according to claim 2, wherein, The second CNN network comprises two convolution layers and one pooling layer, and the convolution kernels of the two convolution layers are 3×7 and 7×3 respectively.

5. The primary-secondary network speech enhancement system with fusion attention mechanism according to claim 2, wherein, The first CNN network comprises three convolution layers.

6. The primary-secondary network speech enhancement system with fusion attention mechanism according to claim 2, wherein, During training, the auxiliary network takes the difference between the output signal of the main network and the target training signal as the learning target.

Images