Ultra-lightweight speech enhancement neural network system and method based on adaptive convolution

Abstract

The application discloses an ultra-light speech enhancement neural network system and method based on adaptive convolution. The system comprises an encoder: a spectrum compression module, which is used for compressing the dynamic range and frequency dimension of the noisy speech spectrum; an adaptive convolution attention module, which is used for performing frequency dimension down-sampling and feature extraction analysis on the compressed feature map through an adaptive convolution layer and a time channel attention, and then sending the feature map into an enhancer; the enhancer: a double-path grouping recurrent neural network, which is used for modeling the sub-band time features and the intra-frame frequency features of the speech; a decoder: an adaptive transpose convolution attention module, which is used for performing frequency dimension up-sampling and feature reconstruction on the feature map output by the enhancer; and a spectrum expansion module, which is used for expanding the frequency dimension of the feature map output by the adaptive transpose convolution attention module. The application can realize high-performance speech enhancement with extremely low computational complexity.

Description

Technical Field

[0001] This invention belongs to the field of speech enhancement, specifically relating to an ultra-lightweight speech enhancement neural network system and method based on adaptive convolution. Background Technology

[0002] Speech enhancement is one of the core problems in audio signal processing. Its main purpose is to recover speech from signals contaminated by environmental noise, interfering human voices, reverberation, and echoes, thereby improving speech quality and intelligibility. It has been widely used in mobile communications, teleconferencing, Bluetooth headsets, hearing aids, and speech recognition front-ends. In recent years, deep learning models have been widely used in speech enhancement. Through a data-driven approach, deep neural networks directly learn the nonlinear mapping relationship from mixed speech to clean speech, achieving performance far exceeding traditional methods in many scenarios. However, current state-of-the-art speech enhancement neural networks suffer from excessively high memory and computational resource consumption, making them difficult to deploy on widely used edge devices. In scenarios with limited computational resources and high real-time requirements, ultra-lightweight neural networks are needed.

[0003] Currently, ultra-lightweight speech enhancement neural networks have been proposed, such as the Grouped Temporal Convolutional Recurrent Network (GTCRN), which employs a grouping strategy, sub-band feature extraction module, and temporal recurrent attention mechanism to reduce model complexity and improve performance (Rong X, Sun T, Zhang X, et al. GTCRN: A Speech Enhancement Model Requiring Ultralow Computational Resources[C] / / ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP).IEEE,2024:971-975.). However, although these models have extremely low computational complexity, there is still room for further optimization in terms of performance.

[0004] Typical designs that improve performance while maintaining low computational overhead include modern convolutional neural network designs (Liu Z, Mao H, Wu CY, et al. A convnet for the 2020s[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2022:11976-11986.) and dynamic convolution (Chen Y, Dai X, Liu M, et al. Dynamic convolution: Attention overconvolution kernels[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:11030-11039.). However, these strategies have not yet been fully applied in ultra-lightweight speech enhancement models. Among them, dynamic convolution is an advanced convolutional neural network design in the field of computer vision. It uses an attention mechanism to aggregate information from an image, obtains a set of attention scores, uses these scores to weight several candidate convolution kernels, obtains a convolution kernel independent for each image, and uses this convolution kernel to convolve the image. However, in the field of speech enhancement, performing dynamic convolution on the entire spectrogram does not satisfy causality and cannot adapt to rapidly changing speech spectral features. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes an ultra-lightweight speech enhancement neural network system and method based on adaptive convolution, which can achieve high-performance speech enhancement with extremely low computational complexity.

[0006] The technical solution adopted by the system of this invention is as follows:

[0007] An ultra-lightweight speech enhancement neural network system based on adaptive convolution is provided. The system includes an encoder, an enhancer, and a decoder. The encoder includes: a spectrum compression module for compressing the dynamic range and frequency dimension of the noisy speech spectrum; and an adaptive convolutional attention module for downsampling and feature extraction analysis of the compressed feature map in the frequency dimension through adaptive convolutional layers and temporal channel attention, and then feeding the feature map into the enhancer.

[0008] The enhancer includes a dual-path grouped recurrent neural network for modeling subband temporal features and intra-frame frequency features of the speech.

[0009] The decoder includes: an adaptive transposed convolutional attention module, which is used to upsample the feature map output by the enhancer in the frequency dimension and reconstruct the features through an adaptive transposed convolutional layer and temporal channel attention; and a spectral dilation module, which is used to dilate the frequency dimension of the feature map output by the adaptive transposed convolutional attention module.

[0010] This invention also provides a method for utilizing the above-mentioned ultra-lightweight speech enhancement neural network system based on adaptive convolution, the specific steps of which are as follows:

[0011] In the encoding stage, the input noisy signal is first transformed into a spectrum using a short-time Fourier transform, and the spectrum compression module compresses the input spectrum in terms of dynamic range and frequency dimension. Then, a two-layer adaptive convolutional attention module is used to downsample and extract features from the feature map in the frequency dimension, while mapping the feature channels to a high-dimensional space. Finally, a three-layer adaptive convolutional attention module is used to further extract and analyze features from the feature map.

[0012] During the enhancement phase, the enhancer is used to analyze and model the sub-band temporal features and intra-frame frequency features of the feature map output by the encoder;

[0013] In the decoding stage, a 3-layer adaptive transposed convolutional attention module is first used to analyze and reconstruct the feature map; then a 2-layer adaptive transposed convolutional attention module is used to upsample the feature map in the frequency dimension and further reconstruct the features; finally, a spectrum dilation module is used to dilate the feature map in the frequency dimension to obtain a spectrum mask with the same spectrum dimension as the input. The spectrum mask is multiplied with the input noisy spectrum and an inverse short-time Fourier transform is performed to obtain the enhanced speech signal.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0015] (1) This invention uses adaptive convolution to enable convolution to obtain a stronger adaptive representation of frame-level local features;

[0016] (2) This invention uses channel and kernel joint attention to enhance the network’s attention to key channels and key candidate kernels;

[0017] (3) This invention uses an organic combination of depthwise convolution and pointwise convolution to improve the feature extraction and reconstruction capabilities of the network while ensuring low computational complexity.

[0018] (4) The system of the present invention can achieve advanced speech enhancement performance with extremely low parameter and computational requirements. Attached Figure Description

[0019] Figure 1 This is a network system architecture diagram of the present invention;

[0020] Figure 2 This is a structural diagram of the adaptive convolutional attention module in an embodiment of the present invention;

[0021] Figure 3 This is a structural diagram of the adaptive transposed convolutional attention module in an embodiment of the present invention;

[0022] Figure 4 This is a structural diagram of the time-cyclic channel convolution kernel joint attention generation module in an embodiment of the present invention. Detailed Implementation

[0023] The technical solution of the present invention will be described in complete and clear form below with reference to the accompanying drawings and embodiments.

[0024] The network system architecture diagram of this invention is as follows: Figure 1 As shown, the system includes an encoder, an enhancer, and a decoder. The encoder includes: a spectral compression module for compressing the dynamic range and frequency dimension of the noisy speech spectrum; and an adaptive convolutional attention module for downsampling the frequency dimension and performing feature extraction analysis on the compressed feature map through adaptive convolutional layers and temporal channel attention, before feeding the feature map into the enhancer. The enhancer includes a dual-path grouped recurrent neural network for modeling the sub-band temporal features and intra-frame frequency features of the speech. The decoder includes: an adaptive transposed convolutional attention module for upsampling the frequency dimension and performing feature reconstruction on the feature map output by the enhancer through adaptive transposed convolutional layers and temporal channel attention; and a spectral expansion module for expanding the frequency dimension of the feature map output by the adaptive transposed convolutional attention module.

[0025] The main implementation method of the system of this invention is as follows:

[0026] In the encoding stage, the input noisy signal is first transformed into a spectrum using a short-time Fourier transform, and the spectrum compression module compresses the input spectrum in terms of dynamic range and frequency dimension. Then, a two-layer adaptive convolutional attention module is used to downsample and extract features from the feature map in the frequency dimension, and simultaneously map the feature channels to a high-dimensional space. Finally, a three-layer adaptive convolutional attention module is used to further extract and analyze features from the feature map.

[0027] During the enhancement phase, the enhancer is used to analyze and model the sub-band temporal features and intra-frame frequency features of the feature map output by the encoder;

[0028] In the decoding stage, a 3-layer adaptive transposed convolutional attention module is first used to analyze and reconstruct the feature map; then a 2-layer adaptive transposed convolutional attention module is used to upsample the feature map in the frequency dimension and further reconstruct the features; finally, a spectrum dilation module is used to dilate the feature map in the frequency dimension to obtain a spectrum mask with the same spectrum dimension as the input. The spectrum mask is multiplied with the input noisy spectrum and an inverse short-time Fourier transform is performed to obtain the enhanced speech signal.

[0029] The spectrum compression module described in this embodiment includes dynamic range compression and frequency dimension compression. Dynamic range compression uses logarithmic and exponential operations to compress the spectrum values. Frequency dimension compression keeps the frequency points within the 0-2kHz band unchanged, and uses a triangular filter bank based on an equivalent rectangular bandwidth frequency scale to compress the frequency points within the 2-8kHz band. Specifically, assuming the single-frame input spectrum is X = X... R +jX I , where X R X is the real part of the spectrum. I For the imaginary part of the spectrum, the spectrum compression module maps the input complex spectrum into three channels, each using... Indicate, then

[0030]

[0031] Among them, log 10 (·) represents logarithmic operation, and ERB[·] represents filtering operation based on a triangular filter bank with an equivalent rectangular bandwidth frequency scale. Finally, Shifting each channel one frequency point towards both low and high frequencies creates feature maps for another six channels, which are then compared with... They are spliced ​​together along the channel dimension and used as input for subsequent modules, which is used to expand the receptive field in the frequency dimension.

[0032] In this embodiment, the spectrum spreading module is implemented by interpolating the frequency dimension of the feature map using the transpose of the equivalent matrix of the triangular filter bank based on the equivalent rectangular bandwidth frequency scale.

[0033] The structure diagram of the adaptive convolutional attention module described in this embodiment is as follows: Figure 2As shown, the encoder includes a layer normalization module with affine transformation, a joint attention generation module for temporal cyclic convolutional kernels, and sequentially connected input temporal attention, adaptive depth convolutional layers, a first batch normalization, a first parameter rectified linear unit activation function, a first adaptive pointwise convolutional layer, a Gaussian error linear unit activation function, a second adaptive pointwise convolutional layer, a second batch normalization, a second parameter rectified linear unit activation function, and output temporal attention. The first two adaptive convolutional attention modules of the encoder have frequency dimension downsampling functionality, with the frequency dimension stride of the convolutional kernel in the adaptive depth convolutional layer set to 2, while the stride of the subsequent three adaptive convolutional attention modules is set to 1. The temporal recurrent convolutional kernel, along with the attention generation module, analyzes the input features to obtain the input temporal channel attention map, the output temporal channel attention map, the frame-level convolutional kernel attention map of the adaptive depthwise convolutional layer, the frame-level convolutional kernel attention map of the first adaptive pointwise convolutional layer, and the frame-level convolutional kernel attention map of the second adaptive pointwise convolutional layer. The input and output temporal channel attention maps are multiplied by the feature map, respectively. Each frame-level convolutional kernel attention map is then input into the corresponding adaptive depthwise convolutional layer and adaptive pointwise convolutional layer. The adaptive depthwise convolutional layer and adaptive pointwise convolutional layer are depthwise and pointwise convolutional layers implemented adaptively. A depthwise convolutional layer is a convolutional layer with the number of groups equal to the number of input and output channels, and a pointwise convolutional layer is a convolutional layer with a 1×1 kernel size. The adaptive depthwise convolutional layer does not change the number of channels in the feature map. The first adaptive pointwise convolutional layer maps the number of channels in the feature map from the number of input channels to the number of hidden channels, and the second adaptive pointwise convolutional layer maps the number of channels in the feature map from the number of hidden channels to the number of output channels.

[0034] The structure diagram of the adaptive transposed convolutional attention module described in this embodiment is as follows: Figure 3 As shown, its structure differs from the adaptive convolutional attention module only in the adaptive depth-transposed convolutional layer and the output activation function. The adaptive depth-transposed convolutional layer is an adaptively implemented depth-transposed convolutional layer, where the number of groups equals the number of input and output channels. Specifically, the last two layers of the decoder's adaptive transposed convolutional attention module have frequency-dimensional upsampling capabilities, with the kernel's frequency-dimensional stride set to 2. The output activation function in the last layer of the adaptive transposed convolutional attention module is a learnable sigmoid activation function, while in other layers it is a parametric rectified linear unit activation function. The formula for calculating the learnable sigmoid activation function is:

[0035] F out = 2 × sigmoid(W sigmoid *F in )

[0036] Among them, F in and F out These represent the input and output feature maps, respectively, both with a shape of B×C×T×F, where B, C, T, and F represent the batch size, number of channels, temporal dimension, and frequency domain dimension, respectively; W sigmoid The affine transformation weights are learnable and have a shape of 1×1×1×F; * indicates element-wise multiplication; sigmoid(·) indicates the regular sigmoid activation function.

[0037] The temporally looping channel convolution kernel combined with attention generation module described in this embodiment is as follows: Figure 4 As shown, the system consists of an energy-average pooling layer along the frequency dimension, a recurrent neural network, a fully connected layer, and an attention generation layer connected sequentially. The recurrent neural network performs temporal analysis and modeling on the feature maps from the energy-average pooling layer. The attention generation layer first divides the output features of the fully connected layer into five parts along the channel dimension. The first two parts use the sigmoid activation function to obtain the input and output temporal channel attention maps, while the last three parts use a softmax activation function with a temperature coefficient to obtain the frame-level convolutional kernel attention maps for the adaptive depthwise convolutional layer, the first adaptive pointwise convolutional layer, and the second adaptive pointwise convolutional layer. The softmax activation function with a temperature coefficient divides the input features by a temperature coefficient during neural network training. This temperature coefficient changes with training annealing to ensure the efficiency of training optimization for all candidate convolutional kernels; however, during testing, this division by the temperature coefficient is not required.

[0038] In this embodiment, the adaptive depthwise convolutional layer, adaptive depthwise transposed convolutional layer, first adaptive pointwise convolutional layer, and second adaptive pointwise convolutional layer all belong to adaptive convolution, which is a causal, frame-by-frame dynamic convolution with adaptive capabilities to the spectral features of each frame. Specifically, the adaptive depthwise convolutional layer, adaptive depthwise transposed convolutional layer, first adaptive pointwise convolutional layer, and second adaptive pointwise convolutional layer in different modules each receive corresponding frame-level convolutional kernel attention maps as input, and use these attention maps to perform a weighted summation of the candidate convolutional kernels of the corresponding layers to obtain the frame-level adaptive convolutional kernel W. L (t), that is

[0039]

[0040] Where L represents the layer to which the adaptive convolutional kernel belongs; W i L ,i=0,1,…K-1 represents the i-th candidate convolutional kernel of this layer. The candidate convolutional kernel is a fixed set of learnable parameters for this layer. There are K such kernels in each layer. The shape of each candidate convolutional kernel is the same as the shape of the convolutional kernel of the corresponding regular convolutional layer. This represents the frame-level convolutional kernel attention map of the i-th candidate convolutional kernel in the input of this layer; then, the adaptive convolutional kernel W is used. i L (t) Perform convolution or transpose convolution operations on the feature map input to this layer.

[0041] The enhancer described in this embodiment includes two cascaded dual-path grouped recurrent neural networks. Each dual-path grouped recurrent neural network is composed of an intra-frame grouped recurrent neural network for modeling intra-frame frequency features of speech and an inter-frame grouped recurrent neural network for modeling sub-band temporal features of speech, which are connected in sequence.

[0042] The specific settings in this embodiment are as follows: the speech enhancement object is audio data with a sampling rate of 16kHz; the frame length and frame shift of the short-time Fourier transform are set to 512 (32ms) and 256 (16ms) respectively; the number of Fourier transform points is 512; and the window function is a square root Hanning window. The spectrum compression module keeps 65 frequency bands in the 0-2kHz range unchanged and maps 192 frequency bands in the 2-8kHz range into 64 frequency bands. The first two layers of the encoder's adaptive convolutional attention module have the following hyperparameters: input channels [9, 16, 16, (1, 5), (1, 2)] and kernel size [16, 16, 16, (1, 5), (1, 2)], stride [16, 16, 16, (1, 5), (1, 2)]. The kernel size and stride correspond to the time and frequency dimensions, respectively. The last three layers of the encoder's adaptive convolutional attention module and the first three layers of the decoder's adaptive transposed convolutional attention module have the following hyperparameters: [16, 16, 16, (3, 3), (1, 1)]. The last two layers of the decoder's adaptive transposed convolutional attention module have the following hyperparameters: [16, 16, 16, (1, 5), (1, 2)] and [16, 1, 4, (1, 5), (1, 2)]. The number of candidate convolutional kernels in all adaptive convolutional layers is set to 8. The recurrent neural network (RNN) of all time-recurrent channel convolutional kernels and the attention generation module has 32 hidden units. The inter-frame grouping RNN of the two dual-path grouping RNN modules has 16 hidden units, and the intra-frame grouping RNN has 8 hidden units, with 2 groups in each module. With these parameter settings, the model has 135K parameters and 38 MMACs per second. During training, the temperature coefficient of the softmax activation function with a temperature coefficient gradually decays exponentially from 30 to 1, with a decay exponent of 0.9. The calculation formula is as follows:

[0043] τ(e) = 30 × 0.9 e

[0044] Where τ(e) is the temperature coefficient of the e-th training epoch.

[0045] To illustrate the effectiveness of this invention, approximately 200 hours of speech and noise data from the 5th Deep Noise Suppression Challenge (DNS5) dataset were synthesized for experiments. The signal-to-noise ratio was set to -5 to 15 dB, and all noisy and clean speech data were downsampled to 16 kHz. The GTCRN model (Rong X, Sun T, Zhang X, et al. GTCRN: ASpeech Enhancement Model Requiring Ultralow Computational Resources [C] / / ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024: 971-975.) was selected as a baseline model for comparison. Evaluation metrics for speech enhancement performance include Scale-Invariant Signal-to-Noise Ratio (SISNR), Perceptual Evaluation of Speech Quality (PESQ), Extended Short-Time Objective Intelligence (ESTOI), and Deep Noise Suppression Mean Opinion Score (DNSMOS), which includes Signal Preservation Quality (SIG), Noise Suppression Quality (BAK), and Overall Quality (OVRL). Higher values ​​for these metrics are generally better. Comparative results on the test set are shown in Table 1, where computational complexity is measured using Multiply-Accumulate Operations (MACs) per second.

[0046] Table 1 Comparison Results

[0047]

[0048] The results in Table 1 demonstrate that the computational complexity of the system of the present invention is comparable to that of GTCRN, while achieving significantly better performance enhancement.

Claims

1. An ultra-lightweight speech enhancement neural network system based on adaptive convolution, comprising an encoder, an enhancer, and a decoder, characterized in that, The encoder includes: a spectrum compression module for compressing the dynamic range and frequency dimension of the noisy speech spectrum; and an adaptive convolutional attention module for downsampling and feature extraction analysis of the compressed feature map in the frequency dimension through an adaptive convolutional layer and temporal channel attention, and then feeding the feature map into the enhancer. The enhancer includes a dual-path grouped recurrent neural network for modeling subband temporal features and intra-frame frequency features of the speech. The decoder includes: an adaptive transposed convolutional attention module, used to upsample the frequency dimension of the feature map output by the enhancer and reconstruct the features through an adaptive transposed convolutional layer and temporal channel attention; and a spectral dilation module, used to dilate the frequency dimension of the feature map output by the adaptive transposed convolutional attention module. The adaptive convolutional attention module includes a layer normalization module with affine transformation, a joint attention generation module for time-cyclic channel convolution kernels, and sequentially connected input time-channel attention, an adaptive depth convolutional layer, a first batch normalization, a first parameter rectified linear unit activation function, a first adaptive pointwise convolutional layer, a Gaussian error linear unit activation function, a second adaptive pointwise convolutional layer, a second batch normalization, a second parameter rectified linear unit activation function, and an output time-channel attention.

2. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 1, characterized in that, The temporal recurrent channel convolutional kernel joint attention generation module consists of an energy average pooling layer along the frequency dimension, a recurrent neural network, a fully connected layer, and an attention generation layer connected sequentially. The recurrent neural network performs temporal analysis and modeling on the feature map of the energy average pooling layer. The attention generation layer first divides the output feature of the fully connected layer into 5 parts along the channel dimension. The first two parts use the sigmoid activation function to obtain the temporal channel attention map of the input and output. The last three parts use the softmax activation function with a temperature coefficient to obtain the frame-level convolutional kernel attention map of the adaptive depth convolutional layer, the first adaptive pointwise convolutional layer, and the second adaptive pointwise convolutional layer.

3. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 2, characterized in that, The softmax activation function with a temperature coefficient divides the input features by a temperature coefficient during the training process of the neural network. This temperature coefficient changes with training annealing to ensure the training optimization efficiency of all candidate convolutional kernels.

4. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 1, characterized in that, The adaptive convolutional layer and the adaptive transposed convolutional layer each accept the corresponding frame-level convolutional kernel attention map input. The candidate convolutional kernels are weighted and summed using the frame-level convolutional kernel attention map to obtain the frame-level adaptive convolutional kernel, so that the convolutional kernel has a stronger adaptive representation ability for frame-level local features. Then, the adaptive convolutional kernel is used to perform convolution operation or transposed convolution operation on the feature map.

5. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 1, characterized in that, The adaptive transposed convolutional attention module includes a layer normalization module with affine transformation, a joint attention generation module for temporal cyclic channel convolution kernels, and sequentially connected input temporal channel attention, an adaptive depth transposed convolutional layer, a first batch normalization, a parameter rectified linear unit activation function, a first adaptive pointwise convolutional layer, a Gaussian error linear unit activation function, a second adaptive pointwise convolutional layer, a second batch normalization, an output activation function, and an output temporal channel attention.

6. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 5, characterized in that, The output activation function is a learnable sigmoid activation function in the last layer and a parametric rectified linear unit activation function in other layers.

7. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 1, characterized in that, The spectrum compression module includes dynamic range compression and frequency dimension compression. Dynamic range compression uses logarithmic and exponential operations to compress the spectrum values. Frequency dimension compression keeps the frequency points in the 0-2kHz band unchanged, and uses a triangular filter bank based on the equivalent rectangular bandwidth frequency scale to compress the frequency points in the 2-8kHz band.

8. The ultra-lightweight speech enhancement neural network system based on adaptive convolution according to claim 1, characterized in that, The enhancer includes two cascaded dual-path grouped recurrent neural networks, each consisting of an intra-frame grouped recurrent neural network for modeling intra-frame frequency features of speech and an inter-frame grouped recurrent neural network for modeling sub-band temporal features of speech, connected in sequence.

9. The method using the ultra-lightweight speech enhancement neural network system based on adaptive convolution as described in claim 1, characterized in that, The specific steps of this method are as follows: In the encoding stage, the input noisy signal is first transformed into a spectrum using a short-time Fourier transform, and the spectrum compression module compresses the input spectrum in terms of dynamic range and frequency dimension. Then, a two-layer adaptive convolutional attention module is used to downsample and extract features from the feature map in the frequency dimension, while mapping the feature channels to a high-dimensional space. Finally, a three-layer adaptive convolutional attention module is used to further extract and analyze features from the feature map. During the enhancement phase, the enhancer is used to analyze and model the sub-band temporal features and intra-frame frequency features of the feature map output by the encoder; In the decoding stage, a 3-layer adaptive transposed convolutional attention module is first used to analyze and reconstruct the feature map; then a 2-layer adaptive transposed convolutional attention module is used to upsample the feature map in the frequency dimension and further reconstruct the features; finally, a spectrum dilation module is used to dilate the feature map in the frequency dimension to obtain a spectrum mask with the same spectrum dimension as the input. The spectrum mask is multiplied with the input noisy spectrum and an inverse short-time Fourier transform is performed to obtain the enhanced speech signal.

Images