Far-field single-channel speech enhancement method
The far-field single-channel speech enhancement method, which utilizes the MambAttention model and supervised adversarial training, solves the problems of signal attenuation and noise reverberation in far-field speech enhancement, and achieves efficient improvement in the clarity and intelligibility of speech signals.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-03
AI Technical Summary
Far-field speech enhancement suffers severe signal attenuation as the distance between the speaker and microphone increases, and is also subject to noise and reverberation interference, making it difficult to effectively suppress and restore clear speech.
A far-field single-channel speech enhancement method based on MambAttention is adopted. The clean compression amplitude and phase spectrum are predicted by feature encoder and decoder. Combined with supervised adversarial training, a multi-resolution discriminator is used to optimize the model, so as to achieve effective suppression of noise and reverberation.
It improves the clarity and intelligibility of speech signals, effectively overcomes the problem of inconsistency between amplitude and phase compensation in traditional methods, and enhances the naturalness and fidelity of speech.
Smart Images
Figure CN121789703A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech enhancement technology, and in particular to a far-field single-channel speech enhancement method. Background Technology
[0002] Currently, speech enhancement refers to eliminating or suppressing interference components in noisy speech signals and recovering the original clean speech signal from the noisy speech. Its purpose is to improve speech intelligibility and perceptual quality. Far-field speech enhancement refers to achieving speech enhancement in scenarios where the speaker is more than 1 meter away from the microphone. As the distance between the speaker and microphone increases, the difficulty of speech enhancement also increases sharply. The main challenge of far-field speech enhancement is the rapid attenuation of the signal, while also facing noise and reverberation interference that is the same as or even more severe than in near-field scenarios. However, many researchers are still dedicated to achieving breakthroughs in far-field speech enhancement even in these extremely challenging scenarios. Summary of the Invention
[0003] The purpose of this invention is to overcome the aforementioned technical problems and provide a far-field single-channel speech enhancement method. This far-field single-channel speech enhancement method can simultaneously and effectively suppress noise and reverberation while completely preserving the speech signal, thereby improving the clarity and intelligibility of the speech signal. To achieve the above objective, this invention provides a far-field single-channel speech enhancement method, which includes:
[0004] Step 1: Process the far-field speech signal to obtain the complex spectrum of the noisy speech signal;
[0005] Step 2: Input the compressed complex spectrum into the feature encoder, whose output is processed by N MambAttention blocks, and then fed into the amplitude mask decoder and phase decoder to predict the clean compressed amplitude mask and phase spectrum respectively.
[0006] Step 3: First, warm up and train the MambAttention model, then use the MambAttention model as the generator and the multi-resolution discriminator as the discriminator to perform supervised adversarial training.
[0007] Step 4: Use the supervised adversarial training model with the test speech input to achieve far-field single-channel speech enhancement.
[0008] In some embodiments, the complex spectrum is obtained by short-time Fourier transform of the far-field noisy speech time-domain signal.
[0009] In some embodiments, the feature encoder comprises two convolutional blocks and an extended DenseNet, each convolutional block consisting of a 2D convolutional layer, instance normalization, and PreLU activation; the magnitude mask decoder and the phase decoder both comprise an extended DenseNet, 2D transposed convolution, instance normalization, and PreLU activation; the magnitude mask decoder is followed by a deconvolutional block and a learnable sigmoid function after activation, while the phase decoder is followed by two parallel 2D convolutional layers and a two-parameter arctangent function after activation.
[0010] In some embodiments, the expanded DenseNet contains four convolutional layers with dilation rates of 1, 2, 4, and 8, respectively. The expanded convolutions increase the receptive field by introducing interval sampling. Furthermore, all convolutional layers are interconnected through dense connections to avoid gradient vanishing during training.
[0011] In some embodiments, the MambAttention block processing flow includes shaping the input B×C×T×F into BF×T×C, where B is the batch, C is the number of channels, T is the number of time frames, and F is the number of frequency points, layer normalization, time-domain multi-head attention block, bidirectional time-domain Mamba block, shaping the previous step output into BT×F×C, layer normalization, frequency-domain multi-head attention block, bidirectional frequency-domain Mamba block, and finally shaping the previous step output into B×C×T×F.
[0012] In some embodiments, the temporal multi-head attention block maps the input sequence into a query, key, and value matrix through a multi-head attention mechanism and calculates attention weights, which are responsible for capturing long-term dependencies in the time dimension; the bidirectional temporal Mamba block is based on a state-space model and processes the sequence in a bidirectional manner in the time domain, efficiently modeling long-distance temporal dependencies and capturing causal and non-causal relationships in the time series through a state transition mechanism.
[0013] In some embodiments, the frequency domain multi-head attention block is symmetrical to the time domain multi-head attention block, focusing only on feature interactions in the frequency dimension; the bidirectional frequency domain Mamba block is symmetrical to the bidirectional time domain Mamba block, applying bidirectional Mamba processing only in the frequency dimension.
[0014] In some embodiments, the preheating training of the MambAttention model is performed on the DNS Challenge 3 dataset; the noisy speech in the dataset is clean speech convolved with random RIR and mixed with noise, so that its SNR ranges from -5dB to 15dB; the target speech is direct sound with the first 100ms of reverberation preserved in order to reduce the trailing effect and enhance the auditory experience.
[0015] In some embodiments, the multi-resolution discriminator comprises three different modules sharing the same network architecture, which consists of three sets of short-time Fourier transforms (STFTs) with different parameters, followed by six sets of 2D convolutional layers with Leaky ReLU activation functions, and then a 2D convolutional layer with a Sigmoid activation function.
[0016] The present invention provides a far-field single-channel speech enhancement method based on MFSE (MambAttention-based Far-field Speech Enhancement), comprising: Step 1: Processing the far-field speech signal to obtain a complex spectrum of the noisy speech signal; Step 2: Inputting the compressed complex spectrum into a feature encoder, the output of which is processed by N MambAttention blocks and then fed into an amplitude mask decoder and a phase decoder to predict a clean compressed amplitude mask and phase spectrum, respectively; Step 3: Preheating and training the MambAttention model, then using the MambAttention model as a generator and a multi-resolution discriminator as a discriminator for supervised adversarial training; Step 4: Inputting test speech into the trained model to achieve far-field single-channel speech enhancement.
[0017] The beneficial effects of this invention are as follows: In the above-mentioned solution of this invention, the MambAttention model used as the generator combines the linear complexity of the Mamba module with a multi-head attention mechanism. Through weight sharing and block processing strategies, it significantly reduces computational complexity and memory usage while maintaining high performance. At the same time, under the structure of amplitude and phase parallel decoding, it effectively overcomes the problems of inconsistent amplitude and phase compensation, overfitting to high reverberation and noise in traditional speech enhancement methods. By jointly optimizing the amplitude spectrum and phase spectrum, it improves the naturalness and intelligibility of speech. Subsequently, the MambAttention model is combined with a supervised adversarial training strategy. Through a multi-resolution discriminator and weight sharing mechanism, it effectively captures the time-frequency features of the speech signal, enhancing the fidelity and clarity of speech. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart of a far-field single-channel speech enhancement method provided in an embodiment of the present invention;
[0020] Figure 2This is a diagram of the MambAttention model architecture corresponding to the far-field single-channel speech enhancement method provided in this embodiment of the invention.
[0021] Figure 3 A schematic diagram of the supervised adversarial training framework corresponding to the far-field single-channel speech enhancement method provided in the embodiments of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] Figure 1 A flowchart illustrating a far-field single-channel speech enhancement method provided in an embodiment of the present invention. Figure 1 As shown, the present invention provides a far-field single-channel speech enhancement method, the far-field single-channel speech enhancement method comprising:
[0024] Step 1: Process the far-field speech signal to obtain the complex spectrum of the noisy speech signal;
[0025] Furthermore, the complex spectrum is obtained by short-time Fourier transform of the far-field noisy speech time-domain signal;
[0026] Step 2: Input the compressed complex spectrum into the feature encoder, whose output is processed by N MambAttention blocks, and then fed into the amplitude mask decoder and phase decoder to predict the clean compressed amplitude mask and phase spectrum respectively;
[0027] Furthermore, the feature encoder comprises two convolutional blocks and an expanded DenseNet, each convolutional block consisting of a 2D convolutional layer, instance normalization, and PreLU activation; the magnitude mask decoder and phase decoder both comprise an expanded DenseNet, 2D transposed convolution, instance normalization, and PreLU activation, the magnitude mask decoder is followed by a deconvolutional block and a learnable sigmoid function after activation, while the phase decoder is followed by two parallel 2D convolutional layers and a two-parameter arctangent function after activation;
[0028] Furthermore, the expanded DenseNet contains four convolutional layers with dilation rates of 1, 2, 4, and 8, respectively. The expanded convolutions increase the receptive field by introducing interval sampling. In addition, all convolutional layers are interconnected through dense connections to avoid gradient vanishing during training.
[0029] Furthermore, the MambAttention block processing flow includes shaping the input B×C×T×F into BF×T×C, where B is the batch, C is the number of channels, T is the number of time frames, and F is the number of frequency points, layer normalization, time-domain multi-head attention block, bidirectional time-domain Mamba block, shaping the previous step output into BT×F×C, layer normalization, frequency-domain multi-head attention block, bidirectional frequency-domain Mamba block, and finally shaping the previous step output into B×C×T×F;
[0030] Furthermore, the temporal multi-head attention block maps the input sequence into a query, key, and value matrix through a multi-head attention mechanism and calculates attention weights, which are responsible for capturing long-term dependencies in the time dimension; the bidirectional temporal Mamba block is based on a state-space model and processes the sequence in a bidirectional manner in the time domain, efficiently modeling long-distance temporal dependencies and capturing causal and non-causal relationships in the time series through a state transition mechanism.
[0031] Furthermore, the frequency domain multi-head attention block is symmetrical to the time domain multi-head attention block, focusing only on the feature interaction in the frequency dimension; the bidirectional frequency domain Mamba block is symmetrical to the bidirectional time domain Mamba block, applying bidirectional Mamba processing only in the frequency dimension; Step 3: First, preheat and train the MambAttention model, then use the MambAttention model as a generator and the multi-resolution discriminator as a discriminator to perform supervised adversarial training.
[0032] Furthermore, the preheating training of the MambAttention model is conducted on the DNS Challenge 3 dataset; the noisy speech in the dataset is clean speech convolved with random RIR and mixed with noise, so that its SNR range is -5dB to 15dB; the target speech is the direct sound with the first 100ms of reverberation preserved, in order to reduce the trailing effect and enhance the auditory experience.
[0033] Furthermore, the multi-resolution discriminator comprises three different modules that share the same network architecture, which consists of three sets of short-time Fourier transforms (STFTs) with different parameters, followed by six sets of 2D convolutional layers with Leaky ReLU activation function, and then a 2D convolutional layer with Sigmoid activation function.
[0034] Step 4: Use the model trained with supervised adversarial input of the test speech to achieve far-field single-channel speech enhancement.
[0035] In practical implementation, this invention proposes a far-field single-channel speech enhancement method based on MFSE. It utilizes a MambAttention model as the generator and a multi-resolution discriminator as the discriminator, performing supervised adversarial training to achieve efficient and high-fidelity far-field speech enhancement. The MambAttention model architecture is as follows: Figure 2 As shown. The signal processing in the MambAttention model involves the following steps:
[0036] Step 1: Process the far-field speech time-domain signal to generate a complex spectrum of the noisy speech signal. Specifically, take 1024 sampling points as one frame of signal each time. If the last frame is shorter than 1024 points, pad it with zeros. A Hamming window is used to reduce spectral leakage. Perform STFT on each frame of signal to obtain a complex spectrum, including amplitude and phase spectra. The STFT parameters are set as follows: sampling rate 16kHz, FFT points 512, window length 400, hop size 100. The complex spectrum can be represented as shown in Equation 1. Compress the amplitude spectrum with a compression factor c of 0.3 to obtain the compressed amplitude spectrum (Y). m ) c This is to enhance the model's sensitivity to low-energy components.
[0037] Y(t,f)=Y m (t,f)·e jφ(t,f) (1)
[0038] Where Y m (t,f) is the amplitude spectrum, and φ(t,f) is the phase spectrum.
[0039] Step 2: The compressed complex spectrogram is input into the feature encoder, whose output is processed by N MambAttention blocks, and then the decoder predicts the amplitude mask and phase spectrum of the clean speech; specifically, it contains two convolutional blocks and an expanded DenseNet. Each convolutional block consists of a 2D convolutional layer, instance normalization, and a PReLU activation function, with a kernel size of 3×3, a stride of 1, and padding of 1. The expanded DenseNet contains four convolutional layers with dilation rates of 1, 2, 4, and 8, respectively. Dense connections prevent gradient vanishing and expand the receptive field. The feature encoder increases the number of input channels from 2 (amplitude and phase) to K=64 and halves the frequency dimension. The MambAttention block processing flow includes shaping the input B×C×T×F into BF×T×C, where B is the batch size, C is the number of channels, T is the number of time frames, and F is the number of frequency points; layer normalization; temporal multi-head attention block; bidirectional temporal Mamba block; shaping the previous output into BT×F×C; layer normalization; frequency domain multi-head attention block; and bidirectional frequency domain Mamba block. The MambAttention block reshapes the output of the previous step into B×C×T×F. The forward propagation of the MambAttention block is shown in equations 2, 3, 4, 5, 6, 7, and 8. The decoder includes an amplitude mask decoder and a phase decoder. The amplitude mask decoder contains an expanded DenseNet, 2D transposed convolution, instance normalization, and PReLU activation, followed by a deconvolution block (reducing the number of channels from K to 1) and a learnable sigmoid function (parameter β = 2). The phase decoder has a similar structure to the amplitude mask decoder, but uses two parallel 2D convolutional layers to predict the phase spectrum and calculates the enhanced phase spectrum using a two-parameter arctangent function.
[0040] X Time =reshape(X,[M·F,T,K]), (2)
[0041] X1 = X Time +T-MHA(LN(X Time (3)
[0042] X2 = X1 + T - Mamba(X1), (4)
[0043] X Freq. =reshape(X2,[M·T,F,K]), (5)
[0044] X3 = X Freq. +F-MHA(LN(X Freq. (6)
[0045] X4 = X3 + F - Mamba(X3), (7)
[0046] Y=reshape(X4,[M,K,T,F]), (8)
[0047] Where reshape() is an integer function; T and F refer to the time dimension and frequency dimension, respectively; MHA is a multi-head attention block; and Mamba is a bidirectional Mamba block.
[0048] This invention uses the preheated MambAttention model as the generator and introduces a multi-resolution discriminator as the debating agent for supervised adversarial training; a schematic diagram of the supervised adversarial training framework is shown below. Figure 3 As shown, the multi-resolution discriminator comprises three modules (D1, D2, and D3), sharing the same network architecture but using different STFT parameters (D1: FFT = 2048, hop = 240, win = 1200; D2: FFT = 1024, hop = 120, win = 600; D3: FFT = 512, hop = 50, win = 240). Each module consists of seven Conv2D layers, with the first six layers followed by LeakyReLU activation and the last layer by Sigmoid activation. The generator loss incorporates an adversarial loss L. A The total loss is During training, the generator and discriminator are updated alternately, and the learning rate is scheduled using exponential decay.
[0049]
[0050] in It is the discriminator's predicted label for the generated speech; λ = 0.1, the SHIFT operation alleviates the near and far field signal alignment error by time-domain frame shift.
[0051] In summary, this invention proposes a far-field single-channel speech enhancement method based on MFSE. This method first processes the far-field speech signal to obtain a complex spectrogram of the noisy speech signal. Then, the compressed complex spectrogram is input into a feature encoder, whose output is processed by N MambAttention blocks. Subsequently, it is fed into an amplitude mask decoder and a phase decoder to predict clean compressed amplitude and phase spectra, respectively. The MambAttention model is pre-trained, then used as a generator and a multi-resolution discriminator as a discriminator for supervised adversarial training. Finally, the test speech is input into the trained model to achieve far-field single-channel speech enhancement. This method can effectively suppress noise and reverberation simultaneously while completely preserving the speech signal, thereby improving the clarity and intelligibility of the speech signal.
[0052] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A far-field single-channel speech enhancement method, characterized in that, The method includes the following steps: Step 1: Process the far-field speech signal to obtain the complex spectrum of the noisy speech signal; Step 2: Input the compressed complex spectrum into the feature encoder, whose output is processed by N MambAttention blocks, and then fed into the amplitude mask decoder and phase decoder to predict the clean compressed amplitude mask and phase spectrum respectively. Step 3: First, warm up and train the MambAttention model, then use the MambAttention model as the generator and the multi-resolution discriminator as the discriminator to perform supervised adversarial training. Step 4: Use the supervised adversarial training model with the test speech input to achieve far-field single-channel speech enhancement.
2. The method according to claim 1, characterized in that, The complex spectrum is obtained by short-time Fourier transform of the far-field noisy speech time-domain signal.
3. The method according to claim 1, characterized in that, The feature encoder comprises two convolutional blocks and an extended DenseNet. Each convolutional block consists of a 2D convolutional layer, instance normalization, and PreLU activation. The magnitude mask decoder and phase decoder both include an extended DenseNet, 2D transposed convolution, instance normalization, and PreLU activation. The magnitude mask decoder is followed by a deconvolutional block and a learnable sigmoid function after activation, while the phase decoder is followed by two parallel 2D convolutional layers and a two-parameter arctangent function after activation.
4. The method according to claim 3, characterized in that, The expanded DenseNet contains four convolutional layers with dilation rates of 1, 2, 4, and 8, respectively. The expanded convolutions increase the receptive field by introducing interval sampling. In addition, all convolutional layers are interconnected through dense connections to avoid gradient vanishing during training.
5. The method according to claim 1, characterized in that, The MambAttention block processing flow includes shaping the input B×C×T×F into BF×T×C, where B is the batch size, C is the number of channels, T is the number of time frames, and F is the number of frequency points; layer normalization; time-domain multi-head attention block; bidirectional time-domain Mamba block; shaping the previous step output into BT×F×C; layer normalization; frequency-domain multi-head attention block; bidirectional frequency-domain Mamba block; and finally shaping the previous step output into B×C×T×F.
6. The method according to claim 4, characterized in that, The temporal multi-head attention block maps the input sequence into a query, key, and value matrix through a multi-head attention mechanism and calculates attention weights, which are responsible for capturing long-term dependencies in the temporal dimension. The bidirectional time-domain Mamba block is based on the state-space model and processes sequences in a bidirectional manner in the time domain, efficiently modeling long-distance temporal dependencies and capturing causal and non-causal relationships in time series through a state transition mechanism.
7. The method according to claim 4, characterized in that, The frequency-domain multi-head attention block is symmetrical to the time-domain multi-head attention block, focusing only on the feature interactions in the frequency dimension; the bidirectional frequency-domain Mamba block is symmetrical to the bidirectional time-domain Mamba block, applying bidirectional Mamba processing only in the frequency dimension.
8. The method according to claim 1, characterized in that, The preheating training of the MambAttention model is performed on the DNSChallenge 3 dataset. The noisy speech in the dataset is clean speech convolved with random RIR and mixed with noise, so that its SNR ranges from -5dB to 15dB. The target speech is the direct sound with the first 100ms of reverberation preserved in order to reduce the trailing effect and enhance the auditory experience.
9. The method according to claim 1, characterized in that, The multi-resolution discriminator comprises three different modules that share the same network architecture, which consists of three sets of short-time Fourier transforms (STFTs) with different parameters, followed by six sets of 2D convolutional layers with Leaky ReLU activation functions, and then a 2D convolutional layer with a Sigmoid activation function.
Citation Information
Cited By
ECG signal denoising method, device, equipment and medium
CN122096816A
Ecg signal denoising method, device, equipment and medium
CN122096816B