Speech enhancement method based on lightweight attention-driven network ADNet

Abstract

The invention relates to the technical field of voice signal processing, and discloses a voice enhancement method based on a lightweight attention driven network ADNet. The method comprises the following steps: firstly, constructing a multi-noise voice data set, executing short-time Fourier transform (STFT) on a voice signal, and extracting an amplitude spectrum and a phase spectrum as input features; then designing a lightweight ADNet network structure which comprises an encoder, a bottleneck layer and a decoder, wherein the encoder extracts multi-scale time-frequency features by using adaptive embedding and a multi-branch attention mechanism; the bottleneck layer fuses a CAE channel attention module and an FDAB cavity convolution attention module to strengthen voice related features and expand an effective receptive field; a-Gate attention guidance jump connection is adopted by the decoder, and cross-layer feature screening and noise suppression are achieved. The trained ADNet can output an enhanced amplitude spectrum and an enhanced phase spectrum, and reconstructs time domain speech through inverse short time Fourier transform (iSTFT). Experiments show that excellent enhancement performance can be obtained under the conditions of low signal-to-noise ratio, no noise and real recording, the model is lightweight and efficient, and the method is suitable for mobile terminals and real-time voice processing application and has good engineering application value.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing and intelligent voice interaction technology, specifically relating to a single-channel speech enhancement method based on the lightweight attention-driven network ADNet. Background Technology

[0002] Speech enhancement aims to recover a signal as close as possible to clean speech from noisy speech. It is a key technology in scenarios such as voice communication, speech recognition, far-field voice interaction, in-vehicle systems, and hearing aids. With the widespread use of mobile devices and smart terminals, speech enhancement systems not only need to have good noise suppression and speech fidelity capabilities, but also must meet the requirements of real-time performance, low power consumption, and deployability. However, the types of noise in real-world environments are complex and varied, such as traffic noise, human voice interference, and mechanical noise, which are typical non-stationary noises. Traditional statistical model methods such as spectral subtraction and Wiener filtering rely on fixed noise assumptions and are difficult to maintain stable enhancement performance in scenarios with low signal-to-noise ratio (SNR) and rapidly changing noise.

[0003] In recent years, deep learning has brought significant progress to speech enhancement tasks. Methods based on models such as U-Net, DNN, RNN, and TSTNN can directly learn the mapping relationship between noise and speech in the time or time-frequency domains, achieving enhancement results far superior to traditional methods under matched noise conditions. However, in order to improve the ability to model complex noise, these models generally adopt deep networks, multi-scale convolutions, and large-size attention structures, which significantly increases the number of model parameters and computational cost. While large models have outstanding performance, they are difficult to run in real time on mobile devices, embedded devices, and low-power platforms; while lightweight models are easy to deploy, they often show significant performance degradation under low SNR, no-noise, and real-noise conditions, manifesting as increased speech distortion, prominent residual noise, and insufficient enhancement stability.

[0004] Therefore, the current field of speech enhancement faces a prominent technical bottleneck: there is a significant contradiction between lightweight models and enhancement performance, especially in low signal-to-noise ratio and non-stationary noise scenarios, where lightweight models struggle to balance speech fidelity, noise suppression, and generalization ability. How to improve the robustness of lightweight speech enhancement models under various noise and scenario conditions by using efficient attention mechanisms, improved information fusion methods, and superior feature representation capabilities, while maintaining low model structural complexity, has become a critical technical problem that urgently needs to be solved. Summary of the Invention

[0005] To address the performance degradation of existing speech enhancement techniques under lightweight conditions and insufficient robustness in low signal-to-noise ratio and non-stationary noise environments, this invention proposes a speech enhancement method based on the lightweight attention-driven network ADNet. This method constructs a multi-noise speech dataset and designs a lightweight encoder-bottleneck-decoder neural network to jointly model the amplitude and phase spectra of the speech, thereby significantly improving speech enhancement performance in noisy environments while reducing the number of model parameters. The method includes the following steps:

[0006] Step 1) Construct a multi-noise speech dataset and perform time-frequency domain preprocessing. Use Short Time Fourier Transform (STFT) to extract the amplitude spectrum and phase spectrum of all speech samples, which will be used as the input for subsequent models and the basis for reconstruction.

[0007] Step 2) Design a lightweight attention-driven network ADNet (Attention-Driven Network for Speech Enhancement), and construct a time-frequency modeling structure consisting of an encoder, a bottleneck layer, and a decoder to achieve efficient joint modeling of speech time-frequency features;

[0008] Step 3) Use the constructed multi-noise speech dataset to train and optimize the parameters of ADNet, and improve the speech enhancement performance and generalization ability of the network under lightweight conditions through iterative updates;

[0009] Step 4) In the speech enhancement inference stage, STFT is performed on the speech to be enhanced to obtain the amplitude spectrum and phase spectrum, and then input into the trained ADNet to obtain the corresponding enhanced amplitude spectrum and enhanced phase spectrum;

[0010] Step 5) The enhanced amplitude spectrum and phase spectrum output by ADNet are reconstructed into time-domain enhanced speech through inverse short-time Fourier transform (iSTFT), and the performance of the speech enhancement results is evaluated using objective metrics such as PESQ and STOI.

[0011] Further, step 1) specifically includes:

[0012] 101) The TIMIT corpus was selected as a clean speech source, and all speech samples were preprocessed in a uniform manner, including sampling rate normalization, amplitude normalization and silence segment clipping, to ensure data consistency.

[0013] 102) Select eight typical noise types (such as white noise, factory noise, engine noise, etc.) from the NOISE-92 dataset and use them as the matching noise source during the training phase;

[0014] 103) Further select other noise types not used for training from NOISE-92, and select various real environmental noises (such as street noise, traffic noise, station noise, etc.) from the Aurora dataset as unseen noise test sets to verify the model's cross-noise generalization ability.

[0015] 104) Mix the above noise with clean speech according to the set signal-to-noise ratio (SNR = -5, 0, 5, 10 dB) to generate noisy speech samples with multiple noises, and construct training and test datasets covering different noise types and different interference intensities.

[0016] 105) The VoiceBank+DEMAND dataset is introduced as a test set for real-world recording of noisy scenes, serving as an evaluation basis for the model's real-world application capabilities;

[0017] 106) Perform short-time Fourier transform (STFT) on all the above speech samples and extract their amplitude spectrum and phase spectrum respectively, which will be used as the basic input features for subsequent model training and inference.

[0018] Furthermore, step 2) specifically involves:

[0019] 201) Construct the overall framework of the lightweight attention-driven network ADNet, adopting a symmetrical structure of encoder – bottleneck layer – decoder, and using the magnitude spectrum and phase spectrum of speech as input features, so that the network can simultaneously utilize information from the time and frequency dimensions for joint modeling.

[0020] 202) Introduce the MET-Transformer (Multi-path Enhanced Transformer) module into the encoder. It integrates multi-head self-attention and feedforward network structure. Multi-head attention is used to model the correlation between different time frames and frequency units to enhance the ability to express long-term dependencies and global speech structure.

[0021] 203) Add a Deformable Embedding structure to the input stage of MET-Transformer. By making a learnable offset on the feature sampling position, the network can still adaptively focus on the region where speech energy is concentrated even when the spectrum shape changes and the noise distribution is uneven, thereby improving the feature representation ability in complex noise environment.

[0022] 204) Embed a CAE (Channel-wise Attention Enhancement Block) module in the bottleneck layer. By performing global statistics on the entire feature map, the importance weight of each channel is automatically learned. The speech-related channels are enhanced and the noise-dominant channels are suppressed, thereby improving the selectivity and discriminativeness of features between channels.

[0023] 205) Further add the FDB (Frequency-Dilated Attention Block) module to the bottleneck layer. Use convolutional kernels with dilation rate to expand the receptive field without significantly increasing the number of parameters. Combine attention mechanism to assign differentiated weights to time-frequency units at different locations, thereby enhancing the network's ability to model contextual information and speech structure under low signal-to-noise ratio and non-stationary noise conditions.

[0024] 206) In the decoder section, an A-Gate (Attention-Guided Skip Connection) module is set up to perform attention filtering and weighted fusion on multi-scale features from each layer of the encoder. Only key information related to speech recovery is passed to the decoder. At the same time, noise components leaked through skip connections are suppressed. Finally, combined with layer-by-layer upsampling operations, the enhanced amplitude spectrum and phase spectrum are output, providing high-quality time-frequency estimation results for subsequent time-domain speech reconstruction.

[0025] Furthermore, step 3) specifically involves:

[0026] 301) The ADNet network structure is built and initialized using a deep learning framework (such as PyTorch). The parameters of each module of the encoder, bottleneck layer and decoder are set, including the kernel size, number of channels, number of heads of multi-head attention, dilation rate and skip connection configuration, so that the network has stable feature representation ability under lightweight constraints.

[0027] 302) Use the amplitude spectrum and phase spectrum extracted in step 1 as model inputs and the corresponding clean speech spectrum as training targets to construct supervised learning data pairs for the speech enhancement task;

[0028] 303) The AdamW optimizer is used to update parameters during training, and a cosine annealing learning rate scheduling strategy is combined to enable the model to converge quickly in the early stage of training and maintain stable detail optimization ability in the later stage, thereby improving the enhancement effect and generalization ability.

[0029] 304) To adapt to different noise distributions and signal-to-noise ratios, data augmentation strategies such as random pruning, normalization, and multi-noise mixing are introduced during the training phase to increase the diversity of training samples and improve the robustness of the model in environments with no visible noise and real recorded noise.

[0030] 305) During training, monitor objective metrics such as PESQ and STOI on the validation set, save the network weight file periodically, and select the best-performing set of parameters as the final model for speech enhancement tasks in the subsequent inference stage.

[0031] 306) After completing the end-to-end training of ADNet, the model is lightweighted and optimized for inference, including redundant channel pruning, activation function optimization and feature fusion path simplification, to ensure that the model can achieve real-time speech enhancement on resource-constrained devices.

[0032] Further, step 4) specifically involves:

[0033] 401) In the speech enhancement inference stage, the input speech signal to be enhanced is subjected to a short-time Fourier transform (STFT) consistent with the training stage, its amplitude spectrum and phase spectrum are extracted, and the amplitude spectrum is normalized to ensure that it is consistent with the training distribution.

[0034] 402) The extracted amplitude spectrum and phase spectrum are input into the trained lightweight attention-driven network ADNet, and processed layer by layer through the encoder, bottleneck layer and decoder to obtain the corresponding enhanced amplitude spectrum and enhanced phase spectrum.

[0035] 403) By using modules such as MET-Transformer, CAE, FCAB and A-Gate in ADNet, the time-frequency structure of noisy speech is denoised, feature-enhanced and context-completed to restore the speech energy distribution and suppress background noise.

[0036] 404) Perform post-processing operations on the enhanced amplitude spectrum and enhanced phase spectrum output by the model, including amplitude inverse normalization, phase matching and boundary smoothing, to improve the stability and reconstructability of the enhanced spectrum.

[0037] 405) The processed enhanced amplitude spectrum and phase spectrum are used as inputs to perform inverse short-time Fourier transform (iSTFT) to reconstruct the time-domain enhanced speech for further application or evaluation.

[0038] Further, step 5) specifically involves:

[0039] 501) Compare the temporal enhanced speech obtained in step 4 with the corresponding clean speech sample, and quantitatively evaluate the enhancement quality using objective indicators;

[0040] 502) The PESQ (Perceptual Evaluation of Speech Quality) index is used to measure the subjective listening quality of the enhanced speech, in order to evaluate the extent to which the model improves speech intelligibility and clarity in low signal-to-noise ratio and complex noise scenarios.

[0041] 503) The STOI (Short-Time Objective Intelligibility) metric was used to calculate the change in speech intelligibility before and after enhancement, in order to verify the network’s generalization ability in environments with no noise and real noise.

[0042] 504) Depending on the actual application requirements, additional indicators such as SNR and LSD (Log-Spectral Distance) can be introduced to comprehensively evaluate the degree of spectrum recovery, noise suppression capability and residual noise level of the enhanced signal;

[0043] 505) Based on the above indicators, the speech enhancement effect under different noise conditions is statistically analyzed to verify the robustness and superior performance of the lightweight attention-driven network ADNet in multiple noise, low signal-to-noise ratio and real-world scenarios.

[0044] Beneficial effects

[0045] (1) This invention provides a speech enhancement method based on a lightweight attention-driven network. By simultaneously utilizing amplitude spectrum and phase spectrum information for enhancement, it effectively overcomes the problems of severe speech distortion and obvious residual noise in traditional methods under low signal-to-noise ratio and non-stationary noise environments. Compared with existing deep learning models, this invention significantly improves noise suppression capability and speech recovery quality while maintaining extremely low parameter quantity, and has stronger engineering usability.

[0046] (2) This invention proposes a multi-noise data construction method that covers matched noise, unseen noise and real recorded noise scenarios, so that the training data has higher noise diversity; this data system improves the generalization performance of the model under unknown noise conditions, and makes the speech enhancement algorithm more suitable for complex sound fields in real environment;

[0047] (3) The lightweight speech enhancement network ADNet (Attention-Driven Network for Speech Enhancement) constructed in this invention integrates structures such as MET-Transformer, CAE, FDB and A-Gate, taking into account both global dependency modeling and local detail extraction, forming an efficient, stable and low-computational-cost speech enhancement scheme. This method achieves a good balance between inference speed, model size and enhancement effect, and fully demonstrates its application potential in real-time speech enhancement, mobile terminals and embedded devices. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the system flow of the present invention;

[0049] Figure 2 This is a schematic diagram of the overall structure of the lightweight speech enhancement network ADNet proposed in this invention.

[0050] Figure 3 The following are schematic diagrams of the encoder module structure: (a) shows the specific structure of the encoder; (b) shows the internal structure of the MET-Transformer.

[0051] Figure 4 The schematic diagrams of the bottleneck layer structure are as follows: (a) CAE structure; (b) FDAB structure.

[0052] Figure 5 The following are schematic diagrams of the decoder structure: (a) shows the overall structure of the decoder; (b) shows the A-Gate structure. Detailed Implementation

[0053] This invention discloses a speech enhancement method based on the lightweight attention-driven network ADNet, which addresses the problems of speech quality degradation in complex noise environments and limited computing resources of terminal devices. By constructing a multi-noise speech dataset, designing a lightweight network structure with multi-level attention fusion, and employing a joint amplitude spectrum-phase spectrum enhancement strategy, this invention achieves high-generalization speech enhancement capabilities with excellent performance in low signal-to-noise ratio, unseen noise, and real-world scenarios.

[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make various modifications and substitutions without departing from the concept of the present invention, all of which fall within the scope of protection of the present invention.

[0055] Specifically, referring to Figure 1, the overall method flow of this invention includes the construction and preprocessing of speech data, the design of the lightweight network ADNet, the training and optimization of the model, speech enhancement during the inference stage, and the performance evaluation of the results. This method mainly includes the following steps:

[0056] Step 1) Construct a multi-noise speech dataset and perform time-frequency domain preprocessing, including the following steps:

[0057] 101) Constructing a multi-noise dataset. Using the TIMIT corpus as a clean speech source, eight typical noise types from NOISE-92, other noises not used for training in NOISE-92, and Aurora environmental noise were mixed with the clean speech at different signal-to-noise ratios (SNR = −5 to 20 dB) to generate noisy training and test sets covering multiple noise types and interference intensities. Simultaneously, the VoiceBank+DEMAND dataset was used as independent real-world noise test data to evaluate the model's generalization ability in real-world scenarios.

[0058] 102) Standardize the speech format. Perform resampling, amplitude normalization, and silence clipping on all speech samples to unify the speech sampling rate to 16 kHz and standardize the amplitude range to [−1,1], ensuring a consistent input distribution during network training and improving model convergence stability.

[0059] 103) Perform time-frequency domain transformation. Perform short-time Fourier transform (STFT) on all speech samples in the training and inference phases to obtain the complex time-frequency spectrum representation of the speech.

[0060] Step 2) Design and construct the lightweight attention-driven network ADNet, including the following steps:

[0061] 201) Construct the overall structure of ADNet. Refer to... Figure 2 The overall architecture of ADNet is constructed. This network adopts a symmetrical structure of encoder-bottleneck layer-decoder, using amplitude and phase spectra as joint inputs. The encoder is responsible for extracting multi-scale time-frequency structure information; the bottleneck layer performs deep noise-speech modeling through a series of attention-enhanced modules; the decoder combines upsampling and attention-guided skip connections to recover high-resolution features, ultimately outputting enhanced amplitude and phase spectra.

[0062] The encoder includes a deformable embedding (DE) module and a multi-level MET-Transformer module; the bottleneck layer integrates the MET-Transformer, CAE channel attention module and FDB dilated convolution attention module; the decoder consists of an upsampling operator, MET-Transformer and A-Gate fusion module to achieve cross-scale fusion and detail restoration.

[0063] 202) Reference Figure 3 (a) Adaptive position modeling of the input amplitude and phase spectra based on Deformable Embedding (DE). This module enhances the representation of local speech structures through offset prediction, dynamic sampling, and linear mapping, providing more stable feature inputs for subsequent Transformer processing.

[0064] First, the offset prediction branch is based on the input features. Calculate the spatial offset of each sampling point:

[0065]

[0066] The sampling position is then updated based on the predicted offset:

[0067]

[0068] Then, deformable convolution and weighted sampling are performed at the updated sampling locations to obtain enhanced features:

[0069]

[0070] Finally, the embedded output features are obtained through linear projection:

[0071]

[0072] in, For offset prediction convolution kernel, The spatial offset of the i-th sampling point. and These represent the original and updated sampling positions, respectively. Here, K represents the interpolation weights at the corresponding positions, and K is the number of sampling points. This is a linear projection matrix. This step enables adaptive modeling of local time-frequency features of speech, providing more stable and expressive input features for the subsequent MET-Transformer module.

[0073] 203) Reference Figure 3(b) Multi-branch attention modeling is performed on the embedded features after DE processing based on MET-Transformer. This module includes a temporal multi-head self-attention branch (T-MSA) and a channel-space joint attention branch (CSA), which model the speech time-frequency features from the two dimensions of temporal dependence and spatial structure, respectively.

[0074] First, the input features Perform a linear mapping to obtain the query matrix. Key matrix AND-value matrix :

[0075]

[0076] T-MSA then computes multi-head self-attention in the time dimension to capture the temporal structure relationships of the speech:

[0077]

[0078] Next, channel attention Ac and spatial attention As are constructed to model the importance of amplitude and phase spectra in different spatial regions:

[0079]

[0080] Combining the two yields the channel-space joint attention CSA(Z):

[0081]

[0082] Finally, the outputs of the two branches are fused to form the enhanced feature Zf, and then normalized through a feedforward network to obtain the final output Y of the MET-Transformer:

[0083]

[0084] This step enables joint modeling of the temporal dependence, important region distribution, and cross-dimensional correlation of speech time-frequency features, providing more expressive deep features for subsequent bottleneck layers.

[0085] 204) Downsample the encoder output features and feed them into the bottleneck layer, referring to... Figure 3 (a) The encoder output features processed by the MET-Transformer module are input into the downsampling module. The resolution of the time and frequency dimensions is reduced simultaneously by stride convolution, pooling or other downsampling methods to obtain compressed time-frequency features. These features are then sent to the bottleneck layer for depth modeling.

[0086] This step reduces the computational burden on the bottleneck layer, expands the effective receptive field, and enables the network to model the overall structure of speech and noise over a larger area, thereby providing a more compact and expressive input to the bottleneck layer.

[0087] 205) Embed CAE in the bottleneck layer, refer to Figure 4 (a) A CAE module is added inside the bottleneck layer to model the channel-dimensional importance of the downsampled features. Let the input features of the bottleneck layer be denoted as... Its dimensions are C×T×F s (Where C is the number of channels, T is the number of time frames, and F is the number of time frames) s (This refers to the number of frequency points). First, global average pooling is performed on F along the time and frequency dimensions to obtain a channel-level global description vector. :

[0088] Subsequently, the channel attention weight vector is learned by sequentially applying two layers of fully connected mapping (dimensionality reduction followed by dimensionality increase) and non-linear activation functions such as ReLU and Sigmoid. :

[0089] Finally, After expansion by channel dimension, compared with the original feature Element-wise multiplication yields the enhanced bottleneck layer features. :

[0090]

[0091] in, , These are the parameters for the fully connected layer of the CAE module. (·) represents the ReLU activation function. (·) represents the Sigmoid activation function, " indicates element-wise multiplication along the channel dimension. The CAE module can enhance speech-related channels and suppress noise-dominant channels, thereby improving the discriminativeness and robustness of bottleneck layer features and providing more effective speech structure information for subsequent FDB and decoder modules.

[0092] 206) Reference Figure 4 (b) An FDB module is added to the bottleneck layer to expand the receptive field through dilated convolution and combine it with an attention mechanism to model the importance of different spatial locations. Let the input features be... The dimensions are C×T×F sFirst, FDAB uses dilated convolution kernels to perform dilated convolutions on the features, in order to obtain a wider range of contextual relationships and intermediate features without significantly increasing the number of parameters. :

[0093]

[0094] Subsequently, Normalization and ReLU activation are performed to form a location-dependent attention response:

[0095]

[0096] Next, respond with attention. Compared with the original input features The FDB output characteristics are obtained by adding them element-wise. To achieve enhanced positional attention:

[0097]

[0098] in, (·) represents a dilated convolution with an inflation rate of d. (·) indicates batch normalization operation. (·) for Activation function, " indicates element-wise addition. Through the coupling mechanism of dilated convolution and attention, the FDAB module can enhance the ability of bottleneck layer features to model long-range time-frequency dependence and speech structure, enabling the network to maintain higher robustness under low signal-to-noise ratio and non-stationary noise.

[0099] 207) Reference Figure 5 (a) and Figure 5 (b) Set the A-Gate in the decoder to adaptively filter and weightedly fuse cross-layer features from the encoder. Let the upsampled features of the current layer of the decoder be... The jump feature from the encoder is .

[0100] First, and Through weighted convolution respectively , After performing a linear transformation with batch normalization (BN), the results are added together and then activated by ReLU to obtain the fused intermediate features. :

[0101]

[0102] Then, intermediate features Input convolution kernel Attention weights are generated after Sigmoid activation. This is used to suppress noise components in skip connections and highlight speech-related regions.

[0103]

[0104] Finally, attention weights With jumping characteristics Element-wise multiplication followed by addition back to the main branch feature , get output ; then, Features of the main branch Concatenate the data along the channel dimension to form the input features for subsequent modules of the decoder:

[0105]

[0106]

[0107] The A-Gate module effectively suppresses noise leakage and enhances speech structure information by adjusting the attention of cross-layer jump features, providing a cleaner and more discriminative time-frequency representation for subsequent reconstruction by the decoder.

[0108] 208) The enhanced amplitude and phase spectra are output. After layer-by-layer upsampling by the decoder, A-Gate skip connection fusion, and MET-Transformer reconstruction modeling, the network finally obtains an enhanced feature map with full time-frequency resolution. This feature map is then separated into enhanced amplitude and enhanced phase spectra, which are used as inputs for time-domain reconstruction.

[0109] The enhanced amplitude spectrum is used to recover the energy distribution of the speech, while the enhanced phase spectrum is used to ensure the temporal continuity and naturalness of the speech. By performing inverse short-time Fourier transform (iSTFT) on both, the final temporal-domain enhanced speech can be generated, providing high-quality input for subsequent auditory evaluation and objective indicators (such as PESQ, STOI, etc.).

[0110] Step 3) Train and optimize the parameters of ADNet using the constructed multi-noise dataset, including the following steps:

[0111] 301) The complete network structure of ADNet is built using deep learning frameworks (such as Python and PyTorch), and the parameters of the encoder, bottleneck layer, and decoder are initialized and configured, including kernel size, channel dimension, number of multi-head attention heads, dilation rate of dilated convolutions, and A-Gate skip connection mode. By setting the parameters reasonably, ADNet can still have stable feature representation and noise suppression capabilities under lightweight constraints.

[0112] 302) The amplitude spectrum and phase spectrum extracted in step 1 are used as model inputs, and together with the corresponding clean speech amplitude spectrum and phase spectrum, they form supervised learning target pairs to generate speech enhancement training data. During training, the model uses the difference between the enhanced spectrum and the reference clean spectrum as the loss to guide the network to learn noise suppression and speech recovery capabilities.

[0113] 303) During training, the AdamW optimizer is used to update network parameters, and a cosine annealing learning rate strategy is applied to enable the model to quickly decrease to a better region in the early stages of training and maintain a stable convergence of the learning rate in the later stages, thereby improving the stability of the enhancement effect and the generalization ability. The change of the loss value is monitored throughout the entire training process to ensure that the network can efficiently learn time-frequency features.

[0114] 304) In order to enhance the adaptability of the model under different noise types and different signal-to-noise ratios, data augmentation strategies such as random pruning, amplitude normalization, and multi-noise mixing are added during the training phase to enable the training samples to cover more scene features, thereby improving the robustness of the network to unseen noise and real recorded noise.

[0115] 305) During the training phase, continuously monitor the performance of objective metrics such as PESQ and STOI on the validation set and save the weight file regularly; when the validation performance reaches the optimal level, use this set of parameters as the final model weights to ensure that the best speech enhancement performance can be obtained during the inference phase.

[0116] 306) After completing the end-to-end training of ADNet, the model is optimized and lightweighted by removing redundant channels, adapting fast activation functions, and simplifying feature fusion paths to improve inference speed and enable the model to achieve real-time speech enhancement in resource-constrained scenarios such as mobile devices and embedded platforms.

[0117] Step 4) In the speech enhancement inference stage, STFT is performed on the speech to be enhanced to obtain the amplitude spectrum and phase spectrum, and then input into the trained ADNet to obtain the corresponding enhanced amplitude spectrum and enhanced phase spectrum, including the following steps:

[0118] 401) Perform a short-time Fourier transform (STFT) on the input speech to be enhanced, extract its amplitude spectrum and phase spectrum, and process them according to the normalization method of the training stage to ensure that the feature distribution in the inference stage is consistent with that in the training stage, thereby improving the stability of the enhancement results.

[0119] 402) Input the normalized amplitude spectrum and phase spectrum into the trained ADNet network, and then sequentially process the encoder's multi-scale time-frequency feature extraction, the bottleneck layer's multi-level attention enhancement modeling, and the decoder's upsampling reconstruction and skip connection fusion to obtain the corresponding enhanced amplitude spectrum and enhanced phase spectrum.

[0120] 403) Inside ADNet, MET-Transformer is used to complete multi-branch attention modeling of time dependence and spatial structure; CAE module is used to enhance speech-related channel response; FDB is used to expand the receptive field and improve the time-frequency context modeling capability; and A-Gate is used to suppress noise and restore speech details of cross-layer skip features, so that the network can obtain enhanced features with more complete structure and less noise.

[0121] 404) Based on the decoder output of ADNet, the enhanced amplitude spectrum and enhanced phase spectrum with complete structure and noise suppression are obtained, providing high-quality time-frequency features for subsequent time-domain speech reconstruction.

[0122] 405) The enhanced amplitude spectrum and enhanced phase spectrum are output as the final enhancement result for use in time-domain reconstruction or downstream speech processing tasks.

[0123] Step 5) Reconstruct the enhanced amplitude spectrum and phase spectrum output by ADNet into time-domain enhanced speech using inverse short-time Fourier transform (iSTFT), and evaluate the performance of the speech enhancement results using various objective metrics, including the following steps:

[0124] 501) Perform inverse short-time Fourier transform (iSTFT) on the enhanced amplitude spectrum and enhanced phase spectrum output by ADNet to reconstruct the time-domain enhanced speech, which is used as the input signal for subsequent objective index evaluation.

[0125] 502) The reconstructed enhanced speech is compared with the corresponding clean speech samples. Multiple objective speech quality indicators are used to quantitatively evaluate the enhancement effect and measure the overall performance of the model under multiple noise and scenario conditions.

[0126] 503) PESQ (Perceptual Speech Quality Assessment) is used to calculate the quality difference between enhanced speech and reference clean speech. It is used to evaluate the clarity, naturalness and subjective improvement of the enhancement result, and is the core indicator for measuring the subjective quality of speech enhancement algorithms.

[0127] 504) The STOI (Short-Time Intelligibility Index) is used to evaluate the intelligibility of enhanced speech in noisy environments, improve the ability to quantify the fidelity of speech content, and effectively reflect the generalization ability of the model in no-noise and real-recorded noisy scenarios.

[0128] 505) Based on actual application needs, auxiliary evaluation indicators such as signal-to-noise ratio improvement (SNR-Improvement) and logarithmic spectral distance (LSD) can be further introduced to supplement the analysis of the degree of spectral recovery of speech, noise suppression capability and residual noise level, so as to comprehensively characterize the enhancement performance.

[0129] 506) Under various noise types, signal-to-noise ratios, and scenario testing conditions, statistical analysis was performed on the above evaluation indicators to verify the stability and superior performance of the lightweight attention-driven network ADNet in complex noise environments, low signal-to-noise ratio conditions, and real recording scenarios.

[0130] This invention proposes a speech enhancement method based on multi-noise data construction and a lightweight attention-driven network (ADNet). By constructing a multi-noise dataset for training and using the amplitude and phase spectra obtained from STFT as input, robust speech enhancement in complex sound fields is achieved. The ADNet designed in this invention adopts an encoder-bottleneck layer-decoder structure, combining deformable embedding (DE), MET-Transformer, CAE channel attention, FDB dilated convolutional attention, and A-Gate cross-layer fusion mechanism to effectively improve noise suppression and speech detail recovery quality. While maintaining a lightweight model, it achieves joint modeling of clean speech. This invention achieves significant performance improvements in various noise types, low signal-to-noise ratios, and real-world recording scenarios, with stable improvements in PESQ, STOI, and other metrics for enhanced speech. Compared with contrasting speech enhancement models, this invention has higher model efficiency and stronger cross-scenario generalization ability, making it suitable for resource-constrained applications such as mobile devices and real-time speech processing, and possesses good engineering application value and promising prospects for widespread adoption.

[0131] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech enhancement method based on a lightweight attention-driven network (ADNet), characterized in that, The method comprises the following steps: Step 1) constructing a multi-noise speech data set and performing time-frequency domain preprocessing, adopting a TIMIT corpus as a clean speech source, mixing eight types of NOISE-92 noises, NOISE-92 unused noises and Aurora environmental noises according to different signal-to-noise ratios to generate multi-noise noisy speech, simultaneously introducing a VoiceBank+DEMAND data set as a real recording noise test set, and performing short-time Fourier transform (STFT) on all speech to extract amplitude spectrum and phase spectrum; Step 2) constructing a light-weight attention-driven network ADNet, the network comprising an encoder, a bottleneck layer and a decoder; the encoder being composed of a deformable embedding (DE) module and a multi-level MET-Transformer, for extracting multi-scale time-frequency features; the bottleneck layer comprising a CAE channel attention module and an FDAB empty convolution attention module, for enhancing speech effective features and suppressing noise dominant features; the decoder comprising an up-sampling structure and an A-Gate attention-guided jump connection module, for restoring time-frequency resolution and realizing selective fusion of cross-layer features; Step 3) training the ADNet in an end-to-end manner by using the constructed multi-noise data set, and optimizing model parameters in a supervised learning manner, so that the network has stable noise suppression ability under the constraint of light weight; Step 4) in the speech enhancement inference stage, performing STFT on the speech to be enhanced, inputting the obtained amplitude spectrum and phase spectrum into the trained ADNet, and obtaining enhanced amplitude spectrum and enhanced phase spectrum through the encoder, the bottleneck layer and the decoder; Step 5) reconstructing the enhanced amplitude spectrum and the enhanced phase spectrum output by the ADNet into time-domain enhanced speech through inverse short-time Fourier transform (iSTFT), and evaluating the performance of the enhanced result by using objective indexes such as PESQ and STOI.

2. The method of claim 1, wherein, The multi-noise data construction of step 1 specifically comprises: 201) mixing eight types of NOISE-92 noises, unused noises and Aurora real environmental noises according to different signal-to-noise ratios and clean speech to form training samples covering different noise types and different noise intensities; 202) using noise types not used for training and VoiceBank+DEMAND real recording noises as unseen noise test sets to evaluate the cross-noise generalization ability of the model; 203) performing uniform resampling, amplitude normalization and feature alignment operations on all speech to ensure data consistency in the training and inference stages and stability of model input.

3. The method of claim 1, wherein, The encoder construction of step 2 comprises the following contents: 301) performing adaptive position sampling based on offset prediction on the input amplitude spectrum and phase spectrum to realize dynamic modeling of local time-frequency structure, thereby improving the response capability to speech details; 302) setting parallel time correlation modeling units and space-channel joint importance modeling units at multiple feature scales to simultaneously model the time sequence dependence, channel significance and spatial structure of the input features; 303) The multi-dimensional attention modeling results are integrated at the feature level through an inter-layer gradual fusion mechanism to obtain a multi-scale time-frequency representation that combines local detail expression and global correlation modeling capabilities.

4. The method of claim 1, wherein, The bottleneck layer of step 2 further includes: 401) CAE channel attention module, which generates channel weight coefficients through global statistics and nonlinear mapping to enhance speech-related effective features and suppress noise-dominant channels, improving the discriminability of intermediate features; 402) FDAB dilated convolution attention module, which expands the effective receptive field through dilated convolution and introduces a domain attention mechanism to strengthen the network's modeling ability for long-range time-frequency dependencies without significantly increasing the number of parameters, thereby improving speech recovery performance under low SNR conditions.

5. The method of claim 1, wherein, The decoder of step 2 includes: 501) Layer-wise upsampling structure for restoring the resolution of time and frequency dimensions; 502) A-Gate attention-guided skip connection module that filters correlation and suppresses noise in cross-layer features from the encoder through convolution, normalization, and weight generation mechanisms; 503) Concatenate the skip features processed by A-Gate with the decoder main branch features in the channel dimension to enhance speech details and structural continuity.

6. The method of claim 1, wherein, The inference and result evaluation of steps 4 and 5 include: 601) Perform inverse normalization on the enhanced amplitude spectrum output by ADNet and perform continuity correction on the enhanced phase spectrum to improve the smoothness and physical consistency of the reconstructed speech in the time domain; 602) Combine the enhanced amplitude spectrum and the enhanced phase spectrum to reconstruct the time-domain speech signal based on the reversible transformation from the frequency domain to the time domain to ensure the integrity of the speech's time structure and energy consistency; 603) Quantitatively evaluate the intelligibility, intelligibility, and noise suppression ability of the enhanced speech by comparing the differences between the reconstructed speech and the reference speech to verify the robustness and generalization ability of the proposed method in multiple noise, low SNR, and real-world scenarios.

Images