Speech enhancement method and system based on deformable embedding and joint time-frequency channel

Abstract

The present application belongs to the technical field of speech signal processing, and particularly relates to a speech enhancement method and system based on deformable embedding and time-frequency channel jointing. The method adopts multi-scale dense connection processing to capture the context relationship of speech and noise at different time-frequency levels. Adaptive feature embedding is realized in the multi-scale space through deformable convolution, thereby improving the robustness to speech structure changes and the generalization ability to different noise conditions. Through time-frequency channel jointing processing, not only the calculation advantage of lightweight convolution is retained, but also the structure perception ability is enhanced, and the speech dominant area can be focused and irrelevant interference can be suppressed in a complex noise environment.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, specifically relating to a speech enhancement method and system based on deformable embedding and time-frequency channel combination. Background Technology

[0002] Speech signals are highly susceptible to degradation in complex acoustic environments, especially due to the accumulation of ambient noise and the limitations of recording equipment (such as narrowband channels). These factors can degrade the performance of automatic speech recognition (ASR), voice communication, and teleconferencing systems. Speech enhancement techniques aim to separate clean speech from noisy mixed signals, thereby improving the perceptual quality and intelligibility of speech. Although deep learning methods perform well in this task, their high computational cost hinders their deployment on edge devices such as mobile phones or embedded systems.

[0003] Current research on lightweight speech enhancement typically includes the following categories: (1) model compression, such as pruning and quantization; (2) efficient structures, such as one-dimensional convolution or depthwise separable convolution; and (3) hybrid designs, which combine lightweight models with pitch-based post-processing methods. However, these methods often suffer from performance degradation or increased design complexity. Summary of the Invention

[0004] To address the problems in the background technology, this invention provides a speech enhancement method and system based on deformable embedding and time-frequency channel combination.

[0005] The technical solution of the present invention is as follows:

[0006] This invention provides a speech enhancement method based on deformable embedding and time-frequency channel combination, comprising:

[0007] S1: Extract the amplitude spectrum and phase spectrum of the speech signal to be processed, and concatenate the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map;

[0008] S2: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map;

[0009] The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then stitching together the results of multiple convolutions one by one.

[0010] S3: Multi-scale feature maps calculate the offset of each time-frequency position through deformable convolution, and based on the offset, align features to generate an adaptive embedded feature map;

[0011] S4: The adaptive embedding feature map is processed by multiple depthwise separable convolutions and residual processing to generate intermediate feature maps. The intermediate feature maps are processed by channel attention, temporal attention, and frequency attention respectively. After weighted fusion, they are applied to the intermediate feature maps to generate enhanced feature maps.

[0012] S5: After multiple upsamplings, the enhanced feature map generates a complex spectrum, which is then reconstructed into a time-domain waveform through inverse short-time Fourier transform.

[0013] The adaptive embedding feature map described in S4 undergoes multiple depthwise separable convolutions and residual processing to generate an intermediate feature map, specifically:

[0014] After the adaptive embedding feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to obtain the first feature map. After the first feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to generate the intermediate feature map.

[0015] Furthermore, the multiple depthwise separable convolutions are specifically as follows:

[0016] The feature map to be processed is sequentially processed by depthwise separable convolution, group normalization, and PReLU activation function, and then the process is repeated sequentially.

[0017] The generation of the adaptive embedding feature map described in S3 is specifically as follows:

[0018] After the multi-scale feature map is processed by depthwise separable convolution, the offset of each time-frequency position is calculated by deformable convolution. Based on the offset, irregular sampling is performed on the multi-scale feature map, and the sampling results are reprojected onto the feature map after depthwise separable convolution to generate an adaptive embedding feature map.

[0019] The phase spectrum described in S1 is concatenated with the compressed amplitude spectrum to generate a time-frequency feature map, specifically:

[0020] The amplitude spectrum is compressed using a power law to obtain the compressed amplitude spectrum;

[0021] After the phase spectrum is converted into a dual-channel form of sine and cosine, it is spliced ​​with the compressed amplitude spectrum to generate a time-frequency feature map.

[0022] The initial convolution described in S2 is the processing of the time-frequency feature map through a first two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function.

[0023] The downsampling described in S2 involves sequentially processing the data through a second two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function.

[0024] The present invention also provides a speech enhancement system based on deformable embedding and time-frequency channel combination, comprising:

[0025] Time-frequency feature map generation module: Extracts the amplitude spectrum and phase spectrum of the speech signal to be processed, and concatenates the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map;

[0026] Multi-scale feature map generation module: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map;

[0027] The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then stitching together the results of multiple convolutions one by one.

[0028] Adaptive Embedded Feature Map Generation Module: The multi-scale feature map calculates the offset of each time-frequency position through deformable convolution. Based on the offset, the features are aligned to generate an adaptive embedded feature map.

[0029] Enhanced feature map generation module: The adaptive embedded feature map is processed by multiple depthwise separable convolutions and residual processing to generate intermediate feature maps. The intermediate feature maps are then processed by channel attention, temporal attention, and frequency attention respectively, and weighted and fused and applied to the intermediate feature maps to generate enhanced feature maps.

[0030] Reconstruction module: After multiple upsampling of the enhanced feature map to generate a complex spectrum, it is reconstructed into a time-domain waveform through inverse short-time Fourier transform.

[0031] Beneficial effects

[0032] This invention employs multi-scale dense connection processing to capture the contextual relationship between speech and noise at different time-frequency levels; it achieves adaptive feature embedding in multi-scale space through deformable convolution, thereby improving robustness to changes in speech structure and generalization ability under different noise conditions; through joint processing of time and frequency channels, it not only retains the computational advantages of lightweight convolution but also enhances the structure perception capability, enabling it to focus on speech-dominant regions and suppress irrelevant interference in complex noise environments. Detailed Implementation

[0033] The following examples are intended to illustrate the present invention, and not to further limit the invention.

[0034] Example 1

[0035] This invention provides a speech enhancement method based on deformable embedding and time-frequency channel combination, comprising:

[0036] S1: Extract the amplitude and phase spectra of the speech signal to be processed. Concatenate the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map. The specific steps are as follows:

[0037] First, the speech signal to be processed (noisy speech signal) undergoes short-time Fourier transform (STFT) processing, which maps it to a two-dimensional time-frequency space, generating the corresponding amplitude spectrum. Phase spectrum This step transforms the time-series signal into a time-frequency matrix, providing a foundation for subsequent capture of the structural features of speech in the time and frequency dimensions.

[0038] Subsequently, preferably, the phase spectrum is concatenated with the compressed amplitude spectrum to generate a time-frequency feature map, specifically as follows:

[0039] The amplitude spectrum is compressed using power law to obtain the compressed amplitude spectrum. This operation aims to weaken the dominant role of high-energy features, increase the weight of low-energy speech segments, and make the spectral energy distribution more balanced.

[0040] Phase spectrum After being converted to a dual-channel format of sine and cosine, it is compared with the compressed amplitude spectrum. The data are concatenated to generate a time-frequency feature map. This time-frequency feature map contains both speech energy information and phase structure information, representing a real-valued equivalent expression of the complete complex spectral structure of noisy speech in the time-frequency domain.

[0041] S2: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map;

[0042] The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then sequentially concatenating the results of multiple convolutions.

[0043] Preferably, the initial convolution is achieved by sequentially processing the time-frequency feature map through a first two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function to increase the channel dimension.

[0044] In one specific embodiment, the time-frequency feature map is first subjected to a 1×1 convolution to linearly map and compress the channel dimension, thereby achieving feature integration and dimensionality reduction, and providing a more efficient input representation for subsequent deep convolutions.

[0045] Next, the feature distribution of each sample is standardized by two-dimensional instance normalization and LeakyReLU activation function to reduce speaker differences and noise energy imbalance.

[0046] Regarding the aforementioned multi-scale dense connection processing, in one specific embodiment, four dilated convolutions (with dilation rates of 1, 2, 4, and 8) are used. By successively concatenating the results of multiple convolutions, the contextual relationships between speech and noise are captured at different time-frequency levels, achieving feature reuse and multi-scale context aggregation. This allows shallow local time-frequency information to directly participate in high-level feature modeling. This not only alleviates the gradient vanishing problem but also enhances the diversity of spectral-temporal features. Simultaneously, using different dilation rates in each convolution expands the receptive field, thereby capturing the multi-scale dependencies of the speech signal in the time and frequency dimensions.

[0047] Regarding residual processing, it not only ensures the stable propagation of gradients in deep structures and prevents performance degradation, but also enhances the semantic consistency of high-level layers while preserving low-level details.

[0048] Preferably, the downsampling is performed by sequentially passing through a second two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function.

[0049] In one specific embodiment, the feature map obtained after residual processing is subjected to 1×3 convolution, and local feature aggregation and sideband refinement are performed in the frequency direction to enhance the ability to characterize speech formants and fine-grained spectral structures.

[0050] Finally, the stability of the feature distribution is further improved by two-dimensional instance normalization and LeakyReLU activation function processing.

[0051] This invention maintains low computational complexity while taking into account both time-frequency detail modeling and global semantic representation, enabling it to provide robust and information-intensive high-level feature representations for subsequent applications.

[0052] S3: The multi-scale feature map calculates the offset at each time-frequency position through deformable convolution. Based on the offset, features are aligned to generate an adaptive embedding feature map. Specifically:

[0053] After the multi-scale feature map is processed by depthwise separable convolution, the offset of each time-frequency position is calculated by deformable convolution. Based on the offset, irregular sampling is performed on the multi-scale feature map, and the sampling results are reprojected onto the feature map after depthwise separable convolution to generate an adaptive embedding feature map.

[0054] Under noisy or reverberant conditions, the local energy distribution of spectrograms often undergoes spatial shifts (such as source movement or reflection delay), which are difficult to capture with fixed convolutional kernels. This invention adaptively adjusts the sampling position based on local changes in the spectrogram structure of multi-scale feature maps, thereby effectively modeling the non-stationarity and reverberation distortion of speech in complex acoustic environments. The resulting adaptively embedded feature map has its spatial distribution adaptively corrected according to the local structure of the speech.

[0055] Specifically, the multi-scale feature maps are first processed by depthwise separable convolution, achieving channel-wise convolution and linear combination to reduce computational complexity and extract local time-frequency patterns. Subsequently, deformable convolution adaptively shifts the sampling position of the convolution kernel, dynamically adjusting the receptive field based on the time-varying and non-stationary nature of the speech signal, thereby capturing fine-grained speaker features while maintaining lightweight computation. To avoid gradient instability issues caused by deep deformability, constant-dimensional residual connections are also implemented to preserve original feature information and ensure stable gradient propagation. Furthermore, a Hardswish activation function is introduced to enhance nonlinear expressiveness, facilitating smoother and more efficient modeling of complex acoustic patterns.

[0056] Compared to fixed convolutional structures, this invention enables adaptive feature embedding in a multi-scale space, thereby improving robustness to changes in speech structure and generalization ability under different noise conditions. Furthermore, by employing deformable convolutions, it achieves a balance between local structure learning and global pattern modeling, providing a more flexible and stable high-dimensional semantic representation for speech enhancement tasks.

[0057] S4: The adaptive embedded feature map is processed by multiple depthwise separable convolutions and residual processing to generate an intermediate feature map. The intermediate feature map is then processed by channel attention, temporal attention, and frequency attention, and then weighted and fused and applied to the intermediate feature map to generate an enhanced feature map.

[0058] Specifically, the adaptive embedding feature map undergoes multiple depthwise separable convolutions and residual processing to generate an intermediate feature map, as follows:

[0059] After the adaptive embedding feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to obtain the first feature map. After the first feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to generate the intermediate feature map.

[0060] Furthermore, the multiple depthwise separable convolutions are specifically as follows:

[0061] The feature map to be processed is sequentially processed by depthwise separable convolution, group normalization, and PReLU activation function, and then the process is repeated sequentially.

[0062] In one specific embodiment, the adaptive embedding feature map is first processed by depthwise separable convolution, followed by channel-wise spatial convolution (kernel size 3×3, stride 1, padding 1), and then group normalization and PReLU activation. After further processing by depthwise separable convolution, group normalization, and PReLU activation, the resulting feature map is added to the adaptive embedding feature map by residual addition to obtain the first feature map.

[0063] Then, the first feature map is processed by depthwise separable convolution, followed by channel-wise spatial convolution (kernel size 3×3, stride 1, padding 1), and then group normalization and PReLU activation. After further processing by depthwise separable convolution, group normalization, and PReLU activation, the resulting feature map is added to the first feature map by residual addition to generate an intermediate feature map with the ability to fuse local and global features.

[0064] The intermediate feature maps are processed by channel attention, temporal attention, and frequency attention respectively, and then weighted and fused before being applied to the intermediate feature maps to generate enhanced feature maps. Preferably, the channel attention is achieved by performing two linear mappings sequentially on the intermediate feature maps after global average pooling to generate channel attention weights.

[0065] The temporal attention processing involves processing the intermediate feature map through a first depthwise convolution to generate temporal attention weights.

[0066] The frequency attention processing involves processing the intermediate feature map through a second depth convolution to generate frequency attention weights.

[0067] Channel attention weights, temporal attention weights, and frequency attention weights are weighted and fused with the intermediate feature map, and then processed by the Sigmoid activation function before being multiplied element-wise with the intermediate feature map to generate an enhanced feature map.

[0068] In one specific embodiment, the three parallel branches are configured as follows:

[0069] Temporal attention processing branch: 1×3 depthwise convolution is used to extract time series dependencies and obtain global statistical features of the time series. Then, a lightweight one-dimensional depthwise convolution is used to capture local dependencies in the time dimension.

[0070] Frequency attention processing branch: 3×1 depth convolution is used to focus on changes in energy distribution, capture the energy distribution patterns of different frequency bands, thereby strengthening the main frequency region of speech and suppressing noise frequency bands.

[0071] Channel attention processing branch: It consists of global average pooling and two fully connected processing steps. Global average pooling extracts the statistical information of each channel in the full time-frequency plane, and generates channel weights through two linear mappings (ReLU and Sigmoid) to redistribute the importance between channels.

[0072] The processing results from the three branches are weighted and fused before being applied to the intermediate feature map. This invention's joint time-frequency channel processing not only retains the computational advantages of lightweight convolution but also enhances structure perception capabilities, enabling it to focus on speech-dominant regions and suppress irrelevant interference in complex noise environments.

[0073] After weighted fusion and application to the intermediate feature map, a 1×1 convolution is used for channel mapping to integrate the processing results of each attention branch and compress the feature dimension. The channel mapping result can also be added to or concatenated with skip connection features in the channel dimension to maintain the consistency of multi-scale features. The final enhanced feature map retains both the local time-frequency structure and embeds global channel dependencies.

[0074] In the process of enhancing feature map generation, this invention performs feature extraction and multi-scale fusion through multiple depthwise separable convolutions and residual processing; the joint processing of time and frequency channels completes the adaptive weighting of the time and frequency dimensions; 1×1 convolution and skip connections realize information recombination and cross-layer integration, thereby achieving efficient joint modeling of time and frequency channels with low computational cost.

[0075] S5: After multiple upsampling of the enhanced feature map to generate a complex spectrum, it is reconstructed into a time-domain waveform through inverse short-time Fourier transform. The specific operation is as follows:

[0076] The enhanced feature map fuses high-level semantic features with low-level detail features through successive upsampling and skip connections, gradually recovering a high-resolution spectroscopic representation. The two branches obtained—one for predicting the enhanced amplitude spectrum and the other for predicting the phase correction term—are fused to generate a complex spectrogram. The complex spectrogram is then reconstructed into a time-domain waveform using an inverse short-time Fourier transform.

[0077] This completes the full mapping process from noisy speech signals to enhanced speech.

[0078] This invention employs multi-scale dense connection processing to capture the contextual relationship between speech and noise at different time-frequency levels; it achieves adaptive feature embedding in multi-scale space through deformable convolution, thereby improving robustness to changes in speech structure and generalization ability under different noise conditions; through joint processing of time and frequency channels, it not only retains the computational advantages of lightweight convolution but also enhances the structure perception capability, enabling it to focus on speech-dominant regions and suppress irrelevant interference in complex noise environments.

[0079] Example 2

[0080] Based on Embodiment 1, the present invention also provides a speech enhancement system based on deformable embedding and time-frequency channel combination, comprising:

[0081] Time-frequency feature map generation module: Extracts the amplitude spectrum and phase spectrum of the speech signal to be processed, and concatenates the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map;

[0082] Multi-scale feature map generation module: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map;

[0083] The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then stitching together the results of multiple convolutions one by one.

[0084] Adaptive Embedded Feature Map Generation Module: The multi-scale feature map calculates the offset of each time-frequency position through deformable convolution. Based on the offset, the features are aligned to generate an adaptive embedded feature map.

[0085] Enhanced feature map generation module: The adaptive embedded feature map is processed by multiple depthwise separable convolutions and residual processing to generate intermediate feature maps. The intermediate feature maps are then processed by channel attention, temporal attention, and frequency attention respectively, and weighted and fused and applied to the intermediate feature maps to generate enhanced feature maps.

[0086] Reconstruction module: After multiple upsampling of the enhanced feature map to generate a complex spectrum, it is reconstructed into a time-domain waveform through inverse short-time Fourier transform.

[0087] The system provided by this invention can achieve high-quality voice reconstruction under limited computing power conditions, and is particularly suitable for typical application scenarios such as in-vehicle voice interaction systems, mobile terminal voice communication, and conferencing and IoT voice front-ends.

[0088] In in-vehicle voice interaction scenarios, engine noise, wind noise, and multi-source interference are common in the cockpit, posing significant challenges to voice recognition and command control. The system provided by this invention can serve as a front-end voice enhancement module in an in-vehicle infotainment (IVI) system, deployed on a digital signal processor (DSP) or dedicated AI chip. Its enhanced feature map generation module effectively suppresses low-frequency continuous noise and utilizes an adaptive embedding mechanism to dynamically adjust feature distribution to adapt to different vehicle speeds and environmental conditions. The system's low parameter count (0.12M) ensures millisecond-level latency even on resource-constrained platforms, achieving a robust interactive experience for in-vehicle voice assistants.

[0089] In mobile terminal voice communication scenarios, environmental noise is complex and variable, and device power consumption is limited. The system provided by this invention can be embedded in the audio signal link of mobile devices, serving as an intermediate processing module between the system audio HAL and the application layer, directly acting on the microphone acquisition channel. The system's depthwise separable convolutional structure significantly reduces the computational load, while time and frequency signals, through joint processing, can adaptively suppress complex interference such as urban noise and overlapping human voices, thereby improving voice call clarity and user experience.

[0090] In conferencing and IoT voice front-end scenarios, the system provided by this invention can be deployed on edge computing nodes or voice acquisition gateways as a front-end module for real-time voice enhancement. It receives digital audio streams from single microphones or multi-microphone arrays, performs feature transformation, enhancement, and reconstruction, and then outputs the results to a speech recognition engine or a cloud-based semantic analysis system. The system provided by this invention features flexible interfaces and a customizable structure, enabling dynamic adjustment of the encoding level and the number of embedded modules based on the terminal's computing power, achieving system integration of "lightweight deployment—high-efficiency processing—real-time output."

[0091] experiment

[0092] 1. Dataset

[0093] In the experiments, two widely used speech enhancement datasets, VoiceBank+DEMAND and DNSChallenge 2020, were used to evaluate the performance of the corresponding model of the method of the present invention under standard benchmark and real-world scenario conditions.

[0094] The VoiceBank+DEMAND dataset consists of clean speech samples from 28 speakers in the VoiceBank corpus, mixed with 10 types of noise from the DEMAND database at different signal-to-noise ratios (SNRs) (0-15 dB). Its test set contains 824 speech samples from 2 unseen speakers under 5 unseen noise conditions, with SNRs ranging from 2.5 to 17.5 dB. All audio samples were resampled to 16 kHz.

[0095] The DNS Challenge 2020 dataset contains approximately 500 hours of clean speech from 2150 speakers and 180 hours of noisy speech. Approximately 3000 hours of noisy-clean speech pairs were generated using the official script, with a signal-to-noise ratio (SNR) ranging from -5 to 15 dB. Evaluation was performed on a reverberant-free subset of the non-blind test set, containing 4270 speech samples from 20 speakers, uniformly sampled with an SNR ranging from 0 to 25 dB.

[0096] Furthermore, to enhance the model's adaptability in typical applications such as in-vehicle voice interaction, mobile terminal communication, and conferencing and IoT front-ends, this experiment constructed a multi-scenario composite training set based on the aforementioned publicly available data. Specifically, using actual collected in-vehicle cabin noise (engine and wind noise), urban and indoor conversation noise (subway, shopping mall, and home backgrounds), and meeting scene noise (human conversation, keyboard typing, and electrical interference), these were randomly mixed with clean speech at a signal-to-noise ratio of 0-15dB to generate noisy speech samples matching the target deployment scenario. Through this data construction method, the model can be exposed to multiple types of real acoustic interference during the training phase, improving its robustness and generalization ability across different devices and environments.

[0097] 2. Implementation details

[0098] The model corresponding to the method of this invention uses the Adam optimizer during training, with an initial learning rate of 0.001. The learning rate is halved if the validation set loss does not improve within five consecutive epochs. The batch size is set to 4, and 40,000 pairs (8 seconds per segment) of noisy-clean speech are randomly sampled per epoch. Unseen noise types and signal-to-noise ratio conditions are used during the testing phase to evaluate the model's generalization ability.

[0099] To better align with real-world application environments, a phased scenario-adaptive strategy is employed during training:

[0100] General pre-training was performed on the VoiceBank+DEMAND data to enable the model to learn basic speech-noise separation capabilities;

[0101] Training was performed on the DNS Challenge data in multi-noise environments to enhance the model's robustness to various complex sound sources;

[0102] We performed targeted fine-tuning on three self-constructed subsets of scenarios (vehicle, mobile, and conferencing / IoT). Specifically, vehicle scenario samples were used to optimize the low-frequency convolutional path to suppress continuous engine noise; mobile terminal samples were used to adjust the time-frequency attention branch to adapt to non-stationary backgrounds; and conferencing scenario samples were used to optimize the adaptive embedding module of the bottleneck layer to improve reverberation recovery capabilities.

[0103] The entire training process was conducted over 100 epochs, employing a cosine annealing strategy to gradually decay the learning rate. The WB-PESQ metric was monitored on the validation set to evaluate model convergence. During the testing phase, unseen noise types and signal-to-noise ratio conditions were used to verify the model's generalization ability across different deployment scenarios.

[0104] 3. Evaluation Indicators

[0105] To evaluate the effectiveness of the model corresponding to the method of this invention, this experiment employed multiple objective evaluation metrics. Wideband PESQ (WB-PESQ) was used to evaluate the perceptual quality of wideband speech, with a score range of -0.5 to 4.5. Similarly, Narrowband PESQ (NB-PESQ) was used to evaluate the quality of narrowband speech, with the same score range. STOI was used to measure speech intelligibility, with a score range of 0 to 1.

[0106] In addition, this experiment also used three comprehensive indicators based on subjective opinion scores (MOS): CSIG (predicted signal distortion), CBAK (background noise interference), and COVL (overall quality), all of which have a score range of 1 to 5.

[0107] To quantify computational efficiency, this experiment reports the number of model parameters and floating-point operations (FLOPS). The FLOPS figures are based on processing an audio sample with a sampling rate of 16kHz and a duration of 2 seconds on a GPU.

[0108] 4. Experimental Results

[0109] Table 1 shows the comparison results of the SE-AdaUNet model corresponding to the method of this invention with several state-of-the-art speech enhancement models on the VoiceBank+DEMAND dataset. SEGAN proposes an end-to-end speech enhancement framework based on generative adversarial networks, achieving noise suppression through joint training of convolutional encoder-decoder and discriminator. However, its structure mainly relies on fixed convolutional kernels, making it difficult to capture local shift features in complex acoustic environments, and its model parameter scale is large. DEMUCS combines convolutions and bidirectional recurrent networks to recover temporal waveforms, performing well in long-term dependency modeling, but its reliance on LSTM structures leads to significant computational overhead, making it difficult to deploy in lightweight scenarios. TSTNN and MetricGAN+ optimize speech quality based on time-frequency Transformers or adversarial learning based on perceptual metrics, respectively, but their structural complexity is high, mainly improving performance through deep layer stacking, and they do not adaptively align to the spatial non-stationarity of the spectrogram. Furthermore, the large-scale models ROSE and MANNER-S-5.3GF improve expressive power through multi-branch design and large convolutional stacking, resulting in high computational cost and parameter count, making them unsuitable for resource-constrained devices. SE-Conformer, based on the Conformer attention structure, captures long and short-term dependencies by combining convolutional modules and multi-head self-attention. It requires global computation of the feature sequence, and its inference complexity increases linearly or even approximately quadratically with the sequence length, resulting in significant overall computational overhead. MetricGAN-OKD, on the other hand, introduces an online knowledge distillation (OKD) strategy on top of MetricGAN. It relies on multi-layer convolutions for deep feature encoding and requires training a discriminator network to participate in adversarial learning. Its overall training and inference structure is complex, and its parameter scale and operational efficiency are insufficient for resource-constrained scenarios.

[0110] This table specifically highlights models with a parameter size close to 1.0M to emphasize the advantages of the corresponding model in lightweight design. The corresponding model achieved a score of 3.31 on the WB-PESQ metric, outperforming models such as SEGAN (2.16) and DEMUCS (3.07). Notably, the corresponding model also outperformed other comparative methods on the CSIG, CBAK, and COVL metrics, demonstrating its effectiveness in improving speech quality and reducing distortion and noise interference. Furthermore, the corresponding model requires only 0.12M parameters, significantly reducing computational and storage overhead while maintaining high performance, making it highly suitable for deployment in resource-constrained real-world scenarios.

[0111] Table 1 Test results of each model on the VoiceBank+DEMAND dataset

[0112]

[0113] "–" indicates that the original paper did not report this result.

[0114] Table 2 presents the comparison results of the SE-AdaUNet model corresponding to the method of this invention with representative state-of-the-art (SOTA) speech enhancement models on the VoiceBank+DEMAND dataset. Among them, CMGAN, MP-SENet, MUSE, and SEMamba typically rely on large-scale convolutional stacking or multi-head attention structures to improve speech quality, thus having a high number of parameters and computational cost.

[0115] Among all methods, the method of this invention has the lowest computational cost for the corresponding model, using only 0.12M parameters and 4.49 GFLOPs. Compared with large-scale architectures such as MP-SENet and CMGAN, it reduces the number of parameters by more than 90% and requires only about 6-7% of the computation.

[0116] Regarding speech intelligibility, the model corresponding to the method of this invention has an STOI of 0.95, which is almost equivalent to the best-performing large model while consuming very few computational resources. Regarding perceptual quality, the model corresponding to the method of this invention has a WB-PESQ of 3.31. Although slightly lower than some high-capacity models, this result remains highly competitive considering the significant reduction in model size and complexity.

[0117] Overall, the model corresponding to the method of this invention maintains near-state-of-the-art (SOTA) understandability while significantly reducing computational requirements, providing a solution for resource-constrained, real-time deployment scenarios such as embedded platforms and edge devices.

[0118] Table 2 Comparison results on VoiceBank+DEMAND datasets

[0119]

[0120] Table 3 compares the performance of the SE-AdaUNet model corresponding to the method of this invention with several recent lightweight speech enhancement models on the DNS Challenge 2020 dataset. This dataset covers diverse real-world noise scenarios, multi-speaker speech, and complex acoustic backgrounds, providing a more comprehensive benchmark for evaluating the generalization ability and enhancement performance of models in practical applications. The recent lightweight speech enhancement models compared in Table 3 (such as DCCRN, FullSubNet-Lite, TF-GridNet-Lite, etc.) typically reduce the number of parameters by decreasing network depth or the number of channels, but they often employ fixed convolutional structures or a single time-frequency processing path, making it difficult to fully model the non-stationary features in real-world noise scenarios.

[0121] Experimental results show that the model corresponding to the method of this invention achieves WB-PESQ=3.25 and NB-PESQ=3.66, respectively, and reaches 98.10 in the STOI index, significantly outperforming several larger-scale models in terms of speech quality and intelligibility. These results demonstrate that the model corresponding to the method of this invention can achieve high-fidelity speech enhancement while maintaining a lightweight structure.

[0122] Compared to VoiceBank+DEMAND, the DNS Challenge 2020 dataset is significantly more complex and diverse, containing over 2000 speakers and a wide range of signal-to-noise ratios (SNR) covering various acoustic conditions. Despite the more challenging task, the model corresponding to the method in this invention maintains stable and competitive performance, further validating the effectiveness of its architectural design.

[0123] Therefore, with only 0.12M parameters, the model corresponding to the method of this invention has achieved excellent results on two benchmarks with significant differences in data distribution: VoiceBank+DEMAND and DNSChallenge 2020. This highlights the adaptability and generalization ability of the model in real deployment scenarios and proves that it has achieved an effective balance between lightweight design and high performance.

[0124] Table 3 Test results of each model in the DNS Challenge 2020 dataset

[0125]

[0126] "–" indicates that the original paper did not report this result.

[0127] Table 4 analyzes the effects of different modules of the SE-AdaUNet model corresponding to the method of this invention through ablation experiments on the VoiceBank+DEMAND dataset. The complete model achieves the best performance: WB-PESQ=3.31, CSIG=4.51, CBAK=3.70, COVL=3.93 (while maintaining STOI=0.95).

[0128] When the residual processing module (which serves skip connections and avoids the vanishing gradient problem in deep learning networks) is removed from the multi-scale feature map generation module, the WB-PESQ drops slightly to 3.25, while other metrics remain relatively stable, indicating that residual connections are crucial for ensuring feature propagation and training stability. Removing the time-frequency channel joint processing module results in a more significant performance decrease, with WB-PESQ dropping to 3.14, demonstrating the key role of time-frequency channel joint attention processing in improving speech feature extraction capabilities. Removing the adaptive embedding feature map generation module results in a WB-PESQ of 3.28, slightly lower than the complete model, but still better than the configuration lacking other key components, further demonstrating the effectiveness of adaptive embedding in capturing crucial information.

[0129] Table 4 Ablation Experiment

[0130]

[0131] To verify the effectiveness of DenseEncoder and SEAdaConvBlock in the model corresponding to the method of this invention, ablation experiments were conducted based on the VoiceBank+DEMAND dataset, and the results are shown in Table 5.

[0132] Replacing the encoder of the MP-SENET speech enhancement model and the input encoder of the MUSE speech enhancement model with the multi-scale feature map generation module proposed in this invention, denoted as DenseEncoder, yields models denoted as MP-SENET+DenseEncoder and MUSE+DenseEncoder, respectively. It can be observed that WB-PESQ is improved by 0.02 and 0.01, respectively, while the number of parameters and FLOPS remain essentially unchanged. This indicates that DenseEncoder effectively enhances feature extraction capabilities and improves speech enhancement performance without significantly increasing computational complexity.

[0133] Furthermore, to evaluate the impact of the joint time-frequency channel processing module (denoted as SEAdaConvBlock), in the MUSE speech enhancement model, which has a similar multi-scale coding-decoding framework (for fusing time-frequency features at different scales) to the model corresponding to the method of this invention, the MET Transformer module was replaced with SEAdaConvBlock, resulting in a model denoted as MUSE+SEAdaConvBlock. Simultaneously, to ensure fairness, the multi-scale coding-decoding depth of the model corresponding to the method of this invention was increased from two layers to three layers, maintaining consistency with the MUSE speech enhancement model. The results show that although WB-PESQ decreased slightly by 0.04, the number of parameters and FLOPS decreased to approximately 50% of the MUSE speech enhancement model, highlighting the advantage of the model corresponding to the method of this invention in maintaining competitive performance while significantly reducing computational overhead.

[0134] In summary, both DenseEncoder and SEAdaConvBlock can improve speech enhancement quality while optimizing model complexity. The model corresponding to the method in this invention successfully achieves a balance between high performance and high efficiency, making it particularly suitable for low-resource, real-time deployment scenarios.

[0135] Table 5 Ablation experiments based on the VoiceBank+DEMAND dataset

[0136]

[0137] The test results of the models used in the experiments of this invention are derived from the following papers:

[0138] SEGAN [2] From PASCUAL S, BONAFONTE A, SERRÀ J. SEGAN: Speech EnhancementGenerative Adversarial Network[C / OL] / / Interspeech 2017. 2017: 3642-3646. DOI:10.21437 / Interspeech.2017-1428.

[0139] TSTNN [3]From WANG K, HE B, ZHU W P. TSTNN: Two-Stage Transformer Based Neural Network for Speech Enhancement in the Time Domain[C / OL] / / ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). 2021: 7098-7102. DOI:10.1109 / ICASSP39728.2021.9413740.

[0140] MP-SENET [7] From LU Y X, AI Y, LING Z H. MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra[C] / / Proc. Interspeech. 2023: 3834-3838.

[0141] CMGAN [8] From ABDULATIF S, CAO R, YANG B. CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement[J / OL]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 2477-2493. DOI:10.1109 / TASLP.2024.3393718.

[0142] DEMUCS

[18] From DEFOSSEZ A, SYNNAEVE G, ADI Y. Real Time Speech Enhancement in the Waveform Domain[C] / / Interspeech. 2020.

[0143] FullSubNet

[19] From HAO X, SU X, HORAUD R. Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement[C / OL] / / ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021: 6633-6637. DOI:10.1109 / ICASSP39728.2021.9414177.

[0144] CTSNet

[20] From LI A, LIU W, ZHENG C. Two Heads are Better Than One: A Two-Stage Complex Spectral Mapping Approach for Monaural Speech Enhancement[J / OL]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 1829-1843. DOI:10.1109 / TASLP.2021.3079813.

[0145] MetricGAN+

[21] From FU S W, YU C, HSIEH T A, et al. MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement[C / OL] / / Interspeech. 2021. https: / / api.semanticscholar.org / CorpusID:233181974.

[0146] MetricGAN-OKD

[22] From SHIN W, LEE B H, KIM J S. MetricGAN-OKD: multi-metric optimization of MetricGAN via online knowledge distillation for speech enhancement[C] / / International Conference on Machine Learning. PMLR, 2023:31521-31538.

[0147] ROSE

[23] From YU X, GUO D, ZHANG J. ROSE: A Recognition-Oriented Speech Enhancement Framework in Air Traffic Control Using Multi-Objective Learning[J / OL]. IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 3365-3378. DOI:10.1109 / TASLP.2024.3423652.

[0148] MANNER-S-5.3GF

[24] From SHIN W, PARK H J, KIM J S. Multi-View Attention Transfer for Efficient Speech Enhancement[C / OL] / / KO H, HANSEN J H L. Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022. ISCA, 2022:1198-1202. https: / / doi.org / 10.21437 / Interspeech.2022-10251. DOI:10.21437 / Interspeech.2022-10251.

[0149] TaylorSENet

[25] From LI A, YOU S, YU G. Taylor, Can You Hear Me Now? ATaylor-Unfolding Framework for Monaural Speech Enhancement[C / OL] / / RAEDT L D.Proceedings of the Thirty-First International Joint Conference on ArtificialIntelligence, IJCAI-22. International Joint Conferences on ArtificialIntelligence Organization, 2022: 4193-4200. https: / / doi.org / 10.24963 / ijcai.2022 / 582. DOI:10.24963 / ijcai.2022 / 582.

[0150] FRCRN

[26] From ZHAO S, MA B, WATCHARASUPAT K N. FRCRN: Boosting FeatureRepresentation Using Frequency Recurrence for Monaural Speech Enhancement[C / OL] / / ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP). 2022: 9281-9285. DOI:10.1109 / ICASSP43922.2022.9747578.

[0151] SE-Conformer

[36] From KIM E, SEO H. SE-Conformer: Time-Domain SpeechEnhancement Using Conformer[C / OL] / / Interspeech. 2021. https: / / api.semanticscholar.org / CorpusID:239684026.

[0152] MUSE

[37] Source: LIN Z, CHEN X, WANG J. MUSE: Flexible Voiceprint ReceptiveFields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-basedSpeech Enhancement[J / OL]. ArXiv, 2024, abs / 2406.04589. https: / / api.semanticscholar.org / CorpusID:270357676.

[0153] SEMamba

[38] Source: CHAO R, CHENG W H, QUATRA M L. An Investigation ofIncorporating Mamba For Speech Enhancement[C / OL] / / 2024 IEEE Spoken LanguageTechnology Workshop (SLT). 2024: 302-308. DOI:10.1109 / SLT61566.2024.10832332.

Claims

1. A speech enhancement method based on deformable embedding and time-frequency channel combination, characterized in that, include: S1: Extract the amplitude spectrum and phase spectrum of the speech signal to be processed, and concatenate the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map; S2: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map; The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then stitching together the results of multiple convolutions one by one. S3: Multi-scale feature maps calculate the offset of each time-frequency position through deformable convolution, and based on the offset, align features to generate an adaptive embedded feature map; S4: The adaptive embedding feature map is processed by multiple depthwise separable convolutions and residual processing to generate intermediate feature maps. The intermediate feature maps are processed by channel attention, temporal attention, and frequency attention respectively. After weighted fusion, they are applied to the intermediate feature maps to generate enhanced feature maps. S5: After multiple upsamplings, the enhanced feature map generates a complex spectrum, which is then reconstructed into a time-domain waveform through inverse short-time Fourier transform.

2. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 1, characterized in that, The adaptive embedding feature map described in S4 undergoes multiple depthwise separable convolutions and residual processing to generate an intermediate feature map, specifically: After the adaptive embedding feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to obtain the first feature map. After the first feature map is subjected to multiple depthwise separable convolutions, the residuals of the two features are added together to generate the intermediate feature map.

3. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 2, characterized in that, The multiple depthwise separable convolutions are specifically as follows: The feature map to be processed is sequentially processed by depthwise separable convolution, group normalization, and PReLU activation function, and then the process is repeated sequentially.

4. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 1, characterized in that, The generation of the adaptive embedding feature map described in S3 is specifically as follows: After the multi-scale feature map is processed by depthwise separable convolution, the offset of each time-frequency position is calculated by deformable convolution. Based on the offset, irregular sampling is performed on the multi-scale feature map, and the sampling results are reprojected onto the feature map after depthwise separable convolution to generate an adaptive embedding feature map.

5. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 1, characterized in that, The phase spectrum described in S1 is concatenated with the compressed amplitude spectrum to generate a time-frequency feature map, specifically: The amplitude spectrum is compressed using a power law to obtain the compressed amplitude spectrum; After the phase spectrum is converted into a dual-channel form of sine and cosine, it is spliced ​​with the compressed amplitude spectrum to generate a time-frequency feature map.

6. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 1, characterized in that, The initial convolution described in S2 is the processing of the time-frequency feature map through a first two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function.

7. The speech enhancement method based on deformable embedding and time-frequency channel combination according to claim 1, characterized in that, The downsampling described in S2 involves sequentially processing the data through a second two-dimensional convolution, two-dimensional instance normalization, and a linear rectified activation function.

8. A speech enhancement system based on deformable embedding and time-frequency channel combination, characterized in that, include: Time-frequency feature map generation module: Extracts the amplitude spectrum and phase spectrum of the speech signal to be processed, and concatenates the phase spectrum with the compressed amplitude spectrum to generate a time-frequency feature map; Multi-scale feature map generation module: The time-frequency feature map is sequentially processed through initial convolution, multi-scale dense connection processing, residual processing, and downsampling to generate a multi-scale feature map; The multi-scale dense connection processing involves performing a single convolution using convolution kernels with different dilation rates, and then stitching together the results of multiple convolutions one by one. Adaptive Embedded Feature Map Generation Module: The multi-scale feature map calculates the offset of each time-frequency position through deformable convolution. Based on the offset, the features are aligned to generate an adaptive embedded feature map. Enhanced feature map generation module: The adaptive embedded feature map is processed by multiple depthwise separable convolutions and residual processing to generate intermediate feature maps. The intermediate feature maps are then processed by channel attention, temporal attention, and frequency attention respectively, and weighted and fused and applied to the intermediate feature maps to generate enhanced feature maps. Reconstruction Module: After multiple upsampling of the enhanced feature map to generate a complex spectrum, it is reconstructed into a time-domain waveform through inverse short-time Fourier transform.

Images