A real-time speech enhancement method and system based on causal time-frequency attention and parameter sharing frequency-time-frequency modeling

Abstract

The application discloses a real-time speech enhancement method and system based on causal time-frequency attention and parameter sharing frequency-time-frequency modeling, and belongs to the technical field of speech enhancement, and comprises the following steps: introducing time-frequency double-branch attention in a feature extraction stage to realize fine dynamic modeling of a speech signal, adopting a parameter sharing frequency-time-frequency recursive structure in a bottleneck layer to improve modeling efficiency, and guaranteeing that the model only depends on past and current frame information through strict causal design, so that the contradiction between real-time performance and light weight of a traditional speech enhancement model is solved.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement technology, and more specifically to a real-time speech enhancement method and system based on causal time-frequency attention and parameter-sharing time-frequency modeling. Background Technology

[0002] Speech enhancement technology aims to recover clear speech from noisy speech and is a crucial component in voice communication, speech recognition, hearing aids, and intelligent voice interaction systems. Early methods mainly relied on traditional signal processing, such as spectral subtraction and Wiener filtering. While these methods performed reasonably well in stable noisy environments, they were prone to speech distortion and "musical noise" artifacts in complex noisy scenarios. With the development of deep learning, neural network-based speech enhancement methods have gradually become mainstream. Models such as DCCRN, Conv-TasNet, and Demucs have significantly improved speech quality through deep convolutions and recurrent structures. However, these methods still have significant limitations: First, they lack sufficient time-frequency dependency modeling, as most networks only model in a single dimension of time or frequency, making it difficult to fully capture the two-dimensional dynamic features of speech signals. Second, they have a large number of model parameters and high computational complexity, often requiring millions of parameters and billions of multiply-accumulate operations, making it difficult to run in real time on embedded or mobile devices. Third, they lack rigorous causal design, with many structures relying on future frame information, resulting in excessive latency and failing to meet the requirements of real-time communication scenarios.

[0003] Therefore, how to provide a lightweight, low-latency, and causal real-time speech enhancement method and system is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a real-time speech enhancement method and system based on causal time-frequency attention and parameter-sharing time-frequency modeling. By introducing time-frequency dual-branch attention (TFA) in the feature extraction stage, it achieves fine dynamic modeling of speech signals. In the bottleneck layer, it adopts a parameter-sharing frequency-time-frequency recursive structure (FTF) to improve modeling efficiency. Furthermore, through strict causal design, it ensures that the model depends only on past and current frame information, thus resolving the contradiction between real-time performance and lightweight design in traditional speech enhancement models.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: On one hand, this invention provides a real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling, comprising: Acquire the raw speech signal; The temporal features of the original speech signal are extracted using the causal time encoder, and a time feature vector is generated. The original speech signal is mapped to the frequency domain using a short-time Fourier transform; The transformed original speech signal is input into a causal encoder for frequency domain feature extraction. The alternating dependencies of frequency domain features in time and frequency are obtained, and time-frequency joint modeling is performed to obtain bottleneck layer features. The bottleneck layer features are linearly modulated channel by channel using the time feature vector to obtain fused features; The fused features are decoded using a causal decoder, and the complex domain mask is predicted using a complex mask output layer to correct the input complex spectrum. The corrected complex spectrum is then subjected to inverse short-time Fourier transform to obtain a time-domain waveform, resulting in enhanced speech.

[0006] Preferably, the causal temporal coding module includes multiple one-dimensional causal convolutional layers, with each convolutional layer followed by batch normalization and a non-linear activation function; global average pooling is performed on the features output from each of the multiple layers, and the pooling results are concatenated and then passed through a fully connected layer to generate a global temporal feature vector.

[0007] Preferably, the transformed original speech signal is input into a causal encoder for frequency domain feature extraction, specifically including: The causal encoder is composed of multiple layers of causal coding blocks stacked together. Each causal coding block includes two layers of causal depthwise separable convolutions. The depthwise separable convolutions are composed of depthwise convolutions and pointwise convolutions. The depthwise convolutions are used to extract local time-frequency features, and the pointwise convolutions are used for channel fusion. Each layer of the coding block embeds a time-frequency attention mechanism, which is used to calculate attention weights for the input features in the time and frequency dimensions and perform fusion modulation.

[0008] Preferably, the time-frequency attention mechanism includes: The input features are adaptively pooled along the frequency dimension and processed using one-dimensional causal convolution to obtain temporal attention weights, where the convolution output is cropped to future frames to ensure causality. Adaptive pooling is performed along the time dimension on the input features, and one-dimensional convolution is used to obtain frequency attention weights; The time attention weights and frequency attention weights are broadcast and multiplied to generate a two-dimensional time-frequency attention map, and the input features are weighted accordingly.

[0009] Preferably, the step of obtaining the alternating dependencies of frequency domain features in time and frequency, performing time-frequency joint modeling, and obtaining bottleneck layer features includes: The frequency domain features output by the causal encoder are input into the parameter-sharing frequency-time-frequency modeling module; The frequency-time modeling module performs the following three stages: The first stage involves expanding the input frequency domain feature sequence along the frequency dimension, modeling the energy coupling relationship between different frequency bands through the first gated cyclic unit, and obtaining the frequency structure dependence of the speech signal. The second stage involves unfolding the input frequency domain feature sequence along the time dimension and modeling the temporal dynamics and pronunciation change patterns of the speech signal through the second gated cyclic unit. Third stage: Expand along the frequency dimension again, and perform spectral consistency correction on the modeling results through the third gated loop unit; The first gated loop unit, the second gated loop unit, and the third gated loop unit share weight parameters. The output of each stage is mapped back to the original channel dimension through linear transformation and fused with the input features using residual connection.

[0010] Preferably, the parameter-sharing frequency-time-frequency modeling module further includes: Layer normalization units are set before and after the input of each stage to align the statistical characteristics in the frequency domain and time domain; The channel attention unit is activated when the number of input feature channels is greater than or equal to a preset threshold. It adaptively scales the channel response through one-dimensional convolution.

[0011] Preferably, the step of using the time feature vector to perform channel-by-channel linear modulation on the bottleneck layer features to obtain fused features includes: Scaling factors are generated based on the time feature vector through the first fully connected layer; The offset coefficients are generated based on the time feature vector through the second fully connected layer; The bottleneck layer features are linearly modulated channel by channel, and the calculation formula is as follows:

[0012] Where x is the bottleneck feature, scale is the scaling factor, shift is the offset factor, and x′ is the modulated fusion feature.

[0013] Preferably, the step of decoding the fused features using a causal decoder and predicting the complex domain mask using a complex mask output layer includes: The causal decoder is composed of multiple stacked decoder blocks. Each decoder block is upsampled through deconvolution to gradually restore the time-frequency resolution. Starting from the second layer decoder block, each layer decoder block receives skip connection features from the corresponding layer of the encoder and fuses them with the upsampled features; A causal depthwise separable convolutional structure is used to process the fused features; At the decoder output, the real and imaginary part masks are predicted separately using a complex ratio mask prediction layer, and the input complex spectrum is corrected using the following formula:

[0014] Where R and I are the real and imaginary parts of the input complex spectrum, respectively, and M... r and M i These are the predicted real and imaginary part masks, respectively, and α is the balance coefficient. and These are the corrected real and imaginary parts, respectively.

[0015] On the other hand, the present invention provides a real-time speech enhancement system based on causal time-frequency attention and parameter-sharing time-frequency modeling, comprising: The input module is used to acquire the raw speech signal; The causal time coding module is used to extract the temporal features of the original speech signal through the causal time encoder and generate a time feature vector; The STFT module is used to map the original speech signal to the frequency domain through short-time Fourier transform; The causal coding module is used to input the transformed original speech signal into the causal encoder for frequency domain feature extraction; The parameter-sharing time-frequency modeling module is used to obtain the alternating dependencies of frequency domain features in time and frequency, perform time-frequency joint modeling, and obtain bottleneck layer features. The time feature fusion module is used to perform channel-by-channel linear modulation on the bottleneck layer features using the time feature vector to obtain fused features; The causal decoding module is used to decode the fused features using a causal decoder. The output module is used to predict the complex domain mask through the complex mask output layer, correct the input complex spectrum, and perform inverse short-time Fourier transform on the corrected complex spectrum to obtain a time-domain waveform, thereby obtaining enhanced speech.

[0016] As can be seen from the above technical solutions, compared with existing technologies, this invention discloses a real-time speech enhancement method and system based on causal time-frequency attention and parameter-sharing time-frequency modeling. It employs a depthwise separable convolution and parameter-sharing mechanism, resulting in a total model parameter count of only 0.064M and a computational load of approximately 0.241G MACs, significantly lower than existing models such as DCCRN and Demucs, making it suitable for embedded and mobile devices. Through causal convolution and a unidirectional GRU structure, this invention achieves a strict causal inference mechanism without requiring future frame information. At a 16kHz sampling rate, processing 5.13 seconds of audio takes only 0.243 seconds, with a real-time factor RTF of 0.047, and an inference speed approximately 20 times faster than traditional models, achieving true real-time speech enhancement. Furthermore, traditional models often focus on a single dimension (time or frequency). The TFA module proposed in this invention, through joint modeling of time and frequency attention branches, effectively captures the correlation of speech signals across different frequency bands and temporal contexts, improving enhancement quality and speech fidelity. The FTF module achieves low-cost global feature capture by sharing GRU weights and alternating modeling across frequency, time, and frequency stages, effectively improving noise suppression and speech detail preservation. The FiLM temporal feature modulation mechanism, combined with global temporal features extracted by CausalTimeEncoder, enables the model to adaptively adjust to the temporal context, thereby enhancing the naturalness and fluency of the speech. Furthermore, the model structure provided by this invention is general and can be extended to tasks such as speech separation, speech dereverberation, and echo cancellation, demonstrating high potential for practical applications. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the process provided by the present invention.

[0019] Figure 2 This is a structural schematic diagram provided for the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention discloses a real-time speech enhancement method and system based on causal time-frequency attention and parameter-sharing time-frequency modeling. It integrates two approaches: global feature extraction in the time domain and local structure modeling in the frequency domain. The U-Net framework is used to construct the backbone, and feature enhancement is performed through Time-Frequency Attention and parameter-sharing FTF modules, thereby achieving high-quality, low-latency speech noise suppression.

[0022] This invention discloses a real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling, such as... Figure 1 As shown, it includes: Acquire the raw speech signal; The CausalTimeEncoder extracts temporal features from the original speech signal and generates a temporal feature vector. The causal temporal coding module comprises multiple one-dimensional causal convolutional layers with different kernel sizes (7, 3, 3) to achieve multi-scale temporal modeling. Convolutional operations rely only on the current and past signal frames, thus strictly guaranteeing the causality and real-time performance of the model. Each convolutional layer is followed by batch normalization and a non-linear activation function, and deep convolution further captures local temporal dependencies. Global average pooling is performed on the features output from each layer to extract statistical features of the entire speech segment, including energy distribution, rhythmic patterns, and temporal envelope information. The pooling results are concatenated and then passed through a fully connected layer to generate a global temporal feature vector. This vector will be used in the subsequent temporal feature fusion module (FiLM) to dynamically modulate frequency domain bottleneck features, enabling frequency domain processing to have global temporal context awareness. This module's design gives the model strong temporal dependency modeling capabilities at the temporal domain level, providing stable global feature support for real-time speech enhancement.

[0023] After completing time-domain encoding, the system maps the waveform signal to the frequency domain using Short-Time Fourier Transform (STFT). The STFT process converts the time signal into a complex spectrum:

[0024] Where R represents the real part and I represents the imaginary part.

[0025] Subsequently, the real and imaginary parts are treated as two channels and concatenated into a two-dimensional tensor [B,2,F,T], which serves as the input to the frequency domain backbone network.

[0026] The core function of this step is to establish a time-frequency spatial representation, enabling the network to directly learn the structural differences between speech and noise in the frequency dimension, while preserving phase information to facilitate subsequent complex domain enhancement.

[0027] The transformed original speech signal is input into a causal encoder for frequency domain feature extraction; specifically, the features after STFT transformation are input into the causal encoder (CausalEncoderBlock) for frequency domain feature extraction. This module is the basic structural unit of the frequency domain backbone, consisting of two layers of causal depthwise separable convolutions (a combination of depthwise and pointwise convolutions). The depthwise convolutions are responsible for extracting local time-frequency features, while the pointwise convolutions are used for channel fusion, thereby reducing the number of model parameters while maintaining efficiency. Batch normalization and activation functions are applied after each convolution to enhance nonlinear representation capabilities. The module incorporates a time-frequency attention mechanism (TFA). TFA simultaneously branches and calculates attention weights in both the time and frequency dimensions. The time branch retains only time information through adaptive pooling in the frequency dimension and uses convolution to model time dependencies, followed by causal pruning to prevent leakage of information from future frames; the frequency branch models the collaborative relationship between frequency bands through pooling and convolution in the time dimension.

[0028] Finally, the time and frequency attention results are multiplied to form a two-dimensional time-frequency attention map, which performs point-by-point weighted modulation on the input features, thereby highlighting salient speech regions and suppressing background noise. Starting from the second layer, the output of each coded block serves both as input to the next layer and as a skip connection for feature fusion during the decoding stage. Through layer stacking and downsampling operations, this module progressively compresses the spectral resolution while preserving speech structure information, providing multi-scale features for the bottleneck layer.

[0029] The alternating dependencies of frequency domain features in time and frequency are obtained, and time-frequency joint modeling is performed to obtain bottleneck layer features. The bottleneck layer features are linearly modulated channel by channel using the time feature vector to obtain fused features; The fused features are decoded using a causal decoder, and the complex domain mask is predicted using a complex mask output layer to correct the input complex spectrum. The corrected complex spectrum is then subjected to inverse short-time Fourier transform to obtain a time-domain waveform, resulting in enhanced speech.

[0030] Furthermore, a CausalParameterSharedFTF (parameter-shared time-frequency modeling module) is set up after the encoder. This module aims to capture the alternating dependencies of the speech signal in both time and frequency dimensions, achieving a deeper level of time-frequency joint modeling. Internally, it employs a multi-set parameter-shared GRU (CausalSharedParameterGRU) structure. Its operation consists of three stages: First, the input feature sequence is expanded along the frequency dimension, and the energy coupling relationship between different frequency bands is modeled by GRU, thereby capturing the frequency structure dependence of speech; Next, the features are expanded along the time dimension, and another set of GRUs is used to model the temporal dynamics and pronunciation change patterns of speech; Finally, the modeling results are expanded along the frequency dimension again to correct for spectral consistency.

[0031] Each stage's output is mapped back to the original channel dimension via a linear transformation and fused with input features using residual connections to maintain information stability and gradient fluidity. To further enhance the model's dynamic adjustment capability, a lightweight attention module is introduced into the channel dimension when the number of channels is high, adaptively scaling the channel responses. The FTF module significantly reduces the model's computational complexity through a parameter-sharing mechanism, enabling the system to efficiently capture the dual-domain correlation of speech during real-time operation. This module's design allows the model to perform cyclic inference in the time-frequency space, i.e., to model time and frequency dependencies back and forth, enhancing the integrity and naturalness of the speech signal structure.

[0032] Following the bottleneck layer, the system employs the FiLM (Feature-wise Linear Modulation) module to achieve interactive modulation of time-domain and frequency-domain features. The FiLM module receives the global time vector output from the CausalTimeEncoder, generates scaling and shift coefficients through two fully connected layers, and performs linear modulation on the bottleneck features channel by channel. The specific calculation formula is as follows:

[0033] This fusion approach enables the frequency domain backbone to dynamically adjust its response based on the temporal context of the entire speech segment, adaptively enhancing speech at different speeds, intonations, and in noisy environments. Through this module, the model achieves deep fusion of global temporal semantics and local frequency domain structure, ensuring that the speech enhancement process not only focuses on instantaneous features but also maintains overall temporal coherence.

[0034] The temporally modulated features enter the CausalDecoderBlock decoding and reconstruction module, which progressively upsamples through multiple deconvolution layers to restore the time-frequency resolution. Starting from the second decoder block, each decoder block receives skip connection features from the corresponding layer of the encoder, ensuring sufficient fusion of high- and low-level information. The decoder internally employs the same causal depthwise separable convolutional structure as the encoder, ensuring that details are restored without introducing information from future frames, thus maintaining system real-time performance. The main task of this stage is to map the high-dimensional features abstracted from the bottleneck layer back to a feature space consistent with the original spectral size, preparing for output mask prediction.

[0035] After feature recovery is complete, the system performs complex domain enhancement through the Complex Ratio Mask (CRM) output module. The output layer predicts the real part mask M. r With the imaginary part mask M i The real and imaginary parts of the input complex spectrum are corrected using the following formula:

[0036] Here, α is the balance coefficient (0.2). The advantage of the CRM module lies in its ability to simultaneously correct the amplitude and phase information of speech, significantly improving speech naturalness and listening quality compared to traditional masking methods that only enhance amplitude. The enhanced complex spectrum is then converted back to a time-domain waveform via inverse short-time Fourier transform (iSTFT) to generate the final enhanced speech output. Since both STFT and iSTFT are differentiable operations, this system can be trained end-to-end, directly minimizing perceptual loss or time-domain signal-to-noise ratio loss at the waveform level.

[0037] A lightweight time-frequency attention mechanism with causality and two-dimensional decoupling fusion features: Compared with existing TFA-like structures, the TFA module used in this invention has three significant differences in its implementation: Causal pruning mechanism: In the time branch, by performing "future frame pruning" (removing look-ahead information) on the convolution output, a fully causal attention mapping is achieved, thereby ensuring the deployability of real-time speech enhancement; Lightweight structure: Adopting the AdaptiveAvgPool2d+ (1×3) and (3×1) small convolutional kernel structure, the computational overhead of traditional multi-head time-frequency cross attention is avoided, and the number of parameters is only 1 / 10 of that of traditional TFA; Branch decoupling and re-fusion design: Temporal attention and frequency attention are extracted to extract dynamic correlations respectively. Finally, a two-dimensional attention map is generated through broadcast multiplication, so that it naturally corresponds to the spectrogram structure and is suitable for embedding causal convolutional encoders.

[0038] Therefore, this module differs from publicly available literature not only in its structural form, but also in its real-time performance, computational efficiency, and dual-branch fusion strategy.

[0039] This invention provides a frequency-time-frequency alternating recursive modeling structure based on parameter sharing and its implementation method in a lightweight speech enhancement system: The FTF module is the core innovation of this invention, distinguishing it from traditional speech enhancement networks that only use RNNs in a single dimension. The uniqueness of this module is reflected in: Three-stage alternating modeling mechanism: adopts a three-stage modeling process of "frequency → time → frequency" to continuously capture global dependencies in the spectral and temporal domains; Parameter sharing design: The three-stage GRU submodules share weights to achieve semantic consistency and significant parameter compression; Cross-domain normalization strategy: Insert LayerNorm before and after the input at each stage to ensure the alignment of statistical characteristics in the frequency domain and time domain, thereby improving model stability; Lightweight attention modulation: When the number of channels is high (≥32), 1D convolutional attention modulation channel weights are automatically enabled to further improve information selectivity.

[0040] Experimental results show that this module only requires about 16 hidden_dim and 4 shared GRUs to achieve performance comparable to the traditional frequency-time model at 0.24G MACs.

[0041] A global time adaptive fusion mechanism combining causal time coding and FiLM modulation: In the field of speech enhancement, the FiLM modulation mechanism is introduced, using time-coded features as a dynamic adjustment signal: Temporal encoder design: Extract global features in the temporal domain through stacked 1D causal convolutions, and generate compressed temporal representations through global pooling and fully connected layers; FiLM modulation implementation: The time representation generates scale and shift parameters through a bilinear layer, and the bottleneck layer features are modulated channel by channel to achieve "adaptive enhancement of time context"; No future frame dependency: CausalTimeEncoder uses zero-latency convolutions to ensure strict causality and real-time feasibility of the model.

[0042] On the other hand, the present invention provides a real-time speech enhancement system based on causal time-frequency attention and parameter-sharing time-frequency modeling, such as... Figure 2 As shown, it includes: The input module is used to acquire the raw speech signal; The causal time coding module is used to extract the temporal features of the original speech signal through the causal time encoder and generate a time feature vector; The STFT module is used to map the original speech signal to the frequency domain through short-time Fourier transform; The causal coding module is used to input the transformed original speech signal into the causal encoder for frequency domain feature extraction; The parameter-sharing time-frequency modeling module is used to obtain the alternating dependencies of frequency domain features in time and frequency, perform time-frequency joint modeling, and obtain bottleneck layer features. The time feature fusion module is used to perform channel-by-channel linear modulation on the bottleneck layer features using the time feature vector to obtain fused features; The causal decoding module is used to decode the fused features using a causal decoder. The output module is used to predict the complex domain mask through the complex mask output layer, correct the input complex spectrum, and perform inverse short-time Fourier transform on the corrected complex spectrum to obtain a time-domain waveform, thereby obtaining enhanced speech.

[0043] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0044] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling, characterized in that, include: Acquire the raw speech signal; The temporal features of the original speech signal are extracted using the causal time encoder, and a time feature vector is generated. The original speech signal is mapped to the frequency domain using a short-time Fourier transform; The transformed original speech signal is input into a causal encoder for frequency domain feature extraction. The alternating dependencies of frequency domain features in time and frequency are obtained, and time-frequency joint modeling is performed to obtain bottleneck layer features. The bottleneck layer features are linearly modulated channel by channel using the time feature vector to obtain fused features; The fused features are decoded using a causal decoder, and the complex domain mask is predicted using a complex mask output layer to correct the input complex spectrum. The corrected complex spectrum is then subjected to inverse short-time Fourier transform to obtain a time-domain waveform, resulting in enhanced speech.

2. The real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling as described in claim 1, characterized in that, The causal temporal coding module contains multiple one-dimensional causal convolutional layers, each followed by batch normalization and a non-linear activation function. Global average pooling is performed on the features output from each layer, and the pooling results are concatenated and then passed through a fully connected layer to generate a global temporal feature vector.

3. The real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling as described in claim 1, characterized in that, The transformed original speech signal is input into a causal encoder for frequency domain feature extraction, specifically including: The causal encoder is composed of multiple layers of causal coding blocks stacked together. Each causal coding block includes two layers of causal depthwise separable convolutions. The depthwise separable convolutions are composed of depthwise convolutions and pointwise convolutions. The depthwise convolutions are used to extract local time-frequency features, and the pointwise convolutions are used for channel fusion. Each layer of the coding block embeds a time-frequency attention mechanism, which is used to calculate attention weights for the input features in the time and frequency dimensions and perform fusion modulation.

4. The real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling according to claim 3, characterized in that, The time-frequency attention quantum mechanism includes: The input features are adaptively pooled along the frequency dimension and processed using one-dimensional causal convolution to obtain temporal attention weights, where the convolution output is cropped to future frames to ensure causality. Adaptive pooling is performed along the time dimension on the input features, and one-dimensional convolution is used to obtain frequency attention weights; The time attention weights and frequency attention weights are broadcast and multiplied to generate a two-dimensional time-frequency attention map, and the input features are weighted accordingly.

5. The real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling according to claim 1, characterized in that, The process of obtaining the alternating dependencies of frequency domain features in time and frequency, performing joint time-frequency modeling, and obtaining bottleneck layer features includes: The frequency domain features output by the causal encoder are input into the parameter-sharing frequency-time-frequency modeling module; In the parameter-shared frequency time-frequency modeling module, the following three stages are performed: The first stage involves expanding the input frequency domain feature sequence along the frequency dimension, modeling the energy coupling relationship between different frequency bands through the first gated cyclic unit, and obtaining the frequency structure dependence of the speech signal. The second stage involves unfolding the input frequency domain feature sequence along the time dimension and modeling the temporal dynamics and pronunciation change patterns of the speech signal through the second gated cyclic unit. Third stage: Expand along the frequency dimension again, and perform spectral consistency correction on the modeling results through the third gated loop unit; The first gated loop unit, the second gated loop unit, and the third gated loop unit share weight parameters. The output of each stage is mapped back to the original channel dimension through linear transformation and fused with the input features using residual connection.

6. A real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling according to claim 5, characterized in that, The parameter-shared frequency time-frequency modeling module also includes: Layer normalization units are set before and after the input of each stage to align the statistical characteristics in the frequency domain and time domain; The channel attention unit is activated when the number of input feature channels is greater than or equal to a preset threshold. It adaptively scales the channel response through one-dimensional convolution.

7. A real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling as described in claim 1, characterized in that, The step of using the time feature vector to perform channel-by-channel linear modulation on the bottleneck layer features to obtain fused features includes: Scaling factors are generated based on the time feature vector through the first fully connected layer; The offset coefficients are generated based on the time feature vector through the second fully connected layer; The bottleneck layer features are linearly modulated channel by channel, and the calculation formula is as follows: Where x is the bottleneck feature, scale is the scaling factor, shift is the offset factor, and x′ is the modulated fusion feature.

8. The real-time speech enhancement method based on causal time-frequency attention and parameter-sharing time-frequency modeling according to claim 1, characterized in that, The process of decoding the fused features using a causal decoder and predicting the complex domain mask using a complex mask output layer includes: The causal decoder is composed of multiple stacked decoder blocks. Each decoder block is upsampled through deconvolution to gradually restore the time-frequency resolution. Starting from the second layer decoder block, each layer decoder block receives skip connection features from the corresponding layer of the encoder and fuses them with the upsampled features; A causal depthwise separable convolutional structure is used to process the fused features; At the decoder output, the real and imaginary part masks are predicted separately using a complex ratio mask prediction layer, and the input complex spectrum is corrected using the following formula: Where R and I are the real and imaginary parts of the input complex spectrum, respectively, and M... r and M i These are the predicted real and imaginary part masks, respectively, and α is the balance coefficient. and These are the corrected real and imaginary parts, respectively.

9. A real-time speech enhancement system based on causal time-frequency attention and parameter-sharing time-frequency modeling, characterized in that, include: The input module is used to acquire the raw speech signal; The causal time coding module is used to extract the temporal features of the original speech signal through the causal time encoder and generate a time feature vector; The STFT module is used to map the original speech signal to the frequency domain through short-time Fourier transform; The causal coding module is used to input the transformed original speech signal into the causal encoder for frequency domain feature extraction; The parameter-sharing time-frequency modeling module is used to obtain the alternating dependencies of frequency domain features in time and frequency, perform time-frequency joint modeling, and obtain bottleneck layer features. The time feature fusion module is used to perform channel-by-channel linear modulation on the bottleneck layer features using the time feature vector to obtain fused features; The causal decoding module is used to decode the fused features using a causal decoder. The output module is used to predict the complex domain mask through the complex mask output layer, correct the input complex spectrum, and perform inverse short-time Fourier transform on the corrected complex spectrum to obtain a time-domain waveform, thereby obtaining enhanced speech.

Images