Two-stage lightweight echo cancellation method and system combining NKF and EMA-GT convolution

Abstract

The invention discloses a two-stage lightweight echo cancellation method and system combining NKF and EMA-GT convolution. In the first stage of the method, the NKF is adopted to effectively remove linear echoes to obtain error signals after preliminary echo cancellation. In the second stage, grouping time convolution of EMA and a DPERNN network are combined to make full use of information in the first stage, and deeper feature extraction and voice signal reconstruction are carried out. In order to better reduce redundant information and improve the capability of a network to eliminate echoes, a TRA-based filter is designed, and the relation between a real part and an imaginary part in a complex spectrum is enhanced, so that the quality of reconstructed voice is improved.

Description

Technical Field

[0001] This invention belongs to the field of acoustic echo cancellation technology, and particularly relates to a two-stage lightweight echo cancellation method and system that combines NKF and EMA-GT convolution. Background Technology

[0002] Acoustic echo cancellation technology aims to suppress echo interference to improve the quality and intelligibility of disturbed speech signals, and is indispensable in scenarios such as real-time voice communication and human-computer voice interaction. Deep learning-based echo cancellation methods, relying on excellent nonlinear fitting capabilities, have improved the processing efficiency for nonlinear echoes and have become the mainstream technical solution. Among them, multi-level echo cancellation network structures have attracted widespread attention due to their superior performance. This model adopts a hierarchical progressive architecture, progressively refining the echo signal through multiple layers of networks. The first level initially separates the main components of speech and echo, and subsequent levels further process residual echoes and subtle features based on the results of the previous level to achieve deep suppression, significantly improving speech purity.

[0003] However, this model has the limitation of high computational resource consumption. The multi-level complex structure requires each level to perform a large number of operations, which places high demands on hardware computing performance. In practical applications, it may increase processing latency and power consumption, limiting its promotion in resource-constrained devices.

[0004] Although researchers have made many improvements to the model network of the two-stage acoustic echo cancellation method and achieved significant echo cancellation effects, the high number of parameters and computational complexity of the two-stage echo cancellation network make it difficult to apply in situations with limited computing resources. Summary of the Invention

[0005] To address the problems of existing technologies, this invention provides a two-stage lightweight echo cancellation method and system that combines NKF and EMA-GT convolution. It aims to solve the technical problem that existing two-stage acoustic echo cancellation methods are difficult to adapt to scenarios with low computing resources due to their large number of parameters and high computational complexity. From the perspective of suppressing linear and nonlinear echoes, a lightweight method for applying acoustic echo cancellation is designed.

[0006] The technical solution of this invention is implemented as follows:

[0007] A two-stage lightweight echo cancellation method combining NKF and EMA-GT convolutions includes a first stage and a second stage, wherein the first stage includes the following steps:

[0008] S11. Use the GCC-PHAT algorithm to perform delay compensation on the far-end reference signal;

[0009] S12, the neural Kalman filter NKF performs filtering in the time-frequency domain through short-time Fourier transform (STFT) to obtain the error signal;

[0010] The second phase includes the following steps:

[0011] S21. Extract STFT spectral features from the near-end speech signal, error signal, and far-end reference signal;

[0012] S22. Construct an echo cancellation network based on the amplitude spectrum and complex spectrum of STFT;

[0013] S23. In the Unet-based echo cancellation network, design an encoder and decoder based on the EMA-GTConv module to extract and reconstruct the features of the clean speech signal;

[0014] S24. Design a dual-path grouped recurrent neural network (DPERNN) as the bottleneck layer of the echo cancellation network;

[0015] S25. Design a TRA-based filter to eliminate feature redundancy and enhance the connection between the real and imaginary parts;

[0016] S26. Simultaneously design a joint loss function to train the two-stage echo cancellation method;

[0017] S27. After the decoder outputs the complex value mask, it is multiplied with the short-time Fourier transform (STFT) spectrum of the near-end microphone signal to obtain the estimated clean speech spectrum after denoising and echo removal.

[0018] S28. Use inverse short-time Fourier transform to convert the clean speech spectrum into a time-domain signal.

[0019] Furthermore, in the first stage, the GCC-PHAT algorithm is used to compensate for the delay of the far-end reference signal, specifically including:

[0020] Calculate the PATH weight function φ(ω):

[0021]

[0022] Where X1(ω) and X2(ω) are the FFT forms of the near-end microphone signal and the far-end reference signal, respectively. * Indicates the calculation of conjugate transpose;

[0023] Calculate the generalized cross-correlation function R[τ]:

[0024]

[0025] The time delay between two signals can be estimated from R[τ]. Finally utilize Delay compensation is applied to the remote reference signal:

[0026]

[0027] in, This is an estimate of the time delay between two signals.

[0028] Furthermore, in the first stage, the neural Kalman filter (NKF) is filtered in the time-frequency domain using a short-time Fourier transform (STFT) to obtain the error signal, specifically including:

[0029] For an echo cancellation system containing a near-end microphone signal x[n], a near-end clean speech signal s[n], a far-end signal y[n], and a room impulse response h(n), its expression at time T is:

[0030]

[0031] Where τ is the delay of the past value of the far-end signal;

[0032] Transformed to the time domain via STFT:

[0033]

[0034] Where S[m,k], X[m,k], and Y[m,k] represent the near-end clean speech signal, microphone signal, and far-end echo signal at frame m and frequency k, respectively, L is the number of taps, * is the complex conjugate operation, and H[l,k] is the convolution transfer function of the echo path;

[0035] To further simplify it, we define:

[0036] y m,k =[y[m,k],…,y[mL,k]] T ;

[0037] h k = [H[0,k],…,H[L-1,k]] T ;

[0038] Among them, y m,k The time delay vector of the far-end reference signal constructed at frequency k in the m-th frame is composed of the STFT coefficients of the current and past L-1 time points, representing the input signal vector of the echo path, used in conjunction with the frequency domain filter h. k Perform inner product operations to estimate echo components; h k For the echo path, represent the frequency domain filter vector of the echo path at frequency k;

[0039] Assuming echo path h k Let be a random vector with zero mean, which follows a simplified first-order Markov model, represented by the state equation:

[0040] h m,k =Ahm-1,k +w m,k ;

[0041] Where A is a transition parameter close to 1, w m,k It is a noise vector, and w m,k ~N(0,Q) k ) follows a pattern with zero mean and a covariance matrix of Q k Gaussian distribution;

[0042] The observation function is expressed as:

[0043]

[0044] Where X[m,k] represents the near-end microphone signal in the observation function, h m,k S[m,k] represents the echo path in the observation function, and S[m,k] represents the near-end clean speech signal in the observation function.

[0045] The Kalman filter solution is expressed as:

[0046]

[0047] in, This represents the echo path vector of the prior estimate (prediction). Let P represent the echo path vector estimated posteriorly (after updating). m,k Denotes the prior error covariance matrix. Let e ​​represent the posterior error covariance matrix. m,k k represents the predicted echo signal. m,k Q represents the Kalman gain vector. k Indicates the variance of observation noise;

[0048] NKF (Neural Kalman Filter) uses a neural network to estimate the Kalman gain k. m,k And update the echo path at each time step, the specific process is as follows:

[0049]

[0050] Where f(·) represents neural network computation, e represents the Kalman gain calculated by the neural network. m,k This represents the predicted echo signal.

[0051] The neural network model consists of one complex fully connected layer (FC), two complex gated recurrent unit (GRU) layers, and two complex fully connected layers. This neural network model can effectively achieve Kalman gain.

[0052] The initial echo cancellation error signal e(n) can be obtained by subtracting the estimated echo signal from the near-end microphone signal; the first stage uses the pre-trained model NKF to estimate the error signal e(n).

[0053] Furthermore, in the second stage, S21 represents the near-end speech signal as a linear sum of clean speech s(t), echo signal e(t), and noise c(t), i.e.:

[0054] x(t) = s(t) + e(t) + c(t);

[0055] Where x(t) represents the near-end speech signal;

[0056] STFT spectral feature extraction is performed on the near-end speech signal, error signal, and far-end reference signal to convert the time-domain signal into frequency-domain information. The formula is as follows:

[0057]

[0058] Where X(n,k) is the complex frequency domain coefficient of the signal at the k-th frequency point in the n-th frame; x(m) is the time-domain input signal (near-end speech, error signal, or far-end reference signal); w(·) is the analysis window function; n is the frame index; k is the frequency point index (k=0,1,…,L-1); L is the FFT length; and j is the imaginary unit.

[0059] Furthermore, in S22, the construction of an echo cancellation network model based on the amplitude spectrum and complex spectrum of STFT specifically includes constructing a UNet echo cancellation network based on EMA-GTConv, extracting the amplitude spectrum and complex spectrum from the echo-containing time-domain signal through STFT, and removing echoes and noise in the time-frequency domain to restore clean speech.

[0060] The amplitude spectrum and complex spectrum features of the near-end speech signal, error signal and far-end reference signal are extracted and input into the echo cancellation network for processing.

[0061] The echo cancellation network adopts a five-layer symmetrical Unet structure. The first two layers of the encoder and decoder are composed of convolutional blocks, and the last three layers are processed using a convolutional architecture based on EMA-GTConv. The redundant input features are reduced by using an equivalent rectangular bandwidth ERB filter bank, and DPERNN is introduced in the bottleneck layer to enhance the feature processing capability of the network. At the same time, a TRA filter is constructed to reduce feature redundancy.

[0062] Furthermore, in S23, in the Unet-based echo cancellation network, the last three layers, based on the EMA-GTConv module, are designed as the encoder and decoder, responsible for complex value masking. cplx The extraction process includes:

[0063] The feature E output by the convolution block i The data is then fed into the EMA-GTConv module for processing.

[0064] Input feature E i The data is divided into two groups. One group remains unchanged without any processing, while the other group's features are processed by the SFE module and then subjected to a series of convolutional operations for deeper feature extraction. Finally, channel shuffling is performed to better fuse the speech feature information. The formula is expressed as follows:

[0065]

[0066] Among them, E i For the input feature map, Output features for each stage, The features are shuffled from the channels; GConv represents group convolution, DWConv represents depthwise separable convolution, BN is batch normalization, PReLU is parameterized linear rectified activation function, and Channel Shuffle is used to rearrange channels to enhance cross-group feature fusion.

[0067] The output features are processed by the efficient multi-scale attention mechanism EMA to further integrate feature information from different levels, thereby enhancing the model's adaptability to complex scenes.

[0068]

[0069] in, Features processed by EMA (Efficient Multiscale Attention Mechanism). These are the output features from the previous stage;

[0070] Obtain output features E i and The features are added along the channel dimension, and then the output features are processed. By shuffling the channels, the features of clean speech signals are further integrated:

[0071]

[0072] in, Features processed by EMA (Efficient Multiscale Attention Mechanism). Channel shuffle is used to rearrange channels to facilitate feature fusion after channel shuffling.

[0073] Furthermore, in S24, DPERNN is designed as the bottleneck layer of the network, dividing the features within each frame into several groups, with each group using a GRU unit to independently perform causal inter-frame modeling:

[0074]

[0075] Among them, E fg For the input features of the f-th group, For the corresponding output features; BiGRU stands for Bidirectional Gated Recurrent Unit, used for inter-frame temporal modeling; FC stands for Fully Connected Layer; LayerNorm is for layer normalization, stabilizing network training; Residual Connection +E fg Preserve the original information flow and improve gradient propagation efficiency.

[0076] Furthermore, in S25, designing a TRA-based filter to eliminate feature redundancy specifically includes:

[0077] The TRASF, or TRA Spectrum Filter, is designed to remove redundant information in the complex number field while strengthening the connection between the real and imaginary number fields, thus promoting the removal of redundant information.

[0078] TRASF first segments the real and imaginary spectra along the channel dimension, then uses the TRA module for feature enhancement, leveraging TRASF to enhance the expressiveness of the complex spectrum. Specific formulas include:

[0079] Real out =TRA(Real) - TRA(Imag);

[0080] Imag out =TRA(Imag) + TRA(Real);

[0081] Where Real and Imag are the real and imaginary features of the input complex spectrum, respectively. out and Imag out The output is enhanced real and imaginary parts; TRA represents the time-frequency domain attention module, which is used to enhance feature representation capabilities; by cross-modulating the real and imaginary part information, the overall expressiveness of the complex spectrum is improved.

[0082] Furthermore, step S26 specifically includes: employing a multi-objective joint loss function to optimize the training process of the signal reconstruction network. This loss function comprehensively considers both the frequency domain complex spectrum reconstruction accuracy and the simultaneous domain speech quality perception characteristics, and its expression is:

[0083] loss=30×(real_loss+imag_loss)+70×mag_loss+sisnr_loss;

[0084] Here, `real_loss` and `imag_loss` represent the real and imaginary part errors of the predicted and target signals in the short-time Fourier transform domain, respectively, used to constrain the overall approximation accuracy of the complex spectrum; `mag_loss` represents the reconstruction error of the amplitude spectrum, emphasizing the optimization of key auditory perception features; `sisnr_loss` is the scale-invariant signal-to-noise ratio loss, used to measure the fidelity and separation quality of the speech signal in the time domain. These terms are weighted and fused using preset weights to achieve coordinated optimization of frequency domain details and perceptual quality.

[0085] Furthermore, S27 specifically includes: outputting a complex-valued mask after speech reconstruction by the decoder. cplx Near-end voice Y cplx Multiplying by the complex mask yields the estimated clean speech spectrum:

[0086]

[0087] Among them, Y cplx For the complex spectrum input of near-end speech, Mask cplx This is used to mask the complex values ​​output by the decoder (containing amplitude and phase correction information). For the estimated clean speech complex spectrum;

[0088] Furthermore, S28 specifically includes: using the Inverse Short-Time Fourier Transform (ISTFT) to... and Reconstructing the time-domain signal; where, and for The real and imaginary parts.

[0089] A two-stage lightweight echo cancellation system combining NKF and EMA-GT convolutions is provided, which applies a two-stage lightweight echo cancellation method combining NKF and EMA-GT convolutions as described in any of the above-mentioned methods.

[0090] Compared with the prior art, the present invention achieves the following beneficial effects:

[0091] This invention provides a two-stage lightweight echo cancellation method and system that combines NKF and EMA-GT convolution. In the first stage, NKF effectively removes linear echoes, obtaining an error signal after preliminary echo cancellation. Targeted processing of linear components reduces the computational burden on subsequent networks. The second stage designs a network combining EMA-based grouped temporal convolution and DPERNN, fully utilizing the information from the first stage for deeper feature extraction and speech signal reconstruction. The lightweight design is achieved through the synergistic effect of EMA and grouped convolution, while DPERNN enhances temporal feature modeling capabilities. To further reduce redundant information and improve the network's echo cancellation capability, this invention designs a TRA-based filter to enhance the correlation between the real and imaginary parts of the complex spectrum, thereby improving the quality of the reconstructed speech. This invention significantly reduces the number of model parameters and computational complexity while maintaining echo cancellation effectiveness, making it suitable for acoustic echo cancellation tasks with low computational resources. Attached Figure Description

[0092] Figure 1 This is a flowchart of a two-stage lightweight echo cancellation method combining NKF and EMA-GT convolution provided in an embodiment of the present invention;

[0093] Figure 2 This is a block diagram of a two-stage lightweight echo cancellation system combining NKF and EMA-GT convolution provided in an embodiment of the present invention;

[0094] Figure 3 This is a network structure diagram based on EMA-GTConv and DPERNN provided in the embodiments of the present invention;

[0095] Figure 4 This is a diagram of the EMA-GTConv module provided in this embodiment of the invention;

[0096] Figure 5 This is a diagram of the DPERNN module provided in the embodiments of the present invention;

[0097] Figure 6 This is a diagram of the TRASF module provided in an embodiment of the present invention. Detailed Implementation

[0098] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0099] Example

[0100] like Figures 1 to 6 This paper presents a two-stage lightweight echo cancellation method and system that combines NKF (Neural Kalman Filter) and EMA-GT convolution. The first stage uses NKF to effectively remove linear echoes, obtaining an error signal after preliminary echo cancellation. The second stage uses STFT-based short-time Fourier transform (STFT) for speech feature extraction and reconstruction. A neural network combining EMA (Efficient Multi-Scale Attention Mechanism) grouped temporal convolution and DPERNN (Dual-Path Grouped RNN with Inter-Frame Path Extension) is designed for deeper feature extraction and speech signal reconstruction. A TRA (Temporal Recursive Attention)-based filter is designed. A joint loss function is used to train the network model, resulting in better echo cancellation performance.

[0101] The relevant technologies and their meanings related to the English abbreviations involved in this invention are explained as follows:

[0102] STFT Short-Time Fourier Transform;

[0103] The Neural Kalman Filter (NKF) is an advanced state estimation method that combines traditional Kalman filtering with neural networks. By integrating the advantages of model-driven and data-driven approaches, it significantly improves the estimation accuracy and robustness of nonlinear systems.

[0104] EMA's efficient multi-scale attention mechanism;

[0105] EMA-GTConv UNet is an improved neural network architecture for echo cancellation tasks, combining the exponential moving average (EMA) mechanism with the gated temporal convolution (GTConv) UNet structure. This network is designed to effectively handle echo problems in real-time audio signals, especially in complex acoustic environments.

[0106] Among them, the EMA (Exponential Moving Average) mechanism introduces EMA modules at different levels of the network to stabilize the feature learning process; it smooths feature representation through adaptive weighting of historical states; and it helps to suppress the influence of instantaneous noise and outliers.

[0107] The GTConv module (Gated Temporal Convolution) uses a gating mechanism to control the flow of temporal information; it expands the receptive field by combining dilated convolution; and it enhances feature extraction capabilities through parallel processing paths.

[0108] The UNet architecture, an encoder-decoder structure, preserves multi-scale information, skip connections maintain high-frequency details, and is implemented using 1D convolution for audio signals.

[0109] DPERNN (Dual-Path Enhanced Recurrent Neural Network) is a hybrid neural network architecture specifically designed for audio signal processing. It combines the advantages of convolutional operations and recurrent neural networks through parallel processing paths, making it particularly suitable for tasks such as echo cancellation that require capturing both local features and long-term dependencies.

[0110] GCC-PHAT Algorithm: GCC-PHAT (Generalized Cross-Correlation Phase Transform) is a time delay estimation algorithm based on generalized cross-correlation, mainly used in sound source localization and signal processing. Its core idea is to improve the robustness of time delay estimation in noisy and reverberant environments by using phase transformation and weighted cross-power spectrum.

[0111] TRASF module: TRA Spectrum Filter;

[0112] TRASF (Temporal-Recurrent Attention Spectrum Filter) is a hybrid filtering technique that combines temporal recursive processing and frequency domain attention mechanisms, specifically designed for speech enhancement and echo cancellation tasks. It improves speech signal processing quality by jointly optimizing time- and frequency-domain features.

[0113] TRA (Temporal-Recurrent Attention) is a hybrid neural network module that combines temporal modeling (RNN / Temporal CNN) and attention mechanisms, primarily used for processing temporal signals (such as speech, audio, and time-series data). Its core idea is to simultaneously capture local temporal dependencies and global contextual relationships, making it suitable for tasks such as speech enhancement, echo cancellation, and speech recognition.

[0114] FFT (Fast Fourier Transform) is an efficient algorithmic implementation of the Discrete Fourier Transform (DFT) used to convert time-domain signals into frequency-domain representations.

[0115] The Convolutional Transfer Function (CTF) is a core tool for describing the input-output relationship of linear time-invariant (LTI) systems in the frequency domain, and is particularly suitable for modeling and analyzing acoustic systems and communication channels.

[0116] ERB (Equivalent Rectangular Bandwidth) is an indicator used in auditory psychology and acoustics to describe the human ear's ability to resolve frequencies. It represents the bandwidth at a specific frequency by which the human ear can simplify the response characteristics of a filter (such as an auditory filter) to that of an ideal rectangular filter with the same peak gain and the same total energy transfer.

[0117] The Speech Feature Extraction Module (SFE) is an algorithm or system module used to extract representative acoustic features from raw speech signals. It is typically used as a front-end processing step in tasks such as speech recognition, speaker recognition, and sentiment analysis. Its core objective is to convert speech signals into low-dimensional, robust feature representations that are more suitable for machine learning models.

[0118] GRU (Gated Recurrent Unit) is a variant of recurrent neural network (RNN) for processing sequential data. It was proposed by Cho et al. in 2014 to solve the gradient vanishing / exploding problem of traditional RNNs, while being simpler and more computationally efficient than LSTM (Long Short-Term Memory) network.

[0119] A two-stage lightweight echo cancellation method combining NKF and EMA-GT convolutions includes a first stage and a second stage, wherein the first stage includes the following steps:

[0120] S11. The GCC-PHAT algorithm is used to compensate for the delay of the far-end reference signal; specifically including:

[0121] Calculate the PATH weight function φ(ω):

[0122]

[0123] Where X1(ω) and X2(ω) are the FFT forms of the near-end microphone signal and the far-end reference signal, respectively. * Indicates the calculation of conjugate transpose;

[0124] Calculate the generalized cross-correlation function R[τ]:

[0125]

[0126] The time delay between two signals can be estimated from R[τ]. Finally utilize Delay compensation is applied to the remote reference signal:

[0127]

[0128] in, This is an estimate of the time delay between two signals.

[0129] S12, the Neural Kalman Filter (NKF) performs filtering in the time-frequency domain using a Short-Time Fourier Transform (STFT) to obtain the error signal; specifically including:

[0130] For an echo cancellation system containing a near-end microphone signal x[n], a near-end clean speech signal s[n], a far-end signal y[n], and a room impulse response h(n), its expression at time T is:

[0131]

[0132] Where τ is the delay of the past value of the far-end signal;

[0133] Transformed to the time domain via STFT:

[0134]

[0135] Where S[m,k], X[m,k], and Y[m,k] represent the near-end clean speech signal, microphone signal, and far-end echo signal at frame m and frequency k, respectively, L is the number of taps, * is the complex conjugate operation, and H[l,k] is the convolution transfer function of the echo path;

[0136] To further simplify it, we define:

[0137] y m,k =[y[m,k],…,y[mL,k]] T ;

[0138]

[0139] Among them, y m,k The time delay vector of the far-end reference signal constructed at frequency k in the m-th frame is composed of the STFT coefficients of the current and past L-1 time points, representing the input signal vector of the echo path, used in conjunction with the frequency domain filter h. k Perform inner product operations to estimate echo components; h k For the echo path, represent the frequency domain filter vector of the echo path at frequency k;

[0140] Assuming echo path h k Let be a random vector with zero mean, which follows a simplified first-order Markov model, represented by the state equation:

[0141] h m,k =Ah m-1,k +w m,k ;

[0142] Where A is a transition parameter close to 1, w m,k It is a noise vector, and w m,k ~N(0,Q) k ) follows a pattern with zero mean and a covariance matrix of Q k Gaussian distribution;

[0143] The observation function (which repeats the far-end echo signal representation above) is expressed as:

[0144]

[0145] Where X[m,k] represents the near-end microphone signal, h m,k The echo path is represented by S[m,k], which represents the clean speech signal at the near end.

[0146] The Kalman filter solution is expressed as:

[0147]

[0148] in, This represents the echo path vector of the prior estimate (prediction). Let P represent the echo path vector estimated posteriorly (after updating). m,k Denotes the prior error covariance matrix. Let e ​​represent the posterior error covariance matrix. m,k k represents the predicted echo signal. m,k Q represents the Kalman gain vector. k Indicates the variance of observation noise;

[0149] NKF (Neural Kalman Filter) uses a neural network to estimate the Kalman gain k. m,k And update the echo path at each time step, the specific process is as follows:

[0150]

[0151] Where f(·) represents neural network computation, e represents the Kalman gain calculated by the neural network. m,k This represents the predicted echo signal.

[0152] The neural network model consists of one complex fully connected layer (FC), two complex gated recurrent unit (GRU) layers, and two complex fully connected layers. This neural network model can effectively achieve Kalman gain.

[0153] Subtracting the estimated echo signal from the near-end microphone signal yields the initial echo cancellation error signal e(n). The first stage uses the pre-trained model NKF to estimate the error signal e(n).

[0154] The second phase includes the following steps:

[0155] S21. Perform STFT spectral feature extraction on the near-end speech signal, error signal, and far-end reference signal; specifically including:

[0156] The near-end speech signal is represented as a linear sum of clean speech s(t), echo signal e(t), and noise c(t), i.e.:

[0157] x(t) = s(t) + e(t) + c(t);

[0158] Where x(t) represents the near-end speech signal;

[0159] STFT spectral feature extraction is performed on the near-end speech signal, error signal, and far-end reference signal to convert the time-domain signal into frequency-domain information. The formula is as follows:

[0160]

[0161] Where X(n,k) is the complex frequency domain coefficient of the signal at the k-th frequency point in the n-th frame; x(m) is the time-domain input signal (near-end speech, error signal, or far-end reference signal); w(·) is the analysis window function; n is the frame index; k is the frequency point index (k=0,1,…,L-1); L is the FFT length; and j is the imaginary unit.

[0162] S22. Construct an echo cancellation network model based on the amplitude spectrum and complex spectrum of STFT; specifically including,

[0163] A UNet echo cancellation network based on EMA-GTConv is constructed. The amplitude spectrum and complex spectrum are extracted from the echo-containing time-domain signal through STFT, and the echo and noise are removed in the time-frequency domain to restore clean speech.

[0164] The amplitude spectrum and complex spectrum features of the near-end speech signal, error signal and far-end reference signal are extracted and input into the echo cancellation network for processing.

[0165] The echo cancellation network adopts a five-layer symmetrical Unet structure. The first two layers of the encoder and decoder are composed of convolutional blocks, and the last three layers are processed using a convolutional architecture based on EMA-GTConv. The redundant input features are reduced by using an equivalent rectangular bandwidth ERB filter bank, and DPERNN is introduced in the bottleneck layer to enhance the feature processing capability of the network. At the same time, a TRA filter is constructed to reduce feature redundancy.

[0166] S23. In the Unet-based echo cancellation network, design an encoder and decoder based on the EMA-GTConv module to extract and reconstruct clean speech signals; the specific process includes:

[0167] The feature E output by the convolution block i The data is then fed into the EMA-GTConv module for processing.

[0168] Input feature E i The data is divided into two groups. One group remains unchanged without any processing, while the other group's features are processed by the SFE module and then subjected to a series of convolutional operations for deeper feature extraction. Finally, channel shuffling is performed to better fuse the speech feature information. The formula is expressed as follows:

[0169]

[0170]

[0171] Among them, E i For the input feature map, Output features for each stage, The features are shuffled channels; GConv represents group convolution, DWConv represents depthwise separable convolution, BN is batch normalization, PReLU is parameterized linear rectified activation function, and Channel Shuffle is used to rearrange channels to enhance cross-group feature fusion.

[0172] The output features are processed by the efficient multi-scale attention mechanism EMA to further integrate feature information from different levels, thereby enhancing the model's adaptability to complex scenes.

[0173]

[0174] in, Features processed by EMA (Efficient Multiscale Attention Mechanism). These are the output features from the previous stage;

[0175] Obtain output features E i and The features are added along the channel dimension, and then the output features are processed. By shuffling the channels, the features of clean speech signals are further integrated:

[0176]

[0177] in, Features processed by EMA (Efficient Multiscale Attention Mechanism). Channel shuffle is used to rearrange channels to facilitate feature fusion after channel shuffling.

[0178] S24. Design a dual-path grouped recurrent neural network (DPERNN) as the bottleneck layer of the echo cancellation network.

[0179] Specifically, DPERNN is designed as the bottleneck layer of the network, dividing the features within each frame into several groups, with each group using a GRU unit to independently perform causal inter-frame modeling:

[0180]

[0181] Among them, E fg For the input features of the f-th group, For the corresponding output features; BiGRU stands for Bidirectional Gated Recurrent Unit, used for inter-frame temporal modeling; FC stands for Fully Connected Layer; LayerNorm is for layer normalization, stabilizing network training; Residual Connection +E fg Preserve the original information flow and improve gradient propagation efficiency.

[0182] S25. Design a TRA-based filter to eliminate feature redundancy and enhance the connection between the real and imaginary parts; specifically including:

[0183] The TRASF, or TRA Spectrum Filter, is designed to remove redundant information in the complex number field while strengthening the connection between the real and imaginary number fields, thus promoting the removal of redundant information.

[0184] TRASF first segments the real and imaginary spectra along the channel dimension, then uses the TRA module for feature enhancement, leveraging TRASF to enhance the expressiveness of the complex spectrum. Specific formulas include:

[0185] Real out =TRA(Real) - TRA(Imag);

[0186] Imag out =TRA(Imag) + TRA(Real);

[0187] Where Real and Imag are the real and imaginary features of the input complex spectrum, respectively. out and Imag out The output is enhanced real and imaginary parts; TRA represents the time-frequency domain attention module, which is used to enhance feature representation capabilities; by cross-modulating the real and imaginary part information, the overall expressiveness of the complex spectrum is improved.

[0188] S26. Simultaneously design a joint loss function to train the two-stage echo cancellation method; specifically including:

[0189] A multi-objective joint loss function is employed to optimize the training process of the signal reconstruction network. This loss function comprehensively considers both the accuracy of frequency domain complex spectrum reconstruction and the perceptual characteristics of speech quality in the simultaneous domain, and its expression is:

[0190] loss=30×(real_loss+imag_loss)+70×mag_loss+sisnr_loss;

[0191] Here, `real_loss` and `imag_loss` represent the real and imaginary part errors of the predicted and target signals in the short-time Fourier transform domain, respectively, used to constrain the overall approximation accuracy of the complex spectrum; `mag_loss` represents the reconstruction error of the amplitude spectrum, emphasizing the optimization of key auditory perception features; `sisnr_loss` is the scale-invariant signal-to-noise ratio loss, used to measure the fidelity and separation quality of the speech signal in the time domain. These terms are weighted and fused using preset weights to achieve coordinated optimization of frequency domain details and perceptual quality.

[0192] S27. After speech reconstruction by the decoder, a complex value mask is output. The estimated clean speech spectrum is obtained by multiplying the complex value mask by the speech spectrum from the near-end microphone. Specifically, this includes:

[0193] After speech reconstruction by the decoder, a complex-valued mask is output. cplx Near-end voice Y cplx Multiplying by the complex mask yields the estimated clean speech spectrum:

[0194]

[0195] Among them, Y cplx For the complex spectrum input of near-end speech, Mask cplx This is used to mask the complex values ​​output by the decoder (containing amplitude and phase correction information). This is the estimated complex spectrum of clean speech.

[0196] S28. Use inverse short-time Fourier transform to convert the clean speech spectrum into a time-domain signal.

[0197] Specifically, the Inverse Short-Time Fourier Transform (ISTFT) is used to... and Reconstruct the time-domain signal; where, where, and for The real and imaginary parts.

[0198] A two-stage lightweight echo cancellation system combining NKF and EMA-GT convolutions is provided, which applies the two-stage lightweight echo cancellation method combining NKF and EMA-GT convolutions as described above.

[0199] This invention provides a two-stage lightweight echo cancellation method and system combining NKF and EMA-GT convolution. Inspired by research on deep learning networks under various low computational resource conditions, the first stage of this method uses NKF to effectively remove linear echoes, obtaining an error signal after preliminary echo cancellation. The second stage combines EMA's grouped temporal convolution and a DPERNN network to fully utilize the information from the first stage, performing deeper feature extraction and speech signal reconstruction. To better reduce redundant information and improve the network's echo cancellation capability, a TRA-based filter is designed to enhance the connection between the real and imaginary parts of the complex spectrum, thereby improving the quality of the reconstructed speech.

[0200] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A two-stage light-weighted echo cancellation method combined with NKF and EMA-GT convolution, characterized in that, The method comprises a first stage and a second stage, wherein the first stage comprises the following steps: S11, performing delay compensation on the far-end reference signal by using a GCC-PHAT algorithm; S12, performing filtering processing on the error signal in a time-frequency domain by a neural Kalman filter NKF through a short-time Fourier transform STFT; The second stage comprises the following steps: S21, performing STFT spectral feature extraction on the near-end speech signal, the error signal and the far-end reference signal; S22, constructing an echo cancellation network based on STFT amplitude spectrum and complex spectrum; S23, designing an encoder and a decoder based on an EMA-GTConv module in the echo cancellation network based on Unet, to extract and reconstruct the clean speech signal features; S24, designing a double-path grouping recurrent neural network as a bottleneck layer of the echo cancellation network; S25, designing a filter based on TRA to eliminate feature redundancy and enhance the connection between the real part and the imaginary part; S26, simultaneously designing a joint loss function to train the two-stage echo cancellation method; S27, multiplying the short-time Fourier transform spectrum of the near-end microphone signal by the complex-valued mask output by the decoder to obtain the estimated clean speech spectrum after denoising and echo cancellation; S28, converting the clean speech spectrum into a time-domain signal by using an inverse short-time Fourier transform.

2. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 1, characterized in that, In the first stage, the GCC-PHAT algorithm is used to perform delay compensation on the far-end reference signal, specifically including: Computing a PATH weight function where X1(ω) and X2(ω) are the FFT forms of the near microphone signal and the far reference signal, respectively, and * denotes the conjugate transpose computation; Calculate the generalized cross-correlation function R[τ]: The time delay estimate between the two signals can be obtained from R[τ] Finally, using Delay compensation for the far-end reference signal: wherein, is the time delay estimate between the two signals.

3. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 2, characterized in that, In the first stage, the neural Kalman filter NKF performs filtering processing in the time-frequency domain through the short-time Fourier transform STFT to obtain the error signal, specifically including: For an echo cancellation system, there are a near-end microphone signal x[n], a near-end clean speech signal s[n], a far-end reference signal y[n] and a room impulse response h[n], and the expression at time T is: Wherein, τ is the delay amount of the past value of the far-end signal; Through STFT transformation to the time domain: Wherein, S[m,k], X[m,k], Y[m,k] represent the near-end clean speech signal, the near-end microphone signal and the far-end echo signal at frame m and frequency k, respectively, L is the number of taps, * is the complex conjugate operation, and H[l,k] is the convolution transfer function of the echo path; To further simplify it, define: y m,k = [y[m, k],..., y[m - L, k]] T ; h k = [H[0, k],..., H[L - 1, k]] T ; where y m,k is the time-delay vector of the far-end reference signal constructed at the m-th frame, frequency k, consisting of the STFT coefficients at the current and past L-1 time instants, represents the input signal vector of the echo path, and is used to perform inner product operation with the frequency-domain filter h k to estimate the echo component; h k is the echo path, representing the echo path frequency-domain filter vector at frequency k. Assume echo path h k is a zero-mean random vector that follows a simplified first-order Markov model, represented by the state equation: h m,k = Ah m-1,k +w m,k ; where A is an overparameterization close to 1, w m,k is a noise vector, and w m,k ~ N(0, Q k ) is Gaussian with mean zero and covariance matrix Q k . The observation function can be expressed as: where X[m, k] represents the near microphone signal in the observation function, h m,k represents the echo path in the observation function, and S[m, k] represents the near clean speech signal in the observation function; The Kalman filter solution is expressed as: wherein, represents an a priori estimated echo path vector, represents an a posteriori estimated echo path vector, P m,k represents an a priori error covariance matrix, represents an a posteriori error covariance matrix, e m,k represents a predicted echo signal, k m,k represents a Kalman gain vector, Q k represents an observation noise variance; The NKF neural Kalman filter uses a neural network to estimate the Kalman gain k m,k and updates the echo path at each time step as follows: where f(·) represents a neural network computation, denotes the Kalman gain after the neural network computation, e m,k denotes the predicted echo signal; The neural network model is composed of 1 complex full connection layer FC, 2 complex gate recurrent unit layers GRU, and 2 complex full connection layers. The neural network model can better complete Kalman gain The estimated echo signal can be obtained by subtracting the estimated echo signal from the near-end microphone signal; the first stage uses a pre-trained model NKF to estimate the error signal e(n).

4. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 1, characterized in that, In the second stage, the S21 represents the near-end speech signal as a linear sum of a clean speech s(t), an echo signal e(t) and a noise c(t), that is: x(t)=s(t)+e(t)+c(t); Wherein, x(t) represents the near-end speech signal; The STFT spectral feature extraction is performed on the near-end speech signal, the error signal and the far-end reference signal, and the time-domain signal is converted into frequency-domain information, and the formula is: Wherein, X(n, k) is the complex frequency domain coefficient of the signal in the nth frame, the kth frequency point; x(m) is the time domain input signal; w(·) is the analysis window function; n is the frame index; k is the frequency point index(k=0, 1, …, L-1); L is the FFT length; j is the imaginary unit.

5. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 1, characterized in that, In the S22, an echo cancellation network model based on the STFT amplitude spectrum and the complex spectrum is constructed, specifically including: constructing an EMA-GTConv-based UNet echo cancellation network, extracting the amplitude spectrum and the complex spectrum from the time-domain signal containing the echo through the STFT, removing the echo and the noise in the time-frequency domain, and restoring the clean speech; The near-end speech signal, the error signal and the far-end reference signal are respectively extracted to input the amplitude spectrum and the complex spectrum features into the echo cancellation network for processing; The echo cancellation network adopts a Unet structure with five layers of symmetry, the first two layers of the encoder and the decoder adopt a convolution block, and the last three layers adopt a convolution architecture based on EMA-GTConv for processing; the input features are reduced through an equivalent rectangular bandwidth ERB filter bank, and the feature processing capability of the DPERNN enhancement network is introduced in the bottleneck layer; meanwhile, a TRA filter is constructed to reduce feature redundancy.

6. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 5, characterized in that, In the S23, in the echo cancellation network based on Unet, the EMA-GTConv module based on the last three layers of the encoder and the decoder is designed, which is responsible for the complex masking Mask cplx Extraction, the specific process includes: The features E output by the convolutional block i are fed into the EMA-GTConv module for processing; Input feature E i The input features are divided into two groups, one of which is not processed and remains unchanged, and the other of which is processed by the SFE module, and then a series of convolution operations are performed on the input features to extract them more deeply, and then a channel scrambling operation is performed to better fuse the speech feature information, which is expressed by the formula: wherein E i is the input feature map, is the output feature of each stage, is the feature after channel shuffle; GConv represents group convolution, DWConv represents depthwise separable convolution, BN is batch normalization, PReLU is parameterized linear rectifier activation function, and Channel Shuffle is used for rearranging channels to enhance cross-group feature fusion; The output features are processed through the EMA efficient multi-scale attention mechanism to further integrate the feature information at different levels, so as to enhance the adaptability of the model to complex scenes, that is: wherein, is the feature processed by the EMA high-efficiency multi-scale attention mechanism, is the output feature of the previous stage. get output features E i and addition in the channel dimension, followed by shuffling of the output features in the channel dimension, further fusing the speech signal features: wherein, is the feature processed by the EMA high-efficiency multi-scale attention mechanism, is the fusion feature after channel shuffling, Channel shuffle is used to rearrange the channels to facilitate feature fusion.

7. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 6, characterized in that, In the S24, the DPERNN is designed as the bottleneck layer of the network, and the features in each frame are divided into several groups, and each group uses a GRU unit to independently perform causal inter-frame modeling: where E fg is the input feature of the f-th group, is the corresponding output feature; BiGRU denotes a bidirectional gated recurrent unit for modeling inter-frame timing; FC is a fully connected layer; LayerNorm is a layer normalization to stabilize network training; the residual connection + E fg retains the original information flow and improves the efficiency of gradient propagation.

8. The two-stage light-weighted echo cancellation method combining NKF and EMA-GT convolution according to claim 1, characterized in that, In the S25, a TRA-based filter is designed to eliminate feature redundancy, specifically including: A TRASF, i.e., a TRA spectrum filter, is designed to remove the redundant information in the complex domain, and at the same time, to deepen the relationship between the real domain and the imaginary domain, and to promote the removal of redundant information; The TRASF first divides the real spectrum and the imaginary spectrum in the channel dimension, and then uses the TRA module for feature enhancement, and uses the TRASF to enhance the expressiveness of the complex spectrum, and the specific formula includes: Real out = TRA(Real) - TRA(Imag); Imag out = TRA(Imag) + TRA(Real); where Real and Imag are the real and imaginary parts of the input complex spectrum, respectively, Real out and Imag out are the output enhanced real and imaginary parts; TRA represents a time-frequency domain attention module for enhancing the feature representation capability; and the overall expressiveness of the complex spectrum is improved by cross-modulating the real and imaginary part information.

9. The two-stage lightweight echo cancellation method combined with NKF and EMA-GT convolution according to claim 1, in S26: a multi-objective joint loss function is used to optimize the training process of the signal reconstruction network; the loss function considers the frequency domain complex spectrum reconstruction accuracy and the time domain speech quality perception characteristics at the same time, and its expression is: Loss=30×(real_loss+imag_loss)+70×mag_loss+sisnr_loss; wherein, The real_loss and imag_loss respectively represent the real part and the imaginary part error of the predicted signal and the target signal in the short-time Fourier transform domain, and are used to constrain the overall approximation accuracy of the complex spectrum; the mag_loss represents the reconstruction error of the amplitude spectrum, and emphasizes the optimization of the key features of auditory perception; the sisnr_loss is a scale-invariant signal-to-noise ratio loss, which is used to measure the fidelity and separation quality of the speech signal in the time domain; each item is weighted and fused through a preset weight, so as to realize the collaborative optimization of the frequency domain details and the perceptual quality; In S27: complex-valued mask Mask is output after decoder speech reconstruction cplx , near-end speech Y cplx The estimated clean speech spectrum is obtained by multiplying the complex-valued mask where Y cplx is the complex spectral input of the near-end speech, Mask cplx is the complex-valued mask output by the decoder, containing both amplitude and phase correction information, is the estimated clean speech complex spectrum.

10. A two-stage lightweight echo cancellation system combined with NKF and EMA-GT convolution, which applies the two-stage lightweight echo cancellation method combined with NKF and EMA-GT convolution according to any one of claims 1 to 9.

Images