A decoupled lightweight echo cancellation method

Abstract

The present application relates to the field of echo cancellation, and specifically relates to a decoupled lightweight echo cancellation method. The method firstly extracts speech features based on STFT for the near-end mixed speech signal and the far-end reference speech, splices to obtain the amplitude spectrum and the complex spectrum, constructs a decoupled double-branch network, and can simultaneously process the amplitude spectrum and the complex spectrum. Then, the dilated dense convolution (Dilated DenseNet) and the DPT module are used to capture local and global long-term dependencies. Then, the Dual-path Skip-Transformer module is designed to replace the subsequent stacked DPT module, which can reuse the output information of the DPT module, reduce the parameter quantity and the calculation quantity, and maintain excellent effect; in addition, a decoder with a redundant removal function is designed, the SCConv is used to remove redundant features, the feature representation is enhanced, the complex version SCCSF is designed in the complex spectrum branch, and double reconstruction is formed. The method only needs a small amount of model parameters and calculation, and is suitable for echo cancellation tasks under low computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of echo cancellation, in particular to a decoupled lightweight echo cancellation method. BACKGROUND

[0002] Echo cancellation technology aims to make voice communication free from echo and noise interference, which is crucial for applications such as long-distance voice calls and network video conferences. Traditional echo cancellation methods work well for linear echo, but they are powerless against nonlinear echo in practical applications. Deep learning-based end-to-end echo cancellation methods have improved their ability to handle nonlinear echo due to their strong nonlinear fitting capability, and have become the mainstream method in the field of echo cancellation. Decoupled echo cancellation methods are commonly used neural network architectures proposed in recent years. They generally decouple complex echo cancellation problems into several subtasks, with each subtask corresponding to a neural subnetwork, so that each neural network focuses on processing its own task. Decoupled echo cancellation methods can make full use of different types of input features, making the input network's message more abundant. Information exchange between subnetworks can also promote the fusion of different types of features and enable other subnetworks to use their information to facilitate their own feature learning. Common decoupled methods generally fall into two categories: one is a series connection, where speech is first roughly removed from linear echo by traditional processing methods, and then refined by neural networks to effectively handle nonlinear echo; the other is a parallel connection, where different features are fed into the network for optimization at the same time, strengthening information exchange between subnetworks and promoting feature fusion, which is beneficial to overall echo cancellation.

[0003] Although researchers have made many improvements to the model network of decoupled echo cancellation methods with relatively significant effects, the high parameter quantity and computational complexity of decoupled networks make it difficult to apply in low computing resource situations. Inspired by the remarkable success of the Transformer in various fields such as natural language processing, its strong context modeling capability can capture long-term dependency information in the time-frequency domain of speech, promoting echo removal in a global dimension. However, the addition of a Transformer module to a decoupled network will further increase the parameter quantity and computational complexity. SUMMARY

[0004] To address the deficiencies in the above problems, the present application provides a decoupled lightweight echo cancellation method. From the perspective of amplitude-phase decoupled neural networks, a lightweight method is designed for speech enhancement.

[0005] The technical solution for achieving the object of the present application is: based on short-time Fourier transform (STFT) to realize feature extraction and reconstruction of speech. Design Dual-path Skip-Transformer to fine-tune the information modeled by the Dual-path Transformer module, promote local and global feature fusion, design feature reconstruction module SCConv Complex Spectrum Filter at the decoding end to remove redundant information, the method comprises the following steps:

[0006] (1) Perform STFT spectral feature extraction on the near-end mixed speech and the far-end reference speech, and the formula of STFT is expressed as:

[0007]

[0008] (2) Construct an echo cancellation network model based on the decoupling of STFT amplitude spectrum and complex spectrum;

[0009] (3) Design Dual-path Skip-Transformer module to fine-tune the output results of the Dual-path Transformer module and enhance the local feature relationship of the speech.

[0010] (4) Design a decoder with redundant feature removal function, introduce SCConv in the amplitude branch, and design SCConv in the complex spectrum branch to guide the decoder to remove redundant features, enhance feature representation, and strengthen the feature fusion of real spectrum and imaginary spectrum in the complex spectrum branch.

[0011] (5) Output M m from the amplitude decoder and (M r , M i ) from the complex spectrum decoder. m p

[0012]

[0013] (6) Mask the spectrum Y m and Y p of the near-end mixed speech by the mask Msk m and Mask p from (5) to obtain the predicted near-end reference speech spectrum (this figure uses doublet speech, so the predicted near-end reference language is blank on both sides):

[0014]

[0015] Finally, use the inverse short-time Fourier transform ISTFT to convert the estimated near-end reference speech into a time-domain speech signal.

[0016] Compared with the prior art, the present application has the following advantages:

[0017] 1. The echo cancellation task is decoupled into a near-end mixed speech amplitude mask estimation and a complex spectrum mask estimation task, and two sub-networks are used to perform the two tasks to optimize the overall speech information. A single-layer Dual-path Transformer (DPT) module is used in the network bottleneck layer to capture the long-term dependency information of the speech time-frequency, and a module named Dual-path Skip-Transformer (DPST) is designed to fine-tune the information modeled by the DPT module, while maintaining excellent performance while reducing the parameter amount and computational complexity. In view of the redundant information generated by information interaction in the dual-path network, Spatial and Channel reconstruction Convolution (SCConv) is introduced at the amplitude decoding end, and SCConv Complex Spectrum Filter (SCCSF) is designed at the complex spectrum decoding end. SCConv removes the redundancy existing in the features from the perspectives of channel reconstruction and time-frequency domain reconstruction, and enhances the representation of important information in the features. In addition, the special convolution method of SCCSF can perform double reconstruction operations, which can simultaneously strengthen the connection between the real spectrum and the imaginary spectrum in the complex spectrum, so as to improve the echo cancellation performance of the network.

[0018] 2. The method first performs STFT-based speech feature extraction on the near-end mixed speech signal and the far-end reference speech, splices the amplitude spectrum and the complex spectrum to obtain the amplitude spectrum and the complex spectrum, and constructs a decoupled dual-branch network capable of simultaneously processing the amplitude spectrum and the complex spectrum. Then, Dilated DenseNet and DPT modules are used to capture local and global long-term dependencies. Then, a Dual-path Skip-Transformer module is designed to replace the subsequent stacked DPT modules. This module can reuse the output information of the DPT module, which reduces the parameter amount and computational complexity while maintaining excellent performance. In addition, a decoder with a redundancy removal function is designed, which uses SCConv to remove redundant features and enhance feature representation. In the complex spectrum branch, a complex version of SCCSF is designed to form double reconstruction, which effectively improves the echo cancellation effect. In order to effectively train the model, a power compression loss is used. Finally, the waveform is reconstructed using the inverse short-time Fourier transform ISTFT to obtain the estimated near-end reference speech signal. The method of the present application only needs a small amount of model parameters and computational complexity, and is suitable for echo cancellation tasks under low computational resources. BRIEF DESCRIPTION OF DRAWINGS

[0019] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings, in which:

[0020] Figure 1is a block diagram of an echo cancellation system of the present application;

[0021] Figure 2 is a Dual-path Skip-Transformer module of the present application;

[0022] Figure 3 is a SCConv module;

[0023] Figure 4 is a SCCSF module of the present application; DETAILED DESCRIPTION

[0024] Embodiments of the present application will be described in detail below with reference to the accompanying drawings:

[0025] Embodiments:

[0026] The embodiment discloses a decoupled lightweight echo cancellation method. From the perspective of amplitude and phase decoupled neural network, a lightweight echo cancellation method is designed to effectively improve the echo cancellation effect and promote local and global feature fusion. The method comprises the following steps:

[0027] Step one: STFT feature extraction is performed on the near-end mixed speech and the far-end reference speech. It is assumed that s(t), c(t) represent the near-end signal, the far-end signal and the impulse response of the room, respectively, and the near-end mixed speech y(t) can be represented as their linear sum:

[0028] y(t) = s(t) + c(t)

[0029] The time domain signal is converted into frequency domain information by STFT. The STFT formula is:

[0030]

[0031] Step two: a decoupled dual-path echo cancellation network is constructed. STFT is used to extract the amplitude spectrum and complex frequency spectrum from the near-end mixed speech and the far-end reference speech. After splicing, the network is fed back, and the near-end reference speech is restored in the time-frequency domain. The upper branch network processes the amplitude spectrum, and the lower branch processes the complex frequency spectrum. The Fusion module is embedded between the upper and lower paths. The module is distributed in different stages of the network, and real-time information interaction is completed in the form of amplitude addition. The two branches learn from each other and maintain their independence.

[0032] Step three: the basic structure of the network is composed of convolution blocks and dilated dense convolution. The architecture of the encoder and the decoder is symmetrical, and is responsible for feature extraction and reconstruction of the speech, respectively.

[0033] Step four: Based on the decoupled network structure, design the Dual-path Skip-Transformer module based on DPT to fine-tune the local and global long-term dependency information generated by the DPT module, reduce the parameter quantity and calculation quantity, DPST consists of two parts: residual iteration module and Skip Transformer module; The module DPST fine-tunes the long-term dependency output by the DPT module, the process of DPST mainly includes the following steps:

[0034] Step a: Save the local and global long-term dependencies generated by the DPT module And Into the DPST module;

[0035] Step b: Send And Into the Skip Transformer module for fine-tuning to strengthen the local information association of the two features:

[0036]

[0037] Step c: Among them, the Skip Transformer module in step b can enhance the adjacent information of the feature after restoring the original channel number, the feature is averaged in the channel dimension, then one-dimensional convolution is performed in the time dimension or frequency dimension, then Sigmoid function is used for activation, mapping to(0,1) Range to get a set of weights, finally, use these weights to scale the input feature;

[0038] Step d: Finally, the feature is normalized and reshaped back to CxTxF after residual connection, and the output feature is obtained, the overall expression is as follows:

[0039]

[0040] Step five: In order to make important features more prominent and remove redundant features more effectively, design a decoder containing embedded SCConv and SCCSF modules, which can perform reconstruction operation on the feature, reduce the parameter quantity and calculation quantity while increasing the convolution efficiency; SCConv can be divided into the following steps:

[0041] Step a: SCConv is composed of spatial reconstruction unit (Spatial Reconstruction Unit, SRU) and channel reconstruction unit (Channel Reconstruction Unit, CRU) in series;

[0042] Step b: The spatial reconstruction unit first evaluates the information content of the features by group normalization to obtain normalized correlation weights, and the reweighted features are mapped to (0, 1) through a Sigmoid function. Then, a gating mechanism is set up, and weights higher than the threshold are regarded as important features W1, and weights lower than the threshold are regarded as redundant features W2. Subsequently, W1, W2 are multiplied with the input features to obtain important features and redundant features Finally, the important features and the redundant features are cross-reconstructed to obtain the TF domain reconstructed features X w :

[0043] W = Gate (Sigmoid (W γ (GN (X))).

[0044] Step c: The features reconstructed by the SRU are divided into two parts from the channel dimension. The upper branch is the important channel dimension, and the lower branch is the relatively redundant features in the channel dimension. First, both branches are compressed from the channel dimension through 1x1 convolution to improve computational efficiency. Then, the upper branch uses group convolution and point-by-point convolution to add the features to obtain important features. The lower branch only uses cheap point-by-point convolution to splice the features before and after convolution to generate relatively redundant features as a supplement to the important features. Subsequently, global average pooling is used to generate channel soft attention. Finally, the features of the upper and lower branches are spliced in the channel dimension to obtain the channel-reconstructed features.

[0045] Step d: In the complex spectrum branch, SCConv cannot be directly used to process complex features. Therefore, SCConvComplex Spectrum Filter (SCCSF) is designed to remove redundant information in the complex domain and deepen the connection between the real and imaginary domains to promote the removal of redundant information. SCCSF first divides the real and imaginary spectra in the channel dimension, then uses SCConv for complex convolution. SCCSF enhances the representation ability of complex spectrum:

[0046] Real out = CUR r (SUR r (Real))-CUR i (SUR i (Imag))

[0047] Imag out = CUR r (SUR r (Imag))+CUR i (SUR i (Real).

[0048] Step six: Previous studies show that although the spectrum of the near-end reference speech can be directly estimated, better echo cancellation effect can be obtained by masking the spectrum of the near-end mixed speech. The amplitude decoder output M m and the complex spectrum decoder output (M r , M i ) are further combined to calculate the mask:

[0049]

[0050] The mask Mask m and Mask p mask the spectrum Y m and Y p of the near-end mixed speech to obtain the estimated spectrum of the near-end reference speech:

[0051]

[0052] Step seven: Finally, the inverse short-time Fourier transform ISTFT is used to reconstruct the time-domain signals from and .

[0053] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, various changes, modifications, replacements and variations can be made to these embodiments without departing from the technical principles of the present application, and these changes, modifications, replacements and variations should also be considered as falling within the protection scope of the present application.

Claims

1. A decoupled lightweight echo cancellation method, characterized in that, From the perspective of amplitude-phase decoupling neural networks, a lightweight method for applying echo cancellation is designed to effectively improve the echo cancellation effect and promote the fusion of local and global features. The method includes the following steps: Step 1: Perform STFT feature extraction on the near-end mixed speech and the far-end reference speech. s(t) and c(t) represent the near-end signal, the far-end signal, and the impulse response of the room, respectively. The near-end mixed speech y(t) can be represented as their linear sum: y(t) = s(t) + c(t) The STFT converts a time-domain signal into a frequency-domain information. The STFT formula is as follows: Step 2: Construct a decoupled dual-path echo cancellation network. STFT is used to extract amplitude spectrum and complex spectrum from near-end mixed speech and far-end reference speech. After splicing, the spectrum is fed into the network and restored to near-end reference speech in the time-frequency domain. The upper branch of the network processes the amplitude spectrum, and the lower branch processes the complex spectrum. A Fusion module is embedded between the upper and lower paths. This module is distributed in different stages of the network and completes real-time information interaction by adding amplitudes. The two branches learn from each other and maintain their independence. Step 3: The basic structure of the network consists of convolutional blocks and dilated dense convolutions. The encoder and decoder have symmetrical architectures and are responsible for the extraction and reconstruction of speech features, respectively. Step 4: Based on the decoupled network structure, design a DPT-based dual-path. The Skip-Transformer module is used to fine-tune the local and global long-term dependency information generated by the DPT module, reducing the number of parameters and computation. DPST consists of two parts: the residual iteration module and the Skip Transformer module. Step 5: In order to more effectively highlight important features and remove redundant features, a decoder containing embedded SCConv and SCCSF modules is designed. It can reconstruct features, increasing convolution efficiency while reducing the number of parameters and computation. Step Six: Further combine the amplitude decoder output M_m and the complex spectrum decoder output (M_r, M_i) to calculate the mask: Mask m and Mask p Spectral Y of masked near-end mixed speech m and Y p The estimated near-end reference speech spectrum is obtained: Step 7: Finally, use the Inverse Short Time Fourier Transform (ISTFT) to... and Reconstruct the time-domain signal.

2. The decoupled lightweight echo cancellation method according to claim 1, characterized in that: Step four involves fine-tuning the long-term dependencies of the DPT module output in the DPST module. The specific steps are as follows: Step a: Configure local and global long-term dependencies generated by the DPT module and Save it and send it to the DPST module; Step b: and The data is then fed into the Skip Transformer module for fine-tuning to enhance the correlation between the local information of the two features: Step c: In step b, the Skip Transformer module can enhance the neighbor information of the feature after the original number of channels is restored. The feature is averaged in the channel dimension, then one-dimensional convolution is performed in the time dimension or frequency dimension, then the Sigmoid function is used for activation, and a set of weights is obtained by mapping to the (0,1) range. Finally, these weights are used to scale the input feature. Step d: Finally, the features are grouped and normalized, then reconstructed back to C×T×F using residual connections to obtain the output features. The overall expression is as follows:

3. The decoupled lightweight echo cancellation method according to claim 1, characterized in that: Step five involves designing a decoder that includes embedded SCConv and SCCSF modules. The specific steps are as follows: Step a: SCConv is composed of Spatial Reconstruction Units (SRUs) and Channel Reconstruction Units (CRUs) connected in series; Step b: The spatial reconstruction unit first evaluates the information content of the features through group normalization to obtain normalized relevant weights. The reweighted features are then mapped to the range (0,1) using the Sigmoid function. Next, a gating mechanism is set, treating weights above a threshold as important features W1 and weights below a threshold as redundant features W2. Subsequently, W1 and W2 are multiplied by the input features to obtain the important features. and redundancy features Finally, the important features and redundant features are cross-reconstructed to obtain the TF domain reconstructed features X. w : W=Gate(Sigmoid(W γ (GN(X)))); Step c: The features reconstructed by the spatial reconstruction unit are divided into two parts from the channel dimension. The upper branch is the important channel dimension, and the lower branch is the relatively redundant feature in the channel dimension. First, both branches compress the features from the channel dimension through 1×1 convolution to improve computational efficiency. Then, the upper branch uses group convolution and pointwise convolution to add the features to obtain the important features. The lower branch only uses the inexpensive pointwise convolution and concatenating the features before convolution to generate relatively redundant features as a supplement to the important features. Then, global average pooling is used to generate channel soft attention. Finally, the features of the upper and lower branches are concatenated according to the channel dimension to obtain the channel reconstructed features. Step d: Since SCConv cannot be directly used to process complex features in the complex spectrum branch, the SCConv ComplexSpectrum Filter (SCCSF) was designed to remove redundant information in the complex domain and deepen the connection between the real and imaginary domains, mutually promoting the removal of redundant information. SCCSF first segments the real and imaginary spectra along the channel dimension, and then uses SCConv for complex convolution. SCCSF enhances the representation capability of the complex spectrum. Real out *CUR r (ON r (Real)-CUR i (ON i (Image) Image out *CUR r (ON r (Imag)+CUR i (ON i (Real)).

Images