Decoupling type lightweight echo cancellation method

Abstract

The invention relates to the field of echo cancellation, in particular to a decoupling type lightweight echo cancellation method. According to the method, firstly, STFT-based voice feature extraction is carried out on near-end mixed voice signals and far-end reference voice, an amplitude spectrum and a complex frequency spectrum are obtained after splicing, a decoupling type double-branch network is constructed, and the amplitude spectrum and the complex frequency spectrum can be processed at the same time. And then, capturing local and global long-term dependencies by using an expanded dense convolution (Dilated DenseNet) module and a DPT (Discrete Potential Transform) module. Secondly, a dual-path Skip-Transform module is designed to replace a subsequently stacked DPT module, the module can reuse output information of the DPT module, the parameter quantity and the calculation quantity are reduced, and meanwhile the excellent effect is kept; in addition, a decoder with a redundancy removal function is designed, SCConv is used for removing redundancy features and enhancing feature representation, and a plurality of versions of SCCSF are designed in a complex frequency spectrum branch to form dual reconstruction. According to the method, only few model parameters and calculation amount are needed, and the method is suitable for echo cancellation tasks under low calculation resources.

Description

Technical Field

[0001] The invention relates to the field of echo cancellation, and in particular to a decoupled lightweight echo cancellation method. Background Art

[0002] Echo cancellation technology aims to free voice communication from echo and noise interference, which is crucial for applications such as remote voice calls and network video conferencing. Traditional echo cancellation methods work well for linear echoes, but they are powerless against nonlinear echoes in practical applications. The end-to-end echo cancellation method based on deep learning has improved the processing ability of nonlinear echoes with its powerful nonlinear fitting ability, and has become the mainstream method in the field of echo cancellation. The decoupled echo cancellation method is a common neural network architecture proposed in recent years. It generally decouples the complex echo cancellation problem and divides it into several subtasks. Each subtask corresponds to a neural subnetwork, so that each neural network focuses on processing its own task. The decoupled echo cancellation method can make full use of different types of input features to make the input network messages richer. The subnetworks can also exchange information, promote the fusion of different types of features, and use the information of other subnetworks to promote the learning of their own features. There are generally two common decoupling methods. One is a series connection, in which the speech is first roughly removed of linear echoes through traditional processing methods, and then refined through a neural network to effectively process nonlinear echoes. The other is a parallel connection, in which different features are fed into the network at the same time for optimization. While processing in parallel, the information interaction between sub-networks is strengthened, feature fusion is promoted, and it is beneficial to the overall elimination of echoes.

[0003] Although researchers have made many improvements to the model network of the decoupled echo cancellation method and the results are quite significant, the decoupled network contains many sub-networks, resulting in a high number of parameters and computational complexity, making it difficult to apply in situations with low computing resources. Inspired by the remarkable success of Transformer in various fields such as natural language processing, its powerful context modeling ability can well capture the long-term dependency information of speech in the time and frequency domain, and promote echo removal in the global dimension. However, adding the Transformer module to the decoupled network will further increase the number of parameters and computational complexity. Summary of the invention

[0004] In view of the shortcomings of the above problems, the present invention provides a decoupled lightweight echo cancellation method. From the perspective of amplitude-phase decoupled neural network, a lightweight method for speech enhancement is designed.

[0005] The technical solution to achieve the purpose of the present invention is: to realize the feature extraction and reconstruction of speech based on short-time Fourier transform (STFT). Dual-path Skip-Transformer is designed to fine-tune the information of Dual-path Transformer module modeling, promote local and global feature fusion, and design a feature reconstruction module SCConv Complex Spectrum Filter at the decoding end to remove redundant information. The method includes the following steps:

[0006] (1) STFT spectrum feature extraction is performed on the near-end mixed speech and the far-end reference speech. The formula of STFT is expressed as:

[0007]

[0008] (2) Construct an echo cancellation network model based on STFT amplitude spectrum and complex spectrum decoupling;

[0009] (3) Design a Dual-path Skip-Transformer module to fine-tune the output results of the Dual-path Transformer module and enhance the local feature relationship of speech.

[0010] (4) Design a decoder with redundant feature removal function. Introduce SCConv in the amplitude branch and design SCCSF in the complex spectrum branch to guide the decoder to remove redundant features, enhance feature representation, and strengthen the feature fusion of real spectrum and imaginary spectrum in the complex spectrum branch.

[0011] (5) The amplitude decoder output M m and the complex spectrum decoder output (M r ,M i ) is further combined to calculate the mask:

[0012]

[0013] (6) The mask Msk in (5) m and Mask p Mask the spectrum Y of the near-end mixed speech m and Y p Get the predicted near-end reference speech spectrum (this figure uses double-talk speech, so the predicted near-end reference language is blank on both sides):

[0014]

[0015] Finally, the estimated near-end reference speech is converted into a time-domain speech signal using the inverse short-time Fourier transform (ISTFT).

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. Decouple the echo cancellation task into near-end mixed speech amplitude mask estimation and complex spectrum mask estimation tasks, and use two sub-networks to perform these two tasks respectively to optimize the overall speech information. The network bottleneck layer uses a single-layer Dual-path Transformer (DPT) module to capture the long-term dependency information of speech time and frequency. A module called Dual-path Skip-Transformer (DPST) is designed to fine-tune the information modeled by the DPT module, while reducing the number of parameters and computational complexity while maintaining excellent performance. In view of the redundant information generated by information interaction in the dual-path network, Spatial and Channel reconstruction Convolution (SCConv) is introduced at the amplitude decoding end, and SCConv Complex Spectrum Filter (SCCSF) is designed at the complex spectrum decoding end. SCConv removes the redundancy in the features from the perspective of channel reconstruction and time-frequency domain reconstruction, and enhances the representation of important feature information. In addition, the special convolution method of SCCSF can perform dual reconstruction operations, which can simultaneously strengthen the connection between the real spectrum and the imaginary spectrum in the complex spectrum to improve the network echo cancellation performance.

[0018] 2. This method first extracts speech features based on STFT for the near-end mixed speech signal and the far-end reference speech, obtains the amplitude spectrum and the complex spectrum after splicing, and constructs a decoupled dual-branch network that can process the amplitude spectrum and the complex spectrum at the same time. Then, the dilated dense convolution (Dilated DenseNet) and DPT modules are used to capture local and global long-term dependencies. Then, the Dual-path Skip-Transformer module is designed to replace the subsequent stacked DPT module. The module can reuse the output information of the DPT module, which not only reduces the number of parameters and the amount of calculation, but also maintains excellent results; in addition, a decoder with redundancy removal function is designed, and SCConv is used to remove redundant features and enhance feature representation. A complex version SCCSF is designed in the complex spectrum branch to form a dual reconstruction, which effectively improves the echo cancellation effect. In order to effectively train the model, power compression loss is used. Finally, the inverse short-time Fourier transform (ISTFT) is used to reconstruct the waveform to obtain the estimated near-end reference speech signal. The method of the present invention requires only a small number of model parameters and calculations, and is suitable for echo cancellation tasks under low computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings, wherein:

[0020] Figure 1is a block diagram of an echo cancellation system of the present invention;

[0021] Figure 2 is the Dual-path Skip-Transformer module of the present invention;

[0022] Figure 3 It is the SCConv module;

[0023] Figure 4 is the SCCSF module of the present invention; DETAILED DESCRIPTION

[0024] The embodiments of the present invention are described in detail below with reference to the accompanying drawings:

[0025] Example:

[0026] This embodiment discloses a decoupled lightweight echo cancellation method. Starting from the perspective of amplitude-phase decoupled neural network, a lightweight echo cancellation method is designed to effectively improve the echo cancellation effect and promote local and global feature fusion. The method includes the following steps:

[0027] Step 1: Perform STFT feature extraction on the near-end mixed speech and far-end reference speech. Assume that s(t) and c(t) represent the impulse response of the near-end signal, far-end signal and room respectively, and the near-end mixed speech y(t) can be expressed as their linear sum:

[0028] y(t)=s(t)+c(t)

[0029] The time domain signal is converted into frequency domain information through STFT. The STFT formula is:

[0030]

[0031] Step 2: Construct a decoupled dual-path echo cancellation network. Use STFT to extract amplitude spectrum and complex spectrum from near-end mixed speech and far-end reference speech, splice them and feed them into the network, and restore them to near-end reference speech in the time-frequency domain. The upper branch network processes the amplitude spectrum, and the lower branch processes the complex spectrum. A Fusion module is embedded between the upper and lower paths. The module is distributed in different stages of the network. Real-time information interaction is completed by adding amplitudes. The two branches assist each other in learning and maintain their independence.

[0032] Step 3: The basic structure of the network consists of convolution blocks and dilated dense convolutions. The architectures of the encoder and decoder are symmetrical, responsible for the extraction and reconstruction of speech features respectively;

[0033] Step 4: Based on the decoupled network structure, a Dual-pathSkip-Transformer module based on DPT is designed to fine-tune the local and global long-term dependency information generated by the DPT module to reduce the number of parameters and calculations. DPST consists of two parts: the residual iteration module and the Skip Transformer module; the DPST module fine-tunes the long-term dependency output of the DPT module. The DPST process is mainly divided into the following steps;

[0034] Step a: Local and global long-term dependencies generated by DPT modules and Save it and send it to the DPST module;

[0035] Step b: and The model is sent to the Skip Transformer module for fine-tuning to strengthen the local information association between the two features:

[0036]

[0037] Step c: The Skip Transformer module in step b can enhance the adjacent information of the feature after restoring the original number of channels. The feature is average pooled in the channel dimension, followed by one-dimensional convolution in the time dimension or frequency dimension, and then activated using the Sigmoid function, mapped to the range of (0,1) to obtain a set of weights, and finally, these weights are used to scale the input features.

[0038] Step d: Finally, the features are group normalized and reshaped back to C×T×F using residual connections to obtain the output features. The overall expression is as follows:

[0039]

[0040] Step 5: In order to more effectively highlight important features and remove redundant features, a decoder containing embedded SCConv and SCCSF modules is designed, which can reconstruct features and increase convolution efficiency while reducing the number of parameters and calculations; SCConv can be divided into the following steps:

[0041] Step a: SCConv is composed of a spatial reconstruction unit (Spatial Reconstruction Unit, SRU) and a channel reconstruction unit (Channel Reconstruction Unit, CRU) connected in series;

[0042] Step b: The spatial reconstruction unit first evaluates the information content of the features through group normalization to obtain normalized correlation weights. The re-weighted features are mapped to (0, 1) through the Sigmoid function. Then, a gating mechanism is set to regard weights above the threshold as important features W1 and weights below the threshold as redundant features W2. Subsequently, W1 and W2 are multiplied by the input features to obtain important features. and redundant features Finally, the important features and redundant features are cross-reconstructed to obtain the TF domain reconstruction feature X w :

[0043] W=Gate(Sigmoid(W γ (GN(X))));

[0044] Step c: The features of CRU reconstructed by SRU are divided into two parts from the channel dimension. The upper branch is the important channel dimension, and the lower branch is the relatively redundant features in the channel dimension. First, both branches compress the features from the channel dimension through 1×1 convolution to improve computational efficiency. Then the upper branch uses grouped convolution and point-by-point convolution to add the features to obtain important features. The lower branch only uses cheap point-by-point convolution and pre-convolution features to generate relatively redundant features as a supplement to important features. Subsequently, global average pooling is used to generate channel soft attention. Finally, the upper and lower features are concatenated according to the channel dimension to obtain the features after channel reconstruction.

[0045] Step d: SCConv cannot be used directly to process complex features in the complex spectrum branch. SCConv Complex Spectrum Filter (SCCSF) is designed to remove redundant information in the complex domain, while deepening the connection between the real domain and the imaginary domain, and promoting the removal of redundant information. SCCSF first splits the real spectrum and the imaginary spectrum in the channel dimension, and then uses SCConv for complex convolution. SCCSF enhances the performance of the complex spectrum:

[0046] Real out =CUR r (SUR r (Real))-CUR i (SUR i (Imag)

[0047] Imag out =CUR r (SUR r (Imag))+CUR i (SUR i (Real).

[0048] Step 6: Previous studies have shown that although the spectrum of the near-end reference speech can be directly estimated, better echo removal can be achieved by masking the near-end mixed speech spectrum. m and the complex spectrum decoder output (M r ,M i ) is further combined to calculate the mask:

[0049]

[0050] Mask m and Mask p Mask the spectrum Y of the near-end mixed speech m and Y p Get the estimated near-end reference speech spectrum:

[0051]

[0052] Step 7: Finally, use the inverse short-time Fourier transform ISTFT to and Reconstruct the signal back into the time domain.

[0053] The above description is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, various changes, modifications, substitutions and variations can be made to these embodiments without departing from the technical principles of the present invention. These changes, modifications, substitutions and variations should also be regarded as the scope of protection of the present invention.

Claims

1. A decoupled lightweight echo cancellation method, characterized in that: From the perspective of amplitude-phase decoupled neural network, a lightweight method of echo cancellation is designed to effectively improve the echo cancellation effect and promote local and global feature fusion. The method includes the following steps: Step 1: Perform STFT feature extraction on the near-end mixed speech and far-end reference speech. s(t) and c(t) represent the impulse response of the near-end signal, far-end signal and room respectively, and the near-end mixed speech y(t) can be expressed as their linear sum: y(t)=s(t)+c(t) The time domain signal is converted into frequency domain information through STFT. The STFT formula is: Step 2: Construct a decoupled dual-path echo cancellation network. Use STFT to extract amplitude spectrum and complex spectrum from near-end mixed speech and far-end reference speech, splice them and feed them into the network, and restore them to near-end reference speech in the time-frequency domain. The upper branch network processes the amplitude spectrum, and the lower branch processes the complex spectrum. A Fusion module is embedded between the upper and lower paths. The module is distributed in different stages of the network. Real-time information interaction is completed by adding amplitudes. The two branches assist each other in learning and maintain their independence. Step 3: The basic structure of the network consists of convolution blocks and dilated dense convolutions. The architectures of the encoder and decoder are symmetrical, responsible for the extraction and reconstruction of speech features respectively; Step 4: Design a dual-path based on DPT based on a decoupled network structure The Skip-Transformer module is used to fine-tune the local and global long-term dependency information generated by the DPT module to reduce the number of parameters and calculations. The DPST consists of two parts: the residual iteration module and the Skip Transformer module; Step 5: In order to more effectively highlight important features and remove redundant features, a decoder containing embedded SCConv and SCCSF modules is designed, which can reconstruct features and increase convolution efficiency while reducing the number of parameters and calculations; Step 6: Further combine the amplitude decoder output M_m and the complex spectrum decoder output (M_r, Mi_i) to calculate the mask: Mask m and Mask p Mask the spectrum Y of the near-end mixed speech m and Y p Get the estimated near-end reference speech spectrum: Step 7: Finally, use the inverse short-time Fourier transform ISTFT to and Reconstruct the signal back into the time domain.

2. The decoupled lightweight echo cancellation method according to claim 1, characterized in that: Step 4: The module DPST fine-tunes the long-term dependency of the output of the DPT module. The specific steps are as follows: Step a: Local and global long-term dependencies generated by DPT modules and Save it and send it to the DPST module; Step b: and The model is sent to the Skip Transformer module for fine-tuning to strengthen the local information association between the two features: Step c: The Skip Transformer module in step b can enhance the adjacent information of the feature after restoring the original number of channels. The feature is average pooled in the channel dimension, followed by one-dimensional convolution in the time dimension or frequency dimension, and then activated using the Sigmoid function, mapped to the range of (0,1) to obtain a set of weights, and finally, these weights are used to scale the input features. Step d: Finally, the features are group normalized and reshaped back to C×T×F using residual connections to obtain the output features. The overall expression is as follows:

3. The decoupled lightweight echo cancellation method according to claim 1, characterized in that: In step 5, we design a decoder that embeds SCConv and SCCSF modules. The specific steps are as follows: Step a: SCConv is composed of a spatial reconstruction unit (Spatial Reconstruction Unit, SRU) and a channel reconstruction unit (Channel Reconstruction Unit, CRU) connected in series; Step b: The spatial reconstruction unit first evaluates the information content of the features through group normalization to obtain normalized correlation weights. The re-weighted features are mapped to (0, 1) through the Sigmoid function. Then, a gating mechanism is set to regard weights above the threshold as important features W1 and weights below the threshold as redundant features W2. Subsequently, W1 and W2 are multiplied by the input features to obtain important features. and redundant features Finally, the important features and redundant features are cross-reconstructed to obtain the TF domain reconstruction feature X w : W=Gate(Sigmoid(W γ (GN(X)))); Step c: The features reconstructed by the spatial reconstruction unit of the channel reconstruction unit are divided into two parts from the channel dimension. The upper branch is the important channel dimension, and the lower branch is the relatively redundant features in the channel dimension. First, both branches compress the features from the channel dimension through 1×1 convolution to improve the computational efficiency. Then the upper branch uses grouped convolution and point-by-point convolution to add the features to obtain important features. The lower branch only uses cheap point-by-point convolution and pre-convolution features to generate relatively redundant features as a supplement to important features. Subsequently, global average pooling is used to generate channel soft attention. Finally, the upper and lower features are concatenated according to the channel dimension to obtain the features after channel reconstruction. Step d: SCConv cannot be used directly to process complex features in the complex spectrum branch. SCConv ComplexSpectrum Filter (SCCSF) is designed to remove redundant information in the complex domain and deepen the connection between the real domain and the imaginary domain to promote the removal of redundant information. SCCSF first splits the real spectrum and the imaginary spectrum in the channel dimension, and then uses SCConv for complex convolution. SCCSF enhances the performance of the complex spectrum: Real out =CUR r (ON r (Real))-CUR i (ON i (Image)) Image out =CUR r (ON r (Imag))+CUR i (ON i (Real)).

Images