A stereo acoustic echo cancellation method and system

Abstract

The application discloses a kind of stereo echo cancellation method and system, method includes: obtaining remote signal 1, remote signal 2 and microphone signal;Utilize the stereo echo cancellation model trained to process the above 3 signals, obtain estimated near-end speech signal;The model includes: signal time-frequency feature extraction module, deep Kalman filter, residual echo and noise suppression module and time-domain signal recovery module, deep Kalman filter includes reference signal estimation module and Kalman filtering process module.Deep Kalman filter utilizes the time-frequency feature of the above 3 signals to estimate nonlinear reference signal, input Kalman filter with two remote reference signals and estimated nonlinear reference signal to filter, while filtering out linear and nonlinear echo in microphone signal, thus, it can be while guaranteeing linear echo cancellation amount, reducing the speech quality damage caused by nonlinear modeling when better coping nonlinear echo.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing, and specifically relates to a stereo echo cancellation method and system. Background Technology

[0002] The coupling between the speaker and the microphone produces an echo. Acoustic echo cancellation, designed to eliminate echoes picked up by the microphone while preserving near-end signals, is an important component of hands-free communication and human-computer interaction systems. Stereo systems are receiving increasing attention for achieving better spatial listening experiences.

[0003] Stereo echo cancellation solutions include adaptive filtering, end-to-end models, and hybrid models. Adaptive filtering eliminates echoes by identifying the echo path between the speaker and microphone. However, stereo echo cancellation suffers from non-uniqueness, causing traditional adaptive filtering algorithms to converge much slower than in single-channel cases when there is high cross-correlation between far-end signals. To address this, decorrelation algorithms are used to reduce the correlation between two far-end signals, but this comes at the cost of stereo imaging and speech quality. End-to-end models directly use neural networks to estimate masks or mapping functions to suppress echoes and noise. This approach eliminates the need for decorrelation algorithms, but performance degrades at low near-end-echo ratios. Hybrid models cascade an echo canceller based on adaptive filtering with a neural network-based residual echo and noise suppressor, combining the advantages of both methods. However, existing adaptive filtering-based echo cancellers are based on linear modeling and cannot effectively handle nonlinear echoes. Summary of the Invention

[0004] The purpose of this invention is to overcome the defects of the prior art and to propose a stereo echo cancellation method. This invention also discloses a stereo echo cancellation system.

[0005] A stereo echo cancellation method includes:

[0006] Before the speaker plays the signal, far-end signal 1 and far-end signal 2 are acquired, and the signal in the near-end room is picked up. The picked-up signal is called the microphone signal, which includes echo signal, near-end speech signal and near-end background noise signal.

[0007] The microphone signal is converted into a corresponding digital signal so that it, along with far-end signal 1 and far-end signal 2, can be output to the trained stereo echo cancellation model.

[0008] The trained stereo echo cancellation model processes the far-end signal 1, far-end signal 2 and microphone signal to obtain the estimated near-end speech signal;

[0009] The trained stereo echo cancellation model includes: a signal time-frequency feature extraction module, a deep Kalman filtering module, a residual echo and noise suppression module, and a time-domain signal recovery module. The deep Kalman filtering module includes: a reference signal estimation module and a Kalman filtering process module. The trained stereo echo cancellation model processes far-end signal 1, far-end signal 2, and the microphone signal as follows:

[0010] The signal time-frequency feature extraction module extracts the time-frequency features of the input remote signal 1, remote signal 2 and microphone signal;

[0011] The deep Kalman filter module utilizes the time-frequency characteristics of far-end signal 1, far-end signal 2, and microphone signal to eliminate part of the echo signal in the microphone signal. Specifically, the elimination process includes: a reference signal estimation module using the time-frequency characteristics of far-end signal 1, far-end signal 2, and microphone signal to estimate a nonlinear reference signal; and a Kalman filter module updating the Kalman filter based on the time-frequency characteristics of the microphone signal, far-end signal 1, far-end signal 2, and the estimated nonlinear reference signal to obtain the time-frequency characteristics that have eliminated part of the echo signal.

[0012] The residual echo and noise suppression module suppresses residual echo and noise in the time-frequency characteristics of the partially eliminated echo signal, thus obtaining the time-frequency characteristics of the processed signal.

[0013] The time-domain signal recovery module restores the time-frequency characteristics of the processed signal into a time-domain signal, thereby obtaining the estimated near-end speech signal.

[0014] In an improved stereo echo cancellation method, the reference signal estimation module includes: a decoupling module, a nonlinear reference signal estimation module, a coupling module, and a feature splicing module;

[0015] The reference signal estimation module uses the time-frequency characteristics of far-end signal 1, far-end signal 2, and microphone signal to estimate the nonlinear reference signal, specifically including:

[0016] The decoupling module decouples the complex time-frequency features output by the signal time-frequency feature extraction module, takes the modulus of the complex time-frequency features to obtain the amplitude time-frequency features, and takes the phase of the complex time-frequency features to obtain the phase time-frequency features;

[0017] The nonlinear reference signal estimation module estimates the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module, and multiplies the amplitude mask by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0018] The coupling module couples the phase time-frequency characteristics of the microphone signal output by the decoupling module with the amplitude time-frequency characteristics of the nonlinear reference signal output by the nonlinear reference signal estimation module to obtain the time-frequency characteristics of the nonlinear reference signal.

[0019] The feature splicing module splices the time-frequency features output by the signal time-frequency feature extraction module and the time-frequency features output by the nonlinear reference signal estimation module to obtain the reference signal of the Kalman filter.

[0020] In an improved stereo echo cancellation method, the nonlinear reference signal estimation module includes: a feature connection layer, an input dimension adjustment layer composed of stacked linear layers, a temporal modeling layer composed of stacked recurrent neural networks, an output dimension adjustment layer, a nonlinear activation layer composed of sigmoid functions, and a multiplication operation layer.

[0021] The nonlinear reference signal estimation module estimates the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module. The amplitude mask is then multiplied by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal, including:

[0022] The feature connection layer connects the amplitude time-frequency features of remote signal 1, remote signal 2 and microphone signal into the input features of the neural network;

[0023] The input dimension adjustment layer performs a linear mapping on the input features of the neural network output by the feature connection layer to obtain the dimension-adjusted input features;

[0024] The timing modeling layer continuously receives the dimension-adjusted features output by the input dimension adjustment layer, thereby obtaining the timing relationship between the remote signal 1, the remote signal 2, and the microphone signal.

[0025] The output dimension adjustment layer performs a linear mapping on the temporal features output by the temporal modeling layer to obtain the dimension-adjusted output features;

[0026] The nonlinear activation layer compresses the value range of the features output by the output dimension adjustment layer to the range of [0, 1], thus obtaining an amplitude mask;

[0027] The multiplication operation layer multiplies the amplitude mask output by the nonlinear activation layer with the amplitude time-frequency characteristics of the microphone signal output by the decoupling module to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0028] In an improved stereo echo cancellation method, the Kalman filtering process module updates the Kalman filter, including:

[0029] The Kalman filtering module utilizes a recurrent neural network to estimate the process noise power spectral density based on the norm of the filter coefficients. The recurrent neural network includes a norm operation layer, a time-series modeling layer composed of stacked recurrent neural networks, a fully connected layer composed of stacked linear layers, and a sigmoid function layer. The norm operation layer calculates the norm of the filter coefficients and then concatenates the calculation results at each frequency point to form a feature vector of the filter coefficients. The time-series modeling layer continuously receives the feature vectors of the filter coefficients output by the norm operation layer, thereby obtaining the time-series features of the filter coefficients. The fully connected layer performs a linear mapping on the features output by the time-series modeling layer to obtain dimension-adjusted output features. The sigmoid function layer processes the features output by the fully connected layer through a sigmoid function, thereby compressing the value range of the features to [0,1] to obtain the process noise power spectral density.

[0030] In an improved stereo echo cancellation method, the stereo echo cancellation model is trained in the following manner:

[0031] Construct training and validation sets. Each set of samples in the training and validation sets includes: input signal and near-end speech signal. The input signal includes far-end signal 1, far-end signal 2 and microphone signal. The near-end speech signal has time-frequency feature labels.

[0032] The training set is divided into batches of samples B1, and the input signals from the divided samples are input into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combined with the time-frequency feature labels of the near-end speech signal, the loss function value is calculated, and the calculated loss function value is used to train the stereo echo cancellation model by backpropagating the gradient of the parameters in the stereo echo cancellation model, where B1 is a positive integer greater than 1.

[0033] The validation set is divided into batches of samples B2, and the input signals in the divided samples are input into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combined with the time-frequency feature labels of the near-end speech signal, the mean of the loss function is calculated. The stereo echo cancellation model used in the validation set uses the model parameters obtained by training the training set in this round, where B2 is a positive integer greater than 1.

[0034] The training and validation sets are repeatedly input into the stereo echo cancellation model in batches for training until convergence. The mean loss function on the validation set is compared at different training rounds, and the parameter with the minimum mean loss function is saved as the trained model parameter.

[0035] In an improved stereo echo cancellation method, before dividing the training set into batches of B1 samples, the method includes:

[0036] In each sample group, a time point is randomly selected to introduce an echo path change into the input signal.

[0037] On the other hand, the present invention also provides a stereo echo cancellation system, including: a loopback acquisition module, a microphone, an A / D module and a stereo echo cancellation module;

[0038] The loopback acquisition module is used to acquire remote signal 1 and remote signal 2 before the speaker plays the signal.

[0039] The microphone is used to pick up signals in the near-end room, wherein the picked-up signals are called microphone signals and include echo signals, near-end speech signals and near-end background noise signals.

[0040] The A / D module is used to convert the microphone signal into a corresponding digital signal, so that it, together with the remote signal 1 and the remote signal 2, is output to the stereo echo cancellation module.

[0041] The stereo echo cancellation module is used to process the far-end signal 1, far-end signal 2 and microphone signal using a trained stereo echo cancellation model to obtain an estimated near-end speech signal.

[0042] The stereo echo cancellation model includes:

[0043] The signal time-frequency feature extraction module is used to extract the time-frequency features of the input remote signal 1, remote signal 2 and microphone signal;

[0044] The deep Kalman filter module utilizes the time-frequency characteristics of far-end signal 1, far-end signal 2, and microphone signal to eliminate part of the echo signal in the microphone signal. The deep Kalman filter module includes a reference signal estimation module, used to estimate a nonlinear reference signal using the time-frequency characteristics of far-end signal 1, far-end signal 2, and microphone signal; and a Kalman filtering process module, used to update the Kalman filter based on the time-frequency characteristics of the microphone signal, far-end signal 1, far-end signal 2, and the estimated nonlinear reference signal, thus obtaining time-frequency characteristics that have eliminated part of the echo signal.

[0045] The residual echo and noise suppression module is used to suppress residual echo and noise in the echo-cancelled signal output from the deep Kalman filter module, obtaining the time-frequency characteristics of the processed signal; and

[0046] The time-domain signal recovery module is used to recover the time-frequency characteristics of the processed signal into a time-domain signal to obtain the estimated near-end speech signal.

[0047] In an improved stereo echo cancellation system, the reference signal estimation module includes: a decoupling module, a nonlinear reference signal estimation module, a coupling module, and a feature splicing module;

[0048] The decoupling module is used to decouple the complex time-frequency features output by the signal time-frequency feature extraction module, take the modulus of the complex time-frequency features to obtain the amplitude time-frequency features, and take the phase of the complex time-frequency features to obtain the phase time-frequency features;

[0049] The nonlinear reference signal estimation module is used to estimate the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module, and multiply the amplitude mask by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0050] The coupling module is used to couple the phase time-frequency characteristics of the microphone signal output by the decoupling module with the amplitude time-frequency characteristics of the nonlinear reference signal output by the nonlinear reference signal estimation module to obtain the time-frequency characteristics of the nonlinear reference signal.

[0051] The feature splicing module is used to splice the time-frequency features output by the signal time-frequency feature extraction module and the time-frequency features output by the nonlinear reference signal estimation module to obtain the reference signal of the Kalman filter.

[0052] In an improved stereo echo cancellation system, the nonlinear reference signal estimation module includes: a feature connection layer, an input dimension adjustment layer composed of stacked linear layers, a temporal modeling layer composed of stacked recurrent neural networks, an output dimension adjustment layer, a nonlinear activation layer composed of sigmoid functions, and a multiplication operation layer.

[0053] The nonlinear reference signal estimation module is used to estimate the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module. The amplitude mask is then multiplied by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal, including:

[0054] The feature connection layer is used to connect the amplitude time-frequency features of remote signal 1, remote signal 2 and microphone signal into the input features of the neural network.

[0055] The input dimension adjustment layer is used to perform a linear mapping on the input features of the neural network output by the feature connection layer to obtain the dimension-adjusted input features;

[0056] The timing modeling layer is used to continuously receive the dimension-adjusted features output by the input dimension adjustment layer, thereby obtaining the timing relationship between the remote signal 1, the remote signal 2 and the microphone signal.

[0057] The output dimension adjustment layer is used to perform a linear mapping on the temporal features output by the temporal modeling layer to obtain the dimension-adjusted output features;

[0058] The nonlinear activation layer is used to compress the value range of the features output by the output dimension adjustment layer to the range of [0, 1] to obtain an amplitude mask;

[0059] The multiplication operation layer is used to multiply the amplitude mask output by the nonlinear activation layer with the amplitude time-frequency characteristics of the microphone signal output by the decoupling module to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0060] In an improved stereo echo cancellation system, the stereo echo cancellation model is trained through a construction module, a training module, a validation module, and a model parameter determination module, wherein:

[0061] The construction module is used to construct training sets and validation sets. Each set of samples in the training set and validation set includes: an input signal and a near-end speech signal. The input signal includes far-end signal 1, far-end signal 2 and microphone signal. The near-end speech signal has time-frequency feature labels.

[0062] The training module is used to divide the training set into batches of samples B1, and input the input signals from the divided samples into batches into the stereo echo cancellation model to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combining the time-frequency feature labels of the near-end speech signal, the loss function value is calculated, and the calculated loss function value is used to train the stereo echo cancellation model by backpropagating the gradient of the parameters in the stereo echo cancellation model, where B1 is a positive integer greater than 1.

[0063] The verification module is used to divide the verification set into batches of samples B2, and input the input signals in the divided samples into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combined with the time-frequency feature labels of the near-end speech signal, the mean of the loss function is calculated. The stereo echo cancellation model used in the verification set uses the model parameters obtained by training the training set in this round, where B2 is a positive integer greater than 1.

[0064] The model parameter determination module is used to trigger the training module and the validation module when the stereo echo cancellation model has not converged. The module repeatedly inputs the samples of the training set and the validation set into the stereo echo cancellation model in batches for training, compares the mean loss function on the validation set at different training rounds, and saves the parameter with the minimum mean loss function as the trained model parameter.

[0065] Compared with the prior art, the advantages of the present invention are:

[0066] 1. In this embodiment of the invention, a deep Kalman filter is used, which is different from traditional filters. The difference is that, firstly, the time-frequency characteristics of far-end signal 1, far-end signal 2 and microphone signal are used to estimate the nonlinear reference signal; secondly, the two far-end reference signals and the estimated nonlinear reference signal are input into the Kalman filter for filtering, thereby filtering out both linear and nonlinear echoes in the microphone signal. Therefore, it can better deal with nonlinear echoes while ensuring the amount of linear echo cancellation and reducing the speech quality loss caused by nonlinear modeling.

[0067] 2. In this embodiment of the invention, a recurrent neural network is used to estimate the power spectral density of process noise, thereby enhancing the tracking capability of the deep Kalman filter. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the stereo echo cancellation model and its workflow in Embodiment 1 of the present invention;

[0069] Figure 2 This is a structural diagram of the deep Kalman filter module in Embodiment 1 of the present invention;

[0070] Figure 3 This is a structural diagram of the reference signal estimation module in Embodiment 1 of the present invention;

[0071] Figure 4 This is a structural diagram of the nonlinear reference signal estimation module in Embodiment 1 of the present invention;

[0072] Figure 5 This is a schematic diagram of the working steps of the Kalman filtering process module in Embodiment 1 of the present invention;

[0073] Figure 6 This is a structural diagram of the process noise power spectral density estimation module in Embodiment 1 of the present invention;

[0074] Figure 7 This is a flowchart of the stereo echo cancellation model training method in Embodiment 2 of the present invention;

[0075] Figure 8 This is a schematic diagram of the stereo echo cancellation system in Embodiment 4 of the present invention;

[0076] Figure 9 This is a schematic diagram of the stereo echo cancellation model training device in Embodiment 4 of the present invention. Detailed Implementation

[0077] This invention proposes a stereo echo cancellation method and system based on a deep hybrid model. The method and system involve two stages of echo cancellation. In the first stage, a Kalman filter based on deep learning is used to eliminate part of the echo. In the second stage, a deep neural network is employed to eliminate residual echo and noise. This method is applicable to stereo echo cancellation in various signal echo and signal-to-noise scenarios. Specific embodiments of the invention are described below.

[0078] Example 1

[0079] The stereo echo cancellation method provided by this invention includes the following steps:

[0080] Step 1: Before the speaker plays the signal, acquire the far-end signal 1 and far-end signal 2, and pick up the signal in the near-end room. The picked-up signal is called the microphone signal, which includes the echo signal and the near-end signal. The near-end signal includes the near-end speech signal and the near-end background noise signal.

[0081] Step 2: Convert the microphone signal into a corresponding digital signal so that it, along with far-end signal 1 and far-end signal 2, can be output to the trained stereo echo cancellation model.

[0082] Step 3: The trained stereo echo cancellation model processes the far-end signal 1, far-end signal 2 and microphone signal to obtain the estimated near-end speech signal.

[0083] The stereo echo cancellation model includes: a signal time-frequency feature extraction module 101, a deep Kalman filter module 102, a residual echo and noise suppression module 103, and a time-domain signal recovery module 104. Its model and workflow diagram are shown below. Figure 1 As shown, where:

[0084] The signal time-frequency feature extraction module 101 extracts the time-frequency features of the remote signal 1, the remote signal 2 and the microphone signal, and uses them as input to the deep Kalman filter 102.

[0085] The working steps of the signal time-frequency feature extraction module 101 are as follows:

[0086] Step 101-1) Frame the input signal; in this embodiment, the length of each frame is 20ms and the frame shift is 10ms;

[0087] Step 101-2) Multiply each frame of signal obtained by a Hann window to reduce spectral leakage;

[0088] Step 101-3) Perform an F-point discrete Fourier transform on each frame of the processed signal to obtain the complex spectrum of the signal; the time-frequency characteristics of the far-end signal 1 are denoted as... The time-frequency characteristics of the remote signal 2 are denoted as: The time-frequency characteristics of the microphone signal are denoted as: The time frame index is k∈{1,2,...,K}, the frequency block index is k, and K=F / 2+1 is the total number of frequency blocks to be processed.

[0089] The deep Kalman filter module 102 is connected to the output of the signal time-frequency feature extraction module 101. It uses the extracted input time-frequency features to perform deep Kalman filtering to eliminate some echo components in the time-frequency features of the microphone signal. For example... Figure 2 As shown, the depth Kalman filter module 102 can be divided into two parts, including a reference signal estimation module 201 and a Kalman filtering process module 202.

[0090] The reference signal estimation module 201 uses the time-frequency characteristics of the remote signal 1, the remote signal 2 and the microphone signal to estimate the reference signal of the Kalman filter. The nonlinear reference signal contained in the reference signal plays an important role in this invention.

[0091] The Kalman filtering process module 202 updates the time-frequency characteristics of the error signal based on the time-frequency characteristics of the microphone signal and the time-frequency characteristics of the reference signal, and then updates the observation noise power spectral density, Kalman gain, filter coefficients, process noise power spectral density, and state error covariance.

[0092] The specific details of the reference signal estimation module 201 are as follows: Figure 3 As shown, it includes a decoupling module 301, a nonlinear reference signal estimation module 302, a coupling module 303, and a feature splicing module 304.

[0093] Decoupling module 301 extracts the amplitude and phase of the time-frequency features output by signal time-frequency feature extraction module 101; the amplitude time-frequency features of the far-end signal 1 are denoted as... The phase time-frequency characteristic is denoted as The amplitude time-frequency characteristics of the far-end signal 2 are denoted as: The phase time-frequency characteristic is denoted as The amplitude-time frequency characteristic of the microphone signal is denoted as The phase time-frequency characteristic is denoted as θ Y(l,k) ∈[-π,π].

[0094] The nonlinear reference signal estimation module 302 estimates the amplitude mask M(l,k)∈[0,1] using a neural network based on the amplitude time-frequency characteristics of the far-end signal 1, far-end signal 2, and microphone signal output from the decoupling module 301. The estimated amplitude mask is then multiplied by the amplitude time-frequency characteristics of the microphone signal output from the decoupling module 301 to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0095] The specific details of the nonlinear reference signal estimation module 302 are as follows: Figure 4 As shown, it is composed of a neural network, including a feature connection layer 401, an input dimension adjustment layer 402, a temporal modeling layer 403, an output dimension adjustment layer 404, a nonlinear activation layer 405, and a multiplication operation layer 406.

[0096] Feature connection layer 401 connects the amplitude time-frequency features of remote signal 1, remote signal 2 and microphone signal. and Input features connected into a neural network

[0097] The input dimension adjustment layer 402 is composed of stacked linear layers. It performs a linear mapping on the input features of the neural network output from the feature connection layer 401 to obtain the dimension-adjusted input features. Assuming the number of linear layers is 1, the input dimension is 3K, and the output dimension is 300, then the dimension-adjusted input features are:

[0098] The temporal modeling layer 403 is composed of stacked recurrent neural networks. It continuously receives the dimension-adjusted features output by the dimension adjustment layer 402 to obtain the temporal relationship between the remote signal 1, the remote signal 2 and the microphone signal.

[0099] It should be noted that recurrent neural networks (RNNs) can take many forms, including Long Short-Term Memory (LSTM) networks, gated recurrent units (GRUs), and other variations. In this example, an LSM network is chosen as the basic unit for the temporal modeling layer. With 4 layers, 300 input neurons, 300 hidden neurons, and 300 output neurons, the temporal features are:

[0100] The output dimension adjustment layer 404 is composed of stacked linear layers. It performs a linear mapping on the temporal features output by the temporal modeling layer 403 to obtain the dimension-adjusted output features. Setting the number of linear layers to 1, the input dimension to 300, and the output dimension to K, the dimension-adjusted output features are:

[0101] The nonlinear activation layer 405, composed of sigmoid functions, compresses the feature range of the output dimension adjustment layer 404 to the range [0,1], thus obtaining the amplitude mask.

[0102] The multiplication operation layer 406 multiplies the amplitude mask output by the nonlinear activation layer 405 with the amplitude time-frequency characteristics of the microphone signal output by the decoupling module 301 to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0103] The coupling module 303 couples the phase time-frequency characteristics of the microphone signal output from the decoupling module 301 with the amplitude time-frequency characteristics of the nonlinear reference signal output from the nonlinear reference signal estimation module 302 to obtain the time-frequency characteristics of the nonlinear reference signal.

[0104] The feature splicing module 304 splices together the time-frequency features of the remote signal 1 and the remote signal 2 output by the signal time-frequency feature extraction module 101 and the time-frequency features of the nonlinear reference signal output by the coupling module 303 to obtain the reference signal of the Kalman filter, which is used as the input signal of the Kalman filter.

[0105] The working steps of the feature splicing module 304 are as follows:

[0106] Step 304-1) Concatenate the time-frequency features of the current time frame l and the far-end signal 1 from the past M1-1 time frames to form the input vector x1(l,k) = [X1(l,k), X1(l-1,k), ..., X1(l-M1+1,k)] corresponding to the far-end signal 1. T ; The filter length corresponding to the far-end signal 1 is recommended to be selected within the range of {1,2,...,16} in this embodiment;

[0107] Step 304-2) Concatenate the time-frequency features of the current time frame l and the far-end signal 2 from the past M2-1 time frames to form the input vector x2(l,k) = [X2(l,k), X2(l-1,k), ..., X2(l-M2+1,k)] corresponding to the far-end signal 2. T ; The filter length corresponding to the far-end signal 2 is recommended to be selected within the range of {1,2,...,16} in this embodiment;

[0108] Step 304-3) Combine the current time frame l and the past M NL The time-frequency features of the far-end signal 2 from -1 time frames are concatenated to form the input vector x corresponding to the nonlinear reference signal. NL (l,k)=[X NL (l,k),X NL (l-1,k),...,X NL (lM NL +1,k)] T ; The filter length is the nonlinear reference signal, and in this embodiment, it is recommended to select it within the range of {1,2,...,16}.

[0109] Step 304-4) Concatenate the input vectors corresponding to far-end signal 1, far-end signal 2, and the nonlinear reference signal to obtain the input vector of the Kalman filter.

[0110] The working steps of Kalman filtering module 202 are as follows: Figure 5 As shown, the specific details are as follows:

[0111] Step 501) Update the time-frequency characteristics E(l,k) of the error signal based on the time-frequency characteristics Y(l,k) of the microphone signal and the input signal x(l,k) of the Kalman filter:

[0112]

[0113] Wherein, the Kalman filter vector It is formed by concatenating the filter vectors corresponding to the far-end signal 1, the far-end signal 2, and the nonlinear reference signal;

[0114] Step 502) Update the observation noise power spectral density φ based on the time-frequency characteristics E(l,k) of the error signal. V (l,k):

[0115] φ V (l,k)=αφ V (l-1,k)+(1-α)||E(l,k)|| 2

[0116] Where α is the forgetting factor, φ V (l-1,k) is the observed noise power spectral density estimated in the (l-1)th iteration;

[0117] Step 503) Based on the input signal x(l,k) of the Kalman filter and the observed noise power spectral density φ V (l,k) and the state noise error covariance φ estimated in the (l-1)th iteration ε Update the Kalman gain G(l,k) to (l-1,k):

[0118]

[0119] Step 504) Update the Kalman filter coefficients based on the Kalman gain G(l,k) and the time-frequency characteristics E(l,k) of the error signal.

[0120]

[0121] Step 505) The specific details of the noise power spectral density update process are as follows: Figure 6 As shown, it includes a norm operation layer 601, a timing modeling layer 602, a fully connected layer 603, and a sigmoid function layer 604;

[0122] Norm operation layer 601, for filter coefficients Calculate the norm, and then concatenate the calculation results of K frequency points into the eigenvector φ of the filter coefficients. h (l):

[0123]

[0124] Where L = M1 + M2 + M NL It is the filter coefficient vector The total length;

[0125] The temporal modeling layer 602 is composed of stacked recurrent neural networks, continuously receiving feature vectors of the filter coefficients output by the norm operation layer 601, thereby obtaining the temporal features of the filter coefficients. In this example, a Long Short-Term Memory (LSTM) network is selected as the basic unit of the temporal modeling layer. With a layer number of 1, input neurons of K, hidden neurons of 50, and output neurons of 50, the temporal features of the filter coefficients are as follows:

[0126] The fully connected layer 603 is composed of stacked linear layers. It performs a linear mapping on the features output by the temporal modeling layer 602 to obtain the dimension-adjusted output features. Setting the number of linear layers to 1, the input dimension to 50, and the output dimension to K, the dimension-adjusted output features are as follows:

[0127] The S-shaped function layer 604 passes the features output from the fully connected layer 603 through an S-shaped function, thereby compressing the value range of the features to [0,1], and obtaining the process noise power spectral density.

[0128] Step 506) Based on the filter input signal x(l,k), Kalman gain G(l,k), and process noise power spectral density φ Δh (l,k) Update the state error covariance φ ε (l,k):

[0129]

[0130] Where φ ε (l-1,k) is the state error covariance estimated in the (l-1)th iteration.

[0131] The residual echo and noise suppression module 103 is connected to the output of the signal time-frequency feature extraction module 101 and the deep Kalman filter module 102. It uses a neural network to further suppress residual echoes and noise in the echo-cancelled signal to obtain the time-frequency features of the target signal.

[0132] The time-domain signal recovery module 104 is connected to the output of the residual echo and noise suppression module 104. It transforms the time-frequency characteristics of the target signal to the time domain to obtain the time-domain representation of the target signal. In this example, the inverse short-time Fourier transform is selected as the inverse transform.

[0133] Unless otherwise specified, all of the above neural network models employ conventional methods familiar to those skilled in the art.

[0134] Example 2

[0135] This embodiment 2 describes the training method for the stereo echo cancellation model, such as... Figure 7 As shown, it includes the following steps:

[0136] Step 701: Construct training and validation sets. Each set of samples in the training and validation sets includes: input signal and near-end speech signal.

[0137] The input signals here include far-end signal 1, far-end signal 2 and microphone signal, and the near-end speech signal has time-frequency feature labels;

[0138] Ideally, to improve the ability of the stereo echo cancellation model to cope with changes in the echo path, after constructing the training set, an echo path change can be introduced into the input signal at a random time in each set of samples.

[0139] Step 702: Divide the training set into batches of samples B1, and input the input signals from the divided samples into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combine the time-frequency feature labels of the near-end speech signal to calculate the loss function value, and use the calculated loss function value to train the stereo echo cancellation model by backpropagating the gradient of the parameters in the stereo echo cancellation model, where B1 is a positive integer greater than 1.

[0140] Step 703: Divide the validation set into batches of samples B2, and input the input signals in the divided samples into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combine the time-frequency feature labels of the near-end speech signal to calculate the mean of the loss function.

[0141] The stereo echo cancellation model used in the validation set uses the model parameters obtained from training the training set in this round, where B2 is a positive integer greater than 1;

[0142] Step 704: Repeatedly input the samples from the training set and validation set into the stereo echo cancellation model in batches for training until convergence, and compare the mean loss function on the validation set at different training rounds, and save the parameter with the minimum mean loss function as the trained model parameter.

[0143] Since Example 1 has already described in detail the stereo echo cancellation model and the signal processing process after inputting far-end signal 1, far-end signal 2, microphone signal and near-end speech signal, this Example 2 will not repeat the description.

[0144] Example 3

[0145] This embodiment 3 provides a detailed explanation of the training method for a stereo echo cancellation model through a specific example. In this embodiment, a simulation dataset is constructed using collected speech and noise materials to train and evaluate a stereo echo cancellation model.

[0146] First, a training set for stereo echo cancellation is constructed, with each sample group including far-end signal 1, far-end signal 2, microphone signal, and near-end speech signal. Specific steps include:

[0147] (a) Acquire speech and noise signals separately. In this example, the speech signal comes from LibriSpeech, and the noise signal comes from the noise datasets in the INTERSPEECH 2021DNS-Challenge and NoiseX-92 datasets. The subsets train-clean-100, dev-clean, and test-clean from LibriSpeech are used to construct the training, validation, and test sets, respectively. The noise dataset from the INTERSPEECH 2021DNS-Challenge is used to construct the training and validation sets in an 8:2 ratio. The NoiseX-92 dataset is used to construct the test set.

[0148] (b) The acquired speech signals are divided into far-end and near-end speech signals. For the speech dataset, the speech is first randomly shuffled alphabetically, and then the first 50% of the speech is taken as the far-end signal and the last 50% of the speech is taken as the near-end signal;

[0149] (c) Generating room impulse responses, including generating room impulse responses using the mirror method or extracting a segment of a certain duration from recorded room impulse response samples. In this example, the mirror method is used to generate room impulse responses. For each sample, four room impulse responses need to be generated, including two far-end room impulse responses and two near-end room impulse responses. Far-end room impulse response 1 and far-end room impulse response 2 are obtained by setting the far-end room dimensions, the position of the single far-end speaker, and the position and directivity of the two far-end microphones. Near-end room impulse response 1 and near-end room impulse response 2 are obtained by setting the near-end room dimensions, the position of the two near-end speakers, and the position and directivity of the single near-end microphone.

[0150] (d) Convolve the randomly selected far-end signal in (b) with the far-end room impulse response 1 and far-end room impulse response 2 generated in (c) respectively to obtain the far-end signal 1 and far-end signal 2 that the near-end speaker needs to play. In addition to simulating the far-end room, far-end signal 1 and far-end signal 2 can also be obtained by collecting readily available dual-channel audio from the network, such as dual-channel speech and music;

[0151] (e) Distortion processing is applied to far-end signal 1 and far-end signal 2 to simulate speech distortion when the signal is played by a loudspeaker. The distortion includes two parts: distortion caused by the power amplifier and distortion caused by the loudspeaker. The distortion caused by the power amplifier is simulated using hard clipping or soft clipping, and the loudspeaker distortion can be simulated using functions such as Tanh or Sigmoid. For two different loudspeakers, different distortion modes and distortion parameters are randomly selected to obtain the nonlinearly transformed far-end signal 1 and far-end signal 2.

[0152] (f) Convolve the near-room impulse response 1 and near-room impulse response 2 generated in (c) with the distorted far-room signal 1 and far-room signal 2 from (e) respectively to obtain echo signal 1 and echo signal 2. Superimpose these two echo signals to obtain the stereo echo signal received by the microphone. In addition to the distortion simulation in (e) and the near-room propagation simulation in (f), echo signal 1 and echo signal 2 can also be obtained by directly playing the far-room signal 1 and far-room signal 2 obtained in (d) through two loudspeakers;

[0153] (g) The near-end speech signal, noise, and stereo echo signal are mixed according to a certain signal-to-noise ratio and signal-to-echo ratio to obtain the microphone signal. The corresponding far-end signal 1, far-end signal 2, microphone signal, and near-end speech signal form a sample, where far-end signal 1, far-end signal 2, and microphone signal are the input signals, and the near-end speech signal is a time-frequency feature label;

[0154] (h) During training and validation, an echo path change is introduced at a random time point in each sample. Methods include abrupt echo path changes and gradual echo path changes. Abrupt echo path changes include two cases: first, directly replacing the original echo path with another random echo path; second, multiplying the original echo path by -1. Gradual echo path changes refer to slowly transforming the original echo path into a new echo path within a specified time range by changing the coefficients of the linear combination.

[0155] Secondly, the training set is divided into... The samples are divided into batches, and the batches of samples are input into the stereo echo cancellation model to obtain the estimated near-end speech signal for that batch. Specifically, each batch of samples contains the time-domain signals of far-end signal 1, far-end signal 2, and microphone signal, which are sequentially processed by the signal time-frequency feature extraction module 101, the deep Kalman filter module 102, the residual echo and noise suppression module 103, and the time-domain signal recovery module 104. After obtaining the estimated near-end speech signal, the loss function is calculated by combining the label of the near-end speech signal, and the gradient of the parameters in the network is calculated through backpropagation to train the neural network.

[0156] Next, the validation set is arranged according to... The samples are divided into batches, and the divided samples are input into the stereo echo cancellation model according to the batch to obtain the estimated near-end speech signal of the batch; after obtaining the estimated near-end speech signal, the mean of the loss function is calculated in combination with the label of the near-end speech signal.

[0157] Finally, the samples from the training set and validation set are repeatedly input into the stereo echo cancellation model in batches for training until convergence. The mean loss function on the validation set is compared at different training rounds, and the parameter with the minimum mean loss function is saved as the final model for stereo echo cancellation.

[0158] The loss function used in this invention is the Mean Square Error (MSE) loss function:

[0159]

[0160] Where N is the total number of time frames of the time-frequency characteristics obtained after performing a short-time Fourier transform on the time-domain signal.

[0161] Specifically, during training and validation, the echo path is randomly changed at a time between 4 and 6 seconds. Introducing abrupt changes in the echo path helps improve the stereo echo cancellation model's ability to adapt to these changes. The training set is divided into batches of 32 samples, and these batches are input into the stereo echo cancellation model described in Example 1 to obtain the estimated near-end speech signal for each batch. After obtaining the estimated near-end speech signal, the loss function is calculated using the label of the near-end speech signal, and the gradient of the parameters in the network is calculated through backpropagation to train the neural network. The validation set is divided into batches of 8 samples, and these batches are input into the stereo echo cancellation model described in Example 1 to obtain the estimated near-end speech signal. After obtaining the estimated near-end speech signal, the mean of the loss function is calculated using the label of the near-end speech signal. The batches of training and validation set samples are repeatedly input into the neural network for training until convergence. The loss function on the validation set is compared at different training epochs, and the parameter with the minimum mean loss function is saved as the final model for time delay estimation.

[0162] Finally, the trained stereo echo cancellation model is used to perform the stereo echo cancellation task.

[0163] Example 4

[0164] like Figure 8 As shown, the present invention also provides a stereo echo cancellation system, which includes: a loopback acquisition module 801, a microphone 802, an A / D module 803, and a stereo echo cancellation module 804.

[0165] The loopback acquisition module 801 is used to acquire reference signals (digital signals) before the speaker plays, including remote signal 1 and remote signal 2.

[0166] The microphone 802 is used to pick up signals from the near end of the room. The picked-up signal is called the microphone signal, which includes an echo signal and a near-end signal. The near-end signal includes a near-end speech signal and a near-end background noise signal. The echo signal is the target of the echo cancellation system, while the near-end speech signal is the signal that needs to be preserved.

[0167] The A / D module 803 is used to convert the microphone signal (analog signal) into a corresponding digital signal so that it, together with the remote signal 1 and the remote signal 2, can be output to the stereo echo cancellation module.

[0168] The stereo echo cancellation module 804 is used to process the far-end signal 1, far-end signal 2 and microphone signal using a trained stereo echo cancellation model to obtain an estimated near-end speech signal. The trained stereo echo cancellation model can be obtained using the stereo echo cancellation model training method in Example 2.

[0169] In addition, in practical application scenarios, the near-end voice signal needs to be transmitted to the far end for playback, which requires a signal transmission module 805, a signal receiving module 806, a D / A module 807, and a speaker 808.

[0170] The signal transmission module 805 is used to transmit the near-end speech signal output by the stereo echo cancellation system to the remote device.

[0171] The signal receiving module 806 is used to receive near-end voice signals transmitted from the near end to the far end.

[0172] The D / A module 807 is used to convert near-end voice signals (digital signals) into corresponding analog signals so that the signals can be played by speakers in a distant room.

[0173] The speaker 808 is used to play the analog signal of the near-end speech signal obtained after processing by the stereo echo cancellation model.

[0174] The stereo echo cancellation model includes: a signal time-frequency feature extraction module, a deep Kalman filter module including a reference signal estimation module and a Kalman filter process module, a residual echo and noise suppression module, and a time-domain signal recovery module.

[0175] The signal time-frequency feature extraction module is used to extract the time-frequency features of the input remote signal 1, remote signal 2 and microphone signal;

[0176] The deep Kalman filter module is used to perform a deep Kalman filter using the time-frequency features output by the signal time-frequency feature extraction module to obtain time-frequency features that have partially eliminated the echo signal. The deep Kalman filter module includes a reference signal estimation module, used to estimate a nonlinear reference signal using the time-frequency features of far-end signal 1, far-end signal 2, and the microphone signal; and a Kalman filtering process module, used to update the Kalman filter based on the time-frequency features of the microphone signal, far-end signal 1, far-end signal 2, and the estimated nonlinear reference signal to obtain time-frequency features that have partially eliminated the echo signal.

[0177] The residual echo and noise suppression module is used to suppress residual echo and noise in the echo-cancelled signal output by the deep Kalman filter module, thereby obtaining the time-frequency characteristics of the processed signal.

[0178] The time-domain signal recovery module is used to recover the time-frequency characteristics of the processed signal output by the residual echo and noise suppression module into a time-domain signal, thereby obtaining the estimated near-end speech signal.

[0179] The reference signal estimation module includes: a decoupling module, a nonlinear reference signal estimation module, a coupling module, and a feature splicing module;

[0180] The reference signal estimation module is used to estimate a nonlinear reference signal using the time-frequency characteristics of remote signal 1, remote signal 2, and the microphone signal, specifically including:

[0181] The decoupling module is used to decouple the complex time-frequency features output by the signal time-frequency feature extraction module, take the modulus of the complex time-frequency features to obtain the amplitude time-frequency features, and take the phase of the complex time-frequency features to obtain the phase time-frequency features;

[0182] The nonlinear reference signal estimation module is used to estimate the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module, and multiply the amplitude mask by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0183] The coupling module is used to couple the phase time-frequency characteristics of the microphone signal output by the decoupling module with the amplitude time-frequency characteristics of the nonlinear reference signal output by the nonlinear reference signal estimation module to obtain the time-frequency characteristics of the nonlinear reference signal.

[0184] The feature splicing module is used to splice the time-frequency features output by the signal time-frequency feature extraction module and the time-frequency features output by the nonlinear reference signal estimation module to obtain the reference signal of the Kalman filter.

[0185] Specifically, the nonlinear reference signal estimation module includes: a feature connection layer, an input dimension adjustment layer composed of stacked linear layers, a temporal modeling layer composed of stacked recurrent neural networks, an output dimension adjustment layer, a nonlinear activation layer composed of sigmoid functions, and a multiplication operation layer;

[0186] The nonlinear reference signal estimation module is used to estimate the amplitude mask using the amplitude time-frequency characteristics output by the decoupling module. The amplitude mask is then multiplied by the amplitude time-frequency characteristics of the microphone signal to obtain the amplitude time-frequency characteristics of the nonlinear reference signal, including:

[0187] The feature connection layer is used to connect the amplitude time-frequency features of remote signal 1, remote signal 2 and microphone signal into the input features of the neural network.

[0188] The input dimension adjustment layer is used to perform a linear mapping on the input features of the neural network output by the feature connection layer to obtain the dimension-adjusted input features;

[0189] The timing modeling layer is used to continuously receive the dimension-adjusted features output by the input dimension adjustment layer, thereby obtaining the timing relationship between the remote signal 1, the remote signal 2 and the microphone signal.

[0190] The output dimension adjustment layer is used to perform a linear mapping on the temporal features output by the temporal modeling layer to obtain the dimension-adjusted output features;

[0191] The nonlinear activation layer is used to compress the value range of the features output by the output dimension adjustment layer to the range of [0, 1] to obtain an amplitude mask;

[0192] The multiplication operation layer is used to multiply the amplitude mask output by the nonlinear activation layer with the amplitude time-frequency characteristics of the microphone signal output by the decoupling module to obtain the amplitude time-frequency characteristics of the nonlinear reference signal.

[0193] The Kalman filtering module specifically utilizes a recurrent neural network to estimate the process noise power spectral density based on the norm of the filter coefficients. The recurrent neural network includes a norm operation layer, a time-series modeling layer composed of stacked recurrent neural networks, a fully connected layer composed of stacked linear layers, and a sigmoid function layer. The norm operation layer calculates the norm of the filter coefficients and then concatenates the calculation results at each frequency point to form a feature vector of the filter coefficients. The time-series modeling layer continuously receives the feature vectors of the filter coefficients output by the norm operation layer, thereby obtaining the time-series features of the filter coefficients. The fully connected layer performs a linear mapping on the features output by the time-series modeling layer to obtain dimension-adjusted output features. The sigmoid function layer processes the features output by the fully connected layer through a sigmoid function, thereby compressing the value range of the features to [0,1] to obtain the process noise power spectral density.

[0194] like Figure 9 The stereo echo training device shown is used to train the stereo echo cancellation model. The stereo echo training device includes: a construction module 91, a training module 92, a verification module 93, and a model parameter determination module 94; wherein:

[0195] The construction module 91 is used to construct a training set and a validation set. Each set of samples in the training set and the validation set includes: an input signal and a near-end speech signal. The input signal includes a far-end signal 1, a far-end signal 2 and a microphone signal. The near-end speech signal has noise and echo and has time-frequency feature labels.

[0196] The training module 92 is used to divide the training set into batches of samples B1, and input the input signals in the divided samples into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combined with the time-frequency feature labels of the real near-end speech signal, the loss function value is calculated, and the calculated loss function value is used to train the stereo echo cancellation model by backpropagating the gradient of the parameters in the stereo echo cancellation model, where B1 is a positive integer greater than 1.

[0197] The verification module 93 is used to divide the verification set into batches of samples B2, and input the input signals in the divided samples into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each batch of samples. Combined with the time-frequency feature labels of the near-end speech signal, the mean of the loss function is calculated. The stereo echo cancellation model used in the verification set uses the model parameters obtained by training the training set in this round, where B2 is a positive integer greater than 1.

[0198] The model parameter determination module 94 is used to trigger the training module and the validation module when the stereo echo cancellation model has not converged, repeatedly inputting the samples of the training set and validation set into the stereo echo cancellation model in batches for training, comparing the mean loss function on the validation set at different training rounds, and saving the parameter with the smallest mean loss function as the trained model parameter.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A stereo acoustic echo cancellation method, comprising: collecting a far-end signal 1 and a far-end signal 2 before a speaker plays, and picking up a signal in a near-end room, wherein the picked-up signal is called a microphone signal, and contains an echo signal, a near-end speech signal and a near-end background noise signal; converting the microphone signal into a corresponding digital signal, so as to output the same together with the far-end signal 1 and the far-end signal 2 to a trained stereo acoustic echo cancellation model; the trained stereo acoustic echo cancellation model processing the far-end signal 1, the far-end signal 2 and the microphone signal to obtain an estimated near-end speech signal; wherein the trained stereo acoustic echo cancellation model comprises a signal time-frequency feature extraction module, a deep Kalman filtering module, a residual echo and noise suppression module and a time-domain signal recovery module, the deep Kalman filtering module comprises a reference signal estimation module and a Kalman filtering process module, and the processing of the far-end signal 1, the far-end signal 2 and the microphone signal by the trained stereo acoustic echo cancellation model comprises: the signal time-frequency feature extraction module extracting time-frequency features of the input far-end signal 1, far-end signal 2 and microphone signal; the deep Kalman filtering module eliminating part of the echo signal in the microphone signal by using the time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal; wherein the elimination process specifically comprises: the reference signal estimation module estimating a nonlinear reference signal by using the time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal; and the Kalman filtering process module updating a Kalman filter according to the time-frequency features of the microphone signal and the far-end signal 1, the far-end signal 2 and the estimated nonlinear reference signal, to obtain time-frequency features of the microphone signal from which part of the echo signal is eliminated; the residual echo and noise suppression module suppressing residual echo and noise by using a deep neural network on the time-frequency features of the microphone signal from which part of the echo signal is eliminated, to obtain time-frequency features of a processed signal; the time-domain signal recovery module recovering the time-frequency features of the processed signal into a time-domain signal to obtain the estimated near-end speech signal; the reference signal estimation module comprises a decoupling module, a nonlinear reference signal estimation module, a coupling module and a feature splicing module; the reference signal estimation module estimates a nonlinear reference signal by using the time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal, specifically comprising: the decoupling module decouples the complex time-frequency features output by the signal time-frequency feature extraction module, and obtains amplitude time-frequency features by taking modulus of the complex time-frequency features, and obtains phase time-frequency features by taking phase of the complex time-frequency features; the nonlinear reference signal estimation module estimates an amplitude mask by using the amplitude time-frequency features output by the decoupling module, and multiplies the amplitude mask with the amplitude time-frequency features of the microphone signal to obtain amplitude time-frequency features of the nonlinear reference signal; the coupling module couples the phase time-frequency features of the microphone signal output by the decoupling module with the amplitude time-frequency features of the nonlinear reference signal output by the nonlinear reference signal estimation module to obtain time-frequency features of the nonlinear reference signal; The feature splicing module splices the time-frequency features output by the signal time-frequency feature extraction module and the time-frequency features output by the nonlinear reference signal estimation module to obtain a reference signal of a Kalman filter.

2. The stereo acoustic echo cancellation method of claim 1, wherein, The nonlinear reference signal estimation module comprises a feature connection layer, an input dimension adjustment layer stacked by linear layers, a time series modeling layer stacked by recurrent neural networks, an output dimension adjustment layer, a nonlinear activation layer composed of a sigmoid function, and a multiplication operation layer. The nonlinear reference signal estimation module estimates an amplitude mask using the amplitude time-frequency features output by the decoupling module, multiplies the amplitude mask with the amplitude time-frequency features of the microphone signal to obtain the amplitude time-frequency features of the nonlinear reference signal, which comprise: The feature connection layer connects the amplitude time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal into input features of the neural network; The input dimension adjustment layer linearly maps the input features of the neural network output by the feature connection layer to obtain dimension-adjusted input features; The time series modeling layer continuously receives the dimension-adjusted features output by the input dimension adjustment layer to obtain the time series relationship of the far-end signal 1, the far-end signal 2 and the microphone signal; The output dimension adjustment layer linearly maps the time series features output by the time series modeling layer to obtain dimension-adjusted output features; The nonlinear activation layer compresses the range of values of the features output by the output dimension adjustment layer to a range of values, resulting in an amplitude mask; The multiplication operation layer multiplies the amplitude mask output by the nonlinear activation layer with the amplitude time-frequency features of the microphone signal output by the decoupling module to obtain the amplitude time-frequency features of the nonlinear reference signal.

3. The stereo acoustic echo cancellation method of claim 1, wherein, The Kalman filter process module updates the Kalman filter, which comprises: The Kalman filtering process module estimates the process noise power spectral density according to the norm of the filter coefficients by using a recurrent neural network; wherein the recurrent neural network comprises a norm operation layer, a time series modeling layer stacked by a recurrent neural network, a fully connected layer stacked by a linear layer, and an S-shaped function layer; the norm operation layer calculates the norm of the filter coefficients, and then splices the calculation results of each frequency point into a feature vector of the filter coefficients; the time series modeling layer continuously receives the feature vector of the filter coefficients output by the norm operation layer, thereby obtaining the time series characteristics of the filter coefficients; the fully connected layer performs linear mapping on the features output by the time series modeling layer, thereby obtaining the output features with adjusted dimensions; and the S-shaped function layer compresses the value range of the features output by the fully connected layer through the S-shaped function, thereby compressing the value range of the features to , and obtaining the process noise power spectral density.

4. The stereo acoustic echo cancellation method of claim 1, wherein, The stereo echo cancellation model is trained in the following manner: A training set and a validation set are constructed, and each group of samples in the training set and the validation set comprises an input signal and a near-end speech signal, the input signal comprising a far-end signal 1, a far-end signal 2 and a microphone signal, and the near-end speech signal having a time-frequency feature label; The training set is divided into B1 groups of samples at a time, and the input signals in the divided samples are input into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each group of samples in the batch, and the loss function value is calculated in combination with the time-frequency feature label of the near-end speech signal, and the stereo echo cancellation model is trained by back-propagating the gradient of the parameters in the stereo echo cancellation model using the calculated loss function value, wherein B1 is a positive integer greater than 1; The validation set is divided into B2 groups of samples at a time, and the input signals in the divided samples are input into the stereo echo cancellation model in batches to obtain the estimated time-frequency features of the near-end speech signal corresponding to each group of samples in the batch, and the loss function mean value is calculated in combination with the time-frequency feature label of the near-end speech signal, wherein the stereo echo cancellation model used by the validation set uses the model parameters obtained by training the training set in this round, and B2 is a positive integer greater than 1; The samples in the training set and the validation set are repeatedly input into the stereo echo cancellation model in batches for training until convergence, and the loss function mean values on the validation set at different training rounds are compared, and the parameters with the minimum loss function mean value are saved as the trained model parameters.

5. The stereo acoustic echo cancellation method of claim 4, wherein, Before dividing the training set according to a batch of B1 group samples, the method comprises: Randomly selecting a time point in each group of samples to introduce echo path changes in the input signal.

6. A stereo acoustic echo cancellation system characterized by, Comprise: Loop acquisition module, microphone, A / D module and stereo echo cancellation module; The loop acquisition module is used for collecting the far-end signal 1 and the far-end signal 2 before the loudspeaker plays; The microphone is used for picking up the signal in the near-end room, wherein the picked-up signal is called the microphone signal, which contains the echo signal, the near-end speech signal and the near-end background noise signal; The A / D module is used for converting the microphone signal into a corresponding digital signal, so as to output it together with the far-end signal 1 and the far-end signal 2 to the stereo echo cancellation module; The stereo echo cancellation module is used for processing the far-end signal 1, the far-end signal 2 and the microphone signal by using the trained stereo echo cancellation model, to obtain the estimated near-end speech signal; The stereo echo cancellation model comprises: The signal time-frequency feature extraction module is used for extracting the time-frequency features of the input far-end signal 1, the far-end signal 2 and the microphone signal; The deep Kalman filtering module is used for eliminating part of the echo signal in the microphone signal by using the time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal; wherein the deep Kalman filtering module comprises a reference signal estimation module, which is used for estimating the nonlinear reference signal by using the time-frequency features of the far-end signal 1, the far-end signal 2 and the microphone signal; and a Kalman filtering process module, which is used for updating the Kalman filter according to the time-frequency features of the microphone signal and the far-end signal 1, the far-end signal 2 and the estimated nonlinear reference signal, to obtain the time-frequency features of the signal from which part of the echo signal is eliminated; The residual echo and noise suppression module is used for suppressing the residual echo and noise of the echo-canceled signal output by the deep Kalman filtering module by using the deep neural network, to obtain the time-frequency features of the processed signal; and The time-domain signal recovery module is used for recovering the time-frequency features of the processed signal into a time-domain signal, to obtain the estimated near-end speech signal; The reference signal estimation module comprises a decoupling module, a nonlinear reference signal estimation module, a coupling module and a feature splicing module; The decoupling module is used for decoupling the complex time-frequency features output by the signal time-frequency feature extraction module, taking the modulus of the complex time-frequency features to obtain the amplitude time-frequency features, and taking the phase of the complex time-frequency features to obtain the phase time-frequency features; The nonlinear reference signal estimation module is used for estimating the amplitude mask by using the amplitude time-frequency features output by the decoupling module, multiplying the amplitude mask with the amplitude time-frequency features of the microphone signal, to obtain the amplitude time-frequency features of the nonlinear reference signal; The coupling module is used for coupling the phase time-frequency features of the microphone signal output by the decoupling module with the amplitude time-frequency features of the nonlinear reference signal output by the nonlinear reference signal estimation module, to obtain the time-frequency features of the nonlinear reference signal; The feature splicing module is used for splicing the time-frequency features output by the signal time-frequency feature extraction module and the time-frequency features output by the nonlinear reference signal estimation module, to obtain the reference signal of the Kalman filter.

7. The stereo acoustic echo cancellation system of claim 6, wherein, The nonlinear reference signal estimation module comprises a feature connection layer, an input dimension adjustment layer stacked by linear layers, a time series modeling layer stacked by recurrent neural networks, an output dimension adjustment layer, a nonlinear activation layer composed of a sigmoid function, and a multiplication operation layer; The nonlinear reference signal estimation module is configured to estimate an amplitude mask using the amplitude time-frequency features output by the decoupling module, multiply the amplitude mask and the amplitude time-frequency features of the microphone signal, and obtain the amplitude time-frequency features of the nonlinear reference signal, and the nonlinear reference signal estimation module comprises: The feature connection layer is configured to connect the amplitude time-frequency features of the far-end signal 1, the far-end signal 2, and the microphone signal into input features of the neural network. The input dimension adjustment layer is configured to perform linear mapping on the input features of the neural network output by the feature connection layer to obtain dimension-adjusted input features. The time series modeling layer is configured to continuously receive the dimension-adjusted features output by the input dimension adjustment layer to obtain the time series relationship of the far-end signal 1, the far-end signal 2, and the microphone signal. The output dimension adjustment layer is configured to perform linear mapping on the time series features output by the time series modeling layer to obtain dimension-adjusted output features. The nonlinear activation layer is configured to compress a value range of the features output by the output dimension adjustment layer to a range to obtain an amplitude mask; The multiplication operation layer is configured to multiply the amplitude mask output by the nonlinear activation layer and the amplitude time-frequency features of the microphone signal output by the decoupling module to obtain the amplitude time-frequency features of the nonlinear reference signal.

8. The stereo acoustic echo cancellation system of claim 6, wherein, The stereo acoustic echo cancellation model is trained through a construction module, a training module, a verification module, and a model parameter determination module, wherein: The construction module is configured to construct a training set and a verification set, and each group of samples in the training set and the verification set comprises an input signal and a near-end speech signal, the input signal comprises a far-end signal 1, a far-end signal 2, and a microphone signal, and the near-end speech signal has a time-frequency feature label; The training module is configured to divide the training set into B1 groups of samples at a time, input the input signal in the divided samples into the stereo acoustic echo cancellation model in batches, obtain the time-frequency features of the estimated near-end speech signal corresponding to each group of samples in the batch, calculate a loss function value in combination with the time-frequency feature label of the near-end speech signal, and train the stereo acoustic echo cancellation model by back-propagating the gradient of the parameters in the stereo acoustic echo cancellation model using the calculated loss function value, wherein B1 is a positive integer greater than 1; The verification module is configured to divide the verification set into B2 groups of samples at a time, input the input signal in the divided samples into the stereo acoustic echo cancellation model in batches, obtain the time-frequency features of the estimated near-end speech signal corresponding to each group of samples in the batch, calculate a loss function mean value in combination with the time-frequency feature label of the near-end speech signal, wherein the stereo acoustic echo cancellation model used by the verification set uses the model parameters obtained by training the training set in the current round, and B2 is a positive integer greater than 1. The model parameter determination module is configured to, when the stereo echo cancellation model does not converge, trigger the training module and the verification module to repeatedly input the samples of the training set and the verification set in batches to the stereo echo cancellation model for training, and compare the loss function mean values on the verification set at different training rounds, and save the parameters with the minimum loss function mean value as the trained model parameters.

Images