Systems and methods for acoustic echo cancellation using deep multi-task recurrent neural networks

Abstract

A system for performing echo cancellation, comprising: a processor configured to: receive a far-end signal; record a microphone signal, the microphone signal comprising: a near-end signal, and an echo signal corresponding to the far-end signal; extract far-end features from the far-end signal; extract microphone features from the microphone signal; calculate estimated near-end features by supplying the microphone features and the far-end features to an acoustic echo cancellation module, the acoustic echo cancellation module comprising a recursive neural network, the recursive neural network comprising: an encoder, the encoder comprising a plurality of gated recursive units, and a decoder, the decoder comprising a plurality of gated recursive units; calculate an estimated near-end signal from the estimated near-end features; and transmit the estimated near-end signal to a far-end device. The recursive neural network may include a contextual attention module; and the recursive neural network may take as input a plurality of error features calculated based on the far-end features, the microphone features, and the acoustic path parameters.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Patent Application No. 16 / 573,573 filed with the U.S. Patent and Trademark Office on September 17, 2019, U.S. Provisional Patent Application No. 62 / 825,681 filed with the U.S. Patent and Trademark Office on March 28, 2019, and U.S. Provisional Patent Application No. 62 / 838,146 filed with the U.S. Patent and Trademark Office on April 28, 2019, and this application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 914,875 filed with the U.S. Patent and Trademark Office on October 14, 2019, the entire disclosures of which are incorporated herein by reference. Technical Field

[0003] Various aspects of the embodiments of the present disclosure relate to the field of audio signal processing, and in particular include systems and methods for providing acoustic echo cancellation using deep multi-task recurrent neural networks. Background Art

[0004] During a real-time or near real-time audio or video conference interaction, a far-end participant may notice an acoustic echo in the signal received from a near-end participant. An acoustic echo may be generated when a far-end signal played out from a near-end speaker couples back into a microphone at the near-end. When the signal from the near-end microphone is transmitted to the far-end, the far-end user will hear a mixture of the near-end signal (e.g., the voice from the near-end participant) and a delayed and modified version of their own far-end voice as an acoustic echo. An acoustic echo canceller (AEC) or acoustic echo suppressor (AES) attempts to reduce or remove this echo while avoiding distorting the near-end user's voice (e.g., without distorting the near-end signal). Summary of the invention

[0005] Aspects of embodiments of the present disclosure relate to using deep multi-task recurrent neural networks to cancel acoustic echo in audio communication systems.

[0006] According to one embodiment of the present disclosure, a system for performing echo cancellation includes: a processor; and a memory storing instructions, which, when executed by the processor, cause the processor to: receive a far-end signal from a far-end device at a near-end device; record a microphone signal at the near-end device, the microphone signal including: a near-end signal and an echo signal corresponding to the far-end signal; extract a plurality of far-end features from the far-end signal; extract a plurality of microphone features from the microphone signal; calculate a plurality of estimated near-end features from the microphone signal by providing the microphone features and the far-end features to an acoustic echo cancellation module, the acoustic echo cancellation module including a contextual attention A context attention) neural network, the context attention neural network comprising: an encoder, the encoder comprising a plurality of gated recurrent units, the encoder being configured to encode microphone features and far-end features into encoded features; a context attention module, the context attention module being configured to calculate context attention features from the encoded features; and a decoder, the decoder comprising a plurality of gated recurrent units, the decoder being configured to decode estimated near-end features from the context attention features; calculate estimated near-end signals from the estimated near-end features; and transmit the estimated near-end signals to a far-end device.

[0007] The encoder may include one or more layers of gated recurrent units, and the encoded features may correspond to the output of the last layer of the one or more layers of gated recurrent units of the encoder, and the encoded features may be supplied to a multihead self-attention layer of the contextual attention module; and the decoder may include one or more layers of gated recurrent units, and the estimated proximal features correspond to the output of the last layer of the one or more layers of gated recurrent units of the decoder.

[0008] The contextual attention module may further include a multi-head attention layer configured to receive an output of the multi-head self-attention layer.

[0009] The contextual attention module may further include a first normalization layer between the multi-head self-attention layer and the multi-head attention layer.

[0010] The contextual attention module may further include a second normalization layer configured to calculate contextual attention features based on outputs from other layers of the contextual attention module.

[0011] The memory may further store instructions which, when executed by the processor, cause the processor to: calculate a plurality of acoustic path parameters based on previous acoustic path parameters, previous far-end features, and previous microphone features; and calculate a plurality of error features based on the far-end features, the microphone features, and the acoustic path parameters, and the acoustic echo cancellation module may be further configured to calculate a plurality of estimated near-end features from the microphone signal based on the error features.

[0012] The distal features supplied to the contextual attention neural network may include a current frame of the distal features and a causal window of multiple previous frames of the distal features, and the microphone features supplied to the contextual attention neural network may include a current frame of the microphone features and a causal window of multiple previous frames of the microphone features.

[0013] The estimated proximal features may include a current frame of the estimated proximal features and a causal window of multiple previous frames of the estimated proximal features, and the contextual attention neural network may be trained by iteratively training multiple parameters that configure the contextual attention neural network to minimize the difference between the estimated proximal features and multiple ground truth proximal features of multiple training data.

[0014] Training data may be generated by: loading a corpus of training data comprising recorded utterances from a plurality of different human speakers; selecting a plurality of pairs of human speakers; for each pair of human speakers: concatenating a plurality of utterances of a first speaker in the pair of human speakers to generate a training far-end signal; transforming the concatenated utterances to simulate an acoustic path to generate a training echo signal; padding an utterance of a second speaker in the pair of human speakers to generate a training near-end signal having a length equal to the training far-end signal; and mixing the training echo signal with the training near-end signal to generate a training microphone signal.

[0015] Transforming the cascade of speech may include convolving the far-end signal with a room impulse response simulating the room.

[0016] Transforming the cascaded speech may further include: applying hard clipping to the far-end signal to generate a clipped far-end signal; and applying sigmoidal distortion (S-shaped distortion) to the clipped far-end signal.

[0017] The far-end features, microphone features, and estimated near-end features may include logarithmic short-time Fourier transform features in a logarithmic spectral space.

[0018] According to one embodiment of the present disclosure, a system for performing echo cancellation includes: a processor; and a memory storing instructions, which, when executed by the processor, causes the processor to: receive a far-end signal from a far-end device; record a microphone signal, the microphone signal including: a near-end signal, and an echo signal corresponding to the far-end signal; extract a plurality of far-end features from the far-end signal; extract a plurality of microphone features from the microphone signal; calculate a plurality of acoustic path parameters based on previous acoustic path parameters, previous far-end features, and previous microphone features; calculate a plurality of error parameters based on the far-end features, the microphone features, and the acoustic path parameters. Features; by supplying microphone features, far-end features and error features to an acoustic echo cancellation module, a plurality of estimated near-end features are calculated from the microphone signal, the acoustic echo cancellation module comprising a recursive neural network, the recursive neural network comprising: an encoder comprising a plurality of gated recursive units, the encoder being configured to encode the microphone features and the far-end features into encoded features; and a decoder comprising a plurality of gated recursive units, the decoder being configured to decode the estimated near-end features based on the encoded features; calculate an estimated near-end signal from the estimated near-end features; and transmit the estimated near-end signal to a far-end device.

[0019] The acoustic echo cancellation module may further include a contextual attention module configured to calculate contextual attention features from the encoded features, and the decoder may be configured to calculate the estimated proximal features based on the contextual attention features.

[0020] The far-end features supplied to the acoustic echo cancellation module may include a causal window of a current frame of the far-end features and multiple previous frames of the far-end features, the microphone features supplied to the acoustic echo cancellation module may include a causal window of a current frame of the microphone features and multiple previous frames of the microphone features, and the error features supplied to the acoustic echo cancellation module may include a causal window of a current frame of the error features and multiple previous frames of the error features.

[0021] The estimated proximal features may include a current frame of the estimated proximal features and a causal window of multiple previous frames of the estimated proximal features, and the recursive neural network may be trained by iteratively training multiple parameters that configure the recursive neural network to minimize the difference between the estimated proximal features and multiple true value proximal features of multiple training data.

[0022] Training data may be generated by: loading a corpus of training data comprising recorded utterances from a plurality of different human speakers; selecting a plurality of pairs of human speakers; for each pair of human speakers: concatenating a plurality of utterances of a first speaker in the pair of human speakers to generate a training far-end signal; transforming the concatenated utterances to simulate an acoustic path to generate a training echo signal; padding an utterance of a second speaker in the pair of human speakers to generate a training near-end signal having a length equal to the training far-end signal; and mixing the training echo signal with the training near-end signal to generate a training microphone signal.

[0023] Transforming the cascade of speech may include convolving the far-end signal with a room impulse response simulating the room.

[0024] Transforming the cascaded speech may include: applying hard clipping to the far-end signal to generate a clipped far-end signal; and applying a sigmoidal distortion to the clipped far-end signal.

[0025] The far-end features, microphone features, and estimated near-end features may include logarithmic short-time Fourier transform features in a logarithmic spectral space. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings, together with the specification, illustrate exemplary embodiments of the present disclosure, and, together with the description, serve to explain the principles of the present disclosure.

[0027] Figure 1A is a schematic diagram illustrating a communication system including an acoustic echo cancellation (AEC) module configured to reduce or remove acoustic echo.

[0028] Figure 1B is a schematic diagram illustrating a communication system including an acoustic echo cancellation (AEC) module and a residual echo suppression (RES) module configured to reduce or remove acoustic echo.

[0029] Figure 2 is a block diagram illustrating an acoustic echo cancellation system according to one embodiment of the present disclosure.

[0030] Figure 3 is a block diagram illustrating an input layer of an unfolded deep gated recurrent unit (GRU) network according to one embodiment of the present disclosure.

[0031] Figure 4 is a block diagram illustrating the architecture of an acoustic echo cancellation (AEC) neural network according to one embodiment of the present disclosure.

[0032] Figure 5Ais a flow chart depicting a method for training a recursive neural network of an acoustic echo cancellation module according to one embodiment of the present disclosure.

[0033] Figure 5B is a flowchart of a method for generating training data according to an embodiment of the present disclosure.

[0034] Figure 5C is a flowchart of a method for generating a simulated echo signal y(t) according to one embodiment of the present disclosure.

[0035] Fig. 6A is a block diagram illustrating the architecture of an acoustic echo cancellation (AEC) neural network operating in inference mode according to one embodiment of the present disclosure.

[0036] Figure 6B is a flow chart depicting a method for computing an estimated near-end signal from a received far-end signal and a microphone signal using a neural network in an inference mode according to one embodiment of the present disclosure.

[0037] Figure 7 is a block diagram illustrating a hybrid system including a deep multi-task acoustic echo cancellation system according to one embodiment of the present disclosure.

[0038] Fig. 8A is a schematic block diagram of an acoustic echo cancellation system according to one embodiment of the present disclosure, in which a deep neural network (denoted as “EchoDNN”) is used to cancel echoes from microphone signals.

[0039] Figure 8B is an architectural block diagram of a deep neural network configured to cancel echo from a microphone signal according to one embodiment of the present disclosure.

[0040] Fig. 9 is a block diagram illustrating an acoustic echo cancellation system using a generative network according to one embodiment of the present disclosure.

[0041] Fig.10 is a block diagram illustrating an acoustic echo cancellation system including a contextual attention neural network according to one embodiment of the present disclosure.

[0042] Fig.11A is a block diagram describing a contextual attention-aware neural network according to one embodiment of the present disclosure.

[0043] Fig. 11B is a more detailed block diagram depicting the layers of a contextual attention neural network according to one embodiment of the present disclosure.

[0044] Fig. 11Cis a block diagram depicting a contextual attention module according to one embodiment of the present disclosure.

[0045] Fig.12 is a flow chart depicting a method for computing an estimated near-end signal from a received far-end signal and a microphone signal using a contextual attention neural network in an inference mode according to one embodiment of the present disclosure.

[0046] Fig.13 is a block diagram illustrating an acoustic echo cancellation system according to one embodiment of the present disclosure, the acoustic echo cancellation system including a hybrid of a Frequency Domain Normalized Mean Square (FDNLMS) technique combined with a contextual attention neural network.

[0047] Fig.14A is a block diagram depicting a contextual attention neural network according to one embodiment of the present disclosure, wherein the contextual attention neural network is configured to accept far-end features, error features, and microphone features as inputs.

[0048] Fig. 14B is a more detailed block diagram depicting the layers of a contextual attention neural network configured to accept far-end features, error features, and microphone features as inputs according to one embodiment of the present disclosure.

[0049] Fig.15 is a flow chart depicting a method for computing an estimated near-end signal from a received far-end signal and a microphone signal using a hybrid of a classical acoustic echo cancellation technique and a contextual attention neural network in an inference mode according to one embodiment of the present disclosure.

[0050] Fig.16 is a block diagram of a proximal device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In the detailed description below, only by way of illustration, certain exemplary embodiments of the present disclosure are shown and described. As will be appreciated by those skilled in the art, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings and the following discussion, the same reference numerals represent the same components.

[0052] Figure 1A is a schematic diagram showing a communication system including an acoustic echo cancellation (AEC) module configured to reduce or remove acoustic echo. Figure 1AAs shown, a far-end signal x(t) is received and played back on a near-end speaker 12 of a near-end device 10. Typically, the far-end signal x(t) contains speech from a far-end user or participant in an interaction, where the far-end signal x(t) is captured by a far-end communication device that transmits the far-end signal x(t) to be received by the near-end communication device (e.g., via a communication network such as a cellular communication network). Thus, for convenience, the far-end signal x(t) may sometimes be referred to herein as "far-end speech." The sound produced by the near-end speaker 12 may be detected by the near-end microphone 14 as an acoustic echo y(t), and the sound may travel to the near-end microphone 14 over a plurality of different echo paths 16, such as after the sound bounces off walls and other surfaces 18 in the physical environment in which the near-end device 10 is located. The near-end microphone 14 may also detect a desired near-end signal s(t), such as speech from a near-end user or participant in an interaction. Therefore, the signal d(t) generated by the near-end microphone 14 may contain both the desired near-end signal s(t) and the undesired acoustic echo y(t). The microphone signal d(t) is provided to an acoustic echo cancellation (AEC) module 100, which attempts to cancel the echo signal y(t) and generate an output signal q(t) that approximates the desired near-end signal s(t) (e.g., an output signal q(t) corresponding to the microphone signal d(t) with the acoustic echo y(t) removed). The processed output signal q(t) can then be sent to the far end as an estimated desired near-end signal s(t) with the echo signal y(t) removed.

[0053] For clarity, as used herein, given v(t) as an arbitrary time-domain signal at time t: the short-time Fourier transform (STFT) complex-valued spectrum v(t) in frame k and frequency bin f is given by V k，f Indicated by ∠V k，f Expressed as: Its logarithmic amplitude is expressed as express. A vector representing the logarithmic magnitudes of all frequency bins f and frames k.

[0054] Figure 1B 1 is a schematic diagram showing a communication system including an acoustic echo cancellation (AEC) module and a residual echo suppression (RES) module configured to reduce or remove acoustic echo. A microphone signal d(t) includes a near-end speech signal s(t) and an acoustic echo y(t):

[0055] d(t)=s(t)+y(t)

[0056] In some embodiments, the microphone signal d(t) also includes other components, such as additive noise n(t) (e.g., d(t) = s(t) + y(t) + n(t)). The acoustic echo signal y(t) is a modified version of the far-end speech signal x(t), including the room impulse response (RIR) and speaker distortion, both of which may cause nonlinearity in the relationship between x(t) and y(t).

[0057] Broadly speaking, the acoustic echo cancellation (AEC) problem is to retrieve the net near-end signal s(t) after removing the acoustic echo due to the far-end signal x(t) detected by the near-end microphone 14. Figure 1B As shown, the comparative system estimates a model of the echo path using a linear adaptive filter 110, and then subtracts the estimated echo v(t) from the microphone signal d(t). In addition, to achieve echo estimation, some comparative AEC methods use a double talk detector (DTD) to stop filter adaptation during double talk when both near-end and far-end signals are present (compared to a one-way talk period, during which only the near-end signal or only the far-end signal is present or not negligible). Typically, after applying AEC, a certain amount of residual echo (d(t)-v(t)) is still present at the output of the AEC method. One of the reasons why the comparative AEC methods cannot completely remove all echo noise is that even when the echo path 16 is completely linear, the echo path 16 is not a linear function of the far-end speech signal x(t). For example, the audio amplifier (e.g., power amplifier) ​​and the near-end speaker 12 may cause nonlinearity in the relationship between the far-end speech signal x(t) and the echo signal y(t). In addition, the length of the adaptive linear filter may not be sufficient to remove longer echoes (e.g., echoes in large rooms or corridors). Therefore, a residual echo suppressor (RES) 150 may be further applied to improve the near-end signal, wherein the RES may be implemented by a Wiener filter or spectral subtraction in the frequency domain. The final output of the AEC system is the estimated near-end signal q(t).

[0058] Various aspects of embodiments of the present disclosure relate to recurrent neural network (RNN) architectures for acoustic echo cancellation (AEC). Some embodiments relate to using deep gated recurrent unit (GRU) networks in encoder / decoder architectures (e.g., see K. Cho, B. van C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwen and Y. Bengio, “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation” in Proc. Empirical Methods in Natural Language Processing, 2014, pp. 1724-1734 and J. Chung, C. Gulcehre, K. Cho and Y. Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling” in Proc. NIPS Deep Learning Workshop, 2014) are used to map the spectral features of the microphone signal d(t) and the far-end signal x(t) to a hyperspace (e.g., a feature space such as log-frequency space), and then decode the target spectral features of the near-end signal s(t) from the encoded hyperspace. In some embodiments, a RNN acoustic echo cancellation module is trained using multi-task learning to learn an auxiliary task of estimating an echo signal y(t) in order to improve the primary task of estimating a net near-end speech signal s(t) as an estimated near-end signal q(t). As discussed in more detail below, experimental results show that embodiments of the present disclosure eliminate acoustic echo in both one-way talk and double-talk periods with nonlinear distortion without the need for a separate double-talk detector.

[0059] Figure 2 200 is a block diagram illustrating an acoustic echo cancellation system according to one embodiment of the present disclosure. The microphone signal d(t)=s(t)+y(t) is supplied to the acoustic echo cancellation system 200 to cancel or remove the echo signal y(t). For convenience, various aspects of the embodiments of the present disclosure will be described herein in the context of an audio signal sampled at 16 kHz. However, the embodiments of the present disclosure are not limited thereto and may be applied with audio signals sampled at a rate higher than 16 kHz or at a rate lower than 16 kHz.

[0060] exist Figure 2In the embodiment shown in FIG. 1 , the microphone signal d(t) is supplied to a microphone signal feature extraction module 210, which includes a short-time Fourier transform (STFT) module 212, an absolute value module 214 for removing the conjugate symmetric half of the output of the STFT module 212, and a final logarithmic magnitude spectrum feature vector or logarithmic spectrum feature or microphone signal feature in a "feature space" or "hyperspace" or "log spectrum space". The STFT module 212 can also calculate the phase ∠D of the microphone signal d(t) k，f .

[0061] For convenience, various aspects of embodiments of the present disclosure will be described herein where a spectral feature vector is calculated using a 512-point short-time Fourier transform (STFT) with a frame offset of 256 points (assuming a 16 kHz sampling rate, each frame corresponding to 32 milliseconds, and a 16 millisecond offset between frames, resulting in a 16 millisecond overlap between frames). In some embodiments, the absolute value module 214 reduces the 512-point STFT magnitude vector to 257 points by removing the conjugate symmetric halves. In some embodiments, features (e.g., microphone signal features) are calculated using scalars calculated from training data. ) is standardized to have zero mean and unit variance, as discussed in more detail below. As will be appreciated by those skilled in the art, the spectral feature vectors may be calculated with more than 512 points or less than 512 points and with longer or shorter frame offsets (e.g., more overlap or less overlap between frames).

[0062] Similar to the above Figure 1B In a more systematic manner, some aspects of embodiments of the present disclosure relate to estimating acoustic echo using a received far-end signal x(t). In more detail, in some embodiments, the far-end signal x(t) may also be applied to a far-end signal feature extraction module 220. In some embodiments, the far-end signal feature extraction module 220 is substantially similar to the microphone signal feature extraction module 210 and includes an STFT module 222, an absolute value module 224, and a logarithmic operation module 226. The far-end signal feature extraction module 220 calculates the far-end signal feature from the far-end signal x(t). (in feature space or hyperspace). According to some embodiments of the present disclosure, the far-end signal feature is supplied to the echo estimator 230, which is configured to calculate the estimated echo characteristics (in feature space or hyperspace).

[0063] In some embodiments of the present disclosure, the near-end estimator 250 receives the microphone signal characteristics Far-end signal characteristics and the estimated echo characteristics (or another output of the echo estimator 230) to calculate the estimated near-end speech feature Estimated near-end speech features The signal may then be supplied to a feature inversion module (or signal synthesis module) 270, which may include an exponential operation module 272 (to invert the logarithmic operation applied to the input signal) and an inverse short-time Fourier transform (iSTFT) module 274 to convert the estimated near-end speech features into The feature space or hyperspace is transformed into a time domain signal q(t), which is the estimated near-end speech or near-end signal s(t).

[0064] In various speech processing applications, using past and / or future data frames can help compute estimated features for the current frame. In some such speech processing applications, a fixed context window is used as input to the fully connected first layer of a deep neural network. In these comparative methods, after this first layer, context information may be lost as the information flows through deeper layers.

[0065] Therefore, some aspects of embodiments of the present disclosure use contextual features for both the input and output of the neural network in order to keep contextual information available throughout the neural network. According to some embodiments, the input features of the current frame include the feature vector of the current frame k and the feature vectors of the six previous frames or causal frames (k-1, k-2, ..., k-6) According to some embodiments of the present disclosure, a causal window (using only data from previous frames, not data from future frames) is selected to prevent additional delays (e.g., when a causal window of frames is used, there is no need to wait for the arrival of future frames k+1, k+2, ... before processing the current frame k). The seven frames with 50% overlap of the embodiments discussed above produce a receptive field of 112 milliseconds, which is generally long enough for processing speech signals. In order to incorporate context awareness, some aspects of embodiments of the present disclosure involve using an expanded deep gated recurrent unit (GRU) network with seven time steps (or frames) for both the echo estimation module and the near-end estimation module. However, embodiments of the present disclosure are not limited to this and may be implemented with more than six previous data frames or less than six previous data frames.

[0066] Figure 3 is a block diagram showing the input layer of an expanded deep gated recurrent unit (GRU) network according to one embodiment of the present disclosure. Figure 3As shown, each of the seven GRUs in the input GRU layer 232 is input from the corresponding current frame and the previous six frames Each of the seven GRUs of the input GRU layer 232 calculates a hidden feature vector or activation h. For example, the GRU of the input GRU layer 232 corresponding to the k-6th frame receives the corresponding feature vector And calculate the activation The activations from each previous GRU from the input GRU layer 232 are supplied to the next GRU in the sequence. For example, the activations from the GRU corresponding to the k-6th frame is supplied as the input of the GRU of the k-5th frame. Therefore, the GRU of the k-5th frame is fed from its corresponding input feature vector and the activation from the previous frame Calculate its activation This recursive arrangement allows contextual information from previous frames to be used when processing the current frame.

[0067] According to some embodiments of the present disclosure, each GRU calculates its output activation according to the following formula:

[0068]

[0069] where ⊙ is an element-wise multiplication and updates the gate z k for:

[0070]

[0071] Where σ is the sigmoid function. Candidate hidden states Calculated by the following formula

[0072]

[0073] where elu is the exponential linear unit function and the reset gate r k Calculated by the following formula

[0074]

[0075] In the formula U, W, U r , W r , U z and W z is the internal weight matrix of the GRU. In some embodiments, each of the GRUs in a given layer (e.g., each of the GRUs in the input GRU layer 232) uses the same set of weights (i.e., the "recursive" nature of the neural network). In some embodiments, the values ​​of the internal weight matrix are learned through a training process, which will be described in more detail below.

[0076] Figure 4 2 is a block diagram showing the architecture of an acoustic echo cancellation (AEC) neural network according to one embodiment of the present disclosure. The acoustic echo cancellation (AEC) neural network 228 includes a deep neural network model including two stacked GRU networks. The first stack corresponds to Figure 2 The echo estimator 230 shown in FIG. 1 and the context-aware frame As input to each GRU in the input GRU layer 232, and using a fully connected (FC) output layer 236 with linear activation to estimate the echo features exist Figure 4 In the embodiment shown in FIG. 1 , the first stack includes a hidden GRU layer 234. In more detail, the first GRU layer or input GRU layer 232 calculates the first activation As mentioned above Figure 3 The first activation is supplied to the second GRU layer 234 to calculate the second activation exist Figure 4 In the embodiment shown in FIG. 2 , the second GRU layer 234 is also the last GRU layer of the stack. Thus, the activation of the last GRU layer of the stack (here, the second activation ) is supplied to the fully connected output layer 236 to calculate the estimated echo features However, embodiments of the present disclosure are not limited thereto, and more than one hidden layer may be included in the neural network of the echo estimator 230. For example, one or more additional GRU layers may be inserted between the input GRU layer 232 and the last GRU layer 234, wherein each i-th additional layer will be based on the activation of the previous layer. To calculate its own activation

[0077] exist Figure 2 In the embodiment shown in FIG. 1 , the output from the last GRU layer 234 of the first stack (echo estimator 230) is is supplied to the cascade layer 251 to correspond to the far-end signal The characteristics of the microphone signal d(t) The context-aware frames of are concatenated to create the concatenated input c to the GRU layer of the second stack (proximal estimator 250). Figure 4 In the embodiment shown in FIG. 1 , the second stack (proximal estimator 250) includes three GRU layers: a third GRU layer 253 (which receives the cascaded input c), a fourth GRU layer 254, and a fifth GRU layer 255, which calculate the corresponding third activation Fourth Activation and the fifth activation The activation from the last hidden GRU layer (in Figure 4 In the example shown in FIG. 1 , the fifth activation from the fifth GRU layer 255 is ) is supplied to the fully connected output layer 259 to estimate the context-aware frame of the features of the estimated near-end speech like Figure 4 As shown, the network can also calculate the previous six frames The estimated near-end speech of , which can be used to calculate the loss during training according to the loss function, as discussed in more detail below.

[0078] As mentioned above, in Figure 4 In the embodiment shown in FIG. 1 , data from the current frame k and six previous frames k-6, k-5, ..., k-1 are used to calculate the features of the estimated near-end speech Also as discussed above, Figure 4 The embodiment shown in FIG. 1 utilizes a 257-point eigenvector. Therefore, in general, the eigenvector of the far-end signal x(t) is The seven frames of have a dimension of 7×257. Figure 4 In the embodiment shown in , the activation or output h of each of the GRU units is a vector of length 1000, but the embodiments of the present disclosure are not limited thereto. Figure 4 The embodiment uses seven data frames, so the output or activation of each of the GRU layers 232 and 234 has a dimension of 7×1000. To match the shape of the input feature vector, the output estimated echo feature Each of has a length of 257, and thus the output of the fully connected output layer 236 of the echo estimator has a dimension of 7×257.

[0079] exist Figure 4 In the embodiment shown in FIG. 1 , the microphone signal d(t) is supplied to a feature extractor substantially identical to the feature extractor used with the far-end signal x(t), and thus the microphone signal feature vector for each frame is For each frame, there is also a feature vector of length 257 values, so that the microphone signal features The seven frames of 251 produce a total dimension of 7×257. Each of the cascade units of the cascade layer 251 combines the output of the last GRU layer 234 of the first stack (having a length of 1000) (e.g., output activation h2), the distal signal feature (with a length of 257 values) and the corresponding microphone signal features of the frame (having a length of 257) is cascaded so that the output c of each of the cascaded units has a length of 1000+257+257=1514, and the output of the cascaded layer 251 has a dimension of 7×1514.

[0080] exist Figure 4 In the embodiment shown in FIG. 1 , in a manner similar to the echo estimator 230, each GRU layer 253, 254, and 255 of the near-end estimator 250 produces an output activation h of dimension 7×1000 (e.g., each GRU unit map computes an activation or activation vector of length 1000 values), and the final fully connected layer 259 produces a feature vector of the estimated near-end speech of the current frame and the previous six frames. (having length 257), where the dimension of the output of the fully connected layer 259 is 7×257.

[0081] Figure 5A is a flowchart depicting a method for training a recursive neural network of an acoustic echo cancellation module according to one embodiment of the present disclosure. Typically, the supervised learning process of training a neural network includes calculating various parameters of the neural network, such as weights and biases of the neurons of the network, by adjusting the parameters to minimize a loss function between the output of the neural network and the "ground truth" data of a labeled training data set for a given set of inputs in the training data. The training can be performed by a computer system that accesses the training data (the computer system includes a processor and a memory, and can be accelerated using a vector processor such as a graphics processing unit), where the training data can be divided into a training set, a test set, and in some embodiments, a validation set. Typically, the training set is used to learn the parameters of the network, the validation set is used to determine various hyperparameters of the network (e.g., architectural parameters, such as the number of neurons in each unit and the number of layers in the neural network), and the test set is used to evaluate the overall performance of the trained system.

[0082] In the specific domain of acoustic echo cancellation described herein, the training data may include: a far-end signal x(t); a near-end signal s(t); and an echo signal y(t). In some embodiments of the present disclosure, a method 500 for training a recursive neural network of an acoustic echo cancellation module includes: at 510, a computer system generates training data in a manner similar to that described in "Deep Learning for Acoustic Echo Cancellation in Noisy and Double-Talk Scenarios" by H. Zhang and D. Wang in Proc. Annual Conference of the International Speech Communication Association, 2018, pp. 3239-3243. In some embodiments, the TIMIT dataset is used to generate training data (e.g., see "Speech database development: Design and analysis of the acoustic-phonetic corpus," by F. Lamel, RH Kassel, and S. Seneff in Speech Input / Output Assessment and Speech Databases, 1989).

[0083] Figure 5B is a flowchart of a method for generating training data according to one embodiment of the present disclosure. In more detail, in some embodiments, the input data set may include recorded speech from multiple different human speakers or voices. The method 510 for generating training data includes: At 511, multiple pairs of human speakers may be selected (e.g., randomly) to serve as near-end speakers and far-end speakers. At 513, from each pair, three utterances of the far-end speaker are randomly selected and concatenated to create a true far-end signal x(t). At 515, for each pair, the far-end signal x(t) is converted to produce a corresponding echo signal y(t) by simulating the effects of the acoustic path. The following discusses in more detail Figure 5C A method for generating a simulated true-value echo signal y(t) according to one embodiment of the present disclosure is described.

[0084] At 517, a true near-end signal s(t) is generated by padding zeros before and after the utterance to have the same size as the far-end signal, padding or extending each utterance of the near-end speaker in the pair to the same length as the length of its corresponding far-end signal x(t) (e.g., for each concatenated far-end signal generated from the paired far-end human speakers). (Embodiments of the present disclosure are not limited in this regard, and in some embodiments, noise is added to the entire padding signal.) In some embodiments, more than one pair of far-end signal x(t) and near-end signal s(t) is selected for each near-end far-end pair.

[0085] At 519, the computer system mixes (e.g., adds) the true value echo signal y(t) and the true value near-end signal s(t) calculated for each pair to generate a corresponding training microphone signal d(t). For the training mixture, in some embodiments, at 519, the computer system generates the training microphone signal d(t) by mixing the near-end speech signal and the echo signal to a signal-to-echo ratio (SER) level randomly selected from {6, 3, 0, 3, 6} dB. The SER level is calculated during double talk as follows:

[0086]

[0087] Figure 5C 5 is a flow chart of a method 515 according to one embodiment of the present disclosure for generating a simulated or true value echo signal y(t) in a manner similar to that described in “State-space frequency-domain adaptive filtering for nonlinear acoustic echo cancellation,” by S. Malik and G. Enzner, IEEE Transactions on audio, speech, and language processing, vol. 20, no. 7, pp. 2065-2079, 2012. For a nonlinear model of the acoustic path, at 515-1, the computer system applies hard clipping to simulate a power amplifier of a loudspeaker (in one embodiment, x max is set to 80% of the maximum volume of the input signal):

[0088]

[0089] At 515-3, to simulate speaker distortion, the computer system applies a sigmoidal function, such as:

[0090]

[0091] Where, if b(t)>0, then b(t)=1.5xclip (t)-0.3x clip (t) 2 and a=4, and otherwise a=0.5.

[0092] According to one embodiment, at 515-5, a room impulse response (RIR) g(t) is randomly selected from the set of RIRs, where each of the RIRs has a length of 512, the simulated room size is 4 meters × 4 meters × 3 meters, and the simulated microphone is fixed at a position of [221.5] meters (at the center of the room). The simulated speaker is placed at seven random positions at a distance of 1.5 meters from the microphone. In some embodiments of the present disclosure, multiple different RIRs are also generated using different room sizes and different placements of simulated microphones and / or simulated speakers.

[0093] In some embodiments, RIR is obtained using an image method with a reverberation time (T 60 ) (e.g., see JB Allen, DA Berkley, "Image method for efficiently simulating small-room acoustics" in The Journal of Acoustic Society of America, vol. 65, no. 4, pp. 943-950, 1979). In some embodiments, from the generated RIRs, some of the RIRs are used to generate training data (e.g., can be randomly selected), while other RIRs are reserved for generating test data.

[0094] At 515-7, the output of the sigmoidal function is convolved with a randomly selected room impulse response (RIR) g(t) to simulate a distorted (non-linear) far-end signal x played through the speakers in the room. nl Acoustic transmission of (t):

[0095] y nl (t) = x nl (t)*g(t)

[0096] Where * indicates a convolution operation.

[0097] In some embodiments, the linear acoustic path y is simulated simply by convolving the original far-end signal x(t) with RIRg(t) lin (t) to generate an echo signal where nonlinearities such as clipping and speaker distortion are not applicable to this model:

[0098] y lin (t) = x(t)*g(t)

[0099] Back to Figure 5A At 520, the computer system uses the feature extractor described above to calculate feature vectors (true near-end feature vectors) from the respective parts of the training data (true near-end signal s(t), true far-end signal x(t), true echo signal y(t), and microphone signal d(t)). Ground truth distal features True echo feature and microphone characteristics ).

[0100] At 530, the computer system trains the neural network of the AEC 228 based on the training data. In more detail, as discussed above, each of the GRUs is based on an internal weight matrix U, W, U r , W r , U z and W z Its corresponding activation h is calculated from its input. In addition, each of the fully connected units includes a plurality of internal weights W and biases b (e.g., applying an affine function of the form Wx+b) for mapping the input of the fully connected unit to an output in a feature space (e.g., STFT space).

[0101] Training the neural network involves learning the internal weights of the GRU and fully connected units so that the output feature vector (the estimated proximal feature and the estimated echo characteristics ) is close to the true feature vector (true value proximal feature and ground truth echo features ). Output feature vector and Between and true feature vector and The difference between them can be measured using a loss function, which represents how well the neural network configured with the current set of internal weights approximates the underlying data.

[0102] In one embodiment, a mean absolute error (MAE) loss function is used to train the neural network. In the feature domain (e.g., STFT domain, as discussed above), the mean absolute error between the true source (near-end signal s(t)) and the network estimated output (estimated near-end signal q(t)) is calculated. Some embodiments use a weighted loss function that takes into account the near-end signal s(t) and the echo path signal y(t) to calculate the network weights. Therefore, in one embodiment, the loss for a given frame k is calculated based on the current frame and the previous six frames according to the following formula:

[0103]

[0104] where β is the weighting factor between the loss associated with the near-end signal and the loss associated with the echo signal, Corresponding to the true proximal feature of the i-th frame, The estimated proximal features corresponding to the i-th frame, corresponds to the true echo feature of the i-th frame, and Echo features corresponding to the estimate of the i-th frame. In embodiments where m previous data frames are used for context (eg, a causal window of length m frames), the sum is from n=0 to m. For convenience, in the embodiments described in detail herein, m=6.

[0105] In some embodiments of the present disclosure, gradient descent and back propagation are used to calculate weights. In particular, the weights are iteratively adjusted based on the difference between the current output of the neural network and the true value. In some embodiments of the present disclosure, AMS Grad optimization (e.g., see J. Reddi, S. Kale, and S. Kumar, “On the convergence of Adam and beyond” at International Conference on Learning Representations (ICLR), 2018) and in particular Adam variants (see, e.g., DPKingma and JLBa, “Adam: a method for stochastic optimization” at International Conference on Learning Representations (ICLR), 2015) are used by setting β for 100 epochs. 1 =0.9, β 2 = 0.999 and ∈ = 10 -3 , and the model is trained with a batch size of 100. In some embodiments, the weights of all layers are initialized using the Xavier method (e.g., see X. Glorot and Y. Bengio in Proc. International Conference on Artificial Intelligence and Statistics, 2010, pp. 249-256, "Understanding the difficulty of training deepfeedforward neural networks") and using biases initialized to zero. In some embodiments, L2 regularization of all weights with a regularization constant of 0.000001 is used to prevent overfitting.

[0106] After training the weights of the neural network, the trained network can be tested using a test set of the training data to verify the accuracy of the network. As described above, the test set can be formed using utterances from speakers not used in the training set and / or using RIRs and / or other distortions not present in the training set. Thus, the test set can be used to evaluate whether the training process has trained the neural network to perform a generalized function of acoustic echo cancellation, rather than overfitting to specific features of the training data (e.g., removing acoustic echo features of a specific human speaker or RIRs of the training data).

[0107] After training the neural network and determining that the performance of the trained network is sufficient (e.g., based on a test set), the weights can be saved and used to configure the neural network running on an end-user device (such as a smartphone or tablet). In various embodiments of the present disclosure, the neural network of the acoustic echo cancellation module is implemented on at least one processor 1620 of the proximal device 10 (e.g., see Figure 11), where the processor can be: a general-purpose central processing unit; a graphical processing unit (GPU); a field programmable gate array (FPGA); a neural processing unit (NPU) or a neural network processor (NNP) (e.g., a processor with an architecture customized to perform inference using a neural network); or a neuromorphic processor. For example, the parameters of the neural network (e.g., weights and biases) and the neural network architecture can be stored in a non-transitory memory connected to the processor, where the processor performs inference using the network by loading the parameters and network architecture from the memory. As another example, in the case of an FPGA, the FPGA can be configured with the network architecture and weights in a non-transitory manner using a bit file. Because the training process can be considered complete or stable, in some embodiments of the present disclosure, the end-user device may operate the neural network only in inference mode to calculate the currently estimated proximal features. Or the estimated near-end signal q(t).

[0108] Fig. 6A is a block diagram illustrating the architecture of an acoustic echo cancellation (AEC) neural network operating in inference mode according to one embodiment of the present disclosure. Fig. 6A As shown, during inference, it is not necessary to calculate the previous frame The estimated near-end features of the AEC neural network are, for example, because when using the AEC neural network in the field, the near-end device 10 only needs to calculate the current estimated frame and send it to the far-end (e.g., the previously estimated near-end frame has already been sent to the far-end). (During training, the previous frame helps provide additional information to the loss function). Similarly, Fig. 6A As shown, the fully connected output layer 236 of the first stack of the echo estimator 230 can be omitted from the inference mode neural network because the second stack of the proximal estimator 250' in the inference mode depends only on the output of the last GRU layer 234 of the first stack of the echo estimator 230'.

[0109] Figure 6B 6 is a flow chart depicting a method 600 for computing an estimated near-end signal from a received far-end signal and a microphone signal using a neural network in an inference mode according to one embodiment of the present disclosure. Figure 6B As shown, at 602, the acoustic echo cancellation system 200 receives a far-end signal x(t), and at 604, the far-end signal feature extraction module 220 extracts a far-end feature from the far-end signal x(t). Additionally, at 606, a first stack of neural networks corresponding to the echo estimator 230′ is generated from the distal features The echo estimator output is calculated. (In some embodiments, the echo estimator output corresponds to the output of the last GRU layer 234 of the first stack of the echo estimator 230'. ).

[0110] Similarly, at 612, the acoustic echo cancellation system 200 receives the microphone signal d(t), and at 614, the microphone signal feature extraction module 210 extracts the microphone signal feature d(t) from the microphone signal d(t).

[0111] At 620, a second stack of neural networks corresponding to the proximal estimator 250′ is generated from the distal features Echo estimator characteristics (e.g., h) and microphone characteristics Compute estimated proximal features like Fig. 6A As shown, when calculating the estimated proximal features of the current frame k When , in addition to the current frame k, features from the context of previous frames (e.g., six previous frames k-6, k-5, ..., k-1) are also supplied to the near-end estimator 250'. In particular, Fig. 6A As shown, when calculating the estimated proximal features of the current frame k When the distal features from the current frame k and the six previous frames k-6, k-5, ..., k-1 The echo estimator output h and microphone characteristics (e.g., distal features Echo Estimator Output and microphone characteristics ) is used to estimate the proximal features

[0112] At 622, the feature inversion module 270 of the acoustic echo cancellation system 200 extracts the estimated near-end features of the current frame from the near-end features of the current frame. Calculate the estimated near-end signal q(t) of the current frame. As described above, the features (e.g., far-end signal features Microphone characteristics and the estimated proximal features ) can be in a feature space or hyperspace, such as STFT space (e.g., spectral features or spectral domain). Therefore, in some embodiments, the feature inversion module 270 converts the estimated spectral features The feature space is converted into a time domain signal q(t) suitable for playback on the speaker at the remote device. Figure 2 As shown, when calculating the estimated near-end signal q(t), the phase LD of the microphone signal d(t) k，f It may also be used by the inverse short-time Fourier transform (iSTFT) module 274 .

[0113] Figure 7 is a block diagram illustrating a hybrid system including a deep multi-task acoustic echo cancellation system according to one embodiment of the present disclosure. In these embodiments, the multi-task GRU network has the same architecture as described above. However, for the training of the network and during inference, the output of the frequency domain normalized least mean square (NLMS) filter 700 is used instead of the microphone signal d(t). The far-end signal x(t) remains as another input to the acoustic echo cancellation system 200 (including the multi-task GRU network), and the output target is still the near-end feature (If estimated That) and echo characteristics (If estimated like that).

[0114] Experimental results using neural network technology

[0115] To evaluate the performance of the acoustic echo cancellation system 200 as described above, experiments were performed using training data generated from the TIMIT dataset (e.g., see F.Lamel, RHKassel, and S.Seneff, "Speech database development: Design and analysis of the acoustic-phonetic corpus" in Speech Input / OutputAssessment and Speech Databases, 1989). In some embodiments of the present disclosure, seven utterances of a near-end speaker are used to generate 3500 training mixtures, where each near-end signal is mixed with five different far-end signals. From the remaining 430 speakers, 100 pairs of speakers are randomly selected as far-end speakers and near-end speakers. To generate 300 test mixtures, the same process as above is used but with only three near-end speakers' utterances, where each near-end signal is mixed with one far-end signal. Therefore, the test mixtures are from human speakers who are not part of the training set.

[0116] The Perceptual Evaluation of Speech Quality (PESQ) scores of the unprocessed test mixtures for the linear and non-linear models (without echo cancellation) are shown in Table 1. The unprocessed PESQ scores are calculated by comparing the microphone signal with the near-end signal during double talk.

[0117] Table 1: PESQ scores for the untreated test mixture in the linear and nonlinear models of the acoustic path

[0118]

[0119] In some cases, echo return loss enhancement (ERLE) is used to evaluate the echo reduction achieved by the acoustic echo cancellation system 200 according to an embodiment of the present disclosure in a one-way talk situation where only echo is present, where ERLE is defined as:

[0120]

[0121] Where E is the statistical expectation operation achieved through averaging.

[0122] To evaluate the performance of the system during double talk, we use Perceptual Evaluation of Speech Quality (PESQ). In some embodiments, PESQ is calculated by comparing the estimated near-end speech q(t) with the true near-end speech s(t) during double talk periods only. PESQ scores range from -0.5 to 4.5, and higher scores indicate better quality.

[0123] In the following discussion, frequency domain normalized least mean square (FDNLMS) (e.g., see C. Faller and J. Chen, "Suppressing acoustic echo in a spectral envelope space" in IEEE Transactions on Acoustic, Speech and Signal Processing, vol. 13, no. 5, pp. 10481062, 2005) is used as a comparative example. A double-talk detector (DTD) is used based on the energy of the microphone signal d(t) and the far-end signal x(t). In some cases, the post-processing algorithm is further based on the method presented in “The echo shaping approach to acoustic echo control” by R. Martin and S. Gustafsson in Speech Communication, vol. 20, no. 3-4, pp. 181-190, 1996. The embodiments of the present disclosure are also compared with the bidirectional long short-term memory (BLSTM) method described in “Deep Learning for Acoustic Echo Cancellation in Noisy and Double-Talk Scenarios” by H. Zhang and D. Wang in Proc. Annual Conference of the International Speech Communication Association, 2018, pp. 3239-3243.

[0124] The embodiments of the present disclosure are compared with a comparison method using a linear model of the acoustic path (e.g., linear acoustic echo). Table 2 shows the average ERLE values ​​and PESQ gains of a traditional NLMS filter, a BLSTM, and a context-aware multi-task GRU (denoted as "CA multi-task GRU") according to an embodiment of the present disclosure. The PESQ gain is calculated as the difference in the PESQ value of each method relative to its unprocessed PESQ value. Table 2 also shows the results of a context-aware single-task GRU (denoted as "CA single-task GRU") according to an embodiment of the present disclosure, which uses only a single-task GRU with and The second stack of GRU layers takes as input, where the loss function is calculated by taking the true feature vector s(t) with respect to only the near-end speech s(t). The results show that the multi-task GRU according to some embodiments of the present disclosure outperforms the single-task GRU according to some embodiments of the present disclosure in both PESQ and ERLE. It also shows that the embodiments of the present disclosure outperform both the traditional NLMS+ post-processing and BLSTM methods under all conditions.

[0125] Table 2: ERLE and PESQ scores in the linear model of the acoustic path

[0126]

[0127] The embodiments of the present disclosure are also compared with a comparative method using a nonlinear model of the acoustic path (e.g., nonlinear acoustic echo). In this set of experiments, the nonlinear ground truth echo signal y nl (t) is used to generate the microphone signal d(t), so the model includes both power amplifier clipping and loudspeaker distortion (e.g., corresponding to Figure 5C 515-3 and 515-7 of ). The results of the embodiments of the present disclosure are compared with the comparative ES+RES method including NLMS. In the nonlinear acoustic path, the performance is also compared with the deep neural network (DNN)-based residual echo suppression (RES) system described in "DNN-based residual echo suppression" by CMLee, JWShin and NSKim in Proc. Annual Conference of the International Speech Communication Association, 2015, pp. 17751779 and recorded as "AES+DNN" in the table. The results in Table 3 show that the embodiments of the present disclosure outperform the other two comparison methods in both PESQ and ERLE.

[0128] Table 3: ERLE and PESQ scores in the nonlinear model of the acoustic path

[0129]

[0130] Embodiments of the present disclosure achieve excellent echo reduction without significant near-end distortion (eg, spectra corresponding to the estimated near-end signal and the actual near-end signal are very similar).

[0131] The performance of embodiments of the present disclosure was also evaluated in the presence of additive noise and a nonlinear model of the acoustic path. In these embodiments, when generating training data, white noise with an SNR of 10 dB was added to the near-end signal s(t), while the nonlinear acoustic path was at a SER level of 3.5 dB. The embodiments of the present disclosure were then compared to a conventional NLMS+ post-processing system. As shown in Table 4 below, various aspects of the embodiments of the present disclosure outperform the comparative methods by a large margin.

[0132] Table 4: ERLE and in the nonlinear model of acoustic path (SER = 3.5 dB) and additive noise (SNR = 10 dB) PESQ score

[0133]

[0134] In addition, the alternative hybrid embodiments discussed above were evaluated for unseen RIRs for different reverberation times and speaker distances from the microphone. In this evaluation, the model was trained and tested using the same RIRs as discussed above (corresponding to a room size of 4 meters × 4 meters × 3 meters, a reverberation time of 200ms, and a random speaker distance of 1.5 meters from the microphone, and a total length of 512 samples). During testing of the hybrid system according to an embodiment of the present disclosure, the speaker distance was changed to 15cm. The results of the frequency domain NLMS and the hybrid method of the NLMS and multi-task GRU according to an embodiment of the present disclosure trained with the above RIRs are shown in Table 5. The multi-task GRU was further fine-tuned using the RIRs generated under various room sizes (small, medium, large), different reverberation times (from 250 milliseconds to 900 milliseconds) and a speaker distance of 15cm. The fine-tuning results are also shown in Table 5 below. These results suggest that the hybrid approach according to some embodiments of the present disclosure may perform better if the model is fine-tuned using the impulse response of the target device (eg, the target end-user near-end device).

[0135] Table 5: ERLE and PESQ scores of the hybrid method

[0136]

[0137] Additional Embodiments

[0138] Some embodiments of the present disclosure relate to different architectures of the neural network of the acoustic echo cancellation system 200 . Fig. 8Ais a schematic block diagram of an acoustic echo cancellation system according to one embodiment of the present disclosure, in which a deep neural network (denoted as "EchoDNN") is used to cancel the echo from the microphone signal d(t). According to some embodiments of the present disclosure, EchoDNN 900 uses only a plurality of fully connected (FC) layers.

[0139] Figure 8B FIG. 1 is a block diagram of an architecture of a deep neural network configured to cancel echo from a microphone signal according to an embodiment of the present disclosure. Figure 8B As shown, the features extracted from the microphone signal d(t) and the far-end signal x(t) and is fed into the input layer 902 in the fully connected deep network (EchoDNN) 900. Figure 8B In the embodiment shown in , the network includes three hidden layers 904, 906 and 908, and an output layer 910. Each of these layers can be a fully connected layer that implements, for example, an affine transformation. In some embodiments, each of the three fully connected hidden layers 904, 906 and 908 includes 4000 units (e.g., neurons). In some embodiments, the output layer 910 is also a fully connected layer with 257 (equal to the number of frequency bins of the extracted Log-Mag feature) units. For the Log-Mel-Mag feature, in some embodiments, the output layer uses 80 units. In some embodiments, an exponential linear unit (ELU) is used as the activation function of each unit.

[0140] The estimated features 912 of the near-end signal are obtained directly from the output of the fully connected deep network (EchoDNN) 900. At 972, these features are converted back to the time domain to synthesize the estimated near-end speech signal, for example, using the above-mentioned feature inversion module 270. In some embodiments, for the microphone signal d(t) and the near-end signal s(t) sampled at a rate of 16kHz, a frame size of 512 samples with 50% overlap is used. A 512-point short-time Fourier transform (STFT) is then applied to each frame of the input signal, resulting in 257 frequency bins. After calculating the logarithm operation on the amplitude value, the final logarithmic magnitude (Log-Mag) feature is calculated. In some embodiments of the present disclosure, log-mel magnitude (Log-Mel-Mag) is used as the final feature 912 to reduce the dimensionality of the feature space and thus reduce the complexity of the technology applied in these embodiments. In some embodiments, the features are compressed by using an 80-dimensional Mel-transform matrix.

[0141] In order to use context information, in some embodiments, features of context frames of two input signals are also extracted and concatenated as input features.

[0142] In various embodiments, during training, the logarithmic magnitude (Log-Mag) features or the Log-mel magnitude (or Log-Mel-Mag) of the near-end speech signal are used as target labels.

[0143] In some embodiments, AMSGRAD is used as an optimizer during training. In some embodiments, the mean absolute error (MAE) between the target label and the output of the network is used as a loss function.

[0144] Fig. 9 is a block diagram illustrating an acoustic echo cancellation system using a generative network according to one embodiment of the present disclosure. Fig. 9 In the embodiment shown in , a generative model 990, such as Wavenet (see, e.g., Oord, Aaron vanden et al., “Wavenet: A generative model for raw audio.” arXiv preprint arXiv:1609.03499 (2016)) replaces the feature inversion module 270 at the output of the AEC network (e.g., at the output of the near-end estimator 250). A generative network (such as Wavenet) can enhance the output of the acoustic echo cancellation system 228 based on information (e.g., mel-log-amplitude) provided by the AEC output or “Mel-spec”. The generative model 990 reconstructs the near-end speech signal q(t) with much higher accuracy than the inverse STFT (iSTFT) 274.

[0145] Echo Removal Using Contextual Attention Neural Network

[0146] Attention-based models provide a class of methods for performing sequence-to-sequence learning tasks (e.g., from a sequence of input audio frames to a sequence of output audio frames). Some aspects of embodiments of the present disclosure relate to contextual attention neural networks (e.g., multi-head attention networks) for acoustic echo cancellation (AEC). An example embodiment includes an encoder / decoder architecture with a multi-head contextual self-attention mechanism (or neural network) that maps spectral features of microphone signals and far-end signals to estimate near-end signals. Experiments using both simulated and real recorded room impulse responses (RIRs) show that the contextual attention method according to some embodiments of the present disclosure achieves better performance in echo return loss enhancement (ERLE) scores and perceptual evaluation of speech quality (PESQ) scores compared to baseline echo removal techniques (such as encoder / decoder architectures lacking attention-based components).

[0147] Fig.10 is a block diagram illustrating an acoustic echo cancellation system including a contextual attention neural network according to one embodiment of the present disclosure. Fig.10 The embodiment shown in Figure 2 , but differs in that the echo canceller 1028 does not include a separate echo estimator (e.g., similar to Figure 2 and Figure 4 ). Instead, as described in more detail below, the echo canceller 1028 includes a contextual attention neural network configured to extract information from the microphone signal features. and far-end signal characteristics Compute estimated near-end speech features No need to train a network to estimate echo features Fig.10 Like reference numerals in Figure 2 Similar components to the embodiments shown in FIG. Fig.10 Similar components will not be described in detail.

[0148] like Fig.10 As shown, similar to Figure 2 In an embodiment of the present invention, the microphone signal d(t)=s(t)+y(t) is supplied to the acoustic echo cancellation system 1000 to cancel or remove the echo signal y(t). The microphone signal d(t) is supplied to a microphone signal feature extraction module 1010, which includes a short-time Fourier transform (STFT) module 1012, an absolute value module 1014 for removing the conjugate symmetric half of the output of the STFT module 1012, and a final logarithmic magnitude spectrum feature vector or logarithmic spectrum feature or microphone signal feature in a "feature space" or "hyperspace" or "log spectral space". The STFT module 1012 can also calculate the phase ∠D of the microphone signal d(t). k，f .

[0149] like Fig.10 As shown, similar to Figure 2 In some embodiments, the far-end signal x(t) may also be applied to the far-end signal feature extraction module 1020. In some embodiments, the far-end signal feature extraction module 1020 is substantially similar to the microphone signal feature extraction module 1010 and includes an STFT module 1022, an absolute value module 1024, and a logarithmic operation module 1026. The far-end signal feature extraction module 1020 calculates the far-end signal feature from the far-end signal x(t). (in feature space or hyperspace).

[0150] exist Fig.10 In the embodiment shown in FIG. 1 , the microphone signal characteristic and far-end signal characteristics is supplied to an echo canceller 1028 (or acoustic echo canceller), which includes a contextual attention neural network (which may be referred to as a contextual attention-aware model).

[0151] Fig.11A is a block diagram describing a contextual attention neural network according to one embodiment of the present disclosure. As described above, in some embodiments of the present disclosure, the contextual attention neural network 1100 is Fig.10 The components of the echo canceller 1028 are shown in FIG. Fig.10 , Microphone signal characteristics Know the remote signal characteristics Calculate proximal features used in the process.

[0152] Fig. 11B is a more detailed block diagram depicting the layers of a gated recurrent unit (GRU) of a contextual attention neural network according to one embodiment of the present disclosure.

[0153] like Fig.11A and Fig. 11B As shown, the contextual attention neural network 1100 includes an encoder module 1110, a contextual attention module 1130, and a decoder module 1190, wherein each of these modules includes one or more layers of neurons (e.g., layers of gated recurrent units), as discussed in more detail below. The encoder module 1110 is configured to receive a microphone signal feature and far-end signal characteristics As input. Fig.11A In the embodiment shown in , each of the two inputs has dimensions of T×257, where T is the number of frames of input used in the causal window or context window. Figure 3 , Figure 4 and Fig. 6A In the particular example embodiment shown and described, the number of frames T is set to seven frames (T=7), where the current frame and six previous causal frames are used to perform echo removal. As described above, embodiments of the present disclosure are not limited to the case where the window has a size of seven frames (T=7). As described above, embodiments of the present disclosure are described herein where features are represented as 257 point vectors. Thus, Fig.11A and Fig. 11B As shown, the microphone signal characteristics and far-end signal characteristics Each frame of includes 257×2=514 features, and thus the total input dimension of network 1100 is T×514.

[0154] like Fig.11A As shown, the encoder module 1110 obtains the microphone signal characteristics and far-end signal characteristics and map them into a hyperspace or latent space to generate the encoding feature h, where Fig.11A The encoding features in the embodiment shown in have dimensions of T×1000:

[0155]

[0156] In more detail, Fig. 11B In some embodiments of the present disclosure, the encoder module 1110 includes a cascade layer 1112 configured to cascade input microphone signal features. and far-end signal characteristics (each with dimensions T×257) to compute concatenated features of dimension T×514. The concatenated features are then fed to a Figure 3 The gated recurrent unit (GRU) layer 1114 of the GRU layer shown in FIG. 1 , wherein each unit of the GRU layer 1114 calculates the encoding feature h according to the following formula:

[0157]

[0158] Where f represents the GRU function, and h t represents the output of GRU at time t.

[0159] The contextual attention module 1130 uses the encoded feature h to calculate the contextual attention (or contextual attention feature) c that identifies the important regions of the hyperspace, where Fig.11A The context attention c in the illustrated embodiment has a dimension of T×1000:

[0160] c=Attention(h)

[0161] In more detail, the contextual attention mechanism allows the model to learn the importance of each speech frame in context. This is achieved by weighting the speech frames when constructing representations of the frames in a hyperspace or latent space. Applying contextual attention according to embodiments of the present disclosure is particularly effective during double talk periods. As described above, although the attention mechanism can be applied to both past frames and future frames, in order to avoid or reduce delays, some embodiments of the present disclosure only use past frames.

[0162] Fig. 11C is a block diagram depicting a contextual attention module according to one embodiment of the present disclosure. Fig. 11CThe context-afternoon attention module 1130 shown in FIG. 1 includes two multi-head attention (MHA) layers (or modules) 1131 and 1134, wherein the first multi-head attention layer 1131 implements a context-self-attention layer. In general, a context-self-attention layer (such as the multi-head self-attention layer 1131) is used to capture the similarity of a speech frame relative to any hyperspace representation of all past neighboring speech frames in the input sequence.

[0163] For convenience, the computations performed by the multi-head self-attention (MHSA) layer will be described in more detail below in the context of the multi-head self-attention layer 1131. In some embodiments, the multi-head attention (MHA) module 1134 is substantially identical in architecture to the MHSA module 1131, but uses different inputs than the MHSA module 1131, such as Fig. 11C As shown in FIG. 1 and described in more detail below, the weight parameter a t，t′ Capture the logarithmic amplitude of speech frames at time step t and time step t′ as input (e.g., concatenated and h t and h t ′) The similarity between the hidden state representations supplied to the layer. In some embodiments, the contextual self-attention mechanism calculates the weight parameter a according to the following formula t，t′ :

[0164]

[0165] where e t，t′ is the attention importance score calculated for each time step. In some embodiments of the present disclosure, the attention importance score e is calculated using the “additive” method t，t′ :

[0166]

[0167] In some embodiments of the present disclosure, the “general” method is used to calculate the attention importance score e t，t′ :

[0168]

[0169] In some embodiments, the attention importance score e t，t′ After being computed to construct the probability distribution over the speech frames, it is normalized (e.g., by applying a softmax activation function to the scores).

[0170] In some embodiments, the output of the multi-head self-attention layer 1131 is represented by the hidden state h of all past frames at time step t′ t′ And its hidden state representation h with the current frame t Similarity of a t，t′ The weighted sum of gives the focused hidden state representation of the frame at time step t. More informally, the output of the multi-head self-attention layer 1131 represents the degree to which we should pay attention to the speech frame at any time step based on the neighboring context:

[0171] ∑ t′ a t，t′ h t′

[0172] In order to improve the calculation of the importance score e t，t′ In some embodiments, a temperature parameter is used in the softmax activation function, and three learnable weights (W a , W b , W c ) instead of just one weight W a :

[0173]

[0174] (See, e.g., G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network” in NIPS Deep Learning and Representation Learning Workshop, 2015).

[0175] In some embodiments of the present disclosure, Fig. 11C As shown, the multi-head self-attention layer is used together with the multi-head attention layer 1134 (for example, see A. Vaswani et al., "Attention is All you Need" in Advances in Neural Information Processing Systems, I. Guyon, UV Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 59986008), where both the multi-head self-attention layer 1131 and the multi-head attention layer 1134 have residual connections (skip connections) and layer normalization.

[0176] In more detail, Fig. 11C As shown, the contextual attention module 1130 (or contextual attention layer) according to one embodiment includes a multi-head self-attention module 1131, which receives three copies of the encoded feature h. The output of the multi-head self-attention module 1131 is supplied to a first summing node 1132, which adds the output of the multi-head self-attention module 1131 to another copy of the encoded feature h. Then, the output of the first summing node 1132 is supplied to a first layer normalization module 1133 (e.g., see Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer normalization" in arXiv preprint arXiv: 1607.06450 (2016)), and the layer normalization activation is supplied to the multi-head attention module 1134 as input together with the two additional copies of the encoded feature h. In other words, the query to the multi-head attention layer 1134 is the output of the first layer normalization module 1133 and the output of the encoder (encoded feature h). The output of the multi-head attention module 1134 is supplied to a summing node 1135, which adds a copy of the output of the first layer normalization module 1133 before outputting the sum to the second layer normalization layer 1136. The output of the second layer normalization layer is then output from the contextual attention module 1130 as the contextual attention (or contextual attention feature) c.

[0177] In some embodiments of the present disclosure, the multi-head attention module 1134 and the second layer normalization module 1136 are omitted, so that the output of the first layer normalization module 1133 is regarded as the output c of the context attention module 1130.

[0178] Return to reference Fig.11A and Fig. 11B , and then uses the decoder module 1190 to calculate the near-end speech from the context attention c The estimated logarithmic magnitude of Fig.11A and Fig. 11B In the embodiment shown in FIG. Has dimensions of T×257 (eg, the same dimensions as the input microphone signal features and the far-end signal features).

[0179] In more detail, in some embodiments, such as Fig. 11B In the illustrated embodiment, the decoder includes two GRU layers 1192 and 1194. According to one embodiment, the first GRU layer 1192 is based on the context attention c (e.g., c k-T+1 , c k-T+2 , ..., ck ), calculate its output s (for example, s k-T+1 ,s k-T+2 , ..., s k ):

[0180] s t =g 1 (c t ,s t-′ )

[0181] And the second GRU layer 1194 calculates the estimated near-end speech according to the following formula (For example )’s spectral characteristics:

[0182]

[0183] In some embodiments, an exponential linear unit (elu) activation function is used with the first GRU layer 1192 and a linear activation function is used with the second GRU layer 1194.

[0184] like Fig.10 As shown, near-end speech The estimated features are converted back into an estimated near-end speech signal q(t), which can be sent to the far-end communication device.

[0185] In some embodiments, Fig.11A , Fig. 11B and Fig. 11C The contextual attention neural network 1100 shown in FIG. 1 uses the same Figure 5A , Figure 5B and Figure 5C The contextual attention neural network 1100 can be trained using substantially the same training process as described above, and the resulting trained contextual attention neural network 1100 can be deployed into a communication device (e.g., an end-user device such as a smartphone or tablet) in a manner similar to that described above.

[0186] Figure 6B 1 is a flow chart depicting a method 1200 for computing an estimated near-end signal from a received far-end signal and a microphone signal using a contextual attention neural network 1100 in an inference mode according to one embodiment of the present disclosure. Fig.12 As shown, at 1202, the acoustic echo cancellation system 1000 receives a far-end signal x(t), and at 1204, the far-end signal feature extraction module 1020 extracts a far-end feature from the far-end signal x(t).

[0187] Similarly, at 1206, the acoustic echo cancellation system 1000 receives the microphone signal d(t), and at 1208, the microphone signal feature extraction module 1010 extracts the microphone signal feature from the microphone signal d(t).

[0188] At 1210, the contextual attention neural network 1100 (e.g., a component of the echo canceller 1028) extracts information from the distal features. and microphone characteristics Compute estimated proximal features For example, Fig. 11B As shown, when calculating the estimated proximal features of the current frame k , in addition to the current frame k, features from the context of the previous T frames (e.g., six previous frames k-6, k-5, ..., k-1) are also supplied to the context attention neural network 1100. In particular, Fig. 11B As shown, when calculating the estimated proximal features of the current frame k When the distal features from the current frame k and T-1 previous frames k-T+1, k-T+2, ..., k-1 and microphone characteristics (e.g., distal features and microphone characteristics ) is used to calculate the estimated proximal features

[0189] At 1212, the feature inversion module 1070 of the acoustic echo cancellation system 1000 extracts the estimated near-end features of the current frame from the near-end features of the current frame. To calculate the estimated near-end signal q(t) of the current frame. As described above, the features (e.g., far-end signal features Microphone characteristics and the estimated proximal features ) can be in a feature space or hyperspace, such as STFT space (e.g., spectral features or spectral domain). Therefore, in some embodiments, the feature inversion module 1070 converts the estimated spectral features The feature space is converted into a time domain signal q(t) suitable for playback on the speaker of the remote communication device, wherein the feature inversion module 1070 may include an exponential module 1072 and an inverse short-time Fourier transform (iSTFT) module 1074. Fig.10 As shown, when calculating the estimated near-end signal q(t), the phase ∠D of the microphone signal d(t) k，f It can also be used by the inverse short-time Fourier transform (iSTFT) module 1074.

[0190] Experimental results of echo cancellation using contextual attention neural network technology

[0191] An embodiment of the acoustic echo cancellation system 1000 for removing acoustic echo using the contextual attention neural network 1100 according to an embodiment of the present disclosure is trained and tested in a manner similar to that described above for testing the acoustic echo cancellation system 200 .

[0192] To create a benchmark for comparison, training data generated from the TIMIT dataset was used to measure the performance of a frequency-domain normalized least mean square (FDNLMS) model using double talk detection (DTD) based microphone and far-end signal energy (see, e.g., C. Faller and J. Chen, “Suppressing acoustic echo in a spectral envelope space” in IEEE Trans. Speech Audio Process., vol. 13, no. 5, pp. 10481061, 2005).

[0193] Some additional tests were performed on the effect of applying the post-filtering method to FDNMLS audio (FDNMLS+post-filtering), based on "The echo shaping approach to acoustic echo control" by R. Martin and S. Gustafsson, Speech Commun., vol. 20, no. 34, pp. 181 190, 1996. Additional tests were performed using a deep neural network (DNN) instead of post-filtering, using the method described in "DNN-based residual echo suppression" by CMLee, JWShin, and NSKim, INTERSPEECH, 2015, vol. 2015-Janua, pp. 17751779, where the parameters of the DNN were set according to the values ​​given in the paper. (FDNLMS+DNN).

[0194] The fourth benchmark uses basically the same Fig.11A The same encoder / decoder GRU network architecture is shown in , but the contextual attention module 1130 is excluded (e.g., the output of the encoder module 1110 is directly supplied as input to the decoder module 1190). (Encoder / Decoder GRU).

[0195] Embodiments of the present disclosure corresponding to five different variations for calculating attention are described above. They include using a general attention mechanism, an additive attention mechanism, or a scaled dot product attention mechanism, as well as using multi-head self-attention (MHSA) alone and using MHSA with multi-head attention (MHSA+MHA). Table 6 shows the ERLE and PESQ scores in the linear model of the acoustic path using synthetic RIR

[0196] Table 6: ERLE and PESQ scores in the linear model of the acoustic path

[0197]

[0198]

[0199] As shown in Table 6, the method using contextual attention according to an embodiment of the present disclosure generally outperforms the baseline echo removal system, including FDNLMS, FDNLMS+post-filtering, FDNLMS+DNN, and a bare encoder / decoder GRU without a contextual attention module. In a variant of an embodiment of the present disclosure, the use of MHSA and MHA together shows the best ERLE and PESQ scores.

[0200] The disclosed embodiments are also compared with FDNLMS+post-filtering and encoder / decoder GRU baselines when removing echo due to nonlinear acoustic paths (including analog power amplifier clipping and speaker distortion). Table 7 shows the ERLE and PESQ scores in the acoustic path nonlinear model (SER=3.5dB) using synthetic RIR.

[0201] Table 7: ERLE and PESQ scores in the nonlinear model of the acoustic path

[0202]

[0203] As shown in Table 7 above, when experiments were conducted using synthetic RIR, the contextual attention GRU echo canceller according to an embodiment of the present disclosure outperformed FDNLMS+post-filtering and encoder / decoder GRU (without contextual attention module) in both ERLE and PESQ scores.

[0204] Table 8 below further illustrates the use of impulse response data from the Aachen impulse response database (see, for example, M. Jeub, M. and P. Vary, “A binaural room impulse response database for the evaluation of dereverberation algorithms”, International Conference on Digital Signal Processing, 2009), ERLE and PESQ scores of the contextual attention GRU echo canceller compared to the same FDNLMS+postfiltering and encoder / decoder GRU (without the contextual attention module) baseline model.

[0205] Table 8: ERLE and PESQ scores for real recorded room impulse responses

[0206]

[0207]

[0208] As shown in Table 8 above, when tested on real recorded RIRs, the contextual attention GRU echo canceller according to an embodiment of the present disclosure outperforms FDNLMS+post-filtering and encoder / decoder GRU (without contextual attention module) in both ERLE and PESQ scores.

[0209] Thus, some embodiments of the present disclosure relate to using a contextual attention neural network as a component of an echo cancellation system. Such a contextual attention neural network provides increased performance (e.g., reducing the perceived echo in a transmitted near-end signal) over comparable echo removal techniques.

[0210] Echo Removal Using Frequency Domain Normalized Mean Square (FDNLMS) Using Contextual Attention Neural Network

[0211] Some aspects of embodiments of the present disclosure relate to using frequency domain normalized mean square (FDNLMS) in conjunction with a contextual attention neural network similar to the contextual attention neural network described above.

[0212] In some comparative acoustic echo cancellation (AEC) techniques, the acoustic path is estimated using frequency domain normalized mean square (FDNLMS).The received far-end signal is then modified by the acoustic path estimated using FDNLMS to calculate an estimated echo signal that is subtracted from the microphone signal.

[0213] Embodiments of the present disclosure described in this section relate to a hybrid approach that combines a classical echo estimation technique (FDNLMS is used as an example here) with a recursive deep neural network for AEC. In more detail, some aspects of embodiments of the present disclosure use far-end signals and microphone signals and error signals between them in a feature domain to estimate near-end signals using an encoder / decoder gated recurrent unit (GRU) network. In some embodiments, the encoder / decoder GRU network further includes a contextual attention module between the encoder and decoder modules. The FDNLMS update rule is used to calculate the echo estimate. The advantages of classical acoustic echo cancellation (e.g., good handling of previously unseen environments) are combined with the superior performance of deep learning methods using a hybrid approach. Experiments using simulated and real recorded room impulse responses (RIRs) show that, compared with various baseline classical and deep learning techniques, a hybrid approach according to an embodiment of the present disclosure (e.g., using FDNLMS with an encoder / decoder GRU network) can consistently achieve better performance in terms of echo return loss enhancement (ERLE) and perceptual evaluation of speech quality (PESQ) scores.

[0214] Fig.13 is a block diagram illustrating an acoustic echo cancellation system according to one embodiment of the present disclosure, the acoustic echo cancellation system including a hybrid of a Frequency Domain Normalized Mean Square (FDNLMS) technique combined with a contextual attention neural network. Fig.13 The embodiment shown in FIG. 1 is generally similar to Figure 2 , but the difference is that the echo canceller 1328 uses the error characteristic calculated according to the FDNLMS technique As an additional input. According to some embodiments of the present invention, the echo canceller 1328 includes a recursive neural network. As described in more detail below, in some embodiments, the recursive neural network of the echo canceller 1328 is a contextual attention neural network configured to extract information from the microphone signal features. Far-end signal characteristics and error signal characteristics To calculate the estimated near-end speech features Fig.13 Like reference numerals in Figure 2 and Fig.10 Similar components to the embodiments shown in FIG. Fig.13 Similar components will not be described in detail.

[0215] like Fig.13 As shown, similar to Figure 2 and Fig.10In an embodiment of the present invention, the microphone signal d(t)=s(t)+y(t) is supplied to the acoustic echo cancellation system 1300 to cancel or remove the echo signal y(t). The microphone signal d(t) is supplied to a microphone signal feature extraction module 1310, which includes a short-time Fourier transform (STFT) module 1312, an absolute value module 1314 for removing the conjugate symmetric half of the output of the STFT module 1312, and a final logarithmic magnitude spectral feature vector or logarithmic spectral feature or microphone signal feature in a "feature space" or "hyperspace" or "log spectral space". The STFT module 1312 can also calculate the phase ∠D of the microphone signal d(t). k，f .

[0216] like Fig.13 As shown in Figure 2 and Fig.10 In some embodiments, the far-end signal x(t) may also be applied to the far-end signal feature extraction module 1320. In some embodiments, the far-end signal feature extraction module 1320 is substantially similar to the microphone signal feature extraction module 1310 and includes an STFT module 1322, an absolute value module 1324, and a logarithmic operation module 1326. The far-end signal feature extraction module 1320 calculates the far-end signal feature from the far-end signal x(t). (in feature space or hyperspace).

[0217] Fig.13 Further depiction is configured to calculate the error characteristics According to one embodiment of the present disclosure, the log spectrum error feature is calculated from the far-end signal x(t), the microphone signal d(t) and the estimated acoustic path, Fig.13 In the illustrated embodiment, the estimated acoustic path is based on a frequency domain normalized mean square (FDNLMS) estimate. More specifically, the absolute error characteristic |E k，f | can be calculated according to the following formula:

[0218] |E k，f |=|D k，f |-G k，f |X k，f |

[0219] Among them, |D k，f | is the absolute microphone feature (eg, the output of the absolute value module 1314 of the microphone signal feature extraction module 1310), |X k，f | is an absolute far-end feature (eg, the output of the absolute value module 1324 of the far-end signal feature extraction module 1320), and Gk，f are parameters representing the current estimate of the acoustic path estimated using classical techniques. Fig.13 As shown, the product node 1332 of the classical echo estimator 1330 calculates the acoustic path parameter G k，f and the absolute distal feature |X k，f The product of |G k，f |X k，f |. This product represents the estimated value of the echo signal y(t), and Fig.13 The echo characteristics are depicted as absolute estimates in When FDNLMS is used, the acoustic path parameter G is calculated and updated for each frame according to the following formula: k，f :

[0220]

[0221] The step size μ is determined by the average power P of the remote signal x(t). k，f Normalized and obtained recursively by the following formula:

[0222] P k，f =(1-α)P k-1，f +α|X k，f | 2

[0223] where α is a forgetting factor between 0 and 1.

[0224] like Fig.13 As shown, the summing node 1334 is derived from the absolute microphone characteristic |D k，f |Subtract the absolute estimated echo feature from To calculate the absolute error characteristic |E k，f |, and the logarithmic operation module 1336 is derived from the absolute error characteristic |E k，f | Calculate the logarithmic error characteristic

[0225] exist Fig.13 In the embodiment shown in FIG. 1 , the microphone signal characteristic Far-end signal characteristics Know the error characteristics is provided to the echo canceller 1328. In some embodiments of the invention, the echo canceller 1328 comprises a recursive neural network. As described above, in some embodiments, the recursive neural network is or comprises a contextual attention perception model or a contextual attention neural network 1400, such as Fig.14A and Fig. 14BIn some embodiments of the present disclosure, instead of using the contextual attention neural network 1400, the recurrent neural network is or includes an encoder / decoder GRU network without a contextual attention module (e.g., where the encoder supplies its output directly to the decoder).

[0226] Fig.14A 1 is a block diagram depicting a contextual attention neural network according to one embodiment of the present disclosure, wherein the contextual attention neural network is configured to accept far-end features, error features, and microphone features as inputs. The contextual attention neural network 1400 can be used as Fig.13 The components of the echo canceller 1328 are shown in FIG. 1 and are configured to receive far-end characteristics. Error characteristics and microphone characteristics as input. Fig. 14B is a more detailed block diagram depicting the layers of a contextual attention neural network configured to accept far-end features, error features, and microphone features as inputs according to one embodiment of the present disclosure. The contextual attention neural network 1400 is configured to accept far-end features Error characteristics and microphone characteristics as input. Fig.14A and 14B The contextual attention neural network 1400 shown in FIG. 1 is substantially similar to Fig.11A and Fig. 11B The contextual attention neural network 1100 shown in FIG. 1 is a schematic diagram of a neural network 1100 for detecting a condition of an image, and therefore the description of the basic components will not be repeated herein.

[0227] and Fig.11A and Fig. 11B One difference of the contextual attention neural network 1100 shown in FIG. 1 is that the contextual attention neural network 1400 obtains the microphone signal features discussed above. Far-end signal characteristics and error characteristics As input. Fig.14A and Fig. 14B Example of a view showing microphone signal characteristics Far-end signal characteristics And the error characteristics Each of has a dimension of T×257. Therefore, when these features are cascaded (e.g., through the cascade layer 1412 of the encoder module 1410 of the contextual attention neural network 1400), the resulting features have a dimension of T×771.

[0228] Similar to Fig.11A and Fig. 11B By way of example only, Fig.14Aand 14B The contextual attention neural network 1400 of the embodiment shown in FIG. 1 includes an encoder module 1410, a contextual attention module 1430, and a decoder module 1490, wherein each of these modules includes one or more layers of neurons. The encoder module 1410 is configured to receive microphone signal features. Far-end signal characteristics and error characteristics As described above, in some embodiments, the recurrent neural network of the echo canceller 1328 includes only an encoder / decoder GRU network having an encoder module 1410 directly connected to a decoder module 1490 (e.g., without a contextual attention module 1430 in between).

[0229] like Fig.14A As shown, the encoder module 1410 obtains the microphone signal characteristics Far-end signal characteristics and error characteristics The logarithmic magnitude of , and map them to the hyperspace to generate the encoding feature h, where Fig.11A The encoding features in the embodiment shown in have dimensions of T×1000:

[0230]

[0231] In more detail, Fig. 14B In some embodiments of the present disclosure shown, the encoder module 1410 (or encoder layer) includes a cascade layer 1412, which is configured to cascade input microphone signal features. Far-end signal characteristics and error characteristics (each with dimensions T×257) to compute concatenated features of dimension T×771. The concatenated features are then fed to a Figure 3 and Fig. 11B The gated recurrent unit (GRU) layer 1114 of the GRU layer shown in FIG. 1 , wherein each unit of the GRU layer 1414 calculates the encoding feature h according to the following formula:

[0232]

[0233] Where f represents the GRU function, and h t represents the output of GRU at time t.

[0234] In embodiments of the present disclosure where the echo canceller 1328 includes a contextual attention neural network, the contextual attention module 1430 employs the encoded features h to compute contextual attention c that identifies important regions of the hyperspace, where Fig.14A The context attention c in the embodiment shown in has a dimension of T×1000:

[0235] c=Attention(h)

[0236] In some embodiments, the contextual attention module 1430 includes a multi-head self-attention (MHSA) layer and an additional multi-head attention (MHA) layer, which is substantially similar to Fig. 11C Therefore, reference Fig. 11C The various embodiments of the described contextual attention module 1130 may also be used for the contextual attention module 1430 in embodiments of the present disclosure corresponding to hybrid classical and contextual attention neural networks, and the contextual attention module 1430 will not be described in additional detail herein.

[0237] refer to Fig.14A and Fig. 14B , where the echo canceller 1328 includes a contextual attention neural network, the decoder module 1490 is then used to compute the near-end speech from the contextual attention c The estimated logarithmic magnitude of Fig.14A and Fig. 14B The near-end speech in the embodiment shown in has a dimension of T×257 (e.g., the same dimension as the input microphone signal features and the far-end signal features). In an embodiment of the present disclosure, the echo canceller 1328 includes an encoder / decoder GRU network, and the decoder calculates the near-end speech based on the encoded features h The estimated logarithmic magnitude of .

[0238] In more detail, in some embodiments, such as Fig. 14B In the embodiment shown in FIG. 1 , the decoder module 1490 (or decoder layer) includes two GRU layers 1492 and 1494. According to one embodiment, the first GRU layer 1492 is based on the context attention c (c k-T+1 , c k-T+2 , ..., c k ) is calculated according to the following formula: k-T+1 ,s k-T+2 , ..., s k ):

[0239] s t =g 1 (c t ,s t-, )

[0240] In some embodiments, the first GRU layer 1492 is based on the encoded feature h(h k-T+1 ,h k-T+2 , ..., h k ) The output s(s) is calculated as follows: k-T+1 ,s k-T+2 , ..., s k ):

[0241] s t =g 1 (h t ,s t-′ )

[0242] The second GRU layer 1494 calculates the estimated near-end speech according to the following formula Spectral characteristics:

[0243]

[0244] In some embodiments, an exponential linear unit (elu) activation function is used with the first GRU layer 1492 and a linear activation function is used with the second GRU layer 1494.

[0245] like Fig.13 As shown, near-end speech The estimated features are converted back into an estimated near-end speech signal q(t), which can be sent to the far-end communication device.

[0246] In some embodiments, Fig.14A and Fig. 14B The encoder / decoder GRU network or contextual attention neural network 1400 shown in FIG. 1 uses the same Figure 5A , Figure 5B and Figure 5C The contextual attention neural network 1400 can be trained using substantially the same training process as described above, and the resulting trained contextual attention neural network 1400 can be deployed into a communication device (e.g., an end-user device such as a smartphone or tablet) in a manner similar to that described above.

[0247] Fig.15is a flow chart depicting a method for computing an estimated near-end signal from a received far-end signal and a microphone signal using a hybrid of classical acoustic echo cancellation techniques and a deep convolutional neural network in an inference mode according to one embodiment of the present disclosure. As described above, in some embodiments, the deep recurrent neural network is an encoder / decoder GRU network, and in some embodiments, the deep recurrent neural network is a contextual attention neural network. Fig.15 As shown, at 1502, the acoustic echo cancellation system 1300 receives a far-end signal x(t), and at 1504, the far-end signal feature extraction module 1320 extracts a far-end feature from the far-end signal x(t).

[0248] Similarly, at 1506, the acoustic echo cancellation system 1300 receives the microphone signal d(t), and at 1508, the microphone near-end signal feature extraction module 1310 extracts the microphone signal feature from the microphone signal d(t).

[0249] At 1510, the acoustic path parameters G are calculated using a suitable classical automatic echo cancellation technique, such as FDNLMS as discussed above. At 1512, the far-end features are used to calculate the acoustic path parameters G. Microphone signal characteristics and acoustic path parameter G to calculate the error characteristic

[0250] At 1514, a trained recurrent neural network (e.g., a component of the echo canceller 1328) is trained from the far-end features Microphone characteristics and error characteristics Compute estimated proximal features For example, in Fig. 14B When calculating the estimated proximal features of the current frame k When , in addition to the current frame k, features from the context of the previous T frames (e.g., six previous frames k-6, k-5, ..., k-1) are also supplied to the recurrent neural network. In particular, Fig. 14B As shown, when calculating the estimated proximal features of the current frame k When the distal features from the current frame k and T-1 previous frames k-T+1, k-T+2, ..., k-1 Know the characteristics of microphone (e.g., distal features Microphone characteristics and error characteristics ) is used to calculate the estimated proximal features

[0251] At 1516, the feature inversion module 1370 of the acoustic echo cancellation system 1300 extracts the estimated near-end features of the current frame from the near-end features of the current frame. To calculate the estimated near-end signal q(t) of the current frame. As described above, the features (e.g., far-end signal features Microphone characteristics and the estimated proximal features ) can be in a feature space or hyperspace, such as STFT space (e.g., spectral features or spectral domain). Therefore, in some embodiments, the feature inversion module 1370 converts the estimated spectral features The feature space is converted into a time domain signal q(t) suitable for playback on a speaker of a remote communication device, wherein the feature inversion module 1370 may include an exponential module 1372 and an inverse short-time Fourier transform (iSTFT) module 1374. Fig.13 As shown, when calculating the estimated near-end signal q(t), the phase ∠D of the microphone signal d(t) k,f It can also be used by the inverse short-time Fourier transform (iSTFT) 1074.

[0252] Experimental results on echo cancellation using hybrid classical and contextual attention neural network techniques

[0253] In a manner similar to that described above, in the following discussion, frequency domain normalized least mean square (FDNLMS) (e.g., see C. Faller and J. Chen, "Suppressing acoustic echo in a spectral envelope space" in IEEE Transactions on Acoustic, Speech and Signal Processing, vol. 13, no. 5, pp. 10481062, 2005) is used as a comparative example for measuring the impact of the contextual attention neural network. Another comparative example of the encoder / decoder GRU (e.g., the encoder module 1410 and the decoder module 1490 without the contextual attention module 1430) is also used to measure the impact of the contextual attention module 1430.

[0254] In particular, the training data is generated from the TIMIT dataset as described above, and then modified from synthetic and real room impulse responses (RIRs). In the experiments described in this paper, for the NLMS update rule, μ is set to 0.5 when there is no double talk and to a very small value otherwise. The forgetting factor α is set to 0.6.

[0255] A linear model of the acoustic path was used to evaluate the hybrid neural AEC according to some embodiments of the present disclosure. In this set of experiments, real measured RIRs in "Office", "Conference Room", "Stairway 1", "Stairway 2", "Bathroom", and "Report Room" were used for training of the hybrid neural AEC, while the "Hallway" state was used for testing with the phone in a handheld position (HHP). Here, the mismatch between the training RIRs and the test RIRs was small because the same recording device was used to capture both RIRs. The similarity between the training and test RIRs based on the average normalized cross correlation (NCC) was 0.97.

[0256] Table 9 shows the average ERLE values ​​and PESQ scores for the classic (FDNLMS) baseline, encoder / decoder GRU, and hybrid encoder / decoder according to some embodiments of the present disclosure:

[0257] Table 9: ERLE values ​​and PESQ scores in the linear model of the acoustic path

[0258]

[0259]

[0260] As shown in Table 9, the hybrid encoder / decoder outperforms the two baseline AEC models, but outperforms the encoder / decoder GRU by a smaller margin, probably because the conditions of the training and test sets are similar in this experiment and thus reduce the benefit of using the FDNLMS predictor as a component of the hybrid encoder / decoder.

[0261] Therefore, the performance of the embodiments of the present disclosure was further compared with the encoder / decoder GRU benchmark under different training and test conditions. To this end, seven synthetic RIRs were generated for training and tested again on data created from real measured "corridor" RIRs. The "corridor" environment was set to have a reverberation time (T60) selected from {0.2, 0.4, 0.6, 0.8, 0.9, 1.0, 1.25} seconds. Therefore, with an average NCC of approximately 0.58, the training and test RIRs in this experiment are significantly different. The comparison results are given in Table 10 below.

[0262] Table 10: ERLE values ​​and PESQ scores in the acoustic path linear model when trained on synthetic RIRs

[0263]

[0264] As shown in Table 10 above, in these experiments, the hybrid method according to some embodiments of the present disclosure outperforms the "encoder / decoder GRU" method by a large margin. Thus, compared to the clean near-end speech, the hybrid encoder / decoder method according to some embodiments of the present disclosure achieves excellent echo reduction without significant near-end distortion.

[0265] Additional experiments were performed to study the impact of nonlinear models of the acoustic path on embodiments of the present disclosure. In this set of experiments, we used a nonlinear (distorted) echo signal y in generating the microphone signal nl (t), so the model contains both power amplifier clipping and speaker distortion. Synthetic RIRs are used for training, and "corridor" RIRs are used for testing. As before, these results are compared with the classic FDNLMS alone and the non-hybrid encoder / decoder GRU. The results presented in Table 11 below show that embodiments of the present disclosure outperform both baselines:

[0266] Table 11: ERLE and PESQ scores for the nonlinear model of acoustic paths and actual measurements of the "corridor" RIRs

[0267]

[0268] Thus, some embodiments of the present disclosure relate to an architecture for AEC that uses a hybrid of classic AEC techniques (e.g., FDNLMS) to guide an encoder / decoder GRU network (e.g., contextual attention neural network) by providing adaptive information. The benefits of the hybrid techniques according to embodiments of the present disclosure are particularly observed in situations where there is a mismatch between the training conditions and test conditions (e.g., field deployment conditions) used to train the encoder / decoder GRU network, and embodiments of the present disclosure provide performance benefits over comparative baseline techniques.

[0269] Fig.161 is a block diagram of a near-end device according to one embodiment of the present disclosure. As described above, in some embodiments, the near-end device 10 is a communication device, such as a smart phone or a tablet computer. The near-end device 10 may include a modem 1610, which is configured to send and receive data, such as voice communication data. The modem may include a signal processor, which is configured to convert the received encoded data into audio data (e.g., pulse code modulation data) corresponding to the far-end signal x(t). The decoded data is supplied to the processor 1620, which may implement the acoustic echo cancellation system 200. As described above, in various embodiments of the present disclosure, the processor implementing the acoustic echo cancellation system 200 may be: a general-purpose central processing unit; a graphics processing unit (GPU); a field programmable gate array (FPGA); a neural processing unit (NPU) or a neural network processor (NNP) (e.g., a processor having an architecture customized to perform reasoning using a neural network); or a neuromorphic processor, but the embodiments of the present disclosure are not limited thereto. The received far-end signal x(t) may be played back through the speaker 12, which emits sound waves into the environment, some of which may be reflected and received by the near-end microphone 14, which also receives the near-end signal s(t). The generated microphone signal d(t) is supplied to the processor 1620 and the acoustic echo cancellation system 200 to calculate an estimated near-end signal q(t), which is sent to the far-end device via the modem 1610. Typically, the near-end device 10 may include Fig.10 Other components not shown, such as user interface components (e.g., buttons, touch screen display, etc.), batteries or other power sources, additional general or special purpose processors, temporary and non-temporary memory, etc.

[0270] Therefore, aspects of embodiments of the present disclosure relate to recurrent neural networks for acoustic echo cancellation (AEC), including deep multi-task recurrent neural networks. As shown in experimental results, embodiments of the present disclosure perform well in both one-way calls and two-way calls. Some aspects of embodiments of the present disclosure relate to end-to-end multi-task learning of echo signals and near-end signals at the same time, which improves the overall performance of the trained AEC system. In addition, some aspects of the embodiments relate to using low-latency causal context windows to improve context awareness when estimating near-end signals while removing acoustic echoes. When compared based on a reference data set, embodiments of the present disclosure can reduce echo more significantly than comparative techniques and are robust to added background noise. Further, the hybrid method according to some embodiments of the present disclosure is more robust to changes in room impulse response (RIR) and can perform well when fine-tuned by adding data simulated with the impulse response of a target device in use (e.g., near-end device 10).

[0271] Thus, aspects of embodiments of the present disclosure relate to echo cancellation or echo suppression using a trained deep recurrent neural network. Although the present disclosure has been described in conjunction with certain exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed embodiments, but on the contrary is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims and their equivalents.

Claims

1. A system for performing echo cancellation, the system comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the processor to: receiving, at the near-end device, a far-end signal from the far-end device; A microphone signal is recorded at the near-end device, the microphone signal comprising: near-end signal; and an echo signal corresponding to the far-end signal; extracting a plurality of far-end features from the far-end signal; extracting a plurality of microphone features from the microphone signal; computing a plurality of estimated near-end features from the microphone signal by supplying the microphone features and the far-end features to an acoustic echo cancellation module, the acoustic echo cancellation module comprising a contextual attention neural network, the contextual attention neural network comprising: an encoder, the encoder comprising a plurality of gated recursive units, the encoder configured to encode the microphone features and the far-end features into encoded features; a contextual attention module configured to compute contextual attention features from the encoded features; and a decoder comprising a plurality of gated recurrent units, the decoder configured to decode the estimated proximal features from the contextual attention features; calculating an estimated near-end signal from the estimated near-end features; and The estimated near-end signal is transmitted to the far-end device.

2. The system according to claim 1, wherein: The encoder comprises one or more layers of gated recurrent units, and the encoded features correspond to outputs of a last layer of the one or more layers of gated recurrent units of the encoder, wherein the encoded features are supplied to a multi-head self-attention layer of the contextual attention module; and The decoder comprises one or more layers of gated recurrent units, and the estimated proximal features correspond to outputs of a last layer of the one or more layers of gated recurrent units of the decoder.

3. The system of claim 2, wherein the contextual attention module further comprises a multi-head attention layer configured to receive the output of the multi-head self-attention layer.

4. The system according to claim 3, wherein: The contextual attention module also includes a first normalization layer between the multi-head self-attention layer and the multi-head attention layer.

5. The system according to claim 4, wherein: The contextual attention module also includes a second normalization layer, which is configured to calculate the contextual attention features based on outputs from other layers of the contextual attention module.

6. The system of claim 1 , wherein the memory further stores instructions that, when executed by the processor, cause the processor to: calculating a plurality of acoustic path parameters based on previous acoustic path parameters, previous far-end characteristics, and previous microphone characteristics; and calculating a plurality of error features based on the far-end features, the microphone features, and the acoustic path parameters, and in, The acoustic echo cancellation module is further configured to compute the plurality of estimated near-end features from the microphone signal based on the error feature.

7. The system according to claim 1, wherein: The distal features supplied to the contextual attention neural network include a current frame of the distal features and a causal window of a plurality of previous frames of the distal features, and Wherein, the microphone features supplied to the contextual attention neural network include a current frame of microphone features and a causal window of multiple previous frames of microphone features.

8. The system according to claim 7, wherein: The estimated proximal features include a current frame of estimated proximal features and a causal window of a plurality of previous frames of estimated proximal features, and The contextual attention neural network is trained by iteratively training a plurality of parameters, wherein the plurality of parameters configure the contextual attention neural network to minimize the difference between the estimated proximal features and a plurality of true proximal features of a plurality of training data.

9. The system according to claim 8, wherein: The training data is generated in the following way: loading a corpus of training data, the corpus comprising recorded utterances from a plurality of different human speakers; Select multiple pairs of human speakers; For each pair of human speakers: concatenating a plurality of utterances of a first speaker of the pair of human speakers to generate a training far-end signal; transforming the cascaded utterances to simulate an acoustic path, thereby generating a training echo signal; padding an utterance of a second speaker in the pair of human speakers to generate a training near-end signal having a length equal to that of the training far-end signal; as well as The training echo signal is mixed with the training near-end signal to generate a training microphone signal.

10. The system according to claim 9, wherein: The transforming the concatenated utterance includes convolving the far-end signal with a room impulse response simulating a room.

11. The system according to claim 10, wherein: The transforming of the cascaded utterances further comprises: applying hard clipping to the far-end signal to produce a clipped far-end signal; and A sigmoidal distortion is applied to the clipped far-end signal.

12. A system for performing echo cancellation, the system comprising: processor; as well as a memory storing instructions that, when executed by the processor, cause the processor to: receiving a remote signal from a remote device; Record microphone signals, including: near-end signal; and an echo signal corresponding to the far-end signal; extracting a plurality of far-end features from the far-end signal; extracting a plurality of microphone features from the microphone signal; calculating a plurality of acoustic path parameters based on previous acoustic path parameters, previous far-end characteristics, and previous microphone characteristics; calculating a plurality of error features based on the far-end features, the microphone features, and the acoustic path parameters; computing a plurality of estimated near-end features from the microphone signal by supplying the microphone features, the far-end features, and the error features to an acoustic echo cancellation module, the acoustic echo cancellation module comprising a recurrent neural network, the recurrent neural network comprising: an encoder comprising a plurality of gated recursive units, the encoder configured to encode the microphone features and the far-end features into encoded features; and a decoder comprising a plurality of gated recurrent units, the decoder being configured to decode the estimated proximal features based on the encoded features; calculating an estimated near-end signal from the estimated near-end features; and The estimated near-end signal is transmitted to the far-end device.

13. The system according to claim 12, wherein: The acoustic echo cancellation module further includes a context attention module configured to calculate a context attention feature from the encoded feature, and Wherein, the decoder is configured to calculate the estimated proximal feature based on the contextual attention feature.

14. The system according to claim 12, wherein: The far-end features supplied to the acoustic echo cancellation module include a current frame of far-end features and a causal window of a plurality of previous frames of far-end features, wherein the microphone features supplied to the acoustic echo cancellation module include a current frame of microphone features and a causal window of a plurality of previous frames of microphone features, and Wherein, the error feature supplied to the acoustic echo cancellation module includes a current frame of error features and a causal window of a plurality of previous frames of error features.

15. The system of claim 12, wherein: The estimated proximal features include a current frame of estimated proximal features and a causal window of a plurality of previous frames of estimated proximal features, and The recursive neural network is trained by iteratively training a plurality of parameters, wherein the plurality of parameters configure the recursive neural network to minimize the difference between the estimated proximal features and a plurality of true proximal features of a plurality of training data.

16. The system of claim 15, wherein: The training data is generated in the following way: loading a corpus of training data, the corpus comprising recorded utterances from a plurality of different human speakers; Select multiple pairs of human speakers; For each pair of human speakers: concatenating a plurality of utterances of a first speaker of the pair of human speakers to generate a training far-end signal; transforming the cascaded utterances to simulate an acoustic path, thereby generating a training echo signal; padding an utterance of a second speaker in the pair of human speakers to generate a training near-end signal having a length equal to that of the training far-end signal; as well as The training echo signal is mixed with the training near-end signal to generate a training microphone signal.

17. The system of claim 16, wherein: The transforming the concatenated utterance includes convolving the far-end signal with a room impulse response simulating a room.

18. The system of claim 17, wherein: The transforming of the cascaded utterances comprises: applying hard clipping to the far-end signal to produce a clipped far-end signal; and A sigmoidal distortion is applied to the clipped far-end signal.

19. The system of claim 12, wherein: The far-end features, the microphone features, and the estimated near-end features include logarithmic short-time Fourier transform features in a logarithmic spectral space.

Images