Method, apparatus and program for unified acoustic echo suppression using recurrent neural networks

Abstract

A method of acoustic echo suppression using a recurrent neural network executed by at least one processor is provided, the method including the steps of receiving a microphone signal and a far-end reference signal, estimating an echo suppressed signal and an echo signal based on the microphone signal and the far-end reference signal, estimating an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal, generating an enhancement signal based on the enhancement filter, and adjusting the enhancement signal using an automatic gain control (AGC) and outputting an adjusted signal.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 17 / 840,188, filed with the U.S. Patent and Trademark Office on June 14, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Field Apparatus and methods consistent with exemplary embodiments of the present disclosure generally relate to processing data using neural networks and deep learning for speech enhancement. Specifically, embodiments relate to training a NeuralEcho model and automatic gain control (AGC) tasks in a unified model to perform acoustic echo cancellation (AEC) and improve echo and noise suppression. [Background technology]

[0003] Acoustic echo cancellation (AEC) plays an important role in full-duplex voice communication as well as in front-end speech enhancement for recognition in loudspeaker playback situations.

[0004] In the related art, adaptive filtering methods for estimating the acoustic echo path and linear echo cancellation have been studied. Normalized least mean square filters (e.g., frequency domain adaptive filter and multi-delay block frequency domain adaptive filter) are widely used due to their robustness and low complexity. Nonlinear post-processing is typically cascaded for residual echo suppression. However, these methods are not efficient for nonlinear echo distortion, echo path changes, and non-stationary noise. Therefore, deep neural networks (e.g., complex-valued DNN, long short-term memory network, multi-head self-attention) have been adopted to develop echo suppression systems that better handle nonlinear echo distortion and echo path delay. In the related art, residual echo suppression based on neural networks followed by linear adaptive filtering has been adopted to form a hybrid system for AEC system design and has achieved promising results.

[0005] Related work also describes a NN3A model that supports two-stage multi-channel joint AEC and beamforming, and audio front-end AEC, noise suppression, and AGC. AEC is also formulated as an end-to-end supervised speech enhancement problem, where a neural network is employed to predict a mask to extract the near-end target speaker. In some related work, a dual signal transformation LSTM network (DTLN) is employed for the AEC task, and a Wave-U-Net-based acoustic echo cancellation with attention mechanism is proposed to jointly suppress acoustic echo and background noise. The use of a complex neural network together with frequency-time-LSTM (FT-LSTM) provides significant phase information modeling and temporal modeling, outperforming the best methods in AEC. Summary of the Invention [Problem to be solved by the invention]

[0006] According to an embodiment, a method is provided for acoustic echo suppression via a NeuralEcho model trained with a unified model and automatic gain control (AGC) to perform acoustic echo cancellation (AEC) and improve echo and noise suppression. [Means for solving the problem]

[0007] According to an aspect of the present disclosure, a method for acoustic echo suppression using a recurrent neural network is executed by at least one processor, the method including receiving a microphone signal and a far-end reference signal, estimating an echo suppressed signal and an echo signal based on the microphone signal and the far-end reference signal, estimating an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal, generating an enhancement signal based on the enhancement filter, adjusting the enhancement signal using an automatic gain control (AGC), and outputting the adjusted signal.

[0008] The method may further include a first stage and a second stage, where the echo suppressed signal and the echo signal are estimated by the first stage and the enhancement filter is estimated by the second stage.

[0009] The method may further include, in a first stage, calculating a first covariance matrix based on the microphone signal and the far-end reference signal, processing the first covariance matrix through a first linear projection filter and encoding a result of the first linear projection filter to estimate a first stage filter, applying the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal, and outputting an echo suppressed signal and an echo signal.

[0010] The method may further include, in a second stage, generating second stage input features based on the first linear projection filter and the normalized log power spectrum of the echo suppressed signal and the echo signal, processing the second stage input features through a second linear projection filter, encoding a result of the second linear projection filter to estimate a second stage filter, applying the second stage filter to estimate target speech and microphone noise channels, calculating a second covariance matrix based on the estimated target speech and microphone noise channels, and processing the second covariance matrix through a recurrent neural network to output an enhancement filter.

[0011] The method may further include the microphone signal including the target speech, the distorted far-end reference signal, and microphone noise.

[0012] The method may further include computing a loss function based on a scale-invariant signal-to-distortion ratio in the time domain and an L1 norm of the spectral magnitude difference between the enhanced signal and the target signal.

[0013] The method may further include adjusting the enhancement signal including encoding a magnitude of the enhancement signal to predict a magnitude of the AGC, generating an AGC signal in the time domain based on a phase of the enhancement signal, and adjusting the enhancement signal based on the AGC signal.

[0014] According to another aspect of the disclosure, an apparatus for acoustic echo suppression using a recurrent neural network includes at least one memory configured to store program code and at least one processor configured to read and operate according to instructions of the program code, the program code including receiving code configured to cause the at least one processor to receive a microphone signal and a far-end reference signal, signal estimation code configured to cause the at least one processor to estimate an echo suppressed signal and an echo signal based on the microphone signal and the far-end reference signal, filter estimation code configured to cause the at least one processor to estimate an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal, generating code configured to cause the at least one processor to generate an enhancement signal based on the enhancement filter, and adjusting code configured to cause the at least one processor to adjust the enhancement signal using an automatic gain control (AGC) and output the adjusted signal.

[0015] The apparatus may further include a first stage and a second stage, where the echo suppressed signal and the echo signal are estimated by the first stage and the enhancement filter is estimated by the second stage.

[0016] The apparatus may further include, in a first stage, computation code configured to cause the at least one processor to compute a first covariance matrix based on the microphone signal and the far-end reference signal; processing code configured to cause the at least one processor to process the first covariance matrix through a first linear projection filter and encode a result of the first linear projection filter to estimate a first stage filter; application code configured to cause the at least one processor to apply the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal; and output code configured to cause the at least one processor to output an echo suppressed signal and an echo signal.

[0017] The apparatus may further include, in a second stage, a second generation code configured to cause the at least one processor to generate second stage input features based on a first linear projection filter and a normalized log power spectrum of the echo suppressed signal and the echo signal; a second processing code configured to cause the at least one processor to process the second stage input features through a second linear projection filter, encode a result of the second linear projection filter to estimate a second stage filter, and apply the second stage filter to estimate a target speech and microphone noise channel; a second calculation code configured to cause the at least one processor to calculate a second covariance matrix based on the estimated target speech and microphone noise channels; and a third processing code configured to cause the at least one processor to process the second covariance matrix through a recurrent neural network and output an enhancement filter.

[0018] The apparatus may further include, wherein the microphone signal includes the target speech, the distorted far-end reference signal, and microphone noise.

[0019] The apparatus may further include computational code configured to cause the at least one processor to compute a loss function based on a scale-invariant signal-to-distortion ratio in the time domain and an L1 norm of a spectral magnitude difference between the enhanced signal and the target signal.

[0020] The apparatus may further include an encoding code configured to cause the at least one processor to encode a magnitude of the enhancement signal to predict an AGC magnitude in an adjustment code, an AGC generation code configured to cause the at least one processor to generate an AGC signal in the time domain based on a phase of the enhancement signal, and a second adjustment code configured to cause the at least one processor to adjust the enhancement signal based on the AGC signal.

[0021] According to another aspect of the disclosure, a non-transitory computer-readable medium has stored thereon instructions for execution by at least one processor of an apparatus for acoustic echo suppression using a recurrent neural network, the instructions causing the at least one processor to perform the steps of generating a microphone signal and a far-end reference signal, estimating an echo suppressed signal and an echo signal based on the microphone signal and the far-end reference signal, estimating an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal, generating an enhancement signal based on the enhancement filter, and adjusting the enhancement signal using an automatic gain control (AGC) and outputting the adjusted signal.

[0022] The non-transitory computer readable medium may further include a first stage and a second stage, where the echo suppressed signal and the echo signal are estimated by the first stage and the enhancement filter is estimated by the second stage.

[0023] The non-transitory computer-readable medium may further include instructions causing the at least one processor to perform, in a first stage, calculating a first covariance matrix based on a microphone signal and a far-end reference signal; processing the first covariance matrix through a first linear projection filter to encode a result of the first linear projection filter to estimate a first stage filter; applying the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal; and outputting an echo suppressed signal and an echo signal.

[0024] The non-transitory computer-readable medium may further include causing the at least one processor to perform, in a second stage, generating second stage input features based on a first linear projection filter and a normalized log power spectrum of the echo suppressed signal and the echo signal; processing the second stage input features through a second linear projection filter; encoding a result of the second linear projection filter to estimate a second stage filter; applying the second stage filter to estimate target speech and microphone noise channels; calculating a second covariance matrix based on the estimated target speech and microphone noise channels; and processing the second covariance matrix through a recurrent neural network to output an enhancement filter.

[0025] The non-transitory computer-readable medium may further include, wherein the microphone signal includes the target speech, the distorted far-end reference signal, and microphone noise.

[0026] The non-transitory computer-readable medium may further include the instructions causing the at least one processor to calculate a loss function based on a scale-invariant signal-to-distortion ratio in the time domain and an L1 norm of a spectral magnitude difference between an enhanced signal and a target signal.

[0027] Additional embodiments will be set forth in the description that follows, and in part will be obvious from the description, and / or may be learned by practice of the presented embodiments of the present disclosure. [Brief description of the drawings]

[0028] The above and other aspects, features, and aspects of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0029] [Figure 1] FIG. 2 is a diagram of a device of a system according to one or more embodiments.

[0030] [Diagram 2] FIG. 2 is a diagram of components of one or more of the devices of FIG. 1.

[0031] [Diagram 3] FIG. 1 is a block diagram of a system according to an embodiment.

[0032] [Figure 4] 1 illustrates a NeuralEcho model with two-stage single-channel acoustic echo cancellation (AEC) and noise reduction, according to one embodiment.

[0033] [Diagram 5] 1 illustrates a NeuralEcho model with automatic gain control (AGC), according to one embodiment.

[0034] [Figure 6] 1 is a flowchart of a method for end-to-end speaker diarization (EESD), according to one embodiment.

[0035] [Figure 7] FIG. 1 is a block diagram of end-to-end speaker diarization (EESD) according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The following detailed description of the exemplary embodiments refers to the accompanying drawings, in which the same reference numbers in different drawings may identify the same or similar elements.

[0037] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of an embodiment may be incorporated in or combined with another embodiment (or one or more features of the other embodiments). Furthermore, in the flowcharts and operational descriptions provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed simultaneously (at least in part), and the order of one or more operations may be switched.

[0038] It will be apparent that the systems and / or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not a limitation of the implementation. Thus, the operation and behavior of the systems and / or methods have been described herein without reference to any specific software code. It will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

[0039] Although specific combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claims.

[0040] No element, act, or instruction used herein should be construed as critical or required unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms "have," "has," "having," "including," "including," and the like are intended to be open-ended terms. Furthermore, the phrase "based on..." is intended to mean "based at least in part on..." unless specifically stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood as including only A, only B, or both A and B.

[0041] Exemplary embodiments of the present disclosure provide, among other things, methods, apparatus, and computer programs for performing acoustic echo suppression. With the increasing use of hands-free devices and teleconferencing, speech-based human-machine dialogue and voice communication is becoming more prevalent, and thus improved acoustic echo cancellation (AEC) models may be advantageous.

[0042] Acoustic echo is caused by the sound from the loudspeaker that is received by the near-end microphone and then transmitted to the far-end listener or speech recognition engine. The quality of the speech signal is degraded by acoustic echo, environmental noise, room reverberation, hardware receivers, digital signal processing, and network transmission. Such interfering signals degrade the voice quality and limit the performance of AEC algorithms in teleconferencing systems, mobile communications, and hands-free human-machine dialogue.

[0043] Exemplary embodiments described below provide a system, method, and computer program that uses an all-deep-learning based AEC system with a recurrent neural network for acoustic echo suppression. An end-to-end two-stage single-channel AEC and noise reduction model (i.e., NeuralEcho model) is used to perform acoustic echo suppression. In the first stage, the model implicitly estimates the echo signal and the AEC output. The echo / noise covariance matrix and the target speech covariance matrix are estimated through the estimated echo channel, the AEC output channel, and the input microphone channel. In the second stage, the speech enhancement filter is predicted given the second-order statistics of the echo / noise and the target speech. The model exploits cross-channel second-order statistics for better echo / noise reduction and target speaker preservation. Along with the RNN model structure, multi-head self-attention is used over time to dynamically highlight relevant features.

[0044] In addition, feature-wise linear modulation (FiLM) is used to combine the target speaker embedding vector with other acoustic features, which allows the network to better separate the target speech signal from echo and other interferences. An AGC branch is also integrated into the system to amplify the speech signal to a clear sound level, which is in high demand in real-time communication (RTC). In particular, a traditional AGC algorithm is placed in the post-processing stage. By training the NeuralEcho model and the AGC branch together as part of a unified model, the output of the AGC branch not only adjusts the processed signal to a proper sound level, but also improves the speech recognition accuracy.

[0045] FIG. 1 is a diagram of an environment 100 in which methods, apparatus, and systems for acoustic echo suppression described herein may be implemented, according to embodiments.

[0046] 1, environment 100 may include user devices 110, a platform 120, and a network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0047] User device 110 includes one or more devices that can receive, generate, store, process, and / or provide information related to platform 120. For example, user device 110 can include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., smart glasses, a smart watch, etc.), or a similar device. In some implementations, user device 110 can receive information from platform 120 and / or transmit information to platform 120.

[0048] Platform 120 includes one or more devices as described elsewhere herein. In some implementations, platform 120 may include a cloud server or a group of cloud servers. In some implementations, platform 120 may be designed in a modular manner, such that software components may be swapped in or out. In this manner, platform 120 may be easily and / or quickly reconfigured for different uses.

[0049] In some implementations, as shown, platform 120 may be hosted in a cloud computing environment 122. In particular, although the implementations described herein describe platform 120 as being hosted in a cloud computing environment 122, in some implementations platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0050] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide services, such as computing, software, data access, storage, etc., that do not require end-user (e.g., user equipment 110) knowledge of the physical location and configuration of the systems and / or devices that host platform 120. As illustrated, cloud computing environment 122 can include a group of computing resources 124 (collectively referred to as “computing resources 124” and individually referred to as “computing resource 124”).

[0051] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communications devices. In some implementations, the computing resources 124 may host the platform 120. Cloud resources may include computing instances running on the computing resources 124, storage devices provided on the computing resources 124, data transfer devices provided by the computing resources 124, etc. In some implementations, the computing resources 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0052] As further shown in FIG. 1, computing resources 124 include a group of cloud resources, such as one or more applications (“APP”) 124-1, one or more virtual machines (“VM”) 124-2, virtualized storage (“VS”) 124-3, and one or more hypervisors (“HYP”) 124-4.

[0053] Application 124-1 includes one or more software applications that may be provided to or accessed by user device 110 and / or platform 120. Application 124-1 may eliminate the need to install and run a software application on user device 110. For example, application 124-1 may include software associated with platform 120 and / or any other software that may be provided via cloud computing environment 122. In some implementations, one application 124-1 may send information to or receive information from one or more other applications 124-1 via virtual machine 124-2.

[0054] The virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. The virtual machine 124-2 can be either a system virtual machine or a process virtual machine, depending on the extent to which the virtual machine 124-2 uses or corresponds to any real machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). A process virtual machine can execute a single program and support a single process. In some implementations, the virtual machine 124-2 can run on behalf of a user (e.g., user device 110) and manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0055] Virtualized storage 124 includes one or more storage systems and / or one or more devices of computing resources 124 that use virtualization techniques within the storage systems or devices. In some implementations, within the context of a storage system, types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage such that the storage system may be accessed without regard to the physical storage or heterogeneous structure. This separation allows administrators of the storage system flexibility in how they manage storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and where the file is physically stored. This may enable storage usage optimization, server consolidation, and / or performance of non-disruptive file migration.

[0056] The hypervisor 124-4 may provide hardware virtualization techniques that allow multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer, such as the computing resource 124. The hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of diverse operating systems may share virtualized hardware resources.

[0057] Network 130 may include one or more wired and / or wireless networks. For example, network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic based network, etc., and / or a combination of these or other types of networks.

[0058] The number and arrangement of devices and networks shown in Figure 1 are provided as an example. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different arrangements of devices and / or networks than shown in Figure 1. Furthermore, two or more devices shown in Figure 1 may be implemented within a single device, or a single device shown in Figure 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0059] FIG. 2 is a block diagram of example components of one or more devices of FIG.

[0060] The device 200 may correspond to the user device 110 and / or the platform 120. As shown in FIG. 2, the device 200 may include a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface 270.

[0061] The bus 210 includes components that enable communication between the components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0062] The storage component 240 stores information and / or software related to the operation and use of the device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium along with a corresponding drive.

[0063] Input components 250 include components that enable device 200 to receive information, for example, via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 can include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0064] Communications interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 200 to communicate with other devices, e.g., via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communications interface 270 enables device 200 to receive information from and / or provide information to another device. For example, communications interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0065] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer-readable medium, such as memory 230 and / or storage component 240. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0066] Software instructions may be loaded into memory 230 and / or storage component 240 from another computer-readable medium or from another device via communications interface 270. When executed, the software instructions stored in memory 230 and / or storage component 240 can cause processor 220 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of, or in combination with, software instructions to perform one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0067] The number and arrangement of components shown in Figure 2 is provided as an example. In practice, device 200 may include additional, fewer, different, or differently arranged components than those shown in Figure 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0068] In embodiments, any one of the operations or processes of FIGS. 3 through 7 may be performed by or using any one of the elements shown in FIGS.

[0069] 3 is a block diagram of an exemplary audio system 300 according to one embodiment. The system includes a far-end speaker 310 (hereafter referred to as the "speaker") and a near-end microphone 320 (hereafter referred to as the "microphone").

[0070] For example, microphone 320 receives near-end target speech signal 330 from speaker 310. Speaker 310 can be any audio source (e.g., loudspeaker, recorded audio, spoken audio, etc.) apparent to one skilled in the art without departing from the scope of the described embodiments. Target speech signal 330 includes, for example, reverberation, far-end echo signal from speaker 310, and background noise. Let s and x represent the dry clean signal from near-end microphone 320 and the dry clean signal from far-end speaker 310, respectively. The signal captured by microphone 320 at time t is y(t)=s r (t)+x r (t)+v(t) (Equation 1) It can be expressed as:

[0071] Here, s r (t)=h s (t)*s(t), where s r (t) is the reverberant target speech signal, and x r (t)=h x (t)*g(x(t)), where x r (t) is the distorted far-end signal emitted by the speaker 310, v(t) represents the near-end noise signal, and * represents the convolution. The function g represents the nonlinear distortion, and the echo path delay and near-end reverberation are expressed as h x Included in (t).

[0072] FIG. 4 illustrates a NeuralEcho model 400 with two-stage single-channel AEC and noise reduction for acoustic echo suppression. As shown in FIG. 4, given a microphone signal y(t) and a far-end reference signal x(t), the reverberant target speech signal s(t) is generated while suppressing the far-end echo signal and background noise. r The NeuralEcho model 400 predicts the reverberant target speech signal s r 4, a line representing the far-end reference signal x(t) is designated by reference numeral 401, a line representing the microphone signal y(t) is designated by reference numeral 402, and a fusion line illustrating where one or more output signals are combined and input to another layer is designated by reference numeral 405.

[0073] First, the audio samples of the microphone signal y(t) and the far-end reference signal x(t) are transformed into the frequency domain as Y(k,n) and X(k,n), respectively, by a short-time Fourier transform (STFT), where k represents the frequency bin and n represents the frame index. To decode the audio samples, a 512-point fast Fourier transform (FFT) (e.g., using a Hanning window with a length of 32 ms and a hop size of 16 ms) is used. The stacking of the frequency representations Y(k,n) and X(k,n) results in Z(k,n), where Z(k,n)=[Y(k,n),X(k,n)]. T The covariance matrix of Z is Φ C (k,n)∈C 2×2 The covariance matrix Φ is used as the input feature, which is represented by the Channel Cov layer shown in Figure 4. It describes the cross-correlation between the far-end reference signal x(t) and the microphone signal y(t), and the power spectral density (PSD) of the microphone signal y(t) and the far-end reference signal x(t). C (k,n) is Φ C (k,n)=(Z(k,n)-μ z )(Z(k,n)-μ z ) H (Formula 2) It is calculated as:

[0074] Where:

number

[0075] The upper half of the complex symmetric matrix is ​​discarded to reduce computational cost and memory usage. The lower triangular matrix is ​​flattened and the real and imaginary parts are concatenated. Furthermore, correlation matrices across time frames and frequency bins are calculated separately for the microphone signal y(t) and the far-end reference signal x(t). Specifically, the temporal correlation Φ T and frequency correlation Φ F is calculated as follows in the Temporal Corr layer and the Frequency Corr layer shown in FIG.

number

[0076] where i represents the microphone and the far-end reference channel, and n τ and k τ corresponds to the maximum shift along the time and frequency axes, respectively. In this example, we use the reverberant target speech signal s r The maximum shift in the model (to predict t) is set to 9. Similarly, the lower triangular matrix is ​​flattened and the real and imaginary parts of both the microphone and the far-end reference channel are concatenated. The three features (i.e., the covariance matrix Φ C , the temporal correlation Φ T , and frequency correlation Φ F ) are concatenated together with the normalized log-power spectra (LPS) of the microphone signal y(t) and the far-end reference signal x(t) shown by the LayerNorm&Stack layer in Figure 4 to generate the first stage input features.

[0077] The feature vector for each time-frequency bin of the first stage input features is projected to a one-dimensional space (e.g., from 368 dimensions to 1 dimension) through a linear FiLM projection layer. The feature vector for each time-frequency bin may be projected to speaker information (e.g., information about the speaker 310). The speaker information is optional (hence shown by the dotted line) and is not required for the NeuralEcho model 400. After a gated recurrent unit (GRU) encoder layer (e.g., a GRU encoder layer with 257 hidden units), the NeuralEcho model 400 includes a x-dimensional complex-valued ratio filter cRF for the first stage AEC output and echo estimation, respectively. AEC (k,n) and cRF echo Estimate the (k,n)-dimensional complex-valued ratio filter cRF AEC (k,n) and cRF echo (k,n) may be estimated based on (2K+1)×(2N+1), denoted as Cov1D AEC cRF and Cov1D Echo cRF in FIG. 4. Equation 4 expresses the estimated cRF AEC (k,n) to a time-frequency shifted version of the input microphone signal Y(k,n) to produce an echo-suppressed signal Y(k,n), denoted as AEC out in FIG. AEC Similar operations are applied to the far-end reference signal X(k,n) to produce the echo estimate X(k,n), denoted Echo est in FIG. echo 4, the line indicating the Echo est signal is denoted by reference numeral 403, and the line indicating the AEC out signal is denoted by reference numeral 404.

number

[0078] The second stage uses the raw microphone input channel, the AEC processed channel from the first stage, and the echo estimation channel.

number

number

[0079] Here, MHSA stands for multi-head self attention with, for example, four parallel attention heads. The time-frequency index (k,n) is omitted in the above formula. Attention is performed to select important features across time frames, and is performed as an attention query Φ proj is a dot product with itself and performed as the attention key and value to generate the attention weights. The values ​​are weighted-averaged together with the residual connections using the attention weights and fed to the GRU encoder. A second self-attention module and a subsequent linear projection layer are performed to estimate single-tap speech enhancement filters W∈CF×T×3 for the microphone channel, the first stage AEC channel, and the echo estimation channel, respectively. The results are then summed to give a complex-valued matrix. The complex-valued matrix is ​​then transformed to the time domain by an inverse short-time Fourier transform (iSTFT) to generate the enhanced signal ̂Sr.

[0080] The loss function for the two-stage single-channel AEC and NeuralEcho model 400 is defined as follows:

number

[0081] Equation 6 is calculated using a combination of the scale-invariant signal-to-distortion ratio (SI-SDR) in the time domain and the L1-norm of the spectral magnitude difference between the enhanced signal ^Sr and the target signal Sr. The L1-loss on the spectral magnitude serves to regularize the scale of the output signal.

[0082] In both stages, the speaker embeddings are combined with other acoustic features using feature-wise linear modulation (FiLM) as shown in Figure 4. Speech separation using speaker embeddings improves the user's speech quality not only in multi-speaker situations but also in ambient noise. Residual connections are added after the FiLM layer to fuse the features and ensure that the NeuralEcho model 400 can perform well in the absence of speaker embeddings. Mathematically, this configuration of feature fusion transforms the input acoustic features θ and speaker embeddings ds to generate output features ξ as follows:

number

[0083] where r(·) and h(·) are linear projections from the 128-dimensional speaker embeddings to 257-dimensional vectors that match the size of the input acoustic features θ. The speaker embeddings d provide global biases for the target speakers. s is given by averaging the speaker vectors over the five enrolled utterances in the NeuralEcho Model 400 architecture setup.

[0084] 5 is an illustration of an AGC branch implemented using the NeuralEcho model 400, according to one embodiment. The AGC branch includes:

[0085] As shown in FIG. 5, an AGC branch is added at the output stage of the NeuralEcho model 400 to adjust and amplify the enhanced signal ^Sr or (the speech segment of the enhanced signal) to a clear sound level. The magnitude of the enhanced signal passes through a GRU layer (257 hidden units) to predict the post-AGC magnitude, which is then combined with the phase of the output signal of the NeuralEcho model to synthesize the post-AGC signal estimate in the time domain. The AGC tool is used to process the original dry clean data and generate the post-AGC dry clean target signal for training purposes. Thus, the output of the AGC branch is expected to adjust the sound level as well as suppress the reverberation. Training Loss SI-SDR and MAG L1 520 calculates the loss function defined in Equation 6 using a combination of the SI-SDR in the time domain and the L1-norm of the spectral magnitude difference between the enhanced signal ^Sr and the target signal Sr. SI-SDR leads to very slow convergence and poor performance in dereverberation tasks. Thus, the training loss Mag L1 510 uses the L1-loss on the spectral magnitude error between the estimated signal and the dry clean target signal after AGC to condition and amplify the speech signal while improving speech recognition accuracy.

[0086] 6 is an example flowchart illustrating a method 600 performed by at least one processor for acoustic echo suppression using a recurrent neural network, according to an embodiment. In some implementations, one or more process blocks of FIG. 6 may be performed by platform 120. In some implementations, one or more process blocks of FIG. 6 may be performed by another device or group of devices, such as user device 110, separate from or including platform 120.

[0087] At operation 610, the method 600 includes receiving a microphone signal and a far-end reference signal. The microphone signal may include a target speech, a distorted far-end reference signal, and microphone noise.

[0088] At operation 620, the method 600 includes estimating the echo suppressed signal and the echo signal based on the microphone signal and the far-end reference signal.

[0089] At operation 630, the method 600 includes estimating an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal.

[0090] At operation 640, the method 600 includes generating an enhanced signal (enhanced signal) based on the enhancement filter.

[0091] The method 600 includes a first stage and a second stage, where operation 620 is performed by the first stage and operation 630 is performed by the second stage. The first stage may include calculating a first covariance matrix based on a microphone signal and a far-end reference signal, processing the first covariance matrix through a first linear projection filter, encoding a result of the first linear projection filter to estimate a first stage filter, applying the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal, and outputting an echo suppressed signal and an echo signal. The second stage may include generating second stage input features based on a first linear projection filter and a normalized log power spectrum of the echo suppressed signal and the echo signal, processing the second stage input features through a second linear projection filter, encoding a result of the second linear projection filter to estimate a second stage filter, applying the second stage filter to estimate channels of target speech and microphone noise, calculating a second covariance matrix based on the estimated channels of target speech and microphone noise, and processing the second covariance matrix through a recurrent neural network to output an enhancement filter.

[0092] At operation 650, the method 600 includes adjusting the enhancement signal using an automatic gain control (AGC) and outputting the adjusted signal. Operation 650 may further include encoding a magnitude of the enhancement signal to predict a magnitude of the AGC, generating an AGC signal in the time domain based on a phase of the enhancement signal, and adjusting the enhancement signal based on the AGC signal.

[0093] The functional loss may be calculated based on the scale-invariant signal-to-distortion ratio in the time domain and the L1 norm of the spectral magnitude difference between the enhanced signal and the target signal.

[0094] Although Figure 6 illustrates example operations of a method, in some implementations a method may include additional, fewer, different, or differently arranged operations than those illustrated in Figure 6. Additionally or alternatively, two or more of the operations of the method may be performed in parallel or combined.

[0095] 7 is a block diagram of an example of computer code for acoustic echo suppression using a recurrent neural network, according to one or more embodiments. According to embodiments of the present disclosure, an apparatus / device may be provided that includes at least one processor with a memory that stores computer code. The computer code, when executed by the at least one processor, may be configured to perform any number of aspects of the present disclosure.

[0096] As shown in FIG. 7, computer code 700 includes receiving code 710, signal estimation code 720, filter estimation code 730, generating code 740, and adjustment code 750.

[0097] A receiving code 710 is configured to cause the at least one processor to receive a microphone signal and a far-end reference signal.

[0098] The signal estimation code 720 is configured to cause the at least one processor to estimate an echo suppressed signal and an echo signal based on a microphone signal and a far-end reference signal.

[0099] The filter estimation code 730 is configured to cause the at least one processor to estimate an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal.

[0100] Generation code 740 is configured to cause the at least one processor to generate an enhancement signal based on the enhancement filter.

[0101] The adjustment code 750 is configured to cause the at least one processor to adjust the enhancement signal using an automatic gain control (AGC) and output an adjusted signal. The adjustment code may further include an encoding code configured to cause the at least one processor to encode a magnitude of the enhancement signal to predict an AGC magnitude, an AGC generation code configured to cause the at least one processor to generate an AGC signal in the time domain based on a phase of the enhancement signal, and a second adjustment code configured to cause the at least one processor to adjust the enhancement signal based on the AGC signal.

[0102] The computer code 700 may further include, in a first stage of computer processing, computation code configured to cause the at least one processor to compute a first covariance matrix based on the microphone signal and the far-end reference signal; processing code configured to cause the at least one processor to process the first covariance matrix through a first linear projection filter and encode a result of the first linear projection filter to estimate a first stage filter; application code configured to cause the at least one processor to apply the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal; and output code configured to cause the at least one processor to output an echo suppressed signal and an echo signal.

[0103] The computer code 700 may further include, in a second stage of computer processing, a second generation code configured to cause the at least one processor to generate second stage input features based on the first linear projection filter and the normalized log power spectrum of the echo suppressed signal and the echo signal; a second processing code configured to cause the at least one processor to process the second stage input features through a second linear projection filter, encode a result of the second linear projection filter to estimate a second stage filter, and apply the second stage filter to estimate a target speech and microphone noise channel; a second calculation code configured to cause the at least one processor to calculate a second covariance matrix based on the estimated target speech and microphone noise channels; and a third processing code configured to cause the at least one processor to process the second covariance matrix through a recurrent neural network and output an enhancement filter.

[0104] The computer code 700 may further include computational code configured to cause the at least one processor to compute a loss function based on a scale-invariant signal-to-distortion ratio in the time domain and an L1 norm of the spectral magnitude difference between the enhanced signal and the target signal.

[0105] Although Figure 7 illustrates example blocks of computer code 700 for a system or apparatus according to embodiments, in some embodiments, a system may include additional, fewer, different, or differently arranged blocks than those illustrated in Figure 7. Additionally or alternatively, two or more of the blocks of the system may be combined. In other words, although Figure 7 illustrates separate blocks of code, the various code instructions need not be separate and may be interwoven.

[0106] The methods and systems described according to the embodiments of the present disclosure provide significant AEC feature learning by leveraging second-order statistics across signal channels, time and frequency axes compared to existing deep learning-based approaches. A self-attention RNN neural network is also implemented to perform AEC on the microphone channel, intermediate AEC processed channel, and echo estimation channel to dynamically highlight relevant features. The proposed model is further developed to work with speaker information and perform AGC for better target speaker enhancement. Experimental results show that the NeuralEcho model according to the embodiments and its derived models provide significant improvement over traditional signal processing-based methods and state-of-the-art neural network methods in terms of both audio quality and speech recognition accuracy.

[0107] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations may be possible in light of the above disclosure or may be acquired from practice of the implementations.

[0108] Some embodiments relate to systems, methods, and / or computer-readable media at any possible level of technical detail integration. Furthermore, one or more of the above-mentioned components may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0109] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-limiting list of more specific examples of computer-readable storage media may include portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards or mechanically encoded devices such as ridge-in-groove structures, and any suitable combination of the above. A computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0110] The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in the respective computing / processing device.

[0111] The computer readable program code / instructions for performing the operations may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages. Such programming languages ​​include object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may utilize state information of the computer readable program instructions to execute computer readable program instructions to personalize the electronic circuitry in order to perform aspects or operations.

[0112] These computer-readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to cause a machine such that the instructions, executed via the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in the flowchart and / or block diagram block(s). These computer-readable program instructions can be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that a computer-readable storage medium having instructions stored therein includes an article of manufacture that includes instructions that perform aspects of the functions / operations specified in the flowchart and / or block diagram block(s).

[0113] The computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps resulting in a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the function / operation specified in the flowchart and / or block diagram block(s).

[0114] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or part of instructions, including one or more executable instructions for implementing the specified logical functions. The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than those shown in the figures. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may in fact be executed simultaneously or substantially simultaneously, depending on the functionality involved, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that executes the specified functions or operations, or executes a combination of special-purpose hardware and computer instructions.

[0115] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware may be designed to implement the systems and / or methods based on the description herein.

Claims

1. 1. A method of acoustic echo suppression using a recurrent neural network, executed by at least one processor, comprising: receiving a microphone signal and a far-end reference signal; estimating an echo suppressed signal and an echo signal based on the microphone signal and the far-end reference signal; estimating an enhancement filter for the microphone signal based on the echo suppressed signal and the echo signal; generating an enhancement signal based on the enhancement filter; adjusting the enhanced signal using an automatic gain control (AGC) and outputting an adjusted signal. method.

2. The method of claim 1 , further comprising a first stage and a second stage, wherein the echo suppressed signal and the echo signal are estimated by the first stage and the enhancement filter is estimated by the second stage.

3. The first stage: calculating a first covariance matrix based on the microphone signal and the far-end reference signal; processing the first covariance matrix through a first linear projection filter and encoding a result of the first linear projection filter to estimate a first stage filter; applying the first stage filter to the time-frequency shifted microphone signal and the far-end reference signal; and outputting the echo suppressed signal and the echo signal. The method of claim 2.

4. The second stage: generating second stage input features based on the first linear projection filter and the normalized log power spectrum of the echo suppressed signal and the echo signal; processing the second stage input features through a second linear projection filter, encoding a result of the second linear projection filter to estimate a second stage filter, and applying the second stage filter to estimate target speech and microphone noise channels; calculating a second covariance matrix based on the estimated channels of the target speech and microphone noise; and processing the second covariance matrix through the recurrent neural network to output the enhancement filter. The method according to claim 3.

5. The method of claim 1 , wherein the microphone signals include a target speech, a distorted far-end reference signal, and microphone noise.

6. 2. The method of claim 1, further comprising: computing a loss function based on a scale-invariant signal-to-distortion ratio in the time domain and an L1 norm of a spectral magnitude difference between the enhanced signal and a target signal.

7. Adjusting the enhancement signal comprises: encoding a magnitude of an enhancement signal to predict the magnitude of the AGC; generating an AGC signal in the time domain based on a phase of the enhancement signal; and adjusting the enhancement signal based on the AGC signal. The method of claim 1.

8. at least one processor; and at least one memory storing a computer program for causing said at least one processor to carry out the method according to any one of claims 1 to 7. Device.

9. A computer program product for causing at least one processor to carry out the method according to any one of claims 1 to 7.

Images