Generalized automatic speech recognition for integrated acoustic echo cancellation, speech enhancement, and voice separation.

Abstract

A method for training a generalized automatic speech recognition model for integrated acoustic echo cancellation, speech enhancement, and sound separation includes receiving a plurality of training utterances paired with corresponding training context signals. The training context signals include a training context noise signal including noise before the training utterances, a training reference audio signal, and a training speaker vector including voice characteristics of the speaker who spoke the training utterances. The operations also include training a contextual front-end processing model on the training utterances to learn how to predict the enhanced speech features using a context signal dropout strategy. The context signal dropout strategy drops out each of the training context signals using a predetermined probability during training of the contextual front-end processing model.

Description

[Technical field]

[0001] This disclosure relates to generalized automatic speech recognition for integrated acoustic echo cancellation, speech enhancement, and voice separation. [Background technology]

[0002] The robustness of automatic speech recognition (ASR) systems has improved significantly over the years with the advent of neural network-based end-to-end models, large-scale training data, and improved strategies for expanding the training data. However, various conditions such as echo, stronger background noise, and competing voices significantly degrade the performance of ASR systems. An integrated ASR model can be trained to handle these conditions. However, in use, the integrated ASR model does not encounter all conditions that occur simultaneously. Therefore, it is not practical to train the integrated ASR model on all conditions that exist. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method for training a generalized automatic speech recognition model for combined echo cancellation, speech enhancement, and voice separation, which, when executed on data processing hardware, causes the data processing hardware to perform operations including receiving a plurality of training utterances paired with corresponding training context signals, the training context signals including a training context noise signal including noise prior to the corresponding training utterances, a training reference audio signal, and a training speaker vector including voice characteristics of a target speaker who spoke the corresponding training utterances, and training a contextual front-end processing model on the training utterances to learn how to predict enhanced speech features using a context signal dropout strategy, where the context signal dropout strategy drops out each of the training context signals using a predetermined probability during training of the contextual front-end processing model.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the signal dropout strategy drops out each training context signal by replacing the corresponding context signal with all zeros. In these implementations, replacing the training reference audio signal with all zeros includes replacing the training reference audio signal with all-zero features of the same length and feature dimensions as the corresponding training utterance. Additionally or alternatively, replacing the training context noise signal includes replacing the training context noise signal with all-zero features having a predetermined length and the same feature dimensions as the corresponding training utterance. Further, in these implementations, replacing the training speaker vectors includes replacing the training speaker vectors with all-zero features having an all-zero vector. In some examples, the signal dropout strategy drops out each training context signal by replacing the corresponding context signal with a frame-level learned representation.

[0005] In some implementations, the trained contextual front-end processing model includes a primary encoder, a noise context encoder, a cross-attention encoder, and a decoder. The primary encoder receives as input input speech features corresponding to the target speech and generates as output a primary input encoding. The noise context encoder receives as input a contextual noise signal including noise prior to the target speech and generates as output a contextual noise encoding. The cross-attention encoder receives as input the primary input encoding generated as output from the primary encoder and the contextual noise encoding generated as output from the noise context encoder and generates as output a cross-attention embedding. The decoder decodes the cross-attention embedding into enhanced input speech features corresponding to the target speech. In these implementations, the primary encoder is further configured to receive as input reference features corresponding to a reference audio signal and generate as output a primary input encoding by processing the input speech features stacked with the reference features. Alternatively, the primary encoder is further configured to receive as input a speaker embedding including voice characteristics of the target speaker who spoke the target utterance, and generate as output a primary input encoding by combining the input speech characteristics with the speaker embedding using feature-based linear modulation (FiLM). Additionally or alternatively, the cross-attention encoder is further configured to receive as input a primary input encoding modulated by the speaker embedding using feature-based linear modulation (FiLM), where the speaker embedding includes voice characteristics of the target speaker who spoke the target utterance, and process the primary input encoding modulated by the speaker embedding and the context noise encoding to generate as output a cross-attention embedding. In some implementations, the primary encoder includes N modulation conformer blocks, the context noise encoder includes N conformer blocks, and runs in parallel with the primary encoder, and the cross-attention encoder includes M modulation cross-attention conformer blocks.

[0006] In some examples, the contextual front-end processing model is trained in conjunction with a back-end automatic speech recognition (ASR) model using a spectral loss and an ASR loss. In these examples, the spectral loss may be based on a distance between an estimated ratio mask and an ideal ratio mask of an L1 loss function and an L2 loss function, where the ideal ratio mask is calculated using reverberant speech and reverberant noise. Furthermore, in these examples, the ASR loss is calculated by receiving a predicted output of an ASR encoder of the enhanced speech features using the enhanced speech features predicted by the contextual front-end processing model of the training utterance as input, generating a target output of an ASR encoder of the target speech features using an ASR encoder configured to receive as input the target speech features of the training utterance, and calculating an ASR loss based on the predicted output of the ASR encoder of the enhanced speech features and the target output of the ASR encoder of the target speech features.

[0007] Another aspect of the present disclosure provides a system for training a generalized automatic speech recognition model for integrated echo cancellation, speech enhancement, and voice separation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a plurality of training utterances paired with corresponding training context signals. The training context signals include a training context noise signal including noise prior to the corresponding training utterances, a training reference audio signal, and a training speaker vector including voice characteristics of a target speaker who spoke the corresponding training utterances. The operations also include training a contextual front-end processing model on the training utterances to learn how to predict the enhanced speech features using a context signal dropout strategy, where the context signal dropout strategy drops out each of the training context signals using a predetermined probability during training of the contextual front-end processing model.

[0008] This aspect may include one or more of the following optional features: In some implementations, the signal dropout strategy drops out each training context signal by replacing the corresponding context signal with all zeros. In these implementations, replacing the training reference audio signal with all zeros includes replacing the training reference audio signal with all-zero features of the same length and feature dimensions as the corresponding training utterance. Additionally or alternatively, replacing the training context noise signal includes replacing the training context noise signal with all-zero features having a predetermined length and the same feature dimensions as the corresponding training utterance. Additionally, in these implementations, replacing the training speaker vectors includes replacing the training speaker vectors with all-zero features. In some examples, the signal dropout strategy drops out each training context signal by replacing the corresponding context signal with a frame-level learned representation.

[0009] In some implementations, the trained contextual front-end processing model includes a primary encoder, a noise context encoder, a cross-attention encoder, and a decoder. The primary encoder receives as input input speech features corresponding to the target utterance and generates as output a primary input encoding. The noise context encoder receives as input a contextual noise signal including noise before the target utterance and generates as output a contextual noise encoding. The cross-attention encoder receives as input the primary input encoding generated as output from the primary encoder and the contextual noise encoding generated as output from the noise context encoder and generates as output a cross-attention embedding. The decoder decodes the cross-attention embedding into enhanced input speech features corresponding to the target utterance. In these implementations, the primary encoder is further configured to receive as input reference features corresponding to a reference audio signal and generate as output a primary input encoding by processing the input speech features stacked with the reference features. Alternatively, the primary encoder is further configured to receive as input a speaker embedding including voice characteristics of the target speaker who spoke the target utterance and generate as output a primary input encoding by combining the input speech features with the speaker embedding using feature-based linear modulation (FiLM). Additionally or alternatively, the cross-attention encoder is further configured to receive as input a primal input encoding modulated by a speaker embedding using feature-wise linear modulation (FiLM), where the speaker embedding comprises voice characteristics of a target speaker who spoke the target utterance, and process the primal input encoding modulated by the speaker embedding and the context noise encoding to generate a cross-attention embedding as output. In some implementations, the primary encoder comprises N modulated conformer blocks, the context noise encoder comprises N conformer blocks and runs in parallel with the primary encoder, and the cross-attention encoder comprises M modulated cross-attention conformer blocks.

[0010] In some examples, the contextual front-end processing model is trained in conjunction with a back-end automatic speech recognition (ASR) model using a spectral loss and an ASR loss. In these examples, the spectral loss may be based on a distance between an estimated ratio mask and an ideal ratio mask of an L1 loss function and an L2 loss function, where the ideal ratio mask is calculated using reverberant speech and reverberant noise. Additionally, in these examples, the ASR loss is calculated by receiving a predicted output of an ASR encoder of the enhanced speech features using the enhanced speech features predicted by the contextual front-end processing model of the training utterance as input, generating a target output of an ASR encoder of the target speech features using an ASR encoder configured to receive a target speech feature of the training utterance as input, and calculating an ASR loss based on the predicted output of the ASR encoder of the enhanced speech features and the target output of the ASR encoder of the target speech features.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of a system including a user communicating a spoken target utterance to a voice-enabled user device. [Diagram 2] FIG. 2 is a schematic diagram of the contextual front-end processing model of FIG. 1. [Diagram 3] FIG. 2 is a schematic diagram of a modulation conformer block. [Figure 4] FIG. 1 is a schematic diagram of a modulation conformer block architecture implemented by a cross-attention encoder of the context front-end processing model. [Diagram 5] FIG. 1 is a schematic diagram of an example training process for training a contextual front-end processing model. [Figure 6]FIG. 1 is a schematic diagram of an example training process for jointly training a contextual front-end processing model and an automatic speech recognition model. [Figure 7] 1 is an exemplary flowchart of an exemplary arrangement of operations for a method of automatic speech recognition using a contextual front-end processing model. [Figure 8] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference symbols in the various drawings indicate like elements.

[0014] The robustness of automatic speech recognition (ASR) systems has improved significantly over the years with the advent of neural network-based end-to-end models, large-scale training data, and improved strategies for expanding the training data. Nevertheless, background interference can significantly degrade the ability of an ASR system to accurately recognize speech directed at it. Background interference can be broadly categorized into three groups: device echo, background noise, and competing voices. To treat each of these background interference groups in isolation, separate ASR models may be trained, but maintaining ASR models specific to multiple tasks / conditions and switching between models on the fly during use is difficult and impractical.

[0015] Device echo may correspond to the playback audio output from a device, such as a smart home speaker, whereby the playback audio is recorded as an echo and can affect the performance of back-end speech systems, such as ASR systems. In particular, the degradation of the performance of the back-end speech system is particularly severe when the playback audio includes audible voice, for example, a text-to-speech (TTS) response from a digital assistant. This problem is typically addressed via acoustic echo cancellation (AEC) techniques. A unique property of AEC is that a reference signal corresponding to the playback audio is typically available and can be used for suppression.

[0016] Background noise with non-voice characteristics is usually adequately handled using data augmentation strategies such as multi-style training (MTR) of ASR models, where noise is added to the training data using a room simulator, and then they are carefully weighted with clean data during training to balance performance between clean and noisy conditions. As a result, large-scale ASR models are robust to moderate levels of non-voice noise. However, in the presence of low signal-to-noise ratio (SNR) conditions, background noise can still impact the performance of the back-end speech system.

[0017] Unlike non-speech background noise, competing speech is very challenging for ASR models trained to recognize a single speaker. Training an ASR model with multiple talker speech can be problematic in itself, as it is difficult to disambiguate which speaker to focus on during inference. Using a model that recognizes multiple speakers is also suboptimal, since it is difficult to know in advance how many users to support. Moreover, such multi-speaker models usually perform poorly in a single-speaker setting, making them undesirable.

[0018] The three aforementioned classes of background interference are typically addressed in isolation from one another, each using a separate modeling strategy. In recent literature, speech separation using speaker embeddings, along with techniques such as deep clustering, permutation invariant learning, has received much attention. When using speaker embeddings, the target speaker of interest is assumed to be known a priori. Techniques developed for speaker separation are also applied to remove non-speech noise, modifying the training data. AEC has also been studied, alone or together, in the presence of background noise. It is well known that improving speech quality does not necessarily improve ASR performance, since distortions introduced by nonlinear processing can adversely affect ASR performance. One way to mitigate the mismatch between the augmented front-end that first processes the incoming audio and the resulting ASR performance is to integrate and train the augmented front-end together with a back-end ASR model.

[0019] Furthermore, as the application of large-scale multi-domain and multi-lingual ASR models continues to attract interest, the training data of these ASR models usually covers various acoustic and linguistic use cases (e.g., voice search and video captioning), which makes it difficult to simultaneously deal with more challenging noise conditions. As a result, it is often convenient to train and maintain a separate front-end feature processing model that can deal with unfavorable conditions without combining it with a back-end ASR model. Furthermore, although various types of data are available for training the ASR model, the ASR model also needs to perform well when one or more of the aforementioned background interference groups (e.g., device echo, background noise, and competing voices) are missing from the training examples.

[0020] Embodiments herein are directed to training a contextual front-end processing model to improve ASR robustness by implementing an integrated implementation of acoustic echo cancellation (AEC), speech enhancement, and speech separation modules in a single model. The single integrated model is practical in view of the fact that it is difficult, if not impossible, to know in advance which class of background interference to address, especially in streaming ASR settings. Specifically, the contextual front-end processing model includes a context-enhanced neural network (CENN) that can optionally utilize three different types of side context inputs: a reference signal associated with the playback audio, a noise context, and a speaker embedding representing the voice characteristics of the target speaker of interest. Embodiments herein are more specifically directed to using a context signal dropout strategy for training the contextual front-end processing model to improve the performance of the model during inference when one or more context inputs are missing. As will become apparent, the reference signal associated with the playback audio is necessary to provide echo cancellation, while the noise context is useful for speech enhancement. Additionally, a speaker embedding (when available) representing the voice characteristics of the target speaker is not only important for speech separation, but also useful for echo cancellation and speech enhancement. For speech enhancement and separation, the noise context, i.e., a few seconds of speech preceding the target utterance to be recognized, conveys useful information about the acoustic context. The CENN uses a respective neural network architecture configured to take in the corresponding context-side inputs, respectively, to generate enhanced input speech features that can be passed to a back-end speech system, such as an ASR model, which can process the enhanced input speech features to generate speech, and generate speech recognition results for the target utterance. In particular, since the noise context and reference features are optional context-side inputs, the noise context and reference features are assumed by the CENN to be respective unannounced silence signals when unavailable.

[0021] 1, in some implementations, the system 100 includes a user 10 in an audio environment communicating a spoken target utterance 12 to a voice-enabled user device 110 (also referred to as device 110 or user device 110). The user 10 (i.e., the speaker of the utterance 12) may speak the target utterance 12 as a query or command that solicits a response from the device 110. The device 110 is configured to capture sounds from one or more users 10, 11 within the audio environment. Here, audio sounds may refer to the utterance 12 spoken by the user 10 functioning as an audible query, a command to the device 110, or an audible communication captured by the device 110. The device 110 or a voice-enabled system associated with the device 110 may address the query of the command by causing the query and / or command to be executed.

[0022] Various types of background interference may interfere with the ability of the back-end speech system 180 to process the target utterance 12 specifying a query or command to the device 110. As mentioned above, the background interference may include one or more of device echo corresponding to the playback audio 154 (also referred to as the reference audio signal 154) output from the user device (e.g., smart speaker) 110, competing voices 13 such as utterances other than the target utterance 12 spoken by one or more other users 111 not directed to the user device 110, and background noise having non-voice characteristics. In the embodiments herein, a contextual front-end processing model 200 (also referred to as the model 200) running on the device 110 and configured to receive as input input voice features corresponding to the target utterance 12 and one or more context input features 213, 214, 215 is used to process the input voice features 212 and one or more context input features 213, 214, 215 to generate as output enhanced input voice features 250 corresponding to the target utterance 12. As described in more detail below (e.g., FIG. 5), the model 200 may be trained using a context signal dropout strategy to improve the performance of the model 200 during inference when one or more of the context input features 213, 214, 215 are missing. The back-end speech system 180 may then process the enhanced input speech features 250 to generate the output 182. In particular, the context front-end processing model 200 effectively removes the presence of background interference recorded by the device 110 when the user 10 spoke the target utterance 12, such that the enhanced input speech features 250 provided to the back-end speech system 180 convey the speech intended for the device 110 (i.e., the target utterance 12) such that the output 182 generated by the back-end speech system 180 is not degraded by the background interference.

[0023] In the depicted example, the backend speech system 180 includes an ASR system 190 that uses an ASR model 192 to process the enhanced speech features 250 and generate a speech recognition result (e.g., a transcription) for the target utterance 12. The ASR system 190 may further include a natural language understanding (NLU) module (not shown) that performs semantic interpretation on the transcription of the target utterance 12 to identify a query / command directed to the device 110. Thus, the output 182 from the backend speech system 180 may include a transcription and / or instructions for satisfying the query / command identified by the NLU module.

[0024] The backend speech system 180 may additionally or alternatively include a hotword detection model (not shown) configured to detect whether the enhanced input speech features 250 include the presence of one or more hot / warm words that the hotword detection model was trained to detect. For example, the hotword detection model may output a hotword detection score indicating the likelihood that the enhanced input speech features 250 corresponding to the target utterance 12 include a particular hot / warm word. Detection of a hotword may trigger a wake-up process that may wake the device 110 from a sleep state. For example, the device 110 may wake up and process the hotword and / or one or more terms preceding / following the hotword.

[0025] In an additional example, the background audio system 180 includes an audio or audio-video calling application (e.g., a video conferencing application) where the enhanced input audio features 250 corresponding to the target utterance 12 are used by the audio or audio-video calling application to filter the voice of the target speaker 10 for communication to a recipient during an audio or audio-video communication session. The background audio system 180 may additionally or alternatively include a speaker identification model configured to perform speaker identification using the enhanced input audio features 250 to identify the user 10 who spoke the target utterance 12.

[0026] In the illustrated example, the device 110 captures a noisy audio signal 202 (also referred to as audio data) of a target utterance 12 spoken by a user 10 in the presence of background interference emanating from one or more sources other than the user 10. The device 110 may correspond to any computing device associated with the user 10 and capable of receiving the noisy audio signal 202. Some examples of the user device 110 include, but are not limited to, a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a computer, a wearable device (e.g., a smart watch), a smart appliance, an Internet of Things (IoT) device, a smart speaker, and the like. The device 110 includes a data processing hardware 112 and a memory hardware 114 in communication with the data processing hardware 112, which stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The context front-end processing model 200 may be executed on the data processing hardware 112. In some examples, the back-end voice system 180 runs on the data processing hardware 112 .

[0027] In some examples, device 110 includes one or more applications (i.e., software applications), each of which may utilize enhanced input speech features 250 generated by contextual front-end processing model 200 to perform various functions within the application. For example, device 110 includes an assistant application configured to communicate synthesized playback audio 154 to user 10 to assist user 10 with various tasks.

[0028] Device 110 further includes an audio subsystem with an audio capture device (e.g., a microphone) 116 for capturing and converting spoken utterances 12 into electrical signals within the audio environment, and an audio output device (e.g., a speaker) 118 for communicating an audible audio signal (e.g., a composite playback signal 154 from device 110). In the illustrated example, device 110 implements a single audio capture device 116, but device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, where one or more audio capture devices 116 of the array may communicate with the audio subsystem (e.g., peripherals of device 110) without physically residing at device 110. For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones located throughout the vehicle.

[0029] In some examples, the device 110 is configured to communicate with a remote system 130 over a network (not shown). The remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). The user device 110 may utilize the remote resources 132 to perform various functions related to voice processing and / or synthesis playback communication. The contextual front-end processing model 200 and the back-end voice system 180 may reside on the device 110 (referred to as an on-device system) or may reside remotely while in communication with the device 110 (e.g., may reside on the remote system 130). In some examples, one or more back-end voice systems 180 reside locally or on the device, while one or more other back-end voice systems 180 reside remotely. In other words, the one or more backend speech systems 180 utilizing the enhanced input speech features 250 output from the contextual frontend processing model 200 may be local or remote in any combination. For example, if the size or processing requirements of a system 180 are significant, the system 180 may reside on a remote system 130. Furthermore, if the device 110 can support the size or processing requirements of the one or more systems 180, the one or more systems 180 may reside on the device 110 using the data processing hardware 112 and / or memory hardware 114. Optionally, one or more systems 180 may reside both locally / on-device and remotely. For example, the backend speech system 180 may run on the remote system 130 by default when a connection between the device 110 and the remote system 130 is available, but when the connection is lost or unavailable, the system 180 runs locally on the device 110 instead.

[0030] In some implementations, device 110 or a system associated with device 110 identifies text that device 110 communicates to user 10 in response to a query spoken by user 10. Device 110 may then use a text-to-speech (TTS) system to convert the text into corresponding synthetic playback audio 154 for device 110 to communicate to user 10 (e.g., audibly communicate with user 10) in response to the query. Once generated, the TTS system communicates synthetic playback audio 154 to device 110, enabling device 110 to output synthetic playback audio 154. For example, device 110 outputs synthetic playback audio 154 of "It's sunny today" on speaker 118 of device 110 in response to user 10 providing a verbal query regarding today's weather forecast.

[0031] Continuing to refer to FIG. 1, when the device 110 outputs the synthetic playback audio 154, the synthetic playback audio 154 generates an echo 156 captured by the audio capture device 116. The synthetic playback audio 154 corresponds to a reference audio signal. Meanwhile, the synthetic playback audio 154 illustrates a reference audio signal in the example of FIG. 1, which may include other types of playback audio 154 including media content output from the speaker 118, or a communication from a remote user 10 with whom the user 10 is conversing via the device 110 (e.g., a voice-over-IP call or a video conference call). Unfortunately, in addition to the echo 156, the audio capture device 116 may also simultaneously capture a target utterance 12 spoken by the user 10, which includes a supplemental query further inquiring about the weather by saying, "How is tomorrow?" For example, FIG. 1 illustrates a target utterance 12 spoken by the user 10, which further inquires about the weather by saying, "How is tomorrow?" to the device 110, as the device 110 outputs the synthetic playback audio 154. Here, the spoken utterance 12 and the echo 156 are both captured simultaneously by the audio capture device 116 to form a noisy audio signal 202. In other words, the audio signal 202 includes an overlapping audio signal in which a portion of the target utterance 12 spoken by the user 10 overlaps with a portion of the reference audio signal (e.g., synthetic playback audio) 154 output from the speaker 118 of the device 110. In addition to the synthetic playback audio 154, a competing voice 13 spoken by another user 11 in the environment may also be captured by the audio capture device 116 and may contribute background interference that overlaps with the target utterance 12.

[0032] 1 , the back-end speech system 180 may have problems processing the target utterance 12 corresponding to the supplemental weather query, "How is tomorrow?", due to the presence of background interference in the noisy audio signal 202 caused by at least one of the playback audio 154, competing speech 13, or non-speech background noise interfering with the target utterance 12. The contextual front-end processing model 200 is used to improve the robustness of the back-end speech system 180 by integrating and implementing acoustic echo cancellation (AEC), speech enhancement, and speech separation models / modules into a single model.

[0033] To perform acoustic echo cancellation (AEC), the single model 200 uses a reference signal 154 being played by the device as an input to the model 200. The reference signal 154 is assumed to be time-aligned with and of the same length as the target utterance 12. In some examples, a feature extractor (not shown) extracts reference features 214 corresponding to the reference audio signal 154. The reference features 214 may include log-mel filter bank energy (LFBE) features of the reference audio signal 154. Similarly, the feature extractor may extract input speech features 212 corresponding to the target utterance 12. The input speech features 212 may include LFBE features. As described in more detail below, the input speech features 212 may be stacked with the reference features 214 and provided as an input to a primary encoder 210 (FIG. 2) of the single model 200 to perform AEC. When there is no reference audio signal 154 played by the device, an all-zero reference signal may be used, such that only the input speech features 212 are received as input to the primary encoder 210.

[0034] In the case where the single model 200 processes a contextual noise signal 213 associated with a predetermined period of a noise segment captured by the audio capture device 116 prior to the target utterance 12 spoken by the user 10, the single model 200 may further perform speech enhancement in parallel with the AEC by applying noise context modeling. In some examples, the predetermined period includes six (6) seconds of the noise segment. Thus, the contextual noise signal 213 provides a noise context. In some examples, the contextual noise signal 213 includes LFBE features of the noise context signal for use as context information.

[0035] Optionally, the single model 200 may further perform target speaker modeling for speech separation, integrated with AEC and speech enhancement. Here, speaker embeddings 215 are received as input by the single model 200. The speaker embeddings 215 may include speech features of the target speaker 10 who spoke the target utterance 12. The speaker embeddings 215 may include d-vectors. In some examples, the speaker embeddings 215 are computed using a text-independent speaker identification (TI-SID) model trained with a generalized end-to-end augmented set softmax loss. The TI-SID may include three long short-term memory (LSTM) layers with 768 nodes and a projection size of 256. The output of the final frame of the last LSTM layer is then linearly transformed into a final 256-dimensional d-vector.

[0036] For training and evaluation, each target utterance may be paired with a separate "enrollment" utterance from the same speaker. The enrollment utterance may be randomly selected from the pool of available utterances of the target speaker. The d-vector is then calculated for the enrollment utterance. In most practical applications, the enrollment utterance is usually obtained via a separate offline process.

[0037] FIG. 2 illustrates the context front-end processing model 200 of FIG. 1. The context front-end processing model 200 uses a modified version of a conformal neural network architecture that combines convolution and self-attention to model short-range and long-range interactions. The model 200 includes a primary encoder 210, a noise context encoder 220, a cross-attention encoder 400, and a decoder 240. The primary encoder 210 may include N modulated conformer blocks. The noise context encoder 220 may include N conformer blocks. The cross-attention encoder 230 may include M modulated cross-attention conformer blocks. The primary context encoder 210 and the noise context encoder 220 may run in parallel. As used herein, each conformer block may use localized and causal self-attention to enable streaming capabilities.

[0038] The primary encoder 210 may be configured to receive as input input speech features 212 corresponding to a target utterance and generate as output a primary input encoding 218. If a reference audio signal 154 is available, the primary encoder 210 is configured to receive as input input speech features 212 stacked with reference features 214 corresponding to the reference audio signal and generate a primary input encoding by processing the input speech features 212 stacked with the reference features 214. The input speech features and the reference features may each include a respective sequence of LFBE features.

[0039] The primary encoder 210 may further be configured to receive as input a speaker embedding 215 (i.e., if available) that includes voice characteristics of the target speaker (i.e., user) 10 who spoke the target utterance 12, and generate as output a primary input encoding 218 by combining the input speech features 212 (or input speech features stacked with reference features 214) using a feature-based linear modulation (FiLM) layer 310 (FIG. 3). FIG. 3 provides an exemplary modulation conformer block 320 used by the primary encoder 210. Here, before each conformer block 320 in the primary encoder 210, the speaker embedding 215 (e.g., d-vector) is combined with the input speech features 212 (or stack of input speech and reference features 214) using the FiLM layer 310 to generate the output 312. FiLM allows the primary encoder 210 to adjust its encoding based on the speaker embedding 215 of the target speaker 10. After the FiLM layer 310, residual connections 314 are added to combine the input speech features 212 (or the input speech features 212 stacked with the reference features 214) with the output 312 of the FiLM layer 310 to generate modulated input features 316 for the conformer block 320 as input to ensure that the architecture can perform well when the speaker embedding 215 is not present. Mathematically, the modulation conformer block 320 transforms the input features x using the modulation features m to generate output features y as follows:

number

[0040] where h(·) and r(·) are affine transformations. FFN, Conv, and MHSA represent the feedforward module, the convolution module, and the multi-head self-attention module, respectively. Equation 1 shows the feature-based linear modulation (FiLM) layer 310 with residual connections.

[0041] Referring again to FIG. 2, the noise context encoder 220 is configured to receive as input a context noise signal 213 including noise before the target utterance and generate as output a context noise encoding 222. The context noise signal 213 may include LFBE features of the context noise signal. The noise context encoder 220, unlike the primary and cross attention encoders 210, 400, includes standard conformer blocks without modulation by the speaker embeddings 215. Since the context noise signal 213 is associated with an acoustic noise context before the target utterance 12 is spoken, the noise context encoder 220 does not modulate the context noise signal 213 with the speaker embeddings 215 and is therefore assumed to include information to be forwarded to the cross attention encoder 400 to aid noise suppression.

[0042] 2, the cross-attention encoder 400 may be configured to receive as input the main input encodings 218 generated as output from the primary encoder 210 and the context noise encodings 222 generated as output from the noise context encoder 220, and generate as output the cross-attention embeddings 480. The decoder 240 is then configured to decode the cross-attention embeddings 480 into enhanced input speech features 250 corresponding to the target utterance 12. The context noise encodings 222 may correspond to an auxiliary input. The decoder 240 may include a simple projection decoder with a single-layer frame-wise fully connected network with sigmoid activation.

[0043] As shown in FIG. 4, the cross-attention encoder 400 may use a set of M modulation conformer blocks each of which receives as input the primary input encoding 218 modulated by the speaker embedding 215 using FiLM as described in FIG. 3, and the context noise encoding 222 output from the noise context encoder 220. The cross-attention encoder 400 first processes the modulation input 218 and the auxiliary input 222 independently using a half feedforward net 402, a first residual connection 404, a convolution block 406, and a second residual connection 408. Specifically, the modulation input 218 is processed by a half feedforward net 402a, which generates an output 403a. The first residual connection 404a then combines the modulation input 218 with the output 403a of the half feedforward net 402a to generate the modulation input features 405a. The modulation input features 405a are input to a convolution block 406a, which generates a convolution output 407a. A second residual connection 408a combines the convolution output 407a of the convolution block 406a with the modulated input features 405a to produce an output that includes the query vector 409a.

[0044] Similarly, the auxiliary input 222 is processed by a half feedforward net 402b which generates an output 403a. A first residual connection 404b then combines the auxiliary input 222 with the output 403b of the half feedforward net 402b to generate modulated input features 405b. The modulated input features 405b are input to a convolution block 406b which generates a convolution output 407b. A second residual connection 408b combines the convolution output 407b of the convolution block 406b with the modulated input features 405b to generate an output including a first key vector 409b and a first value vector 409c.

[0045] Subsequently, a multi-head cross-attention (MHCA) module 410 receives as input the query vector 409a, the first key vector 409b, and the first value vector 409c, and summarizes these vectors 409a-c to generate a noise summary 412. Intuitively, the role of the MHCA module 410 is to summarize the noise context separately for each input frame to be enhanced. The noise summary 412 output by the MHCA module 410 is then merged with the query vector 409a using a FiLM layer 420, which generates a FiLM output 422.

[0046] The multi-head self-attention (MHSA) layer 430 receives the FiLM output 422 as input and merges the FiLM output 422 with the query vector 409a to generate the attention output 432. The third residual connection 434 receives the query vector 409a and the attention output 432 and combines the query vector 409a and the attention output 432 to generate the residual output 436. The feedforward module 440 then receives the residual output 436 of the third residual connection 434 as input and generates the feature output 442. The fourth residual connection 444 then combines the feature output 422 with the residual output 436 of the third residual output 434 to generate the merged input features 446. The merged input features 446 are then processed as input by the LayerNorm 450, which is sent to the convolution block 406b to generate the cross-attention embedding 480.

[0047] Mathematically, if x, m, and n are the encoded input, d vector, and the encoded noise context from the previous layer, the cross-attention encoder 400 performs the following:

number

[0048] The cross-attention encoder 400 produces as output a cross-attention embedding 480, which is passed along with the d-vector m and the encoded noise context n to the next layer of M modulation conformer blocks. Thus, the input is modulated by each of the M conformer blocks by both the speaker embedding 215 associated with the target speaker and the noise context encoding 222.

[0049] 5 illustrates an exemplary training process 500 for training the contextual front-end processing model 200 to generate enhanced input speech features 250 when one or more of the contextual input features 213, 214, 215 are absent. The training process 500 may be executed on the remote system 130 of FIG. 1. As shown, the training process obtains one or more training data sets 520 stored in a data store 510 and trains the contextual front-end processing model 200 on the training data sets 520. The data store 510 may reside on the memory hardware 136 of the remote system 130. Each training data set 520 may include multiple training examples 530, 530a-n, and each training example 530 may include a training utterance 532 paired with a corresponding training context signal 534, 534a-c. Specifically, the training context signal 534 includes a training context noise signal 534a that includes noise prior to the corresponding training utterance 532, a training reference audio signal 534b, and a training speaker vector 534c that includes the voice characteristics of the target speaker who spoke the corresponding training utterance 532.

[0050] As discussed above with respect to FIG. 1, during inference, the contextual front-end processing model 200 may not receive all of the contextual input features 213, 214, 215 at the same time. By training the contextual front-end processing model 200 with one or more missing training context signals 534, the contextual front-end processing model 200 is more likely to utilize alternatives of the contextual input features 213, 214, 215, rather than over-relying on the most relevant contextual input features 213, 214, 215. As a result, the contextual front-end processing model 200 can accurately predict the enhanced input speech features 250 when one or more of the contextual input features 213, 214, 215 are absent. To keep the contextual front-end processing model 200 static, any missing training context signals 534 need to be input to the contextual front-end processing model 200 in some way.

[0051] The training process 500 may also utilize a signal dropout model 550. The signal dropout model 550 receives the training context signals 534 as input from the data store 510 and uses a context signal dropout strategy to drop out one or more of the training context signals 534 before training the context front-end processing model 200. The context signal dropout strategy of the signal dropout model 550 may include a predetermined probability (e.g., 50%, 20%, etc.) of dropping out each of the training context signals 534, with the same predetermined probability being used for each of the training context signals 534. In other words, in a given training example 530, the signal dropout model 550 may use the context signal dropout strategy to drop out the training context noise signal 534a with a predetermined probability of 50%, the training reference audio signal 534b with a predetermined probability of 50%, and the training speaker vector 534c with a predetermined probability of 50%. Similarly, for a given training example, the signal dropout model 550 may use a context signal dropout strategy to drop out the training context noise signal 534a with a predetermined probability of 20%, the training reference audio signal 534b with a predetermined probability of 20%, and the training speaker vector 534c with a predetermined probability of 20%.

[0052] In addition to the signal dropout strategy, the signal dropout model 550 may trim the length of the training context noise signal 534a to include noise preceding the corresponding training utterance 532 with a uniformly distributed length of zero to six (0-6) seconds. In other words, the signal dropout model 550 implements the signal dropout strategy while simultaneously trimming the training context noise signal 534a. For example, for a given training example 530, even if the signal dropout model 550 does not drop out the training context noise signal 534a, the signal dropout model 550 may still trim the length of the training context noise signal 534a.

[0053] In some implementations, the signal dropout model 550 uses a signal dropout strategy to drop out each training context signal 534 by replacing the corresponding training context signal 534 with all zeros based on a predetermined probability. In these implementations, the signal dropout model 550 may replace the training context noise signal 534a with an all-zero feature having a predetermined length and the same feature dimension as the corresponding training utterance 532. For example, the signal dropout strategy may include creating an all-zero feature that is six (6) seconds in length and has the same dimension as the LFBE feature. Similarly, the signal dropout model 550 may use a signal dropout strategy to replace the training reference audio signal 534b with an all-zero feature having the same length and feature dimension as the corresponding training utterance 532. Here, the all-zero feature dimension of the training reference audio signal 534b corresponds to the LFBE feature of the training reference audio signal 534b if the signal dropout strategy did not drop out the training reference audio signal 534b. Similarly, the signal dropout model 550 may use a signal dropout strategy to replace the training speaker vector 534c with an all-zero feature with an all-zero vector, where the training speaker vector 534c is replaced with a 256-dimensional all-zero vector. In other implementations, the signal dropout model 550 uses a signal dropout strategy to drop out each training context signal 534 by replacing the corresponding training context signal 534 with a frame-level learned representation based on a predetermined probability.

[0054] 5, the signal dropout model 550 receives the training context signals 534a-c as inputs and, using a given probability of a context signal dropout strategy, drops out the training reference audio signal 534b by replacing it with all-zero features of the same length and feature dimension as the corresponding training utterance 532. In other words, at this time step, the context front-end processing model 200 is trained only on the training context signals 534a, 534c, which approximates the conditions that the model 200 may encounter during inference when the context input features 213, 214, 215 only include the context noise signal 213 and the speaker embeddings 215.

[0055] After the signal dropout model 550 drops out the training reference audio signal 534b, the training utterance 532 and the training context signal 534 including the training reference audio signal 534b are replaced with all-zero features and dimensions to simulate the training reference audio signal 534b as the dropouts are provided to train the contextual front-end processing model 200. The contextual front-end processing model 200 receives as input the training utterance 532 and the training context signal 534 simulating the training reference audio signal 534b as the dropouts, and outputs the output prediction y r Generate the output prediction y r contains the enhanced input speech features 250 that are being tested for their accuracy. At each time step during the training process 500, the contextual front-end processing model 200 calculates the r-1 It is additionally trained using the output predictions of

[0056] 6 shows an example training process 600 for calculating the ASR loss 640 when the context front-end processing model 200 is trained jointly with the ASR model 192. Here, only the encoder 620 of the ASR model 192 is used to calculate the loss. The ASR loss 640 is calculated as the l2 distance between the output of the ASR encoder 620 for the target features 540 of the training utterances 532 and the enhanced input speech features 250. The ASR encoder 620 is not updated during the training process 600. In particular, the training process 600 calculates the ASR loss 640 by using an ASR encoder 620 of the ASR model 192 configured to receive as input the enhanced input speech features 250 predicted by the context front-end processing model 200 of the training utterance 532 to generate a predicted output 622 of the ASR encoder 620 of the enhanced input speech features 250, and using an ASR encoder 620 configured to receive as input the target speech features 540 of the training utterance 532 to generate a target output 624 of the ASR encoder 620 of the target speech features 540. The predicted output 622 of the enhanced input speech features 250 and the target output 624 of the target speech features 540 may each include a respective sequence of LFBE features. The training process 600 then calculates, via a loss module 630, an ASR loss 640 based on the predicted output 622 of the ASR encoder 620 for the enhanced input speech features 250 and the target output 624 of the ASR encoder 620 for the target speech features 540. The goal of using the ASR loss 640 is to enhance the contextual front-end model 200 to better match the ASR model 192, which is important for obtaining the best performance from the contextual front-end model 200. By keeping the parameters of the ASR model 192 fixed, the ASR model 192 is decoupled from the contextual front-end model 200, thereby allowing each to be trained and deployed independently of one another.

[0057] In some implementations, the contextual front-end model 200 is trained jointly with the ASR model 192 of the back-end automatic speech recognition system 180 using the spectral and ASR losses 640. The training targets 540 for training the contextual front-end model 200 use Ideal Ratio Masks (IRMs). The IRMs are computed using reverberant speech and reverberant noise under the assumption that speech and noise are uncorrelated in Mel spectral space, as follows:

number

number

number

number

number

[0058] During prediction, the estimated IRM is scaled and floored to reduce speech distortion at the expense of reduced noise suppression. This is particularly important since the ASR model 192 is susceptible to speech distortion and nonlinear front-end processing, which is one of the main challenges in improving the performance of robust ASR models using an enhancement front-end. The enhancement features are derived as follows:

number

number

number

[0059] 7 includes an exemplary flow chart of an exemplary arrangement of operations for a method 700 of training a generalized automatic speech recognition model using the contextual front-end processing model 200. At operation 702, the method 700 includes receiving a plurality of training utterances 532 paired with corresponding training context signals 534, 534a-c. The training context signals 534 include a training context noise signal 534a including noise prior to the corresponding training utterances 532, a training reference audio signal 534b, and a training speaker vector 534c including voice characteristics of a target speaker who spoke the corresponding training utterances 532. The method 700 also includes, at operation 704, training the contextual front-end processing model 200 with the training utterances 532 to learn how to predict the enhanced speech features 250 using a context signal dropout strategy. Here, the context signal dropout strategy simulates one or more of the training context signals 534 as missing to teach the model 200, during training of the context front-end processing model 200, with a predetermined probability of dropping out each of the training context signals 534, and learns how to robustly generate enhanced speech features 250 when any of the corresponding context input features are missing during prediction.

[0060] 8 is a schematic diagram of an exemplary computing device 800 that can be used to implement the systems and methods described herein. The computing device 800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the present disclosure described and / or claimed in this document.

[0061] Computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 that connects to memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 that connects to a low-speed bus 870 and storage device 830. Each of components 810, 820, 830, 840, 850, and 860 are interconnected using various buses and may be mounted on a common motherboard or exist in other manners as needed. Processor 810 (e.g., data processing hardware 112, 134 of FIG. 1) can process instructions for execution within computing device 800, including instructions stored in memory 820 or storage device 830, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 880 connected to high-speed interface 840. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple types of memories. Additionally, multiple computing devices 800 may be connected together (eg, as a bank of servers, a group of blade servers, or a multi-processor system) with each device providing multiple portions of the required operations.

[0062] The memory 820 (e.g., memory hardware 114, 136 of FIG. 1) stores information non-transiently within the computing device 800. The memory 820 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 820 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0063] The storage device 830 can provide mass storage for the computing device 800. In some embodiments, the storage device 830 is a computer-readable medium. In various different implementations, the storage device 830 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 820, the storage device 830, or a memory on the processor 810.

[0064] The high-speed controller 840 manages bandwidth-intensive operations of the computing device 800, and the low-speed controller 860 manages lower bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 840 is coupled to a memory 820, a display 880 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 850 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 860 is coupled to a storage device 830 and a low-speed expansion port 890. The low-speed expansion port 890 may include various communication ports (USB, Bluetooth, Ethernet, wireless Ethernet, etc.) and can connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0065] The computing device 800 may be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 800a, or multiple times within a group of such servers 800a, as a laptop computer 800b, or as part of a rack server system 800c.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0067] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0068] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0069] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0070] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0071] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, spoken language, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0072] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (700) that, when executed on data processing hardware (134), causes the data processing hardware (134) to perform operations, the operations comprising: receiving a plurality of training utterances (532) paired with corresponding training context signals (534, 534a-c), the training context signals (534a-c) comprising: a training context noise signal (534a) including noise preceding the corresponding training utterance (532); A training reference audio signal (543b); and a training speaker vector (534c) including voice characteristics of a target speaker who spoke the corresponding training utterance (532); training a context front-end processing model (200) with the training utterances (532) to learn how to predict enhanced speech features (250) using a context signal dropout strategy, the context signal dropout strategy dropping out each of the training context signals (534) with a predetermined probability during training of the context front-end processing model (200); A computer-implemented method (700), comprising:

2. 2. The computer-implemented method of claim 1, wherein the signal dropout strategy drops out each training context signal by replacing the corresponding training context signal with all zeros.

3. 3. The computer-implemented method of claim 2, wherein replacing the training reference audio signal with all zeros comprises replacing the training reference audio signal with all-zero features of the same length and feature dimension as the corresponding training utterance.

4. 4. The computer-implemented method of claim 2 or 3, wherein replacing the training context noise signal comprises replacing the training context noise signal with all-zero features having a predetermined length and the same feature dimensions as the corresponding training utterances.

5. The computer-implemented method (700) of any of claims 2-4, wherein replacing the training speaker vector (534c) comprises replacing the training speaker vector (534c) with an all-zero feature with an all-zero vector.

6. 2. The computer-implemented method of claim 1, wherein the signal dropout strategy drops out each training context signal by replacing the corresponding training context signal with a frame-level learned representation.

7. The trained contextual front-end processing model (200) A primary encoder (210), receiving as input input speech features (212) corresponding to a target utterance (12); a primary encoder (210) configured to generate as an output a primary input encoding (218); A noise context encoder (220), comprising: receiving as input a context noise signal (213) comprising noise preceding said target utterance (12); a noise context encoder (220) configured to generate as an output a context noise encoding (222); A cross attention encoder (400), comprising: receiving as input the primary input encoding (218) generated as output from the primary encoder (210) and the context noise encoding (222) generated as output from the noise context encoder (220); a cross-attention encoder (400) configured to generate as output a cross-attention embedding (480); a decoder (240) configured to decode the cross-attention embeddings (480) into enhanced speech features (250) corresponding to the target utterance (12); The computer-implemented method (700) of any of claims 1 to 6, comprising:

8. The primary encoder (210) further comprises: receiving as input reference features (214) corresponding to a reference audio signal (154); As an output, generating the primary input encoding (218) by processing the input speech features (212) stacked with the reference features (214). The computer-implemented method (700) of claim 7, configured to:

9. The primary encoder (210) further comprises: receiving as input a speaker embedding (215) including voice characteristics of a target speaker (10) who spoke the target utterance (12); As an output, the input speech features (212) are combined with the speaker embeddings (215) using Feature-Based Linear Modulation (FiLM) to generate the primal input encoding (218). A computer-implemented method (700) according to claim 7 or 8, adapted to:

10. The cross attention encoder (400) further comprises: receiving as an input the primary input encoding (218) modulated by a speaker embedding (215) using feature-based linear modulation (FiLM), the speaker embedding (215) comprising voice characteristics of a target speaker (10) who spoke the target utterance (12); processing the primary input encodings (218) modulated by the speaker embeddings (215) and the context noise encodings (222) to produce, as output, the cross-attention embeddings (480); The computer-implemented method (700) of any of claims 7 to 9, configured to:

11. The primary encoder (210) includes N modulation conformer blocks (320); The noise context encoder (220) includes N conformer blocks and runs in parallel with the primary encoder (210); The cross-attention encoder (400) comprises M modulated cross-attention conformer blocks. A computer implemented method (700) according to any of claims 7 to 10.

12. 12. The computer-implemented method (700) of any of claims 1 to 11, wherein the contextual front-end processing model (200) is trained in conjunction with a back-end automatic speech recognition (ASR) model (192) using spectral and ASR losses (640).

13. 13. The computer-implemented method (700) of claim 12, wherein the spectral loss is based on an L1 loss function and an L2 loss function distance between an estimated ratio mask and an ideal ratio mask, the ideal ratio mask being computed using reverberant speech and reverberant noise.

14. The ASR loss (640) is, for each training utterance (532), using an ASR encoder (620) of the ASR model (192) configured to receive as input enhanced speech features (250) predicted by the context front-end processing model (200) of the training utterance (532) using the context signal dropout strategy to generate a predicted output (622) of the ASR encoder (620) for the enhanced speech features (250); using the ASR encoder (620) configured to receive as input the target speech features (540) of the training utterances (532), generating a target output (624) of the ASR encoder (620) for the target speech features (540); calculating the ASR loss (640) based on the predicted output (622) of the ASR encoder (620) for the enhanced speech features (250) and the target output (624) of the ASR encoder (620) for the target speech features (540); 14. The computer-implemented method (700) of claim 12 or 13, wherein:

15. A system (100), comprising: Data processing hardware (134); and memory hardware (136) in communication with the data processing hardware (134) and configured to store instructions that, when executed on the data processing hardware (134), cause the data processing hardware (134) to: receiving a plurality of training utterances (532) paired with corresponding training context signals (534, 534a-c), the training context signals (534a-c) comprising: a training context noise signal (534a) including noise preceding the corresponding training utterance (532); A training reference audio signal (543b); and a training speaker vector (534c) including voice characteristics of a target speaker who spoke the corresponding training utterance (532); training a context front-end processing model (200) on the training utterances (532) to learn how to predict enhanced speech features (250) using a context signal dropout strategy, the context signal dropout strategy dropping out each of the training context signals (534) using a predetermined probability during training of the context front-end processing model (200); System (100).

16. 16. The system (100) of claim 15, wherein the signal dropout strategy drops out each training context signal (534) by replacing the corresponding training context signal (534) with all zeros.

17. 17. The system of claim 16, wherein replacing the training reference audio signal with all zeros comprises replacing the training reference audio signal with all-zero features of the same length and feature dimension as the corresponding training utterance.

18. 18. The system (100) of claim 16 or 17, wherein replacing the training context noise signal (534a) comprises replacing the training context noise signal (534a) with all-zero features having a predetermined length and the same feature dimensions as the corresponding training utterances (532).

19. The system (100) of any of claims 16 to 18, wherein replacing the training speaker vector (534c) comprises replacing the training speaker vector (534c) with an all-zero feature with an all-zero vector.

20. 16. The system (100) of claim 15, wherein the signal dropout strategy drops out each training context signal (534) by replacing the corresponding training context signal (534) with a frame-level learned representation.

21. The trained contextual front-end processing model (200) A primary encoder (210), receiving as input input speech features (212) corresponding to a target utterance (12); a primary encoder (210) configured to generate as an output a primary input encoding (218); A noise context encoder (220), comprising: receiving as input a context noise signal (213) comprising noise preceding said target utterance (12); a noise context encoder (220) configured to generate as an output a context noise encoding (222); A cross attention encoder (400), comprising: receiving as input the primary input encoding (218) generated as output from the primary encoder (210) and the context noise encoding (222) generated as output from the noise context encoder (220); a cross-attention encoder (400) configured to generate as output a cross-attention embedding (480); a decoder (240) configured to decode the cross-attention embeddings (480) into enhanced speech features (250) corresponding to the target utterance (12); The system (100) according to any one of claims 15 to 20, comprising:

22. The primary encoder (210) further comprises: receiving as input reference features (214) corresponding to a reference audio signal (154); 22. The system (100) of claim 21 configured to generate the primary input encoding (218) by processing the input speech features (212) stacked with the reference features (214) as output.

23. The primary encoder (210) further comprises: receiving as input a speaker embedding (215) including voice characteristics of a target speaker (10) who spoke the target utterance (12); 23. The system (100) of claim 21 or 22, configured to generate the primary input encoding (218) by combining the input speech features (212) with the speaker embeddings (215) using Feature-Based Linear Modulation (FiLM) as output.

24. The cross attention encoder (400) further comprises: receiving as an input the primary input encoding (218) modulated by a speaker embedding (215) using feature-based linear modulation (FiLM), the speaker embedding (215) comprising voice characteristics of a target speaker (10) who spoke the target utterance (12); processing the primary input encodings (218) modulated by the speaker embeddings (215) and the context noise encodings (222) to produce, as output, the cross-attention embeddings (480); The system (100) according to any one of claims 21 to 23, configured to:

25. The primary encoder (210) includes N modulation conformer blocks (320); The noise context encoder (220) includes N conformer blocks and runs in parallel with the primary encoder (210); The system (100) of any of claims 21 to 24, wherein the cross-attention encoder (400) comprises M modulated cross-attention conformer blocks.

26. The system (100) of any of claims 15 to 25, wherein the contextual front-end processing model (200) is trained in conjunction with a back-end automatic speech recognition (ASR) model (192) using spectral and ASR losses.

27. 27. The system (100) of claim 26, wherein the spectral loss is based on an L1 loss function and an L2 loss function distance between an estimated ratio mask and an ideal ratio mask, the ideal ratio mask being calculated using reverberant speech and reverberant noise.

28. The ASR loss is, for each training utterance (532), using an ASR encoder (620) of the ASR model (192) configured to receive as input enhanced speech features (250) predicted by the context front-end processing model (200) of the training utterance (532) using the context signal dropout strategy to generate a predicted output (622) of the ASR encoder (620) for the enhanced speech features (250); using the ASR encoder (620) configured to receive as input the target speech features (540) of the training utterances (532), generating a target output (624) of the ASR encoder (620) for the target speech features (540); calculating the ASR loss (640) based on the predicted output (622) of the ASR encoder (620) for the enhanced speech features (250) and the target output (624) of the ASR encoder (620) for the target speech features (540); 28. The system (100) of claim 26 or 27, wherein the calculated

Images