Universal automatic speech recognition for joint acoustic echo cancellation, speech enhancement, and speech separation

By adopting a field situation front-end processing model in the automatic speech recognition system, combining acoustic echo cancellation, speech enhancement and speech separation modules, the field situation signal temporary retreat strategy and feature linear modulation are used, and the performance degradation problem under complex conditions is solved and improved System robustness and speech recognition accuracy.

CN118985025APending Publication Date: 2024-11-19GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380034894.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-20
Filing Date
2023-02-19
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing automatic speech recognition system performs poorly in the face of complex conditions such as echo, background noise and competitive speech, and it is difficult to train and maintain models under multiple conditions at the same time, resulting in performance degradation.

Method used

The field-context front-end processing model is adopted, and the combined implementation of acoustic echo cancellation, speech enhancement and speech separation modules is implemented. The model is trained using the field-context signal temporary retreat strategy to improve the model's performance when some field-context input is missing, and combined with feature linearity Modulation and cross attention encoder to generate enhanced input speech features.

Benefits of technology

It improves the robustness of the automatic speech recognition system under complex background interference, enhances the processing ability of echo, noise and competitive speech, and improves the accuracy and robustness of speech recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118985025A_ABST
    Figure CN118985025A_ABST
Patent Text Reader

Abstract

A method (700) for training a generic automatic speech recognition model for joint acoustic echo cancellation, speech enhancement, and speech separation includes receiving a plurality of training utterances (532) paired with corresponding training context signals (534, 534a-c). The training context signal includes: a training context noise signal including noise prior to a corresponding training utterance; training the reference audio signal; and training speaker vectors including speech characteristics of the target speaker speaking the corresponding training utterance. The operations also include training the contextual front-end processing model (200) on the training utterance using the contextual signal sag strategy to learn how to predict the enhanced speech features (250). Here, the contextual signal sag strategy uses a predetermined probability to sag each of the training contextual signals during training of the contextual front-end processing model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to general automatic speech recognition for joint acoustic echo cancellation, speech enhancement and voice separation. Background Art

[0002] With the advent of end-to-end models based on neural networks, large-scale training data, and improved strategies for strengthening training data, the robustness of automatic speech recognition (ASR) systems has improved significantly over the years. However, various conditions such as echo, harsher background noise, and competing speech significantly degrade the performance of ASR systems. Joint ASR models can be trained to handle these conditions. However, in use, the joint ASR model may not encounter all conditions that occur simultaneously. Therefore, it is not practical to train a joint ASR model when all conditions exist. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method for training a general automatic speech recognition model for joint echo cancellation, speech enhancement, and voice separation, which, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving a plurality of training utterances paired with corresponding training context signals. The training context signals include: a training context noise signal, which includes noise preceding the corresponding training utterances; a training reference audio signal; and a training speaker vector, which includes voice characteristics of a target speaker who speaks the corresponding training utterances. The operations also include training a context front-end processing model on the training utterances using a context signal dropout strategy to learn how to predict enhanced speech features. Here, the context signal dropout strategy uses a predetermined probability to dropout each of the training context signals during training of the context front-end processing model.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the signal backoff strategy backs off each training context signal by replacing the corresponding context signal with all zeros. In these implementations, replacing the training reference audio signal with all zeros includes replacing the training reference audio signal with all zero features of the same length and feature dimension as the corresponding training utterance. Additionally or alternatively, replacing the training context noise signal includes replacing the training context noise signal with all zero features having a predetermined length and the same feature dimension as the corresponding training utterance. Furthermore, in these implementations, replacing the training speaker vector includes replacing the training speaker vector with all zero features having an all zero vector. In some examples, the signal backoff strategy backs off each training context signal by replacing the corresponding context signal with a representation learned at the frame level.

[0005] In some implementations, the trained context front-end processing model includes a main encoder, a noise context encoder, a cross-attention encoder, and a decoder. The main encoder receives input speech features corresponding to the target speech as input and generates a main input encoding as output. The noise context encoder receives a context noise signal including noise before the target speech as input and generates a context noise encoding as output. The cross-attention encoder receives the main input encoding generated from the main encoder as output and the context noise encoding generated from the noise context encoder as output as input, and generates a cross-attention embedding as output. The decoder decodes the cross-attention embedding into an enhanced input speech feature corresponding to the target speech. In these implementations, the main encoder is also configured to receive a reference feature corresponding to a reference audio signal as input, and generate a main input encoding as output by processing the input speech features stacked with the reference features. Alternatively, the main encoder is also configured to receive a speaker embedding including a voice feature of a target speaker who speaks the target speech as input, and generate a main input encoding as output by combining the input speech features with the speaker embedding using feature linear modulation (FiLM). Additionally or alternatively, the cross-attention encoder is further configured to receive as input a primary input encoding modulated by a speaker embedding using feature linear modulation (FiLM). Here, the speaker embedding includes voice characteristics of a target speaker speaking a target utterance, and processes the primary input encoding modulated by the speaker embedding and the context noise encoding to generate a cross-attention embedding as output. In some implementations, the main encoder includes N modulated conformer blocks, the context noise encoder includes N conformer blocks and is executed in parallel with the main encoder, and the cross-attention encoder includes M modulated cross-attention conformer blocks.

[0006] In some examples, the context front-end processing model is jointly trained with the back-end automatic speech recognition (ASR) model using spectral loss and ASR loss. In these examples, the spectral loss can be based on the L1 loss function and L2 loss function distance between the estimated ratio mask and the ideal ratio mask. Here, the ideal ratio mask is calculated using reverberant speech and reverberant noise. In addition, in these examples, the ASR loss is calculated by: receiving the enhanced speech features predicted by the context front-end processing model for the training speech as input, receiving the predicted output of the ASR encoder for the enhanced speech features, using the ASR encoder configured to receive the target speech features of the training speech as input to generate the target output of the ASR encoder for the target speech features, and calculating the ASR loss based on the predicted output of the ASR encoder for the enhanced speech features and the target output of the ASR encoder for the target speech features.

[0007] Another aspect of the present disclosure provides a system for training a general automatic speech recognition model for joint echo cancellation, speech enhancement, and voice separation. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving a plurality of training utterances paired with corresponding training context signals. The training context signals include: a training context noise signal including noise preceding the corresponding training utterances; a training reference audio signal; and a training speaker vector including voice characteristics of a target speaker who speaks the corresponding training utterances. The operation also includes training a context front-end processing model on the training utterances using a context signal back-off strategy to learn how to predict enhanced speech features. Here, the context signal back-off strategy backs off each of the training context signals using a predetermined probability during training of the context front-end processing model.

[0008] This aspect may include one or more of the following optional features. In some implementations, the signal backoff strategy backs off each training context signal by replacing the corresponding context signal with all zeros. In these implementations, replacing the training reference audio signal with all zeros includes replacing the training reference audio signal with all zero features of the same length and feature dimension as the corresponding training utterance. Additionally or alternatively, replacing the training context noise signal includes replacing the training context noise signal with all zero features having a predetermined length and the same feature dimension as the corresponding training utterance. Furthermore, in these implementations, replacing the training speaker vector includes replacing the training speaker vector with all zero features having an all zero vector. In some examples, the signal backoff strategy backs off each training context signal by replacing the corresponding context signal with a representation learned at the frame level.

[0009] In some implementations, the trained context front-end processing model includes a main encoder, a noise context encoder, a cross-attention encoder, and a decoder. The main encoder receives input speech features corresponding to the target speech as input and generates a main input encoding as output. The noise context encoder receives a context noise signal including noise before the target speech as input and generates a context noise encoding as output. The cross-attention encoder receives the main input encoding generated from the main encoder as output and the context noise encoding generated from the noise context encoder as output as input, and generates a cross-attention embedding as output. The decoder decodes the cross-attention embedding into an enhanced input speech feature corresponding to the target speech. In these implementations, the main encoder is also configured to receive a reference feature corresponding to a reference audio signal as input, and generate a main input encoding as output by processing the input speech features stacked with the reference features. Alternatively, the main encoder is also configured to receive a speaker embedding including a voice feature of a target speaker who speaks the target speech as input, and generate a main input encoding as output by combining the input speech features with the speaker embedding using feature linear modulation (FiLM). Additionally or alternatively, the cross-attention encoder is further configured to receive as input a primary input encoding modulated by a speaker embedding using feature linear modulation (FiLM). Here, the speaker embedding includes voice characteristics of a target speaker speaking a target utterance, and processes the primary input encoding modulated by the speaker embedding and the context noise encoding to generate a cross-attention embedding as output. In some implementations, the main encoder includes N modulated conformer blocks, the context noise encoder includes N conformer blocks and is executed in parallel with the main encoder, and the cross-attention encoder includes M modulated cross-attention conformer blocks.

[0010] In some examples, the context front-end processing model is jointly trained with the back-end automatic speech recognition (ASR) model using spectral loss and ASR loss. In these examples, the spectral loss can be based on the L1 loss function and L2 loss function distance between the estimated ratio mask and the ideal ratio mask. Here, the ideal ratio mask is calculated using reverberant speech and reverberant noise. In addition, in these examples, the ASR loss is calculated by: receiving the enhanced speech features predicted by the context front-end processing model for the training speech as input, receiving the predicted output of the ASR encoder for the enhanced speech features, using the ASR encoder configured to receive the target speech features of the training speech as input to generate the target output of the ASR encoder for the target speech features, and calculating the ASR loss based on the predicted output of the ASR encoder for the enhanced speech features and the target output of the ASR encoder for the target speech features.

[0011] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of a system including a user communicating a spoken target utterance to a speech-enabled user device.

[0013] Figure 2 yes Figure 1 Schematic diagram of the scene front-end processing model.

[0014] Figure 3 is a schematic diagram of the modulated conformer block.

[0015] Figure 4 Schematic diagram of the modulated conformer block architecture implemented by the criss-cross attention encoder of the context front-end processing model.

[0016] Figure 5 is a schematic diagram of an example training process for training a context front-end processing model.

[0017] Figure 6 is a schematic diagram of an example training process for jointly training a context front-end processing model and an automatic speech recognition model.

[0018] Figure 7 is an example flow chart of an example arrangement of operations of a method of automatic speech recognition using a contextual front-end processing model.

[0019] Figure 8 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] With the advent of neural network-based end-to-end models, large-scale training data, and improved strategies for strengthening training data, the robustness of automatic speech recognition (ASR) systems has improved significantly over the years. However, background interference can significantly degrade the ability of an ASR system to accurately recognize speech intended for the ASR system. Background interference can be roughly classified into three groups: device echo; background noise; and competing speech. Although separate ASR models can be trained to handle each of these background interference groups in isolation, the difficulty of maintaining multiple task / condition-specific ASR models and switching between running models during use is not practical.

[0022] Device echo can correspond to the playback audio output from a device such as a smart home speaker, whereby the playback audio is recorded as an echo and can affect the performance of a backend voice system such as an ASR system. Specifically, if the playback audio contains audible speech, such as a text-to-speech (TTS) response from a digital assistant, the degradation of the performance of the backend voice system is particularly severe. This problem is typically addressed via acoustic echo cancellation (AEC) technology. A unique feature of AEC is that a reference signal corresponding to the playback audio is typically available and can be used for suppression.

[0023] Background noise with non-speech characteristics is often handled well using data enhancement strategies such as multi-style training (MTR) of ASR models. Here, a room simulator is used to add noise to the training data, which is then carefully weighted with clean data during training to obtain a good performance balance between clean and noisy conditions. As a result, large-scale ASR models are robust to moderate levels of non-speech noise. However, in the presence of low signal-to-noise ratio (SNR) conditions, background noise can still affect the performance of the backend speech system.

[0024] Unlike non-speech background noise, competing speech is quite challenging for ASR models trained to recognize a single speaker. Training ASR models with multi-speaker speech can itself cause problems because it is difficult to disambiguate which speaker to focus on during inference. Using models that recognize multiple speakers is also suboptimal because it is difficult to know in advance how many users to support. In addition, such multi-speaker models usually have degraded performance in a single-speaker setting, which is undesirable.

[0025] These three aforementioned background interference classifications are usually solved in isolation from each other, each using a separate modeling strategy. Speech separation has received much attention in recent literature, using techniques such as deep clustering, permutation-invariant training, and using speaker embeddings. When speaker embeddings are used, it is assumed that the target speaker of interest is known a priori. Techniques developed for speaker separation have also been applied to remove non-speech noise, wherein the training data is modified. AEC has also been studied in isolation or together in the presence of background noise. It is well known that improving speech quality does not always improve ASR performance, because the distortion introduced by nonlinear processing can adversely affect ASR performance. One way to alleviate the difference between the enhanced front end that initially processes the incoming audio and the resulting ASR performance is to jointly train the enhanced front end and back end ASR models.

[0026] Furthermore, as the application of large-scale multi-domain and multi-lingual ASR models continues to gain interest, the training data for these ASR models typically covers a variety of acoustic and linguistic use cases (e.g., voice search and video subtitling), making it challenging to simultaneously address more severe noise conditions. Therefore, it is often convenient to train and maintain a separate front-end feature processing model that can handle adverse conditions without combining it with a back-end ASR model. Furthermore, while various types of data are available for training of the ASR model, the ASR model must also perform well when one or more of the aforementioned background interference groups (e.g., device echo, background noise, and competing speech) are missing from the training examples.

[0027] The implementations herein are directed to training a context front-end processing model to improve the robustness of ASR by jointly implementing acoustic echo cancellation (AEC), speech enhancement, and speech separation modules into a single model. A single joint model is practical from the perspective that it is difficult (if not impossible) to know in advance which type of background interference to address, especially in a streaming ASR setting. Specifically, the context front-end processing model includes a context enhancement neural network (CENN) that can optionally utilize three different types of secondary context inputs: a reference signal associated with the playback audio; a noise context; and a speaker embedding representing the voice characteristics of the target speaker of interest. The implementations herein are more specifically directed to training the context front-end processing model using a context signal backoff strategy to improve the performance of the model during inference when one or more context inputs are missing. As will be apparent, the reference signal associated with the playback audio is necessary to provide echo cancellation, while the noise context is useful for speech enhancement. In addition, the speaker embedding representing the voice characteristics of the target speaker (when available) is not only crucial for speech separation, but also helps with echo cancellation and speech enhancement. For speech enhancement and separation, the noise context (i.e., the audio several seconds before the target speech to be recognized) carries useful information about the acoustic context. CENN employs a corresponding neural network architecture that is configured to ingest each corresponding context side input to produce enhanced input speech features, which can be passed to a backend speech system, such as an ASR model that can process the enhanced input speech features to generate speech recognition results for the target speech. It is worth noting that since the noise context and reference features are optional context side inputs, when unavailable, the noise context and reference features are assumed by CENN to be corresponding uninformative silent signals.

[0028] refer to Figure 1In some implementations, the system 100 includes a user 10 communicating a spoken target utterance 12 to a speech-enabled user device 110 (also referred to as device 110 or user device 110) in a speech environment. The user 10 (i.e., the speaker of the utterance 12) may utter the target utterance 12 as a query or command to solicit a response from the device 110. The device 110 is configured to capture sound from one or more users 10, 11 within the speech environment. Here, audio sound may refer to a spoken utterance 12 of the user 10 that serves as an audible query, a command to the device 110, or an audible communication captured by the device 110. A speech-enabled system of or associated with the device 110 may process the query or command by answering the query and / or causing the command to be executed.

[0029] Various types of background interference can interfere with the ability of the backend speech system 180 to process the target speech 12, which specifies a query or command for the device 110. As described above, background interference can include: device echo corresponding to the playback audio 154 (also referred to as the reference audio signal 154) output from the user device (e.g., smart speaker) 110; competing speech 13, such as speech other than the target speech 12, spoken by one or more other users 111, not directed to the device 110; and background noise with non-speech characteristics. The implementation method of this article adopts a context front-end processing model 200 (also referred to as model 200), which is executed on the device 110 and is configured to receive input speech features corresponding to the target speech 12 and one or more context input features 213, 214, 215 as input, and generate enhanced input speech features 250 corresponding to the target speech 12 as output by processing the input speech features 212 and one or more context input features 213, 214, 215. As described in more detail below (for example, Figure 5 ), the model 200 may be trained using a context signal backoff strategy to improve the performance of the model 200 during inference when one or more of the context input features 213, 214, 215 are missing. The back-end speech system 180 may then process the enhanced input speech features 250 to generate the output 182. Notably, the context front-end processing model 200 effectively removes the presence of background interference recorded by the device 110 when the user 10 speaks the target utterance 12, such that the enhanced input speech features 250 provided to the back-end speech system 180 convey the speech intended for the device 110 (i.e., the target utterance 12), such that the output 182 generated by the back-end speech system 180 is not degraded by the background interference.

[0030] In the example shown, the backend speech system 180 includes an ASR system 190 that processes the enhanced input speech features 250 using an ASR model 192 to generate a speech recognition result (e.g., a transcription) of the target utterance 12. The ASR system 190 may also include a natural language understanding (NLU) module (not shown) that performs semantic interpretation on the transcription of the target utterance 12 to identify a query / command for the device 110. Thus, the output 182 from the backend speech system 180 may include a transcription and / or instructions to fulfill the query / command identified by the NLU module.

[0031] The backend speech system 180 may additionally or alternatively include a hot word detection model (not shown) that is configured to detect whether the enhanced input speech features 250 include the presence of one or more hot words / warm words that the hot word detection model is trained to detect. For example, the hot word detection model may output a hot word detection score that indicates the likelihood that the enhanced input speech features 250 corresponding to the target utterance 12 include a particular hot word / warm word. Detection of a hot word may trigger a wake-up process that wakes the device 110 from a sleep state. For example, the device 110 may wake up and process the hot word and / or one or more terms before / after the hot word.

[0032] In an additional example, the background speech system 180 includes an audio or audio-video call application (e.g., a video conferencing application). Here, the enhanced input speech features 250 corresponding to the target utterance 12 are used by the audio or audio-video call application to filter the speech of the target speaker 10 for transmission to the recipient during the audio or audio-video communication session. The background speech system 180 may additionally or alternatively include a speaker recognition model that is configured to perform speaker recognition using the enhanced input speech features 250 to identify the user 10 who spoke the target utterance 12.

[0033] In the example shown, the device 110 captures a noisy audio signal 202 (also referred to as audio data) of a target utterance 12 spoken by the user 10 in the presence of background interference from one or more sources other than the user 10. The device 110 may correspond to any computing device associated with the user 10 and capable of receiving the noisy audio signal 202. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, and Internet of Things (IoT) devices, smart speakers, etc. The device 110 includes data processing hardware 112 and memory hardware 114 that communicates with the data processing hardware 112 and stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations. The context front-end processing model 200 may be executed on the data processing hardware 112. In some examples, the back-end speech system 180 is executed on the data processing hardware 112.

[0034] In some examples, the device 110 includes one or more applications (i.e., software applications), each of which can utilize the enhanced input speech features 250 generated by the context front-end processing model 200 to perform various functions within the application. For example, the device 110 includes an assistant application that is configured to transmit the synthesized playback audio 154 to the user 10 to assist the user 10 in completing various tasks.

[0035] The device 110 also includes an audio subsystem having an audio capture device (e.g., a microphone) 116 for capturing spoken speech 12 within a speech environment and converting the spoken speech into electrical signals; and a speech output device (e.g., a speaker) 118 for transmitting an audible audio signal (e.g., a synthesized playback signal 154 from the device 110). Although in the illustrated example, the device 110 implements a single audio capture device 116, the device 110 may implement an array of audio capture devices 116 without departing from the scope of the present disclosure, wherein one or more of the audio capture devices 116 in the array may not physically reside on the device 110, but may communicate with the audio subsystem (e.g., a peripheral device of the device 110). For example, the device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones positioned throughout the vehicle.

[0036] In some examples, the device 110 is configured to communicate with the remote system 130 via a network (not shown). The remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). The device 110 can utilize the remote resources 132 to perform various functions related to speech processing and / or synthesized playback communication. The context front-end processing model 200 and the back-end speech system 180 may reside on the device 110 (referred to as an on-device system), or reside remotely (e.g., reside on the remote system 130) but communicate with the device 110. In some examples, one or more back-end speech systems 180 reside locally or on the device, while one or more other back-end speech systems 180 reside remotely. In other words, one or more back-end speech systems 180 that utilize the enhanced input speech features 250 output from the context front-end processing model 200 can be local or remote in any combination. For example, when a system 180 is substantial in size or processing requirements, the system 180 may reside in a remote system 130. However, when the device 110 can support the size or processing requirements of one or more systems 180, the one or more systems 180 may reside on the device 110 using the data processing hardware 112 and / or the memory hardware 114. Alternatively, one or more of the systems 180 may reside both locally / on the device and remotely. For example, when a connection between the device 110 and the remote system 130 is available, the backend voice system 180 may execute on the remote system 130 by default, but when the connection is lost or unavailable, the system 180 executes locally on the device 110 instead.

[0037] In some implementations, the device 110 or a system associated with the device 110 identifies text that the device 110 will transmit to the user 10 as a response to a query spoken by the user 10. The device 110 can then use a text-to-speech (TTS) system to convert the text into corresponding synthesized playback audio 154 for the device 110 to transmit to the user 10 as a response to the query (e.g., audibly transmit to the user 10). Once generated, the TTS system transmits the synthesized playback audio 154 to the device 110 to allow the device 110 to output the synthesized playback audio 154. For example, in response to the user 10 providing a spoken query for today's weather forecast, the device 110 outputs the synthesized playback audio 154 of "today is sunny" at the speaker 118 of the device 110.

[0038] Continue to refer Figure 1When the device 110 outputs the synthesized playback audio 154, the synthesized playback audio 154 generates an echo 156 captured by the audio capture device 116. The synthesized playback audio 154 corresponds to the reference audio signal. Figure 1 , but the reference audio signal may include other types of played audio 154, including media content output from the speaker 118 or communications from a remote user with whom the user 10 is talking through the device 110 (e.g., an IP voice call or a video conference call). Unfortunately, in addition to the echo 156, the audio capture device 116 may also simultaneously capture the target utterance 12 spoken by the user 10, which includes a follow-up query asking more about the weather by stating "what about tomorrow?" For example, Figure 1 It is depicted that when the device 110 outputs the synthesized playback audio 154, the user 10 asks more about the weather by stating "what about tomorrow?" in the spoken utterance 12 to the device 110. Here, both the spoken utterance 12 and the echo 156 are captured at the same time at the audio capture device 116 to form a noisy audio signal 202. In other words, the audio signal 202 includes overlapping audio signals, wherein certain portions of the target utterance 12 spoken by the user 10 overlap certain portions of the reference audio signal (e.g., synthesized playback audio) 154 output from the speaker 118 of the device 110. In addition to the synthesized playback audio 154, a competing speech 13 spoken by another user 11 in the environment may also be captured by the audio capture device 116 and cause background interference that overlaps with the target utterance 12.

[0039] exist Figure 1 In the example, the backend speech system 180 may have problems processing the target utterance 12 corresponding to the subsequent weather query "what about tomorrow?" in the noisy audio signal 202 due to the presence of background interference due to at least one of the played audio 154, the competing speech 13, or the non-speech background noise interfering with the target utterance 12. The context front-end processing model 200 is adopted to improve the robustness of the backend speech system 180 by jointly implementing the acoustic echo cancellation (AEC), speech enhancement, and speech separation models / modules into a single model.

[0040] To perform acoustic echo cancellation (AEC), the single model 200 uses the reference signal 154 being played by the device as an input to the model 200. It is assumed that the reference signal 154 is aligned in time with the target speech 12 and has the same length. In some examples, a feature extractor (not shown) extracts reference features 214 corresponding to the reference audio signal 154. The reference features 214 may include logarithmic Mel filter bank energy (LFBE) features of the reference audio signal 154. Similarly, the feature extractor may extract input speech features 212 corresponding to the target speech 12. The input speech features 212 may include LFBE features. As described in more detail below, the input speech features 212 may be stacked with the reference features 214 and provided as input to the main encoder 210 ( Figure 2 ) to perform AEC. When no reference audio signal 154 is being played by the device, an all-zero reference signal may be used so that only the input speech features 212 are received as input to the main encoder 210.

[0041] The single model 200 can additionally perform speech enhancement in parallel with AEC by applying noise context modeling, wherein the single model 200 processes a context noise signal 213 associated with a noise segment of a predetermined duration captured by the audio capture device 116 prior to the target utterance 12 spoken by the user 10. In some examples, the predetermined duration includes a noise segment of six (6) seconds. Thus, the context noise signal 213 provides a noise context. In some examples, the context noise signal 213 includes LFBE features of the noise context signal to serve as context information.

[0042] Optionally, a single model 200 can additionally perform target speaker modeling for speech separation in conjunction with AEC and speech enhancement. Here, a speaker embedding 215 is received as input by a single model 200. The speaker embedding 215 can include voice characteristics of the target speaker 10 who speaks the target utterance 12. The speaker embedding 215 can include a d-vector. In some examples, the speaker embedding 215 is calculated using a text-independent speaker identification (TI-SID) model trained with a general end-to-end extended set softmax loss. TI-SID can include three long short-term memory (LSTM) layers with 768 nodes and a projection size of 256. The output of the final frame of the last LSTM layer is then linearly converted to a final 256-dimensional d-vector.

[0043] For training and evaluation, each target utterance can be paired with a separate "enrollment" utterance from the same speaker. The enrollment utterance can be randomly selected from the pool of available utterances of the target speaker. The d-vector is then computed on the enrollment utterance. For most practical applications, the enrollment utterances are typically obtained via a separate offline process.

[0044] Figure 2 Shows Figure 1 Context front-end processing model 200. The context front-end processing model 200 uses a modified version of the conformer neural network architecture that combines convolution and self-attention to model short-range and long-range interactions. Model 200 includes a main encoder 210, a noise context encoder 220, a cross-attention encoder 400, and a decoder 240. The main encoder 210 may include N modulated conformer blocks. The noise context encoder 220 may include N conformer blocks. The cross-attention encoder 230 may include M modulated cross-attention conformer blocks. The main encoder 210 and the noise context encoder 220 may be executed in parallel. As used herein, each conformer block may use local causal self-attention to allow streaming functionality.

[0045] The primary encoder 210 may be configured to receive as input an input speech feature 212 corresponding to the target utterance and to generate as output a primary input encoding 218. When the reference audio signal 154 is available, the primary encoder 210 is configured to receive as input the input speech feature 212 stacked with a reference feature 214 corresponding to the reference audio signal and to generate the primary input encoding by processing the input speech feature 212 stacked with the reference feature 214. The input speech feature and the reference feature may each include a corresponding LFBE feature sequence.

[0046] The primary encoder 210 may also be configured to receive as input a speaker embedding 215 (i.e., when available) including voice characteristics of a target speaker (i.e., user) 10 speaking the target utterance 12, and to decode the target utterance 12 using a feature linear modulation (FiLM) layer 310 ( Figure 3 ) combines the input speech features 212 (or the input speech features stacked with the reference features 214) to generate a primary input encoding 218 as output. Figure 3An example modulated conformer block 320 employed by the primary encoder 210 is provided. Here, before each conformer block 320 at the primary encoder 210, a speaker embedding 215 (e.g., a d-vector) is combined with an input speech feature 212 (or a stack of input speech and reference features 214) using a FiLM layer 310 to generate an output 312. FiLM permits the primary encoder 210 to adjust its encoding based on the speaker embedding 215 of the target speaker 10. A residual connection 314 is added after the FiLM layer 310 to combine the input speech feature 212 (or the input speech feature 212 stacked with the reference feature 214) with the output 312 of the FiLM layer 310 to generate a modulated input feature 316 as an input to the conformer block 320 to ensure that the architecture can perform well when the speaker embedding 215 is not present. Mathematically, the modulated conformer block 320 transforms the input feature x using the modulation feature m to produce the output feature y as follows:

[0047] here, and is an affine transformation. FFN, Conv, and MHSA represent feed-forward module, convolution module, and multi-head self-attention module, respectively. Equation 1 shows a feature linear modulation (FiLM) layer 310 with residual connection.

[0048] Return to reference Figure 2 , the noise context encoder 220 is configured to receive as input a context noise signal 213 including noise before the target utterance, and to generate a context noise encoding 222 as output. The context noise signal 213 may include LFBE features of the context noise signal. Unlike the main encoder 210 and the cross-attention encoder 400, the noise context encoder 220 includes a standard conformer block that is not modulated by the speaker embedding 215. The noise context encoder 220 does not modulate the context noise signal 213 with the speaker embedding 215 because the context noise signal 213 is associated with the acoustic noise context before the target utterance 12 is spoken, and is therefore assumed to contain information that should be passed forward to the cross-attention encoder 400 to assist in noise suppression.

[0049] Continue to refer Figure 2, the cross-attention encoder 400 can be configured to receive as input the main input encoding 218 generated as output from the main encoder 210 and the context noise encoding 222 generated as output from the noise context encoder 220, and generate a cross-attention embedding 480 as output. Thereafter, the decoder 240 is configured to decode the cross-attention embedding 480 into an enhanced input speech feature 250 corresponding to the target utterance 12. The context noise encoding 222 can correspond to the auxiliary input. The decoder 240 may include a simple projection decoder having a single-layer, frame-by-frame fully connected network with a sigmoid activation.

[0050] like Figure 4 As shown, the cross-attention encoder 400 can use a corresponding set of M modulated conformer blocks, each of which receives Figure 3 The main input encoding 218 modulated by the speaker embedding 215 using FiLM and the context noise encoding 222 output from the noise context encoder 220 as described in . The cross-attention encoder 400 first uses the semi-feedforward network 402, the first residual connection 404, the convolution block 406 and the second residual connection 408 to independently process the modulated input 218 and the auxiliary input 222. Specifically, the modulated input 218 is processed by the semi-feedforward network 402a, which generates an output 403a. Next, the first residual connection 404a combines the modulated input 218 with the output 403a of the semi-feedforward network 402a to generate a modulated input feature 405a. The modulated input feature 405a is input to the convolution block 406a, which generates a convolution output 407a. The second residual connection 408a combines the convolution output 407a of the convolution block 406a with the modulated input features 405a to generate an output including a query vector 409a.

[0051] Similarly, the auxiliary input 222 is processed by the semi-feedforward network 402b, which generates an output 403b. Next, the first residual connection 404b combines the auxiliary input 222 with the output 403b of the semi-feedforward network 402b to generate a modulated input feature 405b. The modulated input feature 405b is input to the convolution block 406b, which generates a convolution output 407b. The second residual connection 408b combines the convolution output 407b of the convolution block 406b with the modulated input feature 405b to generate an output including a first key vector 409b and a first value vector 409c.

[0052] Subsequently, a multi-head cross attention (MHCA) module 410 receives the query vector 409a, the first key vector 409b, and the first value vector 409c as input, and summarizes these vectors 409a-c to generate a noise summary 412. Intuitively, the role of the MHCA module 410 is to summarize the noise environment separately for each input frame to be enhanced. The noise summary 412 output by the MHCA module 410 is then merged with the query vector 409a using a FiLM layer 420, which generates a FiLM output 422.

[0053] The multi-head self-attention (MHSA) layer 430 receives the FiLM output 422 as input and merges the FiLM output 422 with the query vector 409a to generate an attention output 432. The third residual connection 434 receives the query vector 409a and the attention output 432 and combines the query vector 409a and the attention output 432 to generate a residual output 436. Then, the feedforward module 440 receives the residual output 436 of the third residual connection 434 as input and generates a feature output 442. Next, the fourth residual connection 444 combines the feature output 422 with the residual output 436 of the third residual output 434 to generate a merged input feature 446. Then, the merged input feature 446 is processed as input by the layer normalization (layernorm) 450 and transmitted to the convolution block 406b, which generates a cross-attention embedding 480.

[0054] Mathematically, if x, m, and n are the encoded input from the previous layer, the d vector, and the encoded noise context, the criss-cross attention encoder 400 performs the following:

[0055] The crisscross attention encoder 400 generates a crisscross attention embedding 480 as an output, which is passed to the next layer of M modulated conformer blocks along with the d vector m and the encoded noise context n. Thus, the input is modulated by each of the M conformer blocks by both the speaker embedding 215 associated with the target speaker and the noise context encoding 222.

[0056] Figure 5 An example training process 500 is shown for training the context front-end processing model 200 to generate enhanced input speech features 250 when one or more of the context input features 213, 214, 215 are not present. The training process 500 may be performed in Figure 1530a-c. As shown, the training process obtains one or more training data sets 520 stored in the data storage 510, and trains the context front-end processing model 200 on the training data sets 520. The data storage 510 may reside on the memory hardware 136 of the remote system 130. Each training data set 520 includes a plurality of training examples 530, 530a-n, wherein each training example 530 may include a training utterance 532 paired with a corresponding training context signal 534, 534a-c. Specifically, the training context signal 534 includes: a training context noise signal 534a, which includes noise before the corresponding training utterance 532; a training reference audio signal 534b; and a training speaker vector 534c, which includes the voice characteristics of the target speaker who speaks the corresponding training utterance 532.

[0057] As mentioned above about Figure 1 As discussed, during inference, the context front end processing model 200 may not receive all context input features 213, 214, 215 simultaneously. Training the context front end processing model 200 with one or more missing training context signals 534 encourages the context front end processing model 200 to utilize alternatives among the context input features 213, 214, 215, rather than overly relying on the most relevant context input features among the context input features 213, 214, 215. Therefore, when one or more of the context input features 213, 214, 215 are not present, the context front end processing model 200 can accurately predict the enhanced input speech features 250. In order for the context front end processing model 200 to remain static, any missing training context signals 534 still need to be input to the context front end processing model 200 in some way.

[0058] The training process 500 may also utilize a signal backoff model 550. The signal backoff model 550 receives the training context signals 534 as input from the data store 510 and uses a context signal backoff strategy to back off one or more of the training context signals 534 before training the context front-end processing model 200. The context signal backoff strategy of the signal backoff model 550 may include a predetermined probability (e.g., 50%, 20%, etc.) of backing off each of the training context signals 534, where the same predetermined probability is used for each of the training context signals 534. In other words, in a given training example 530, the signal backoff model 550 may use the context signal backoff strategy to back off the training context noise signal 534a with a predetermined probability of 50%, back off the training reference audio signal 534b with a predetermined probability of 50%, and back off the training speaker vector 534c with a predetermined probability of 50%. Similarly, in a given training example, the signal back-off model 550 can use the context signal back-off strategy to back-off the training context noise signal 534a with a predetermined probability of 20%, back-off the training reference audio signal 534b with a predetermined probability of 20%, and back-off the training speaker vector 534c with a predetermined probability of 20%.

[0059] In addition to the signal backoff strategy, the signal backoff model 550 can also trim the length of the training environment noise signal 534a to include noise with a uniformly distributed length of zero to six (0-6) seconds before the corresponding training speech 532. In other words, the signal backoff model 550 implements the signal backoff strategy and trims the training environment noise signal 534a at the same time. For example, for a given training example 530, even if the signal backoff model 550 does not back off the training environment noise signal 534a, the signal backoff model 550 can still trim the length of the training environment noise signal 534a.

[0060] In some implementations, the signal backoff model 550 uses a signal backoff strategy to backoff each training context signal 534 based on a predetermined probability by replacing the corresponding training context signal 534 with all zeros. In these implementations, the signal backoff model 550 can replace the training context noise signal 534a with an all-zero feature having a predetermined length and the same feature dimension as the corresponding training utterance 532. For example, the signal backoff strategy includes creating an all-zero feature having a length of six (6) seconds and the same dimension as the LFBE feature. Similarly, the signal backoff model 550 using the signal backoff strategy can replace the training reference audio signal 534b with an all-zero feature having the same length and feature dimension as the corresponding training utterance 532. Here, if the signal backoff strategy does not backoff the training reference audio signal 534b, the feature dimension of the all-zero training reference audio signal 534b corresponds to the LFBE feature of the training reference audio signal 534b. Similarly, the signal backoff model 550 using the signal backoff strategy can replace the training speaker vector 534c with an all-zero feature having an all-zero vector. Here, the training speaker vector 534c is replaced with a 256-dimensional all-zero vector. In some implementations, the signal backoff model 550 uses a signal backoff strategy to backoff each training context signal 534 by replacing the corresponding training context signal 534 with a frame-level learned representation based on a predetermined probability.

[0061] exist Figure 5 In the example shown, the signal back-off model 550 receives the training context signals 534a-c as input and backs off the training reference audio signal 534b by replacing the training reference audio signal 534b with all-zero features of the same length and feature dimension as the corresponding training utterance 532, using a predetermined probability of the context signal back-off strategy. In other words, at this time step, the context front-end processing model 200 is trained only on the training context signals 534a, 534c, which approximates the situation that the model 200 may encounter during inference, where the context input features 213, 214, 215 only include the context noise signal 213 and the speaker embedding 215.

[0062] After the signal-temporal-retirement model 550 temporarily retires the training reference audio signal 534b, the training utterance 532 and the training context signal 534 are provided to train the context front-end processing model 200, the training context signal 534 including replacing the training reference audio signal 534b with all zero features and dimensions to simulate the training reference audio signal 534b as missing. The context front-end processing model 200 receives the training utterance 532 and the training context signal 534 simulating the training reference audio signal 534b as missing as input, and generates an output prediction y r Output prediction y rThe output predictions are tested for their accuracy, including the enhanced input speech features 250. At each time step during the training process 500, the previous time step y r–1 The output predictions of are used to additionally train the context front-end processing model 200.

[0063] Figure 6 An example training process 600 for calculating an ASR loss 640 when the context front-end processing model 200 is jointly trained with the ASR model 192 is shown. Here, only the encoder 620 of the ASR model 192 is used to calculate the loss. The ASR loss 640 is calculated as the l2 distance between the output of the ASR encoder 620 for the target features 540 of the training utterance 532 and the enhanced input speech features 250. The ASR encoder 620 is not updated during the training process 600. Specifically, the training process 600 calculates the ASR loss 640 by: generating a prediction output 622 of the ASR encoder 620 for the enhanced input speech features 250 using the ASR encoder 620 of the ASR model 192, the ASR encoder 620 of the ASR model 192 being configured to receive as input the enhanced input speech features 250 predicted by the context front-end processing model 200 for the training utterance 532; and generating a target output 624 of the ASR encoder 620 for the target speech features 540 using the ASR encoder 620, the ASR encoder 620 being configured to receive as input the target speech features 540 of the training utterance 532. The prediction output 622 of the enhanced input speech features 250 and the target output 624 of the target speech features 540 may each include a corresponding LFBE feature sequence. Thereafter, the training process 600 calculates an ASR loss 640 via a loss module 630 based on the predicted output 622 of the ASR encoder 620 for the enhanced input speech features 250 and the target output 624 of the ASR encoder 620 for the target speech features 540. The goal of using the ASR loss 640 is to enhance the context front end processing model 200 to be more compatible with the ASR model 192, which is critical to obtaining the best performance from the context front end processing model 200. By keeping the parameters of the ASR model 192 fixed, the ASR model 192 is separated from the context front end processing model 200, allowing each to be trained and deployed independently of the other.

[0064] In some implementations, the context front-end processing model 200 is jointly trained with the ASR model 192 of the back-end automatic speech recognition system 180 using spectral loss and ASR loss 640. The training target 540 for training the context front-end processing model 200 uses ideal ratio masking (IRM). Based on the assumption that speech and noise are uncorrelated in Mel-spectrogram space, the IRM is calculated using reverberant speech and reverberant noise as follows. Here, X and N are the reverberant speech and reverberant noise mel spectrograms, respectively. t and c represent the time and mel frequency bin indices. The choice of estimated IRM is based on the target being bounded between [0, 1], thereby simplifying the estimation process. In addition, the ASR model used for evaluation can be trained on real and simulated reverberant data, thereby obtaining a trained ASR model that is relatively robust to reverberant speech. Therefore, the IRM derived using reverberant speech as the target still provides a significant gain in performance. The following is based on the IRM and the estimated IRM The L1 and L2 losses between are used to calculate the spectral loss during training.

[0065] During inference, the estimated IRM is scaled and floored to reduce speech distortion at the expense of reduced noise suppression. This is particularly important because the ASR model 192 is sensitive to speech distortion and nonlinear front-end processing, which is one of the main challenges in improving the performance of robust ASR models using enhanced front-ends. The enhanced feature derivation is as follows. Here, Y is the noise mel spectrogram, is an estimate of the clean Mel spectrogram, α and β are the exponential masking scalar and the masking lower limit. In some examples, α is set to 0.5 and β is set to 0.01. The enhanced features can be logarithmically compressed, i.e. , and is passed to the ASR model 192 for evaluation.

[0066] Figure 7 A flowchart including an example arrangement of operations of a method 700 for training a general automatic speech recognition model using a context front-end processing model 200. At operation 702, the method 700 includes receiving a plurality of training utterances 532 paired with corresponding training context signals 534, 534a-c. The training context signals 534 include: a training context noise signal 534a including noise preceding the corresponding training utterances 532; a training reference audio signal 534b; and a training speaker vector 534c including voice characteristics of a target speaker who spoke the corresponding training utterance 532. The method 700 also includes: at operation 704, training the context front-end processing model 200 on the training utterances 532 using a context signal back-off strategy to learn how to predict enhanced input speech features 250. Here, the context signal back-off strategy uses a predetermined probability to back-off each of the training context signals 534 during training of the context front-end processing model 200 to simulate one or more of the training context signals 534 as missing to teach the model 200 to learn how to robustly generate enhanced speech features 250 when any of the corresponding context input features are missing during inference.

[0067] Figure 8 800 is a schematic diagram of an example computing device 800 that can be used to implement the systems and methods described in this document. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the disclosed implementations described and / or claimed in this document.

[0068] The computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of the components 810, 820, 830, 840, 850, and 860 are interconnected using various buses and may be mounted on a common motherboard or in other ways as appropriate. The processor 810 (e.g., Figure 1 The data processing hardware 112, 134) can process instructions for execution within the computing device 800 (including instructions stored in the memory 820 or on the storage device 830) to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 880 coupled to the high-speed interface 840. In other implementations, multiple processors and / or multiple buses as well as multiple memories and multiple types of memories can be used as appropriate. Moreover, multiple computing devices 800 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0069] Memory 820 (e.g., Figure 1 The memory hardware 114, 136) stores information non-temporarily within the computing device 800. The memory 820 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 820 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0070] The storage device 830 is capable of providing mass storage for the computing device 800. In some implementations, the storage device 830 is a computer-readable medium. In various implementations, the storage device 830 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 820, a storage device 830, or a memory on a processor 810.

[0071] The high-speed controller 840 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 860 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 840 is coupled to a memory 820, a display 880 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 850 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 860 is coupled to a storage device 830 and a low-speed expansion port 890. The low-speed expansion port 890, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0072] As shown, computing device 800 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 800a or multiple implementations in a group of such servers 800a, implemented as a laptop computer 800b, or implemented as part of a rack server system 800c.

[0073] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0074] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0075] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0076] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0077] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0078] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0079] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (700) which, when executed on data processing hardware (134), causes the data processing hardware (134) to perform operations comprising: Receiving a plurality of training utterances (532) paired with corresponding training context signals (534, 534a-c), the training context signals (534a-c) comprising: A training environment noise signal (534a), which includes noise preceding the corresponding training utterance (532); training reference audio signal (543b); and a training speaker vector (534c) comprising voice characteristics of a target speaker who spoke the corresponding training utterance (532); and A context front-end processing model (200) is trained on the training utterance (532) to learn how to predict enhanced speech features (250) using a context signal back-off strategy, wherein the context signal back-off strategy uses a predetermined probability to back-off each of the training context signals (534) during training of the context front-end processing model (200).

2. The computer-implemented method (700) of claim 1, wherein: The signal muting strategy mutes each training context signal (534) by replacing the corresponding training context signal (534) with all zeros.

3. The computer-implemented method (700) of claim 2, wherein: Replacing the training reference audio signal (543b) with all zeros includes replacing the training reference audio signal (543b) with all zero features of the same length and feature dimension as the corresponding training utterance (532).

4. The computer-implemented method (700) of claim 2 or 3, wherein: Replacing the training context noise signal (534a) includes replacing the training context noise signal (534a) with all-zero features having a predetermined length and the same feature dimension as the corresponding training utterance (532).

5. The computer-implemented method (700) of any one of claims 2 to 4, wherein: Replacing the training speaker vector (534c) includes replacing the training speaker vector (534c) with an all-zero feature having an all-zero vector.

6. The computer-implemented method (700) of claim 1, wherein: The signal backoff strategy backs off each training context signal (534) by replacing the corresponding training context signal (534) with a frame-level learned representation.

7. The computer-implemented method (700) of any one of claims 1 to 6, wherein: The trained context front-end processing model (200) comprises: A main encoder (210) is configured to: receiving as input input speech features (212) corresponding to a target utterance (12); and generating as output the primary input encoding (218); A noise context encoder (220) configured to: receiving as input an ambient noise signal (213) including noise preceding the target utterance (12); and generating as output a context noise code (222); and A criss-cross attention encoder (400), which is configured to: receiving as input the primary input code (218) generated as output from the primary encoder (210) and the context noise code (222) generated as output from the noise context encoder (220); and generating a criss-cross attention embedding (480) as output; and A decoder (240) configured to decode the cross-attention embedding (480) into enhanced speech features (250) corresponding to the target utterance (12).

8. The computer-implemented method (700) of claim 7, wherein: The main encoder (210) is further configured to: receiving as input a reference feature (214) corresponding to a reference audio signal (154); and The primary input encoding (218) is generated as an output by processing the input speech features (212) stacked with the reference features (214).

9. The computer-implemented method (700) of claim 7 or 8, wherein: The main encoder (210) is further configured to: receiving as input a speaker embedding (215) comprising speech characteristics of a target speaker (10) speaking the target utterance (12); and The primary input encoding (218) is generated as output by combining the input speech features (212) with the speaker embedding (215) using a feature linear modulation (FiLM).

10. The computer-implemented method (700) of any one of claims 7 to 9, wherein: The criss-cross attention encoder (400) is further configured to: receiving as input the primary input encoding (218) modulated using a feature linear modulation (FiLM) by a speaker embedding (215), the speaker embedding (215) comprising voice characteristics of a target speaker (10) speaking the target utterance (12); and The primary input encoding (218) modulated by the speaker embedding (215) and the context noise encoding (222) is processed to generate the cross-attention embedding (480) as an output.

11. The computer-implemented method (700) of any one of claims 7 to 10, wherein: The main encoder (210) includes N modulated conformer blocks (320); The noise context encoder (220) includes N conformer blocks and is executed in parallel with the main encoder (210); and The criss-cross attention encoder (400) comprises M modulated criss-cross attention conformer blocks.

12. The computer-implemented method (700) of any one of claims 1 to 11, wherein: The context front-end processing model (200) is jointly trained with the back-end automatic speech recognition (ASR) model (192) using spectral loss and ASR loss (640).

13. The computer-implemented method (700) of claim 12, wherein: The spectral loss is based on L1 loss function and L2 loss function distances between an estimated ratio mask and an ideal ratio mask computed using reverberant speech and reverberant noise.

14. The computer-implemented method (700) of claim 12 or 13, wherein: For each training utterance (532), the ASR loss (640) is calculated as follows: Generate a prediction output (622) of the ASR encoder (620) for the enhanced speech feature (250) using an ASR encoder (620) of the ASR model (192), wherein the ASR encoder (620) of the ASR model (192) is configured to receive as input the enhanced speech feature (250) predicted by the context front-end processing model (200) for the training utterance (532) using the context signal backoff strategy; generating a target output (624) of the ASR encoder (620) for the target speech feature (540) using the ASR encoder (620), the ASR encoder (620) being configured to receive the target speech feature (540) of the training utterance (532) as input; as well as The ASR loss (640) is calculated based on the predicted output (622) of the ASR encoder (620) for the enhanced speech feature (250) and the target output (624) of the ASR encoder (620) for the target speech feature (540).

15. A system (100), comprising: Data processing hardware (134); as well as Memory hardware (136) in communication with the data processing hardware (134) and storing instructions that, when executed on the data processing hardware (134), cause the data processing hardware (134) to perform operations including: Receiving a plurality of training utterances (532) paired with corresponding training context signals (534, 534a-c), the training context signals (534a-c) comprising: A training environment noise signal (534a), which includes noise preceding the corresponding training utterance (532); training reference audio signal (543b); and a training speaker vector (534c) comprising voice characteristics of a target speaker who spoke the corresponding training utterance (532); and A context front-end processing model (200) is trained on the training utterance (532) to learn how to predict enhanced speech features (250) using a context signal back-off strategy, wherein the context signal back-off strategy uses a predetermined probability to back-off each of the training context signals (534) during training of the context front-end processing model (200).

16. The system (100) of claim 15, wherein: The signal muting strategy mutes each training context signal (534) by replacing the corresponding training context signal (534) with all zeros.

17. The system (100) of claim 16, wherein: Replacing the training reference audio signal (543b) with all zeros includes replacing the training reference audio signal (543b) with all zero features of the same length and feature dimension as the corresponding training utterance (532).

18. The system (100) of claim 16 or 17, wherein: Replacing the training context noise signal (534a) includes replacing the training context noise signal (534a) with all-zero features having a predetermined length and the same feature dimension as the corresponding training utterance (532).

19. The system (100) according to any one of claims 16 to 18, wherein: Replacing the training speaker vector (534c) includes replacing the training speaker vector (534c) with an all-zero feature having an all-zero vector.

20. The system (100) of claim 15, wherein: The signal backoff strategy backs off each training context signal (534) by replacing the corresponding training context signal (534) with a frame-level learned representation.

21. The system (100) according to any one of claims 15 to 20, wherein: The trained context front-end processing model (200) comprises: A main encoder (210) is configured to: receiving as input input speech features (212) corresponding to a target utterance (12); and generating as output the primary input encoding (218); A noise context encoder (220) configured to: receiving as input an ambient noise signal (213) including noise preceding the target utterance (12); and generating as output a context noise code (222); and A criss-cross attention encoder (400), which is configured to: receiving as input the primary input code (218) generated as output from the primary encoder (210) and the context noise code (222) generated as output from the noise context encoder (220); and generating a criss-cross attention embedding (480) as output; and A decoder (240) configured to decode the cross-attention embedding (480) into enhanced speech features (250) corresponding to the target utterance (12).

22. The system (100) of claim 21, wherein: The main encoder (210) is further configured to: receiving as input a reference feature (214) corresponding to a reference audio signal (154); and The primary input encoding (218) is generated as an output by processing the input speech features (212) stacked with the reference features (214).

23. The system (100) of claim 21 or 22, wherein: The main encoder (210) is further configured to: receiving as input a speaker embedding (215) comprising speech characteristics of a target speaker (10) speaking the target utterance (12); and The primary input encoding (218) is generated as output by combining the input speech features (212) with the speaker embedding (215) using a feature linear modulation (FiLM).

24. The system (100) of any one of claims 21 to 23, wherein: The criss-cross attention encoder (400) is further configured to: receiving as input the primary input encoding (218) modulated using a feature linear modulation (FiLM) by a speaker embedding (215), the speaker embedding (215) comprising voice characteristics of a target speaker (10) speaking the target utterance (12); and The primary input encoding (218) modulated by the speaker embedding (215) and the context noise encoding (222) is processed to generate the cross-attention embedding (480) as an output.

25. The system (100) of any one of claims 21 to 24, wherein: The main encoder (210) includes N modulated conformer blocks (320); The noise context encoder (220) includes N conformer blocks and is executed in parallel with the main encoder (210); and The criss-cross attention encoder (400) comprises M modulated criss-cross attention conformer blocks.

26. The system (100) of any one of claims 15 to 25, wherein: The context front-end processing model (200) is jointly trained with the back-end automatic speech recognition (ASR) model (192) using spectrum loss and ASR loss.

27. The system (100) of claim 26, wherein: The spectral loss is based on L1 loss function and L2 loss function distances between an estimated ratio mask and an ideal ratio mask computed using reverberant speech and reverberant noise.

28. The system (100) of claim 26 or 27, wherein: For each training utterance (532), the ASR loss is calculated as follows: Generate a prediction output (622) of the ASR encoder (620) for the enhanced speech feature (250) using an ASR encoder (620) of the ASR model (192), wherein the ASR encoder (620) of the ASR model (192) is configured to receive as input the enhanced speech feature (250) predicted by the context front-end processing model (200) for the training utterance (532) using the context signal backoff strategy; generating a target output (624) of the ASR encoder (620) for the target speech feature (540) using the ASR encoder (620), the ASR encoder (620) being configured to receive the target speech feature (540) of the training utterance (532) as input; as well as The ASR loss (640) is calculated based on the predicted output (622) of the ASR encoder (620) for the enhanced speech feature (250) and the target output (624) of the ASR encoder (620) for the target speech feature (540).