Sub-model for neural context biasing with attention and embedding space

Abstract

A method (700) for training a sub-model (215) for context biasing for speech recognition includes obtaining a base speech recognition model (200) trained on unbiased data (510). The method includes obtaining a set of training utterances representing a particular domain, each training utterance in the set of training utterances (560) including audio data (561) characterizing the training utterance and a true value transcription (563) of the training utterance. The method further includes, for each corresponding training utterance in the set of training utterances, determining, using an embedding encoder (515), a corresponding document embedding (567) from a true value transcription of the corresponding training utterance. The method includes training a sub-model using corresponding document embedding determined from true value transcriptions of a training utterance set to bias a base speech recognition model to recognize speech in a particular domain.

Description

Technical Field

[0001] The present disclosure relates to training a sub-model for contextually biasing the results of a base machine learning model using an embedding space. Background Art

[0002] Automatic speech recognition (ASR) is a category of natural language processing (NLP) that involves processing audio containing human speech. ASR models are typically used to recognize spoken language and / or translate spoken language into text. One way to generate an ASR model is to train the model on a large data set by using machine learning. Due to the amount of data used for training and the amount of time spent on training, ASR models are typically generalized for many domains and users, which makes the model inflexible. Attempts to make ASR models more flexible (such as by using multiple smaller models) may be computationally expensive (e.g., due to redundancy in training multiple models) or provide skewed results (e.g., models with less training data will be less robust). Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method for training a sub-model for biasing speech recognition results based on context. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations, the operations comprising obtaining a basic speech recognition model trained on unbiased data. The operations comprise obtaining a set of training utterances representing a specific domain, each training utterance in the training utterance set comprising audio data representing the training utterance and a true value transcription of the training utterance. The operations further comprise, for each corresponding training utterance in the training utterance set, using an embedding encoder to determine a corresponding document embedding from the true value transcription of the corresponding training utterance. The operations comprise training a sub-model using the corresponding document embedding determined from the true value transcription of the training utterance set to bias the basic speech recognition model to recognize speech in a specific domain.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, training a sub-model includes, for each corresponding training utterance in a training utterance set: processing audio data representing the training utterance using a base speech recognition model to generate a predicted speech recognition result, the base speech recognition model being configured to receive a sub-model output of the sub-model based on a corresponding document embedding determined from a true value transcription of the corresponding training utterance; and determining a supervised loss term based on the predicted speech recognition result and the true value transcription of the corresponding training utterance. In these implementations, training the sub-model additionally includes, for each corresponding training utterance in the training utterance set: updating the parameters of the sub-model based on the supervised loss term to teach the sub-model to learn how to bias the base speech recognition model to recognize speech in a specific domain. In these implementations, the operation may also include projecting a one-hot vector into a phrase set embedding in an embedding space. In these implementations, the sub-model output of the sub-model based on the corresponding document embedding may also be based on a history of predicted speech recognition results generated by the base speech recognition model during one or more previous output steps.

[0005] Parameters of the base speech recognition model may be frozen while the submodel is being trained. In some implementations, the operation further comprises, for at least one training utterance in the training utterance set, converting a corresponding true-value transcription of at least one training utterance using a text-to-speech (TTS) system to generate audio data comprising a corresponding synthetic speech representation of the corresponding at least one training utterance. In these implementations, a true-value transcription of the corresponding at least one training utterance may be generated using a background language model and an in-domain language model trained on transcribed speech utterances associated with a particular domain. In other implementations, the operation further comprises, for at least one training utterance in the training utterance set, applying data enhancement to the audio data representing the at least one training utterance. In these implementations, the data enhancement applied may include at least one of adding noise, adding reverberation, or manipulating timing.

[0006] The submodel may include one or more neural network layers. Alternatively, the submodel may be arranged in a layer of a basic speech recognition model. In some implementations, the basic speech recognition model may include an encoder and a decoder, and the submodel is arranged between two layers of the encoder of the basic speech recognition model.

[0007] In some implementations, the operation also includes, after training the sub-model, deploying the base speech recognition model and the trained sub-model for execution on a user device, the user device being configured to receive a speech recognition request, the speech recognition request including audio data representing an utterance captured in streaming audio. In these implementations, the user device is configured to determine that the speech recognition request includes a context indicator indicating a specific domain. In these implementations, the user device is also configured to use the trained sub-model to bias the base speech recognition model toward a specific domain, and to use the biased base speech recognition model to generate a transcription of the utterance by processing the audio data, the transcription being biased toward one or more terms in the specific domain.

[0008] In other implementations, the operations further include, after training the sub-model, receiving a speech recognition request from a user device in communication with the data processing hardware, the speech recognition request including audio data representing an utterance captured by the user device in streaming audio. In these implementations, the operations include determining that the speech recognition request includes a context indicator indicating a specific domain. In these implementations, the operations further include using the trained sub-model to bias a base speech recognition model toward a specific domain, and using the biased base speech recognition model to generate a transcription of the utterance by processing the audio data, the transcription being biased toward one or more terms in the specific domain.

[0009] Another aspect of the present disclosure provides a system for training a sub-model for biasing speech recognition results based on context. The system includes data processing hardware and memory hardware, and the memory hardware communicates with the data processing hardware. The memory hardware stores instructions, which, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a basic speech recognition model trained on unbiased data. The operations include obtaining a set of training utterances representing a specific domain, each training utterance in the training utterance set including audio data representing the training utterance and a true value transcription of the training utterance. The operations also include, for each corresponding training utterance in the training utterance set, using an embedding encoder to determine a corresponding document embedding from the true value transcription of the corresponding training utterance. The operations include training a sub-model using the corresponding document embedding determined from the true value transcription of the training utterance set to bias the basic speech recognition model to recognize speech in a specific domain.

[0010] This aspect may include one or more of the following optional features. In some implementations, training the submodel includes, for each corresponding training utterance in the training utterance set: processing audio data representing the training utterance using a base speech recognition model to generate a predicted speech recognition result, the base speech recognition model being configured to receive a submodel output of the submodel based on a corresponding document embedding determined from a true value transcription of the corresponding training utterance; and determining a supervised loss term based on the predicted speech recognition result and the true value transcription of the corresponding training utterance. In these implementations, training the submodel additionally includes, for each corresponding training utterance in the training utterance set: updating the parameters of the submodel based on the supervised loss term to teach the submodel to learn how to bias the base speech recognition model to recognize speech in a specific domain. In these implementations, the operation may also include projecting the unique hot vector into a phrase set embedding in the embedding space. In these implementations, the submodel output of the submodel based on the corresponding document embedding may also be based on a history of predicted speech recognition results generated by the base speech recognition model during one or more previous output steps.

[0011] Parameters of the base speech recognition model may be frozen while the submodel is being trained. In some implementations, the operation further comprises, for at least one training utterance in the training utterance set, converting a corresponding true-value transcription of at least one training utterance using a text-to-speech (TTS) system to generate audio data comprising a corresponding synthetic speech representation of the corresponding at least one training utterance. In these implementations, a true-value transcription of the corresponding at least one training utterance may be generated using a background language model and an in-domain language model trained on transcribed speech utterances associated with a particular domain. In other implementations, the operation further comprises, for at least one training utterance in the training utterance set, applying data enhancement to the audio data representing the at least one training utterance. In these implementations, the data enhancement applied may include at least one of adding noise, adding reverberation, or manipulating timing.

[0012] The submodel may include one or more neural network layers. Alternatively, the submodel may be arranged in a layer of a basic speech recognition model. In some implementations, the basic speech recognition model may include an encoder and a decoder, and the submodel is arranged between two layers of the encoder of the basic speech recognition model.

[0013] In some implementations, the operation also includes, after training the sub-model, deploying the base speech recognition model and the trained sub-model for execution on a user device, the user device being configured to receive a speech recognition request, the speech recognition request including audio data representing an utterance captured in streaming audio. In these implementations, the user device is configured to determine that the speech recognition request includes a context indicator indicating a specific domain. In these implementations, the user device is also configured to use the trained sub-model to bias the base speech recognition model toward a specific domain, and to use the biased base speech recognition model to generate a transcription of the utterance by processing the audio data, the transcription being biased toward one or more terms in the specific domain.

[0014] In other implementations, the operations further include, after training the sub-model, receiving a speech recognition request from a user device in communication with the data processing hardware, the speech recognition request including audio data representing an utterance captured by the user device in streaming audio. In these implementations, the operations include determining that the speech recognition request includes a context indicator indicating a specific domain. In these implementations, the operations further include using the trained sub-model to bias a base speech recognition model toward a specific domain, and using the biased base speech recognition model to generate a transcription of the utterance by processing the audio data, the transcription being biased toward one or more terms in the specific domain.

[0015] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of an example system for a context biasing system including an automatic speech recognition (ASR) model.

[0017] Figure 2 is a schematic diagram of an ASR model for using sub-models to produce biased speech recognition results.

[0018] Figure 3A is a schematic diagram of an ASR model for producing biased speech recognition results using a sub-model as a residual adapter layer.

[0019] Figure 3B is a schematic diagram of an ASR model for producing biased speech recognition results using sub-models in the layers of an encoder.

[0020] Figure 4A Figure 1 is a schematic diagram of the ASR model generating unbiased speech recognition results.

[0021] Figure 4Bis a schematic diagram of an ASR model using sub-models to generate biased speech recognition results.

[0022] Figure 5A is a diagram of an example training scheme for an ASR model.

[0023] Figure 5B is a schematic diagram of an example training scheme for a sub-model used to bias speech recognition results output from an ASR model based on context.

[0024] Fig. 6A is a schematic example text-to-speech (TTS) and data augmentation module for generating synthetic speech utterances.

[0025] Figure 6B is a schematic diagram of a comparative unspoken text selection process for selecting unspoken text utterances used to train a sub-model to bias speech recognition results.

[0026] Figure 7 is a flow diagram of an example operational arrangement of a method for context biasing using a sub-model.

[0027] Figure 8 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0028] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0029] Automatic speech recognition (ASR) is a growing field of language processing that has a variety of uses, from automatic translation and transcription of speech to processing voice commands for computing devices. Recently, it has been found that neural networks for machine learning perform well as the basis for ASR systems and models. Using machine learning techniques, ASR models can be trained on large training data sets including audio samples of speech to produce robust models for speech recognition. Generally, these ASR models are large because the more extensively the model is trained, the better the model performs. However, there are disadvantages in using such large models, such as applying a single model to a variety of users with different characteristics. For example, even though English speakers may have many different accents or colloquialisms based on regions, a single ASR model may still be built for the English language. Then, for some user groups, the ASR model may not perform so accurately. In addition, retraining or updating the model is difficult to perform due to size because it is computationally expensive. This may make the ASR model obsolete and perform poorly for new / emerging words / phrases (e.g., slang, new TV shows).

[0030] The inflexibility of large ASR models hinders the potential feasibility of speech recognition because the ASR model may not perform well for part of the user base. In particular, these large ASR models may not be able to utilize contextual signals that can help provide information about the voice. As described above, the user's location can provide information about the accent, or add / eliminate certain words or phrases that inform the ASR model output. In another example, when the alarm rings from a smart device, the user is more likely to say a voice command related to the alarm (e.g., "stop alarm", "cancel", "snooze") than usual. Current ASR models (e.g., large or general models) cannot use this contextual information to affect the output.

[0031] Conventional attempts to "personalize" ASR models based on context are difficult and may cause problems during implementation. One technique for combining contextual information with an ASR model is to use multiple smaller ASR models, each of which is associated with a specific context or domain. However, training multiple smaller ASR models will be computationally expensive, especially because a large amount of training will be redundant between models. Additionally, training multiple models will be time consuming because the process may take weeks to train each model from scratch. Even if all ASR models are built and trained, some ASR models will perform poorly due to a lack of available training data. Managing and implementing a large number of models will also be cumbersome. Another way to consider contextual information is to modify the ASR model to receive bias terms to influence the results. However, this typically involves significant manual intervention (e.g., when building user domain-specific models) and may cause catastrophic forgetting, where the general business of the ASR model may over-trigger towards bias terms when the bias terms are not spoken, or not trigger at all even when the bias terms exist.

[0032] The implementations herein relate to a base ASR model that uses a sub-model to bias the base ASR model based on context so that the result or output of the ASR model is directed to a specific context or domain. The sub-model includes a set of parameters that can be added to or replaced from a general base ASR model. The sub-model can be loaded / enabled / disabled when necessary, thereby allowing the base ASR model to be used without bias, thereby eliminating the catastrophic forgetting problem. In some examples, the base ASR model is trained, and then the parameters of the base ASR model are frozen during operation (i.e., inference). In this way, the base ASR model remains stable and can continue to work for general business or multiple fields. However, when context information is available, the base ASR model can activate the sub-model for context information in order to bias the speech recognition results toward the terms related to the context information. One advantage of these implementations includes the ability to train sub-models separately without having to train or retrain a large base ASR model. In addition, the sub-model can be trained on an embedding space, and context indicators can be projected onto the embedding space to activate parts of the sub-model related to the context. The base ASR model remains unchanged with or without the use of the sub-models, which eliminates the problem that the model will be damaged due to use and / or updating.

[0033] As used herein, and unless otherwise specified, the terms "speech recognition system" and "speech recognition model" may refer to any combination of ASR systems / models in which speech is recognized and processed by a computing device. As will become apparent, the ASR models of the present disclosure and techniques for training ASR and sub-models will enable biasing speech recognition based on contextual information.

[0034] Figure 1A sub-model training and context biasing system 100 is shown, which includes an automatic speech recognition (ASR) model 200, a sub-model 215, and an embedding encoder 565. The ASR model 200 using the sub-model 215 is configured to process a speech recognition request 105. The speech recognition request 105 includes input audio data 102 corresponding to an utterance 108 spoken by a source speaker 104 and captured by a user device 110. The speech recognition request 105 may also include a context indicator 103. Using the audio data 102 and the context indicator 103, the ASR model 200 and the sub-model 215 generate or predict an unbiased speech recognition result 222 or a biased speech recognition result 224. The biased speech recognition result 224 is more likely to include words or phrases associated with the domain toward which the ASR model 200 is biased using the sub-model 215 (e.g., based on the context indicator 103). In some examples, the input audio data 102 includes an input spectrogram corresponding to the utterance 108. The context indicator 103 can provide an indication of a particular domain in a plurality of different domains, each of which indicates or represents a corresponding context of the utterance 108. The submodel 215 can be trained on some or all of the plurality of domains. In some implementations, the speech recognition results 222, 224 include probability density functions 226, 226A-226B representing the density of probabilities of terms recognized from the utterance 108.

[0035] Although not shown, an acoustic front end resident on the user device 110 may convert a time-domain audio waveform of an utterance 108 captured via a microphone of the user device 110 into an input spectrogram 102 or other type or form of audio data 102. Additionally, the front end device may be configured to determine or obtain data representing contextual indicators 103 contributing to the utterance 108 and / or other relevant information corresponding to the source speaker 104 and / or the client device 110.

[0036] A user device 110 associated with a source speaker 104 may capture an utterance 108 spoken by the source speaker 104 and provide the corresponding input audio data 102 to the ASR model 200 as part of a speech recognition request 105. Additionally, the user device 110 may determine that the contextual indicator 103 is to be included with the speech recognition request 105. The user device 110 may include, but is not limited to, a smartphone, a tablet, a desktop / laptop computer, a smart speaker, a smart display, a smart home appliance, an assistant-enabled wearable device (e.g., a smart watch, a smart headset, smart glasses, etc.), or a vehicle infotainment system. Alternatively, the remote server 112 may process the audio data 102 and any other additional data or metadata from the user device 110 to determine the contextual indicator 103.

[0037] The context biasing system 100 may be distributed across multiple devices such that the ASR model 200 resides on one of the user device 110 or a remote system 150 (also referred to herein as a cloud computing environment) that communicates with the user device 110 via a network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic resources 152, including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). The data store 158 (i.e., a remote storage device) may be overlaid on the storage resources 146 to allow scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The ASR model 200 and the sub-model 215 may be executed on the remote system 150 or the user device 110. The sub-model 215 may be stored locally at the user device 110 or on a remote system (e.g., at the data store 158), or some combination therebetween.

[0038] The sub-model training and context biasing system 100 implements a dynamic model including two or more components or models to generate biased speech recognition results 224 based on the received speech recognition request 105, wherein the two or more components or models include a common base model (e.g., ASR model 200) and a sub-model 215. The ASR model 200 can be trained on a large speech data set. Once trained, the ASR model 200 can be frozen so that the parameters of the ASR model 200 remain constant during operation. The ASR model 200 can be updated, retrained, or replaced as needed or when additional training data becomes available.

[0039] The sub-model 215 may be trained using one or more training utterance sets 560, each of which belongs to a particular domain in the plurality of domains. The training utterances 560 may include audio data 561 representing the training utterances 560 and a true value transcription 563 of the training utterances 560 (i.e., an accurate transcription of the audio data 561). The training utterances 560 may be stored, for example, at a data store 158 of a remote system 150. In some implementations, before providing the training utterances 560 to the sub-model 215, an embedding encoder 565 processes the training utterances 560 to generate corresponding document embeddings 567 from the corresponding true value transcriptions 563 of the training utterances 560. The training process for the sub-model 215 is discussed in more detail below ( Figure 5B ).

[0040] In some implementations, a single sub-model 215 is used to bias the ASR model 200 by activating certain parameters based on the contextual indicator 103 of the speech recognition request 105. For example, the contextual indicator 103 can be a one-hot vector indicating a specific domain in multiple domains. In use, the one-hot vector of the contextual indicator 103 can be projected onto the embedding space on which the sub-model 215 is trained, so that the sub-model 215 activates one or more parameters corresponding to the embedding space (i.e., the specific domain indicated by the contextual indicator 103). When the speech recognition request 105 does not have a contextual indicator 103, or when the sub-model 215 is not trained (or insufficiently trained) on the specific domain corresponding to the contextual indicator 103, in some examples, the ASR model 200 generates an unbiased speech recognition result 222. That is, the unbiased speech recognition result 222 is generated only by the ASR model 200 and is not acted upon or influenced by the sub-model 215.

[0041] The contextual indicator 103 may be based on any signal or data that may be used to improve the accuracy of the biased speech recognition results 224. The contextual indicator 103 may be based on information related to the source speaker 104. For example, the source speaker 104 has a particular dialect, native language, mannerisms, speech patterns, speech disfluencies, etc. Thus, the system 100 may train a sub-model 215 corresponding to the source speaker 104, wherein the sub-model 215 biases / personalizes the ASR model 200 to make predictions specifically suited for the source speaker 104. In some implementations, the contextual indicator 103 includes a one-hot vector, and the system uses the contextual indicator 103 to activate a portion (i.e., one or more parameters) of the sub-model 215 (i.e., the portion of the sub-model 215 corresponding to the source speaker 104).

[0042] In some implementations, the contextual indicator 103 is based on information related to the user device 110. For example, the user device 110 may include a smart device equipped with sensors (such as GPS, accelerometers, gyroscopes, microphones, proximity sensors, cameras, etc.). The contextual indicator 103 may indicate a domain related to the user device 110 as inferred from one of the sensors. For example, the contextual indicator 103 may indicate the geographic location of the user device 110 as inferred from GPS data (in the case where the user 104 explicitly agrees to share his / her geographic location, the sharing can be revoked at any time). Here, the contextual indicator 103 may correspond to a larger geographic location (e.g., a city, such as Chicago) or to a more specific location (e.g., a gym). In either instance, the sub-model 215 may bias the biased speech recognition results 224 towards a specific domain based on location. In particular, the contextual indicator 103 identifying the location Chicago can activate a portion of a sub-model 215 trained on data from users in that city, state, and / or region, which can be biased based on the accent or other features of the speech of users from that area and named entities unique to that area (e.g., restaurants, sports teams, street names, etc.). Thus, the sub-model 215 can bias the ASR model 200 to generate biased speech recognition results 224 toward a prediction bias appropriate for that domain. For example, the biased speech recognition results 224 can increase the likelihood that the ASR model 200 predicts that the utterance 108 includes a reference to a restaurant or street in Chicago.

[0043] Similarly, the contextual indicator 103 indicating that the source speaker 104 is in a gym can activate a portion of a sub-model 215 that is trained based on speech from a user who is exercising or in a similar location. Here, the speech may be affected by labored breathing, or involve specific words or phrases (e.g., voice commands for operating a music player on a smart device). Thus, the sub-model 215 can take these contextual elements into account when processing the audio data 102 to bias the base ASR model 200 to produce the biased speech recognition results 224.

[0044] In another example, the contextual indicator 103 may indicate a software application currently executing on the user device 110, such as a music player application. In this example, the contextual indicator 103 activates a portion of the sub-model 215 corresponding to the software application indicated by the contextual indicator 103 (e.g., the music player) in order to bias the ASR model 200 to recognize terms / phrases such as "next song" or "pause". In other examples, the sub-model 215 is generally trained for that type of application or domain. Thus, when the source speaker 104 utters the utterance 108 "pause playback", the ASR model 200 biased by the sub-model 215 will generate a biased speech recognition result 224 that is skewed or biased toward a result related to the music player relative to an unbiased speech recognition result 222 determined without activating the sub-model 215.

[0045] In some implementations, the context indicator 103 indicates that multiple domains apply to the utterance 108. In this scenario, a single sub-model 215 can bias the ASR model 200 to generate biased speech recognition results 224 toward each of the multiple domains. For example, when the source speaker 104 is located in a gym and a music player is executing a user device, the sub-model 215 can bias the output of the ASR model 200 toward each of these domains.

[0046] Output 190 can receive unbiased speech recognition results 222 and biased speech recognition results 224 generated by ASR model 200. In some examples, output 190 includes natural language understanding (NLU) that performs query interpretation on the speech recognition results. NLU can also instruct downstream applications / services to perform actions based on the results. Output 190 can also include a user interface generator that is configured to display the speech recognition results as a transcription on a screen of user device 110 and / or another device.

[0047] Figure 1 The system 100 is presented for illustrative purposes only and is not intended to be limiting. For example, although only a single example of each component is shown, the system 100 may include any number of components 110, 112, 140, 150, 200, 215, and 565. In addition, although some components are described as being located in the cloud computing environment 150, in some implementations, those components may be hosted locally on the user device 110. In addition, in various implementations, some or all of the components 112, 200, 215, and 565 are hosted locally on the user device 110, remotely hosted (such as in the cloud computing environment 150), or some combination thereof.

[0048] Reference now Figure 2 A, the exemplary ASR model 200 implements a sub-model 215 to produce a biased speech recognition result 224 (e.g., a probability density function 226 and / or a transcription of an utterance 108 corresponding to the audio data 102). Here, the ASR model 200 receives a speech recognition request 105 including the audio data 102 and the context indicator 103. In this example, the sub-model 215 includes a single model trained on various inputs and contexts corresponding to a variety of domains (i.e., multiple training utterance sets 560). The context indicator 103 can be a one-hot vector indicating one or more specific domains corresponding to the context of the audio data 102. The one-hot vector can be concatenated and projected into a phrase set embedding before being transmitted to the sub-model 215. In some implementations, the one-hot vector is looked up in an embedding matrix before being projected into the phrase set embedding. Alternatively, the sub-model 215 can project the one-hot vector into the phrase set embedding. The sub-model 215 can then activate one or more parameters corresponding to one or more specific domains indicated by the contextual indicator 103 based on the phrase set embedding to process the audio data 102. When the speech recognition request 105 does not include the contextual indicator 103, or when the contextual indicator 103 is not applicable to the sub-model 215 (i.e., the sub-model is not trained on the embedding space corresponding to the contextual indicator 103), the ASR model 200 processes the audio input 102 without activating or enabling the sub-model 215 to produce an unbiased speech recognition result 222 ( Figure 1 ).

[0049] Sub-model 215 may be implemented to bias the output of ASR model 200 in any number of ways. Figure 3AA schematic diagram 300a of an ASR model 200 for generating a biased speech recognition result 224 using a sub-model 215 implemented as a residual adapter layer is shown. The ASR model 200 can be a recurrent neural network (RNN) comprising an encoder 310 configured to encode input audio data 102 (and / or audio data 561 or training utterances 560) into an encoded output 312 (e.g., a hidden feature representation comprising a series of vectors) and a decoder 320 configured to decode the encoded output 312 into a biased speech recognition result 224. Typically, the encoded output 312 is sent directly to the decoder 320 to generate the biased speech recognition result 224. However, in this example, the sub-models 215 operate in parallel to process the speech recognition request 105. Subsequently, the sub-model 215 can generate a sub-model output 325 based on the received audio input 102 of the speech recognition request 105. The ASR model 200 may combine the sub-model output 325 and the encoded output 312 to produce a biased encoded output 314 to send to the decoder 320 .

[0050] In another example, Figure 3B A schematic diagram 300b of a submodel 215 implemented between layers of an encoder 310 is shown. The encoder 310 may include a plurality of components 360, and the submodel 215 may be disposed between layers of the components 360 such that the encoder 310 produces a biased encoded output 319. The components 360 of the encoder may include a stack of multi-headed attention blocks (i.e., conformer blocks), which may include a conformer or a transformer. In some implementations, each multi-headed attention block includes a multi-headed attention mechanism. In lieu of a multi-headed attention block, the encoder 310 may include a stack of long short-term memory (LSTM). The decoder 320 may receive the biased encoded output 319 and generate a biased speech recognition result 224.

[0051] Figure 3A and Figure 3BThe above examples are for illustrative purposes only and are not intended to be limiting. The ASR model 200 and the submodel 215 may include any suitable structure / architecture for performing speech recognition and generating biased speech recognition results in response to the context indicator 103. In addition, the submodel 215 and the ASR model 200 may work in any suitable combination to produce a biased speech recognition result 224. For example, the submodel 215 is set within the ASR model 200, at any suitable location in the architecture of the ASR model 200. For example, the submodel 215 is deployed within a layer of the ASR model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a prediction network, as a joint network, etc. Alternatively, the submodel 215 and the ASR model 200 can generate outputs independently of each other, and the results can be combined by the ASR model 200 or another suitable component of the system to determine a biased speech recognition result 224. It is noteworthy that when the sub-model 215 is disabled from the original frozen state of the ASR model 200, the ASR model 200 remains unchanged. That is, when the sub-model 215 is disabled (e.g., because the context indicator 103 is not present), the ASR model 200 produces an unbiased speech recognition result 222 that is not affected by the sub-model 215.

[0052] Figure 4A A schematic diagram 400a is shown of the ASR model 200 generating a first probability density function 226, 226A corresponding to an unbiased speech recognition result 222. Here, the source speaker 104 speaks an utterance 108 ("Pause playback") captured by the user device 110. The user device 110 transmits a speech recognition request 105 (including an audio input 102 representing the utterance 108) to the ASR model 200. The ASR model 200 processes the audio input 102 to generate an unbiased speech recognition result 222. In some implementations, the ASR model 200 determines whether the speech recognition request 105 includes a context indicator 103 before continuing to generate an unbiased speech recognition result 222 (e.g., a first probability density function 226A). In this example, the ASR model 200 predicts with a relatively low probability 410 that the audio input 102 includes the phrase "pause playback", making it unlikely that the utterance 108 will be accurately transcribed.

[0053] Figure 4B Schematic diagram 400b shows another exemplary ASR model 200 using the sub-model 215 to generate a second probability density function 226, 226B corresponding to the biased speech recognition result 224. Figure 4ASimilarly, source speaker 104 speaks utterance 108 ("Pause playback") captured by user device 110, which is then included in speech recognition request 105 as audio input 102 to ASR model 200 representing utterance 108. Here, speech recognition request 105 includes context indicator 103. In this example, context indicator 103 corresponds to a music player application executed on user device 110. ASR model 200 activates a portion of sub-model 215 corresponding to the domain of music players, such that the output of ASR model 200 is biased toward that domain. For example, sub-model 215, and subsequently ASR model 200, is biased toward words or phrases related to music players, such as "stop", "play", "pause", artist name, song name, etc. ASR model 200 generates biased speech recognition results 224, as shown in probability density function 226B. As shown, ASR model 200 predicts with high probability 412 that audio input 102 includes the phrase "pause playback" because the bias provided by sub-model 215 has caused probability density function 226B to be larger than probability density function 226A ( Figure 4A ) toward terms defined by the domain of the sub-model 215. In some implementations, the output 224 is a transcription of the audio data 102, where the transcription is biased toward a specific domain indicated by the contextual indicator 103 (i.e., the likelihood that words in the transcription belong to a specific domain will be higher).

[0054] That is, the biased speech recognition result 224 is different from the unbiased speech recognition result 222. For example, even in the case of an audio input 102 in which the probability density function 226A has a high confidence around a single word or term, if the single word or term is within a specific domain associated with the context indicator 103 (and subsequently the activated portion of the sub-model 215), the probability density function 226B can have an even higher confidence around the word or term. In some examples, the bias changes the probability density of the biased speech recognition result 224. Here, the probability density function 226B has a steeper gradient relative to the probability density function 226A, indicating that in this example, the distribution is concentrated around a smaller number of possibilities.

[0055] Figure 4A and Figure 4BThe above examples are for illustrative purposes only and are not intended to be limiting. For example, the speech recognition results 222, 224 can be in any suitable format, such as transcription, spectrogram, etc. In some implementations, the speech recognition results 222, 224 are generated as instructions for a computing device to perform an action (e.g., instructions for pausing a music application being executed on the user device 110).

[0056] Figure 5A A training process 500 for an ASR model 200 is shown. In some implementations, the process 500a employs a two-step training technique including a pre-training phase and a training phase. A pre-trained model is a technique for initializing a model that can then be further fine-tuned based on additional training data 510. For the ASR model 200, pre-training can include initiating the ASR model 200 with pre-training data 505 including a plurality of spoken utterances spoken by one or more speakers. The pre-training data 505 can also include spoken utterances paired with corresponding true value synthetic speech representations of the spoken utterances. The speech samples used for pre-training can be speech synthesized from reference transcriptions in a predetermined voice and / or speech synthesized from non-synthesized speech samples spoken by real humans.

[0057] After pre-training is completed, the process 500a can fine-tune the parameters of the pre-trained ASR model 200. The training process 500a includes, for example, training the encoder 310 and / or the decoder 320 ( Figure 3A ). Process 500a includes feeding training input 510 (also referred to as unbiased data 510) to the ASR model 200. In some implementations, the training input 510 includes a plurality of speech samples spoken by a variety of different speakers. In addition, the training input 510 can be paired with a label 520 indicating a target output associated with the training input 510. In other words, the training input 510 can include a plurality of speech samples corresponding to utterances spoken by different speakers, and each speech sample can be paired with a corresponding label 520 indicating a transcription of the corresponding utterance. Upon receiving the training input 510, the ASR model 200 can generate an output 515 (e.g., an unbiased speech recognition result 222). The ASR model 200 can generate an output 515 (e.g., an unbiased speech recognition result 222) with respect to the target output. Figure 2 The training input 510 may be processed in any of the manners described above with reference to FIG. 4 or in any other suitable manner for speech recognition.

[0058] In some implementations, the loss function 530 generates a loss 540 based on the output 512 and the true value label 520. That is, the loss function 530 compares the output 515 with the label 520 to generate a loss 540, wherein the loss 540 indicates the difference between the label 520 (i.e., the target output) and the output 515. The loss function 530 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 540 can be back-propagated through the ASR model 200 to update parameters via various techniques (such as stochastic gradient descent). Here, the ASR model 200 processes the loss 540 and adjusts one or more parameters of the ASR model 200 to take into account the loss 540. In some implementations, when the ASR model 200 is properly trained, the model is frozen. In other words, the parameters remain unchanged for a period of time until it is deemed that the ASR model 200 needs to be retrained (eg, when sufficient new training data 510 is obtained) or replaced.

[0059] Figure 5B A training process 500b is shown for training a sub-model 215 on a specific domain. In some implementations, before the training process 500b trains the sub-model 215 on the specific domain, the sub-model 215 is pre-trained on multiple different domains in the multiple domains (i.e., the sub-model 215 has been trained on one or more domains using process 500b). The sub-model 215 can be pre-trained or pre-configured in other ways so that the sub-model 215 is ready to be trained on a specific domain.

[0060] In some examples, process 500b includes training sub-model 215 using training utterances 560 belonging to a specific domain. Here, each training utterance 560 includes corresponding audio data 561 representing the training utterance 560 and a true value transcription 563 of the training utterance. In some implementations, the audio data 561 representing each training utterance 560 is associated with speech spoken by a different speaker than the audio data 561 representing other training utterances 560. The true value transcription 563 can be a manually generated text representing the corresponding audio data 561. In some implementations, the true value transcription 563 is machine generated. The true value transcription 563 should accurately reflect the corresponding speech sample (i.e., audio data 561) so that the true value transcription 563 is the target output of the sub-model 215. In some implementations, the training utterances 560 are collected based on the corresponding domain and / or terms. The sub-model 215 can therefore be trained on the training utterances 560 so that the sub-model 215 is biased towards terms or phrases corresponding to the specific domain associated with the training utterances 560. In the example of a single sub-model 215 adapted to bias multiple domains, the true value transcription 563 may be concatenated and projected into a phrase set embedding, which may then be used to train the sub-model 215. Thus, in use, the one-hot vectors of the contextual indicators 103 may be similarly concatenated and projected into a phrase set embedding that activates the portion of the sub-model 215 trained in a particular domain.

[0061] The process 500b may include feeding the true value transcription 563 of each training utterance 560 to an embedding encoder 565. The embedding encoder 565 may then generate a document embedding 567 for the training utterance 560 based on the true value transcription 563. Thereafter, the sub-model 215 receives the document embedding 567 generated for each training utterance 560 as a side input. More specifically, the training process 500b uses the document embedding 567 to train the sub-model 215 on a specific domain by activating a portion of the sub-model 215 corresponding to the specific domain associated with the training utterance 560. Here, the document embedding 567 may be related to a phrase set embedding of the embedding space. Thus, during operation, the contextual indicator 103 in the form of a one-hot vector indicating a specific domain may be projected to a phrase set embedding of the embedding space that activates the portion of the sub-model 215 corresponding to the specific domain (i.e., the portion of the sub-model 215 trained on the document embedding 567 for the specific domain). The document embedding 567 may be insensitive to phrase order because the embedding encoder 565 may treat phrases in different orders equally.

[0062] The embedding encoder 565 may include one or more components, such as a stack of multi-headed attention blocks, which may include a conformer or transformer block. In some implementations, each multi-headed attention block of the embedding encoder 565 may include a multi-headed attention mechanism. Instead of a multi-headed attention block, the embedding encoder 565 may include a stack of long short-term memories (LSTMs). Alternatively, the embedding encoder 565 may be a Conv net with different types of pooling. The embedding encoder 565 may be an encoder of any suitable form that can extract a single vector (i.e., a document embedding 567) from a set of phrases (i.e., a true value transcription 563).

[0063] The sub-model 215 can receive the document embedding 567 and the audio data 561 corresponding to the training utterance 560 to generate a sub-model output 569. In some implementations, the sub-model output 569 of the sub-model 565 is based on the corresponding document embedding 567 and on the history of the predicted speech recognition results 565 generated by the base speech recognition model 200 during one or more previous output steps. Figure 2 ), the sub-model 215 can be set in a layer of the base ASR model 200, and the sub-model output 569 can be combined with the output of the ASR model 200, or the sub-model output 569 can be generated in a layer of the encoder 310 of the ASR model. The base ASR model 200 can receive the sub-model output 569 and the audio data 561 representing the training utterance 560 to produce a predicted speech recognition result 565 (i.e., the biased speech recognition result 224).

[0064] In some implementations, the predicted speech recognition result 565 is used by the loss function 580 to generate a supervised loss term 590. That is, the loss function 580 compares the predicted speech recognition result 565 with the true value transcription 563 of the corresponding training utterance 560 to generate a supervised loss term 590, wherein the loss 590 indicates the difference between the true value transcription 563 (i.e., the target output) and the predicted speech recognition result 565. The loss function 580 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The supervised loss term 590 can then be fed directly to the sub-model 215. Here, the sub-model 215 processes the supervised loss term 590 and adjusts and / or updates one or more parameters of the sub-model 215 to take into account the supervised loss term 590. In some implementations, the base ASR model 200 is frozen during the training of the sub-model. Thus, the sub-model 215 adjusts parameters so that the sub-model output has the intended biasing effect on the base ASR model 200. In other words, the training process 500b uses the supervised loss term 590 to teach the sub-model to learn how to bias the base speech recognition model to recognize speech in a specific domain.

[0065] In some implementations, the training process 500b trains (or retrains / fine-tunes) a sub-model 215 that is continuously trained as it receives a new set of training utterances 560. For example, while the parameters of the ASR model 200 are frozen, the sub-model 215 can continue to receive the set of training utterances 560. In this way, the sub-model 215 can be trained on multiple domains. In other words, a single sub-model 215 can be used to bias the base ASR model 200 because the sub-model 215 is also trained on the embedding space (based on the document embeddings 567 of the embedding encoder 565) for each domain on which the sub-model 215 is trained.

[0066] Although the examples herein relate to biasing the ASR model 200 for a sub-model 215 for speech detection, it should be understood that the sub-model 215 can be used to bias any kind of model for any purpose. For example, the sub-model 215 can bias an image recognition model, a recommendation model, a filtering (e.g., email) model, a medical diagnosis model, or any other model in which contextual information can be used to bias the results to increase accuracy. As described above, the sub-model 215 can be trained on appropriate context indicators 103 to appropriately bias the underlying base model.

[0067] Training utterances 560 can be obtained in a variety of different ways. Typically, training utterances 560 are collected manually, where audio samples of the utterances are manually transcribed. However, manually labeling training data can be tedious and difficult to collect enough samples of labeled data for training. Fig. 6A and Figure 6B Various techniques are described for collecting training utterances 560 to train sub-models 215. Fig. 6A 600a, in some implementations, the data enhancement module 610 receives audio data 561 representing a non-synthesized speech representation 622 and a synthesized speech representation 624 of the same training utterance 560. Here, the training utterance set 560, 560a to 560n includes a non-synthesized speech representation 622 of each training utterance 560 in the training utterance set. The training utterance 560 may include a non-synthesized speech representation 622 of each training utterance 560 in the training utterance set. Figure 1 ) utterance 108 spoken by the user. In addition, each training utterance 560 can be paired with a corresponding true value transcription 563. The text-to-speech (TTS) system 620 is configured to receive the true value transcription 563 and generate audio data 561 corresponding to a synthesized speech representation 624 of the corresponding training utterance 560. In some implementations, the TTS system 620 generates multiple different synthesized speech representations 624 for a single training utterance 560, such that the different synthesized speech representations 624 are acoustically different from each other but lexically the same. For example, the synthesized speech representation 624 of a single training utterance 560 can be varied by adding noise, adding reverberation, or manipulating timing for enhancement. In some examples, the TTS system 620 receives unspoken training text utterances that include only text data (i.e., unpaired data), such that each unspoken text utterance is not paired with any synthetic or non-synthesized speech representation.

[0068] Thus, the data enhancement module 610 receives audio data 561 of a non-synthesized representation 622 of a training utterance 560 and / or receives audio data 561 of a synthesized speech representation 624 of the same training utterance 560. Thus, the data enhancement module 610 generates a pair of non-synthesized positive audio data examples 612, 612N using the non-synthesized representation 622, and generates a pair of synthesized positive audio data examples 612, 612S using the synthesized speech representation 624. Notably, both the pair of non-synthesized positive audio data examples 612N and the pair of synthesized positive audio data examples 612S correspond to the same training utterance 560, thereby greatly increasing the training process 500b ( Figure 5B ) can be used to train the sub-model 215. That is, the audio data 561 including the non-synthesized speech representation 622, the synthesized speech representation 624, or some combination thereof can be used by the training process 500b ( Figure 5B) is used. As previously mentioned, the "pair" of non-synthesized positive audio data examples 612N generated by the data enhancement module 610 is not limited to two examples, but can include any number of positive audio data examples generated for the same non-synthesized speech representation 622. Similarly, the "pair" of synthesized positive audio data examples 612S generated by the data enhancement module 610 can include any number of positive audio data examples generated for the same synthesized speech representation 624.

[0069] refer to Figure 6B , in contrast, the unspoken text selection process 600b can select unspoken text utterances 670 for training the sub-model 215 from the large unspoken text corpus 652, whereby the selected unspoken text utterances 670 are most similar to the specific domain that the sub-model 215 is being trained to learn. That is, the text selection process 600b is able to identify in-domain and near-domain unspoken text from the unspoken text corpus 652 for inclusion in the unspoken text utterances 670 for use in training the sub-model 215. Notably, the unspoken text utterances 670 selected by the text selection process 600b enable the on-the-fly synthesis of different utterances during batch construction, such that new speaker embeddings z and latent variables z can be sampled each time an unspoken text utterance 670 is in the batch.

[0070] The unspoken text corpus 652 includes a large number of unspoken training text utterances 670, 670a to 670n across a wide range of domains, and includes much greater linguistic diversity than the specific domain that the base ASR model 200 is trained to learn. The unspoken text corpus 652 can be stored in the same or different data store 158 as the spoken transcribed non-synthesized speech utterances 664 to pertain to the specific domain that the sub-model 215 is being trained to learn. Each spoken transcribed non-synthesized speech utterance 664 is paired with a corresponding transcription 663. The unspoken text corpus 652 can be dynamically changed to incorporate new unspoken text utterances 670. Simply using all the unspoken text utterances 670 in the unspoken text corpus 652 is not feasible for the following reasons: i) the speech modality requires much more memory to encode than text for each sentence, making it impractical to convert all the text in the unspoken text corpus 652; and ii) the large differences between the transcriptions 663 paired with the transcribed non-synthesized speech utterances 664 and the unspoken text utterances 670 in the unspoken text corpus 652 require intelligent strategies to balance their contributions.

[0071] The text selection process 600b is intended to select a subset of available unspoken text utterances 670 from the unspoken text corpus 652 as data for TTS synthesis, thereby generating the data generated for use in the above reference. Figure 5BThe synthesized speech representations (i.e., training utterances 560) of the sub-model 215 are trained during the supervised loss of the described training process 500b. In other words, the text selection process 600b aims to improve the match between the selected subset of available unspoken text utterances 670 and the specific domain being targeted, which in turn reduces the computational resources required to utilize large amounts of non-domain-specific data. Thus, the text selection process 600b reduces computational and memory costs by selecting unspoken text utterances 670 that best match the specific domain that the sub-model 215 is being trained to learn.

[0072] In some examples, the text selection process 600b selects a subset of available unspoken text utterances 670 that best matches a particular domain from the unspoken text corpus 652 by simply providing a domain identifier (not shown) associated with the particular domain as input to a background LM 686 previously trained on the entire unspoken text corpus 652. As previously mentioned, the unspoken text corpus 652 spans a large number of different domains. In these examples, the background LM 686 may include a maximum entropy (MaxEnt LM) that can optionally accept a domain identifier as input, as described in U.S. Patent No. 9,842,592 filed on February 12, 2014, the contents of which are incorporated herein by reference in their entirety. Here, the domain identifier associated with the particular domain can allow the MaxEnt LM to output a subset of available unspoken text utterances 670 from the unspoken text corpus 652 that are likely to include words and / or phrases related to the particular domain. In some configurations, rather than evaluating the likelihood of words, the statistical language model operates in reverse mode to randomly generate text phrases that match the statistical distribution of words related to the particular domain.

[0073] In the additional example, and as Figure 6B As depicted, the text selection process 600b uses the transcription 663 paired with the transcribed non-synthesized speech utterance 664 spoken by the human speaker to select a subset of the available unspoken text utterances 670 that best match the specific domain from the unspoken text corpus 652. Here, the transcribed non-synthesized speech utterance 664 includes words, phrases, and / or other terms related to the specific domain. Optionally, in addition to or in lieu of the transcription 663 paired with the transcribed non-synthesized speech utterance 664, a different set of transcribed utterances related to the specific domain can be used to select the unspoken text utterances 670. This will provide the advantage of not requiring all transcribed non-synthesized speech utterances 664 to belong to the specific domain.

[0074] During the first phase (Phase A), the unspoken text selection process 600b builds two language models 684, 686 to enable comparative selection of unspoken text utterances 670. Here, the domain-specific LM 680 is trained on each transcription 663 in the set of transcribed non-synthetic speech utterances 664. It is assumed that the set of transcribed non-synthetic speech utterances 664 belongs to the specific domain that the sub-model 215 is being trained to learn. On the other hand, the background LM 686 is trained on each unspoken text utterance 670 in the entire unspoken text corpus 652. As previously mentioned, the unspoken text corpus 652 spans a large number of different domains. In some examples, the first phase uses n-gram language model training to build two language models 684, 686. In other examples, the first phase uses neural network language model training to build two language models 684, 686.

[0075] During the second state (Phase B), the unspoken text selection process 600b uses two contrastive LMs 684, 686 to evaluate each unspoken text utterance 670 in the unspoken text corpus 652 by determining a first probability associated with each word in the unspoken text utterance 670 that appears in the domain-specific LM 684. and determining a second probability associated with each word in the unspoken text utterance 670 that appears in the background LM 686 Thereafter, for each unspoken text utterance 670 in the unspoken text corpus 652, the text selection process 600b determines a score S at the scorer 688 based on the first probability, the second probability, and the number of words #(w) that appear in the corresponding unspoken text utterance 670. For example, the score S for each unspoken text utterance 670 may be calculated as follows.

[0076]

[0077] After determining the scores, the unspoken text selection process 600b selects the unspoken text utterances 670 with the N-best scores S because these unspoken text utterances 670 best match the specific domain. The unspoken text corpus 652 may include billions of unspoken text utterances 670. The unspoken text utterances 670 selected by the text selection process 600b may include millions of utterances and, therefore, far exceed the number of transcribed non-synthesized speech utterances 664 spoken by human speakers. As discussed above, the content of the unspoken text utterances 670 increases the linguistic diversity of the specific domain that the sub-model is being trained to learn, while the corresponding synthesized speech representations (i.e., training utterances 560) generated from the unspoken text utterances 670 increase the acoustic / lexical diversity of the speech used to train the sub-model 215.

[0078] Figure 7 2 is a flow chart of an exemplary operational arrangement of a method 700 for training a sub-model 215 to contextually bias the results of an ASR model 200. The method 700 may be performed, for example, by Figure 1 The method 700 is performed by various elements of the context bias system 100. At operation 702, the method 700 includes obtaining a base speech recognition model 200 trained on unbiased data 510. At operation 704, the method 700 includes obtaining a set of training utterances 560 representing a specific domain, each training utterance 560 in the set of training utterances 560 including audio data 561 representing the training utterance 560 and a true value transcription 563 of the training utterance 560. At operation 706, the method 700 includes, for each corresponding training utterance 560 in the set of training utterances 560, using an embedding encoder 565 to determine a corresponding document embedding 567 from the true value transcription 563 of the corresponding training utterance 560. At operation 708, the method 700 includes training a sub-model 215 to bias the base speech recognition model 200 to recognize speech in a specific domain using the corresponding document embedding 567 determined from the true value transcription 563 of the set of training utterances 560.

[0079] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or persistent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0080] Figure 8 800 that can be used to implement the systems and methods described in this document. Computing device 800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0081] The computing device 800 includes a processor 810, a memory 820, a storage device 830, a high-speed interface / controller 840 connected to the memory 820 and a high-speed expansion port 850, and a low-speed interface / controller 860 connected to a low-speed bus 870 and the storage device 830. Each of the components 810, 820, 830, 840, 850, and 860 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 810 can process instructions for execution within the computing device 800, including instructions stored in the memory 820 or on the storage device 830, to display graphical information for a graphical user interface (GUI) on an external input / output device (such as a display 880 coupled to the high-speed interface 840). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Moreover, multiple computing devices 800 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0082] The memory 820 stores information non-temporarily within the computing device 800. The memory 820 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 820 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 800. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0083] The storage device 830 is capable of providing mass storage for the computing device 800. In some implementations, the storage device 830 is a computer-readable medium. In various implementations, the storage device 830 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory 820, a storage device 830, or a memory on a processor 810.

[0084] The high-speed controller 840 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 860 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 840 is coupled to a memory 820, a display 880 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 850 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 860 is coupled to a storage device 830 and a low-speed expansion port 890. The low-speed expansion port 890, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0085] As shown, computing device 800 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 800a or multiple implementations in a group of such servers 800a, implemented as a laptop computer 800b, or implemented as part of a rack server system 800c.

[0086] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0087] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0088] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0089] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0090] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0091] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (700) which, when executed by data processing hardware (154), causes the data processing hardware (154) to perform operations comprising: Obtaining a basic speech recognition model (200) trained on unbiased data; A training utterance set (560) representing a specific domain is obtained, each training utterance (560) in the training utterance set (560) comprising: audio data (561) representing the training utterance (560); and a ground-truth transcription (563) of said training utterance (560); for each corresponding training utterance (560) in the set of training utterances (560), determining, using an embedding encoder (515), a corresponding document embedding (567) from the ground-truth transcription (563) of the corresponding training utterance (560); and The sub-model (215) is trained using the corresponding document embeddings (567) determined from the true value transcriptions (563) of the training utterance set (560) to bias the base speech recognition model (200) to recognize speech in the specific domain.

2. The computer-implemented method (700) of claim 1, wherein training the sub-model (215) comprises, for each corresponding training utterance (560) in the set of training utterances (560): processing the audio data (561) representing the training utterance (560) using the base speech recognition model (200) to generate a predicted speech recognition result, the base speech recognition model being configured to receive a sub-model output (569) of the sub-model (215) based on the corresponding document embedding (567) determined from the ground truth transcription (563) of the corresponding training utterance (560); determining a supervision loss term (590) based on the predicted speech recognition result and the ground truth transcription (563) of the corresponding training utterance (560); and The parameters of the sub-model (215) are updated based on the supervised loss term (590) to teach the sub-model (215) to learn how to bias the basic speech recognition model (200) to recognize speech in the specific domain.

3. A computer-implemented method (700) as described in claim 2, wherein the sub-model output (569) of the sub-model (215) based on the corresponding document embedding (567) is also based on a history of predicted speech recognition results (565) generated by the base speech recognition model (200) during one or more previous output steps.

4. A computer-implemented method (700) as claimed in any one of claims 1 to 3, wherein parameters of the base speech recognition model (200) are frozen when training the sub-model (215).

5. The computer-implemented method (700) of any one of claims 1 to 4, wherein the operation further comprises, for at least one training utterance (560) in the set of training utterances (560), converting the true value transcription (563) of the corresponding at least one training utterance (560) using a text-to-speech (TTS) system (620) to generate the audio data (561) comprising a corresponding synthesized speech representation (624) of the corresponding at least one training utterance (560).

6. A computer-implemented method (700) as described in claim 5, wherein the ground-truth transcription (563) of the corresponding at least one training utterance (560) is generated using a background language model (686) and an in-domain language model (684) trained on transcribed speech utterances associated with the specific domain.

7. The computer-implemented method (700) of any one of claims 1 to 6, wherein the operation further comprises, for at least one training utterance (560) in the set of training utterances (560), applying data augmentation to the audio data (561) representing the at least one training utterance (560).

8. The computer-implemented method (700) of claim 7, wherein the applied data enhancement comprises at least one of adding noise, adding reverberation, or manipulating timing.

9. The computer-implemented method (700) of any one of claims 1 to 8, wherein the sub-model (215) comprises one or more neural network layers.

10. The computer-implemented method (700) of any one of claims 1 to 9, wherein the sub-model (215) is provided in a layer of the base speech recognition model (200).

11. The computer-implemented method (700) of any one of claims 1 to 10, wherein: The basic speech recognition model (200) includes an encoder (310) and a decoder (320); and The sub-model (215) is arranged between two layers of the encoder (310) of the basic speech recognition model (200).

12. The computer-implemented method (700) of any one of claims 1 to 11, wherein the operations further comprise, after training the sub-model (215), deploying the base speech recognition model (200) and the trained sub-model (215) for execution on a user device (110), the user device (110) being configured to: receiving a speech recognition request (105) comprising audio data (561) representing utterances captured in streaming audio; determining that the speech recognition request (105) includes a context indicator (103) indicating the specific domain; Using the trained sub-model (215) to bias the base speech recognition model (200) toward the specific domain; as well as A transcription of the utterance is generated by processing the audio data (561) using a biased base speech recognition model (200), the transcription being biased towards one or more terms in the particular domain.

13. The computer-implemented method (700) of any one of claims 1 to 12, wherein the operations further comprise, after training the sub-model (215): receiving a speech recognition request (105) from a user device (110) in communication with the data processing hardware, the speech recognition request comprising audio data (561) representing utterances captured in streaming audio by the user device (110); determining that the speech recognition request (105) includes a context indicator (103) indicating the specific domain; Using the trained sub-model (215) to bias the base speech recognition model (200) toward the specific domain; as well as A transcription of the utterance is generated by processing the audio data (561) using the biased base speech recognition model (200), the transcription being biased towards one or more terms in the particular domain.

14. The computer-implemented method (700) of any one of claims 1 to 13, wherein the sub-model (215) comprises the embedding encoder (515).

15. A system (100), comprising: Data processing hardware (154); as well as Memory hardware (156) in communication with the data processing hardware (154), the memory hardware (156) storing instructions that, when executed on the data processing hardware (154), cause the data processing hardware (154) to perform operations, the operations comprising: Obtaining a basic speech recognition model (200) trained on unbiased data; A training utterance set (560) representing a specific domain is obtained, each training utterance (560) in the training utterance set (560) comprising: audio data (561) representing the training utterance (560); and a ground-truth transcription (563) of said training utterance (560); for each corresponding training utterance (560) in the set of training utterances (560), determining, using an embedding encoder (515), a corresponding document embedding (567) from the ground-truth transcription (563) of the corresponding training utterance (560); and The sub-model (215) is trained using the corresponding document embeddings (567) determined from the true value transcriptions (563) of the training utterance set (560) to bias the base speech recognition model (200) to recognize speech in the specific domain.

16. The system (100) of claim 15, wherein training the sub-model (215) comprises, for each corresponding training utterance (560) in the set of training utterances (560): processing the audio data (561) representing the training utterance (560) using the base speech recognition model (200) to generate a predicted speech recognition result, the base speech recognition model being configured to receive a sub-model output (569) of the sub-model (215) based on the corresponding document embedding (567) determined from the ground truth transcription (563) of the corresponding training utterance (560); determining a supervision loss term (590) based on the predicted speech recognition result and the ground truth transcription (563) of the corresponding training utterance (560); and The parameters of the sub-model (215) are updated based on the supervised loss term (590) to teach the sub-model (215) to learn how to bias the basic speech recognition model (200) to recognize speech in the specific domain.

17. A system (100) as claimed in claim 16, wherein the sub-model output (569) of the sub-model (215) based on the corresponding document embedding (567) is also based on a history of predicted speech recognition results (565) generated by the base speech recognition model (200) during one or more previous output steps.

18. The system (100) of any one of claims 15 to 17, wherein parameters of the base speech recognition model (200) are frozen when training the sub-model (215).

19. The system (100) of any one of claims 15 to 18, wherein the operation further comprises, for at least one training utterance (560) in the set of training utterances (560), using a text-to-speech (TTS) system (620) to convert the true value transcription (563) of the corresponding at least one training utterance (560) to generate the audio data (561) comprising a corresponding synthesized speech representation (624) of the corresponding at least one training utterance (560).

20. The system (100) of claim 19, wherein the ground truth transcription (563) of the corresponding at least one training utterance (560) is generated using a background language model (684) and an in-domain language model (686) trained on transcribed speech utterances associated with the particular domain.

21. The system (100) of any one of claims 15 to 20, wherein the operation further comprises, for at least one training utterance (560) in the set of training utterances (560), applying data augmentation to the audio data (561) representing the at least one training utterance (560).

22. The system (100) of claim 21, wherein the applied data enhancement comprises at least one of adding noise, adding reverberation, or manipulating timing.

23. The system (100) of any one of claims 15 to 22, wherein the sub-model (215) comprises one or more neural network layers.

24. The system (100) of any one of claims 15 to 23, wherein the sub-model (215) is arranged in a layer of the basic speech recognition model (200).

25. The system (100) of any one of claims 15 to 24, wherein: The basic speech recognition model (200) includes an encoder (310) and a decoder (320); and The sub-model is arranged between two layers of the encoder (310) of the basic speech recognition model (200).

26. The system (100) of any one of claims 15 to 25, wherein the operations further comprise, after training the sub-model, deploying the base speech recognition model (200) and the trained sub-model for execution on a user device (110), the user device (110) being configured to: receiving a speech recognition request (105) comprising audio data (561) representing utterances captured in streaming audio; determining that the speech recognition request (105) includes a context indicator (103) indicating the specific domain; Using the trained sub-model to bias the base speech recognition model (200) toward the specific domain; as well as A transcription of the utterance is generated by processing the audio data (561) using a biased base speech recognition model (200), the transcription being biased towards one or more terms in the particular domain.

27. The system (100) of any one of claims 15 to 26, wherein the operations further comprise, after training the sub-model: receiving a speech recognition request (105) from a user device (110) in communication with the data processing hardware, the speech recognition request comprising audio data (561) representing utterances captured in streaming audio by the user device (110); determining that the speech recognition request (105) includes a context indicator (103) indicating the specific domain; Using the trained sub-model to bias the base speech recognition model (200) toward the specific domain; as well as A transcription of the utterance is generated by processing the audio data (561) using the biased base speech recognition model (200), the transcription being biased towards one or more terms in the particular domain.

28. The system (100) of any one of claims 15 to 27, wherein the sub-model comprises the embedding encoder (515).

Images