Submodel for neural context bias in speech recognition

By introducing sub-models into the ASR model and using field context indicators for bias, the problem of poor performance of the ASR model in different fields and situations is solved, and more accurate and flexible speech recognition is achieved, reducing computational cost.

CN119072744APending Publication Date: 2024-12-03GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380035071.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-19
Filing Date
2023-04-19
Publication Date
2024-12-03

Smart Images

  • Figure CN119072744A_ABST
    Figure CN119072744A_ABST
Patent Text Reader

Abstract

A method (600) includes obtaining a base ASR model (200) trained on unbiased data and a sub-model (250) trained on biased data representing a particular domain. The method comprises receiving a speech recognition request (105), the speech recognition request comprising audio data (102) characterizing an utterance (108) captured in the streaming audio. The method further comprises determining whether the speech recognition request comprises a contextual indicator indicative of the specific domain (103). When the speech recognition request does not include the contextual indicator, the method includes generating a first speech recognition result of the utterance by processing the audio data using the base ASR model (222). When the speech recognition request includes the contextual indicator, the method includes biasing the base ASR model toward the domain-specific model using the sub-model, and generating a second speech recognition result of the utterance by processing the audio data using the biased base ASR model (224).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to using sub-models to contextually bias the results of a base machine learning model. Background Art

[0002] Automatic speech recognition (ASR) is a category of natural language processing (NLP) that involves processing audio containing human speech. ASR models are typically used to recognize spoken language and / or translate spoken language into text. One way to generate an ASR model is to train the model on a large data set by using machine learning. Due to the amount of data used for training and the amount of time spent on training, ASR models are typically generalized for many domains and users, which makes the model inflexible. Attempts to make ASR models more flexible, such as by using multiple smaller models, can be computationally expensive (e.g., by training redundancy in multiple models) or provide skewed results (e.g., a model with less training data will be less robust). Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method for biasing speech recognition results based on context. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform an operation, the operation including obtaining a basic automatic speech recognition (ASR) model trained on unbiased data. The operation includes obtaining a sub-model trained on biased data, the biased data representing a specific domain. The operation also includes receiving a speech recognition request, the speech recognition request including audio data representing an utterance captured in a streaming audio. The operation includes determining whether the speech recognition request includes a context indicator indicating a specific domain. When the speech recognition request does not include a context indicator, the operation includes using a basic ASR model to generate a first speech recognition result of the utterance by processing the audio data. When the speech recognition request includes a context indicator, the operation includes using a sub-model to bias the basic ASR model toward a specific domain, and using a biased basic ASR model to generate a second speech recognition result of the utterance by processing the audio data, the second speech recognition result being biased toward one or more terms in the specific domain.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the context indicator includes a one-hot vector. In these implementations, the one-hot vector may indicate a particular sub-model to be activated from a plurality of sub-models, each of the plurality of sub-models being associated with a different domain. In these implementations, the operation may also include projecting the one-hot vector into a phrase set embedding in an embedding space. These implementations may additionally include projecting the one-hot vector into the phrase set embedding, thereby causing the phrase set embedding to activate a portion of the sub-model.

[0005] In some implementations, the submodel is set in a layer of a base ASR model. In these implementations, the base ASR model may include an encoder and a decoder, and the submodel is set between two layers of the encoder. In addition, one or more parameters of the base ASR can be frozen. In some implementations, the operation also includes retraining the submodel based on a speech recognition request (including audio data and context indicators) and a transcription of the audio data. In some implementations, the first speech recognition result is different from the second speech recognition result.

[0006] Another aspect of the present disclosure provides a system for biasing speech recognition results based on context. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a basic automatic speech recognition (ASR) model trained on unbiased data. The operations include obtaining a sub-model trained on biased data, the biased data representing a specific domain. The operations also include receiving a speech recognition request that includes audio data representing an utterance captured in streaming audio. The operations include determining whether the speech recognition request includes a context indicator indicating a specific domain. When the speech recognition request does not include a context indicator, the operations include using the basic ASR model to generate a first speech recognition result of the utterance by processing the audio data. When the speech recognition request includes a context indicator, the operation includes using a sub-model to bias a base ASR model toward a specific domain, and using the biased base ASR model to generate a second speech recognition result of the utterance by processing audio data, the second speech recognition result being biased toward one or more terms in the specific domain.

[0007] This aspect may include one or more of the following optional features. In some implementations, the context indicator includes a one-hot vector. In these implementations, the one-hot vector may indicate a particular sub-model to be activated from a plurality of sub-models, each of the plurality of sub-models being associated with a different domain. In these implementations, the operation may also include projecting the one-hot vector into a phrase set embedding in an embedding space. These implementations may additionally include projecting the one-hot vector into the phrase set embedding, thereby causing the phrase set embedding to activate a portion of the sub-model.

[0008] In some implementations, the submodel is set in a layer of a base ASR model. In these implementations, the base ASR model may include an encoder and a decoder, and the submodel is set between two layers of the encoder. In addition, one or more parameters of the base ASR can be frozen. In some implementations, the operation also includes retraining the submodel based on a speech recognition request (including audio data and context indicators) and a transcription of the audio data. In some implementations, the first speech recognition result is different from the second speech recognition result.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an example system for a context biasing system including an automatic speech recognition (ASR) model.

[0011] Figure 2A is a schematic diagram of an ASR model for using sub-models to produce biased speech recognition results.

[0012] Figure 2B is a schematic diagram of an ASR model for generating biased speech recognition results using a selected sub-model.

[0013] Figure 3A is a schematic diagram of an ASR model for producing biased speech recognition results using a sub-model as a residual adapter layer.

[0014] Figure 3B is a schematic diagram of an ASR model for using sub-models in the layers of an encoder to produce biased speech recognition results.

[0015] Figure 4A This is a schematic diagram of the ASR model generating unbiased speech recognition results.

[0016] Figure 4B It is a schematic diagram of the ASR model using sub-models to generate biased speech recognition results.

[0017] Figure 5A is a diagram of an example training scheme for an ASR model.

[0018] Figure 5B is a schematic diagram of an example training scheme for a sub-model that is used to bias speech recognition results based on context.

[0019] Figure 6 is a flow diagram of an example operational arrangement of a method for context biasing using sub-models.

[0020] Figure 7 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0021] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0022] Automatic speech recognition (ASR) is a growing field of language processing that has a variety of uses, from automatic translation and transcription of speech to processing voice commands for computing devices. Recently, neural networks for machine learning have been found to perform well as the basis for ASR systems and models. Using machine learning techniques, ASR models can be trained on large training data sets including audio samples of speech to produce robust models for speech recognition. Typically, these ASR models are large because the more extensively the model is trained, the better the model performs. However, there are disadvantages to using such large models, such as a single model for a variety of users with different characteristics. For example, even though English speakers may have many different accents or colloquialisms based on region, a single ASR model can still be built for the English language. Then, for some user groups, the ASR model may not perform so accurately. In addition, because of the computational cost, it is difficult to retrain or update the model due to size. This can make the ASR model outdated and not perform well for new / emerging words / phrases (e.g., slang, new TV shows).

[0023] The inflexibility of large ASR models hinders the potential feasibility of speech recognition because the ASR model may not perform well for part of the user base. Specifically, these large ASR models may not be able to utilize contextual signals that can help provide information about the voice. As described above, the user's location can provide information about the accent or add / eliminate certain words or phrases that inform the ASR model output. In another example, when an alarm rings from a smart device, the user is more likely to say a voice command related to the alarm (e.g., "stop al arm", "cancel", "snooze") than usual. Current ASR models (e.g., large or general models) cannot use this contextual information to affect the output.

[0024] Conventional attempts to "personalize" ASR models based on context are difficult and can cause problems during implementation. One technique for combining contextual information with an ASR model is by using multiple smaller ASR models, each of which is associated with a specific context or domain. However, training multiple smaller ASR models will be computationally expensive, especially because a large amount of training will be redundant between models. In addition, training multiple models will be time-consuming because the process can take weeks to train each model from scratch. Even if all ASR models in an ASR model are built and trained, some ASR models will perform poorly due to a lack of available training data, and managing models can be cumbersome. Another way to consider contextual information is to modify the ASR model to receive bias terms to influence the results. However, this typically involves significant manual intervention (e.g., when building user domain-specific models) and can cause catastrophic forgetting, where the general business of the ASR model may be over-triggered towards bias terms when the bias terms are not spoken, or not triggered at all even when the bias terms exist.

[0025] The implementations herein relate to a base ASR model that includes one or more sub-models to bias the base ASR model based on context so that the results or outputs of the model are directed to a specific context or domain. A sub-model is a set of parameters that can be added to or replaced from a general base ASR model. The sub-model can be loaded / enabled / disabled when necessary, thereby allowing the base ASR model to be used without bias, thereby eliminating the catastrophic forgetting problem. In some examples, the base ASR model is trained, and then the parameters of the base ASR model are frozen during operation (i.e., inference). In this way, the base ASR model remains stable and can continue to work for general business or multiple domains. However, when context information is available, the base ASR model can activate the appropriate sub-model for the context information in order to bias the speech recognition results toward terms related to the context information. One advantage of these implementations includes that the sub-models can be trained individually without having to train or retrain a large base ASR model. In addition, one or more sub-models can be loaded and replaced dynamically and in real time upon request (e.g., using a memory cache). With or without the use of the sub-models, the base ASR model remains unchanged, which eliminates the issue that the model will be compromised through use and / or updating.

[0026] As used herein, and unless otherwise specified, the terms "speech recognition system" and "speech recognition model" may refer to any combination of ASR systems / models in which speech is recognized and processed by a computing device. As will become apparent, the ASR models of the present disclosure and techniques for training ASR and sub-models will enable biasing speech recognition based on contextual information.

[0027] Figure 1 A context bias system 100 including an automatic speech recognition (ASR) model 200 and a sub-model 215 is shown. The ASR model 200 using the sub-model 215 is configured to process a speech recognition request 105. The speech recognition request 105 includes input audio data 102 corresponding to an utterance 108 spoken by a source speaker 104 and captured by a user device 110. The speech recognition request 105 may also include a context indicator 103. Using the audio data 102 and the context indicator 103, the ASR model 200 and the sub-model 215 generate or predict an unbiased speech recognition result 222 or a biased speech recognition result 224. The biased speech recognition result 224 is more likely to include words or phrases associated with the domain toward which the ASR model 200 is biased using the sub-model 215 (e.g., based on the context indicator 103). In some examples, the input audio data 102 includes an input spectrogram corresponding to the utterance 108. The context indicator 103 may provide an indication of a particular domain among a plurality of different domains, each indicating or representing a respective context of the utterance 108. In some implementations, the speech recognition results 222, 224 include a probability density function 226, 226A-B representing the density of probabilities of text translations of the utterance 108.

[0028] Although not shown, the acoustic front end resident on the user device 110 can convert the time-domain audio waveform of the speech 108 captured via the microphone of the user device 110 into an input spectrogram 102 or other type or form of audio data 102. In addition, the front-end device can be configured to determine or obtain data representing context indicators 103 affecting the speech 108 and / or other relevant information corresponding to the source speaker 104 and / or the client device 110.

[0029] A user device 110 associated with a source speaker 104 may capture an utterance 108 spoken by the source speaker 104 and provide the corresponding input audio data 102 to the context biasing system 100 as part of a speech recognition request 105. In addition, the user device 110 may determine a context indicator 103 to include with the speech recognition request 105. The user device 110 may include, but is not limited to, a smartphone, a tablet computer, a desktop / laptop computer, a smart speaker, a smart display, a smart home appliance, an assistant-enabled wearable device (e.g., a smart watch, a smart headset, smart glasses, etc.), or a vehicle infotainment system. Alternatively, the remote server 112 may process the audio data 102 and any other additional data or metadata from the user device 110 to determine the context indicator 103.

[0030] The context biasing system 100 may be distributed across multiple devices such that the ASR model 200 resides on one of the user device 110 or a remote system 150 (also referred to herein as a cloud computing environment) that communicates with the user device 110 via a network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic resources 152, including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). The data repository 158 (i.e., a remote storage device) may be overlaid on the storage resources 146 to allow scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The ASR model 200 and the sub-model 215 may be executed on the remote system 150 or the user device 110. The sub-model 215 may be stored locally at the user device 110 or on a remote system (e.g., stored at the data repository 158), or some combination thereof.

[0031] The context bias system 100 implements a dynamic model including two or more components or models, including a common base model (e.g., ASR model 200) and a "swappable" sub-model 215 to generate biased speech recognition results 224 based on the received speech recognition request 105. The ASR model 200 can be trained on a large speech data set. Once trained, the ASR model 200 can be frozen so that the parameters of the ASR model 200 remain constant during operation. The ASR model 200 can be updated, retrained, or replaced as needed or when additional training data becomes available. In some implementations, a single sub-model 215 is used to bias the ASR model 200 by activating certain parameters based on the context indicator 103 of the speech recognition request 105. In other implementations, the system 100 includes multiple sub-models 215, each of which corresponds to a specific domain. In these implementations, a specific sub-model 215 is selected / activated based on the context indicator 103. For example, when the context indicator 103 indicates that the domain including the user's alarm clock is applicable, a corresponding sub-model 215 that is biased toward a specific term (e.g., snooze, stop, etc.) related to the indicated domain (i.e., the user's alarm clock) is selected. When the speech recognition request 105 does not have a context indicator 103 or when the sub-model 215 corresponding to the context indicator does not exist or is underdeveloped, in some examples, the ASR model 200 generates an unbiased speech recognition result 222. That is, the unbiased speech recognition result 222 is generated only by the ASR model 200 and is not affected or affected by any sub-model 215.

[0032] The context indicator 103 may be based on any signal or data that can be used to improve the accuracy of the biased speech recognition results 224. The context indicator 103 may be based on information related to the source speaker 104. For example, the source speaker 104 has a particular dialect, native language, mannerisms, patterns of speech, speech disfluencies, etc. Therefore, the system 100 may generate and train a sub-model 215 corresponding to the source speaker 104, wherein the sub-model 215 biases / personalizes the ASR model 200 to make predictions that are particularly suitable for the source speaker 104. In some implementations, the context indicator 103 includes a one-hot vector, and the system uses the context indicator 103 to select an appropriate sub-model 215 (i.e., the sub-model 215 corresponding to the source speaker 104) from a plurality of different sub-models 215.

[0033] In some implementations, the context indicator 103 is based on information related to the user device 110. For example, the user device 110 may include a smart device equipped with sensors such as GPS, accelerometers, gyroscopes, microphones, proximity sensors, cameras, etc. The context indicator 103 may indicate a domain related to the user device 110, such as inferred from one of the sensors. For example, the context indicator 103 may indicate the geographic location of the user device 110 as inferred from GPS data (which may be revoked at any time if the user 104 explicitly agrees to share his / her geographic location). Here, the context indicator 103 may correspond to a more geographic location (e.g., a city such as Chicago) or to a more specific location (e.g., a gym). In either instance, the sub-model 215 may bias the biased speech recognition results 224 toward a specific domain based on location. Specifically, a contextual indicator 103 identifying the location Chicago can activate a sub-model 215 trained on data from users from that city, state, and / or region, which can be biased based on the accent or other features of the speech of users from that region and named entities unique to that region (e.g., restaurants, sports teams, street names, etc.). Thus, the sub-model 215 can bias the ASR model 200 to generate biased speech recognition results 224 that are biased toward a prediction bias appropriate for that domain. For example, the biased speech recognition results 224 can increase the likelihood that the ASR model 200 predicts that the speech 108 includes a reference to a restaurant or street in Chicago. Similarly, a contextual indicator 103 indicating that the source speaker 104 is at a gym can activate a sub-model 215 trained based on speech from users who are exercising or in a similar location. Here, the speech may be affected by labored breathing, or involve specific words or phrases (e.g., voice commands for operating a music player on a smart device). Thus, the sub-model 215 can take these contextual elements into account when processing the audio data 102 to produce biased speech recognition results 224. In another example, the context indicator 103 can indicate a software application currently being executed on the user device 110, such as a music player application. In this example, the context indicator 103 identifies the sub-model 215 corresponding to the software application (e.g., music player) indicated by the context indicator 103 in order to bias the ASR model 200 to recognize terms / phrases such as "next song" or "pause". In other examples, the sub-model 215 is generally trained for that type of application or domain.Thus, when the source speaker 104 utters the utterance 108 “pause playback,” the ASR model 200 biased by the sub-model 215 will generate a biased speech recognition result 224 that is skewed or biased toward a result related to the music player relative to the unbiased speech recognition result 222 determined without activating the sub-model 215.

[0034] In some implementations, the context indicator 103 indicates that multiple domains are applicable to the utterance 108. In this scenario, a single sub-model 215 can bias the ASR model 200 to generate biased speech recognition results 224 toward each of the multiple domains. Alternatively, multiple sub-models 215 (each trained for one or more of the multiple domains) can bias the ASR model 200 to generate biased speech recognition results 224 toward each of the multiple domains. For example, when the source speaker 104 is located in a gym and a music player is executing on a user device, one or more sub-models 215 can bias the output of the ASR model 200 toward each of these domains.

[0035] Output 190 can receive unbiased speech recognition results 222 and biased speech recognition results 224 generated by ASR model 200. In some examples, output 190 includes natural language understanding (NLU) that performs query interpretation on the speech recognition results. NLU can further instruct downstream applications / services to perform actions based on the results. Output 190 can also include a user interface generator that is configured to display the speech recognition results as a transcription on a screen of user device 110 and / or another device.

[0036] Figure 1 The system 100 is presented for illustration purposes only and is not intended to be limiting. For example, although only a single example of each component is illustrated, the system 100 may include any number of components 110, 112, 140, 150, 200, and 215. In addition, although some components are described as being located in the cloud computing environment 150, in some implementations, those components may be hosted locally on the user device 110. In addition, in various implementations, some or all of the components 112, 200, and 215 are hosted locally on the user device 110, remotely (such as in the cloud computing environment 150), or some combination thereof.

[0037] Reference now Figure 2A, the exemplary ASR model 200 implements a sub-model 215 to produce a biased speech recognition result 224 (e.g., a probability density function 226). Here, the ASR model 200 receives a speech recognition request 105 including audio data 102 and a context indicator 103. In this example, the sub-model 215 includes a single model trained on various inputs and contexts corresponding to various domains. The context indicator 103 can be a one-hot vector that indicates one or more specific domains corresponding to the context of the audio data 102. The one-hot vector can be concatenated and projected into a phrase set embedding before being transmitted to the sub-model 215. In some implementations, the one-hot vector is looked up in the embedding matrix before being projected into the phrase set embedding. Alternatively, the sub-model 215 can project the one-hot vector into the phrase set embedding. Then, the sub-model can activate one or more parameters corresponding to one or more specific domains indicated by the context indicator 103 based on the phrase set embedding to process the audio data 102.

[0038] Reference now Figure 2B, another exemplary ASR model 200 implements the selected sub-model 215, 215b to produce a biased speech recognition result 224 (e.g., a probability density function 226). Here, the ASR model 200 receives a speech recognition request 105 including audio data 102 and a context indicator 103. In this example, the context indicator 103 is received by a sub-model selector 218. The sub-model selector 218 selects a specific sub-model 215 from a plurality of sub-models 215, 215a-n, wherein each sub-model 215 corresponds to one or more different domains. The sub-models 215a-n can be trained on data reflecting any domain of the context provided for processing the audio input 102. For example, the selected sub-model 215b is based on a domain such as location, activity, user profile, software application, or any other domain that may affect the audio data 102. The context indicator 103 can be a unique hot vector indicating a specific domain corresponding to the context of the speech recognition request 105. Upon receiving the context indicator 103, the sub-model selector 218 may perform a lookup to determine the appropriate sub-model 215 to use when processing the speech recognition request 105. The various sub-models 215a-n are easily swappable such that when processing the streaming audio input 102 from the user device 110, the ASR model 200 may swap in or out the various sub-models 215 from the plurality of sub-models 215a-n (i.e., disable or enable the bias provided by the sub-models 215) while processing the streaming input when a new context indicator 103 is received. When the speech recognition request 105 does not include the context indicator 103 or when the context indicator 103 is not applicable to any of the available sub-models 215, then the ASR model 200 processes the audio input 102 without activating or enabling any sub-model 215 to produce an unbiased speech recognition result 222 ( Figure 1 ).

[0039] Sub-model 215 may be implemented to bias the output of ASR model 200 in any number of ways. Figure 3AA schematic diagram 300a of an ASR model 200 for generating a biased speech recognition result 224 using a sub-model 215 implemented as a residual adapter layer is shown. The ASR model 200 can be a recurrent neural network (RNN) comprising an encoder 310 and a decoder 320, the encoder being configured to encode the input audio data 102 into an encoded output 312 (e.g., a hidden feature representation comprising a series of vectors), the decoder being configured to decode the encoded output 312 into a biased speech recognition result 224. Typically, the encoded output 312 is sent directly to the decoder 320 to generate the biased speech recognition result 224. However, in this example, the sub-models 215 operate in parallel to process the speech recognition request 105. Subsequently, the sub-models 215 can generate the sub-model output 325 based on the received audio input 102 of the speech recognition request 105. The ASR model 200 may merge the sub-model output 325 and the encoded output 312 to produce a biased encoded output 314 to send to the decoder 320 .

[0040] In another example, the sub-model 215 is implemented between layers of the encoder 310 . Figure 3B An example schematic diagram 300b of such an implementation is shown. The encoder 310 may include multiple components 360, and the submodel 215 may be set between the layers of the components 360 so that the encoder 310 produces a biased encoded output 319. The components 360 of the encoder may include a stack of multi-headed attention blocks (i.e., convolution-enhanced transformer blocks), which may include a convolution-enhanced transformer or transformer. In some implementations, each multi-headed attention block may include a multi-headed attention mechanism. The encoder 310 may include a stack of long short-term memory (LSTM) instead of a multi-headed attention block. The decoder 320 may receive the biased encoded output 319 and generate a biased speech recognition result 224.

[0041] Figure 3A and Figure 3BThe above examples are for illustrative purposes only and are not intended to be limiting. The ASR model 200 and the submodel 215 may include any suitable structure / architecture for performing speech recognition and generating biased speech recognition results in response to the context indicator 103. In addition, the submodel 215 and the ASR model 200 may work in any suitable combination to produce a biased speech recognition result 224. For example, the submodel 215 is set within the ASR model 200 at any suitable location in the architecture of the ASR model 200. For example, the submodel 215 is deployed within a layer of the ASR model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a prediction network, as a joint network, etc. Alternatively, the submodel 215 and the ASR model 200 can generate outputs independently of each other, and the results can be combined by the ASR model 200 or another suitable component of the system to determine a biased speech recognition result 224. It is noteworthy that when the sub-model 215 is disabled from the original frozen state of the ASR model 200, the ASR model 200 remains unchanged. That is, when the sub-model 215 is disabled (e.g., because the context indicator 103 is not present), the ASR model 200 produces an unbiased speech recognition result 222 that is not affected by the sub-model 215.

[0042] Figure 4A A schematic diagram 400a is shown in which the ASR model 200 generates a first probability density function 226, 226A corresponding to an unbiased speech recognition result 222. Here, the source speaker 104 speaks an utterance 108 ("Pause play back") captured by the user device 110. The user device 110 transmits a speech recognition request 105 (including an audio input 102 representing the utterance 108) to the ASR model 200. The ASR model 200 processes the audio input 102 to generate an unbiased speech recognition result 222. In some implementations, the ASR model 200 determines whether the speech recognition request 105 includes a context indicator 103 before continuing to generate the unbiased speech recognition result 222 (e.g., the first probability density function 226A). In this example, the ASR model 200 predicts that the audio input 102 includes the phrase "pause play back" with a relatively low probability 410, making it unlikely that the utterance 108 will be accurately transcribed.

[0043] Figure 4B Schematic diagram 400b shows another exemplary ASR model 200 using the sub-model 215 to generate a second probability density function 226, 226B corresponding to the biased speech recognition result 224. Figure 4ASimilarly, the source speaker 104 speaks an utterance 108 ("Pause playback") captured by the user device 110, which is then included in the speech recognition request 105 as the audio input 102 to the ASR model 200 representing the utterance 108. Here, the speech recognition request 105 includes the context indicator 103. In this example, the context indicator 103 corresponds to a music player application executed on the user device 110. The ASR model 200 activates a sub-model 215 corresponding to the domain of music players, so that the output of the ASR model 200 is biased toward the domain. For example, the sub-model 215 and subsequently the ASR model 200 are biased toward words or phrases related to music players, such as "stop", "play", "pause", artist name, song name, etc. The ASR model 200 generates a biased speech recognition result 224, as shown in the probability density function 226B. As shown, ASR model 200 predicts that audio input 102 includes the phrase "pause playback" with a high probability 412 because the bias provided by sub-model 215 has shifted probability density function 226B relative to probability density function 226A ( Figure 4A ) towards a term “shifted” by the domain of sub-model 215.

[0044] That is, biased speech recognition result 224 is different from unbiased speech recognition result 222. For example, even in the case of audio input 102, where probability density function 226A has a high confidence around a single word or term, if the word or term is within a specific domain associated with submodel 215, probability density function 226B can have an even higher confidence around a single word or term. In some examples, biasing changes the probability density of biased speech recognition results. Here, probability density function 226B has a steeper gradient relative to probability density function 226A, indicating that in this example, the distribution is concentrated around a smaller number of possibilities.

[0045] Figure 4A and Figure 4B The above examples are for illustration purposes and are not intended to be limiting. For example, the speech recognition results 222, 224 may be in any suitable format, such as transcription, spectrogram, etc. In some implementations, the speech recognition results 222, 224 are generated as instructions for a computing device to perform an action (e.g., an instruction to pause a music application being executed on the user device 110).

[0046] Figure 5AA training process 500 for an ASR model 200 is shown. In some implementations, the process 500a employs a two-step training technique including pre-training and training. A pre-trained model is a technique for initializing a model, which can then be further fine-tuned based on additional training data 510. For the ASR model 200, pre-training can include initiating the ASR model 200 using pre-training data 505 including a plurality of spoken utterances by one or more speakers. The pre-training data 505 can also include spoken utterances paired with corresponding true value synthetic speech representations of the spoken utterances. The speech samples used for pre-training can be speech synthesized from reference transcriptions in a predetermined voice and / or non-synthesized speech samples spoken by real humans.

[0047] After pre-training is complete, process 500a may fine-tune the parameters of the pre-trained ASR model 200. The training process 500a includes training, for example, the encoder 310 and / or the decoder 320 ( Figure 3A ). Process 500 includes feeding training input 510 to ASR model 200. In some implementations, training input 510 includes multiple speech samples spoken by various different speakers. In addition, training input 510 can be labeled with a label 520 indicating a target output associated with training input 510. Upon receiving training input 510, ASR model 200 can generate output 515 (e.g., unbiased speech recognition result 222). ASR model 200 can process training input 510 in the manner described with respect to any of Figures 2 to 4 or in any other suitable manner for speech recognition.

[0048] In some implementations, the output 515 is used by the loss function 530 to generate a loss 540. That is, the loss function 530 compares the output 515 with the label 520 to generate a loss 540, where the loss 540 indicates the difference between the label 520 (i.e., the target output) and the output 515. The loss function 530 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 540 can then be fed directly to the ASR model 200. Here, the ASR model 200 processes the loss 540 and adjusts one or more parameters of the ASR model 200 to account for the loss 540. In some implementations, when the ASR model 200 is properly trained, the model is frozen. In other words, the parameters remain unchanged for a period of time until it is deemed that the ASR model 200 needs to be retrained (e.g., when enough new training data 510 is obtained) or replaced.

[0049] Figure 5BA training process 500b for a sub-model 215 is shown. In some implementations, the process 500b employs a two-step training technique including pre-training and training. Pre-training is a technique for initializing a model, which can then be further fine-tuned based on additional training data 560. For the sub-model 215, pre-training can include initiating the sub-model 215 using pre-training data 555 including a plurality of spoken utterances by a speaker. The pre-training data 555 can also include spoken utterances paired with corresponding true value synthetic speech representations of the spoken utterances. The speech samples used for pre-training can be speech synthesized from a reference transcription in a predetermined voice. For the sub-model 215, pre-training can also include adjusting one or more parameters of the sub-model 215 so as to bias the sub-model 215 toward words and / or phrases corresponding to a specific domain associated with the sub-model 215.

[0050] In some examples, process 500b includes fine-tuning the parameters of the pre-trained sub-model 215. In these examples, process 500b includes feeding training input 560 to sub-model 215. In some implementations, training input 560 includes multiple speech samples spoken by various speakers. Training input 560 may also include context indicators 103 and transcriptions under each speech sample. The transcription may be a manually generated text representing the corresponding speech sample. In some implementations, the transcription is machine generated. The transcription should accurately reflect the corresponding speech sample so that the transcription is the target output of the sub-model. In some implementations, the context indicators 103 of the training input 560 may include domains and / or terms. In other implementations, the context indicators 103 of the training input 560 may include a unique hot vector. The sub-model 215 corresponding to the context indicator 103 may be trained on the speech sample and the transcription so that the sub-model 215 is biased toward the terms or phrases corresponding to the domain associated with the context indicator 103. In the example of a single sub-model 215 suitable for biasing multiple domains, the one-hot vectors of the context indicators 103 can be concatenated and projected into a phrase set embedding, which can then be used to train the sub-model 215. In addition, the training input 560 can be labeled with a label 570 indicating a target output associated with the training input 560. Upon receiving the training input 560, the sub-model 215 can generate an output 565 (e.g., a biased speech recognition result). The sub-model 215 can process the training input 560 in the manner described with respect to any of Figures 2 to 4 or in any other suitable manner for speech recognition.

[0051] In some implementations, the output 565 is used by the loss function 580 to generate a loss 590. That is, the loss function 580 compares the output 565 to the label 570 to generate a loss 590, where the loss 590 indicates the difference between the label 570 (i.e., the target output) and the output 565. The loss function 580 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 590 can then be fed directly to the sub-model 215. Here, the sub-model 215 processes the loss 590 and adjusts one or more parameters of the sub-model 215 to account for the loss 590. In some implementations, the sub-model 215 continues to be trained (or retrained) when additional speech recognition requests are received. For example, when the ASR model 200 is frozen, the sub-model 215 may continue to adjust parameters based on the received speech recognition request 105 (including the audio data 102, the context indicator 103), and the biased speech recognition result 224. Here, in the case of multiple sub-models 215, each sub-model 215 may be trained when used by the ASR model 200 for biasing to the associated domain.

[0052] Although the examples herein relate to biasing the ASR model 200 for a sub-model 215 for speech detection, it should be understood that the sub-model 215 and the context indicators 103 can be used to bias any kind of model for any purpose. For example, the sub-model 215 can bias an image recognition model, a recommendation model, a filtering (e.g., email) model, a medical diagnosis model, or any other model in which context information can be used to bias the results to increase accuracy. As described above, the sub-model 215 can be trained on the appropriate context indicators 103 to appropriately bias the underlying base model.

[0053] Figure 6 is a flow chart of an exemplary operational arrangement of a method 600 for contextually biasing the results of the ASR model 200 using the sub-model 215. The operations of the method 600 may be performed on the data processing hardware 710 ( Figure 7 ), which data processing hardware may reside on the user device 110 or include data processing hardware 154 residing on the remote system 150. Thus, the method 600 may be performed, for example, by Figure 1The method 600 is performed by various elements of the context bias system 100. At operation 602, the method 600 includes obtaining a basic automatic speech recognition (ASR) model 200 trained on unbiased data 510. At operation 604, the method 600 includes obtaining a sub-model 215 trained on biased data 560, and the biased data 560 represents a specific domain. At operation 606, the method 600 includes receiving a speech recognition request 105, which includes audio data 102 representing an utterance 108 captured in a streaming audio. At operation 608, the method 600 includes determining whether the speech recognition request 105 includes a context indicator 103 indicating a specific domain. At operation 610, when the speech recognition request 105 does not include the context indicator 103, the method 600 includes using the basic ASR model 200 to generate a first speech recognition result 222 (i.e., an unbiased speech recognition result 222) of the utterance 108 by processing the audio data 102. At operation 612, when the speech recognition request 105 includes a context indicator 103, the method 600 includes biasing the base ASR model 200 toward a specific domain using the sub-model 215, and using the biased base ASR model 200 to generate a second speech recognition result 224 (i.e., a biased speech recognition result 224) of the utterance 108 by processing the audio data 102, the second speech recognition result 224 being biased toward one or more terms in the specific domain.

[0054] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] Figure 7 700 that can be used to implement the systems and methods described in this document. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0056] The computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor (e.g., data processing hardware) 710 can process instructions for execution within the computing device 700, including instructions stored in the memory 720 (e.g., memory hardware) or on the storage device 730 to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 780 coupled to the high-speed interface 740). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Furthermore, multiple computing devices 700 may be connected (eg, as a server bank, a group of blade servers, or a multi-processor system), with each device providing portions of the necessary operations.

[0057] The memory 720 stores information non-temporarily within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 720 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0058] The storage device 730 is capable of providing mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various implementations, the storage device 730 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0059] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 740 is coupled to the memory 720, the display 780 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner; or to a networking device, such as a switch or a router, for example, through a network adapter.

[0060] As shown, computing device 700 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 700a or multiple implementations in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0061] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be special-purpose or general-purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0062] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0063] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0064] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system (for example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). For example, a processor suitable for executing a computer program includes both a general-purpose microprocessor and a special-purpose microprocessor, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0066] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (600) which, when executed by data processing hardware (710), causes the data processing hardware (710) to perform operations comprising: Obtain a basic automatic speech recognition (ASR) model trained on unbiased data; Obtaining a sub-model (215) trained on biased data, wherein the biased data represents a specific domain; receiving a speech recognition request (105) comprising audio data (102) representing speech (108) captured in streaming audio; determining whether the speech recognition request (105) includes a context indicator (103) indicating the specific domain; When the speech recognition request (105) does not include the context indicator (103), generating a first speech recognition result (222) of the utterance (108) by processing the audio data (102) using the basic ASR model (200); and When the speech recognition request (105) includes the context indicator (103): biasing the base ASR model (200) toward the specific domain using the sub-model (215); as well as A second speech recognition result (224) of the utterance (108) is generated by processing the audio data (102) using a biased base ASR model (200), wherein the second speech recognition result (224) is biased toward one or more terms in the specific domain.

2. The computer-implemented method (600) of claim 1, wherein the context indicator (103) comprises a one-hot vector.

3. The computer-implemented method (600) of claim 2, wherein the one-hot vector indicates a particular sub-model (215) to be activated from a plurality of sub-models (215), each of the plurality of sub-models (215) being associated with a different domain.

4. The computer-implemented method (600) of claim 2 or 3, wherein the operations further comprise projecting the one-hot vector into a phrase set embedding of an embedding space.

5. The computer-implemented method (600) of claim 4, wherein projecting the one-hot vector into the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model (215).

6. The computer-implemented method (600) according to any one of claims 1 to 5, wherein the sub-model (215) is provided in a layer of the base ASR model (200).

7. The computer-implemented method (600) of claim 6, wherein: The basic ASR model (200) includes an encoder (310) and a decoder (320); and The sub-model (215) is arranged between two layers of the encoder (310).

8. A computer-implemented method (600) according to any one of claims 1 to 7, wherein one or more parameters of the basic ASR are frozen when generating the first speech recognition result (222) and the second speech recognition result (224).

9. The computer-implemented method (600) of any one of claims 1 to 8, wherein: The data processing hardware (710) resides on a user device (110) that captures the speech (108) in the streaming audio; or The data processing hardware (710) resides on a remote system (150) that communicates with the user device (110) via a network.

10. The computer-implemented method (600) of any one of claims 1 to 9, wherein the first speech recognition result (222) is different from the second speech recognition result (224).

11. A system (100), comprising: Data processing hardware (710); as well as Memory hardware (720) in communication with the data processing hardware (710), the memory hardware (720) storing instructions that, when executed on the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations comprising: Obtain a basic automatic speech recognition (ASR) model trained on unbiased data; Obtaining a sub-model (215) trained on biased data, wherein the biased data represents a specific domain; receiving a speech recognition request (105) comprising audio data (102) representing speech (108) captured in streaming audio; determining whether the speech recognition request (105) includes a context indicator (103) indicating the specific domain; When the speech recognition request (105) does not include the context indicator (103), generating a first speech recognition result (222) of the utterance (108) by processing the audio data (102) using the basic ASR model (200); and When the speech recognition request (105) includes the context indicator (103): biasing the base ASR model (200) toward the specific domain using the sub-model (215); and A second speech recognition result (224) of the utterance (108) is generated by processing the audio data (102) using a biased base ASR model (200), wherein the second speech recognition result (224) is biased toward one or more terms in the specific domain.

12. The system (100) of claim 11, wherein the context indicator (103) comprises a one-hot vector.

13. The system (100) of claim 12, wherein the one-hot vector indicates a specific sub-model (215) to be activated from a plurality of sub-models (215), each of the plurality of sub-models (215) being associated with a different domain.

14. The system (100) of claim 12 or 13, wherein the operations further comprise projecting the one-hot vector into a phrase set embedding in an embedding space.

15. The system (100) of claim 14, wherein projecting the one-hot vector into the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model.

16. The system (100) according to any one of claims 11 to 15, wherein the sub-model is arranged in a layer of the base ASR model (200).

17. The system (100) of claim 16, wherein: The basic ASR model (200) includes an encoder (310) and a decoder (320); and The sub-model is arranged between two layers of the encoder (310).

18. The system (100) according to any one of claims 11 to 17, wherein one or more parameters of the basic ASR are frozen when generating the first speech recognition result (222) and the second speech recognition result (224).

19. The system (100) according to any one of claims 11 to 18, wherein: The data processing hardware resides on a user device (110) that captures the speech (108) in the streaming audio; or The data processing hardware resides on a remote system (150) that communicates with the user device (110) via a network.

20. The system (100) according to any one of claims 11 to 19, wherein the first speech recognition result (222) is different from the second speech recognition result (224).