Submodels of neural context bias in speech recognition

Abstract

The method includes obtaining a base automatic speech recognition ASR model (200) trained with unbiased data and a sub-model (250) trained with biased data representative of a particular domain. The method includes receiving a speech recognition request (105) comprising audio data (102) characterizing an utterance (108) captured in streaming audio. The method includes determining whether the speech recognition request comprises a context indicator (103) indicative of the particular domain. If the context indicator is not present, the audio data is processed with the base automatic speech recognition ASR model to generate a first speech recognition result (222) of the utterance. If the speech recognition request comprises the context indicator, the sub-model is used to bias the base automatic speech recognition ASR model toward the particular domain and the audio data is processed with the biased base automatic speech recognition ASR model to generate a second speech recognition result (224) of the utterance.

Description

[Technical Field]

[0001] The present disclosure relates to using sub-models to account for contextual bias in the results of a base machine learning model. [Background technology]

[0002] Automatic speech recognition (ASR) is a category of natural language processing (NLP) that involves processing audio comprising human speech. Automatic speech recognition (ASR) models are often used to recognize and / or convert spoken language into text. One way to generate an automatic speech recognition (ASR) model is to use machine learning to train the model on a large dataset. Due to the amount of data used for training and the time it takes to train, automatic speech recognition (ASR) models are typically generalized to many domains and users, which reduces the flexibility of the model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Chinese Patent No. 112992127 Summary of the Invention [Problem to be solved by the invention]

[0004] Attempting to make automatic speech recognition (ASR) models more flexible, such as by using several smaller models, can be computationally expensive (e.g., due to redundancy in training multiple models) or produce distorted results (e.g., models with less training data are less robust). [Means for solving the problem]

[0005] One aspect of the present disclosure provides a computer-implemented method for biasing speech recognition results based on context. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations including obtaining a base automatic speech recognition (ASR) model trained with unbiased data. The operations include obtaining a sub-model trained on biased data, the biased data representing a particular domain. The operations further include receiving a speech recognition request including audio data characterizing an utterance captured in streaming audio. The operations include determining whether the speech recognition request includes a context indicator indicating the particular domain. If the speech recognition request does not include the context indicator, the operations include generating a first speech recognition result for the utterance by processing the audio data using the base automatic speech recognition ASR model. If the speech recognition request includes a context indicator, the operations include biasing a base automatic speech recognition ASR model toward the particular domain using the sub-model, and generating a second speech recognition result of the utterance by processing the audio data using the biased base automatic speech recognition ASR model, where the second speech recognition result is biased toward one or more terms of the particular domain.

[0006] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the context indicator comprises a one-hot vector. In these implementations, the one-hot vector may indicate a particular sub-model from a plurality of sub-models to be activated, each sub-model of the plurality of sub-models being associated with a different region. In these implementations, the operations may further include projecting the one-hot vector onto an embedding of the phrase set in the embedding space. These implementations may further include projecting the one-hot vector onto the embedding of the phrase set, whereby the embedding of the phrase set activates a portion of the sub-models.

[0007] In some embodiments, the sub-model is disposed in a layer of a base automatic speech recognition ASR model. In these embodiments, the base automatic speech recognition ASR model may include an encoder and a decoder, with the sub-model disposed between two layers of the encoder. Furthermore, one or more parameters of the base automatic speech recognition ASR may be frozen. In some embodiments, the operations further include retraining the sub-model based on a speech recognition request comprising audio data and a context indicator, and a transcript of the audio data. In some embodiments, the first speech recognition result differs from the second speech recognition result.

[0008] Another aspect of the present disclosure provides a system for biasing speech recognition results based on context. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a base automatic speech recognition (ASR) model trained with unbiased data. The operations include obtaining a sub-model trained on biased data, where the biased data represents a particular domain. The operations further include receiving a speech recognition request including audio data characterizing an utterance captured in streaming audio. The operations include determining whether the speech recognition request includes a context indicator indicating the particular domain. If the speech recognition request does not include the context indicator, the operations include generating a first speech recognition result for the utterance by processing the audio data using the base automatic speech recognition ASR model. If the speech recognition request includes a context indicator, the operations include biasing a base automatic speech recognition ASR model toward the particular domain using the sub-model, and generating a second speech recognition result of the utterance by processing the audio data using the biased base automatic speech recognition ASR model, where the second speech recognition result is biased toward one or more terms of the particular domain.

[0009] This aspect may include one or more of the following optional features: In some implementations, the context indicator comprises a one-hot vector. In these implementations, the one-hot vector may indicate a particular sub-model from a plurality of sub-models to be activated, each sub-model of the plurality of sub-models being associated with a different region. In these implementations, the operations may further include projecting the one-hot vector onto an embedding of the phrase set in the embedding space. These implementations may further include projecting the one-hot vector onto the embedding of the phrase set, whereby the embedding of the phrase set activates a portion of the sub-models.

[0010] In some embodiments, the sub-model is disposed in a layer of a base automatic speech recognition ASR model. In these embodiments, the base automatic speech recognition ASR model may include an encoder and a decoder, and the sub-model is disposed between two layers of the encoder. Further, one or more parameters of the base automatic speech recognition ASR may be frozen. In some embodiments, the operations further include retraining the sub-model based on a speech recognition request comprising audio data and a context indicator, and a transcript of the audio data. In some embodiments, the first speech recognition result differs from the second speech recognition result.

[0011] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of an exemplary system for a context bias system comprising an automatic speech recognition (ASR) model. [Figure 2A] FIG. 1 is a schematic diagram of an automatic speech recognition (ASR) model for generating biased speech recognition results by using sub-models. [Figure 2B] 1 is a schematic diagram of an automatic speech recognition (ASR) model for generating biased speech recognition results by using selected sub-models. [Figure 3A] 1 is a schematic diagram of an automatic speech recognition (ASR) model for generating biased speech recognition results by using sub-models as residual adapter layers. [Figure 3B] FIG. 1 is a schematic diagram of an automatic speech recognition (ASR) model for generating biased speech recognition results by using sub-models in the encoder layer. [Figure 4A] FIG. 1 is a schematic diagram of an automatic speech recognition (ASR) model that produces unbiased speech recognition results. [Figure 4B] FIG. 1 is a schematic diagram of an automatic speech recognition (ASR) model that uses sub-models to generate biased speech recognition results. [Figure 5A] FIG. 1 is a schematic diagram of an exemplary training scheme for an automatic speech recognition (ASR) model. [Figure 5B] FIG. 1 is a schematic diagram of an exemplary training scheme for sub-models for biasing speech recognition results based on context. [Figure 6] 10 is a flowchart of an exemplary configuration of operation of a method for contextual bias using sub-models. [Figure 7] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference symbols in the various drawings indicate like elements. Automatic speech recognition (ASR) is a growing field of language processing with a wide variety of applications, from automatic speech translation and transcription to processing voice commands for computing devices. Recently, neural networks for machine learning have been found to work well as the basis for automatic speech recognition (ASR) systems and models. Using machine learning techniques, it is possible to create robust models for speech recognition by training automatic speech recognition (ASR) models with large training datasets comprising speech audio samples. These automatic speech recognition (ASR) models generally grow in size because the more broadly the model is trained, the better its performance. However, using such large models has drawbacks, such as using a single model for a wide variety of users with different characteristics. For example, a single automatic speech recognition (ASR) model may be built for the English language, even though English speakers may have many different accents or colloquialisms based on region. As a result, the automatic speech recognition (ASR) model may not perform accurately for certain groups of users. Furthermore, due to its size and computational cost, it is difficult to retrain or update the model. This can cause automatic speech recognition (ASR) models to become outdated and perform poorly on new / emerging words / phrases (e.g., slang, new TV shows).

[0014] The lack of flexibility of large-scale automatic speech recognition (ASR) models hinders the potential viability of speech recognition because they may not perform well for a portion of the user base. In particular, these large-scale automatic speech recognition (ASR) models may be unable to utilize contextual signals that can help provide information about speech. As noted above, a user's location may provide information about accent or add / remove specific words or phrases that inform the automatic speech recognition (ASR) model output. In another example, if an alarm is sounding from a smart device, the user may utter a voice command related to the alarm (e.g., "stop alarm," "cancel," or "snooze"), which is more likely than usual. Current automatic speech recognition (ASR) models (e.g., large or generic models) are unable to use that contextual information to influence the output.

[0015] Previous attempts to "individualize" automatic speech recognition (ASR) models based on context have been difficult and can encounter problems during implementation. One technique for incorporating context information into an automatic speech recognition (ASR) model is by using several smaller automatic speech recognition (ASR) models, each relevant to a specific context or domain. However, training many smaller automatic speech recognition (ASR) models is computationally expensive, especially since much of the training is redundant across models. Furthermore, training each model from scratch can take weeks, making training many models time-consuming. Even when all automatic speech recognition (ASR) models are built and trained, some may perform poorly due to a lack of available training data, making model management unwieldy. Another way to consider context information is to modify an automatic speech recognition (ASR) model to receive bias terms that influence the results. However, this typically involves significant manual intervention (e.g., when building domain-specific models of users), can lead to catastrophic forgetting, and typical traffic in ASR models can cause bias terms to be over-triggered when they are not uttered, or may not trigger at all even when bias terms are present.

[0016] Embodiments herein are directed to a base automatic speech recognition ASR model that includes one or more sub-models that bias the base automatic speech recognition ASR model based on context so that the model's results or output are directed toward a specific context or domain. A sub-model is a set of parameters that can be added to or replaced by a general base automatic speech recognition ASR model. The sub-models can be loaded / enabled / disabled as needed, allowing the base automatic speech recognition ASR model to be used without bias, thereby eliminating concerns about catastrophic forgetting. In some examples, the base automatic speech recognition ASR model is trained, and then the parameters of the base automatic speech recognition ASR model are frozen during operation (i.e., inference). In this way, the base automatic speech recognition ASR model remains stable and can continue to function in general traffic or multiple domains. However, when context information is made available, the base automatic speech recognition ASR model is enabled to activate sub-models appropriate to the context information to bias speech recognition results toward terms related to the context information. One advantage of these embodiments includes the ability to train sub-models independently without the need to train or retrain a large base automatic speech recognition ASR model. Furthermore, one or more sub-models can be dynamically loaded and replaced in real time (e.g., by using a memory cache) for each request. The base automatic speech recognition ASR model remains unchanged regardless of whether a sub-model is used, eliminating concerns that its use and / or update will compromise the model.

[0017] As used herein, unless otherwise specified, the terms "speech recognition system" and "speech recognition model" can refer to any combination of automatic speech recognition (ASR) system / model in which speech is recognized and processed by a computing device. As will become apparent, the disclosed techniques for training automatic speech recognition (ASR) models, as well as automatic speech recognition (ASR) and sub-models, enable biasing of speech recognition based on contextual information.

[0018] FIG. 1 illustrates a context bias system 100 including an automatic speech recognition (ASR) model 200 and a sub-model 215. The automatic speech recognition ASR model 200 is configured to process a speech recognition request 105 using the sub-model 215. The speech recognition request 105 includes input audio data 102 corresponding to an utterance 108 generated by a source speaker 104 and captured by a user device 110. The speech recognition request 105 may also include a context indicator 103. Using the audio data 102 and the context indicator 103, the automatic speech recognition ASR model 200 and the sub-model 215 generate or predict an unbiased speech recognition result 222 as an unbiased speech recognition result or a biased speech recognition result 224 as a biased speech recognition result. The biased speech recognition result 224 is likely to include words or phrases related to a region to which the automatic speech recognition ASR model 200 is biased using the sub-model 215 (e.g., based on the context indicator 103). In some examples, the input audio data 102 comprises an input spectrogram corresponding to the utterance 108. The context indicator 103 may provide an indication of a particular region among a plurality of different regions, each of which signifies or represents a respective context of the utterance 108. In some implementations, the speech recognition results 222, 224 comprise probability density functions 226, 226A-226B representing the density of probabilities of text transformations of the utterance 108.

[0019] Although not shown, an acoustic front end present in the user device 110 is enabled to convert the time-domain audio waveform of the speech 108 captured via the microphone of the user device 110 into an input spectrogram 102 or other type or format of audio data 102. Additionally, the front end device may be configured to determine or obtain data representing context indicators 103 affecting the speech 108 and / or other relevant information corresponding to the source speaker 104 and / or the client device 110.

[0020] A user device 110 associated with a source speaker 104 may capture an utterance 108 produced by the source speaker 104 and provide corresponding input audio data 102 to the context bias system 100 as part of a speech recognition request 105. The user device 110 may further determine a context indicator 103 to include in the speech recognition request 105. The user device 110 may include, but is not limited to, a smartphone, a tablet, a desktop / laptop computer, a smart speaker, a smart display, a smart appliance, an assistant-enabled wearable device (e.g., a smart watch, smart headphones, smart glasses, etc.), or a vehicle infotainment system. Alternatively, a remote server 112 may be enabled to determine the context indicator 103 by processing the audio data 102 and any other additional data or metadata from the user device 110.

[0021] The context-biasing system 100 may be distributed across multiple devices, such that the automatic speech recognition ASR model 200 resides on one of the user devices 110 or on a remote system 150 (also referred to herein as a cloud computing environment) that communicates with the user device 110 via the network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 152 comprising computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A data store 158 (i.e., a remote storage device) may overlay the storage resources 146, allowing for scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The automatic speech recognition ASR model 200 and sub-models 215 may execute on the remote system 150 or the user device 110. The sub-models 215 may be stored locally on the user device 110, or may be stored on a remote system (eg, in the data store 158), or some combination therebetween.

[0022] The context bias system 100 generates biased speech recognition results 224 based on a received speech recognition request 105 by implementing a dynamic model comprising two or more component parts, or by implementing a model comprising a general base model (e.g., the automatic speech recognition ASR model 200) and “swappable” sub-models 215. The automatic speech recognition ASR model 200 can be trained with a large set of speech data. Once trained, the automatic speech recognition ASR model 200 can be frozen so that the parameters of the automatic speech recognition ASR model 200 remain constant during operation. The automatic speech recognition ASR model 200 can be updated, retrained, or replaced as needed or when additional training data becomes available. In some implementations, a single sub-model 215 is used to bias the automatic speech recognition ASR model 200 by activating specific parameters based on the context indicators 103 of the speech recognition request 105. In other embodiments, system 100 includes multiple submodels 215, each corresponding to a specific domain. In these embodiments, a specific submodel 215 is selected / activated based on context indicator 103. For example, if context indicator 103 indicates that a domain including a user's alarm is enabled, each submodel 215 is selected to bias specific terms (e.g., snooze, stop, etc.) associated with the indicated domain (i.e., user's alarm). If speech recognition request 105 does not include context indicator 103, or if a submodel 215 corresponding to the context indicator does not exist or is still in development, automatic speech recognition ASR model 200 may, in some instances, generate unbiased speech recognition results 222. That is, unbiased speech recognition results 222 are generated solely by automatic speech recognition ASR model 200 and are not influenced or affected by any submodels 215.

[0023] The context indicators 103 may be based on any signal or data that can be used to improve the accuracy of the biased speech recognition results 224. The context indicators 103 may be based on information related to the source speaker 104. For example, the source speaker 104 may have a particular dialect, native language, mannerisms, speaking patterns, speech disorders, etc. Thus, the system 100 may generate and train sub-models 215 corresponding to the source speaker 104, which bias / personalize the automatic speech recognition ASR model 200 to make predictions that are particularly suited to the source speaker 104. In some implementations, the context indicators 103 comprise one-hot vectors, and the system uses the context indicators 103 to select an appropriate sub-model 215 (i.e., the sub-model 215 corresponding to the source speaker 104) from multiple different sub-models 215.

[0024] In some implementations, the context indicator 103 is based on information related to the user device 110. For example, the user device 110 may include a smart device equipped with sensors such as a GPS, an accelerometer, a gyroscope, a microphone, a proximity sensor, a camera, etc. The context indicator 103 may indicate a region associated with the user device 110 as inferred from one of the sensors. For example, the context indicator 103 may indicate the geographic location of the user device 110 as inferred from GPS data (with the user 104 explicitly consenting to sharing their geographic location, which may be overridden at any time). Here, the context indicator 103 may correspond to a more geographic location (e.g., a city such as Chicago) or a more specific location (e.g., a gym). In either example, the sub-model 215 may bias the biased speech recognition results 224 toward a specific region based on the location. In particular, a context indicator 103 identifying a location in Chicago may activate a sub-model 215 trained with data from users from that city, state, and / or region, which may be biased based on accents or other characteristics of the speech of users from that area and named entities unique to that region (e.g., restaurants, sports teams, street names, etc.). As a result, the sub-model 215 is enabled to bias the automatic speech recognition ASR model 200 to generate biased speech recognition results 224 biased toward predictions that fit the region. For example, the biased speech recognition results 224 may increase the likelihood that the automatic speech recognition ASR model 200 predicts that the utterance 108 contains a reference to a restaurant or street in Chicago. Similarly, a context indicator 103 indicating that the source speaker 104 is at a gym may activate a sub-model 215 trained based on speech from users exercising or in a similar location. Here, speech may be affected by labored breathing or may be directed to specific words or phrases (e.g., voice commands for operating a music player on a smart device).Thus, the sub-model 215 is enabled to take these contextual factors into account when processing the audio data 102 to generate the biased speech recognition result 224. In another example, the context indicator 103 may indicate a software application currently running on the user device 110, such as a music player application. In this example, the context indicator 103 identifies the sub-model 215 corresponding to the software application (e.g., music player) indicated by the context indicator 103, such that the automatic speech recognition ASR model 200 recognizes terms / phrases such as "next song" or "pause." In other examples, the sub-model 215 is generally trained for that type of application or domain. Thus, if the source speaker 104 utters the utterance 108 "pause playback," the sub-model 215 will cause the biased automatic speech recognition ASR model 200 to generate a biased speech recognition result 224 that is skewed or biased toward a result directed toward the music player, compared to the unbiased speech recognition result 222 that would have been determined without activating the sub-model 215.

[0025] In some implementations, the context indicator 103 indicates that multiple domains are applicable to the utterance 108. In this scenario, a single sub-model 215 may bias the automatic speech recognition ASR model 200 to generate biased speech recognition results 224 toward each of the multiple domains. Alternatively, multiple sub-models 215 (each trained for one or more of the multiple domains) may bias the automatic speech recognition ASR model 200 to generate biased speech recognition results 224 toward each of the multiple domains. For example, when the source speaker 104 is located at a gym and a music player is running on the user device, one or more sub-models 215 may bias the output of the automatic speech recognition ASR model 200 toward each of these domains.

[0026] Output 190 is enabled to receive unbiased speech recognition results 222 and biased speech recognition results 224 generated by automatic speech recognition ASR model 200. In some examples, output 190 comprises natural language understanding (NLU) that performs query interpretation on the speech recognition results. The natural language understanding NLU is further enabled to instruct downstream applications / services to take action based on the results. Output 190 may also include a user interface generator configured to display the speech recognition results as a transcription on the screen of user device 110 and / or another device.

[0027] 1 is presented for illustrative purposes only and is not intended to be limiting. For example, while only a single instance of each component is shown, system 100 can include any number of components (110, 112, 140, 150, 200, and 215). Furthermore, while some components are described as being located in cloud computing environment 150, in some implementations, these components may be hosted locally at user device 110. Furthermore, in various implementations, some or all of the components (112, 200, and 215) are hosted locally at user device 110, remotely (such as in cloud computing environment 150), or some combination thereof.

[0028] Referring now to FIG. 2A , an exemplary automatic speech recognition (ASR) model 200 implements a sub-model 215 to generate biased speech recognition results 224 (e.g., probability density functions 226). Here, the automatic speech recognition (ASR) model 200 receives a speech recognition request 105 comprising audio data 102 and a context indicator 103. In this example, the sub-model 215 comprises a single model trained with various inputs and contexts corresponding to different domains. The context indicator 103 may be a one-hot vector indicating one or more specific domains corresponding to the context of the audio data 102. The one-hot vector may be concatenated and projected onto a phrase set embedding before being sent to the sub-model 215. In some implementations, the one-hot vector is looked up in an embedding matrix before being projected onto the phrase set embedding. Alternatively, the sub-model 215 may project the one-hot vector onto the phrase set embedding. The sub-model is then enabled to process the audio data 102 by activating one or more parameters corresponding to one or more particular regions indicated by the context indicators 103 based on the embedding of the phrase set.

[0029] Referring now to FIG. 2B , another exemplary automatic speech recognition (ASR) model 200 implements selected submodels 215, 215B to generate biased speech recognition results 224 (e.g., probability density function 226). Here, the automatic speech recognition (ASR) model 200 receives a speech recognition request 105 comprising audio data 102 and a context indicator 103. In this example, the context indicator 103 is received by a submodel selector 218. The submodel selector 218 selects a particular submodel 215 from multiple submodels 215, 215a-215n, each submodel 215 corresponding to one or more different domains. The submodels 215a-215n may be trained with data reflecting any domain that provides a context for processing the audio input 102. For example, the selected submodel 215b is based on a domain such as location, activity, user profile, software application, or any other domain that may affect the audio data 102. The context indicator 103 may be a one-hot vector that indicates a particular region corresponding to the context of the speech recognition request 105. Upon receiving the context indicator 103, the sub-model selector 218 may perform a lookup to determine the appropriate sub-model(s) 215 to use in processing the speech recognition request 105. The various sub-models 215a-215n are easily swappable, such that when processing streaming audio input 102 from the user device 110, the automatic speech recognition ASR model 200 can swap in and out various sub-models 215 from the plurality of sub-models 215a-215n (i.e., disable or enable biases imparted by the sub-models 215) while processing the streaming input when a new context indicator 103 is received.If the speech recognition request 105 does not include a context indicator 103, or if the context indicator 103 is not applicable to any of the available sub-models 215, the automatic speech recognition ASR model 200 processes the audio input 102 without activating or enabling any of the sub-models 215 to produce an unbiased speech recognition result 222 (FIG. 1).

[0030] The sub-model 215 may be implemented to bias the output of the automatic speech recognition ASR model 200 in any number of ways. FIG. 3A shows a schematic diagram (300a) of the automatic speech recognition ASR model 200 for generating the biased speech recognition result 224 by using the sub-model 215 implemented as a residual adapter layer. The automatic speech recognition ASR model 200 may be a recurrent neural network (RNN) comprising an encoder 310 configured to encode the input audio data 102 into a coded output 312 (e.g., a hidden feature representation comprising a series of vectors) and a decoder 320 configured to decode the coded output 312 into the biased speech recognition result 224. Typically, the coded output 312 is sent directly to the decoder 320 to generate the biased speech recognition result 224. However, in this example, the sub-model 215 operates in parallel with processing the speech recognition request 105. In time, the sub-model 215 is enabled to generate a sub-model output 325 based on the received audio input 102 of the speech recognition request 105. The automatic speech recognition ASR model 200 is enabled to merge the sub-model output 325 and the encoded output 312 to generate a biased encoded output 314 and send it to the decoder 320.

[0031] In another example, the sub-model 215 is implemented between layers of the encoder 310. FIG. 3B shows an exemplary schematic diagram (300b) of such an implementation. The encoder 310 can include several components 360, and the sub-model 215 can be placed between layers of the components 360 so that the encoder 310 generates a biased encoded output 319. The encoder components 360 can include a stack of multi-head attention blocks (i.e., conformer blocks), which can include a conformer or a transformer. In some implementations, each multi-head attention block includes a multi-head attention mechanism. Instead of multi-head attention blocks, the encoder 310 can include a stack of long short-term memories (LSTMs). The decoder 320 receives the biased encoded output 319 and can generate a biased speech recognition result 224.

[0032] The above examples in Figures 3A and 3B are for illustrative purposes only and are not intended to be limiting. The automatic speech recognition ASR model 200 and the sub-model 215 perform speech recognition in response to the context indicator 103 and may include any suitable structure / architecture for generating biased speech recognition results. Furthermore, the sub-model 215 and the automatic speech recognition ASR model 200 may function in any suitable combination to generate the biased speech recognition result 224. For example, the sub-model 215 may be located within the automatic speech recognition ASR model 200 at any suitable location in the architecture of the automatic speech recognition ASR model 200. For example, the sub-model 215 may be deployed within a layer of the automatic speech recognition ASR model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a prediction network, as a combination network, etc. Alternatively, sub-model 215 and automatic speech recognition ASR model 200 can generate outputs independently of one another, with the results being combined by automatic speech recognition ASR model 200, or another suitable component of the system, to determine biased speech recognition result 224. Notably, when sub-model 215 is disabled from its original frozen state, automatic speech recognition ASR model 200 remains unchanged. That is, when sub-model 215 is disabled (e.g., because of the absence of context indicator 103), automatic speech recognition ASR model 200 generates unbiased speech recognition result 222 that is not affected by sub-model 215.

[0033] 4A shows a schematic diagram (400a) of an automatic speech recognition ASR model 200 that generates a first probability density function 226, 226A corresponding to an unbiased speech recognition result 222. Here, a source speaker 104 utters an utterance 108 (“pause playback”) that is captured by a user device 110. The user device 110 sends a speech recognition request 105 (comprising an audio input 102 characterizing the utterance 108) to the automatic speech recognition ASR model 200. The automatic speech recognition ASR model 200 processes the audio input 102 to generate the unbiased speech recognition result 222. In some implementations, the automatic speech recognition ASR model 200 determines whether the speech recognition request 105 includes a context indicator 103 before proceeding to generate the unbiased speech recognition result 222 (e.g., the first probability density function 226A). In this example, the automatic speech recognition ASR model 200 predicts with a relatively low probability 410 that the audio input 102 comprises the phrase "pause playback," making it unlikely that the utterance 108 will be accurately transcribed.

[0034] 4B shows a schematic diagram (400b) of another exemplary automatic speech recognition ASR model 200 that uses sub-models 215 to generate second probability density functions 226, 226B corresponding to biased speech recognition results 224. As in FIG. 4A , when a source speaker 104 utters an utterance 108 (“pause playback”), which is captured by a user device 110, it is then included in a speech recognition request 105 as audio input 102 characterizing the utterance 108 to the automatic speech recognition ASR model 200. Here, the speech recognition request 105 includes a context indicator 103. In this example, the context indicator 103 corresponds to a music player application running on the user device 110. The automatic speech recognition ASR model 200 activates a sub-model 215 corresponding to a region of the music player, and the output of the automatic speech recognition ASR model 200 is biased toward that region. For example, submodel 215 may subsequently bias automatic speech recognition ASR model 200 toward words or phrases associated with music players, such as "stop," "play," "pause," artist names, song titles, etc. Automatic speech recognition ASR model 200 generates biased speech recognition results 224, as indicated by second probability density function 226B. As shown, the bias imparted by submodel 215 has "shifted" second probability density function 226B relative to first probability density function 226A toward the terms defined by the domain of submodel 215 ( FIG. 4A ), such that automatic speech recognition ASR model 200 predicts that audio input 102 contains the phrase "pause playback" with a high probability (412).

[0035] That is, biased speech recognition results 224 differ from unbiased speech recognition results 222. For example, even for audio input 102 in which first probability density function 226A has high confidence around a single word or term, second probability density function 226B may have even higher confidence around the single word or term if the word or term is within a particular region associated with sub-model 215. In some instances, bias changes the probability density of biased speech recognition results, where second probability density function 226B has a steeper slope than first probability density function 226A, indicating that in that instance the distribution is concentrated around a smaller number of possibilities.

[0036] 4A and 4B are for illustrative purposes only and are not intended to be limiting. For example, the speech recognition results 222, 224 may be in any suitable format, such as a transcript, a spectrogram, etc. In some implementations, the speech recognition results 222, 224 are generated as instructions for a computing device to perform an action (e.g., an instruction to pause a music application running on the user device 110).

[0037] FIG. 5A shows a training process 500 for the automatic speech recognition ASR model 200. In some implementations, the process 500a employs a two-step training technique comprising pre-training and training. Model pre-training is a technique used to initialize the model. The model may then be further fine-tuned based on additional training data 510. For the automatic speech recognition ASR model 200, pre-training may include starting the automatic speech recognition ASR model 200 with pre-training data 505 comprising a plurality of spoken utterances by one or more speakers. The pre-training data 505 may further include the spoken utterances paired with corresponding ground truth synthesized speech representations of the spoken utterances. The speech samples used for pre-training may be speech synthesized from reference transcripts of given speech and / or non-synthesized speech samples spoken by actual humans.

[0038] Process 500a is enabled to fine-tune parameters of pre-trained automatic speech recognition ASR model 200 after pre-training is complete. Training process 500a may include, for example, training encoder 310 and / or decoder 320 (FIG. 3A) individually or jointly in any suitable combination. Process 500 includes providing training input 510 to automatic speech recognition ASR model 200. In some implementations, training input 510 comprises a plurality of speech samples produced by a variety of different speakers. Further, training input 510 may be labeled using labels 520 indicating target outputs associated with training input 510. Upon receiving training input 510, automatic speech recognition ASR model 200 is enabled to generate output 515 (e.g., unbiased speech recognition results 222). Automatic speech recognition ASR model 200 is enabled to process training input 510 in a manner described with respect to any of FIGS. 2A-4B or any other suitable manner for speech recognition.

[0039] In some implementations, output 515 is used by loss function 530 to generate loss 540. That is, loss function 530 compares output 515 with label 520 to generate loss 540, which indicates the discrepancy between label 520 (i.e., target output) and output 515. Loss function 530 may implement any suitable technique to determine loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross-entropy, hinge loss, multi-class loss, etc. Loss 540 may then be fed directly to automatic speech recognition ASR model 200, which processes loss 540 and adjusts one or more parameters of automatic speech recognition ASR model 200 to take loss 540 into account. In some implementations, once automatic speech recognition ASR model 200 is properly trained, the model is frozen. In other words, the parameters remain unchanged for a period of time until it is determined that the automatic speech recognition ASR model 200 needs to be retrained or replaced (e.g., when sufficient new training data 510 has been acquired).

[0040] FIG. 5B illustrates a training process 500b for the sub-model 215. In some implementations, the process 500b employs a two-step training technique comprising pre-training and training. Pre-training is a technique used to initialize the model, which may then be further fine-tuned based on additional training data 560. For the sub-model 215, pre-training may include starting the sub-model 215 with pre-training data 555 comprising a plurality of spoken utterances by a speaker. The pre-training data 555 may further include the spoken utterances paired with corresponding ground truth synthesized speech representations of the spoken utterances. The speech samples used for pre-training may be speech synthesized from a reference transcript for a given speech. For the sub-model 215, pre-training may further include adjusting one or more parameters of the sub-model 215 such that the sub-model 215 is biased toward words and / or phrases corresponding to a particular domain associated with the sub-model 215.

[0041] In some examples, the process 500b includes fine-tuning parameters of a pre-trained sub-model 215. In these examples, the process 500b includes providing training inputs 560 to the sub-model 215. In some implementations, the training inputs 560 include a plurality of speech samples produced by various speakers. The training inputs 560 may also include context indicators 103 and transcripts comprising each speech sample. The transcripts may be manually generated text representing the corresponding speech samples. In some implementations, the transcripts are machine-generated. The transcripts must accurately reflect the corresponding speech samples so that the transcripts are the target output of the sub-model. In some implementations, the context indicators 103 of the training inputs 560 may include domains and / or terms. In other implementations, the context indicators 103 of the training inputs 560 may include one-hot vectors. The sub-models 215 corresponding to the context indicators 103 may be trained with the speech samples and transcripts such that the sub-models 215 are biased toward terms or phrases corresponding to the domains associated with the context indicators 103. In the example of a single sub-model 215 adapted to bias multiple regions, the one-hot vector of the context indicators 103 may be concatenated and projected onto the phrase set embeddings, which may then be used to train the sub-model 215. Further, the training inputs 560 may be labeled with labels 570 indicating the target outputs associated with the training inputs 560. Upon receiving the training inputs 560, the sub-model 215 may generate outputs 565 (e.g., biased speech recognition results). The sub-model 215 may be enabled to process the training inputs 560 in the manner described with respect to any of Figures 2A-4B or any other suitable manner for speech recognition.

[0042] In some implementations, the output 565 is used by a loss function 580 to generate a loss 590. That is, the loss function 580 compares the output 565 with the label 570 to generate the loss 590. The loss 590 indicates the discrepancy between the label 570 (i.e., the target output) and the output 565. The loss function 580 is enabled to implement any suitable technique to determine the loss, such as a regression loss, a mean squared error, a mean squared logarithmic error, a mean absolute error, a binary classification, a binary cross-entropy, a hinge loss, a multi-class loss, etc. The loss 590 may then be fed directly to the sub-model 215, which processes the loss 590 and adjusts one or more parameters of the sub-model 215 to take the loss 590 into account. In some implementations, the sub-model 215 is continuously trained (or retrained) as additional speech recognition requests are received. For example, while the automatic speech recognition ASR model 200 is frozen, the sub-models 215 can continue to adjust parameters based on received speech recognition requests 105 comprising audio data 102, context indicators 103, and biased speech recognition results 224. Here, in the case of multiple sub-models 215, each sub-model 215 can be trained during use by the automatic speech recognition ASR model 200 to bias the associated region.

[0043] While the examples herein are directed to sub-models 215 biasing automatic speech recognition (ASR) model 200 for speech detection, it is understood that sub-models 215 and context indicators 103 can be used to bias any type of model used for any purpose. For example, sub-models 215 can bias an image recognition model, a recommendation model, a filtering (e.g., email) model, a medical diagnosis model, or any other model that can use context information to bias results and improve accuracy. As described above, sub-models 215 can be trained with appropriate context indicators 103 to appropriately bias the underlying base model.

[0044] FIG. 6 is a flowchart of an exemplary configuration of operations for a method 600 of context biasing the results of an automatic speech recognition (ASR) model 200 using a sub-model 215. The operations of the method 600 may be performed by data processing hardware 710 ( FIG. 7 ), which may be present in the user device 110 or may include data processing hardware 154 present in the remote system 150. Thus, the method 600 may be performed, for example, by various elements of the context bias system 100 of FIG. 1 . At operation 602, the method 600 comprises obtaining a base automatic speech recognition (ASR) model 200 trained with unbiased data 510. At operation 604, the method 600 comprises obtaining a sub-model 215 trained with biased data 560. The biased data 560 represents a particular domain. At operation 606, the method 600 comprises receiving a speech recognition request 105 comprising audio data 102 characterizing speech 108 captured in streaming audio. At operation 608, the method 600 includes determining whether the speech recognition request 105 includes a context indicator 103 that indicates a particular domain. At operation 610, when the speech recognition request 105 does not include the context indicator 103, the method 600 includes generating a first speech recognition result 222 (i.e., an unbiased speech recognition result 222) for the utterance 108 by processing the audio data 102 using the base automatic speech recognition ASR model 200. At operation 612, if the speech recognition request 105 includes the context indicator 103, the method 600 includes biasing the base automatic speech recognition ASR model 200 toward the particular domain using the sub-model 215, and generating a second speech recognition result 224 (i.e., a biased speech recognition result 224) for the utterance 108 by processing the audio data 102 using the biased base automatic speech recognition ASR model 200. The second speech recognition result 224 is biased toward one or more terms of the particular domain.

[0045] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0046] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the invention as described and / or claimed in this document.

[0047] Computing device 700 includes processor 710, memory 720, storage device 730, high-speed interface / controller 740 connecting to memory 720 and high-speed expansion port 750, and low-speed interface / controller 760 connecting to low-speed bus 770 and storage device 730. The components (710, 720, 730, 740, 750, and 760) are interconnected using various buses and may reside on a common motherboard or exist in other ways as needed. Processor (e.g., data processing hardware) 710 processes instructions for execution within computing device 700, including instructions stored in memory 720 (e.g., memory hardware) or storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as display 780 connected to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Also, multiple computing devices 700 may be connected together (eg, as a server bank, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0048] Memory 720 stores information non-transiently within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). Non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0049] Storage device 730 is enabled to provide mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various different implementations, storage device 730 may be a device array comprising a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied on an information carrier. The computer program product comprises instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0050] The high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 760 manages low-bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is coupled to memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (USB, Bluetooth, Ethernet, Wireless Ethernet, etc.) and may connect, via a network adapter or the like, to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as a switch or router.

[0051] Computing device 700 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 700a, or multiple times in a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0052] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general-purpose, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0053] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0054] These computer programs (also known as programs, software, software applications, or code) comprise machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor comprising a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0056] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube) LCD (liquid crystal display) monitor or a touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, verbal, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0057] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (600) that, when executed by data processing hardware (710), causes the data processing hardware (710) to perform operations, the operations comprising: Obtaining a base automatic speech recognition (ASR) model trained with unbiased data; Obtaining a sub-model (215) trained with biased data, the biased data representing a particular region; receiving a speech recognition request (105) comprising audio data (102) characterizing a captured utterance (108) in streaming audio; determining whether the speech recognition request (105) includes a context indicator (103) that indicates the specific region; if the speech recognition request (105) does not include the context indicator (103), generating a first speech recognition result (222) of the utterance (108) by processing the audio data (102) using the base automatic speech recognition (ASR) model (200); and When the speech recognition request (105) includes the context indicator (103), biasing the base automatic speech recognition (ASR) model (200) toward the specific region using the sub-model (215); and generating a second speech recognition result (224) of the utterance (108) by processing the audio data (102) using the biased base automatic speech recognition ASR model (200), the second speech recognition result (224) being biased toward one or more terms of the specific domain; A computer-implemented method (600) comprising: the context indicator (103) comprises a one-hot vector; A computer-implemented method (600).

2. the one-hot vector indicates a particular submodel (215) from a plurality of submodels (215) to be activated; Each sub-model (215) of the plurality of sub-models (215) is associated with a different region. The computer-implemented method (600) of claim 1.

3. The operations further comprise projecting the one-hot vector onto an embedding of the phrase set in an embedding space. The computer-implemented method (600) of claim 1.

4. projecting the one-hot vector onto the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model (215); The computer-implemented method (600) of claim 3.

5. The sub-models (215) are arranged in layers of the base automatic speech recognition (ASR) model (200). The computer-implemented method (600) of any one of claims 1 to 4.

6. The base automatic speech recognition (ASR) model (200) comprises an encoder (310) and a decoder (320); The sub-model (215) is disposed between two layers of the encoder (310). The computer-implemented method (600) of claim 5.

7. one or more parameters of the base automatic speech recognition ASR model are frozen when generating the first and second speech recognition results (222, 224); The computer-implemented method (600) of any one of claims 1 to 4.

8. the data processing hardware (710) is present on a user device (110) that captured the streaming audio utterance (108); or The data processing hardware (710) resides in a remote system (150) that communicates with the user device (110) over a network. The computer-implemented method (600) of any one of claims 1 to 4.

9. The first speech recognition result (222) is different from the second speech recognition result (224). The computer-implemented method (600) of any one of claims 1 to 4.

10. A system (100) comprising: Data processing hardware (710); memory hardware (720) in communication with said data processing hardware (710); It is equipped with The memory hardware (720) stores instructions that, when executed by the data processing hardware (710), cause the data processing hardware (710) to perform operations, the operations being: Obtaining a base automatic speech recognition (ASR) model trained with unbiased data; Obtaining a sub-model (215) trained with biased data, the biased data representing a particular region; receiving a speech recognition request (105) comprising audio data (102) characterizing a captured utterance (108) in streaming audio; determining whether the speech recognition request (105) includes a context indicator (103) that indicates the specific region; if the speech recognition request (105) does not include the context indicator (103), generating a first speech recognition result (222) of the utterance (108) by processing the audio data (102) using the base automatic speech recognition (ASR) model (200); and When the speech recognition request (105) comprises the context indicator (103), biasing the base automatic speech recognition (ASR) model (200) toward the specific region using the sub-model (215); and generating a second speech recognition result (224) of the utterance (108) by processing the audio data (102) using the biased base automatic speech recognition ASR model (200), the second speech recognition result (224) being biased toward one or more terms of the specific domain; A system (100) comprising: the context indicator (103) comprises a one-hot vector; System (100).

11. the one-hot vector indicates a particular submodel (215) from a plurality of submodels (215) to be activated; Each sub-model (215) of the plurality of sub-models (215) is associated with a different region. The system (100) of claim 10.

12. The operations further comprise projecting the one-hot vector onto an embedding of the phrase set in an embedding space. The system (100) of claim 10.

13. projecting the one-hot vector onto the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model (215); The system (100) of claim 12.

14. The sub-models are arranged in layers of the base automatic speech recognition (ASR) model (200). A system (100) according to any one of claims 10 to 13.

15. The base automatic speech recognition (ASR) model (200) comprises an encoder (310) and a decoder (320); The sub-model is disposed between two layers of the encoder (310). The system (100) of claim 14.

16. one or more parameters of the base automatic speech recognition ASR model are frozen when generating the first and second speech recognition results (222, 224); A system (100) according to any one of claims 10 to 13.

17. the data processing hardware is present on a user device (110) that captured the streaming audio utterance (108); or the data processing hardware resides in a remote system (150) that communicates with the user device (110) over a network; A system (100) according to any one of claims 10 to 13.

18. The first speech recognition result (222) is different from the second speech recognition result (224). A system (100) according to any one of claims 10 to 13.

Images