Personalized Probabilistic Models

Abstract

The method (400) includes receiving audio (112) and using a first recognition model (130) to process the audio data and generate non-contextual candidate hypotheses (135) as output from the first recognition model. The non-contextual candidate hypotheses have corresponding likelihood scores (155) assigned by the first recognition model. The method further includes using a second recognition model (140) configured to receive personal context information (126) to generate contextual candidate hypotheses (145) including personal named entities (148). The method further includes scoring the contextual candidate hypotheses against the non-contextual candidate hypotheses based on the personal context information and the corresponding likelihood scores assigned to the non-contextual candidate hypotheses. The method further includes generating a transcription (175) of the utterance by selecting one of the contextual candidate hypotheses or the non-contextual candidate hypotheses.

Description

[Technical field]

[0001] The present disclosure relates to personalizable probabilistic models. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus on providing not only quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., short delay between when a user speaks and when a transcription appears). Due to the sparsity of the generalized training data used to train ASR systems in performing speech recognition tasks, ASR systems have difficulty recognizing specialized phrases, including medical / legal terms and proper nouns such as personal contacts, song / artist names, applications, and emerging entity names. The same observation makes it difficult for ASR systems to recognize terms in heterolingual speech in a language different from the language of the training data. As a result, speech recognition lattices rank these specialized phrases low or omit them entirely. Summary of the Invention

[0003] One aspect of the disclosure provides a method of scoring contextual candidate hypotheses against non-contextual candidate hypotheses, the method causing the data processing hardware to perform operations when executed. The operations include receiving audio data corresponding to an utterance spoken by a user, and processing the audio data using a first recognition model to generate a non-contextual candidate hypothesis as output from the first recognition model, where the non-contextual candidate hypothesis corresponds to a candidate transcription of the utterance and has a corresponding likelihood score assigned to the non-contextual candidate hypothesis by the first recognition model. The operations further include generating a contextual candidate hypothesis including a personal named entity using a second recognition model configured to receive personal context information, where the contextual candidate hypothesis corresponds to another candidate transcription of the utterance. The operations further include scoring the contextual candidate hypothesis against the non-contextual candidate hypothesis based on the personal context information and the corresponding likelihood score assigned to the non-contextual candidate hypothesis, and generating a transcription of the utterance spoken by the user by selecting one of the contextual candidate hypothesis or the non-contextual candidate hypothesis based on the scoring of the contextual candidate hypothesis against the non-contextual candidate hypothesis.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the first recognition model processes the audio data to generate non-contextual candidate hypotheses without incorporating any personal contextual information associated with a user. In some examples, the first recognition model includes an end-to-end speech recognition model configured to generate corresponding likelihood scores for the non-contextual candidate hypotheses.

[0005] In some implementations, the second recognition model includes an end-to-end speech recognition model configured to generate contextual candidate hypotheses, where generating contextual candidate hypotheses including the personal named entities includes processing the audio data using the end-to-end speech recognition model to generate contextual candidate hypotheses based on the personal context information. In these implementations, the end-to-end speech recognition model may be configured to generate the contextual candidate hypotheses after non-contextual candidate hypotheses are generated as output from the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate the contextual candidate hypotheses in parallel with processing the audio data to generate non-contextual candidate hypotheses as output from the first recognition model.

[0006] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user, where the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby place, or a user-defined entity name. In some examples, the personal context information associated with the user further indicates a user history associated with the personal named entity specific to the user.

[0007] In some implementations, the operations further include receiving a training corpus of ground truth transcriptions. For each corresponding ground truth transcription in the training corpus of ground truth transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding ground truth transcription, and replacing the personal named entity with a token of the entity class when the personal named entity is identified in the corresponding ground truth transcription. Here, the operations further include training the first recognition model with the ground truth transcriptions including the personal named entity replaced by the token of the entity class.

[0008] Another aspect of the disclosure provides a system for scoring contextual candidate hypotheses against non-contextual candidate hypotheses. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to an utterance spoken by a user and processing the audio data using a first recognition model to generate a non-contextual candidate hypothesis as output from the first recognition model, where the non-contextual candidate hypothesis corresponds to a candidate transcription of the utterance and has a corresponding likelihood score assigned to the non-contextual candidate hypothesis by the first recognition model. The operations further include generating a contextual candidate hypothesis including a personal named entity using a second recognition model configured to receive personal context information, where the contextual candidate hypothesis corresponds to another candidate transcription of the utterance. The operations further include scoring the contextual candidate hypotheses relative to the non-context candidate hypotheses based on the personal context information and corresponding likelihood scores assigned to the non-context candidate hypotheses, and generating a transcription of the utterance spoken by the user by selecting one of the contextual candidate hypotheses or the non-context candidate hypotheses based on the scoring of the contextual candidate hypotheses relative to the non-context candidate hypotheses.

[0009] This aspect may include one or more of the following optional features: In some implementations, the first recognition model processes the audio data to generate non-context candidate hypotheses without incorporating any personal contextual information associated with a user. In some examples, the first recognition model includes an end-to-end speech recognition model configured to generate corresponding likelihood scores for the non-context candidate hypotheses.

[0010] In some implementations, the second recognition model includes an end-to-end speech recognition model configured to generate contextual candidate hypotheses, where generating contextual candidate hypotheses including the personal named entities includes processing the audio data using the end-to-end speech recognition model to generate contextual candidate hypotheses based on the personal context information. In these implementations, the end-to-end speech recognition model may be configured to generate the contextual candidate hypotheses after non-contextual candidate hypotheses are generated as output from the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate the contextual candidate hypotheses in parallel with processing the audio data to generate non-contextual candidate hypotheses as output from the first recognition model.

[0011] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user, where the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby place, or a user-defined entity name. In some examples, the personal context information associated with the user further indicates a user history associated with the personal named entity specific to the user.

[0012] In some implementations, the operations further include receiving a training corpus of ground truth transcriptions. For each corresponding ground truth transcription in the training corpus of ground truth transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding ground truth transcription, and replacing the personal named entity with a token of the entity class when the personal named entity is identified in the corresponding ground truth transcription. Here, the operations further include training the first recognition model with the ground truth transcriptions including the personal named entity replaced by the token of the entity class.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0014] [Figure 1] 1 is an exemplary system for scoring a first candidate hypothesis generated for an utterance by a first recognizer model and a second candidate hypothesis generated for the same utterance by a second recognizer model. [Figure 2A] FIG. 2 is a schematic diagram showing an example of a word lattice. [Figure 2B] FIG. 2 is a schematic diagram showing an example of a word lattice. [Diagram 3] 2 is a schematic diagram illustrating a training process for training a first speech recognition model used by the system of FIG. 1; [Figure 4] 1 is a flowchart of an example arrangement of operations of a method for scoring a first candidate hypothesis generated for an utterance by a first speech recognition model and a second candidate hypothesis generated for the same utterance by a second speech recognition model. [Diagram 5]FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Automatic speech recognition (ASR) systems have become increasingly prevalent in client devices as they continue to provide more accurate transcriptions of what users say. Recently, end-to-end (E2E) ASR models have gained popularity by achieving state-of-the-art performance in accuracy and latency. In contrast to traditional hybrid ASR systems with separate acoustic, pronunciation, and language models, E2E ASR models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data, e.g., utterance-transcription pairs. Nevertheless, in some instances, ASR models misrecognize what a user actually spoke and produce inaccurate transcriptions. This is often the case when a user speaks a personal named entity that is rare or absent in the training data used to train the ASR model. As a result, the correct hypothesis for the personal named entity has a low rank or is completely missing in the recognition lattice of possible speech recognition hypotheses. As used herein, a personal named entity refers to any of the names of personal contacts, names in a media library (e.g., songs, artists, albums, movies, etc.), installed applications, as well as user-defined names (e.g., myLivingRoomStreamingDevice).

[0017] Existing techniques for improving the recognition of personal named entities involve biasing, which typically involves either favoring relevant speech recognition hypotheses that contain the personal named entity, or creating new candidate hypotheses that inject the personal named entity with close acoustic, graphemic, or embedding proximity to an existing hypothesis. However, an often overlooked consideration is how to accurately score contextual (user-specific or personal) candidate hypotheses for utterances that have a personal named entity against non-contextual candidate hypotheses for utterances that lack user-specific data and thus lack the personal named entity. If contextual hypotheses are scored in isolation from existing non-contextual hypotheses, as is typically the case when applying the existing biasing techniques described above, the ASR system runs the risk of over-triggering contextual hypotheses that were not spoken, or under-recalling non-contextual hypotheses even when spoken. For example, consider the case where an ASR model generates the non-contextual hypothesis "call lee oh need" for the utterance "call Leonid." Accurately scoring a new hypothesis, "Call Leonid," must take into account whether "Leigh-Oh Need" and "Leong" are personal contacts, whether either "Leigh-Oh Need" or "Leong" are common entities that are likely to be spoken in an utterance, the prior likelihood of the non-contextual hypothesis, and whether other, better non-contextual hypotheses exist.

[0018] In order to accurately rank contextual hypotheses in parallel with non-contextual hypotheses, the contextual hypotheses must be scored accurately. An ASR model that generated and scored non-contextual hypotheses is not suitable for scoring contextual hypotheses because it has already failed to recall or assigned low scores during decoding. On the other hand, context-specific models, such as bias models that bias speech recognition results against personal named entities, do not have the ability to score and rank speech recognition hypotheses that contain personal named entities against the general entities on which the ASR model is trained.

[0019] Embodiments herein are directed to a speech recognizer module that includes a first (generic) model (M1) that does not include any user-specific data, and a second (personal) model (M2) that is computed and dynamically updated for a given user. In particular, a computing system receives audio data of an utterance spoken by the user, and the computing system processes the audio data to generate non-contextual candidate hypotheses of the utterance as output from the first model (M1) and contextual candidate hypotheses of the utterance as output from the second model (M2). In some examples, the first model (M1) is configured to generate a plurality of non-contextual hypotheses that represent possible transcriptions for the utterance in a word lattice. As used herein, non-contextual candidate hypotheses refer to possible transcriptions for an utterance generated by the first model (M1) without incorporating any user-specific data or personal information associated with a given user. In this manner, the first model is effective for modeling the recognition of general entities spoken in essentially impersonal speech, but is generally not suitable for the recognition of personal named entities.

[0020] Meanwhile, the second model (M2) is configured to generate context candidate hypotheses including personal named entities by leveraging personal context information related to the user. The context information may indicate user history related to a given personal named entity, such as invocation frequency and usage patterns. More specifically, the second model (M2) corresponds to a user-specific model that assigns probabilities to possible personal named entities for a particular entity class based on the personal context information. In this way, the second model is effective for modeling the recognition of personal named entities spoken in an utterance. The embodiments herein are particularly directed to selecting a transcription of an utterance by combining the first model M1 and the second model M2 to determine likelihood scores of both non-contextual and contextual hypotheses, and selecting the highest likelihood score of either one of the non-contextual or contextual candidate hypotheses. As used herein, a likelihood score indicates the probability that the corresponding candidate hypothesis is a correct transcription of an utterance spoken by the user.

[0021] 1 illustrates an example of a system 100 for automatic speech recognition (ASR) of an utterance 101 spoken by a user 10 using audio data 112 corresponding to the utterance (i.e., query) 101. The system 100 includes a client device 110, a computing system 120, and a network 118. The computing system 120 may be a distributed system (e.g., a cloud computing environment) with scalable elastic resources. The resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 118 may be wired, wireless, or a combination thereof, and may include private and / or public networks, such as the Internet.

[0022] In some examples, the computing system 120 receives the audio data 112 from the client device 110, and the computing system 120 processes the audio data 112 to generate one or more non-contextual candidate hypotheses 135 for the utterance 101 based on the audio data 112 as output from the first recognizer model 130. In some further examples, the client device 110 processes the audio data 112 entirely on-device to generate one or more non-contextual candidate hypotheses 135 for the utterance 101 based on the audio data 112, where each non-contextual candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 generated without using any personal context information 126 and is represented by a respective sequence of hypothesized terms.

[0023] As described in more detail below, the second recognizer model 140 (running on the client device 110 or the computing system 120) leveraging the personal context information 126 generates a context candidate hypothesis 145 for the utterance 101 that was not included in the existing candidate hypotheses 135 or was ranked very low among the existing candidate hypotheses of the lattice 200. Here, the context candidate hypothesis 145 corresponds to a candidate transcription of the same utterance 101 generated by incorporating the personal context information 126. In this way, the personal context information 126 leveraged by the second recognizer model 140 assists in the recognition of the personal named entity. The scoring model 150 then scores the context candidate hypothesis 145 together with one of the non-context candidate hypotheses 135, and the computing system 120 (or the user device 110) generates a transcription 175 of the utterance 101 by selecting the highest ranked hypothesis among the non-context candidate hypotheses 135 or the context candidate hypotheses 145 scored by the scoring model 150.

[0024] 1 illustrates operations (A)-(G) illustrating the flow of data. As described herein, computing system 120 performs operations (B)-(G). However, it is understood that client device 110 may also perform operations (B)-(G) in addition to, or instead of, computing system 120 performing the operations. In some examples, client device 110 performs a first portion of an operation (e.g., operation (C)) and computing system 120 performs a second portion of the operation (e.g., operations (D)-(F)), or vice versa.

[0025] The client device 110 includes data processing hardware 114 and memory hardware 116. The client device 110 may include one or more audio capture devices (e.g., microphone(s)) 103 for capturing and converting speech 101 from the user 10 into audio data 112 (e.g., electrical signals). In some examples, the microphone 103 is separate from the client device 110 and communicates with the client device 110 to provide the recorded speech 101 to the client device 110. The client device 110 can be any computing device capable of communicating with a computing system 120 over a network 118. In lieu of spoken speech 101, the user 10 may input text utterances 101 by means of a keyboard 117, such as a virtual keyboard displayed in a graphical user interface of the client device 110 or a physical keyboard in communication with the client device 110. Client devices 110 include, but are not limited to, desktop computing devices, mobile computing devices such as laptops, tablets, smartphones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., headphones, headsets, and / or watches).

[0026] In the example of FIG. 1, during stage (A), a user 10 speaks an utterance 101, and a microphone 103 of a client device 110 records the utterance 101. In this example, the utterance 101 includes the user 10 saying "call Best Buy." The client device 110 transmits audio data 112 corresponding to the utterance 101 recorded by the microphone 103 to a computing system 120 via a network 118. During stage (B), the computing system 120 processes the audio data 112 to generate one or more non-contextual candidate hypotheses 135 as output from a first recognition model 130, where each candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 and is represented by a respective sequence of hypothesized terms. For example, the computing system 120 may execute a speech recognizer module 30 including a first recognition model 130 (e.g., an automatic speech recognition (ASR) module) to generate a word lattice 200 indicating possible non-contextual candidate hypotheses 135 for the utterance 101 based on the audio data 112 without incorporating any personal context information 126. The first recognition model 130 may evaluate potential paths through the word lattice 200 to determine the non-contextual candidate hypotheses 135.

[0027] Figure 2A shows an example of a word lattice 200, 200a that may be provided by the first recognition model M1 130 of Figure 1. The word lattice 200a represents multiple possible combinations of words that may form different non-context candidate hypotheses 135 for the utterance 101.

[0028] The word lattice 200a includes one or more nodes 202a-202g that correspond to possible boundaries between words. The word lattice 200a includes a number of edges 204a-l of possible words of candidate hypotheses arising from the word lattice 200a. Additionally, each of the edges 204a-204l may have one or more weights or probabilities that the edge is a correct edge from the corresponding node. The weights are determined by M1 130 and may be based, for example, on the confidence of the match between the speech data and the word of that edge, and how well the word fits grammatically and / or lexically with other words in the word lattice 200a.

[0029] For example, initially, the most likely path through the word lattice 200a (e.g., the most likely non-context candidate hypothesis 135) may include edges 204c, 204e, 204i, 204k with the text "we're coming about 11:30." A second optimal path through the word lattice 200a (e.g., the second optimal non-context candidate hypothesis 135) may include edges 204d, 204h, 204j, 204l with the text "deer hunting scouts 7:30."

[0030] Each pair of nodes may have one or more paths corresponding to alternative words in the various non-context candidate hypotheses 135. For example, the initial most likely path between a pair of nodes starting at node 202a and ending at node 202c is edge 204c "we're." This path has alternative paths that include edges 204a, 204b "we are" and edge 204d "deer."

[0031] Figure 2B is an example of a hierarchical word lattice 200, 200b that may be provided by the first recognition model 130 of Figure 1. The word lattice 200b includes nodes 252a-252l that represent words that make up the various non-context candidate hypotheses 135 for the utterance 101. The edges between the nodes 252a to 252l indicate that the possible candidate hypotheses 135 are (1) nodes 252c, 252e, 252i, and 252k “we’re coming about 11:30”, (2) nodes 252a, 252b, 252e, 252i, and 252k “we are cominng about 11:30”, (3) nodes 252a, 252b, 252f, 252g, 252i, and 252k “we are come at about 11:30”, and (4) nodes 252d, 252f, 252g, 252i, and 252k “deer come at about (5) nodes 252d, 252h, 252j, and 252k “deer hunting scouts 11:30”; and (6) nodes 252d, 252h, 252j, and 252l “deer hunting scouts 7:30.”

[0032] Again, the edges between nodes 242a-l may have associated weights or probabilities based on the confidence of the speech recognition (e.g., candidate hypotheses) and grammatical / lexical analysis of the resulting text. In this example, "we're coming about 11:30" may be the current best hypothesis, and "deer hunting scouts 7:30" may be the next best hypothesis. One or more partitions 354a-d may be created in the word lattice 200b that group words and their alternatives together. For example, partition 254a includes the word "we're" and the alternatives "we are" and "deer". Partition 252b includes the word "coming" and the alternatives "come at" and "hunting". Section 254c contains the word "about" and the alternative "scouts," and section 254d contains the word "11:30" and the alternative "7:30."

[0033] Referring again to FIG. 1 , the first recognition model M1 130 may generate a first decoding result 132 that models the non-contextual candidate hypotheses 135 from the word lattice 200 and corresponding likelihood scores 155 assigned to the non-contextual candidate hypotheses 135. Each likelihood score 155 indicates a probability that the candidate hypothesis 135 is correct (e.g., matches the utterance 101). In some implementations, the first recognition model M1 130 includes an end-to-end (E2E) speech recognition model configured to receive the audio data 112 and generate the word lattice 200. In particular, the E2E speech recognition model processes the audio data 112 to generate a first decoding result 132 that indicates a likelihood score 155 for each of the multiple non-contextual candidate hypotheses 135 from the word lattice 200. In some examples, the first recognition model 130 includes a separate acoustic model, language model, and / or pronunciation model.

[0034] In particular, because the first recognition model 130 corresponds to a generic speech recognizer trained on generic data, the first recognition model M1 130 is less accurate at recognizing speech in specialized domains that use specialized phrases (e.g., voice commands, medical terms, legal terms) and / or named entities (e.g., names of personal contacts, song / artist names, application names). The reduced accuracy of recognizing speech in these specialized domains is due to a lack of generic data used to train the speech recognizer module, foreign or rare words, the emergence of new entities, and out-of-vocabulary terms that were not present in the generic data.

[0035] In some examples, the speech recognizer module 130 includes an acoustic model and / or a language model for generating the word lattice 200 or otherwise generating a plurality of candidate hypotheses 135 for the utterance 101 based on the audio data 112, where the likelihood scores 155 of the plurality of candidate hypotheses 135 may include a combination of acoustic modeling scores from the acoustic model and / or prior likelihood scores from the language model. In other words, the likelihood scores 155 include at least one of the acoustic modeling scores output by the acoustic model and / or the prior likelihood scores output by the language model.

[0036] As described in more detail below with reference to FIG. 3, the first recognition model M1 130 may be trained as an aggregate without any user-specific data, and the first recognition model M1 130 may be personalized such that the first recognition model M1 130 recognizes the user context associated with the input utterance 101 but does not incorporate or use any user-specific or personal data associated with a given user when generating the first decoding result 132. For example, M1 130 may be trained on annotated utterances in which personal entities are identified and replaced with tokens 158. In some examples, different tokens 158 representing personal named entity classes are used to replace named entities that may belong to a particular named entity class. In the illustrated example, the first recognition model M1 130 also models / generates individual likelihood scores 157 for a given non-contextual candidate hypothesis 135 having a recognized named entity (e.g., Best Buy) replaced with a corresponding token 158 (e.g., $CONTACT). Other tokens 158 indicative of other personal named entity classes may be used, such as, but not limited to, application (e.g., $APP), song name (e.g., $SONG), artist name (e.g., $ARTIST), and album name (e.g., $ALBUM). Thus, for the non-contextual candidate hypothesis 135 "Call Best Buy," the first decoder result 132 also provides a separate likelihood score 157 for "Call $CONTACT." By modeling both the likelihood score 155 for the non-contextual hypothesis 135 and the separate likelihood score 157 for the non-contextual hypothesis 135 with the named entity replaced with a token 158 representing a personal named entity class, the first recognition model 130 can be used to evaluate the relative value of the personal (contextual) hypothesis 145 and the non-contextual (non-personal) candidate hypothesis 135.In contrast, a traditional voice-biased approach would unilaterally prioritize "Call $CONTACT" without considering the relative value of non-contextual / impersonal alternatives to token 158.

[0037] During stage (C), the computing system 120 generates, as output from the second recognition model M2 140, context candidate hypotheses 145 that are not included in the lattice 200 output by the first recognition model M1 130 or are ranked very low among the non-context candidate hypotheses of the lattice 200 during the first pass. When generating the non-context candidate hypotheses 135, the first recognition model M1 130 does not incorporate any user-specific data, whereas the second recognition model M2 140 is a personal recognition model tailored to a given user by incorporating personal context information 126 to aid in the recognition of personal named entities. More specifically, the second recognition model M2 140 generates a second decoding result 142 that includes a probability distribution 147 regarding personal named entities 148 that are likely to belong to a particular personal named entity class indicated by the token 158. Here, the likely personal named entities 148 may include all personal contacts of the user based on the personal context information 126 that includes the user's contact list. The personal context information 126 may further indicate the invocation frequency and usage pattern of each personal named entity 148 in the probability distribution 147 associated with a particular personal named entity class. In this manner, the probability distribution 147 may assign a higher probability score to a personal named entity 148 that the user accesses frequently (and / or accesses at a time associated with the current time at which the utterance 101 is received) than to a personal named entity 148 that the user never speaks of or speaks of rarely. Thus, the second recognition model 140 may adjust the scores assigned to the context candidate hypotheses 145 that have the personal named entity 148 based on how "contextually relevant" the personal named entity 148 is. In the illustrated example, the personal named entity 148 "Beth Byer" that is highest in the probability distribution 147 includes the user's lower-tier personal contacts that have not been accessed recently.Thus, the resulting score will be lower than if the best buyers were top-tier personal contacts that the user frequently accesses / calls.

[0038] In some examples, the second recognition model M2 140 includes an E2E speech recognition model, similar to the first recognition model M1 130, where the E2E speech recognition model is configured to receive the audio data 112 and the personal context information 126 and generate the second decoding result 142. In these examples, the first recognition model M1 and the second recognition model M2 may process the audio data 112 to generate the corresponding first decoding result 132 and the second decoding result 142 in parallel. Alternatively, the first recognition model M1 130 may generate the non-context candidate hypotheses 135 in the lattice 200 during the first pass, and then the second recognition model M2 140 may generate the context candidate hypotheses 145 during the second pass. In this scenario, the second recognition model M2 140 may also receive one or more of the non-context candidate hypotheses 135 generated in the lattice 200 during the first pass as a basis for generating the context candidate hypotheses 145 during the second pass. Thus, the first recognition model 130 may be a streaming E2E speech recognition model that generates streaming transcriptions, while the second pass second recognition model 140 may include a more computationally intensive speech recognizer and leverage personal context information 126 to generate more accurate descriptions at the expense of increased latency. In some configurations, the first recognition model M1 and the second recognition model M2 share one or more encoder layers, and audio encoding of the audio data generated during the first pass may be used by the second recognition model M2 140 during the second pass, so that M2 does not need to re-encode the audio data 112.

[0039] In other examples, the second recognition model M2 140 corresponds to a correction model that generates context candidate hypotheses 145 after the first recognition model M1 130 generates non-context candidate hypotheses 135 during the first pass. In this case, the correction model may correspond to a second-pass language model. The second-pass language model uses the personal context information 126 to generate context candidate hypotheses 145 as new hypotheses related to the user context that were completely missing from the lattice 200 and / or rescores existing hypotheses of the lattice 200 based on the user context. In some examples, the language model associated with the second recognition model M2 140 includes a personal / user-specific language model trained to recognize named entities 148 specific to the user 10 based on the personal context information 126. In these examples, the personal / user-specific language model may include an auxiliary language model external to the first recognition model M1 130. In this manner, the second recognition model M2 may receive only the lattice 200 as input without using the audio data 110 and may generate contextual candidate hypotheses 145 that are acoustically similar to the non-contextual candidate hypotheses 135. Additionally or alternatively, the second recognition model 140 may include a bias model that injects personal named entities that are acoustically similar to the general / generic terms of the lattice and / or increases the scores 155 of hypotheses in the lattice 200 that include personal named entities associated with the personal context information 126.

[0040] An important aspect of the speech recognition model 130, which combines a non-personal but personalizable first recognition model 130 and a personal second recognition model 140, is the ability to protect potentially sensitive user-specific data (e.g., personal context information 126) incorporated in the second recognition model 140. For example, the personal context information 126 may be stored in the memory hardware 116 of the user device 110, and the second recognition model 140 may run on the data processing hardware 114 of the user device 110 such that the user-specific data is not disclosed to the computing system 120 (e.g., a remote server / distributed system). In some scenarios, the second recognition model 140 runs on the user device 110 to calculate / generate probabilities of situations that are reduced to a simple weighted grammar and transmitted to the computing device 120 upon request. In these scenarios, the values ​​of the probability distribution 147 over possible personal named entities 148 may be transmitted to the computing device 120 over a network without the need to share any of the user-specific data / logic used to calculate these values.

[0041] The first recognition model 130 may execute on the user device 110 or on the computing system 120. Because the first recognition model 130 is impersonal and purposely constructed without any user-specific data, the first recognition model 130 may execute on the computing system 120 without revealing any personal information / data of the user.

[0042] The personal context information 126 may include personal named entities 148 specific to the user 10 who spoke the utterance 101, such as, but not limited to, the user's list of contacts, names of items in the user's media library (songs, artists, albums, movies), names of nearby places, names of installed applications, and user-defined entity names. The personal context information 126 may further indicate user history associated with these personal named entities, such as invocation frequency and usage patterns. For example, a user's media library may contain music files by artists Phish and Hootie and the Blowfish, but the user may play Phish songs almost daily while only playing Hootie and the Blowfish songs once five years ago. As a result, the personal context information 126 causes the second recognition model M2 140 to appropriately assign a higher likelihood / probability to the candidate hypothesis "Play Harpua by Phish" than to "Play Hootie and the Blowfish." The user history indicated by the personal context information 126 may further include previous queries, learned correlations, corrections of previously misrecognized transcriptions, etc.

[0043] The personal context information 126 may be stored in the memory hardware 114 of the user device 110 and / or in the memory hardware 124 of the computing system 120. The information in these different contextual data sets changes from time to time, such as when a user adds or removes contacts, when a user's media library changes, when a user adds or removes installed applications, when a user changes location, etc.

[0044] Continuing with reference to FIG. 1 , the scoring model 150 receives the first decoder result 132 generated by the first recognition model 130 and the second decoder result 142 generated by the second recognition model 140. In some examples, the scoring model 150 receives the first decoder result 132 associated with the 1-best non-context hypothesis 130 from the lattice 200. The scoring model 150 is configured to score the contextual candidate hypothesis 145 together with the non-contextual hypothesis 135 so that both personal named entities may be considered in conjunction with the non-personal alternatives for selecting the best candidate hypothesis 135, 145 as the resulting transcription 175 of the utterance 101. When scoring the contextual candidate hypothesis 145 and the non-contextual candidate hypothesis 135, the scoring model 150 calculates a likelihood score 155, P, modeled by the first recognition model M1 130 for the non-contextual candidate hypothesis 135. M1 (by calling Best Buy), and an individual likelihood score of 157, P M1 (call $CONTACT), as well as the potential personal named entity 148 modeled by the second recognition model M2 140, P M2 Use the probability distribution 147 for (BessBuyer|$CONTACT).

[0045] As will become apparent, scoring the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 output by M1 130 prevents over-triggering of the context candidate hypotheses 145 in non-spoken scenarios and under-recall of the non-context candidate hypotheses 135 in spoken scenarios. Continuing with the example where “Call Best Buy” includes a non-contextual candidate hypothesis 135 with the general named entity “Best Buy” and “Call Beth Buyer” includes a contextual candidate hypothesis 145 with a distinct named entity injected from the user’s list of personal contacts, the personalization of the first model M1 and the incorporation of personal context information 126 leveraged by the second model M2 allows the scoring model 300 to consider whether Beth Buyer is a personal contact of the user, whether Best Buy or Beth Buyer is a common general entity that is likely to be spoken by the user, the acoustic fit to the audio data 112, and other better (e.g., higher likelihood scores 155) available in the lattice 200.

[0046] The scoring model 150 may score the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 as follows: P M1 (h with entity g )=P M1 (h with $entityclass)×P M2 (entity p |$entityclass) (1) In the formula, P M1 (h with entity g ) denotes the likelihood score 155 assigned to the non-contextual candidate hypothesis 135 (e.g., call Best Buy) modeled by the first recognition model M1 130, and P M1(h with $entityclass) denotes the individual likelihood scores 157 for the non-contextual candidate hypotheses 135 with a general named entity class (e.g., Best Buy) replaced with the corresponding token (e.g., $CONTACT) 158 that maps to the individual named entity class (e.g., Personal Contact), P M2 (entity p |$entityclass) indicates the probability distribution 147 over which privately named entities 148 are likely to belong to the particular privately named entity class represented by the token 158. p The term P may represent a probability value summing to a value of 1.0 for each personal named entity associated with a given personal named entity class. M2 (entity p The resulting value of |$entityclass) may simply comprise the value of the personally named entity with the highest ranking in the probability distribution 147.

[0047] Continuing with the above example, while the personal context information 126 may indicate that "Beth Buyer" is indeed one of the user's personal contacts, the context information 126 may further reveal that "Beth Buyer" is a lower-tier personal contact that the user rarely calls or has never called, making the context (personal) candidate hypothesis 145 relatively weaker compared to the strong non-context (non-personal) candidate hypothesis 135 "Call Best Buy." Notably, the existing candidate hypothesis and the new candidate hypothesis are phonetically close to each other. It may indicate that Beth Buyer is a contact in the user's list of personal contacts, but is a lower-tier personal contact that the user rarely calls. As a result, applying Equation 1 to score the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 yields P M1 (Call Best Buy)>P M1 (Call $CONTACTS)*P M2(Beth Byer|$CONTACTS), and the non-context candidate hypothesis 135 is not replaced and is appropriately selected as the transcription 175 for utterance 101.

[0048] In another example where a user 10 speaks the utterance 101 of “Call Leonid” (e.g., a personal contact of the user), the first model M1 130 may generate a non-contextual candidate hypothesis 135 “Call Lee Oh Need” and the second model M2 140 may generate a contextual candidate hypothesis 145 “Call Leonid.” Although the two hypotheses 135, 145 are phonetically close to each other, the likelihood score 155 assigned to the non-contextual candidate hypothesis 135 may indicate that the non-contextual candidate hypothesis 135 is not plausible and is unlikely to have been spoken, while the probability distribution 147 over likely personal named entities 148 belonging to the personal contacts class may include a high probability for “Leonid” based on the personal context information 126 indicating that “Leonid” is one of the user’s personal contacts that the user calls regularly. As a result, applying Equation 1 to score the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 gives P M1 (Call Lee Oh Need) <P M1 (Call $CONTACTS)*P M2 (Leonid|$CONTACTS), and the context candidate hypothesis 145 is appropriately selected as the transcription 175 of the utterance 101.

[0049] The scoring model 150 may evaluate and score the context candidate hypothesis 145 against a single non-context candidate hypothesis 135 and its corresponding likelihood score 155. The single non-context candidate hypothesis 135 may be selected for better scoring accuracy, for example, based on high acoustic proximity to the new candidate hypothesis 145 and / or affinity with similar hypotheses for use as a reference point. The single non-context candidate hypothesis 135 may be selected as the 1-best hypothesis 135 in the lattice. Alternatively, the scoring model 150 may evaluate and score the context candidate hypothesis 145 against multiple non-context candidate hypotheses 135 (or even all hypotheses in the original lattice 200) and their corresponding likelihood scores 155 to provide more accurate scoring at the cost of increased computational load (in addition to increased latency and increased memory requirements).

[0050] 1, during stage (F), the reranker 160 receives the scoring output 152 from the scoring model 150 that scores the contextual candidate hypotheses 145 against the non-contextual candidate hypotheses 135 and their corresponding likelihood scores 155. The reranker 160 is configured to output a reranked result 165 that includes a ranking of the non-contextual candidate hypotheses 135 and the contextual candidate hypotheses 145 based on the scoring output. In the illustrated example, the reranked result 165 includes the non-contextual candidate hypothesis 135, with a score of 0.9, as the most likely correct transcription 175.

[0051] In stage (G), the computing system 120 is configured to generate a transcription 175 of the utterance 101 spoken by the user 10 by selecting the highest ranked candidate from the re-ranked results 165. In the illustrated example, the computing system 120 selects the non-context candidate hypothesis 135 "Call Best Buy" because it has the highest score of 0.9. The computing system 120 may provide the transcription 175 to the user 10 by transmitting the transcription 175 to the client device 110 via the network 118. In some implementations, the client device 110 performs all of stages (A)-(G) on-device without connecting to the computing system 120 via the network 118.

[0052] While the above example illustrates scoring contextual candidate hypotheses against non-contextual candidate hypotheses of an audio-based speech input, aspects of the present disclosure may be similarly applied to scoring contextual candidate hypotheses against non-contextual candidate hypotheses of a non-speech input. For example, the non-speech input 113 may include a user input indication indicating a selection of one or more characters on a keyboard 117 (e.g., a virtual keyboard or a physical keyboard) in communication with the data processing hardware 114 of the client device 110. In this example, the speech recognizer module 130 may be replaced by a keyboard detection module configured to generate a lattice 200 of non-contextual candidate hypotheses 135 for the non-speech input 113. The models M1 and M1 may function similarly to the above example by generating the non-contextual candidate hypotheses 135 and the contextual candidate hypotheses 145 for the non-speech input 113. The scoring model 150 may then score the contextual candidate hypotheses against the non-contextual candidate hypotheses, thereby providing the ability to rank non-speech recognition hypotheses that are personal / user specific against non-personal hypotheses that include general terms for which the keyboard detection module may not have been trained.

[0053] 3, in some implementations, an exemplary training process 300 may train the first recognition model M1 130 as an aggregate without any user-specific data to personalize the first recognition model M1 130. Specifically, the training process 300 may train the first model M1 130 on an ASR training corpus 320 that includes a corpus of ground truth transcriptions 325. In some examples, the personal named entity extractor 302 may replace the identified personal named entity transcriptions 325 with corresponding tokens associated with the personal named entity class to which the identified personal named entity belongs. For example, the extractor 302 may identify "Nick" in the transcription "Call Nick" as a personal named entity belonging to the personal contacts class and replace "Nick" with the corresponding token "$CONTACT." The extractor 302 may identify personal named entities through techniques such as syntactic analysis and / or explicit labels provided in the transcription 325.

[0054] Optionally, the training process 300 may additionally train the first model M1 130 on a generic transcription that includes only generic named entities, where the extractor 302 may determine when a given transcription 325 omits personal named entities and therefore does not replace any terms in the transcription 325 with tokens. In this scenario, the training process 300 also trains the first model M1 130 on the generic transcription 325 to train the model M1 130 to evaluate the relative merits between personal hypotheses that include personal named entities and non-personal hypotheses that include only generic terms. For example, the extractor 302 may determine that "Best Buy" is a generic entity in the transcription "Call Best Buy" and allow the transcription 325 to train the model M1 130 without replacing any terms with tokens.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0057] 4 is a flowchart of an example arrangement of operations of a method 400 for scoring a contextual candidate hypothesis 145 against a non-contextual candidate hypothesis. The data processing hardware 122 (FIG. 1) may execute instructions stored in the memory hardware 124 (FIG. 1) to perform the example arrangement of operations of the method 400. At operation 402, the method 400 includes receiving audio data 112 corresponding to an utterance 101 spoken by the user 10. At operation 404, the method 400 includes processing the audio data 112 using a first recognition model 130 to generate a non-contextual candidate hypothesis 135 as output from the first recognition model 130, where the non-contextual candidate hypothesis 135 corresponds to a candidate transcription of the utterance and has a corresponding likelihood score 155 assigned to the non-contextual candidate hypothesis 135 by the first recognition model 130.

[0058] At operation 406, the method 400 includes using a second recognition model 140 configured to receive the personal context information 126 to generate context candidate hypotheses 145 including the personal named entities, where the context candidate hypotheses correspond to alternative candidate transcriptions of the utterance.

[0059] At operation 408, the method 400 includes scoring the contextual candidate hypotheses 145 against the non-contextual candidate hypotheses 135 based on the personal context information 126 and the corresponding likelihood scores 155 assigned to the non-contextual candidate hypotheses 135. Based on scoring the contextual candidate hypotheses against the non-contextual candidate hypotheses, the method 400 further includes generating a transcription 175 of the utterance 101 spoken by the user 10 by selecting one of the contextual candidate hypotheses 145 or the non-contextual candidate hypotheses 135 at operation 410.

[0060] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0061] Computing device 500 includes a processor 510, memory 520, storage device 530, high-speed interface / controller 540 that connects to memory 520 and high-speed expansion port 550, and low-speed interface / controller 560 that connects to low-speed bus 570 and storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise present as desired. Processor 510 may process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 580 connected to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Also, multiple computing devices 500 may be connected, with each device performing some of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0062] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0063] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0064] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages low-bandwidth-intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be connected, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0065] The computing device 500 may be implemented in many different forms, as shown in the figure. For example, the computing device may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general purpose, at least one input device, and at least one output device, coupled to receive data and instructions from the storage system and to transmit data and instructions to the storage system.

[0067] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0068] The processes and logic flows described herein can be implemented by one or more programmable processors (also referred to as data processing hardware) executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0069] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0070] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400) that, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations including: receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate, as output from the first recognition model, non-contextual candidate hypotheses (135) that correspond to candidate transcriptions (175) of the utterance (101) and have corresponding likelihood scores (155) assigned to the non-contextual candidate hypotheses (135) by the first recognition model (130); generating a context candidate hypothesis (145) including a personal named entity (148) using a second recognition model configured to receive personal context information (126), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypotheses (145) against the non-context candidate hypotheses (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); generating a transcription (175) of the utterance (101) spoken by the user by selecting one of the context candidate hypotheses (145) or the non-context candidate hypotheses (135) based on the scoring of the context candidate hypotheses (145) against the non-context candidate hypotheses (135).

2. 2. The computer-implemented method of claim 1, wherein the first recognition model processes the audio data to generate the non-context candidate hypotheses without incorporating any of the personal context information associated with the user.

3. 3. The computer-implemented method of claim 1, wherein the first recognition model comprises an end-to-end speech recognition model configured to generate the corresponding likelihood scores of the non-contextual candidate hypotheses.

4. the second recognition model (140) includes an end-to-end speech recognition model (140) configured to generate the context candidate hypotheses (145); The computer-implemented method (400) of any of claims 1 to 3, wherein generating the context candidate hypotheses (145) including the personal named entities (148) comprises processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypotheses (145) based on the personal context information (126).

5. 5. The computer-implemented method of claim 4, wherein the end-to-end speech recognition model is configured to generate the contextual candidate hypotheses after the non-contextual candidate hypotheses are generated as output from the first recognition model.

6. 5. The computer-implemented method of claim 4, wherein the end-to-end speech recognition model is configured to generate the contextual candidate hypotheses in parallel with the processing of the audio data to generate the non-contextual candidate hypotheses as output from a first recognition model.

7. The computer-implemented method (400) of any of claims 1 to 6, wherein the second recognition model (140) comprises a language model.

8. The personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) comprising: a contact name in the user's personal contact list; A name in a media library associated with said user; Name of the installed application, Name of a nearby place, or The computer implemented method (400) of any of claims 1 to 7, including at least one of a user defined entity name.

9. The computer-implemented method (400) of any of claims 1 to 8, wherein the personal context information (126) associated with the user further indicates a user history associated with the personal named entities (148) specific to the user.

10. The operation includes: receiving a training corpus of ground truth transcriptions; For each corresponding ground truth transcription (175) of the training corpus of ground truth transcriptions, determining whether a personal named entity (148) is identified in the corresponding ground truth transcription (175); replacing the personal named entities (148) with entity class tokens (158) when the personal named entities (148) are identified in the corresponding ground truth transcriptions (175); 10. The computer-implemented method (400) of claim 1, further comprising: training the first recognition model with the ground truth transcription that includes personal named entities (148) replaced by tokens of the entity classes.

11. Data processing hardware (510); memory hardware (520) in communication with the data processing hardware (510) and storing instructions that, when executed by the data processing hardware (510), cause the data processing hardware (510) to: receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate, as output from the first recognition model (130), non-contextual candidate hypotheses (135) that correspond to candidate transcriptions (175) of the utterance (101) and have corresponding likelihood scores (155) assigned to the non-contextual candidate hypotheses (135) by the first recognition model (130); generating a context candidate hypothesis (145) including a personal named entity (148) using a second recognition model (140) configured to receive personal context information (126), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypotheses (145) against the non-context candidate hypotheses (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); generating a transcription (175) of the utterance (101) spoken by the user by selecting one of the context candidate hypotheses (145) or the non-context candidate hypotheses (135) based on the scoring of the context candidate hypotheses (145) against the non-context candidate hypotheses (135).

12. 12. The system (100) of claim 11, wherein the first recognition model (130) processes the audio data (112) to generate the non-context candidate hypotheses (135) without incorporating any of the personal context information (126) associated with the user.

13. 13. The system (100) of claim 11 or 12, wherein the first recognition model (130) comprises an end-to-end speech recognition model (130) configured to generate the corresponding likelihood scores (155) of the non-contextual candidate hypotheses (135).

14. the second recognition model (140) includes an end-to-end speech recognition model (140) configured to generate the context candidate hypotheses (145); The system (100) of any of claims 11 to 13, wherein generating the context candidate hypotheses (145) including the personal named entities (148) comprises processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypotheses (145) based on the personal context information (126).

15. 15. The system of claim 14, wherein the end-to-end speech recognition model is configured to generate the contextual candidate hypotheses after the non-contextual candidate hypotheses are generated as output from the first recognition model.

16. 15. The system (100) of claim 14, wherein the end-to-end speech recognition model (140) is configured to generate the contextual candidate hypotheses (145) in parallel with the processing of the audio data (112) to generate the non-contextual candidate hypotheses (135) as output from a first recognition model (130).

17. The system (100) of any of claims 11 to 16, wherein the second recognition model comprises a language model.

18. The personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) comprising: a contact name in the user's personal contact list; A name in a media library associated with said user; Name of the installed application, Name of a nearby place, or The system (100) according to any of claims 11 to 17, comprising at least one of the user-defined entity names.

19. The system (100) of any of claims 11 to 18, wherein the personal context information (126) associated with the user further indicates a user history associated with the personal named entities (148) specific to the user.

20. The operation includes: receiving a training corpus of ground truth transcriptions; For each corresponding ground truth transcription (175) of the training corpus of ground truth transcriptions, determining whether a personal named entity (148) is identified in the corresponding ground truth transcription (175); replacing the personal named entities (148) with entity class tokens (158) when the personal named entities (148) are identified in the corresponding ground truth transcriptions (175); 20. The system (100) of claim 11, further comprising: training the first recognition model with the ground truth transcription that includes personal named entities (148) replaced by tokens of the entity classes.

Images