Personalizable probabilistic model

Abstract

A method (400) includes receiving audio (112) and processing the audio data using a first recognition model (130) to generate a non-contextual candidate hypothesis (135) as an output of the first recognition model. The non-contextual candidate hypothesis has a corresponding likelihood score assigned by the first recognition model (155). The method also includes generating a contextual candidate hypothesis (145) including the personal named entity (148) using a second recognition model (140) configured to receive the personal contextual information (126). The method further includes scoring the contextual candidate hypotheses relative to the non-contextual candidate hypotheses based on the personal contextual information and the corresponding likelihood scores assigned to the non-contextual candidate hypotheses. The method also includes generating a transcription of the utterance by selecting one of a contextual candidate hypothesis or a non-contextual candidate hypothesis (175).

Description

Technical Field

[0001] The present disclosure relates to personalizable probabilistic models. Background Art

[0002] Modern automatic speech recognition (ASR) systems focus not only on providing quality / accuracy (e.g., low word error rate (WER)), but also on providing low latency (e.g., a short delay between a user speaking and the transcription appearing). Because the generalized training data used to train ASR systems to perform speech recognition tasks is sparse, ASR systems have difficulty recognizing specialized phrases including medical / legal terminology, as well as occurrences of proper nouns such as personal contacts, song / artist names, applications, and new entity names. Similarly, ASR systems have difficulty recognizing terms in cross-lingual discourse in a language different from the language of the training data. As a result, the speech recognition grid ranks these specialized phrases low or ignores them entirely. Summary of the invention

[0003] One aspect of the present disclosure provides a method for scoring contextual candidate hypotheses relative to non-contextual candidate hypotheses, which, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving audio data corresponding to an utterance spoken by a user, and processing the audio data using a first recognition model to generate a non-contextual candidate hypothesis as an output of the first recognition model. Here, the non-contextual candidate hypothesis corresponds to a candidate transcription of the utterance and has a corresponding likelihood score assigned to the non-contextual candidate hypothesis by the first recognition model. The operations also include generating a contextual candidate hypothesis including a personal named entity using a second recognition model configured to receive personal context information. Here, the contextual candidate hypothesis corresponds to another candidate transcription of the utterance. The operations also include: scoring the contextual candidate hypothesis relative to the non-contextual candidate hypothesis based on the personal contextual information and the corresponding likelihood score assigned to the non-contextual candidate hypothesis; and generating a transcription of the utterance spoken by the user by selecting one of the contextual candidate hypothesis or the non-contextual candidate hypothesis based on scoring the contextual candidate hypothesis relative to the non-contextual candidate hypothesis.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the first recognition model processes the audio data to generate non-contextual candidate hypotheses without incorporating any of the personal contextual information associated with the user. In some examples, the first recognition model includes an end-to-end speech recognition model that is configured to generate corresponding likelihood scores for the non-contextual candidate hypotheses.

[0005] In some implementations, the second recognition model includes an end-to-end speech recognition model that is configured to generate context candidate hypotheses. Here, generating context candidate hypotheses that include personal named entities includes using the end-to-end speech recognition model to process audio data to generate context candidate hypotheses based on personal context information. In these implementations, the end-to-end speech recognition model can be configured to generate context candidate hypotheses after generating non-context candidate hypotheses as an output of the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate context candidate hypotheses in parallel with processing audio data to generate non-context candidate hypotheses as an output of the first recognition model.

[0006] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user. Here, the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby location, or a user-defined entity name. In some examples, the personal context information associated with the user also indicates a user history related to the personal named entity specific to the user.

[0007] In some implementations, the operations further include receiving a training corpus of real-value transcriptions. For each corresponding real-value transcription in the training corpus of real-value transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding real-value transcription, and when the personal named entity is identified in the corresponding real-value transcription, replacing the personal named entity with an entity class token. Here, the operations further include training a first recognition model on the real-value transcriptions including the personal named entity replaced by the entity class token.

[0008] Another aspect of the present disclosure provides a system for scoring contextual candidate hypotheses relative to non-contextual candidate hypotheses. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including: receiving audio data corresponding to an utterance spoken by a user, and processing the audio data using a first recognition model to generate a non-contextual candidate hypothesis as an output of the first recognition model. Here, the non-contextual candidate hypothesis corresponds to a candidate transcription of the utterance and has a corresponding likelihood score assigned to the non-contextual candidate hypothesis by the first recognition model. The operations also include generating a contextual candidate hypothesis including a personal named entity using a second recognition model configured to receive personal contextual information. Here, the contextual candidate hypothesis corresponds to another candidate transcription of the utterance. The operations also include scoring context candidate hypotheses relative to non-context candidate hypotheses based on personal context information and corresponding likelihood scores assigned to non-context candidate hypotheses, and generating a transcription of an utterance spoken by the user by selecting one of the context candidate hypotheses or the non-context candidate hypotheses based on scoring the context candidate hypotheses relative to the non-context candidate hypotheses.

[0009] This aspect may include one or more of the following optional features. In some implementations, the first recognition model processes the audio data to generate non-contextual candidate hypotheses without incorporating any of the personal contextual information associated with the user. In some examples, the first recognition model includes an end-to-end speech recognition model that is configured to generate corresponding likelihood scores for the non-contextual candidate hypotheses.

[0010] In some implementations, the second recognition model includes an end-to-end speech recognition model that is configured to generate context candidate hypotheses. Here, generating context candidate hypotheses that include personal named entities includes using the end-to-end speech recognition model to process audio data to generate context candidate hypotheses based on personal context information. In these implementations, the end-to-end speech recognition model can be configured to generate context candidate hypotheses after generating non-context candidate hypotheses as an output of the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate context candidate hypotheses in parallel with processing audio data to generate non-context candidate hypotheses as an output of the first recognition model.

[0011] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user. Here, the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby location, or a user-defined entity name. In some examples, the personal context information associated with the user also indicates a user history related to the personal named entity specific to the user.

[0012] In some implementations, the operations further include receiving a training corpus of real-value transcriptions. For each corresponding real-value transcription in the training corpus of real-value transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding real-value transcription, and when the personal named entity is identified in the corresponding real-value transcription, replacing the personal named entity with an entity category tag. Here, the operations further include training a first recognition model on the real-value transcriptions including the personal named entity replaced by the entity category tag.

[0013] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an example system for scoring a first candidate hypothesis generated by a first recognizer model for an utterance and a second candidate hypothesis generated by a second recognizer model for the same utterance.

[0015] Figure 2A and Figure 2B is a schematic diagram showing a diagram of an example of a word grid.

[0016] Figure 3 is used to describe the training Figure 1 Schematic diagram of the training process of the first speech recognition model used by the system.

[0017] Figure 4 is a flow diagram of an example operational arrangement of a method for scoring a first candidate hypothesis generated by a first speech recognition model for an utterance and a second candidate hypothesis generated by a second speech recognition model for the same utterance.

[0018] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0019] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0020] Automatic speech recognition (ASR) systems are becoming increasingly popular in client devices as ASR systems continue to provide more accurate transcriptions of what users say. Recently, end-to-end (E2E) ASR models have become increasingly popular in achieving state-of-the-art performance in terms of accuracy and latency. Compared to conventional hybrid ASR systems that include separate acoustic, pronunciation, and language models, E2E ASR models adopt a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data (e.g., utterance-transcription pairs). In addition, in some instances, the ASR model generates inaccurate transcriptions that misidentify what the user actually said. This typically occurs when the user speaks a personal named entity that is sparse or non-existent in the training data used to train the ASR model. As a result, the correct hypothesis for the personal named entity is ranked low in the recognition grid of possible speech recognition hypotheses, or is missing entirely. As used herein, a personal named entity refers to any of a personal contact's name, a name in a media library (eg, a song, artist, album, movie, etc.), an installed application, and a user-defined name (eg, myLivingRoomStreamingDevice).

[0021] Prior art techniques for improving personal named entity recognition are to apply biases that typically involve promoting relevant speech recognition hypotheses containing personal named entities or creating new candidate hypotheses that inject personal named entities with close acoustic, grapheme, or embedding proximity to existing hypotheses. However, an often overlooked consideration is how to accurately score contextual (user-specific or personal) candidate hypotheses for utterances with personal named entities relative to non-contextual candidate hypotheses for utterances that lack any user-specific data and therefore lack personal named entities. When contextual hypotheses are scored separately from existing non-contextual hypotheses, as is often the case when applying the prior art biasing techniques described above, the ASR system runs the risk of over-triggering unspoken contextual hypotheses or under-recalling non-contextual candidate hypotheses even if spoken. For example, consider the spoken utterance "call Leonid", for which the ASR model generates the non-contextual hypothesis "call lee ohneed". Accurately scoring the new hypothesis “call Leonid” must consider whether “lee oh need” and “Leonid” are personal contacts, whether “lee oh need” and “Leonid” are generic entities that might be uttered in the utterance, the a priori likelihood of the non-contextual hypothesis, and whether there are other, better non-contextual hypotheses.

[0022] In order to accurately rank contextual hypotheses against non-contextual hypotheses, contextual hypotheses need to be accurately scored. ASR models that generate and score non-contextual hypotheses are not adequately suited for scoring contextual hypotheses because the contextual hypotheses are already not recalled during decoding or are assigned a low score. On the other hand, context-specific models (such as bias models that bias speech recognition results toward personal named entities) are unable to score and rank speech recognition hypotheses that include personal named entities relative to the generic entities on which the ASR model was trained.

[0023] Implementations herein relate to a speech recognizer module comprising a first (generic) model (M1) that does not contain any user-specific data and a second (personal) model (M2) that is calculated and dynamically updated for a given user. Specifically, a computing system receives audio data of an utterance spoken by a user, and the computing system processes the audio data to generate non-contextual candidate hypotheses for the utterance as an output of the first model (M1), and to generate contextual candidate hypotheses for the utterance as an output of the second model (M2). In some examples, the first model (M1) is configured to generate a plurality of non-contextual hypotheses that represent possible transcriptions of the utterance in a word grid. As used herein, a non-contextual candidate hypothesis refers to a possible transcription of an utterance generated by the first model (M1) without incorporating any user-specific data or personal information associated with a given user. Thus, the first model is effective for recognition modeling of generic entities spoken in utterances that are impersonal in nature, but is generally not suitable for recognizing personal named entities.

[0024] On the other hand, the second model (M2) is configured to generate contextual candidate hypotheses including personal named entities by utilizing personal context information related to the user. The context information may indicate a user history related to a given personal named entity, such as call frequency and usage pattern. More specifically, the second model (M2) corresponds to a user-specific model that assigns probabilities to possible personal named entities of a specific entity category based on personal context information. Thus, the second model is effective for recognition modeling of personal named entities spoken in an utterance. The implementation herein specifically involves combining the first model M1 and the second model M2 to determine likelihood scores for both non-contextual hypotheses and contextual hypotheses, and selecting a transcription of the utterance by selecting either the non-contextual hypothesis or the contextual candidate hypothesis with the highest likelihood score. As used herein, the likelihood score indicates the probability that the corresponding candidate hypothesis is a correct transcription of the utterance spoken by the user.

[0025] Figure 1An example of a system 100 for performing automatic speech recognition (ASR) on an utterance 101 (i.e., a query) spoken by a user 10 using audio data 112 corresponding to the utterance 101. The system 100 includes a client device 110, a computing system 120, and a network 118. The computing system 120 can be a distributed system with scalable elastic resources (e.g., a cloud computing environment). These resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 118 can be wired, wireless, or a combination thereof, and can include private networks and / or public networks, such as the Internet.

[0026] In some examples, the computing system 120 receives the audio data 112 from the client device 110, and the computing system 120 processes the audio data 112 to generate one or more non-contextual candidate hypotheses 135 for the utterance 101 based on the audio data 112 as an output of the first recognizer model 130. In some additional examples, the client device 110 processes the audio data 112 entirely on the device to generate one or more non-contextual candidate hypotheses 135 for the utterance 101 based on the audio data 112. Here, each non-contextual candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 generated without using any personal context information 126 and is represented by a corresponding sequence of hypothesis terms.

[0027] As described in more detail below, the second recognizer model 140 (executing on the client device 110 or computing system 120) utilizes the personal context information 126 to generate a context candidate hypothesis 145 for the utterance 101 that is not included in the existing candidate hypotheses 135 or ranks very low among the existing candidate hypotheses in the grid 200. Here, the context candidate hypothesis 145 corresponds to a candidate transcription of the same utterance 101 generated by incorporating the personal context information 126. Thus, the personal context information 126 utilized by the second recognizer model 140 facilitates recognition of the personal named entity. Thereafter, the scoring model 150 scores the context candidate hypothesis 145, and one of the non-context candidate hypotheses 135, such that the computing system 120 (or user device 110) generates a transcription 175 of the utterance 101 by selecting the highest ranked hypothesis between the non-context candidate hypothesis 135 or the context candidate hypothesis 145 scored by the scoring model 150.

[0028] Figure 1Operations (A) through (G) are shown to illustrate data flow. As described herein, computing system 120 performs operations (B) through (G). However, it should be understood that client device 110 may also perform operations (B) through (G) in addition to or in lieu of computing system 120 performing these operations. In some examples, client device 110 performs a first portion of the operations (e.g., operation (C)) and computing system 120 performs a second portion of the operations (e.g., operations (D) through (F)), or vice versa.

[0029] The client device 110 includes data processing hardware 114 and memory hardware 116. The client device 110 may include one or more audio capture devices (e.g., microphones) 103 for capturing utterances 101 from the user 10 and converting them into audio data 112 (e.g., electrical signals). In some examples, the microphone 103 is separate from the client device 110 and communicates with the client device 110 to provide the recorded utterances 101 to the client device 110. The client device 110 may be any computing device capable of communicating with the computing system 120 via a network 118. The user 10 may input text utterances 101 via a keyboard 117 (such as a virtual keyboard displayed in a graphical user interface of the client device 110 or a physical keyboard in communication with the client device 110) instead of verbal utterances 101. The client device 110 includes, but is not limited to, desktop computing devices and mobile computing devices, such as laptop computers, tablet computers, smart phones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., earphones, headphones, and / or watches).

[0030] exist Figure 1In the example of , during stage (A), user 10 speaks utterance 101 and microphone 103 of client device 110 records utterance 101. In this example, utterance 101 includes user 10 saying "call Best Buy". Client device 110 transmits audio data 112 corresponding to utterance 101 recorded by microphone 103 to computing system 120 via network 118. During stage (B), computing system 120 processes audio data 112 to generate one or more non-contextual candidate hypotheses 135 as output of first recognition model 130. Here, each candidate hypothesis 135 corresponds to a candidate transcription of utterance 101 and is represented by a corresponding sequence of hypothesis terms. For example, computing system 120 may execute speech recognizer module 30 (e.g., automatic speech recognition (ASR) module) including first recognition model 130 to generate word grid 200 indicating non-contextual candidate hypotheses 135 that may be possible for utterance 101 based on audio data 112 without incorporating any personal context information 126. The first recognition model 130 can evaluate potential paths through the word lattice 200 to determine non-contextual candidate hypotheses 135 .

[0031] Figure 2A It shows that it can be Figure 1 130 . The word grid 200 , 200 a represents a plurality of possible word combinations that may form different non-contextual candidate hypotheses 135 for the utterance 101 .

[0032] The word grid 200a includes one or more nodes 202a-g corresponding to possible boundaries between words. The word grid 200a includes multiple edges 204a-l of possible words in the candidate hypothesis generated by the word grid 200a. In addition, each of the edges 204a-l may have one or more weights or probabilities that the edge is the correct edge from the corresponding node. The weights are determined by M1 130 and can be based on, for example, the confidence of the match between the speech data and the word of the edge and the degree of grammatical and / or lexical matching of the word with other words in the word grid 200a.

[0033] For example, initially, the most likely path through the word lattice 200a (e.g., the most likely non-context candidate hypothesis 135) may include edges 204c, 204e, 204i, 204k with the text "we're coming about 11:30". The second best path through the word lattice 200a (e.g., the second best non-context candidate hypothesis 135) may include edges 204d, 204h, 204j, 204l with the text "deerhunting scouts 7:30".

[0034] Each pair of nodes may have one or more paths corresponding to alternative terms in various non-context candidate hypotheses 135. For example, the initial most likely path between the pair of nodes starting at node 202a and ending at node 202c is edge 204c "we're". This path has alternative paths including edges 204a, 204b "we are" and edge 204d "deer".

[0035] Figure 2B It can be done by Figure 1 1. An example of a hierarchical word grid 200, 200b provided by the first recognition model 130 of FIG. The word grid 200b includes nodes 252a-1 representing words of various non-contextual candidate hypotheses 135 that make up the utterance 101. The edges between nodes 252a-l show that possible candidate hypotheses 135 include: (1) nodes 252c, 252e, 252i, 252k "we're coming about 11:30"; (2) nodes 252a, 252b, 252e, 252i, 252k "we're coming about 11:30"; (3) nodes 252a, 252b, 252f, 252g, 252i, 252k "we are come at about 11:30"; (4) nodes 252d, 252f, 252g, 252i, 252k "deer come at about 11:30"; (5) nodes 252d, 252h, 252j, 252k "deer hunting scouts 11:30"; and (6) nodes 252d, 252h, 252j, 252l "deer hunting scouts 7:30".

[0036] Similarly, the edges between nodes 242a-l can have associated weights or probabilities based on the confidence (e.g., candidate hypothesis) of speech recognition and the grammatical / lexical analysis of the resulting text. In this example, "we're coming about11:30" may currently be the best hypothesis, while "deer hunting scouts 7:30" may be the next best hypothesis. One or more partitions 354a-d may be made in the word grid 200b to group words and their alternatives together. For example, partition 254a includes the word "we're" and the alternatives "we are" and "deer". Partition 252b includes the word "coming" and the alternatives "come at" and "hunting". Partition 254c includes the word "about" and the alternative "scouts", and partition 254d includes the word "11:30" and the alternative "7:30".

[0037] Return to reference Figure 1 , the first recognition model M1 130 can generate a first decoding result 132, modeling non-context candidate hypotheses 135 from the word grid 200 and corresponding likelihood scores 155 assigned to the non-context candidate hypotheses 135. Each likelihood score 155 indicates the probability that the candidate hypothesis 135 is correct (e.g., matches the utterance 101). In some implementations, the first recognition model M1 130 includes an end-to-end (E2E) speech recognition model configured to receive the speech data 112 and generate the word grid 200. Specifically, the E2E speech recognition model processes the audio data 112 to generate a first decoding result 132 indicating a likelihood score 155 for each of a plurality of non-context candidate hypotheses 135 from the word grid 200. In some examples, the first recognition model 130 includes separate acoustic models, language models, and / or pronunciation models.

[0038] It is noteworthy that the first recognition model 130 corresponds to a general speech recognizer trained on general data, so that the first recognition model M1 130 has a lower accuracy in recognizing speech in professional fields that use professional phrases (e.g., voice commands, medical terms, legal terms) and / or named entities (e.g., names of personal contacts, song / artist names, application names). The decrease in accuracy in recognizing speech in these professional fields is due to the sparse general data used to train the speech recognizer module, the presence of cross-language or rare words, new entities, and out-of-vocabulary terms that are not present in the general data.

[0039] In some examples, the speech recognizer module 130 includes an acoustic model and / or a language model to generate a plurality of candidate hypotheses 135 of the word grid 200 or otherwise generate the utterance 101 based on the audio data 112. Here, the likelihood scores 155 of the plurality of candidate hypotheses 135 may include a combination of an acoustic modeling score from the acoustic model and / or a priori likelihood score from the language model. In other words, the likelihood score 155 includes at least one of an acoustic modeling score output by the acoustic model and / or a priori likelihood score output by the language model.

[0040] The following will refer to Figure 3Described in more detail, the first recognition model M1 130 can be trained as an aggregate lacking any user-specific data so that the first recognition model 130 can be personalized, such that the first recognition model M1 130 is aware of the user context associated with the input utterance 101 when generating the first decoding result 132, but does not incorporate or otherwise use any user-specific data or personal information associated with a given user. For example, M1 130 can be trained with annotated utterances in which personal entities are identified and replaced with tags 158. In some examples, different tags 158 representing personal named entity categories are used to replace named entities that may belong to a particular named entity category. In the example shown, the first recognition model M1 130 also models / generates a personalized likelihood score 157 for a given non-contextual candidate hypothesis 135 in which the recognized named entity (e.g., Best Buy) has been replaced with a corresponding tag 158 (e.g., $CONTACT). Other tokens 158 representing other personal named entity categories may be used, such as, but not limited to, applications (e.g., $APP), song names (e.g., $SONG), artist names (e.g., $ARTIST), and album names (e.g., $ALBUM). Thus, for the non-contextual candidate hypothesis 135 "call Best Buy," the first decoder result 132 also provides a personalized likelihood score 157 for "call $CONTACT." By modeling both the likelihood score 155 for the non-contextual hypothesis 135 and the personalized likelihood score 157 for the non-contextual hypothesis 135 where the named entities have been replaced with tokens 158 representing the personal named entity category, the first recognition model 130 may be used to evaluate the relative merits of the personal (contextual) hypothesis 145 and the non-contextual (non-personal) candidate hypothesis 135. In contrast, conventional speech biasing methods would unilaterally promote "call $CONTACT" without considering the relative merits of the non-contextual / non-personal alternatives in place of the token 158.

[0041] During stage (C), the computing system 120 generates a context candidate hypothesis 145 as an output of the second recognition model M2 140 that is not included in the grid 200 output by the first recognition model M1 130, or that ranks very low among the non-context candidate hypotheses in the grid 200 during the first pass. While the first recognition model M1 130 lacks any user-specific data incorporated when generating the non-context candidate hypotheses 135, the second recognition model M2 140 instead becomes a personal recognition model tailored to a given user by incorporating personal context information 126 to facilitate recognizing personal named entities. More specifically, the second recognition model M2 140 generates a second decoding result 142 that includes a probability distribution 147 of possible personal named entities 148 belonging to a particular personal named entity category represented by the marker 158. Here, the possible personal named entities 148 may include all of the user's personal contacts based on the personal context information 126 including the user's contact list. The personal context information 126 may also indicate the call frequency and usage pattern of each personal named entity 148 in the probability distribution 147 associated with a particular personal named entity category. Thus, the probability distribution 147 may assign a higher probability score to a personal named entity 148 that is frequently accessed by the user (and / or accessed at a time associated with the current time at which the utterance 101 is received) than to a personal named entity 148 that is never or rarely spoken by the user. Thus, the second recognition model 140 may adjust the score assigned to the context candidate hypothesis 145 having the personal named entity 148 based on how “contextually relevant” the personal named entity 148 is. In the example shown, the top personal named entity 148 “Beth Byer” in the probability distribution 147 includes a low-level personal contact of the user that has not been visited recently. Therefore, the resulting score is lower than if Beth Byer were instead a high-level personal contact that the user frequently visits / calls.

[0042] In some examples, similar to the first recognition model M1 130, the second recognition model M2 140 includes an E2E speech recognition model. Here, the E2E speech recognition model is configured to receive the audio data 112 and the personal context information 126 and generate the second decoding result 142. In these examples, the first recognition model M1 and the second recognition model M2 can process the audio data 112 in parallel and generate their corresponding first decoding results 132 and second decoding results 142. Alternatively, the first recognition model M1 130 can generate the non-context candidate hypotheses 135 in the grid 200 during the first pass, and then, the second recognition model M2 140 can generate the context candidate hypotheses 145 during the second pass. In this scenario, the second recognition model M2 140 can also receive one or more of the non-context candidate hypotheses 135 generated in the grid 200 during the first pass as a basis for generating the context candidate hypotheses 145 during the second pass. Thus, the first recognition model 130 may be a streaming E2E speech recognition model that generates a streaming transcription, while the second pass second recognition model 140 includes a more computationally intensive speech recognizer that can leverage the personal context information 126 to potentially produce a more accurate description, albeit with increased latency. In some configurations, the first recognition model M1 and the second recognition model M2 share one or more encoder layers, such that audio encodings of the audio data generated during the first pass can be used by the second recognition model M2 140 during the second pass, such that M2 does not have to encode the audio data 112 again.

[0043] In other examples, the second recognition model M2 140 corresponds to a correction model of the first recognition model M1 130 generating contextual candidate hypotheses 145 after the non-contextual candidate hypotheses 135 were generated during the first pass. Here, the correction model can correspond to a second-pass language model that uses the personal context information 126 to generate contextual candidate hypotheses 145 as new hypotheses related to the user context that is completely missing from the grid 200, or to re-score existing hypotheses in the grid 200 based on the user context. In some examples, the language model associated with the second recognition model M2 140 includes a personal / user-specific language model that is trained to recognize named entities 148 of the user 10 based on the personal context information 126. In these examples, the personal / user-specific language model can include an auxiliary language model external to the first recognition model M1 130. Thus, the second recognition model M2 may receive only the grid 200 as input without receiving the audio data 110 and generate a contextual candidate hypothesis 145 that is acoustically similar to the non-contextual candidate hypothesis 135. Additionally or alternatively, the second recognition model 140 may include a bias model that injects personal named entities that are acoustically similar to the common / broad terms in the grid 200 and / or boosts the scores 155 of hypotheses that include personal named entities related to the personal context information 126 in the grid.

[0044] An important aspect of the speech recognition model 130 that combines the non-personal but personalizable first recognition model 130 with the personal second recognition model 140 is the ability to protect potentially sensitive user-specific data (e.g., personal context information 126) incorporated into the second recognition model 140. For example, the personal context information 126 can be stored on the memory hardware 116 of the user device 110, and the second recognition model 140 can be executed on the data processing hardware 114 of the user device 110 so that no user-specific data is leaked to the computing system 120 (e.g., a remote server / distributed system). In some scenarios, the second recognition model 140 is executed on the user device 110 to calculate / generate context probabilities, which are reduced to a simple weighted grammar and sent to the computing device 120 as needed. In these scenarios, the values ​​of the probability distribution 147 about possible personal named entities 148 can be sent to the computing device 120 over the network without having to share any of the user-specific data / logic used to calculate these values.

[0045] The first recognition model 130 may be executed on the user device 110 or the computing system 120. Since the first recognition model 130 is impersonal and intentionally constructed without using any user-specific data, the first recognition model 130 may be executed on the computing system 120 without revealing the user's personal information / data.

[0046] The personal context information 126 may include personal named entities 148 specific to the user 10 who spoke the utterance 101, such as, but not limited to, a list of the user's contacts, names of items in the user's media library (songs, artists, albums, movie names), names of nearby locations, names of installed applications, and user-defined entity names. The personal context information 126 may also indicate user history associated with these personal named entities, such as call frequency and usage patterns. For example, a user's media library may have music files for both the artists Phish and Hoot ie and the Blowfish, but a song by Hootie and the Blowfish may have only been played once five years ago, while the user plays songs by Phish almost every day. As a result, the personal context information 126 will allow the second recognition model M2 140 to appropriately assign a higher likelihood / probability to the candidate hypothesis "Play Harpua by Phish" than "Play Hootie and the Blowfish". The user history indicated by the personal context information 126 may also include previous queries, learned correlations, corrections to previously misidentified transcriptions, etc.

[0047] The personal context information 126 may be stored on the memory hardware 114 of the user device 110 and / or on the memory hardware 124 of the computing system 120. The information in these different context data sets will change over time, such as when a user adds or removes contacts, when a user's media library changes, when a user adds or removes installed applications, when a user changes location, and so on.

[0048] Continue to refer Figure 1 , the scoring model 150 receives the first decoder result 132 generated by the first recognition model 130 and the second decoder result 142 generated by the second recognition model 140. In some examples, the scoring model 150 receives the first decoder result 132 associated with 1 best non-contextual hypothesis 130 from the grid 200. The scoring model 150 is configured to score the contextual candidate hypothesis 145 and the non-contextual hypothesis 135 together so that the relationship of both the personal named entity and the non-personal alternative can be considered to select the best candidate hypothesis 135, 145 as the resulting transcription 175 of the utterance 101. When scoring the contextual candidate hypothesis 145 and the non-contextual candidate hypothesis 135, the scoring model 150 uses the likelihood score 155 (P s ) modeled by the first recognition model M1 130 for the non-contextual candidate hypothesis 135. M1 (call Best Buy)) and personalized likelihood score 157 (P M1 (call$CONTACT)) and the probability distribution 147 (PM2 (Beth Byer|$CONTACT)).

[0049] As is apparent, scoring the contextual candidate hypothesis 145 output by M2 140 relative to the non-contextual candidate hypothesis 135 output by M1 130 prevents the contextual candidate hypothesis 145 from being over-triggered in scenarios where it is not spoken, while preventing it from being under-recalled in scenarios where it is spoken. Continuing with the example where "call Best Buy" includes a non-contextual candidate hypothesis 135 with a generic named entity "Best Buy" and "call Beth Byer" includes a contextual candidate hypothesis 145 with a personalized named entity injected from the user's personal contact list, the personalization of the first model M1 and the incorporation of personal contextual information 126 utilized by the second model M2 allow the scoring model 300 to consider whether Beth Byer is a personal contact of the user, whether the user frequently calls either Best Buy or Beth Byer, whether either Beth Byer or BestBuy are common generic entities that are likely to be spoken, acoustically fit with the audio data 112, and whether there are other better (e.g., higher likelihood scores 155) entities available in the grid 200.

[0050] The scoring model 150 may score the contextual candidate hypotheses 145 output by M2 140 relative to the non-contextual candidate hypotheses 135 as follows.

[0051] P M1 (h with entity g )=P M1 (h with$entityclass)×P M2 (entity p |$entityclass) (1)

[0052] Where P M1 (h with entity g ) represents the likelihood score 155 assigned to the non-context candidate hypothesis 135 (e.g., call Best Buy) modeled by the first recognition model M1 130, P M1 h with $entityclass) indicates that the generic named entity (e.g., Best Buy) has been replaced with the personalized likelihood score 157 of the non-contextual candidate hypothesis 135 associated with the personalized named entity class (e.g., personal contact) 158 (e.g., $CONTACT), and P M2 (entity p|$entityclass) represents the probability distribution 147 about possible personal named entities 148 belonging to the specific personal named entity class represented by the tag 158. p The probability values ​​for each personal named entity associated with a given personal named entity category may sum to a value of 1.0. M2 (entity p |$entityclass) may include only the values ​​of the highest ranked personal named entities in probability distribution 147.

[0053] Continuing with the above example, while the personal context information 126 may indicate that "Beth Byer" is indeed one of the user's personal contacts, the context information 126 may also reveal that "Beth Byer" is a low-level personal contact that the user rarely or even never calls, which makes the context (personal) candidate hypothesis 145 relatively weak compared to the strong non-context (non-personal) candidate hypothesis 135 "callBest Buy". It is worth noting that the existing candidate hypothesis and the new candidate hypothesis are close to each other in pronunciation. It can be indicated that although Beth Byer is a contact in the user's personal contact list, Beth Byer is a low-level personal contact that the user rarely calls. As a result, applying Equation 1 to score the context candidate hypothesis 145 output by M2 140 relative to the non-context candidate hypothesis 135 will result in P M1 (call Best Buy)>P M1 (call$CONTACTS)*P M2 (Beth Byer|$CONTACTS), whereby the non-contextual candidate hypothesis 135 will not be replaced and will be appropriately selected as the transcription 175 of the utterance 101.

[0054] In another example in which the user 10 utters the utterance 101 “call Leonid” (e.g., a personal contact of the user), the first model M1 130 may generate the non-contextual candidate hypothesis 135 “call lee oh need” and the second model M2 140 may generate the contextual candidate hypothesis 145 “call Leonid”. Although the two hypotheses 135, 145 are phonetically close to each other, the likelihood score 155 assigned to the non-contextual candidate hypothesis 135 may indicate that the non-contextual candidate hypothesis 135 is meaningless and unlikely to be uttered, while the probability distribution 147 about possible personal named entities 148 belonging to the personal contact category may include “Leonid” with a high probability based on the personal contextual information 126 indicating that “Leonid” is one of the user’s personal contacts that the user calls regularly. As a result, applying Equation 1 to score the contextual candidate hypothesis 145 output by M2 140 relative to the non-contextual candidate hypothesis 135 will result in a probability score of P 155 of 0. M1 (call lee oh need) <P M1 (call$CONTACTS)*P M2 (Leonid|$CONTACTS), whereby the context candidate hypothesis 145 will be appropriately selected as the transcription 175 of the utterance 101.

[0055] The scoring model 150 may evaluate and score the context candidate hypotheses 145 relative to a single non-context candidate hypothesis 135 and its corresponding likelihood score 155. The single non-context candidate hypothesis 135 may be carefully selected for better scoring accuracy based on, for example, high acoustic proximity to the new candidate hypothesis 145 and / or familiarity with similar hypotheses used as a reference point. The single non-context candidate hypothesis 135 may be selected as the 1 best hypothesis 135 in the grid. Alternatively, the scoring model 150 may evaluate and score the context candidate hypotheses 145 relative to multiple non-context candidate hypotheses 135 (or even all hypotheses in the original grid 200) and their corresponding likelihood scores 155 to provide more accurate scoring, but at the expense of increased computational burden (resulting in increased latency and increased memory requirements).

[0056] Continue to refer Figure 1 During stage (F), reranker 160 receives scoring output 152 from scoring model 150, which scores context candidate hypotheses 145 relative to non-context candidate hypotheses 135 and their corresponding likelihood scores 155. Reranker 160 is configured to output a reranking result 165 including rankings of non-context candidate hypotheses 135 and context candidate hypotheses 145 based on the scoring output. In the example shown, reranking result 165 includes non-context candidate hypothesis 135 with a score of 0.9 as the most likely correct transcription 175.

[0057] At stage (G), computing system 120 is configured to generate a transcription 175 of utterance 101 spoken by user 10 by selecting the highest ranked candidate in re-ranked results 165. In the example shown, computing system 120 selects non-contextual candidate hypothesis 135 "call Best Buy" because it has the highest score of 0.9. Computing system 120 may transmit transcription 175 to client device 110 via network 118 to provide transcription 175 to user 10. In some implementations, client device 110 performs all stages (A) to (G) on the device without connecting to computing system 120 via network 118.

[0058] While the above examples depict scoring contextual candidate hypotheses relative to non-contextual candidate hypotheses for audio-based speech input, aspects of the present disclosure are equally applicable to scoring contextual candidate hypotheses relative to non-contextual candidate hypotheses for non-speech input. For example, the non-speech input 113 may include a user input indication indicating the selection of one or more characters of a keyboard 117 (e.g., a virtual or physical keyboard) in communication with the data processing hardware 114 of the client device 110. In this example, the speech recognizer module 130 may be replaced with a keyboard detection module configured to generate a grid 200 of non-contextual candidate hypotheses 135 for the non-speech input 113. Models M1 and M2 may function similarly to the above examples by generating non-contextual candidate hypotheses 135 and contextual candidate hypotheses 145 for the non-speech input 113. The scoring model 150 may then score the contextual candidate hypotheses relative to the non-contextual candidate hypotheses, thereby providing the ability to rank personal / user-specific non-speech recognition hypotheses relative to non-personal hypotheses that include general terms on which the keyboard detection module may not have been trained.

[0059] Reference now Figure 3In some implementations, the example training process 300 trains the first recognition model M1 130 as an aggregate lacking any user-specific data to personalize the first recognition model M1 130. Specifically, the training process 300 may train the first model M1 130 on an ASR training corpus 320 that includes a corpus of true value transcriptions 325. In some examples, the personal named entity extractor 302 may replace the identified personal named entities of the transcription 325 with corresponding tags associated with the personal named entity category to which the identified personal named entity belongs. For example, the extractor 302 may identify "Nick" in the transcription "Call Nick" as a personal named entity belonging to a personal contact category and replace "Nick" with the corresponding tag "$CONTACT". The extractor 302 may identify the personal named entities through techniques such as parsing and / or explicit tags provided in the transcription 325.

[0060] Optionally, the training process 300 can additionally train the first model M1 130 with generic transcriptions that include only generic named entities. Here, the extractor 302 can determine when a given transcription 325 omits a personal named entity and thus does not replace any terms in the transcription 325 with tokens. In this scenario, the training process 300 also trains the first model M1 130 on the generic transcription 325 to teach the model M1 130 to evaluate the relative merits of a personal hypothesis that includes a personal named entity and an impersonal hypothesis that includes only generic terms. For example, the extractor 302 can determine that "Best Buy" is a generic entity in the transcription "Call BestBuy" and thus allow the transcription 325 to train the model M1 130 without replacing any terms with tokens.

[0061] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0062] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0063] Figure 4 1 is a flow chart of an example operational arrangement of a method 400 for scoring a contextual candidate hypothesis 145 relative to a non-contextual candidate hypothesis. Figure 1 ) can execute the stored in the memory hardware 124 ( Figure 1 ) to perform an example operational arrangement of method 400. At operation 402, method 400 includes receiving audio data 112 corresponding to an utterance 101 spoken by user 10. At operation 404, method 400 includes processing the audio data 112 using a first recognition model 130 to generate non-contextual candidate hypotheses 135 as outputs of the first recognition model 130. Here, the non-contextual candidate hypotheses 135 correspond to candidate transcriptions of the utterance and have corresponding likelihood scores 155 assigned to the non-contextual candidate hypotheses 135 by the first recognition model 130.

[0064] At operation 406, the method 400 includes generating a context candidate hypothesis 145 including a personal named entity using the second recognition model 140 configured to receive the personal context information 126. Here, the context candidate hypothesis corresponds to another candidate transcription of the utterance.

[0065] At operation 408, the method 400 includes scoring the context candidate hypothesis 145 relative to the non-context candidate hypothesis 135 based on the personal context information 126 and the corresponding likelihood score 155 assigned to the non-context candidate hypothesis 135. Based on scoring the context candidate hypothesis relative to the non-context candidate hypothesis, the method 400 also includes generating a transcription 175 of the utterance 101 spoken by the user 10 by selecting one of the context candidate hypothesis 145 or the non-context candidate hypothesis 135 at operation 410.

[0066] Figure 55 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0067] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0068] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0069] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0070] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590 (which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)) can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0071] As shown, computing device 500 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 500a or multiple implementations in a group of such servers 500a, implemented as a laptop computer 500b, or implemented as part of a rack server system 500c.

[0072] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be special purpose or general purpose and that may be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0073] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0074] The processes and logic flows described in this specification may be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by dedicated logic circuit systems, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operatively coupled to receive data from a mass storage device or to transfer data to a mass storage device or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0075] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0076] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (400) which, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations comprising: Receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate non-context candidate hypotheses (135) as output of the first recognition model, the non-context candidate hypotheses (135) corresponding to candidate transcriptions (175) of the utterance (101) and having corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135) by the first recognition model (130); generating, using a second recognition model configured to receive personal context information (126), a context candidate hypothesis (145) including a personal named entity (148), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypothesis (145) relative to the non-context candidate hypothesis (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); as well as Based on scoring the context candidate hypothesis (145) relative to the non-context candidate hypothesis (135), a transcription (175) of the utterance (101) spoken by the user is generated by selecting one of the context candidate hypothesis (145) or the non-context candidate hypothesis (135).

2. The computer-implemented method (400) of claim 1, wherein the first recognition model (130) processes the audio data (112) to generate the non-contextual candidate hypothesis (135) without incorporating any of the personal context information (126) associated with the user.

3. A computer-implemented method (400) as described in claim 1 or 2, wherein the first recognition model (130) includes an end-to-end speech recognition model (130), and the end-to-end speech recognition model is configured to generate the corresponding likelihood score (155) for the non-context candidate hypothesis (135).

4. The computer-implemented method (400) of any one of claims 1 to 3, wherein: The second recognition model (140) comprises an end-to-end speech recognition model (140) configured to generate the context candidate hypothesis (145); and Wherein generating the context candidate hypothesis (145) including the personal named entity (148) includes processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypothesis (145) based on the personal context information (126).

5. A computer-implemented method (400) as described in claim 4, wherein the end-to-end speech recognition model (140) is configured to generate the context candidate hypothesis (145) after generating the non-context candidate hypothesis (135) as an output of the first recognition model (130).

6. A computer-implemented method (400) as described in claim 4, wherein the end-to-end speech recognition model (140) is configured to generate the context candidate hypothesis (145) in parallel with processing the audio data (112) to generate the non-context candidate hypothesis (135) as an output of the first recognition model (130).

7. The computer-implemented method (400) of any one of claims 1 to 6, wherein the second recognition model (140) comprises a language model.

8. The computer-implemented method (400) of any one of claims 1 to 7, wherein the personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) including at least one of: The name of a contact in the user's personal contact list; a name in the media library associated with the user; The name of the installed application; the name of a nearby location, or User-defined entity name.

9. The computer-implemented method (400) of any one of claims 1 to 8, wherein the personal context information (126) associated with the user further indicates a user history related to the personal named entity (148) specific to the user.

10. The computer-implemented method (400) of any one of claims 1 to 9, wherein the operations further comprise: receiving a training corpus of ground-truth transcriptions; For each corresponding ground-truth transcription (175) in the training corpus of ground-truth transcriptions: determining whether a personal named entity (148) is identified in the corresponding ground-value transcription (175); and When the personal named entity (148) is identified in the corresponding ground-value transcription (175), replacing the personal named entity (148) with an entity category tag (158); and The first recognition model is trained on the ground-truth transcription including personal named entities (148) replaced by the entity category token.

11. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) in communication with the data processing hardware (510) and storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations comprising: Receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate non-context candidate hypotheses (135) as outputs of the first recognition model (130), the non-context candidate hypotheses (135) corresponding to candidate transcriptions (175) of the utterance (101) and having corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135) by the first recognition model (130); Using a second recognition model (140) configured to receive personal context information (126), generating a context candidate hypothesis (145) including a personal named entity (148), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypothesis (145) relative to the non-context candidate hypothesis (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); and Based on scoring the context candidate hypothesis (145) relative to the non-context candidate hypothesis (135), a transcription (175) of the utterance (101) spoken by the user is generated by selecting one of the context candidate hypothesis (145) or the non-context candidate hypothesis (135).

12. The system (100) of claim 11, wherein the first recognition model (130) processes the audio data (112) to generate the non-contextual candidate hypothesis (135) without incorporating any of the personal context information (126) associated with the user.

13. A system (100) as described in claim 11 or 12, wherein the first recognition model (130) includes an end-to-end speech recognition model (130), and the end-to-end speech recognition model is configured to generate the corresponding likelihood score (155) for the non-context candidate hypothesis (135).

14. The system (100) according to any one of claims 11 to 13, wherein: The second recognition model (140) comprises an end-to-end speech recognition model (140) configured to generate the context candidate hypothesis (145); and Wherein generating the context candidate hypothesis (145) including the personal named entity (148) includes processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypothesis (145) based on the personal context information (126).

15. The system (100) of claim 14, wherein the end-to-end speech recognition model (140) is configured to generate the context candidate hypothesis (145) after generating the non-context candidate hypothesis (135) as an output of the first recognition model (130).

16. The system (100) of claim 14, wherein the end-to-end speech recognition model (140) is configured to generate the context candidate hypothesis (145) in parallel with processing the audio data (112) to generate the non-context candidate hypothesis (135) as an output of the first recognition model (130).

17. The system (100) of any one of claims 11 to 16, wherein the second recognition model comprises a language model.

18. The system (100) of any one of claims 11 to 17, wherein the personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) including at least one of the following: The name of a contact in the user's personal contact list; a name in the media library associated with the user; The name of the installed application; the name of a nearby location, or User-defined entity name.

19. The system (100) of any one of claims 11 to 18, wherein the personal context information (126) associated with the user further indicates a user history related to the personal named entity (148) specific to the user.

20. The system (100) of any one of claims 11 to 19, wherein the operations further comprise: receiving a training corpus of ground-truth transcriptions; For each corresponding ground-truth transcription (175) in the training corpus of ground-truth transcriptions: determining whether a personal named entity (148) is identified in the corresponding ground-value transcription (175); and When the personal named entity (148) is identified in the corresponding ground-value transcription (175), replacing the personal named entity (148) with an entity category tag (158); and The first recognition model is trained on the ground-truth transcription including personal named entities (148) replaced by the entity category token.

Images