Personalizable Probabilistic Models

Abstract

The method (400) includes receiving audio (112) and using a first recognition model (130) to process the audio data and generate non-contextual candidate hypotheses (135) as output from the first recognition model. The non-contextual candidate hypotheses have corresponding likelihood scores (155) assigned by the first recognition model. The method further includes using a second recognition model (140) configured to receive personal context information (126) to generate contextual candidate hypotheses (145) including personal named entities (148). The method further includes scoring the contextual candidate hypotheses against the non-contextual candidate hypotheses based on the personal context information and the corresponding likelihood scores assigned to the non-contextual candidate hypotheses. The method further includes generating a transcription (175) of the utterance by selecting one of the contextual candidate hypotheses or the non-contextual candidate hypotheses.

Description

[Technical Field]

[0001] The present disclosure relates to personalizable probabilistic models. [Background technology]

[0002] Modern automatic speech recognition (ASR) systems focus on providing not only quality / accuracy (e.g., low word error rate (WER)) but also low latency (e.g., a short delay between when a user speaks and when a transcription appears). Due to the sparsity of the generalized training data used to train ASR systems when performing speech recognition tasks, ASR systems have difficulty recognizing specialized phrases, including medical / legal terms and proper nouns such as personal contacts, song / artist names, application names, and emerging entity names. By the same token, ASR systems have difficulty recognizing terms in heterolingual speech in a language different from the language of the training data. As a result, speech recognition lattices often rank these specialized phrases poorly or omit them entirely. Summary of the Invention

[0003] One aspect of the present disclosure provides a method for scoring contextual candidate hypotheses relative to non-contextual candidate hypotheses, the method causing the data processing hardware to perform operations when executed. The operations include receiving audio data corresponding to an utterance spoken by a user and processing the audio data using a first recognition model to generate non-contextual candidate hypotheses as output from the first recognition model, where the non-contextual candidate hypotheses correspond to candidate transcriptions of the utterance and have corresponding likelihood scores assigned to the non-contextual candidate hypotheses by the first recognition model. The operations further include generating contextual candidate hypotheses including personal named entities using a second recognition model configured to receive personal context information, where the contextual candidate hypotheses correspond to other candidate transcriptions of the utterance. The operations further include scoring the contextual candidate hypotheses relative to the non-contextual candidate hypotheses based on the personal context information and the corresponding likelihood scores assigned to the non-contextual candidate hypotheses, and generating a transcription of the utterance spoken by the user by selecting one of the contextual candidate hypotheses or the non-contextual candidate hypotheses based on the scoring of the contextual candidate hypotheses relative to the non-contextual candidate hypotheses.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the first recognition model processes the audio data to generate non-context candidate hypotheses without incorporating any personal contextual information associated with a user. In some examples, the first recognition model includes an end-to-end speech recognition model configured to generate corresponding likelihood scores for the non-context candidate hypotheses.

[0005] In some implementations, the second recognition model includes an end-to-end speech recognition model configured to generate context candidate hypotheses, where generating context candidate hypotheses including personal named entities includes processing the audio data using the end-to-end speech recognition model to generate context candidate hypotheses based on personal context information. In these implementations, the end-to-end speech recognition model may be configured to generate the context candidate hypotheses after non-context candidate hypotheses are generated as output from the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate the context candidate hypotheses in parallel with processing the audio data to generate non-context candidate hypotheses as output from the first recognition model.

[0006] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user, where the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby place, or a user-defined entity name. In some examples, the personal context information associated with the user further indicates a user history associated with the personal named entity specific to the user.

[0007] In some implementations, the operations further include receiving a training corpus of ground truth transcriptions. For each corresponding ground truth transcription in the training corpus of ground truth transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding ground truth transcription, and when the personal named entity is identified in the corresponding ground truth transcription, replacing the personal named entity with a token of the entity class. Here, the operations further include training the first recognition model with the ground truth transcriptions including the personal named entity replaced by the token of the entity class.

[0008] Another aspect of the present disclosure provides a system for scoring contextual candidate hypotheses against non-contextual candidate hypotheses. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including receiving audio data corresponding to an utterance spoken by a user and processing the audio data using a first recognition model to generate non-contextual candidate hypotheses as output from the first recognition model, where the non-contextual candidate hypotheses correspond to candidate transcriptions of the utterance and have corresponding likelihood scores assigned to the non-contextual candidate hypotheses by the first recognition model. The operations further include generating contextual candidate hypotheses including personal named entities using a second recognition model configured to receive personal context information, where the contextual candidate hypotheses correspond to other candidate transcriptions of the utterance. The operations further include scoring the contextual candidate hypotheses relative to the non-contextual candidate hypotheses based on the personal context information and corresponding likelihood scores assigned to the non-contextual candidate hypotheses, and generating a transcription of the utterance spoken by the user by selecting one of the contextual candidate hypotheses or the non-contextual candidate hypotheses based on the scoring of the contextual candidate hypotheses relative to the non-contextual candidate hypotheses.

[0009] This aspect may include one or more of the following optional features: In some implementations, the first recognition model processes the audio data to generate non-context candidate hypotheses without incorporating any personal contextual information associated with a user. In some examples, the first recognition model includes an end-to-end speech recognition model configured to generate corresponding likelihood scores for the non-context candidate hypotheses.

[0010] In some implementations, the second recognition model includes an end-to-end speech recognition model configured to generate context candidate hypotheses, where generating context candidate hypotheses including personal named entities includes processing the audio data using the end-to-end speech recognition model to generate context candidate hypotheses based on personal context information. In these implementations, the end-to-end speech recognition model may be configured to generate the context candidate hypotheses after non-context candidate hypotheses are generated as output from the first recognition model. Alternatively, the end-to-end speech recognition model is configured to generate the context candidate hypotheses in parallel with processing the audio data to generate non-context candidate hypotheses as output from the first recognition model.

[0011] In some examples, the second recognition model includes a language model. In some implementations, the personal context information associated with the user includes a personal named entity specific to the user, where the personal named entity includes at least one of a contact name in the user's personal contact list, a name in a media library associated with the user, a name of an installed application, a name of a nearby place, or a user-defined entity name. In some examples, the personal context information associated with the user further indicates a user history associated with the personal named entity specific to the user.

[0012] In some implementations, the operations further include receiving a training corpus of ground truth transcriptions. For each corresponding ground truth transcription in the training corpus of ground truth transcriptions, the operations further include determining whether a personal named entity is identified in the corresponding ground truth transcription, and when the personal named entity is identified in the corresponding ground truth transcription, replacing the personal named entity with a token of the entity class. Here, the operations further include training the first recognition model with the ground truth transcriptions including the personal named entity replaced by the token of the entity class.

[0013] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is an exemplary system for scoring a first candidate hypothesis generated for an utterance by a first recognizer model and a second candidate hypothesis generated for the same utterance by a second recognizer model. [Figure 2A] FIG. 1 is a schematic diagram showing an example of a word lattice. [Figure 2B] FIG. 1 is a schematic diagram showing an example of a word lattice. [Figure 3] FIG. 2 is a schematic diagram illustrating a training process for training a first speech recognition model used by the system of FIG. 1. [Figure 4] 1 is a flowchart of an example arrangement of operations of a method for scoring a first candidate hypothesis generated for an utterance by a first speech recognition model and a second candidate hypothesis generated for the same utterance by a second speech recognition model. [Figure 5]FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] Automatic speech recognition (ASR) systems have become increasingly prevalent on client devices as they continue to provide increasingly accurate transcriptions of what users say. Recently, end-to-end (E2E) ASR models have gained popularity by achieving state-of-the-art performance in accuracy and latency. In contrast to traditional hybrid ASR systems with separate acoustic, pronunciation, and language models, E2E ASR models apply a sequence-to-sequence approach to jointly learn acoustic and language modeling in a single neural network that is trained end-to-end from training data, e.g., utterance-transcription pairs. Nevertheless, in some instances, ASR models misrecognize what a user actually said and produce inaccurate transcriptions. This is often the case when a user speaks a personal named entity that is rare or absent in the training data used to train the ASR model. As a result, the correct hypothesis for the personal named entity has a low rank or is completely missing in the recognition lattice of possible speech recognition hypotheses. As used herein, a personal named entity refers to any of the names of personal contacts, names in a media library (e.g., songs, artists, albums, movies, etc.), installed applications, and user-defined names (e.g., myLivingRoomStreamingDevice).

[0017] Existing techniques for improving recognition of personal named entities involve biasing, typically by prioritizing relevant speech recognition hypotheses containing the personal named entity or by creating new candidate hypotheses that inject the personal named entity with close acoustic, graphemic, or embedding proximity to existing hypotheses. However, an often-overlooked consideration is how to accurately score contextual (user-specific or personal) candidate hypotheses for utterances that contain the personal named entity relative to non-contextual candidate hypotheses for utterances that lack user-specific data and thus lack the personal named entity. If contextual hypotheses are scored separately from existing non-contextual hypotheses, as is typical when applying the existing biasing techniques described above, the ASR system risks over-triggering contextual hypotheses that were not spoken or under-recalling non-contextual hypotheses even when spoken. For example, consider the case where an ASR model generates the non-contextual hypothesis "call lee oh need" in response to the utterance "call Leonid." Accurately scoring a new hypothesis such as "Call Leonid" must consider whether "Lee Au Need" and "Leonid" are personal contacts, whether either "Lee Au Need" or "Leonid" are common entities likely to be spoken in an utterance, the prior likelihood of the non-contextual hypothesis, and whether other, better non-contextual hypotheses exist.

[0018] Accurately ranking contextual hypotheses alongside non-contextual hypotheses requires accurate scoring of the contextual hypotheses. ASR models that generate and score non-contextual hypotheses are not suitable for scoring contextual hypotheses because they have already failed to recall or assigned low scores during decoding. On the other hand, context-specific models, such as bias models that bias speech recognition results against personal named entities, are not capable of scoring and ranking speech recognition hypotheses that include personal named entities relative to the general entities on which the ASR model is trained.

[0019] Embodiments herein are directed to a speech recognizer module that includes a first (generic) model (M1) that does not include any user-specific data and a second (personal) model (M2) that is calculated and dynamically updated for a given user. In particular, a computing system receives audio data of an utterance spoken by the user, and the computing system processes the audio data to generate non-contextual candidate hypotheses of the utterance as output from the first model (M1) and contextual candidate hypotheses of the utterance as output from the second model (M2). In some examples, the first model (M1) is configured to generate multiple non-contextual hypotheses representing possible transcriptions for the utterance in a word lattice. As used herein, non-contextual candidate hypotheses refer to possible transcriptions for the utterance generated by the first model (M1) without incorporating any user-specific or personal data associated with the given user. In this manner, the first model is effective for modeling the recognition of general entities spoken in essentially impersonal speech, but is not generally suitable for recognizing personal, named entities.

[0020] Meanwhile, the second model (M2) is configured to generate context candidate hypotheses containing personal named entities by leveraging personal context information related to the user. The context information may indicate user history associated with a given personal named entity, such as invocation frequency and usage patterns. More specifically, the second model (M2) corresponds to a user-specific model that assigns probabilities to possible personal named entities for a particular entity class based on the personal context information. In this manner, the second model (M2) is effective for modeling the recognition of personal named entities spoken in an utterance. Embodiments herein are particularly directed to selecting a transcription of an utterance by combining the first model (M1) and the second model (M2) to determine likelihood scores for both non-contextual and contextual hypotheses and selecting the highest likelihood score of either the non-contextual or context candidate hypotheses. As used herein, a likelihood score indicates the probability that the corresponding candidate hypothesis is a correct transcription of the utterance spoken by the user.

[0021] 1 illustrates an example of a system 100 for automatic speech recognition (ASR) of an utterance 101 spoken by a user 10 using audio data 112 corresponding to the utterance (i.e., a query) 101. The system 100 includes a client device 110, a computing system 120, and a network 118. The computing system 120 may be a distributed system (e.g., a cloud computing environment) with scalable, elastic resources. The resources include computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). The network 118 may be wired, wireless, or a combination thereof, and may include private and / or public networks, such as the Internet.

[0022] In some examples, the computing system 120 receives the audio data 112 from the client device 110, and the computing system 120 processes the audio data 112 to generate one or more non-context candidate hypotheses 135 for the utterance 101 based on the audio data 112 as output from the first recognizer model 130. In some further examples, the client device 110 processes the audio data 112 entirely on-device to generate one or more non-context candidate hypotheses 135 for the utterance 101 based on the audio data 112, where each non-context candidate hypothesis 135 corresponds to a candidate transcription of the utterance 101 generated without using any personal context information 126 and is represented by a respective sequence of hypothesized terms.

[0023] As described in more detail below, a second recognizer model 140 (executing on the client device 110 or the computing system 120) leveraging the personal context information 126 generates a context candidate hypothesis 145 for the utterance 101 that was not included in the existing candidate hypotheses 135 or that was ranked very low among the existing candidate hypotheses in the lattice 200. Here, the context candidate hypothesis 145 corresponds to a candidate transcription of the same utterance 101 generated by incorporating the personal context information 126. In this way, the personal context information 126 leveraged by the second recognizer model 140 assists in the recognition of personal named entities. The scoring model 150 then scores the context candidate hypothesis 145 along with one of the non-context candidate hypotheses 135, and the computing system 120 (or the user device 110) generates a transcription 175 of the utterance 101 by selecting the highest-ranked hypothesis from among the non-context candidate hypotheses 135 or the context candidate hypothesis 145 scored by the scoring model 150.

[0024] 1 shows operations (A)-(G) illustrating the flow of data. As described herein, computing system 120 performs operations (B)-(G). However, it is understood that client device 110 may also perform operations (B)-(G) in addition to, or instead of, computing system 120 performing the operations. In some examples, client device 110 performs a first portion of an operation (e.g., operation (C)) and computing system 120 performs a second portion of the operation (e.g., operations (D)-(F)), or vice versa.

[0025] The client device 110 includes data processing hardware 114 and memory hardware 116. The client device 110 may include one or more audio capture devices (e.g., microphone(s)) 103 for capturing and converting utterances 101 from the user 10 into audio data 112 (e.g., electrical signals). In some examples, the microphone 103 is separate from the client device 110 and communicates with the client device 110 to provide the recorded utterances 101 to the client device 110. The client device 110 can be any computing device capable of communicating with a computing system 120 over a network 118. In lieu of spoken utterances 101, the user 10 may input text utterances 101 by means of a keyboard 117, such as a virtual keyboard displayed in a graphical user interface of the client device 110 or a physical keyboard in communication with the client device 110. Client devices 110 include, but are not limited to, desktop computing devices and mobile computing devices such as laptops, tablets, smartphones, smart speakers / displays, smart appliances, Internet of Things (IoT) devices, and wearable computing devices (e.g., headphones, headsets, and / or watches).

[0026] In the example of FIG. 1 , during stage (A), user 10 speaks utterance 101, and microphone 103 of client device 110 records utterance 101. In this example, utterance 101 includes user 10 saying, "call Best Buy." Client device 110 transmits audio data 112 corresponding to utterance 101 recorded by microphone 103 to computing system 120 over network 118. During stage (B), computing system 120 processes audio data 112 to generate one or more non-contextual candidate hypotheses 135 as output from first recognition model 130, where each candidate hypothesis 135 corresponds to a candidate transcription of utterance 101 and is represented by a respective sequence of hypothesized terms. For example, the computing system 120 may execute a speech recognizer module 30 including a first recognition model 130 (e.g., an automatic speech recognition (ASR) module) to generate a word lattice 200 indicating possible non-context candidate hypotheses 135 for the utterance 101 based on the audio data 112 without incorporating any personal context information 126. The first recognition model 130 may evaluate potential paths through the word lattice 200 to determine the non-context candidate hypotheses 135.

[0027] Figure 2A shows an example of a word lattice 200, 200a that may be provided by the first recognition model M1 130 of Figure 1. The word lattice 200a represents multiple possible combinations of words that may form different non-context candidate hypotheses 135 for the utterance 101.

[0028] The word lattice 200a includes one or more nodes 202a-202g corresponding to possible boundaries between words. The word lattice 200a includes multiple edges 204a-l of possible candidate hypothesis words arising from the word lattice 200a. Additionally, each of the edges 204a-204l may have one or more weights or probabilities that the edge is the correct edge from the corresponding node. The weights are determined by M1 130 and may be based, for example, on the confidence in the match between the speech data and the edge's word and how well the word fits grammatically and / or lexically with other words in the word lattice 200a.

[0029] For example, initially, the most likely path through the word lattice 200a (e.g., the most likely non-context candidate hypothesis 135) may include edges 204c, 204e, 204i, and 204k with the text "we're coming about 11:30." A second optimal path through the word lattice 200a (e.g., the second optimal non-context candidate hypothesis 135) may include edges 204d, 204h, 204j, and 204l with the text "deer hunting scouts 7:30."

[0030] Each pair of nodes may have one or more paths corresponding to alternative words in various non-context candidate hypotheses 135. For example, the initial most likely path between a pair of nodes starting at node 202a and ending at node 202c is edge 204c "we're," which has alternative paths including edges 204a, 204b "we are" and edge 204d "deer."

[0031] Figure 2B is an example of a hierarchical word lattice 200, 200b that may be provided by the first recognition model 130 of Figure 1. The word lattice 200b includes nodes 252a-252l that represent words that make up the various non-context candidate hypotheses 135 for the utterance 101. The edges between nodes 252a-252l indicate that the possible candidate hypotheses 135 are (1) nodes 252c, 252e, 252i, and 252k "we're coming about 11:30", (2) nodes 252a, 252b, 252e, 252i, and 252k "we are coming about 11:30", (3) nodes 252a, 252b, 252f, 252g, 252i, and 252k "we are coming at about 11:30", and (4) nodes 252d, 252f, 252g, 252i, and 252k "deer come at about (5) nodes 252d, 252h, 252j, and 252k "deer hunting scouts 11:30," and (6) nodes 252d, 252h, 252j, and 252l "deer hunting scouts 7:30."

[0032] Again, edges between nodes 242a-242l may have associated weights or probabilities based on the confidence of the speech recognition (e.g., candidate hypotheses) and grammatical / lexical analysis of the resulting text. In this example, "we're coming about 11:30" may be the current best hypothesis, and "deer hunting scouts 7:30" may be the next best hypothesis. One or more sections 354a-354d may be created in word lattice 200b that group words and their alternatives together. For example, section 254a includes the word "we're" and the alternatives "we are" and "deer." Section 252b includes the word "coming" and the alternatives "come at" and "hunting." Section 254c contains the word "about" and the alternative "scouts," and section 254d contains the word "11:30" and the alternative "7:30."

[0033] 1 , the first recognition model M1 130 may generate first decoding results 132 that model the non-context candidate hypotheses 135 from the word lattice 200 and corresponding likelihood scores 155 assigned to the non-context candidate hypotheses 135. Each likelihood score 155 indicates the probability that the candidate hypothesis 135 is correct (e.g., matches the utterance 101). In some implementations, the first recognition model M1 130 includes an end-to-end (E2E) speech recognition model configured to receive the audio data 112 and generate the word lattice 200. In particular, the E2E speech recognition model processes the audio data 112 to generate the first decoding results 132 from the word lattice 200, the first decoding results 132 indicating the likelihood scores 155 for each of the multiple non-context candidate hypotheses 135. In some examples, the first recognition model 130 includes separate acoustic models, language models, and / or pronunciation models.

[0034] In particular, because the first recognition model 130 corresponds to a general-purpose speech recognizer trained with general-purpose data, the first recognition model M1 130 will be less accurate at recognizing speech in specialized domains that use specialized phrases (e.g., voice commands, medical terms, legal terms) and / or named entities (e.g., names of personal contacts, song / artist names, application names). The reduced accuracy in recognizing speech in these specialized domains is due to a lack of general-purpose data used to train the speech recognizer module, foreign or rare words, the emergence of new entities, and out-of-vocabulary terms that were not present in the general-purpose data.

[0035] In some examples, the speech recognizer module 130 includes an acoustic model and / or a language model for generating a word lattice 200 or otherwise generating a plurality of candidate hypotheses 135 for the utterance 101 based on the audio data 112. Here, the likelihood scores 155 of the plurality of candidate hypotheses 135 may include a combination of acoustic modeling scores from the acoustic model and / or prior likelihood scores from the language model. In other words, the likelihood scores 155 include at least one of the acoustic modeling scores output by the acoustic model and / or the prior likelihood scores output by the language model.

[0036] As described in more detail below with reference to FIG. 3 , the first recognition model M1 130 can be trained as a collective without any user-specific data, or the first recognition model M1 130 can be personalized such that the first recognition model M1 130 recognizes the user context associated with the input utterance 101 but does not incorporate or use any user-specific or personal data associated with the given user when generating the first decoding result 132. For example, M1 130 can be trained on annotated utterances in which personal entities are identified and replaced with tokens 158. In some examples, different tokens 158 representing personal named entity classes are used to replace named entities that may belong to specific named entity classes. In the illustrated example, the first recognition model M1 130 also models / generates individual likelihood scores 157 for a given non-contextual candidate hypothesis 135 having a recognized named entity (e.g., Best Buy) replaced with a corresponding token 158 (e.g., $CONTACT). Other tokens 158 may be used to indicate other personal named entity classes, such as, but not limited to, application (e.g., $APP), song title (e.g., $SONG), artist name (e.g., $ARTIST), and album name (e.g., $ALBUM). Thus, for the non-contextual candidate hypothesis 135 "Call Best Buy," the first decoder result 132 also provides a separate likelihood score 157 for "Call $CONTACT." By modeling both the likelihood score 155 for the non-contextual hypothesis 135 and the separate likelihood score 157 for the non-contextual hypothesis 135 with the named entity replaced with a token 158 representing a personal named entity class, the first recognition model 130 can be used to assess the relative value of the personal (contextual) hypothesis 145 and the non-contextual (non-personal) candidate hypothesis 135.In contrast, a traditional voice-biased approach would unilaterally prioritize "Call $CONTACT" without considering the relative value of non-contextual / impersonal alternatives to token 158.

[0037] During stage (C), the computing system 120 generates, as output from the second recognition model M2 140, context candidate hypotheses 145 that are not included in the lattice 200 output by the first recognition model M1 130 or that are ranked very low among the non-context candidate hypotheses of the lattice 200 during the first pass. When generating the non-context candidate hypotheses 135, the first recognition model M1 130 does not incorporate any user-specific data, whereas the second recognition model M2 140 is a personalized recognition model tailored to a given user by incorporating personal context information 126 to aid in the recognition of personal named entities. More specifically, the second recognition model M2 140 generates a second decoding result 142 that includes a probability distribution 147 regarding likely personal named entities 148 that belong to a particular personal named entity class indicated by the token 158. Here, the likely personal named entities 148 may include all of the user's personal contacts based on the personal context information 126, including the user's contact list. The personal context information 126 may further indicate the invocation frequency and usage patterns of each personal named entity 148 in the probability distribution 147 associated with a particular personal named entity class. In this manner, the probability distribution 147 may assign a higher probability score to a personal named entity 148 that the user accesses frequently (and / or accesses at a time associated with the current time at which the utterance 101 is received) than to a personal named entity 148 that the user never or rarely speaks. Accordingly, the second recognition model 140 may adjust the score assigned to a context candidate hypothesis 145 having a personal named entity 148 based on how “contextually relevant” the personal named entity 148 is. In the illustrated example, the personal named entity 148 “Beth Byer” highest in the probability distribution 147 includes the user’s lower-tier personal contacts that have not been accessed recently.Therefore, the resulting score will be lower than if the best buyers were high-tier personal contacts that the user frequently visits / calls.

[0038] In some examples, the second recognition model M2 140, like the first recognition model M1 130, includes an E2E speech recognition model, where the E2E speech recognition model is configured to receive the audio data 112 and the personal context information 126 and generate the second decoding results 142. In these examples, the first recognition model M1 and the second recognition model M2 may process the audio data 112 to generate the corresponding first decoding results 132 and second decoding results 142 in parallel. Alternatively, the first recognition model M1 130 may generate the non-context candidate hypotheses 135 in the lattice 200 during a first pass, and then the second recognition model M2 140 may generate the context candidate hypotheses 145 during a second pass. In this scenario, the second recognition model M2 140 may also receive one or more of the non-context candidate hypotheses 135 generated in the lattice 200 during the first pass as a basis for generating the context candidate hypotheses 145 during the second pass. Thus, the first recognition model 130 may be a streaming E2E speech recognition model that generates streaming transcriptions, while the second pass second recognition model 140 may include a more computationally intensive speech recognizer and leverage personal context information 126 to generate more accurate descriptions at the expense of increased latency. In some configurations, the first recognition model M1 and the second recognition model M2 share one or more encoder layers, and audio encoding of the audio data generated during the first pass may be used by the second recognition model M2 140 during the second pass, thereby avoiding the need for M2 to re-encode the audio data 112.

[0039] In other examples, the second recognition model M2 140 corresponds to a correction model that generates context candidate hypotheses 145 after the first recognition model M1 130 generates non-context candidate hypotheses 135 during the first pass. In this case, the correction model may correspond to a second-pass language model. The second-pass language model uses personal context information 126 to generate context candidate hypotheses 145 as new hypotheses related to user context that were entirely missing from the lattice 200, and / or rescores existing hypotheses in the lattice 200 based on the user context. In some examples, the language model associated with the second recognition model M2 140 includes a personal / user-specific language model trained to recognize named entities 148 specific to the user 10 based on the personal context information 126. In these examples, the personal / user-specific language model may include an auxiliary language model external to the first recognition model M1 130. In this manner, the second recognition model M2 may receive only the lattice 200 as input without using the audio data 110 and may generate context candidate hypotheses 145 that are acoustically similar to the non-context candidate hypotheses 135. Additionally or alternatively, the second recognition model 140 may include a bias model that injects personal named entities that are acoustically similar to the general / general-purpose terms in the lattice and / or that increases the scores 155 of hypotheses in the lattice 200 that include personal named entities associated with the personal context information 126.

[0040] An important aspect of the speech recognition model 130, which combines a non-personal but personalizable first recognition model 130 with a personal second recognition model 140, is its ability to protect potentially sensitive user-specific data (e.g., personal context information 126) incorporated into the second recognition model 140. For example, the personal context information 126 may be stored in the memory hardware 116 of the user device 110, and the second recognition model 140 may execute on the data processing hardware 114 of the user device 110 to prevent user-specific data from being disclosed to the computing system 120 (e.g., a remote server / distributed system). In some scenarios, the second recognition model 140 executes on the user device 110 to calculate / generate situational probabilities that are reduced to a simple weighted grammar and transmitted to the computing device 120 upon request. In these scenarios, the values ​​of the probability distributions 147 over possible personal named entities 148 may be transmitted to the computing device 120 over a network without the need to share any of the user-specific data / logic used to calculate these values.

[0041] The first recognition model 130 may execute on the user device 110 or the computing system 120. Because the first recognition model 130 is impersonal and purposely constructed without any user-specific data, the first recognition model 130 can execute on the computing system 120 without revealing any personal information / data of the user.

[0042] Personal context information 126 may include personal named entities 148 specific to the user 10 who spoke the utterance 101, such as, but not limited to, the user's list of contacts, names of items in the user's media library (songs, artists, albums, movies), names of nearby places, names of installed applications, and user-defined entity names. Personal context information 126 may further indicate user history associated with these personal named entities, such as call frequency and usage patterns. For example, a user's media library may contain music files by both artists Phish and Hootie and the Blowfish, but the user may play Phish songs almost daily while only playing Hootie and the Blowfish songs once five years ago. As a result, the personal context information 126 causes the second recognition model M2 140 to appropriately assign a higher likelihood / probability to the candidate hypothesis "Play Harpua by Phish" than to "Play Hootie and the Blowfish." The user history indicated by the personal context information 126 may further include previous queries, learned correlations, corrections of previously misrecognized transcriptions, etc.

[0043] Personal context information 126 may be stored in memory hardware 114 of user device 110 and / or in memory hardware 124 of computing system 120. The information in these different context data sets changes from time to time, such as when a user adds or removes contacts, when a user's media library changes, when a user adds or removes installed applications, when a user changes location, etc.

[0044] Continuing with reference to FIG. 1 , the scoring model 150 receives the first decoder result 132 produced by the first recognition model 130 and the second decoder result 142 produced by the second recognition model 140. In some examples, the scoring model 150 receives the first decoder result 132 associated with the 1-best non-context hypothesis 130 from the lattice 200. The scoring model 150 is configured to score the context candidate hypothesis 145 along with the non-context hypothesis 135 so that both personal named entities may be considered in conjunction with the non-personal alternatives for selecting the best candidate hypothesis 135, 145, as the resulting transcription 175 of the utterance 101. When scoring the context candidate hypothesis 145 and the non-context candidate hypothesis 135, the scoring model 150 calculates a likelihood score 155, P, modeled by the first recognition model M1 130 for the non-context candidate hypothesis 135. M1 (Call Best Buy), and an individual likelihood score of 157, P M1 (call $CONTACT), as well as the potential personal named entity 148 modeled by the second recognition model M2 140, P M2 Use the probability distribution 147 for (BethBuyer|$CONTACT).

[0045] As will become apparent, scoring the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 output by M1 130 prevents excessive triggering of the context candidate hypotheses 145 in non-spoken scenarios and prevents insufficient recall of the non-context candidate hypotheses 135 in spoken scenarios. Continuing with the example where "Call Best Buy" includes a non-contextual candidate hypothesis 135 with the general named entity "Best Buy" and "Call Beth Buyer" includes a contextual candidate hypothesis 145 with a distinct named entity injected from the user's list of personal contacts, the personalization of the first model M1 and the incorporation of personal contextual information 126 leveraged by the second model M2 allows the scoring model 300 to consider whether Beth Buyer is a personal contact of the user, whether the user calls Best Buy or Beth Buyer more frequently, whether either Beth Buyer or Best Buy are common general entities that are likely to be spoken, acoustic fit to the audio data 112, and whether another better (e.g., higher likelihood score 155) is available in the lattice 200.

[0046] Scoring model 150 may score the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 as follows: P M1 (h with entity g )=P M1 (h with $entityclass)×P M2 (entity p |$entityclass) (1) In the formula, P M1 (h with entity g ) denotes the likelihood score 155 assigned to the non-contextual candidate hypothesis 135 (e.g., call Best Buy) modeled by the first recognition model M1 130, and P M1(h with $entityclass) denotes the individual likelihood score 157 for the non-contextual candidate hypothesis 135 with the general named entity class (e.g., Best Buy) replaced with the corresponding token (e.g., $CONTACT) 158 mapped to the individual named entity class (e.g., personal contact), and P M2 (entity p |$entityclass) indicates the probability distribution 147 over which personal named entities 148 are likely to belong to the particular personal named entity class indicated by token 158. p The term P may represent probability values ​​summing to a value of 1.0 for each personal named entity associated with a given personal named entity class. M2 (entity p |$entityclass) may simply contain the value of the highest-ranked personal named entity in the probability distribution 147.

[0047] Continuing with the above example, while personal context information 126 may indicate that "Beth Buyer" is indeed one of the user's personal contacts, the context information 126 may further reveal that "Beth Buyer" is a lower-tier personal contact that the user rarely or never calls, making the context (personal) candidate hypothesis 145 relatively weak compared to the strong non-context (non-personal) candidate hypothesis 135 "Call Best Buy." Notably, the existing candidate hypothesis and the new candidate hypothesis are phonetically close to each other. This may indicate that Beth Buyer is a contact in the user's list of personal contacts, but is a lower-tier personal contact that the user rarely calls. As a result, applying Equation 1 to score the context candidate hypothesis 145 output by M2 140 against the non-context candidate hypothesis 135 yields P M1 (Call Best Buy)>P M1 (Call $CONTACTS)*P M2(Beth Byer|$CONTACTS), and the non-contextual candidate hypothesis 135 is not replaced and is properly selected as transcription 175 for utterance 101.

[0048] In another example, where user 10 speaks utterance 101 of "Call Leonid" (e.g., the user's personal contacts), first model M1 130 may generate non-context candidate hypothesis 135 "Call Lee Oh Need," and second model M2 140 may generate context candidate hypothesis 145 "Call Leonid." Although the two hypotheses 135, 145 are phonetically close to each other, the likelihood score 155 assigned to non-context hypothesis 135 may indicate that non-context candidate hypothesis 135 is unlikely to be plausible and unlikely to have been spoken, while the probability distribution 147 for likely personal named entities 148 belonging to the personal contacts class may include a high probability for "Leonid" based on personal context information 126 indicating that "Leonid" is one of the user's personal contacts that the user calls regularly. As a result, applying Equation 1 to score the context candidate hypotheses 145 output by M2 140 against the non-context candidate hypotheses 135 gives P M1 (Call Lee O'Need) <P M1 (Call $CONTACTS)*P M2 (Leonid|$CONTACTS), and the context candidate hypothesis 145 is appropriately selected as the transcription 175 of the utterance 101.

[0049] The scoring model 150 may evaluate and score a context candidate hypothesis 145 against a single non-context candidate hypothesis 135 and its corresponding likelihood score 155. A single non-context candidate hypothesis 135 may be selected for better scoring accuracy, for example, based on high acoustic proximity to the new candidate hypothesis 145 and / or affinity with similar hypotheses for use as a reference point. A single non-context candidate hypothesis 135 may be selected as the 1-best hypothesis 135 in the lattice. Alternatively, the scoring model 150 may evaluate and score a context candidate hypothesis 145 against multiple non-context candidate hypotheses 135 (or even all hypotheses in the original lattice 200) and their corresponding likelihood scores 155, providing more accurate scoring at the cost of increased computational load (in addition to increased latency and memory requirements).

[0050] 1 , during stage (F), reranker 160 receives scoring output 152 from scoring model 150, which scores context candidate hypotheses 145 against non-context candidate hypotheses 135 and their corresponding likelihood scores 155. Reranker 160 is configured to output reranked results 165, which include a ranking of non-context candidate hypotheses 135 and context candidate hypotheses 145 based on the scoring output. In the illustrated example, reranked results 165 include non-context candidate hypothesis 135, with a score of 0.9, as the most likely correct transcription 175.

[0051] In stage (G), computing system 120 is configured to generate a transcription 175 of utterance 101 spoken by user 10 by selecting the highest-ranked candidate from re-ranked results 165. In the illustrated example, computing system 120 selects non-context candidate hypothesis 135, "Call Best Buy," because it has the highest score of 0.9. Computing system 120 may transmit transcription 175 to client device 110 via network 118 to provide transcription 175 to user 10. In some implementations, client device 110 performs all of stages (A) through (G) on-device without connecting to computing system 120 via network 118.

[0052] While the above example illustrates scoring contextual candidate hypotheses against non-contextual candidate hypotheses of an audio-based speech input, aspects of the present disclosure may be similarly applied to scoring contextual candidate hypotheses against non-contextual candidate hypotheses of a non-speech input. For example, the non-speech input 113 may include a user input indication indicating the selection of one or more characters on a keyboard 117 (e.g., a virtual keyboard or a physical keyboard) in communication with the data processing hardware 114 of the client device 110. In this example, the speech recognizer module 130 may be replaced by a keyboard detection module configured to generate a lattice 200 of non-contextual candidate hypotheses 135 for the non-speech input 113. Models M1 and M1 may function similarly to the above example by generating non-contextual candidate hypotheses 135 and contextual candidate hypotheses 145 for the non-speech input 113. By scoring the contextual candidate hypotheses against the non-contextual candidate hypotheses, the scoring model 150 may then provide the ability to rank non-speech recognition hypotheses that are personal / user-specific against non-personal hypotheses that include general terms for which the keyboard detection module may not have been trained.

[0053] 3, in some implementations, an exemplary training process 300 trains the first recognition model M1 130 as an ensemble without any user-specific data to personalize the first recognition model M1 130. Specifically, the training process 300 may train the first model M1 130 with an ASR training corpus 320 that includes a corpus of ground truth transcriptions 325. In some examples, the personal named entity extractor 302 may replace an identified personal named entity transcription 325 with a corresponding token associated with a personal named entity class to which the identified personal named entity belongs. For example, the extractor 302 may identify “Nick” in the transcription “Call Nick” as a personal named entity belonging to the personal contacts class and replace “Nick” with the corresponding token “$CONTACT.” The extractor 302 may identify personal named entities through techniques such as syntactic analysis and / or explicit labels provided in the transcription 325 .

[0054] Optionally, the training process 300 can additionally train the first model M1 130 with a generic transcription that includes only generic named entities. Here, the extractor 302 can determine when a given transcription 325 omits personal named entities and therefore does not replace any terms in the transcription 325 with tokens. In this scenario, the training process 300 also trains the first model M1 130 with the generic transcription 325 to train the model M1 130 to evaluate the relative merits between personal hypotheses that include personal named entities and non-personal hypotheses that include only generic terms. For example, the extractor 302 can determine that "Best Buy" is a generic entity in the transcription "Call Best Buy" and allow the transcription 325 to train the model M1 130 without replacing any terms with tokens.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0056] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0057] 4 is a flowchart of an example arrangement of operations of a method 400 for scoring context candidate hypotheses 145 against non-context candidate hypotheses. Data processing hardware 122 (FIG. 1) may execute instructions stored in memory hardware 124 (FIG. 1) to perform the example arrangement of operations of method 400. At operation 402, method 400 includes receiving audio data 112 corresponding to an utterance 101 spoken by user 10. At operation 404, method 400 includes processing the audio data 112 using a first recognition model 130 to generate a non-context candidate hypothesis 135 as output from the first recognition model 130, where the non-context candidate hypothesis 135 corresponds to a candidate transcription of the utterance and has a corresponding likelihood score 155 assigned to the non-context candidate hypothesis 135 by the first recognition model 130.

[0058] At operation 406, the method 400 includes using a second recognition model 140 configured to receive the personal context information 126 to generate context candidate hypotheses 145 including the personal named entities, where the context candidate hypotheses correspond to alternative candidate transcriptions of the utterance.

[0059] At operation 408, the method 400 includes scoring the context candidate hypotheses 145 relative to the non-context candidate hypotheses 135 based on the personal context information 126 and the corresponding likelihood scores 155 assigned to the non-context candidate hypotheses 135. Based on scoring the context candidate hypotheses relative to the non-context candidate hypotheses, the method 400 further includes, at operation 410, generating a transcription 175 of the utterance 101 spoken by the user 10 by selecting one of the context candidate hypotheses 145 or the non-context candidate hypotheses 135.

[0060] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems and methods described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the scope of the invention as described and / or claimed in this document.

[0061] Computing device 500 includes a processor 510, memory 520, a storage device 530, a high-speed interface / controller 540 connecting to memory 520 and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 connecting to storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may reside on a common motherboard or exist otherwise as needed. Processor 510 processes instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, and can display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Multiple computing devices 500 may also be connected, each performing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0062] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0063] Storage device 530 can provide mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 can be a device array including a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0064] High-speed controller 540 manages bandwidth-intensive operations for computing device 500, and low-speed controller 560 manages low-bandwidth-intensive operations. This allocation of roles is merely exemplary. In some implementations, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which can accept various expansion cards (not shown). In some implementations, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can connect, for example, via a network adapter, to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as a switch or router.

[0065] The computing device 500 may be implemented in many different forms, as shown in the figure. For example, the computing device may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0066] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general-purpose, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0067] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0068] The processes and logic flows described herein can be implemented by one or more programmable processors (also called data processing hardware) executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0069] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0070] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (400) that, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations, the operations comprising: receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate, as output from the first recognition model, non-context candidate hypotheses (135), the non-context candidate hypotheses (135) corresponding to candidate transcriptions (175) of the utterance (101) and having corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135) by the first recognition model (130); generating, using a second recognition model configured to receive personal context information (126), a context candidate hypothesis (145) including a personal named entity (148), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypotheses (145) relative to the non-context candidate hypotheses (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); generating a transcription (175) of the utterance (101) spoken by the user by selecting one of the context candidate hypotheses (145) or the non-context candidate hypotheses (135) based on the scoring of the context candidate hypotheses (145) relative to the non-context candidate hypotheses (135); receiving a training corpus of ground truth transcriptions; For each corresponding ground truth transcription (175) of the training corpus of ground truth transcriptions, determining whether a personal named entity (148) is identified in the corresponding ground truth transcription (175); replacing the personal named entity (148) with an entity class token (158) when the personal named entity (148) is identified in the corresponding ground truth transcription (175); training the first recognition model with the ground truth transcriptions that include personal named entities (148) replaced by tokens of the entity classes.

2. 2. The computer-implemented method of claim 1, wherein the first recognition model processes the audio data to generate the non-context candidate hypotheses without incorporating any of the personal context information associated with the user.

3. 3. The computer-implemented method of claim 1, wherein the first recognition model comprises an end-to-end speech recognition model configured to generate the corresponding likelihood scores of the non-contextual candidate hypotheses.

4. the second recognition model (140) includes an end-to-end speech recognition model (140) configured to generate the context candidate hypotheses (145); 3. The computer-implemented method of claim 1, wherein generating the context candidate hypotheses (145) including the personal named entities (148) comprises processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypotheses (145) based on the personal context information (126).

5. 5. The computer-implemented method of claim 4, wherein the end-to-end speech recognition model is configured to generate the context candidate hypotheses after the non-context candidate hypotheses are generated as output from the first recognition model.

6. 5. The computer-implemented method of claim 4, wherein the end-to-end speech recognition model is configured to generate the context candidate hypotheses in parallel with the processing of the audio data to generate the non-context candidate hypotheses as output from a first recognition model.

7. The computer-implemented method (400) of claim 1 or 2, wherein the second recognition model (140) comprises a language model.

8. The personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) comprising: a contact name in the user's personal contact list; a name in a media library associated with said user; Name of the installed application, Names of nearby places, or The computer-implemented method (400) of claim 1 or 2, including at least one of a user-defined entity name.

9. 3. The computer-implemented method of claim 1, wherein the personal context information associated with the user indicates a user history associated with the personal named entity specific to the user.

10. data processing hardware (510); memory hardware (520) in communication with the data processing hardware (510) and storing instructions, the instructions, when executed by the data processing hardware (510), causing the data processing hardware (510) to: receiving audio data (112) corresponding to an utterance (101) spoken by a user; processing the audio data (112) using a first recognition model (130) to generate, as output from the first recognition model (130), non-context candidate hypotheses (135), the non-context candidate hypotheses (135) corresponding to candidate transcriptions (175) of the utterance (101) and having corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135) by the first recognition model (130); generating, using a second recognition model (140) configured to receive personal context information (126), a context candidate hypothesis (145) including a personal named entity (148), the context candidate hypothesis (145) corresponding to another candidate transcription (175) of the utterance (101); scoring the context candidate hypotheses (145) relative to the non-context candidate hypotheses (135) based on the personal context information (126) and the corresponding likelihood scores (155) assigned to the non-context candidate hypotheses (135); generating a transcription (175) of the utterance (101) spoken by the user by selecting one of the context candidate hypotheses (145) or the non-context candidate hypotheses (135) based on the scoring of the context candidate hypotheses (145) relative to the non-context candidate hypotheses (135); receiving a training corpus of ground truth transcriptions; For each corresponding ground truth transcription (175) of the training corpus of ground truth transcriptions, determining whether a personal named entity (148) is identified in the corresponding ground truth transcription (175); replacing the personal named entity (148) with an entity class token (158) when the personal named entity (148) is identified in the corresponding ground truth transcription (175); and training the first recognition model with the ground truth transcription that includes personal named entities (148) replaced by tokens of the entity class.

11. 11. The system of claim 10, wherein the first recognition model processes the audio data to generate the non-context candidate hypotheses without incorporating any of the personal context information associated with the user.

12. 12. The system (100) of claim 10 or 11, wherein the first recognition model (130) comprises an end-to-end speech recognition model (130) configured to generate the corresponding likelihood scores (155) of the non-contextual candidate hypotheses (135).

13. the second recognition model (140) includes an end-to-end speech recognition model (140) configured to generate the context candidate hypotheses (145); 12. The system (100) of claim 10 or 11, wherein generating the context candidate hypotheses (145) including the personal named entities (148) comprises processing the audio data (112) using the end-to-end speech recognition model (140) to generate the context candidate hypotheses (145) based on the personal context information (126).

14. 14. The system of claim 13, wherein the end-to-end speech recognition model is configured to generate the context candidate hypotheses after the non-context candidate hypotheses are generated as output from the first recognition model.

15. 14. The system of claim 13, wherein the end-to-end speech recognition model is configured to generate the context candidate hypotheses in parallel with the processing of the audio data to generate the non-context candidate hypotheses as output from a first recognition model.

16. The system (100) of claim 10 or 11, wherein the second recognition model comprises a language model.

17. The personal context information (126) associated with the user includes a personal named entity (148) specific to the user, the personal named entity (148) comprising: a contact name in the user's personal contact list; a name in a media library associated with said user; Name of the installed application, Names of nearby places, or 12. The system (100) of claim 10 or 11, comprising at least one of a user-defined entity name.

18. 12. The system (100) of claim 10 or 11, wherein the personal context information (126) associated with the user indicates a user history associated with the personal named entity (148) specific to the user.

Images