Language model biasing system

Abstract

Methods, systems, and computer-readable media are disclosed. A computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations, including: receiving context data of a user device associated with a user; identifying an initial set of n-grams from the context data; receiving audio data corresponding to utterances detected by the user device; processing the audio data using a speech recognizer to generate speech recognition candidates for utterances spoken by the user, each speech recognition candidate being associated with a corresponding speech recognition score; adjusting one or more of the speech recognition scores associated with the speech recognition candidates using the initial set of n-grams; and, after adjusting one or more speech recognition scores, determining the transcription of the utterance by selecting the speech recognition candidate associated with the highest corresponding speech recognition score.

Description

[0001] This application is a divisional application of the invention patent application filed on October 19, 2017, with application number 201780086257.5 and invention title "Language Model Bias System". Technical Field

[0002] This specification relates to speech recognition, and a particular implementation relates to a biased language model for performing speech recognition. Background Technology

[0003] Voice input is commonly used to perform various operations on electronic devices. For example, some mobile phones or other computing devices allow users to provide voice commands or other voice input to perform actions, instruct users to make selections, launch applications, or initiate communications using the mobile phone. The user's voice input can be detected by the computing device's microphone and processed by an automatic speech recognizer.

[0004] Automatic speech recognizers (ASPR) can perform speech recognition using both acoustic and language models. An ASPR uses an acoustic model to determine the relationship between an audio signal and phonetic units in the user's speech. A language model indicates possible sequences of sounds or words in the user's speech. Using both models, an ASPR can generate candidate transcriptions of the speech input. Summary of the Invention

[0005] This specification describes a system that can improve speech recognition accuracy by biasing a language model used by an automatic speech recognizer. In some implementations, the system biases a general language model based on contextual information. Contextual information may include, for example, one or more n-grams (n-grams) received by the system (e.g., words in an n-gram, such as a single word or a phrase comprising multiple words). The system may submit these n-grams to one or more language extension services and, in response, receive additional n-grams identified by the language extension services. The general language model may be biased based on the n-grams included in the contextual information and the n-grams received from the language extension services. The automatic speech recognizer can then use the biased general language model to perform speech recognition. Biasing the language model in this way allows the model to be dynamically tuned to favor words relevant to the user's current task or situation, thereby allowing the system to transcribe speech more accurately, especially for words uncommon in typical speech.

[0006] For example, developers of applications configured to respond to voice input can provide the system with a list of words and / or phrases associated with the application. This list may include words a user might say when the application is in a specific state (e.g., on a particular menu screen). The system can receive the word list as an initial set of n-grams and submit this initial set of n-grams to various language extension services. These language extension services can generate additional n-grams based on the initial set, which the system can then collect as an extended set of n-grams. The system can then bias a general language model based on the initial set of n-grams and / or the extended set of n-grams. An automatic speech recognizer can use this biased general language model to generate a transcription of the user-provided voice input.

[0007] After a speech recognizer performs speech recognition on a speech input using a biased general language model, the preferences for speech recognition candidates generated by the speech recognizer can be adjusted based on the identified n-grams. For example, the probabilities, recognition scores, or probability quality assigned to speech recognition candidates that include the identified n-grams can be adjusted to increase the preference for those speech recognition candidates. After making these adjustments, the speech recognizer can determine the transcription of a speech input that includes one or more of the speech recognition candidates.

[0008] In some implementations, the general language model can be biased for the user's current context before receiving voice input from the user. In other implementations, the system can update the bias of the general language model as the user provides voice input; for example, the system can periodically update the bias of the general language model as an automatic speech recognizer processes audio data corresponding to the voice input. For example, when words are detected in the voice input, the system can submit these words to a language extension service to obtain an extended set of n-grams for biasing the general language model.

[0009] Compared to other methods that bias a general language model, biasing a general language model allows for more accurate and efficient speech recognition. For example, in terms of processing requirements and storage space, biasing a single general language model may be more efficient than interpolating a general language model using one or more context-specific language models. Rather than selecting a context-specific language model or interpolating a context-specific general language model, biasing a general language model also allows an automatic speech recognizer to use a language model that is biased using more specific contexts and more frequently as the context evolves, thereby improving speech recognition accuracy. Additionally, the same information used to bias the general language model can be used to identify preferred speech recognition candidates generated using the biased general language model. This second form of biasing speech recognition candidates can further improve speech recognition accuracy.

[0010] The inventive aspects of the subject matter described in this specification can be embodied in a method comprising the following actions: receiving audio data corresponding to a user's utterance and contextual data of the user's utterance; identifying an initial set of one or more n-grams from the contextual data; generating an extended set of one or more n-grams based at least on the initial set of n-grams, the extended set of n-grams including one or more n-grams different from those in the initial set of n-grams; adjusting a language model based at least on the extended set of n-grams; determining one or more speech recognition candidates for at least a portion of the user's utterance using the adjusted language model, wherein each speech recognition candidate includes one or more words; after determining the one or more speech recognition candidates, adjusting the scores of specific speech recognition candidates determined to be included in the extended set of n-grams; after adjusting the scores of the specific speech recognition candidates, determining a transcription of the user's utterance including at least one of the one or more speech recognition candidates; and providing the transcription of the user's utterance for output.

[0011] These and other embodiments may each optionally include one or more of the following features. In various examples, adjusting the language model based at least on the extended set of n-grams includes: adjusting the recognition scores of one or more n-grams in the extended set of n-grams in the language model; adjusting the language model based at least on the extended set of n-grams includes: adjusting the probability quality assigned to one or more n-grams in the extended set of n-grams in the language model.

[0012] Each embodiment may also include one or more of the following features. In various examples, adjusting the language model based at least on an extended set of n-grams includes: adding one or more n-grams from the extended set of n-grams to the language model, the language model including one or more placeholder transitions; and adding one or more n-grams from the extended set of n-grams to the language model includes: assigning a specific n-gram from the extended set of n-grams to a specific placeholder transition; assigning a specific n-gram from the extended set of n-grams to a specific placeholder transition includes: adjusting the recognition score of the specific placeholder transition of the specific n-gram in the extended set of n-grams to which it is assigned.

[0013] Each embodiment may also include one or more of the following features. In some examples, generating an extended set of n-grams based at least on an initial set of n-grams includes: sending one or more n-grams from the initial set of n-grams to each of one or more language extension services, and receiving one or more extended sets of n-grams from the one or more language extension services in response to sending one or more n-grams from the initial set of n-grams to each of the one or more language extension services; one or more n-grams from the extended set of n-grams are included in a hash map, and adjusting the score of a particular speech recognition candidate determined to be included in the extended set of n-grams includes determining that the particular speech recognition candidate is included in the hash map; the audio data corresponding to the user's utterance corresponds to a frame of spoken user input comprising multiple frames; the language model is a general language model or a general language model affected during the processing of the preceding frame of spoken user input.

[0014] Each embodiment may also include one or more of the following features. In some cases, the context data includes one or more context words; the one or more context words are each configured to perform an action in response to spoken user input submitted by an application developer; the context data does not include one or more words included in a previous transcription of the user's utterance; the language model is tuned at least based on an expanded set of n-grams before receiving audio data corresponding to the user's utterance; these features may include tuning the language model at least based on an initial set of n-grams.

[0015] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0016] Figure 1 An example system is described for biasing a language model and performing speech recognition using the biased language model.

[0017] Figure 2 An example system for determining the n-grams used in the language model used to perform speech recognition is described.

[0018] Figure 3 An example of implementing language model bias is described.

[0019] Figure 4 This is a flowchart of an example process for biasing a speech recognizer and performing speech recognition using the biased speech recognizer.

[0020] The same reference numerals in different figures denote the same elements. Detailed Implementation

[0021] Figure 1 An example system is described for biasing a language model used by an automatic speech recognizer and performing speech recognition using the biased language model. In short, Figure 1 A user 105 is shown, which has a client device 104 that provides voice input to a user device 104. The user device 104 interacts with a transcription system 100 to obtain a transcription of the voice input.

[0022] In some embodiments, transcription system 100 may be included in user equipment 104, for example, as a software application stored on and operating locally on user equipment 104. In other embodiments, transcription system 100 may be accessed by user equipment 104 via one or more wired or wireless data paths. For example, transcription system 100 may be hosted on a server accessible by user equipment 104 via one or more networks, such as a wired or wireless internet connection.

[0023] User device 104 can be any computing device capable of receiving voice input and performing operations in response to voice input. For example, user device 104 can be a computing device configured with a microphone and one or more processors, wherein the one or more processors are used to process operations triggered based on voice input received by the microphone. For example, user device 104 can be a cellular phone, smartphone, personal digital assistant (PDA), smartwatch, tablet computer, laptop computer, desktop computer, or other computing device. User device 104 may include interfaces for allowing user 105 to interact with user device 104, such as a touchscreen, keyboard, mouse, touchpad, display, or other interfaces that allow user 105 to provide input to user device 104 and / or allow user device 104 to present information to user 105. User device 104 may further include one or more wired or wireless network connections, such as one or more cellular antennas, Wi-Fi, Bluetooth, Ethernet, Z-Wave, ZigBee, or other mechanisms for transmitting data to, for example, transcription system 100 via wired or wireless connections.

[0024] The transcription system 100 includes a speech recognizer 110 for performing speech recognition and a language model biasing engine 120 for biasing the language model used by the speech recognizer 110 when performing speech recognition. The transcription system 100 further includes a general language model 130 and an n-gram cache 140. The language model biasing engine 120 accesses the general language model 130 and biases the general language model 130 based on information received from the n-gram cache 140.

[0025] The speech recognizer 110 can access one or more acoustic models, which are used to process audio data corresponding to the speech input provided by the user 105. To perform speech recognition, the speech recognizer 110 also receives or accesses a language model from the language model bias engine 120. The speech recognizer 110 uses the language model received or accessible at the language model bias engine 120 and one or more acoustic models to process the audio data corresponding to the speech input to generate a transcription of the speech input.

[0026] For example, user 105 can provide the voice input "Italian Ristorante" to user device 104, and user device 104 can send audio data corresponding to the voice input to transcription system 100. Speech recognizer 110 can use the language model and one or more acoustic models received or accessed by language model bias engine 120 to process the audio data corresponding to the voice input "Italian Ristorante" to generate a transcription of the voice input.

[0027] The speech recognizer 110 can provide a transcription of the speech input to the user equipment 104. For example, when the transcription system 100 is located remotely from the user equipment 104, the speech recognizer 110 can provide the transcription of the speech input to the user equipment 104 via one or more wired or wireless data paths (e.g., a network). The user equipment 104 can receive the transcription and can perform operations on the transcription or perform operations related to the transcription. For example, the user equipment 104 or an application operating on the user equipment 104 can trigger one or more actions based on the transcription in response to the speech input, or can provide the transcription for output at a display of the user equipment 104.

[0028] For example, speech recognizer 110 can generate a transcription of "Italian restaurant" by processing audio data received from user device 104 using a language model accessed or received from language model bias engine 120 and one or more acoustic models. Speech recognizer 110 can provide the transcription of "Italian restaurant" to user device 104, wherein user device 104 or an application operating on user device 104 can perform an operation in response to receiving the transcription, providing the transcription of "Italian restaurant" for output, or may otherwise respond to speech input.

[0029] The language model accessed or received from the language model bias engine 120 can be a general language model 130 that the language model bias engine 120 has modified based on an initial set of n-grams and extended n-grams received from a language extension service set. These n-grams can each represent an n-gram that includes one or more words. The general language model 130 can be a probabilistic language model, which is a probability distribution over a sequence of n-grams.

[0030] The general language model 130 is a language model that is not specific to any particular context, such as a language model that has been trained on a large range of speech data and is therefore not specific to any particular speech context. Although it is primarily described as an n-gram language model in this specification, other forms of the general language model 130 may be employed in other embodiments. For example, the general language model 130 may be a unary language model, a continuous space language model, a positional language model, a log-linear language model, or another type of general language model.

[0031] n-gram cache 140 stores a collection of n-grams that can be used to bias the general language model 130. The n-grams included in n-gram cache 140 are those that have been identified as context-dependent. (See reference...) Figure 2 In more detail, the n-grams included in the n-gram cache 140 may include an initial set of n-grams identified from context information received by the transcription system 100, and may include an extended set of n-grams identified from the context information received by the transcription system 100. The identified n-grams may be added to the n-gram cache 140, which is accessible by the language model bias engine 120.

[0032] In some implementations, an n-gram may be included in the n-gram cache 140 in response to other information or events. For example, the context may be identified based on determining the location of user device 104, the current time, day or date, movement of user device 104, background noise included in the audio signal received from user device 104, or other information displayed to user 105 at user device 104. One or more n-grams may be identified as context-dependent and may be included in the n-gram cache 140. When the context of user device 104 changes, the contents of the n-gram cache 140 may change; for example, previously included n-grams in the n-gram cache 140 may be removed and new n-grams may be added.

[0033] In some implementations, the context includes words presented on the user's device screen. The initial set of n-grams can be words and phrases extracted from the text visible on the user's screen. As another example, the initial set of n-grams can be provided by an application running on the user's device. For example, the application can have different conversation states, such as different views corresponding to different tasks or different steps within a task. Each conversation state, which may correspond to different interfaces or views of the application, can have a predetermined set of n-grams not shown on the screen but specified by the application developer. For example, one set of n-grams can be specified as being associated with a view for typing the body of an email message, a second set of n-grams can be specified as being for a view for selecting a contact within the same application, and so on. Similarly, information indicating the context can be provided as identifiers, such as application identifiers and / or conversation state identifiers, which the speech recognition system can use to look up the appropriate n-grams from a database or from another server system.

[0034] In some implementations, before the transcription system 100 receives audio data corresponding to the speech input by the user 105, the language model bias engine 120 may bias the general language model 130 based on the n-grams included in the n-gram cache 140. For example, before receiving audio data corresponding to the speech input "Italian restaurant" from the user device 104, the language model bias engine 120 may access the n-gram cache 140 and identify the n-grams included in the n-gram cache 140. The language model bias engine 120 may access the general language model 130 and bias the general language model 130 based on the n-grams included in the n-gram cache 140.

[0035] For example, the language model bias engine 120 can bias the general language model 130 to indicate a preference for n-grams in the general language model 130 that are also included in the n-gram cache 140. For example, the general language model 130 can be a probabilistic language model, where each edge between nodes of the general language model 130 represents a word or phrase. The language model bias engine 120 can bias the general language model 130 by increasing the probability of associating with n-grams in the general language model 130 that are also included in the n-gram cache 140. Increasing the probability of associating with n-grams identified in the n-gram cache 140 can cause the speech recognizer 110 to be more likely to identify those n-grams. In some implementations, the language model bias engine 120 can also bias the general language model 130 by decreasing the probability of associating with n-grams not identified in the n-gram cache 140, making it less likely that the speech recognizer 110 using the biased version of the general language model 130 will identify those n-grams in the audio data corresponding to the speech input.

[0036] In other implementations, the language model bias engine 120 may use other techniques to bias the general language model 130. The language model bias engine 120 may bias the general language model 130 by adjusting the recognition scores assigned to n-grams in the general language model 130. For example, the language model bias engine 120 may increase the recognition scores assigned to n-grams included in the n-gram cache 140 and optionally decrease the recognition scores assigned to other n-grams not included in the n-gram cache 140.

[0037] In other implementations, the language model bias engine 120 can bias the general language model 130 by adjusting the probability quality assigned to the n-grams included in the general language model 130. For example, the language model bias engine 120 can increase the probability quality assigned to the n-grams included in the general language model 130 and the n-gram cache 140, and can optionally decrease the probability quality of one or more n-grams in the general language model 130 but not included in the n-gram cache 140.

[0038] In some implementations, the language model biasing system 120 receives n-grams directly from one or more sources other than the n-gram cache 140 for biasing in the general language model 130. For example, the language model biasing system 120 may receive or determine an initial set of n-grams from contextual information received by the transcription system 100. The language model biasing system 120 may also receive or determine an extended set of n-grams generated by one or more language extension services based on the initial set of n-grams. Before the transcription system 100 receives audio data corresponding to the speech input “Italian restaurant”, the language model biasing engine 120 may bias the general language model 130 based on the initial set of n-grams and the extended set of n-grams.

[0039] Such bias may include adjusting the probabilities, recognition scores, or probability quality of n-grams in the initial or extended set of n-grams in the general language model 130. In some embodiments, the general language model may include one or more placeholder transitions, i.e., transitions that are not assigned a particular n-gram or word and / or are not assigned a probability, recognition score, or probability quality sufficient for the speech recognizer 110 to select a placeholder transition when generating speech recognition candidates. In these embodiments, biasing the general language model 130 may include assigning one or more n-grams from the initial or extended set to placeholder transitions, adjusting the probabilities, recognition scores, or probability quality assigned to these placeholder transitions, or both.

[0040] In some implementations, the transcription system 100 biases the general language model 130 during or after a user provides voice input. In some implementations, this bias can be performed for each frame or segment of audio data corresponding to the voice input. For example, user 105 may provide the voice input "Italian restaurant" to user device 104. User device 104 may receive the voice input using its microphone and may transmit the audio data corresponding to the voice input to the transcription system 100.

[0041] The speech recognizer 110 of the transcription system 100 receives audio data corresponding to speech input and performs speech recognition on the audio data or audio data frames to generate one or more speech recognition candidates for the audio data or audio data frames. Each speech recognition candidate may be an n-gram detected by the speech recognizer 110 from the audio data or audio data frames. To generate speech recognition candidates, the speech recognizer 110 may use a biased version of the general language model 130, for example, a version of the general language model 130 generated by biasing the general language model before the user 105 provides speech input, or a version of the general language model 130 generated by biasing the general language model during the processing of a first portion (e.g., a frame) of the audio data corresponding to the speech input.

[0042] After generating speech recognition candidates, the speech recognizer 110 can provide the speech recognition candidates to the language model bias engine 120, or the language model bias engine 120 can access the speech recognition candidates at the speech recognizer 110. The language model bias engine 120 can then access the n-gram cache 140 to identify the speech recognition candidates included in the n-gram cache 140, i.e., the n-grams detected from the audio data. For example, the language model bias engine 120 can compare the speech recognition candidates with the n-grams included in the n-gram cache 140 and identify the n-grams in the n-gram cache 140 that match the speech recognition candidates. The language model bias engine 120 or the speech recognizer 110 can adjust the probability, recognition score, or probability quality of the speech recognition candidates that include the n-grams included in the n-gram cache 140. Additionally or alternatively, the language model bias engine 120 may further bias the general language model 130 to select the n-gram in the general language model 130, which is identified based on a comparison of speech recognition candidates and n-grams in the n-gram cache 140.

[0043] For example, speech recognizer 110 can perform speech recognition on a portion of audio data corresponding to the speech input “Italian restaurant” and determine that two speech recognition candidates for that portion of audio data are “Italian” and “pullin”. Language model bias engine 120 can determine that the n-gram “Italian” is included in the n-gram cache 140, but the n-gram “pullin” is not. In response to determining that the n-gram “Italian” matches an n-gram included in the n-gram cache 140, language model bias engine 120 can adjust the preferences (e.g., probability, recognition score, or probability quality) of the speech recognition candidate “Italian” to increase the likelihood that the speech recognition candidate “Italian” is selected as the transcription of that portion of the speech input.

[0044] Additionally or alternatively, the language model bias engine 120 may access the general language model 130 and may adjust the probability, probability quality, or recognition score associated with the n-gram “Italy” in the general language model 130 to bias the general language model 130 toward that n-gram. A similar biasing process may be performed on the general language model 130 when processing portions of audio data corresponding to the n-gram “restaurant”.

[0045] After further biasing the general language model 130 (e.g., after receiving audio data corresponding to the speech input of user 105), the speech recognizer 110 may use a further biased version of the general language model 130 when generating a transcription of the speech input or a portion thereof. For example, the speech recognizer 110 may use a further biased version of the general language model 130 to process a portion or frame of audio data corresponding to the “Italian” part of the speech input received by user device 104 to generate one or more new speech recognition candidates that reflect the further bias of the general language model 130. For example, the candidate transcription of “Italian” may be associated with a higher probability, recognition score, or probabilistic quality compared to a case where the general language model 130 does not have a further bias toward the n-gram “Italian”.

[0046] The speech recognizer 110 may select a specific speech recognition candidate as a transcription of the speech input or as part of the speech input to return to the user device 104, or be selected as part of the transcription of the speech input to be returned to the user device 104. The speech recognition candidate selected as a transcription of the speech input or as part of the transcription may be a speech recognition candidate, or, in the case of further biasing the general language model 130, a new speech recognition candidate identified by the speech recognizer 110 as having the highest correct probability, the highest recognition score, or the highest probabilistic quality.

[0047] When selecting a specific speech recognition candidate, the language model bias engine 120 or the speech recognizer 110 can compare the speech recognition candidate with the n-grams in the n-gram cache 140 and adjust the probability, recognition score, or probability quality of the speech recognition candidate that is also an n-gram in the n-gram cache 140. For example, after biasing the general language model 130 and performing speech recognition on the speech input using the biased general language model 140, the n-grams “pull in” and “Italian” identified as speech recognition candidates can be compared with the n-grams stored in the n-gram cache 140. Based on identifying the n-gram “Italian” in the n-gram cache 140 and not identifying the n-gram “pull in” in the n-gram cache 140, the language model bias engine 120 or the speech recognizer 110 can adjust the score, probability, or probability quality to indicate a preference for the speech recognition candidate “Italian”. For example, the probability, probability quality, or recognition score assigned to the speech recognition candidate "Italy" can be increased, or the probability, probability quality, or recognition score assigned to the speech recognition candidate "Pull In" can be decreased.

[0048] Speech recognizer 110 selects a speech recognition candidate that has been assigned a probability, recognition score, or probability quality indicating the highest confidence level of correctness and provides the speech recognition candidate to user device 104. For example, based on comparing the probability, probability quality, or recognition score of speech recognition candidate “Italy” with the probability, probability quality, or recognition score of speech recognition candidate “pull in”, speech recognizer 110 can identify “Italy” as a transcription of that speech input portion. Transcription system 100 can then perform a similar process on the audio data portion corresponding to the word “restaurant”. Speech recognizer 110 can return the transcription to user device 104 as it processes each portion of the speech input, for example, such that when transcription system 100 processes the speech input portion corresponding to the word “restaurant”, the transcription “Italy” is provided to user device 104, or speech recognizer 110 can return the complete transcription of the speech input to user device 104 at once, for example, such that the transcription “Italian restaurant” is transmitted to user device 104 after processing the entire speech input.

[0049] In some implementations, whenever the language model biasing engine 120 biases the original general language model 130, it performs a bias on the original, unbiased general language model 130. In this case, the language model biasing engine 120 may bias the original general language model 130 and provide the biased version of the general language model 130 to the speech recognizer 110. When the language model biasing engine 120 is invoked to perform biasing again, the language model biasing engine 120 may discard the previous bias of the general language model 130 and may bias the original language model 130 again, such that each bias of the general language model 130 does not take into account the previous bias of the general language model 130.

[0050] In other implementations, the language model bias engine 120 biases the general language model 130 by updating a previously biased version of the general language model 130. For example, after the speech recognizer 110 performs speech recognition using a biased version of the general language model 130, the language model bias engine 120 can perform subsequent biases of the general language model 130 by further biasing the biased version of the general language model 130.

[0051] In some implementations, the n-gram cache 140 includes a hash map of n-grams. In these implementations, comparing speech recognition candidates with the n-grams included in the n-gram cache 140, or identifying n-grams in the general language model 130, may include calculating the hash values ​​of the n-grams and comparing these hash values. For example, to perform an initial bias of the general language model 130 before receiving speech input, the language model bias engine 120 may compare the hash values ​​of the n-grams in the n-gram cache 140 with the hash values ​​of the n-grams in the general language model 130. Once speech input is received, the hash values ​​of speech recognition candidates may be calculated and compared with the hash values ​​of the n-grams in the n-gram cache 140 to determine whether any speech recognition candidate is an n-gram included in the n-gram cache 140.

[0052] Figure 2 An example system for biasing a language model used in performing speech recognition is described. In general, Figure 2 The system can be used to expand the initial set of n-grams and store the expanded set of n-grams in the n-gram cache. The n-grams stored in the n-gram cache can be accessed by the language model biasing engine when biasing the general language model. For example, Figure 1 Language model bias engine 120 can bias Figure 1The general language model 130 accesses the n-gram cache. Once the speech recognizer has determined speech recognition candidates for the speech input using the biased general language model, it can also access the n-gram cache. For example, the speech recognizer 110 or the language model bias engine 120 can access the n-gram cache, compare the n-grams of the speech recognition candidates with the n-grams in the n-gram cache, and adjust the probabilities, probability quality, or recognition scores assigned to those speech recognition candidates based on the matching of the n-grams of the speech recognition candidates with the n-grams in the n-gram cache.

[0053] in short, Figure 2 The system shown includes user equipment 204 associated with user 205. User equipment 204 may be similar to Figure 1 The user equipment 104 communicates with the transcription system 200 via one or more wired or wireless data paths (e.g., via one or more networks).

[0054] Transcription system 200 includes an n-gram extension engine 210 that communicates with an n-gram cache 240, which in some embodiments may be the same as the n-gram cache 140 of transcription system 100. The n-gram extension engine 210 may also communicate with one or more language extension services [1]-[N]. Although in Figure 2 The language extension services [1]-[N] are shown as being included in the transcription system 200, but in some embodiments, the language extension services [1]-[N] may be external to the transcription system 200. For example, the language extension services [1]-[N] may be accessed by the transcription system 200 via one or more wired or wireless networks, or may be accessed by the transcription system 200 in other ways.

[0055] A user 205 (such as a developer of a software application configured to respond to voice input) on user device 204 can submit contextual information to transcription system 200. For example, user 205 may want to bias a general language model toward certain words relevant to the application they are developing. These words may include those appearing in the application menu or other words the user of the application might say, such as words corresponding to objects displayed to the user during use of the application. User 205 can generate a set of these words, and this set of words can be submitted to transcription system 200 as contextual information from user device 205.

[0056] The transcription system 200 can receive context information and, based on the context information, determine the initial set and the extended set of n-grams. The transcription system 200 can store the initial set and / or the extended set of n-grams in the n-gram cache 240. For this purpose, in some embodiments, the n-gram extension engine 210 receives context information from the user device 205, for example, via one or more networks. The n-gram extension engine 210 can extract the initial set of n-grams from the context information and use the initial set of n-grams to obtain the extended set of n-grams.

[0057] In some implementations, the n-gram expansion engine 210 can extract an initial set of n-grams from contextual information received from the user device 204. For example, if the contextual information received from the user device 204 is a list of words, the n-gram expansion engine 210 can extract these words as n-grams to include in the initial set of n-grams. The n-gram expansion engine 210 can also determine additional n-grams to include in the initial set of n-grams by simplifying compound words from the contextual information to their constituent parts, or by combining words from the contextual information to form larger n-grams. For example, the word "eastbound" can be simplified to the single n-grams "east" and "bound," while the words "drive" and "in" can be combined to form the n-gram "drive-in." The n-gram expansion engine 210 can include all these n-grams in the initial set of n-grams.

[0058] The n-gram extension engine 210 can submit one or more n-grams from an initial set of n-grams to each of one or more language extension services [1]-[N]. Each language extension service can receive the n-grams submitted to it, process the received n-grams, and return one or more additional n-grams to the n-gram extension engine 210. The n-gram extension engine 210 can compile the additional n-grams to generate an extension set of n-grams. Each of the language extension services [1]-[N] can be specific to certain types of language extensions and can return an n-gram identified using the specific type of language extension adopted by that language extension service [1]-[N].

[0059] In some implementations, the language extension services [1]-[N] may include a synonym language extension service. The synonym language extension service may identify synonyms of n-grams received from the n-gram extension engine 210. For example, the synonym language extension service may identify the n-gram “pizzeria” as a synonym of the n-gram “restaurant” submitted by user 205. Similarly, the antonym language extension service, the homonym language extension service, or the acronym language extension service may each generate one or more n-grams, respectively, in their full forms as antonyms, homonyms, acronyms, or acronyms, for the n-grams received from the n-gram extension engine 210.

[0060] In some implementations, the language extension service [1]-[N] may include a similar search result language extension service. The similar search result language extension service may identify an n-gram that, when provided as a search query (e.g., an internet search), produces search results similar to other n-grams. For example, n-gram extension engine 210 may provide the n-gram “Monticello” to the similar search result language extension service, and the service may determine that the n-gram “Monte Cello” returns search results similar to “Monticello” based on the user’s frequent misspelling of the term in search queries. The similar search result language extension service may therefore provide the n-gram “Monte Cello” to n-gram extension engine 210 to be included in the set of n-gram extensions.

[0061] In some implementations, the language extension service [1]-[N] includes an ontology language extension service that identifies an n-gram that is in the same category, ontology, or otherwise identified as related to or associated with other n-grams. For example, in response to receiving the n-gram “Italy”, the ontology language extension service may access one or more ontologies that include the n-gram “Italy” to identify additional n-grams. For example, the ontology language extension service may identify the n-grams “Tuscany,” “Rome,” and “Venice” from a first ontology, for example, that is related to Italian style or description, and the n-grams “France,” “Castiel,” and “Romania” from a second ontology, for example, that is related to romantic language or generally to language.

[0062] Similarly, language extension services [1]-[N] may include topic-related language extension services. Topic-related language extension services may identify n-grams that are identified as related to the same topic as a particular n-gram. For example, n-gram extension engine 210 may submit the n-gram “Italian holiday” to the topic-related language extension service, and in response, the topic-related language extension service may identify n-grams that are identified as related to the same topic (e.g., Italian travel), such as “Venice,” “pizza,” “flight,” “vineyard,” etc. Geolocation language extension services may similarly identify n-grams that are identified as related to a particular n-gram based on their geographic proximity. For example, geolocation language extension services may identify n-grams “Roman Colosseum,” “Tiber River,” or “Vatican City” in response to receiving the n-gram “Rome.”

[0063] In some implementations, the language extension service [1]-[N] may include a word clustering language extension service that identifies n-grams that are close to the n-gram received from the n-gram extension engine 210 in the N-dimensional word space. For example, the word clustering language extension service may access n-gram mappings within the n-dimensional word space. For a specific n-gram, the word clustering language extension service may locate the specific n-gram in the n-dimensional word space and identify other n-grams within a threshold distance of the specific n-gram in the word space.

[0064] The language extension service [1]-[N] may further include a morphing language extension service that creates a morphing form of the n-gram specified by the n-gram extension engine 210 and returns the morphing form of the n-gram to the n-gram extension engine 210. The n-gram extension engine 210 may include morphing forms of the n-gram in the initial set of n-grams in the extended set of n-grams.

[0065] n-gram extension engine 210 receives each of the defined n-grams from the language extension services [1]-[N] and compiles the n-grams to generate an extended set of n-grams. For example, n-gram extension engine 210 may receive one or more n-grams from each of the language extension services [1]-[N], and may compile the received n-grams into an extended set of n-grams by removing n-grams received from multiple services, n-grams that are too long or too short to be useful for the language model bias, etc. In some implementations, n-gram extension engine 210 includes an initial set of n-grams in the extended set of n-grams to generate a single set of n-grams that can be provided to n-gram cache 240. In some implementations, instead of compiling the received n-grams into a single set, n-gram extension engine 210 may send or forward n-grams received from the language extension services [1]-[N] to n-gram cache 140 individually or as n-grams are received from a service.

[0066] The n-gram extension engine 210 sends the extended set of n-grams or information about the n-grams identifying the extended set to the n-gram cache 240. The n-gram cache 240 can store n-grams or information identifying n-grams. Subsequently, this can be done, for example, through a language model bias engine (e.g., Figure 1 The language model bias engine 120 accesses the n-gram cache 240 to identify specific speech recognition candidates, including n-grams that appear in the n-gram cache 240, or to bias the general language model used in speech recognition.

[0067] Figure 3 Examples of implementing language model biases are described. For example, Figure 3 The general language model 300 may be biased to generate a biased version 350 representation of the general language model 300. The biased version 350 of the general language model 300 may be obtained, for example, after receiving contextual information and before the user (e.g., user 105 of user device 104) provides the voice input "Italian restaurant". Bias may also occur during voice input processing. For example, the general language model 300 may be biased for the first time before receiving the voice input "Italian restaurant", and may be biased a second time after processing the word "Italian" in the voice input.

[0068] In short, the general language model 300 includes nodes corresponding to transitions between n-grams or words, and edges corresponding to n-grams or words. Some edges may be placeholder transitions that were not initially assigned a specific n-gram or word. These placeholder transitions may have n-grams or words assigned to them during the biasing process, for example, to adjust the general language model 300 to include non-lexical words or other uncommon n-grams. These placeholder transitions may not be given a probability, probability quality, or recognition score before they are assigned an n-gram or word, or may be given a probability, score, or probability quality of zero before being assigned. Therefore, unassigned placeholder transitions are not selected as candidate transcriptions. Placeholder edges can be added or modified for each word decision to dynamically add potential new words that may be identified as candidates.

[0069] Before or after receiving speech input, and at each word decision (e.g., word boundary), an n-gram or word can be assigned to one or more placeholder transitions. Generally, a word from the cached or expanded n-gram set is only assigned to a placeholder edge if it has not yet appeared in the model as a potential choice for the next word. The probabilities, recognition scores, or probability quality of those placeholder edges and other edges in the general language model 300 can be adjusted to generate a biased version 350. The probability scores assigned to the filled placeholders can be based on the frequency of data usage, or can be predetermined or dynamically set base scores. The biased version 350 can be used to generate speech recognition candidates for the speech input or a portion of the speech input (e.g., the next word in the speech input). Because the placeholder transitions are dynamically filled using words from an expanded set of n-grams, additional word options that were not previously present in the model are added. Furthermore, because the words used for padding are derived from the current context of the user's device (e.g., from a cached n-gram or extended set), the words used for padding are likely to be highly relevant to the current task and topic of the software running on the user's device.

[0070] In some implementations, for each word decision, each initial word from the cache can be filled into separate placeholder edges. For example, if the cache comprises 10 n-grams, then the first word of each of the 10 n-grams can be filled as a potential next word. After making that word decision, the same 10 words can be filled as potential words for the next decision, and so on. Of course, as the cache changes, the set of words filled into the placeholders can also change accordingly.

[0071] In some implementations, any word in the cache can be padded into a placeholder as a potential next word. For example, there might be 10 n-grams in the cache, but since some n-grams may contain 2, 3, or more words, there could be a total of 75 different words in the cache. As a result, the system can padded each of these 75 words into a placeholder as a potential next word, regardless of the word's position in the n-gram. These 75 words (or any word in the cache) can be used for multiple word decisions, with placeholders padded for each decision.

[0072] In some implementations, words in the n-gram cache are selectively filled into placeholders based on the similarity of the identifiers of one or more candidate or user utterance words. For example, instead of adding every word from the cache or every initial word, the system can identify a subset of language patterns that match the identifiers in the utterance. For example, one of the speech recognition candidates in a part of the utterance might be the word "Italy". The system can determine whether the cache includes this word, and if so, what the next word in the n-gram in the cache is. For example, the n-gram cache might be determined to include "Italian music", "eating Italian food", and "Italian style", while other n-grams in the cache do not include the word "Italy". From this analysis, the system can extract "music", "food", and "style" as potential next words to fill into placeholders, since these words represent words immediately following "Italy" in the cached n-grams. Thus, a specific set of words identified as relevant can be filled into placeholders, excluding other words from the cache.

[0073] In some implementations, multiple words of the entire n-gram or multiple word phrases within the n-gram can be padded to placeholder edges. For example, if the n-gram “Italian music” appears in the cache, then the entire n-gram can be padded to placeholders as a potential next phrase.

[0074] As described above, padding words or n-grams into placeholder transition edges can be accompanied by assigning a non-zero probability score to each padded edge. The system uses the language model score, along with the acoustic model score indicating the degree to which a word matches the sound of the evaluated utterance, to select a set of candidate words for word determination. Furthermore, preferences for certain speech recognition candidates can be adjusted based on the fact that some candidates include certain n-grams or words (e.g., n-grams included in an n-gram cache). For example, if one of the candidates matches a word in the n-gram cache, the candidate's probability score can be increased in response to identifying a match with a word or n-gram in the n-gram cache, regardless of whether the candidate was determined from placeholder edges or standard features of the language model.

[0075] This assignment of words or n-grams to placeholder edges can be temporary, for example, only for the determination of a specific word in a utterance. Once a candidate word has been identified or a word determination has been made, the placeholder assignment is removed, and the placeholder's score is set to zero. In this way, each word determination can be made using its own set of placeholder assignments, and these assignments can remain unaffected by other word determinations in the utterance, other utterances, or the recognition of other user utterances.

[0076] like Figure 3 As shown, the general language model 300 includes nodes 302a-302e connected by edges 304a-304g. Edges 304a-304g correspond to n-grams in the general language model 300, where each edge 304a-304g is assigned a specific probability, as shown in diagram 300a. Although shown as probabilities in diagram 300a, in other embodiments, edges 304a-304g may be associated with recognition scores or probability weights, as discussed above. The language model 300 may include... Figure 3 Additional nodes and edges not shown.

[0077] In the general language model, edge 304a corresponds to the n-gram "pull" with an associated probability of 0.3. Edge 304b corresponds to the n-gram "in" with an associated probability of 0.3. Edge 304c corresponds to the n-gram "Italian" with an associated probability of 0.3. Edge 304d is associated with the n-gram "rest," and edge 304e is associated with the n-gram "stop," each with a probability of 0.3. Edge 304f is associated with the n-gram "restaurant" with an assigned probability of 0.3.

[0078] Edge 304g is a placeholder edge that can be replaced by a new n-gram, for example, an n-gram to be added to the general language model 300. Figure 3 In the general language model 300, placeholder edges 304g are not assigned probabilities. However, in other applications, placeholder edges 304g can be assigned zero probability or other low probabilities until a specific n-gram is assigned to placeholder edges 304g and assigned a different probability. Placeholder 304g enables biasing of the general language model 300 to include adding new n-grams to the general language model 300.

[0079] The biased general language model 300 creates a biased version 350 of the general language model 300. For example, based on receiving and / or determining the initial set and the extended set of n-grams, Figure 1 The language model biasing engine 120 can bias the general language model 300 to obtain a biased version 350 of the general language model 300. For example, the language model biasing engine 120 can identify the n-grams “Italy” and “restaurant” in the n-gram cache 140, and in response, can update the general language model 300 to obtain the biased version 350. Figure 1 The speech recognizer 110 can use a biased version 350 of the general language model 300 to perform speech recognition on speech input.

[0080] In the biased version 350 of the general language model 300, the probabilities of edges 304a-304g corresponding to the n-grams of the general language model 300 have been updated. Additionally, placeholder edge 304g has been assigned the n-gram “restaurant” and assigned a non-zero probability. As shown in Table 350a, for example, based on the language model bias engine 120 identifying the n-gram “Italy” in the n-gram cache 140 before receiving audio data corresponding to the speech input, the probabilities of edges 304a and 304b have been reduced to 0.2, and the probability of edge 304c for the n-gram “Italy” has been increased to 0.5.

[0081] Edges 304d and 304e have also had their probabilities reduced to 0.2, for example, because they are neither marked "stationary" nor "stopped" in the n-gram cache 140. Edge 304f of the n-gram "restaurant" is increased because it is marked in the n-gram cache 140.

[0082] Additionally, based on this determination, the language model bias engine 140 can add the n-gram “restaurant” to the general language model 300 by assigning an n-gram to the placeholder edge 304g and assigning a probability of 0.4 to edge 304g. After biasing the general language model 300 to obtain a biased version 350, the transcription system 100 can use the biased version 350 to generate a transcription of the speech input. For example, the transcription system 100 can generate the transcription “Italian restaurant” based on processing the speech input using the biased version 350 of the language model 300. To this end, the speech recognizer 110 can use the biased version 350 of the general language model 300 to generate candidate transcriptions for the speech input. The speech recognizer 110 or the language model bias engine 120 can compare the n-grams of the speech recognition candidates with the n-grams in the n-gram cache 140. The speech recognizer 110 or language model bias engine 120 can adjust the probabilities, recognition scores, or probability quality of certain speech recognition candidates based on the fact that they are included in the n-gram cache 140, and can select specific speech recognition candidates for output based on the adjusted probabilities, recognition scores, or probability quality.

[0083] Figure 4 This is a flowchart of an example process 400 for biasing a language model and performing speech recognition using a biased version of the language model. In some implementations, Figure 1 transcription system 100 and / or Figure 2 The transcription system 200 can perform process 400 or a part of process 400.

[0084] The system receives audio data corresponding to the user's speech and contextual data of the user's speech (402). For example, the system can receive contextual information specifying one or more words from user device 204 of user 205 (e.g., a developer). The contextual information can be received by, for example, an n-gram extension engine 210. The system can also receive audio data corresponding to the user's voice input (e.g., voice input provided by user 105 to user device 104). The audio data corresponding to the voice input can be received by, for example, a speech recognizer 110 of transcription system 100.

[0085] The system identifies an initial set of one or more n-grams from the context data (404). For example, the n-gram extension engine 210 may receive context data including one or more words submitted by the user 205, and the n-gram extension engine 210 may extract an initial set of n-grams from these words.

[0086] The system generates at least one or more extended sets of n-grams based on an initial set of n-grams, the extended sets of n-grams including one or more n-grams different from those in the initial set of n-grams (406). For example, n-gram extension engine 210 may submit one or more n-grams from the initial set of n-grams to each of one or more language extension services [1]-[N]. Language extension services [1]-[N] may receive n-grams and may generate one or more n-grams based on the initial set. As described above, language extension services [1]-[N] may include, for example, a synonym language extension service, a similar search result language extension service, an ontology language extension service, a word clustering language extension service, a variant language extension service, a similar topic language extension service, etc. Each language extension service [1]-[N] may generate one or more n-grams based on the n-grams from the initial set. Language extension services [1]-[N] may provide the generated n-grams to n-gram extension engine 210, which may compile the n-grams to generate an extended set of n-grams. The extended set of n-grams will include at least one n-gram that is different from the n-grams in the initial set of n-grams. The n-gram extension engine 210 can store the extended set of n-grams in the n-gram cache 240.

[0087] The system adjusts the language model (408) based at least on an expanded set of n-grams. For example, the language model bias engine 120 can locate a specific n-gram corresponding to an n-gram in the expanded set of n-grams in the general language model 130 or a biased version of the general language model 130. The language model bias engine 120 can adjust the probability, recognition score, or probability quality assigned to a specific n-gram in the general language model 130 or a biased version of the general language model 130. In some embodiments, the system can also adjust the language model based on an initial set of n-grams, for example, by adjusting the probability, recognition score, or probability quality of one or more n-grams in the initial set of n-grams in the general language model 130 or a biased version of the general language model 130.

[0088] The system uses an adjusted language model to determine one or more speech recognition candidates for at least a portion of a user's utterance, wherein each speech recognition candidate includes one or more words (410). For example, a speech recognizer 110 receiving audio data corresponding to a user 105's speech input can perform speech recognition on the audio data or a segment of audio data (e.g., a single frame) to generate one or more speech recognition candidates for the audio data or the segment of audio data. The speech recognizer 110 can generate speech recognition candidates using a biased version of a general language model 130. For example, the speech recognizer 110 can generate speech recognition candidates using a biased version of a general language model 130 created before receiving the audio data corresponding to the audio data. The biased version of the general language model 130 can be the original general language model 130, wherein the original general language model 130 has been biased based on an extended set of n-grams and optionally an initial set of n-grams. Alternatively, the biased version of the general language model 130 may be a version of the general language model 130 that was biased during an earlier speech recognition process (e.g., an earlier segment of audio data corresponding to the user 105's speech input or another speech input) and further biased based on the n-grams identified in the extended set of n-grams.

[0089] After identifying one or more speech recognition candidates, the system adjusts the score (412) of a particular speech recognition candidate determined to be included in an extended set of n-grams. For example, the extended set of n-grams may be stored in an n-gram cache 140, accessible by a language model bias engine 120. The language model bias engine 120 may receive speech recognition candidates from the speech recognizer 110 and may determine that a particular speech recognition candidate is included in the extended set of n-grams by comparing it with the n-grams stored in the n-gram cache 140. For example, the n-gram cache 140 may include hash maps of the n-grams in the extended set of n-grams. The language model bias engine 120 may determine that a particular speech recognition candidate is included in the extended set of n-grams by comparing the hash value corresponding to the speech recognition candidate with the hash value in the hash map. The speech recognizer 110 or the language model bias engine 120 may adjust the score of a particular speech recognition candidate by adjusting the probability, probability quality, or recognition score assigned to that candidate.

[0090] After adjusting the scores of a particular speech recognition candidate, the system determines a transcription of a user utterance that includes at least one of the one or more speech recognition candidates (414). For example, after the speech recognizer 110 or the language model bias engine 120 has adjusted the scores of a particular speech recognition candidate, the speech recognizer 110 or the language model bias engine 120 may select a particular speech recognition from one or more speech recognition candidates, for example, by selecting the speech recognition candidate with the highest probability, probability quality, or recognition score. Because the scores of a particular speech recognition candidate have been adjusted based on its inclusion in an expanded set of n-grams, the system can more likely determine a transcription of a user utterance that includes a particular speech recognition candidate.

[0091] The system provides a transcription of the user's speech for output (418). For example, the speech recognizer 110 can provide a transcription of the speech input for output to the user device 104. The user device 104 can receive the transcription and can perform operations on the transcription, or provide the transcription for display on the user device 104's interface. For example, the user device 104 or an application operating on the user device 104 can process the transcription into voice commands to perform one or more operations, or the user device 104 or an application operating on it can provide the transcription for output on the user device 104's interface.

[0092] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. For example, the various forms of processes shown above can be used, in which steps are reordered, added, or removed. Therefore, other embodiments are also within the scope of the following claims.

[0093] For systems and / or methods discussed herein that may collect personal information about users or instances where personal information may be used, users may be given the opportunity to control whether programs or features collect personal information (e.g., information about a user's social networks, social actions or activities, occupation, preferences, or current location), or the opportunity to control whether and / or how systems and / or methods perform actions more relevant to the user. Furthermore, some data may be anonymized in one or more ways before its storage or use, thereby removing personally identifiable information. For example, a user's identity may be anonymized so that personally identifiable information cannot be determined for the user, or a user's geographic location may be generalized to a city, zip code, or state level where location information is obtained, making it impossible to determine the user's specific location. Therefore, users can control how information about them is collected and used.

[0094] While the foregoing embodiments have been described primarily with reference to the development or processing of voice input for use with applications installed on user devices, the described features can also be used with respect to machines, other devices, robots, or other systems. For example, the described systems and methods can be used to improve user interaction with large machines having associated computing systems; can be used to develop and implement voice actions for interaction with robots or systems having robotic components; can be used to develop and implement voice actions for interaction with electrical appliances, entertainment systems, or other devices; or can be used to develop and implement voice actions for interaction with vehicles or other transportation systems.

[0095] The embodiments and all functional operations described in this specification can be implemented in digital electronic circuit systems, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of these. Embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or control of the operation of a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of materials that implement machine-readable propagated signals, or a combination of one or more of these. The term "data processing apparatus" encompasses all means, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. The propagated signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device.

[0096] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0097] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system, and the device can be implemented as a dedicated logic circuit system, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0098] As an example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory or random access memory, or both.

[0099] The basic components of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices (e.g., hard disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a tablet computer, mobile phone, personal digital assistant (PDA), mobile audio player, GPS receiver, to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), hard disks (e.g., internal hard disks or removable hard disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit system.

[0100] To provide interaction with the user, embodiments can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) for displaying information to the user, as well as a keyboard and indicating devices through which the user can provide input to the computer (e.g., a mouse or trackball). Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0101] The embodiments can be implemented in a computing system that includes, for example, a backend component acting as a data server, or includes middleware components (e.g., an application server), or includes frontend components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with the implementation), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), such as the Internet.

[0102] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through communication networks. The client-server relationship arises from computer programs running on individual computers and having client-server relationships with each other.

[0103] While this specification contains numerous details, these should not be construed as limiting the scope of this disclosure or the claims, but rather as descriptions of features specific to embodiments. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even thereby initially claimed, in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations thereof.

[0104] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential manner, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0105] In each instance where an HTML file is mentioned, other file types or formats can be substituted. For example, an HTML file can be replaced with XML, JSON, plain text, or other file types. Furthermore, when referring to a table or hash table, other data structures (such as spreadsheets, relational databases, or structured files) can be used.

[0106] Therefore, specific embodiments have been described. Other embodiments are within the scope of the following claims. For example, the actions listed in the claims may be performed in a different order and still achieve the desired result.

Claims

1. A computer-implemented method for speech recognition, wherein when executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations, comprising: Receive an initial set of n-grams, the initial set of n-grams being specified by the application developer for the dialogue state; Generating one or more extended sets of n-grams based at least on an initial set of n-grams includes sending one or more n-grams from the initial set of n-grams to one or more language extension services, and receiving the extended set of n-grams from the language extension services in response, wherein the extended set of n-grams includes one or more n-grams that are different from the n-grams in the initial set of n-grams. The language model should be adjusted based on at least an expanded set of n-grams; Receive audio data corresponding to the speech detected by the user equipment; One or more speech recognition candidates are determined for at least a portion of the user’s speech using an adjusted language model, wherein each speech recognition candidate includes one or more words; After identifying one or more speech recognition candidates, the scores of the specific speech recognition candidates that are determined to be included in the extended set of n-grams are adjusted. After adjusting the scores of specific speech recognition candidates, the audio data is processed using a speech recognizer to determine the transcription of the utterance; and Provides transcription of discourse for output.

2. The computer-implemented method according to claim 1, wherein, The operation also includes: Receive context data from the user device. The initial set of receiving n-grams includes the initial set of n-grams identified from context data.

3. The computer-implemented method according to claim 2, wherein: The context data includes an application identifier or a dialogue state identifier; and Identifying the initial set of the n-grams involves retrieving data indicating one or more words or phrases that correspond to the application identifier or dialogue state identifier.

4. The computer-implemented method according to claim 2, wherein, Identifying the initial set of n-grams from the context data includes: Identify a first set of one or more n-grams from the context data; and At least one extended set of n-grams is generated based on a first set of n-grams, wherein the extended set of n-grams includes one or more n-grams that are different from the n-grams in the first set of n-grams.

5. The computer-implemented method according to claim 2, wherein: The speech is detected by the user device that provides the user interface; and The context data includes data indicating the theme corresponding to the interface.

6. The computer-implemented method according to claim 2, wherein: The speech is detected by the user device that provides the user interface; and The context data includes data indicating the tasks to be performed using the interface.

7. The computer-implemented method of claim 2, wherein the context data indication includes one or more phrases in the graphical user interface of the user device.

8. The computer-implemented method according to claim 1, wherein, The initial set of n-grams includes one or more words or phrases displayed on the screen of the user's device.

9. The computer-implemented method according to claim 1, wherein, The initial set of n-grams is provided by an application running on the user device.

10. The computer-implemented method according to claim 1, wherein: The data processing hardware resides on the user equipment; or The data processing hardware resides on a server system that communicates with the user equipment via a communication network.

11. A system for speech recognition, the system comprising: Data processing hardware; and Memory hardware that communicates with the data processing hardware and stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including: Receive an initial set of n-grams, the initial set of n-grams being specified by the application developer for the dialogue state; Generating one or more extended sets of n-grams based at least on an initial set of n-grams includes sending one or more n-grams from the initial set of n-grams to one or more language extension services, and receiving the extended set of n-grams from the language extension services in response, wherein the extended set of n-grams includes one or more n-grams that are different from the n-grams in the initial set of n-grams. The language model should be adjusted based on at least an expanded set of n-grams; Receive audio data corresponding to the speech detected by the user equipment; One or more speech recognition candidates are determined for at least a portion of the user’s speech using an adjusted language model, wherein each speech recognition candidate includes one or more words; After identifying one or more speech recognition candidates, the scores of the specific speech recognition candidates that are determined to be included in the extended set of n-grams are adjusted. After adjusting the scores of specific speech recognition candidates, the audio data is processed using a speech recognizer to determine the transcription of the utterance; and Provides transcription of discourse for output.

12. The system according to claim 11, wherein, The operation also includes: Receive context data from the user device. The initial set of receiving n-grams includes the initial set of n-grams identified from context data.

13. The system according to claim 12, wherein: The context data includes an application identifier or a dialogue state identifier; and Identifying the initial set of the n-grams involves retrieving data indicating one or more words or phrases that correspond to the application identifier or dialogue state identifier.

14. The system according to claim 12, wherein, Identifying the initial set of n-grams from the context data includes: Identify a first set of one or more n-grams from the context data; and At least based on the first set of n-grams, generate one or more extended sets of n-grams, the extended sets of n-grams including one or more n-grams that are different from the n-grams in the first set of n-grams.

15. The system according to claim 12, wherein: The speech is detected by the user device that provides the user interface; and The context data includes data indicating the theme corresponding to the interface.

16. The system according to claim 12, wherein: The speech is detected by the user device that provides the user interface; and The context data includes data indicating the tasks to be performed using the interface.

17. The system of claim 12, wherein the context data indication includes one or more phrases in the graphical user interface of the user equipment.

18. The system according to claim 11, wherein, The initial set of n-grams includes one or more words or phrases displayed on the screen of the user's device.

19. The system according to claim 11, wherein, The initial set of n-grams is provided by an application running on the user device.

20. The system according to claim 11, wherein: The data processing hardware resides on the user equipment; or The data processing hardware resides on a server system that communicates with the user equipment via a communication network.

21. A non-transitory computer-readable medium storing one or more instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform operations, including: Receive an initial set of n-grams, the initial set of n-grams being specified by the application developer for the dialogue state; Generating one or more extended sets of n-grams based at least on an initial set of n-grams includes sending one or more n-grams from the initial set of n-grams to one or more language extension services, and receiving the extended set of n-grams from the language extension services in response, wherein the extended set of n-grams includes one or more n-grams that are different from the n-grams in the initial set of n-grams. The language model should be adjusted based on at least an expanded set of n-grams; Receive audio data corresponding to the speech detected by the user equipment; One or more speech recognition candidates are determined for at least a portion of the user’s speech using an adjusted language model, wherein each speech recognition candidate includes one or more words; After identifying one or more speech recognition candidates, the scores of the specific speech recognition candidates that are determined to be included in the extended set of n-grams are adjusted. After adjusting the scores of specific speech recognition candidates, the audio data is processed using a speech recognizer to determine the transcription of the utterance; and Provides transcription of discourse for output.

Images