Contextual tagging and offsetting of syntax within word grid

Abstract

The invention relates to contextual tagging and offset of syntax within a word grid. In one aspect, a method includes actions of receiving audio data of an utterance. The actions further include generating a word grid including a plurality of candidate transcriptions of the utterance and including transcriptional confidence scores. The actions further include determining a context of the computing device. The actions further include identifying syntax corresponding to a plurality of candidate transcriptions based on a context of the computing device. The actions further include determining a syntax confidence score for each of the plurality of candidate transcriptions by candidate transcriptions, the syntax confidence score reflecting a likelihood that the respective syntax is a match for the respective candidate transcriptions; the actions further include selecting candidate transcriptions from among the candidate transcriptions. The actions further include providing the selected candidate transcription as a transcription of the utterance for output.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980079465.1, filed on November 27, 2019.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This patent application claims the benefit of U.S. application 62 / 591,300, filed December 3, 2018, which is incorporated herein by reference. Technical Field

[0005] This specification relates generally to speech recognition. Background Art

[0006] There is a growing desire to be able to use voice input to perform interactions with computers. This requires developments in input processing, specifically how to program computers to process and analyze natural language data. Such processing may involve speech recognition, a field of computational linguistics that enables computers to recognize and translate spoken language into text. Summary of the invention

[0007] To enable a user to provide input to a computing device via speech, a speech input processing system may use context to identify grammars to be applied to candidate transcriptions generated by an automatic speech recognizer. Each grammar may indicate a different intent of a speaker or an action to be performed by the system for the same candidate transcription. The system may select grammars and candidate transcriptions based on parsing the grammars of the candidate transcriptions, the likelihood that the grammars match the user's intent, and the likelihood that the candidate transcriptions match what the user said. The system may then perform an action corresponding to the grammar by using the details included in the selected candidate transcription.

[0008] More specifically, a speech processing system receives an utterance from a user and generates a word grid. A word grid is a data structure that reflects possible words of an utterance and confidence scores for each word. The system identifies candidate transcriptions and transcription confidence scores for each candidate transcription from the word grid. The system identifies a current context, which may be based on user characteristics, system location, system characteristics, applications running on the system (e.g., currently active applications or applications running in the foreground), or any similar context data. Based on the context, the system generates a grammar confidence score for parsing a grammar for each candidate transcription. In cases where more than one grammar may apply to the same candidate transcription, the system may adjust some grammar confidence scores. The system selects a grammar and a candidate transcription based on a combination of the adjusted grammar confidence scores and transcription confidence scores.

[0009] According to innovative aspects of the subject matter described in the present application, a method for processing speech input includes the following actions: receiving audio data of an utterance by a computing device; generating a word grid by the computing device using an acoustic model and a language model, the word grid including multiple candidate transcriptions of the utterance and including transcription confidence scores, each transcription confidence score reflecting the likelihood that the corresponding candidate transcription is a match for the utterance; determining a context of the computing device by the computing device; based on the context of the computing device, identifying, by the computing device, grammar corresponding to the multiple candidate transcriptions; based on the current context, determining, by the computing device, a grammar confidence score for each of the multiple candidate transcriptions, the grammar confidence score reflecting the likelihood that the corresponding grammar is a match for the corresponding candidate transcription; selecting, by the computing device, a candidate transcription from among the candidate transcriptions based on the transcription confidence scores and the grammar confidence scores; and providing, by the computing device, the selected candidate transcription as a transcription of the utterance for output.

[0010] These and other embodiments can each optionally include one or more of the following features. The action includes: determining that two or more grammars in the grammar correspond to one of the candidate transcriptions; and adjusting the grammar confidence score for the two or more grammars based on determining that the two or more grammars in the grammar correspond to one of the candidate transcriptions. The computing device selects a candidate transcription from the candidate transcriptions based on the transcription confidence score and the adjusted grammar confidence score. The action of adjusting the grammar confidence score for two or more grammars includes: increasing each grammar confidence score by a factor for each grammar in the two or more grammars. The action includes determining the product of the corresponding transcription confidence score and the corresponding grammar confidence score for each candidate transcription in the candidate transcription. The computing device selects the candidate transcription from the candidate transcription based on the product of the transcription confidence score and the corresponding grammar confidence score. The action includes determining, by the computing device, that the context of the computing device is based on the location of the computing device, the application running in the foreground of the computing device, and the time of the day. The language model is configured to identify the probability of a sequence of words included in the word grid. The acoustic model is configured to identify phonemes that match a portion of the audio data. The actions include performing, by the computing device, actions based on the selected candidate transcription and a grammar that matches the selected candidate transcription.

[0011] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the method operations.

[0012] Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A speech recognition system can use both the received speech input and the determined context to select a grammar for further processing the received speech input to cause a computing device to perform an action. In this way, a speech recognition system can reduce delays in a human-computer interface by applying a limited number of grammars to candidate transcriptions. A speech recognition system can use a vocabulary that includes all or nearly all words in a language, enabling the speech recognition system to output a transcription when the system receives unexpected input. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0014] Figure 1 An example system is shown that selects a grammar to apply to an utterance based on context.

[0015] Figure 2 Components of an example system for selecting a grammar to apply to an utterance based on context are shown.

[0016] Figure 3 is a flow chart of an example process based on context selection grammar.

[0017] Figure 4 are examples of computing devices and mobile computing devices.

[0018] Like reference numbers and designations in the various drawings represent like elements. DETAILED DESCRIPTION

[0019] Figure 1 An example system 100 is shown that selects a grammar to apply to an utterance based on context. In short, as described in detail below, a user 102 speaks an utterance 104. A computing device 106 detects the utterance 104 and performs an action 108 in response to the utterance. The computing device 106 can use the context 110 of the computing device 106 to determine the likely intent of the user 102. Based on the likely intent of the user 102, the computing device 106 selects an appropriate action.

[0020] More specifically, user 102 speaks utterance 104 near computing device 106. Computing device 106 may be any type of computing device configured to detect audio. For example, computing device 106 may be a mobile phone, a laptop computer, a wearable device, a smart appliance, a desktop computer, a television, or any other type of computing device capable of receiving audio data. Computing device 106 processes the audio data of utterance 104 and generates word grid 112. Word grid 112 represents different combinations of words that may correspond to the utterance and confidence scores for each word.

[0021] like Figure 1 As shown, user 102 speaks unclearly and utters utterance 104 that sounds similar to "flights out". Computing device 106 detects audio of the user's voice and uses automatic speech recognition to determine what user 102 said. Computing device 106 applies an acoustic model to the audio data. The acoustic model can be configured to determine different phonemes that may correspond to the audio data. For example, the acoustic model can determine that the audio data includes a phoneme of the "f" phoneme in addition to other phonemes. Computing device 106 can apply a language model to the phonemes. The language model can be configured to identify possible words and a series of words that match the phonemes. In some embodiments, computing device 106 can transmit the audio data to another computing device, such as a server. The server can perform speech recognition on the audio data.

[0022] The language model generates a word grid 112 that includes different combinations of words that match the utterance 104. The word grid includes two words, which can be the first words in the utterance 104. The word 114 "lights" can be the first word, or the word 116 "flights" can be the first word. The language model can calculate a confidence score that reflects the likelihood that the words in the word grid are the words spoken by the user. For example, the language model can determine that the confidence score 118 of the word 114 "lights" is 0.65, and the confidence score 120 of the word 116 "flights" is 0.35. The word grid 112 includes possible words for the second word. In this example, the language model identifies only one possible word 122 for the second word, namely "out."

[0023] In some implementations, the computing device 106 may use a sequence-to-sequence neural network or other types of neural networks in place of the acoustic model and the language model. The neural network may have one or more hidden layers and may be trained using machine learning and training data that includes audio data of sample utterances and a transcription corresponding to each sample utterance. In this case, the sequence-to-sequence neural network may generate a word grid 112 and confidence scores 118 and 120. The sequence-to-sequence neural network may not generate separate confidence scores for phonemes and word combinations as the acoustic model and the language model do. Instead, the sequence-to-sequence neural network may generate confidence scores that are a combination of confidence scores for phonemes generated by the acoustic model and confidence scores for word combinations generated by the language model.

[0024] Based on the word grid 112, the computing device 106 identifies two candidate transcriptions of the utterance 104. The first candidate transcription is "lights out" and the second candidate transcription is "flights out". The confidence score of the first candidate transcription is 0.65, and the confidence score of the second candidate transcription is 0.35. The confidence score of each candidate transcription may be the product of the confidence scores of each word in the candidate transcription. The phrase "fights out" may be the closest acoustic match to the utterance 104, but based on the language model, the combination of "fight" and "out" is unlikely to occur. Accordingly, "fights out" is not a candidate transcription. In an embodiment, when a neural network is used in place of an acoustic model and a language model, the neural network may not use the acoustic model and the language model to generate separate confidence scores.

[0025] The computing device 106 may determine a context 110 of the computing device 106. The context 110 may be based on any arbitrary combination of factors present on or around the computing device 106. For example, the context 110 may include that the computing device 106 is located at the home of the user 102. The context 110 may include that the current time is 10:00 PM and the day of the week is Tuesday. The context 110 may include that the computing device 106 is a mobile phone that is executing a digital assistant application in the foreground of the computing device 106.

[0026] In some embodiments, context 110 may include additional information. For example, context 110 may include data about the orientation of computing device 106, such as lying flat on a table or in the user's hand. Context 110 may include applications running in the background. Context may include data displayed on the screen of the computing device or audio output by computing device 106 before receiving speech 104. For example, a display including the prompt "Hey, how can I help you?" may indicate that computing device 106 is executing a virtual assistant application in the foreground. Context 110 may include weather, the identity of user 102, demographic data of user 102, and data stored on computing device 106 or accessible to computing device 106, such as contacts.

[0027] The computing device 106 can use the context to select a grammar to apply to the transcription. A grammar can be any word structure that can be described using common annotation techniques (e.g., Backus-Nohr Form). Each grammar can correspond to a specific user intent. For example, a user intent can be to issue a home automation command or a media playback command. An example of a grammar can include a grammar for an alarm. The alarm grammar can use the annotation $DIGIT = (0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9) to define a number as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 0. The alarm grammar can use the annotation $TIME = $DIGIT $DIGIT : $DIGIT $DIGIT (am | pm), which indicates that the time includes two digits, followed by a colon, then two digits, and then "morning" or "afternoon". The alarm grammar can use the annotation $MODE = (alarm | timer) to define the mode of the alarm, which indicates whether the alarm should be in alarm mode or timer mode. Finally, the alarm grammar may define the alarm syntax as $ALARM = set $MODE for $TIME, which indicates that the user can say "set the alarm to 6:00 am" or "set the timer to 20:00 pm". The computing device 106 uses the grammar to parse spoken or typed commands and identify actions for the computing device 106 to perform. Thus, the grammar provides functional data that causes the computing device 106 to operate in a specific manner in order to parse the command.

[0028] Each grammar can be active for certain contexts. For example, if the application running in the foreground of computing device 106 is an alarm application, the alarm grammar can be active. If the application running in the foreground of computing device 106 is a digital assistant application, the alarm grammar can also be active. Because the likelihood that the user intends to set an alarm in the alarm application can be higher than the digital assistant application, if computing device 106 is running the alarm application, computing device 106 can assign a probability of 0.95 to the likelihood of the alarm grammar matching the command and the user's intent, and a probability of 0.03 if computing device 106 is running the digital assistant application.

[0029] exist Figure 1 In the example shown, the computing device 106 determines the confidence score of each grammar based on different candidate transcriptions that can be applied to the grammar. In other words, the computing device 106 identifies the grammar that parses each candidate transcription. The applied grammar 124 illustrates those grammars that parse the candidate transcription "lights out" and the grammar confidence score that each grammar is the correct grammar for the user's intent. Based on the given transcription and context, a higher grammar confidence score indicates a more likely grammar. If the transcription is "lights out", the grammar confidence score 126 for the home automatic grammar is 0.35. If the transcription is "lights out", the grammar confidence score 128 for the movie command grammar is 0.25. If the transcription is "lightsout", the grammar confidence score 130 for the music command grammar is 0.20. The total score for the grammar confidence score for each transcription should be 1.0. In this case, the computing device 106 can assign the remaining grammar confidence score to the default intent grammar. In this example, if the transcription is "lights out," the grammar confidence score 132 for the default intent grammar is 0.20. The default intent grammar may not be limited to any particular action or intent and may be able to parse all or nearly all transcriptions.

[0030] In some embodiments, all grammars cannot resolve a particular transcription. In this case, the default intent grammar is applied. For example, there may be no grammar to resolve "flights out". Therefore, if the transcription is "flights out", the grammar confidence score for the default intent grammar is 1.0.

[0031] In some embodiments, the computing device 106 selects a grammar and a transcription based on calculating the product of the transcription confidence score and the grammar confidence score. Box 134 shows the result of selecting the transcription and the grammar based on the product of the transcription confidence score and the grammar confidence score. For example, the combined confidence score 136 of "lights out" and the home automation grammar is 0.2275. The combined confidence score 138 of "lights out" and the movie command grammar is 0.1625. The combined confidence score 140 of "lights out" and the music command grammar is 0.130. The combined confidence score 142 of "lights out" and the default intent grammar is 0.130. The combined confidence score 144 of "flights out" and the default intent grammar is 0.350. In this embodiment, the combined confidence score 144 is the highest score. Thus, using this technique, the computing device 106 can perform the action corresponding to the command "flights out". This may be a poor result for the user 102 and may result in the transcription "flights out" to the search engine.

[0032] If searching the Internet for "flights out" was not the intention of user 102, the user would have to repeat utterance 104. Doing so would require computing device 106 to expend additional computing and power resources processing the additional utterance. By activating the screen of computing device 102 for an additional period of time, user 102 ultimately manually enters the desired command into computing device 102, which would use additional processing and power resources.

[0033] In order to reduce the likelihood that the computing device 106 performs an action that does not match the user's intent, the computing device 106 may normalize the grammatical confidence scores of the applied grammar 124. In order to normalize the grammatical confidence scores of the applied grammar 124, the computing device 106 may calculate the factor necessary to increase the highest grammatical confidence score to 1.0. In other words, the product of the factor and the highest grammatical confidence score should be 1.0. The computing device 106 may then multiply the other grammatical confidence scores by the factor to normalize the other grammatical confidence scores. The normalization performed by the computing device 106 may be different from the traditional probability normalization in which the probabilities are adjusted to add up to one. Because the highest grammatical confidence score is increased to 1.0, the normalized probabilities do not add up to one. The increased confidence score may represent a pseudo-probability, rather than a probability in the traditional sense. The normalization process described elsewhere in this document may also generate similar pseudo-probabilities.

[0034] As shown by the normalized grammar confidence score 146, the computing device 106 may identify the grammar confidence score for the home automation grammar as the highest grammar confidence score. To increase the grammar confidence score 126 to 1.0, the computing device multiplies the grammar confidence score 126 by 2.857 (1.0 / 0.35=2.857). The computing device 106 calculates the normalized grammar confidence score 150 by multiplying the grammar confidence score 128 by 2.857. The computing device 106 calculates the normalized grammar confidence score 152 by multiplying the grammar confidence score 130 by 2.857. The computing device 106 calculates the normalized grammar confidence score 154 by multiplying the grammar confidence score 132 by 2.857. If the transcription is "flights out", the grammar confidence score for the default intent grammar is 1.0 because there is no other grammar that parses the transcription "flights out". Therefore, if the transcription is "flights out", there is no need to normalize the syntax confidence score for the default intent syntax because the score is already 1.0.

[0035] In addition to multiplying the grammar confidence score of block 124 by the transcription confidence score of word lattice 112 as shown in block 134, computing device 106 uses the normalized grammar confidence score and the transcription confidence score from word lattice 112 to calculate a combined confidence score. In particular, computing device 106 multiplies each normalized grammar confidence score by the transcription confidence score of the respective transcription. If only the default intent grammar is applicable to the grammar, the corresponding transcription confidence score remains unchanged because the transcription confidence score is multiplied by 1.0.

[0036] As shown in box 156, computing device 106 calculates combined confidence score 158 by multiplying the transcription confidence score of "lights out" by the normalized grammatical confidence score 148, resulting in a result of 0.650. Computing device 106 calculates combined confidence score 160 by multiplying the transcription confidence score of "lights out" by the normalized grammatical confidence score 150, resulting in a result of 0.464. Computing device 106 calculates combined confidence score 162 by multiplying the transcription confidence score of "lights out" by the normalized grammatical confidence score 152, resulting in a result of 0.371. Computing device 106 calculates combined confidence score 164 by multiplying the transcription confidence score of "lights out" by the normalized grammatical confidence score 154, resulting in a result of 0.371. Computing device 106 maintains the transcription confidence score for "flights out" at 0.350 because there is no grammar to parse the transcription "flights out."

[0037] In some embodiments, computing device 106 may adjust the confidence score in box 156 to take into account the syntax that is more likely to occur in the current user context given the current context of computing device 106. For example, user 102 may use computing device 106 to listen to music. Computing device 106 may be a media device that plays songs. In this case, computing device 106 may adjust confidence score 162. Computing device 106 may increase the confidence score by multiplying the confidence score by a factor, by assigning a preset value to the confidence score, or by other techniques. For example, based on the context that user 102 is listening to music through computing device 102 as a media device, the current probability of the music command is 0.9. The probability of the music command in box 124 may be confidence score 130, which is 0.20. In this case, computing device 106 may multiply confidence score 162 by a ratio of 4.5 (0.9 / 0.2=4.5). The resulting confidence score 162 may be 0.371*4.5=1.67. The other confidence scores in box 156 may be adjusted with similar ratios for each corresponding confidence score. For example, the current probability of a home automation command may be 0.04. In this case, the ratio would be 0.16 (0.04 / 0.35=0.16), which is 0.04 divided by the confidence score 126. The computing device 106 may multiply the confidence score 158 by 0.16 to calculate an adjusted or offset confidence score of 0.10. In this case, the highest confidence score ends up being the confidence score corresponding to the music command.

[0038] In some implementations, this additional adjustment step may affect which candidate transcription is selected by computing device 106. For example, if candidate transcription "flights out" is the name of a video game, and the user is expected to launch the video game, computing device 106 may adjust confidence score 166 based on a ratio similar to the ratio calculated above, and use a probability of 0.8 that the user will launch the video game based on the current context of computing device 106 and / or user 102.

[0039] In some implementations, such as when multiple grammars parse the same candidate transcription, computing device 106 may not use this rescoring step and still improve detection of user intent and speech recognition. This rescoring step may allow computing device 106 to select a different candidate transcription that is less likely based on the speech recognition confidence score.

[0040] In some implementations, computing device 106 may use this rescoring step without regard to the confidence scores of block 124. For example, computing device 106 may apply the rescoring step to the transcription confidence scores identified from word lattice 112. In this case, computing device 106 would not perform the adjustments shown in blocks 146 and 156.

[0041] Computing device 106 selects the grammar and transcription with the highest combined confidence score. For example, the home automation transcription and the transcription "lights out" may have the highest combined confidence score of 0.650. In this case, computing device 106 executes "lights out" based on the home automation grammar. Computing device 106 may turn off the lights in the house where computing device 106 is located or in the home of user 102. If computing device 106 uses the movie command syntax, computing device 106 may play the movie "Lights Out". If computing device 106 uses the music command syntax, computing device 106 may play the song "Lights Out".

[0042] like Figure 1 As shown, the display of computing device 106 can indicate the actions performed by computing device 106. Initially, computing device 106 can display prompt 168 for a digital assistant. Computing device 106 receives utterance 104 and indicates by displaying prompt 170 that computing device 106 is turning off the lights.

[0043] Figure 2 Components of an example system 200 for selecting a grammar to apply to an utterance based on context are shown. System 200 may be any type of computing device configured to receive and process speech audio. For example, system 200 may be similar to Figure 1 The components of system 200 may be implemented in a single computing device or distributed across multiple computing devices. For privacy reasons, it may be beneficial for system 200 to be implemented in a single computing device.

[0044] System 200 includes an audio subsystem 205. Audio subsystem 205 may include microphone 210, analog-to-digital converter 215, buffer 220, and various other audio filters. Microphone 210 may be configured to detect sounds in the surrounding area, such as speech. Analog-to-digital converter 215 may be configured to sample audio data detected by microphone 210. Buffer 220 may store sampled audio data for processing by system 200. In some embodiments, audio subsystem 205 may be continuously active. In this case, microphone 210 may continuously detect sounds. Analog-to-digital converter 215 may continuously sample detected audio data. Buffer 220 may store the latest sampled audio data, such as the last ten seconds of sound. If other components of system 200 do not process the audio data in buffer 220, buffer 220 may overwrite previous audio data.

[0045] The audio subsystem 205 provides the processed audio data to the speech recognizer 225. The speech recognizer provides the audio data as input to the acoustic model 230. The acoustic model 230 can be trained to identify possible phonemes corresponding to sounds in the audio data. For example, if the user says "set", the acoustic model 230 can identify phonemes corresponding to the "s" sound, the "e" vowel, and the "t" sound. The speech recognizer 225 provides the identified phonemes as input to the language model 235. The language model 235 generates a word grid. The word grid includes a word confidence score for each candidate word identified by the language model 235. For example, the word grid may indicate that the first word may be "set". The language model 235 may not identify any other possible words for the first word. The language model 235 may identify two possible words for the second word. For example, the word grid may include the words "time" and "chime" as possible second words. The language model 235 may assign a word confidence score to each word. The word confidence score for "time" may be 0.75, and the word confidence score for "chime" may be 0.25.

[0046] The speech recognizer 225 can generate candidate transcriptions based on the word grid. Each candidate transcription can have a transcription confidence score that reflects the likelihood that the speaker said the words in the transcription. For example, a candidate transcription can be "set time" and have a transcription confidence score of 0.75. Another candidate transcription can be "set chime" and have a transcription confidence score of 0.25.

[0047] While the speech recognizer 225 is generating word grids, candidate transcriptions, and transcription confidence scores, the context identifier 240 may be collecting context data indicating the current context of the system 200. The context identifier 240 may collect sensor data 245 from any sensors of the system. The sensors may include position sensors, thermometers, accelerometers, gyroscopes, gravity sensors, time and date, and any other similar sensors. The sensor data 245 may also include data about the state of the system 200. For example, the state may include the battery charge of the system 200, the signal strength, any nearby devices that the system 200 can communicate with or is aware of, and any other similar state of the system 200.

[0048] The context identifier 240 may also collect system process data 250 indicating processes being executed by the system 200. The process data 250 may indicate the memory allocated to each process, the processing resources allocated to each process, the applications being executed by the system 200, the applications running in the foreground or background of the system 200, the interface content on the system display, and any similar process data.

[0049] As an example, the context identifier 240 may receive sensor data 245 and system process data 250 indicating that the system 200 is the user's home, the time is 6:00 PM, the day is Monday, the foreground application is a digital assistant application, and the device is a tablet computer.

[0050] The grammar score generator 255 receives the context of the system 200 from the context identifier 240 and the word lattice from the speech recognizer 225. The grammar score generator 255 identifies a grammar 260 for each candidate transcription that resolves the word lattice. In some cases, none of the grammars 260 resolves the candidate transcription. In this case, the grammar score generator 255 assigns a default intent grammar with a grammar confidence score of 1.0 to the candidate transcription that none of the grammars 260 can resolve.

[0051] In some cases, a single grammar 260 may parse the candidate transcription. In this case, assuming that the candidate transcription is an actual transcription, the grammar score generator 255 may determine a grammar confidence score for the single grammar. Because the grammar confidence score represents a probability, the grammar confidence score may be less than 1.0. The grammar score generator 255 assigns the difference between 1.0 and the grammar confidence score to the grammar confidence score for the default intent grammar for the candidate transcription.

[0052] In some cases, multiple grammars 260 may resolve the candidate transcription. In this case, assuming that the candidate transcription is the actual transcription, the grammar score generator 255 may determine a grammar confidence score for each grammar. Because the grammar confidence scores represent a set of probabilities, the sum of the grammar confidence scores may be less than 1.0. The grammar score generator 255 assigns the difference between 1.0 and the sum of the grammar confidence scores to the grammar confidence score for the default intent grammar for the candidate transcription.

[0053] The grammar score normalizer 265 receives the grammar confidence score for each candidate transcription and normalizes the grammar confidence scores. In some embodiments, the grammar score normalizer 265 normalizes the grammar confidence scores of only grammars other than the default intent grammar. In the event that the grammar score generator 255 generates a grammar confidence score for a grammar for a particular candidate transcription, the grammar score normalizer 265 increases the grammar confidence score to 1.0. In the event that the grammar score generator 255 does not generate a grammar confidence score for any grammar for a particular candidate transcription, the grammar score normalizer 265 maintains the grammar confidence score of the default intent grammar at 1.0.

[0054] In the case where the syntax score generator 255 generates multiple syntax confidence scores for each of the multiple syntaxes for a particular candidate transcription, the syntax score normalizer 265 identifies the highest syntax confidence score for the particular candidate transcription. The syntax score normalizer 265 calculates a factor that increases the highest syntax confidence score to 1.0, such that the product of the factor and the highest syntax confidence score is 1.0. The syntax score normalizer 265 increases other syntax confidence scores for the same particular transcription by multiplying the factor by each other syntax confidence score.

[0055] The grammar and transcription selector 270 receives the normalized grammar confidence score and the transcription confidence score. Using both the grammar confidence score and the transcription confidence score, the grammar and transcription selector 270 identifies the grammar and transcription that are most likely to match the received utterance and the speaker intent by calculating a combined confidence score. The grammar and transcription selector 270 determines each combined confidence score by calculating the product of each normalized grammar confidence score and the transcription confidence score for the corresponding candidate transcription. In the case where the default intent grammar is the only grammar for a particular candidate transcription, the grammar and transcription selector 270 maintains the transcription confidence score as the combined confidence score. The grammar and transcription selector 270 selects the grammar and candidate transcription with the highest combined confidence score.

[0056] Action identifier 275 receives the selected grammar and selected transcription from grammar and transcription selector 270 and identifies the action to be performed by system 200. The selected grammar may indicate the type of action, such as setting an alarm, sending a message, playing a song, calling a person, or any other similar action. The selected transcription may indicate the details of the action type, such as how long to set an alarm, to whom to send a message, which song to play, to whom to call, or any other similar action details. Grammar 260 may include information on the action type for a particular grammar. Grammar 260 may include information on how the action identifier 275 should parse the candidate transcription to determine the action type details. For example, the grammar may be a $ALARM grammar, and action identifier 275 parses the selected transcription to determine that a timer is set to twenty minutes. Action identifier 275 may perform an action or provide instructions to another part of system 200 (e.g., a processor).

[0057] In some embodiments, the user interface generator 280 displays an indication of an action or an indication of performing an action or both. For example, the user interface generator 280 may display a timer that counts down from twenty minutes or a confirmation to turn off the lights in the house. In some cases, the user interface generator 280 may not provide any indication of the action performed. For example, the action may be to adjust the thermostat. The system 200 may adjust the thermostat without generating a user interface for display on the system 200. In some embodiments, the user interface generator 280 may generate an interface for the user to interact with the action or confirm the action. For example, the action may be to call mother. The user interface generator 280 may generate a user interface for the user to confirm the action of calling mother before the system 200 performs the action.

[0058] Figure 3 Flowchart of an example process 300 for selecting a grammar based on context. In general, process 300 performs speech recognition on audio and identifies an action to be performed based on a transcription of the audio and parsing the grammar of the audio. Process 300 normalizes the grammar confidence score to identify an action that the speaker may have intended to take. Process 300 will be described as being performed by a computer system including one or more computers, such as Figure 1 The computing device 106 or Figure 2 system 200.

[0059] The system receives audio data of an utterance (310). For example, a user may say an utterance that sounds like "lights out" or "flights out". The system detects the utterance or receives audio data of the utterance through a microphone. The system may process the audio data using an audio subsystem.

[0060] The system uses an acoustic model and a language model to generate a word grid that includes multiple candidate transcriptions of the utterance and includes a transcription confidence score, each transcription confidence score reflecting the likelihood that the corresponding candidate transcription is a match for the utterance (320). The system uses automatic speech recognition to generate the word grid. The automatic speech recognition process can include providing audio data as input to an acoustic model that identifies different phonemes that match each portion of the audio data. The automatic speech recognition process can include providing the phonemes as input to a language model that generates a word grid that includes a confidence score for each candidate word in the utterance. The language model selects words for the word grid from a vocabulary. The vocabulary can include words of a language that the system is configured to recognize. For example, the system can be configured for English and the vocabulary includes English words. In some embodiments, the actions of process 300 do not include limiting the words in the vocabulary that the system can recognize. In other words, process 300 can generate transcriptions using any word in the language that the system is configured to recognize. Because the system recognizes every word in the language, when a user says something unexpected, the system's speech recognition process will work properly as long as it is in the language the system is configured to recognize.

[0061] The system can use the word grid to generate different candidate transcriptions. Each candidate transcription can include a different transcription confidence score that reflects the likelihood that the user said the transcription. For example, the candidate transcription "lights out" can have a confidence score of 0.65, and the candidate transcription "flights out" can have a confidence score of 0.35.

[0062] The system determines the context of the system (330). In some implementations, the context is based on the location of the system, applications running in the foreground of the computing device, demographics of the user, data stored on or accessible to the system (such as contact data, previous user queries or commands, time of day, date, day of week, weather, system orientation), and any other similar types of information.

[0063] Based on the context of the system, the system identifies grammars corresponding to the plurality of candidate transcriptions (340). The grammars may include different structures for different commands that the system is capable of executing. For example, the grammar may include command structures for setting an alarm, playing a movie, performing an Internet search, checking a user's calendar, or any other similar action.

[0064] Based on the current context, the system determines a grammatical confidence score for each of a plurality of candidate transcriptions, the grammatical confidence score reflecting the likelihood that the corresponding grammar is a match for the corresponding candidate transcription (350). The grammatical confidence score can be considered as a conditional probability of the likelihood of the grammar matching the speaker's intent, given that a candidate transcription is a transcription of an utterance, based on the context of the system. For example, if the transcription is "lights out," the grammatical confidence score for the home automation grammar can be 0.35. In other words, the conditional probability of the home automation grammar is 0.35, given that the transcription is "lights out." The system can generate a grammatical confidence score for each grammar that parses the candidate transcription. In some embodiments, due to the context, the system can generate grammatical confidence scores for only some of the grammars that parse the candidate transcriptions. For each candidate transcription, the sum of the grammatical confidence scores adds up to 1.0, and the system can assign any remaining probability to the default intent grammar.

[0065] Based on the transcription confidence scores and the grammar confidence scores, the system selects a candidate transcription from the candidate transcriptions (360). In some embodiments, the system normalizes the grammar confidence scores by increasing those candidate transcriptions that match more than one grammar. For each candidate transcription, the system may multiply the highest grammar confidence score by a factor required to normalize the grammar confidence score to 1.0. The system may multiply other grammar confidence scores for the same candidate transcription by the same factor. The system may multiply the normalized grammar confidence score by the transcription confidence score to generate a combined confidence score. The system selects the grammar and transcription with the highest combined confidence score.

[0066] The system provides the selected candidate transcription as a transcription of the utterance for output (370). In some implementations, the system performs an action based on the grammar and the candidate transcription. The grammar can indicate an action to be taken. The action can include playing a movie, calling a contact, turning on a light, sending a message, or any other similar type of action. The selected transcription can include a contact to call, a message to send, a recipient of the message, the name of the movie, or any other similar details.

[0067] In some embodiments, the system utilizes a process of labeling a word grid for a given set of grammars using a finite state transducer (FST). In some embodiments, the system may constrain the grammar to be the union of all grammars that match the current context. In some embodiments, the system may match grammars in advance offline or dynamically at runtime.

[0068] In some embodiments, process 300 can weight the edges of a word grid. The system can compile a grammar into a weighted finite state transformer of a constraint grammar. In the finite state transformer, arc weights or edge weights can encode the probability of a word of a given grammar, which can be a negative logarithmic weight. In some embodiments, all or nearly all grammars associated with a word grid can be unioned with each other.

[0069] Process 300 can continue with a system that determines a context-dependent probability for each grammar, which can be the probability of the grammar given the context. The system can determine the context-dependent probability by providing the components and corresponding weights, which can also be encoded as arcs or edges in the grammar constraint, such as its open decorator arc.

[0070] Process 300 may continue with the system clearing the weights of the word lattice. The system may construct the word lattice with the grammar constraints, which may cause spans matching the grammar to be marked. For example, spans matching the grammar may be surrounded by opening and closing decorator tags, such as for spans.<media_commands> play song< / media_commands> , or a transcription of “play song”.

[0071] Process 300 can continue to normalize the probabilities. The system generates a copy of the label grid and removes the decorator tags. The system determines and minimizes the arc costs in the tropical semi-ring, which results in each unique word path through which the highest probability is encoded. The probability can be the probability of the word path given the grammar multiplied by the probability of the grammar. The probability can be inverted by reversing the sign of the arc weight when using negative logarithmic weights, and the inverse grid can be composed of the label grid. For each word sequence or path, using the composition of the inverse grid may be equivalent to dividing the grid weights, in this case, the highest probability amount of the word path given the grammar multiplied by the probability of the grammar. Therefore, the system can divide the probabilities of the best label paths by themselves, and each can become 1.0. In some embodiments, non-optimal paths will receive lower pseudo-probabilities, thereby generating a pseudo-probability of the grammar of the grammar given the grid of the expected amount and the given word path.

[0072] In some embodiments, process 300 may include discarding grammars with low conditional probabilities by using beam pruning. For example, if there are too many grammars that match the utterance "lights out" with a probability of 0.32 for home automation, 0.25 for movie commands, 0.20 for music commands, 0.10 for general search, 0.03 for online transactions, 0.03 for social groups, and 0.02 for online encyclopedia searches. The system can reduce the processing burden by pruning grammars that are below a threshold or below a threshold percentage of the most likely interpretation or grammar. For example, the system can prune those grammars that are one-tenth lower than the most likely grammar. The system can then remove those grammars that are less than 0.032 for a given transcription. The rationale behind pruning is that the probability of a low-scoring label is so low that the lack of a reasonable offset will cause that interpretation to be selected over more likely interpretations. Pruning may require the system to add a finite state transformer that operates on the resulting grid with a desired pruning weight that specifies how far away from the best hypothesis a label must be to be pruned.

[0073] The benefits of process 300 allow the system to shift the grammar directly on the lattice. Process 300 has lower complexity than other word lattice tagging algorithms. Process 300 resolves probability fragments caused by tag ambiguity by implementing maximum normalization. Process 300 allows relatively unlikely suggested tags to be discarded by using beam pruning.

[0074] In some embodiments, process 300 may extract the n best hypotheses from the grid and label each hypothesis separately, and optionally eventually recombine them back into the grid. The system may assume that the labels are otherwise identical. Doing so may produce the same results, but with longer latency and / or lower recall. If the n best hypotheses after the first N hypotheses are removed, such as N=100, then recall may be reduced, which may help manage latency. Because the number of different sentences in a grid may grow exponentially relative to the number of words in it, some solutions may be exponentially slower in the worst case, unless the best-N restriction described above is used, which may limit the number of choices examined.

[0075] Figure 4Examples of computing device 400 and mobile computing device 450 that can be used to implement the techniques described herein are shown. Computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown in this figure, their connections and relationships, and their functions are intended to be exemplary only and are not intended to be limiting.

[0076] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connected to the memory 404 and a plurality of high-speed expansion ports 410, and a low-speed interface 412 connected to a low-speed expansion port 414 and the storage device 406. Each of the processor 402, the memory 404, the storage device 406, the high-speed interface 408, the high-speed expansion port 410, and the low-speed interface 412 is interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 402 is capable of processing instructions for execution within the computing device 400, including instructions stored in the memory 404 or on the storage device 406, to display GUI graphical information on an external input / output device such as a display 416 coupled to the high-speed interface 408. In other embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used appropriately. Moreover, multiple computing devices can be connected, each device providing multiple parts of the necessary operations (for example, as a server array, a blade server group, or a multi-processor system).

[0077] Memory 404 stores information within computing device 400. In some implementations, memory 404 is one or more volatile memory units. In some implementations, memory 404 is one or more non-volatile memory units. Memory 404 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0078] Storage device 406 can provide mass storage for computing device 400. In some embodiments, storage device 406 can be or include computer-readable media, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device or an array of devices (including devices in a storage area network or other configurations). Instructions can be stored in an information carrier. Instructions execute one or more methods, such as those described above, when executed by one or more processing devices (e.g., processor 402). Instructions can also be stored by one or more storage devices (e.g., memory 404, storage device 406, or memory on processor 402) such as a computer or machine-readable medium.

[0079] The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low-speed interface 412 manages less bandwidth-intensive operations. Such functional allocation is for example only. In some embodiments, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 410 that can accept various expansion cards (not shown). In this embodiment, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0080] As shown, computing device 400 can be implemented in several different forms. For example, it can be implemented as a standard server 420 or multiple times in a group of such servers. In addition, it can be implemented in a personal computer (such as laptop computer 422). It can also be implemented as part of a rack server system 424. Alternatively, components from computing device 400 can be combined with other components in a mobile device (not shown) such as mobile computing device 450. Each such device can contain one or more of computing device 400 and mobile computing device 450, and the entire system can be composed of multiple computing devices that communicate with each other.

[0081] The mobile computing device 450 includes a processor 452, a memory 464, an input / output device such as a display 454, a communication interface 466, and a transceiver 468, among other components. The mobile computing device 450 may also be provided with a storage device such as a microdrive or other device for providing additional storage. Each of the processor 452, the memory 464, the display 454, the communication interface 466, and the transceiver 468 are interconnected using various buses, and multiple components may be mounted on a common motherboard or otherwise appropriately mounted.

[0082] Processor 452 is capable of executing instructions within mobile computing device 450, including instructions stored in memory 464. Processor 452 may be implemented as a chipset including independent and multiple analog and digital processor chips. For example, processor 452 may provide cooperation with other components of mobile computing device 450, such as control of a user interface, applications executed by mobile computing device 450, and wireless communications performed by mobile computing device 450.

[0083] Processor 452 can communicate with the user through control interface 458 and display interface 456 coupled to display 454. For example, display 454 can be a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other appropriate display technology. Display interface 456 may include appropriate circuits for driving display 454 to present graphical information and other information to the user. Control interface 458 can receive commands from the user and convert them for submission to processor 452. In addition, external interface 462 can provide communication with processor 452 to enable mobile computing device 450 to communicate with other devices in the vicinity. For example, external interface 462 can provide wired communication in some embodiments or wireless communication in other embodiments, and multiple interfaces can also be used.

[0084] The memory 464 stores information within the mobile computing device 450. The memory 464 can be implemented as one or more computer-readable media or media, one or more volatile storage units, or one or more non-volatile storage units. An expansion memory 474 may also be provided and connected to the mobile computing device 450 via an expansion interface 472, for example, the expansion interface may include a SIMM (Single In-line Memory Module) card interface. The expansion memory 474 may provide additional storage space for the mobile computing device 450, or may also store applications or other information for the mobile computing device 450. Specifically, the expansion memory 474 may include instructions for implementing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 474 may be configured as a security module for the mobile computing device 450, and may be programmed with instructions that allow the mobile computing device 450 to be used securely. In addition, a security application may be provided via a SIMM card together with additional information, such as placing identification information on the SIMM card in an unbreakable manner.

[0085] For example, the memory may include flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some embodiments, the information carrier stores instructions that, when executed by one or more processing devices (e.g., processor 452), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices (e.g., memory 464, expansion memory 474, or memory on processor 452), such as one or more computer or machine readable media. In some embodiments, the instructions can be received in a propagated signal, for example, by transceiver 468 or external interface 462.

[0086] The mobile computing device 450 can communicate wirelessly via a communication interface 466, which may include digital signal processing circuitry as necessary. The communication interface 466 may provide communications in various modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Message Service) or MMS messages (Multimedia Message Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000 or GPRS (General Packet Radio Service), etc. For example, such communications may occur using radio frequencies via a transceiver 468. In addition, short-range communications may occur such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation and location-related wireless data to the mobile computing device 450, which may be used appropriately by applications running on the mobile computing device 450.

[0087] Mobile computing device 450 may also communicate audibly using audio codec 460, which may receive spoken information from a user and convert it into usable digital information. Audio codec 460 may also generate audible sounds for the user, such as through a speaker, such as in a handset of mobile computing device 450. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on mobile computing device 450.

[0088] As shown, mobile computing device 450 can be implemented in many different forms. For example, it can be implemented as a cellular phone 480. It can also be implemented as part of a smart phone 482, a personal digital assistant, or other similar mobile devices.

[0089] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose processor, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0090] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented using high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic input, voice input, or tactile input.

[0092] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser that a user can use to interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet. In some embodiments, the systems and techniques described herein can be implemented on an embedded system where speech recognition and other processing are performed directly on the device.

[0093] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0094] Although some embodiments have been described in detail above, other modifications may be made. For example, although the client application is described as accessing the delegate, in other embodiments, the delegate may be adopted by other applications implemented by one or more processors, such as applications executed on one or more servers. In addition, the logic flow depicted in the accompanying drawings does not require the specific order or sequence shown to obtain the desired results. In addition, other actions can be provided, or actions can be cancelled from the process, and other components can be added to the system or deleted from the system. Accordingly, other embodiments fall within the scope of the appended claims.

Claims

1. A computer-implemented method, which when executed by data processing hardware causes the data processing hardware to perform operations comprising: determining a context indicating that a computing device associated with a user is executing a digital assistant application; receiving audio data of utterances spoken by a user and captured by the computing device; determining, from the audio data, phonemes corresponding to the utterance; generating one or more candidate transcriptions of the utterance using the phonemes corresponding to the utterance, each candidate transcription of the one or more candidate transcriptions having a corresponding transcription likelihood score; as well as For a corresponding candidate transcription among the one or more candidate transcriptions having the highest corresponding transcription likelihood score, parsing is performed using a grammar based on the context indicating that the computing device is executing the digital assistant application to identify an action to be performed by the computing device.

2. The computer-implemented method of claim 1 , wherein: The operations further include instructing the computing device to perform the identified action.

3. The computer-implemented method of claim 1 , wherein: The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device.

4. The computer-implemented method of claim 3, wherein: Each grammar of the plurality of grammars includes a different specified structure for terms.

5. The computer-implemented method of claim 1 , wherein: The computing device comprises a mobile phone or a wearable device.

6. The computer-implemented method of claim 1 , wherein: The grammar includes a default grammar.

7. The computer-implemented method of claim 1 , wherein: Each candidate transcription of the one or more candidate transcriptions includes a plurality of terms.

8. The computer-implemented method of claim 7, wherein: Each term of the plurality of terms includes a corresponding term confidence score.

9. The computer-implemented method of claim 1 , wherein: The transcription likelihood score indicates a likelihood that the candidate transcription matches the utterance spoken by the user.

10. The computer-implemented method of claim 1, wherein: The data processing hardware is located on the computing device.

11. A system comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations comprising: determining a context indicating that a computing device associated with a user is executing a digital assistant application; receiving audio data of utterances spoken by a user and captured by the computing device; determining, from the audio data, phonemes corresponding to the utterance; generating one or more candidate transcriptions of the utterance using the phonemes corresponding to the utterance, each candidate transcription of the one or more candidate transcriptions having a corresponding transcription likelihood score; and For a corresponding candidate transcription among the one or more candidate transcriptions having the highest corresponding transcription likelihood score, parsing is performed using a grammar based on the context indicating that the computing device is executing the digital assistant application to identify an action to be performed by the computing device.

12. The system according to claim 11, wherein: The operations further include instructing the computing device to perform the identified action.

13. The system according to claim 11, wherein: The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device.

14. The system according to claim 13, wherein: Each grammar of the plurality of grammars includes a different specified structure for terms.

15. The system according to claim 11, wherein: The computing device includes a mobile phone or a wearable device.

16. The system according to claim 11, wherein: The grammar includes a default grammar.

17. The system of claim 11, wherein: Each candidate transcription of the one or more candidate transcriptions includes a plurality of terms.

18. The system of claim 17, wherein: Each term of the plurality of terms includes a corresponding term confidence score.

19. The system of claim 11, wherein: The transcription likelihood score indicates a likelihood that the candidate transcription matches the utterance spoken by the user.

20. The system of claim 11, wherein: The data processing hardware is located on the computing device.

21. A computer-implemented method, which when executed by data processing hardware causes the data processing hardware to perform operations comprising: receiving audio data of utterances spoken by a user and captured by a computing device associated with the user; determining, from the audio data, phonemes corresponding to the utterance; generating one or more candidate transcriptions of the utterance using the phonemes corresponding to the utterance, each candidate transcription of the one or more candidate transcriptions having a corresponding transcription likelihood score; determining a context indicating that a user is listening to music via the computing device; selecting, based on the context indicating that the user is listening to music via the computing device, a grammar corresponding to a specified user intent to issue a media play command; as well as For a corresponding candidate transcription of the one or more candidate transcriptions having a highest corresponding transcription likelihood score, the corresponding candidate transcription is parsed using a grammar selected to correspond to the specified user intent to issue the media playback command to identify an action to be performed by the computing device.

22. The computer-implemented method of claim 21, wherein: The operations further include instructing the computing device to perform the identified action.

23. The computer-implemented method of claim 21, wherein: The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device.

24. The computer-implemented method of claim 23, wherein: Each grammar of the plurality of grammars includes a different specified structure for terms.

25. The computer-implemented method of claim 21, wherein: The computing device comprises a mobile phone.

26. The computer-implemented method of claim 21, wherein: The computing device includes a wearable device.

27. The computer-implemented method of claim 21, wherein: Each candidate transcription of the one or more candidate transcriptions includes a plurality of terms.

28. The computer-implemented method of claim 27, wherein: Each term of the plurality of terms includes a corresponding term confidence score.

29. The computer-implemented method of claim 21, wherein: The transcription likelihood score indicates a likelihood that the candidate transcription matches the utterance spoken by the user.

30. The computer-implemented method of claim 21, wherein: The data processing hardware is located on the computing device.

31. A system comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations comprising: receiving audio data of utterances spoken by a user and captured by a computing device associated with the user; determining, from the audio data, phonemes corresponding to the utterance; generating one or more candidate transcriptions of the utterance using the phonemes corresponding to the utterance, each candidate transcription of the one or more candidate transcriptions having a corresponding transcription likelihood score; determining a context indicating that a user is listening to music via the computing device; Based on the context indicating that the user is listening to music via the computing device, selecting a grammar corresponding to a specified user intent to issue a media playback command; and For a corresponding candidate transcription of the one or more candidate transcriptions having a highest corresponding transcription likelihood score, the corresponding candidate transcription is parsed using a grammar selected to correspond to the specified user intent to issue the media playback command to identify an action to be performed by the computing device.

32. The system of claim 31, wherein: The operations further include instructing the computing device to perform the identified action.

33. The system of claim 31, wherein: The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device.

34. The system of claim 33, wherein: Each grammar of the plurality of grammars includes a different specified structure for terms.

35. The system of claim 31, wherein: The computing device comprises a mobile phone or a wearable device.

36. The system of claim 31, wherein: The grammar includes a default grammar.

37. The system of claim 31, wherein: Each candidate transcription of the one or more candidate transcriptions includes a plurality of terms.

38. The system of claim 37, wherein: Each term of the plurality of terms includes a corresponding term confidence score.

39. The system of claim 31, wherein: The transcription likelihood score indicates a likelihood that the candidate transcription matches the utterance spoken by the user.

40. The system of claim 31, wherein: The data processing hardware is located on the computing device.

Images