Voice input processing

The speech input processing system addresses the challenge of ambiguous voice inputs by using context-based grammar selection to enhance accuracy and reduce latency in speech recognition systems.

JP2024163133AActive Publication Date: 2024-11-21GOOGLE LLC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024147124
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2018-12-03
Filing Date
2024-08-29
Publication Date
2024-11-21
Estimated Expiration
2039-11-27

AI Technical Summary

Technical Problem

Existing speech recognition systems struggle to accurately interpret user intent from ambiguous voice inputs due to the lack of context-based grammar selection, leading to potential misinterpretation and increased processing resources for corrections.

Method used

A speech input processing system that utilizes context to develop multiple grammars for candidate transcriptions, adjusting grammar confidence scores based on user and system context, and selects the most likely transcription and grammar combination to reduce misinterpretation and resource consumption.

Benefits of technology

Enhances speech recognition accuracy by reducing human-machine interface latency and minimizing unnecessary processing by applying context-aware grammar selection, effectively handling unexpected inputs and improving user intent detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163133000001_ABST
    Figure 2024163133000001_ABST
Patent Text Reader

Abstract

To execute voice input processing.SOLUTION: The processing includes determining a context indicating that a user is listening to the music through a computing device, receiving audio data of an utterance, generating one or more candidate transcriptions for the utterance, and parsing a candidate transcription from the one or more candidate transcriptions having the highest transcription confidence score using a grammar based on the context indicating that the user is listening to the music through the computing device to identify an action to be performed by the computing device.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This specification relates generally to speech recognition. [Background technology]

[0002] There is an increasing demand to be able to interact with computers using voice input. This requires the development of methods to program computers to process the input, particularly to process and analyze natural language data. Such processing may involve speech recognition, a branch of computational linguistics that allows a computer to recognize spoken language and convert it into text. Summary of the Invention [Problem to be solved by the invention]

[0003] This specification relates generally to speech recognition. [Means for solving the problem]

[0004] To allow a user to provide input to a computing device via voice, a speech input processing system can use context to identify multiple grammars to apply to multiple candidate transcriptions generated by an automatic speech recognizer. Each grammar may indicate a different speaker intent or action for the system to perform on the same candidate transcription. The system can select a grammar and a candidate transcription based on the grammar to parse the candidate transcription and the likelihood that the grammar matches the user intent and the likelihood that the candidate transcription matches what the user said. The system can then use details contained in the selected candidate transcription to perform the action corresponding to the grammar.

[0005] More specifically, the speech processing system receives an utterance from a user and generates a word lattice. The word lattice is a data structure reflecting likely words in the utterance and a confidence score for each word. From the word lattice, the system identifies a plurality of candidate transcriptions and a transcription confidence score for each candidate transcription. The system identifies a current context based on characteristics of the user, a location of the system, characteristics of the system, applications running on the system (e.g., currently active applications or applications running in the foreground), or other similar contextual data. Based on the context, the system generates grammar confidence scores for a plurality of grammars that parse each candidate transcription. If multiple grammars may apply to the same candidate transcription, the system may adjust some of the multiple grammar confidence scores. The system selects a grammar and a candidate transcription based on a combination of the adjusted multiple grammar confidence scores and the multiple transcription confidence scores.

[0006] In accordance with an innovative aspect of the subject matter described in the present application, a method for processing speech input includes the actions of receiving, by a computing device, audio data of an utterance; generating, by the computing device, a word lattice using an acoustic model and a language model, the word lattice including a plurality of candidate transcriptions of the utterance and including a plurality of transcription confidence scores, each of the plurality of transcription confidence scores reflecting a likelihood that a respective candidate transcription matches the utterance; determining, by the computing device, a context of the computing device; identifying, by the computing device, a plurality of grammars corresponding to the plurality of candidate transcriptions based on the context of the computing device; determining, by the computing device, for each of the plurality of candidate transcriptions, a plurality of grammar confidence scores reflecting a likelihood that the respective grammar matches the respective candidate transcription based on the current context; selecting, by the computing device, a candidate transcription from among the plurality of candidate transcriptions based on the plurality of transcription confidence scores and the plurality of grammar confidence scores; and providing, for output by the computing device, the selected candidate transcription as a transcription of the utterance.

[0007] These and other implementations may each optionally include one or more of the following features: The action includes determining that two or more of the plurality of grammars correspond to one of the plurality of candidate transcriptions, and adjusting a plurality of grammar confidence scores for the two or more grammars based on determining that two or more of the plurality of grammars correspond to one of the plurality of candidate transcriptions. The computing device selects a candidate transcription from among the plurality of candidate transcriptions based on the transcription confidence score and the adjusted grammar confidence score. The action of adjusting the plurality of grammar confidence scores for the two or more grammars includes increasing each of the plurality of grammar confidence scores by a factor for each of the two or more grammars. The action includes determining, for each of the plurality of candidate transcriptions, a product of the respective transcription confidence score and the respective grammar confidence score. The computing device selects a candidate transcription from among the plurality of candidate transcriptions based on a product of the transcription confidence score and the respective grammar confidence score. The action of determining, by the computing device, a context of the computing device is based on a location of the computing device, an application running in the foreground of the computing device, and a time of day. The language model is configured to identify a probability for a sequence of words included in the word lattice. The acoustic model is configured to identify phonemes that match the portion of the audio data. The action includes performing, by the computing device, an action based on the selected candidate transcription and a grammar that matches the selected candidate transcription.

[0008] Other embodiments of this aspect include corresponding systems, devices, and computer programs stored on computer storage devices, each configured to perform the operations of the methods.

[0009] Certain embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages: The speech recognition system may use both the received speech input and the determined context to select a grammar used to further process the received speech input and cause the computing device to perform an action. In this manner, the speech recognition system may reduce latency in a human-machine interface by applying a limited number of grammars to multiple candidate transcriptions. The speech recognition system may use a vocabulary that includes all or nearly all words of a language, such that the speech recognition system can output a transcription when the system receives an unexpected input.

[0010] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example system for selecting a grammar to apply to an utterance based on context. [Diagram 2] 1 illustrates components of an exemplary system for selecting a grammar to apply to an utterance based on context. [Diagram 3] 1 is a flow chart illustrating an example process for selecting a grammar based on a context. [Figure 4] 1 illustrates an example of a computing device and a mobile computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Like reference numbers and designations in the various drawings indicate like elements. 1 illustrates an example system 100 for selecting a grammar to apply to an utterance based on a context. As described briefly and in more detail below, a user 102 makes an utterance 104. A computing device 106 detects the utterance 104 and performs an action 108 in response to the utterance. The computing device 106 can use a context 110 of the computing device 106 to determine a probable intent of the user 102. Based on the probable intent of the user 102, the computing device 106 selects an appropriate action.

[0013] More specifically, the user 102 makes an utterance 104 near a computing device 106. The computing device 106 may be any type of computing device configured to detect audio. For example, the computing device 106 may be a mobile phone, a laptop computer, a wearable device, a smart appliance, a desktop computer, a television, or any other type of computing device capable of receiving audio data. The computing device 106 processes the audio data of the utterance 104 and generates a word lattice 112. The word lattice 112 may represent different combinations of words, each of which may correspond to an utterance and a confidence score.

[0014] As shown in FIG. 1, a user 102 does not speak clearly and makes an utterance 104 that sounds like "flights out." A computing device 106 detects audio of the user's voice and uses automatic speech recognition to determine what the user 102 said. The computing device 106 applies an acoustic model to the audio data. The acoustic model may be configured to determine various phonemes that are likely to correspond to the audio data. For example, the acoustic model may determine that the audio data includes a phoneme for an "f" phoneme in addition to other phonemes. The computing device 106 may apply a language model to the phonemes. The language model may be configured to identify likely words and sequences of words that match the phonemes. In some implementations, the computing device 106 may transmit the audio data to another computing device, e.g., a server. The server may perform speech recognition on the audio data.

[0015] The language model generates a word lattice 112 that includes different combinations of words that match the utterance 104. The word lattice includes two words that may be the first word of the utterance 104. In some cases, the word 114 "lights" is the first word, and in other cases, the word 116 "flights" is the first word. The language model may calculate a confidence score that reflects the likelihood that a word in the word lattice is the word spoken by the user. For example, the language model may determine that the word 114 "lights" has a confidence score 118 of 0.65 and the word 116 "flights" has a confidence score 120 of 0.35. The word lattice 112 includes words that may be the second word. In this example, the language model has identified only one possible word 122 "out" as the second word.

[0016] In some implementations, the computing device 106 may use a sequence-to-sequence neural network, or other type of neural network, instead of the acoustic model and the language model. The neural network may have one or more hidden layers and may be trained using machine learning and training data including audio data of sample utterances and transcriptions corresponding to each sample utterance. In this example, the sequence-to-sequence neural network may generate the word lattice 112 and the confidence scores 118 and 120. The sequence-to-sequence neural network may not generate separate confidence scores for phoneme and word combinations as the acoustic model and the language model do. Instead, the sequence-to-sequence neural network may generate a confidence score that is a combination of the phoneme confidence scores generated by the acoustic model and the word combination confidence scores generated by the language model.

[0017] Based on the word lattice 112, the computing device 106 identifies two candidate transcriptions of the utterance 104. The first candidate transcription is "lights out" and the second candidate transcription is "flights out". The first candidate transcription has a confidence score of 0.65 and the second candidate transcription has a confidence score of 0.35. The confidence score of each candidate transcription may be the product of the confidence scores of each word of the candidate transcription. The phrase "fights out" may be the closest acoustic match to the utterance 104, but based on the language model, the combination of "fight" and "out" is unlikely to occur. Therefore, "fights out" is not a candidate transcription. In implementations where a neural network is used instead of an acoustic model and a language model, the neural network may not use the acoustic model and the language model to generate separate confidence scores.

[0018] The computing device 106 can determine a context 110 of the computing device 106. The context 110 may be based on any combination of factors present on or around the computing device 106. For example, the context 110 may include that the computing device 106 is located at the home of the user 102. The context 110 may include that the current time is 10:00 p.m. and the day of the week is Tuesday. The context 110 may include that the computing device 106 is a mobile phone running a digital assistant application in the foreground of the computing device 106.

[0019] In some implementations, the context 110 may include additional information. For example, the context 110 may include data regarding the orientation of the computing device 106, such as flat on a table or on the user's hand. The context 110 may include applications running in the background. The context may include audio output by the computing device 106 or data displayed on the computing device's screen before receiving the utterance 104. For example, a display including a prompt "Hello, how can I help you?" may indicate that the computing device 106 is running a virtual assistant application in the foreground. The context 110 may include data stored on or accessible by the computing device 106, such as the weather, the identity of the user 102, demographic data of the user 102, and contacts.

[0020] The computing device 106 can use the context to select a grammar to apply to the transcription. A grammar can be any structure of multiple words that can be written using a common notation, such as the Backus-Naur form. Each grammar can correspond to a particular user intent. For example, the user intent can be to issue a home automation command or a media playback command. An example of a grammar can include an alarm grammar. An alarm grammar can define digits as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 0 using the notation $DIGIT=(0|1|2|3|4|5|6|7|8|9). An alarm grammar can define the time using the notation $TIME=$DIGIT$DIGIT:$DIGIT$DIGIT(am|pm), which indicates that the time should have two digits, followed by a colon, followed by two digits, followed by "am" or "pm". The alarm grammar can define the mode of the alarm using the notation $MODE=(alarm|timer) which indicates whether the alarm should be in alarm mode or timer mode. Finally, the alarm grammar can define the alarm syntax as $ALARM=set$MODEfor$TIME which indicates that the user can say "set alarm for 6:00 am" or "set timer for 20:00". The computing device 106 uses the grammar to parse spoken or typed commands and identify actions that the computing device 106 should perform. Thus, the grammar provides the capability data that causes the computing device 106 to operate in a particular manner to parse the commands.

[0021] Each grammar may be active in a particular context. For example, an alarm grammar may be active if the application running in the foreground of the computing device 106 is an alarm application. Furthermore, an alarm grammar may be active if the application running in the foreground of the computing device is a digital assistant application. Because the likelihood that a user will attempt to set an alarm in an alarm application may be higher than in a digital assistant application, the computing device 106 may assign a probability of 0.95 to the likelihood of an alarm grammar matching the command and the user's intent if the computing device 106 is running an alarm application, and a probability of 0.03 if the computing device 106 is running a digital assistant application.

[0022] In the example shown in FIG. 1, the computing device 106 determines a confidence score for each grammar based on different candidate transcriptions that may be applied to the grammar. In other words, the computing device 106 identifies a grammar to parse each of the candidate transcriptions. The applied grammars 124 indicate grammars to parse the candidate transcription "lights out", which indicates a grammar confidence score for each grammar that is correct for the user's intent. A higher grammar confidence score indicates that the grammar is more likely to be correct based on the given transcription and context. If the transcription is "lights out", the grammar confidence score 126 for the home automation grammar is 0.35. If the transcription is "lights out", the grammar confidence score 128 for the movie command grammar is 0.25. If the transcription is "lights out", the grammar confidence score 130 for the music command grammar is 0.20. The sum of the grammar confidence scores for each transcription should be 1.0. The computing device 106 may then assign the remaining grammar confidence score to the default intent grammar. In this example, if the transcription is "lights out," the default intent grammar has a grammar confidence score 132 of 0.20. The default intent grammar may not be limited to a particular action or intent and is capable of parsing all or nearly all transcriptions.

[0023] In some implementations, none of the grammars can parse a particular transcription. In this case, the default intent grammar is applied. For example, there may be no grammar to parse "flights out." Thus, if the transcription is "flights out," the default intent grammar has a grammar confidence score 132 of 1.0.

[0024] In some implementations, the computing device 106 selects a grammar and a transcription based on calculating the product of the transcription confidence score and the grammar confidence score. Box 134 shows the result of selecting a transcription and a grammar based on the product of the transcription confidence score and the grammar confidence score. For example, the combined confidence score 136 of "lights out" and the home automation grammar is 0.2275. The combined confidence score 138 of "lights out" and the movie command grammar is 0.1625. The combined confidence score 140 of "lights out" and the music command grammar is 0.130. The combined confidence score 142 of "lights out" and the default intent grammar is 0.130. The combined confidence score 144 of "flights out" and the default intent grammar is 0.350. In this implementation, the combined confidence score 144 is the highest score. Thus, using this approach, the computing device 106 may perform an action corresponding to the command "flights out," which is likely an invalid result for the user 102 and may result in providing the transcription "flights out" to a search engine.

[0025] If it is not the user's 102 intention to search the Internet for "flights out," the user would need to repeat the utterance 104. Doing so would require the computing device 106 to expend additional computing and power resources to process the additional utterance. The user 102 would manually input the desired command into the computing device 102, which may use additional processing and power resources, by activating the screen of the computing device 102 for an additional period of time.

[0026] To reduce the possibility that the computing device 106 performs an action that is inconsistent with the user's intention, the computing device 106 may normalize the grammar confidence scores of the applied grammar 124. To normalize the grammar confidence scores of the applied grammar 124, the computing device 106 may calculate the coefficient required to increase the highest grammar confidence score to 1.0. In other words, the product of the coefficient and the highest grammar confidence score should be 1.0. The computing device 106 may then multiply the other grammar confidence scores by the coefficient to normalize the other grammar confidence scores. The normalization performed by the computing device 106 may differ from traditional probability normalization, in which probabilities are adjusted to sum to 1. Since the highest grammar confidence score is increased to 1.0, the normalized probabilities do not sum to 1. The increased confidence scores may represent pseudo-probabilities rather than probabilities in the traditional sense. The normalization processes described elsewhere in this application may also produce similar pseudo-probabilities.

[0027] As shown in normalized grammar confidence score 146, the computing device 106 may identify the grammar confidence score of the home automation grammar as the highest grammar confidence score. To increase grammar confidence score 126 to 1.0, the computing device multiplies grammar confidence score 126 by 1.0 / 0.35=2.857. The computing device 106 calculates normalized grammar confidence score 150 by multiplying grammar confidence score 128 by 2.857. The computing device 106 calculates normalized grammar confidence score 152 by multiplying grammar confidence score 130 by 2.857. The computing device 106 calculates normalized grammar confidence score 154 by multiplying grammar confidence score 132 by 2.857. The grammar confidence score of the default intent grammar when the transcription is "flights out" is 1.0 because there is no other grammar that parses the transcription "flights out". Therefore, there is no need to normalize the grammar confidence score of the default intent grammar when the transcription is "flights out" since the score is already 1.0.

[0028] As shown in box 134, instead of multiplying the grammar confidence scores of box 124 by the transcription confidence scores of the word lattice 112, the computing device 106 uses the transcription confidence scores from the word lattice 112 and the normalized grammar confidence scores to calculate a combined confidence score. In particular, the computing device 106 multiplies each of the normalized grammar confidence scores by the transcription confidence score of the respective transcription. If only the default intent grammar is applied to the grammar, the transcription confidence scores are multiplied by 1.0, so that the corresponding transcription confidence scores do not change.

[0029] As shown in box 156, the computing device 106 calculates a combined confidence score 158 by multiplying the transcription confidence score of "lights out" by the normalized grammar confidence score 148 to obtain a result of 0.650. The computing device 106 calculates a combined confidence score 160 by multiplying the transcription confidence score of "lights out" by the normalized grammar confidence score 150 to obtain a result of 0.464. The computing device 106 calculates a combined confidence score 162 by multiplying the transcription confidence score of "lights out" by the normalized grammar confidence score 152 to obtain a result of 0.371. The computing device 106 calculates a combined confidence score 164 by multiplying the transcription confidence score of "lights out" by the normalized grammar confidence score 154 to obtain a result of 0.371. The computing device 106 maintains the transcription confidence score of "flights out" at 0.350 because it does not have a grammar to parse the transcription "flights out."

[0030] In some implementations, the computing device 106 can adjust the confidence score in box 156 to accommodate more likely grammars given the current user context of the current context of the computing device 106. For example, the user 102 may be listening to music using the computing device 106. The computing device 106 may be a media device that plays songs. In this case, the computing device 106 can adjust the confidence score 162. The computing device 106 can increase the confidence score by multiplying the confidence score with a factor, by assigning a preset value to the confidence score, or by another technique. For example, the current probability of a music command is 0.9 based on the context of the user 102 listening to music via the computing device 102, which is a media device. The probability of a music command in box 124 can be the confidence score 130, which is 0.20. In this case, the computing device 106 can multiply the confidence score 162 by a ratio of 0.9 / 0.2=4.5. The resulting confidence score 162 is 0.371*4.5=1.67. The other confidence scores in box 156 can be adjusted by a similar ratio for each confidence score. For example, the current probability of a home automation command may be 0.04. In this case, the ratio would be 0.04 / 0.35=0.16, which is 0.04 divided by the confidence score 126. The computing device 106 can multiply the confidence score 158 by 0.16 to calculate an adjusted, or biased, confidence score of 0.10. In this case, the highest confidence score would be the one corresponding to the music command.

[0031] In some implementations, this additional adjustment step may affect the candidate transcription that the computing device 106 selects. For example, if the candidate transcription "flights out" is the name of a video game and the user is expected to launch the video game, the computing device 106 may adjust the confidence score 166 to 0.8 based on a ratio similar to that calculated above and using the probability that the user will launch the video game, which is based on the current context of the computing device 106 and / or the user 102.

[0032] In some implementations, such as when multiple grammars parse the same candidate transcription, the computing device 106 can improve detection of user intent and speech recognition without using this rescoring step. This rescoring step may allow the computing device 106 to select a different candidate transcription that is not likely based on the speech recognition confidence score.

[0033] In some implementations, the computing device 106 may use this rescoring step without considering the confidence scores of box 124. For example, the computing device may apply the rescoring step to the transcription confidence scores determined from the word lattice 112. In this case, the computing device 106 would not perform the adjustments shown in both box 146 and box 156.

[0034] The computing device 106 selects the grammar and transcription with the highest combined confidence score. For example, a home automation transcription and the transcription "lights out" may have a highest combined confidence score of 0.650. In this case, the computing device 106 executes "lights out" based on the home automation grammar. The computing device 106 can turn off the lights in the home in which the computing device 106 is located or in the home of the user 102. If the computing device 106 uses the movie command grammar, the computing device 106 can play the movie "Lights Out". If the computing device 106 uses the music command grammar, the computing device 106 can play the song "Lights Out".

[0035] 1, the display of the computing device 106 may indicate actions to be performed by the computing device 106. First, the computing device 106 may display a prompt 168 for the digital assistant. The computing device 106 receives the utterance 104 and indicates that the computing device 106 is turning off the lights by displaying a prompt 170.

[0036] 2 illustrates components of an example system 200 for selecting a grammar to apply to an utterance based on a context. System 200 may be any type of computing device configured to receive and process speech audio. For example, system 200 may be similar to computing device 106 of FIG. 1. Components of system 200 may be implemented on a single computing device or distributed across multiple computing devices. System 200 implemented on a single computing device may be beneficial for privacy reasons.

[0037] The system 200 includes an audio subsystem 205. The audio subsystem 205 may include a microphone 210, an analog-to-digital converter 215, a buffer 220, and various other audio filters. The microphone 210 may be configured to detect sounds in the surrounding area, such as a voice. The analog-to-digital converter 215 may be configured to sample audio data detected by the microphone 210. The buffer 220 may store the sampled audio data for processing by the system 200. In some implementations, the audio subsystem 205 may be continuously active. In this case, the microphone 210 may be constantly detecting sounds. The analog-to-digital converter 215 may constantly sample the detected audio data. The buffer 220 may store the most recent sampled audio data, such as the last 10 seconds of a sound. If other components of the system 200 do not process the audio data in the buffer 220, the buffer 220 may overwrite the previous audio data.

[0038] The audio subsystem 205 provides the processed audio data to the speech recognizer 225. The speech recognizer provides the speech data as input to the acoustic model 230. The acoustic model 230 may be trained to identify possible phonemes that correspond to sounds in the audio data. For example, if a user says "set," the acoustic model 230 may identify phonemes that correspond to an "s" sound, an "e" vowel, and a "t" sound. The speech recognizer 225 provides the identified phonemes as input to the language model 235. The language model 235 generates a word lattice. The word lattice includes a term confidence score for each of the candidate terms identified by the language model 235. For example, the word lattice may indicate that the first term is likely to be "set." The language model 235 may not identify any other possible terms for the first term. The language model 235 may identify two possible terms for the second term. For example, the word lattice may include the terms "time" and "chime" as possible second terms. The language model 235 may assign a term confidence score to each term. "Time" may have a term confidence score of 0.75, and "chime" may have a term confidence score of 0.25.

[0039] The speech recognizer 225 can generate candidate transcriptions based on the word lattice. Each candidate transcription can have a transcription confidence score that reflects the likelihood that the speaker spoke the terms in the transcription. For example, a candidate transcription can be "set time" and have a transcription confidence score of 0.75. Another candidate transcription can be "set chime" and have a transcription confidence score of 0.25.

[0040] While the speech recognizer 225 is generating the word lattice, candidate transcriptions, and transcription confidence scores, the context determiner 240 may collect contextual data indicative of the current context of the system 200. The context determiner 240 may collect sensor data 245 from any sensors in the system. The sensors may include location sensors, thermometers, accelerometers, gyroscopes, gravity sensors, time and day of the week, and other similar sensors. The sensor data 245 may also include data related to the status of the system 200. For example, the status may include the battery level of the system 200, signal strength, nearby devices with which the system 200 may be in communication or aware, and other similar conditions of the system 200.

[0041] The context identifier 240 may also collect system process data 250 indicative of processes being executed by the system 200. The process data 250 may indicate memory allocated to each process, process resources allocated to each process, applications being executed by the system 200, applications running in the foreground or background of the system 200, the contents of interfaces on the system's display, and similar process data.

[0042] As an example, the context identifier 240 can receive system process data 250 and sensor data 245 that indicate that the system 200 is a user's home, the time is 6:00 p.m., the day of the week is Monday, the foreground application is a digital assistant application, and the device is a tablet.

[0043] The grammar score generator 255 receives the context of the system 200 from the context identifier 240 and the word lattice from the speech recognizer 225. The grammar score generator 255 identifies a grammar 260 that parses each of the candidate transcriptions of the word lattice. In some cases, none of the grammars 260 parses a candidate transcription. In this case, the grammar score generator 255 assigns a grammar confidence score of 1.0 of the default intent grammar to any candidate transcription that cannot be parsed by any of the grammars 260.

[0044] In some examples, a single grammar 260 may parse the candidate transcription. In this case, the grammar score generator 255 may determine a grammar confidence score for the single grammar, assuming that the candidate transcription is an actual transcription. Because the grammar confidence score represents a probability, the grammar confidence score may be less than 1.0. The grammar score generator 255 assigns the difference between 1.0 and the grammar confidence score to the grammar confidence score of the default intent grammar of the candidate transcription.

[0045] In some examples, multiple grammars 260 may parse the candidate transcription. In this case, the grammar score generator 255 may determine a grammar confidence score for each of the multiple grammars, assuming that the candidate transcription is the actual transcription. Because the grammar confidence score represents a collection of probabilities, the sum of the grammar confidence scores may be less than 1.0. The grammar score generator 255 assigns the difference between 1.0 and the sum of the grammar confidence scores to the grammar confidence score of the default intent grammar of the candidate transcription.

[0046] The grammar score normalizer 265 receives the grammar confidence scores for each of the candidate transcriptions and normalizes the grammar confidence scores. In some implementations, the grammar score normalizer 265 normalizes only the grammar confidence scores of grammars other than the default intent grammar. If the grammar score generator 255 generates a grammar confidence score for a grammar for a particular candidate transcription, the grammar score normalizer 265 increases the grammar confidence score to 1.0. If the grammar score generator 255 does not generate grammar confidence scores for multiple grammars for a particular candidate transcription, the grammar score normalizer 265 maintains the grammar confidence score of the default intent grammar at 1.0.

[0047] If the grammar score generator 255 generates multiple grammar confidence scores for each of multiple grammars for a particular candidate transcription, the grammar score normalizer 265 identifies the highest grammar confidence score for the particular candidate transcription. The grammar score normalizer 265 calculates a coefficient to increase the highest grammar confidence score to 1.0 such that the product of the coefficient and the highest grammar confidence score is 1.0. The grammar score normalizer 265 increases the other grammar confidence scores for the same particular transcription by multiplying each of the other grammar confidence scores by the coefficient.

[0048] The grammar and transcription selector 270 receives the normalized grammar confidence scores and the transcription confidence scores. Using both the grammar confidence scores and the transcription confidence scores, the grammar and transcription selector 270 identifies grammars and transcriptions that are likely to closely match the received utterance and the speaker's intent by calculating a combined confidence score. The grammar and transcription selector 270 determines each combined confidence score by calculating the product of each normalized grammar confidence score and the transcription confidence score of the corresponding candidate transcription. If the default intention grammar is the only grammar for a particular candidate transcription, the grammar and transcription selector 270 keeps the transcription confidence score as the combined confidence score. The grammar and transcription selector 270 selects the grammar and candidate transcription with the highest combined confidence score.

[0049] The action specifier 275 receives the selected grammar and the selected transcription from the grammar and transcription selector 270 and specifies an action for the system 200 to perform. The selected grammar may indicate a type of action, such as setting an alarm, sending a message, playing a song, calling a person, or other similar action. The selected transcription may indicate details of the type of action, such as a time to set an alarm, who to send the message to, a song to play, who to call, or other similar details of the action. The grammar 260 may include information about the type of action of the particular grammar. The grammar 260 may include information about how the action specifier 275 should parse the candidate transcription to determine details of the type of action. For example, the grammar may be a $ALARM grammar, and the action specifier 275 parses the selected transcription to determine to set a timer for 20 minutes. The action specifier 275 may perform the action or provide instructions to another part of the system 200, such as a processor.

[0050] In some implementations, the user interface generator 280 displays an indication of the action or an indication of the execution of the action, or both. For example, the user interface generator 280 can display a timer counting down from 20 minutes or a confirmation of turning off the lights in the house. In some examples, the user interface generator 280 may not provide an indication of the action that was executed. For example, the action may be adjusting a thermostat. The system 200 may adjust the thermostat without generating a user interface for display on the system 200. In some implementations, the user interface generator 280 can generate an interface for the user to interact with or confirm the action. For example, the action may be to call mom. The user interface generator 280 can generate a user interface for the user to confirm the action of calling mom before the system 200 executes the action.

[0051] 3 is a flow chart illustrating an example process 300 for selecting a grammar based on a context. In general, the process 300 performs speech recognition on the audio and identifies an action to perform based on a transcription of the audio and a grammar that parses the audio. The process 300 normalizes the grammar confidence score to identify an action that is likely intended by the speaker. The process 300 is described as being performed by a computer system including one or more computers, such as the computing device 106 of FIG. 1 or the system 200 of FIG. 2.

[0052] The system receives 310 audio data of an utterance. For example, a user may speak an utterance that sounds like "lights out" or "flights out." The system detects the utterance via a microphone or receives the audio data of the utterance. The system can process the audio data using an audio subsystem.

[0053] The system uses the acoustic model and the language model to generate a word lattice that includes a plurality of candidate transcriptions of the utterance and includes a plurality of transcription confidence scores, each of the plurality of transcription confidence scores reflecting the likelihood that each of the candidate transcriptions matches the utterance (320). The system generates the word lattice using automatic speech recognition. The automatic speech recognition process may include providing audio data as input to an acoustic model that identifies various phonemes that match each portion of the audio data. The automatic speech recognition process may include providing the phonemes as input to a language model that generates a word lattice that includes a confidence score for each candidate word in the utterance. The language model selects words for the word lattice from a vocabulary. The vocabulary may include words of a language that the system is configured to recognize. For example, the system may be configured for English and the vocabulary may include English words. In some implementations, the actions of the process 300 do not include limiting the words in the vocabulary that the system can recognize. In other words, the process 300 can generate the transcription using any words of the language that the system is configured to recognize. Because the system can recognize every word in a language, the system's speech recognition process can function when the user says something unexpected, as long as it is within the language the system is configured to recognize.

[0054] The system can use the word lattice to generate a variety of candidate transcriptions. Each candidate transcription may include a different transcription confidence score that reflects the likelihood that the user spoke the transcription. For example, the candidate transcription "lights out" may have a confidence score of 0.65, and the candidate transcription "flights out" may have a confidence score of 0.35.

[0055] The system determines a context for the system (330). In some implementations, the context is based on the location of the system, applications running in the foreground of the computing device, user demographic information, data stored on or accessible to the system such as contact data, previous user queries or commands, time, date, day of the week, weather, system orientation, and other similar types of information.

[0056] The system identifies 340 multiple grammars that correspond to multiple candidate transcriptions based on the system's context. The grammars may include various structures for different commands that the system can perform. For example, the grammars may include command structures for setting an alarm, playing a movie, performing an Internet search, checking the user's calendar, or other similar actions.

[0057] The system determines (350) for each of a plurality of candidate transcriptions a plurality of grammar confidence scores reflecting the likelihood that the respective grammar matches the respective candidate transcription based on the current context. The grammar confidence score may be thought of as the conditional probability that the grammar matches the speaker's intent based on the system's context, given that one of the candidate transcriptions is a transcription of the utterance. For example, a home automation grammar with a transcription of "lights out" may have a grammar confidence score of 0.35. In other words, if the transcription is "lights out," the grammar confidence score may be 0.35. The conditional probability of the home automation grammar as "out" is 0.35. The system may generate a grammar confidence score for each grammar that parses the candidate transcription. In some implementations, due to context, the system may generate grammar confidence scores for only some of the grammars that parse the candidate transcription. The system may assign the remaining probabilities to the default intent grammar so that the sum of the grammar confidence scores for each candidate transcription is 1.0.

[0058] The system selects (360) a candidate transcription from among the multiple candidate transcriptions based on the transcription confidence score and the grammar confidence score. In some implementations, the system normalizes the grammar confidence scores by increasing the number of candidate transcriptions that match multiple grammars. For each candidate transcription, the system can multiply the highest grammar confidence score by a factor required to normalize the grammar confidence score to 1.0. The system can multiply other grammar confidence scores for the same candidate transcription by the same factor. The system can multiply the normalized grammar confidence score by the transcription confidence score to generate a combined confidence score. The system selects the grammar and transcription with the highest combined confidence score.

[0059] The system provides the selected candidate transcription as a transcription of the utterance for output (370). In some implementations, the system performs an action based on the grammar and the candidate transcription. The grammar may indicate an action to take. The action may include playing a movie, calling a contact, turning on a light, sending a message, or other similar types of actions. The selected transcription may include the contact to call, the message to send, the recipient of the message, the name of the movie, or other similar details.

[0060] In some implementations, the system uses a process of tagging the word lattice for a given grammar set using a finite state transducer (FST). In some implementations, the system may constrain the grammar as the union of all grammars that match the current context. In some implementations, the system may match offline in advance or dynamically at runtime.

[0061] In some implementations, the process 300 can weight the edges of the word lattice. The system can compile the grammar into a weighted finite-state transducer that constrains the grammar. The weights of the arcs, i.e., the weights of the edges of this finite-state transducer, may encode the amount of probability of a word in a given grammar, which may be negative logarithmic weights. In some implementations, all of the nearly all grammars associated with a word lattice may be integrated together.

[0062] The process 300 can continue with the system determining a context-dependent probability for each grammar, which can be the probability of the grammar given the context. The system can determine the context-dependent probability by the provided components and corresponding weights, which can be encoded into arcs or edges in the grammar constraints, such as the opening decorator arc.

[0063] The process 300 may continue with the system clearing the weights of the word lattice. The system may use grammar constraints to construct the word lattice, which may mark spans that match the grammar. For example, a span that matches the grammar may be<media_commands> Play song< / media_commands> etc. (or a transcription of "play song").

[0064] The process 300 may continue to normalize the probabilities. The system generates a copy of the tagged lattice with the decorator tags removed. The system determines and minimizes the arc costs of the tropical semirings, resulting in each unique word path, where the word path encodes the highest probability through the word path. The probabilities may be the probability of the word path given the grammar multiplied by the probability of the grammar. The probabilities may be inverted by inverting the signs of the arc weights with negative logarithmic weights, and this inverted lattice may be constructed with the tagged lattice. Constructing with an inverted lattice may be effectively the same as performing a division by the weights of the lattice (in this case, for each word sequence or path, the maximum of the word path probabilities given the grammar multiplied by the probability of the grammar). Thus, the system may divide the probability of the optimal tagged path by itself, each of which may be 1.0. In some implementations, non-optimal paths receive lower pseudo-probabilities, generating a lattice that contains a desired amount of pseudo-probability for the grammar of a given word path.

[0065] In some implementations, the process 300 may include discarding grammars with low conditional probabilities by using beam pruning. For example, there are too many grammars that match the utterance "lights out" (where the probabilities are 0.32 for home automation, 0.25 for movie commands, 0.20 for music commands, 0.10 for general search, 0.03 for online business, 0.03 for social groups, and 0.02 for online encyclopedia search). The system can reduce the processing burden by removing grammars that are below a threshold or below a threshold percentage of the most likely interpretation or grammar. For example, the system may remove grammars that are less than one-tenth as likely as the most likely grammar. The system may then remove grammars with a conditional probability of less than 0.032 for a given transcription. The rationale behind pruning is that the probability of low-scoring tagging is so low that there is no rationale for it to be selected over more likely interpretations, even with biasing. In pruning, the system may need to add a finite state transducer that operates on the resulting lattice with the desired pruning weights, which specify which lags the tagging has the best hypothesis to prune.

[0066] Advantages of process 300 include: it allows the system to bias the grammar directly on the lattice; it has lower complexity than other word lattice tagging algorithms; it resolves the probability fragmentation caused by tagging ambiguity by implementing maximal normalization; and it allows the system to discard less likely proposed taggings by using beam pruning.

[0067] In some implementations, the process 300 may extract the n best hypotheses from the lattice, tag all hypotheses and possibilities individually, and optionally recombine them into the lattice at the end. The system may assume that the taggers are otherwise identical. This may yield the same results with higher latency and / or lower recall. If the n best hypotheses after the top N (e.g., N=100) are removed, the recall may be lower. This may help manage latency. The number of distinct sentences in a lattice may be exponential with respect to the number of words in it, so some workarounds may be exponentially slower in the worst case unless the best N limit mentioned above (which may limit the number of options examined) is used.

[0068] 4 illustrates examples of a computing device 400 and a mobile computing device 450 that may be used to implement the techniques described herein. Computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, component connections and relationships, and component functions illustrated herein are intended to be exemplary only and not limiting.

[0069] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 that connects to the memory 404 and multiple high-speed expansion ports 410, and a low-speed interface 412 that connects to the low-speed expansion port 414 and the storage device 406. Each of the processor 402, memory 404, storage device 406, high-speed interface 408, high-speed expansion port 410, and low-speed interface 412 are interconnected via various buses and may be implemented on a common motherboard or in other manners as needed. The processor 402 is capable of processing instructions for execution within the computing device 400, including instructions stored in the memory 404 or storage device 406, to display graphical information of a GUI on an external input / output device, such as a display 416 connected to the high-speed interface 408. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0070] The memory 404 stores information within the computing device 400. In some implementations, the memory 404 is a volatile memory unit or units. In some implementations, the memory 404 is a non-volatile memory unit or units. The memory 404 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0071] The storage device 406 can provide mass storage for the computing device 400. In some implementations, the storage device 406 may be or include a computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory, or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The instructions may be stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., the processor 402), perform one or more methods, such as those described above. The instructions may be stored in one or more storage devices, such as a computer-readable medium or a machine-readable medium (e.g., the memory 404, the storage device 406, or a memory on the processor 402).

[0072] The high-speed interface 408 manages bandwidth-intensive operations of the computing device 400, while the low-speed interface 412 manages less bandwidth-intensive operations. Such an allocation of functions is merely an example. In some implementations, the high-speed interface 408 is connected to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 410 that may receive various expansion cards (not shown). In an implementation, the low-speed interface 412 is connected to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be connected to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, through a network adapter.

[0073] Computing device 400 may be implemented in many different forms, as shown. For example, it may be implemented as a standard server 420 or multiple times in a group of such servers. It may also be implemented in a personal computer, such as a laptop computer 422. It may also be implemented as part of a rack server system 424. Alternatively, components from computing device 400 may be combined with other components in a mobile device (not shown), such as mobile computing device 450. Each such device may include one or more of computing devices 400 and mobile computing devices 450, and the entire system may be made up of multiple computing devices in communication with each other.

[0074] The mobile computing device 450 includes, among other components, a processor 452, memory 464, input / output devices such as a display 454, a communication interface 466, and a transceiver 468. The mobile computing device 450 may also include a storage device such as a microdrive or other device to provide additional storage. The processor 452, memory 464, display 454, communication interface 466, and transceiver 468 are each interconnected via various buses, and multiple components may be implemented on a common motherboard or in other manners as desired.

[0075] The processor 452 can execute instructions within the mobile computing device 450, including instructions stored in the memory 464. The processor 452 may be implemented as a chipset of chips including separate analog and digital processors. The processor 452 may provide coordination of other components of the mobile computing device 450, such as, for example, controlling the user interface, applications run by the mobile computing device 450, and wireless communications by the mobile computing device 450.

[0076] The processor 452 may communicate with a user through a control interface 458 and a display interface 456 connected to a display 454. The display 454 may be, for example, a TFT (thin film transistor liquid crystal display) display, an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 456 may include appropriate circuitry for driving the display 454 to present graphics and other information to the user. The control interface 458 may receive commands from the user and translate the commands to provide to the processor 452. Additionally, an external interface 462 may provide communication with the processor 452 to enable near-field communication between the mobile computing device 450 and other devices. The external interface 462 may provide, for example, wired communication in some implementations and wireless communication in other implementations, and multiple interfaces may be used.

[0077] The memory 464 stores information within the computing device 450. The memory 464 may be implemented as one or more of a computer readable medium, a volatile memory unit, or a non-volatile memory unit. An expansion memory 474 may be provided and connected to the mobile computing device 450 via an expansion interface 472. The expansion interface 472 may include, for example, a SIMM (single in-line memory module) card interface. The expansion memory 474 may provide extra storage space for the mobile computing device 450 or may store applications or other information for the mobile computing device 450. In particular, the expansion memory 474 may include instructions that perform or complement the processes described above and may include security information. Thus, for example, the expansion memory 474 may be provided as a security module for the mobile computing device 450 and may be programmed with instructions that enable secure use of the mobile computing device 450. Additionally, a security application may be provided via a SIMM card, along with additional information such as placing identifying information on the SIMM card in an unhackable manner.

[0078] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, the instructions are stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 452), perform one or more methods, as described above. The instructions may be stored in one or more storage devices, such as one or more computer-readable media or one or more machine-readable media (e.g., memory 464, expansion memory 474, or memory on processor 452). In some implementations, the instructions may be received in a propagated signal, for example through transceiver 468 or external interface 462.

[0079] Mobile computing device 450 may communicate wirelessly through communication interface 466, which may include digital signal processing circuitry as necessary. Communication interface 466 may provide communications under various modes or protocols, such as GSM voice (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communications may occur, for example, through transceiver 468 using radio frequencies. Additionally, short-range communications may occur, such as through the use of Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation- and location-related radio data to the mobile computing device 450 that may be used as appropriate by applications executing on the mobile computing device 450.

[0080] Mobile computing device 450 may communicate audibly using audio codec 460, which may receive verbal information from a user and convert it into usable digital information. Audio codec 460 may also generate audible sounds for the user, such as through a speaker in a handset of mobile computing device 450. Such sounds may include the sounds of a voice call, may include recorded sounds (e.g., voice messages, music files, etc.), and may include sounds generated by applications running on mobile computing device 450.

[0081] The mobile computing device 450 may be implemented in many different forms, as shown in the figure. For example, the mobile computing device 450 may be implemented as a mobile phone 480. The mobile computing device 450 may also be implemented as part of a smartphone 482, a personal digital assistant, or other similar mobile device.

[0082] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or translatable on a programmable system having one or more special-purpose or general-purpose programmable processors coupled to transmit data and instructions to and receive data and instructions from a memory device, one or more input devices, and one or more output devices.

[0083] These computer programs (also known as programs, software, software applications or codes) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic disk, optical disk, memory, PLD (Programmable Logic Device)) including a machine-readable medium that receives machine instructions as a machine-readable signal that is used to provide machine instructions and / or data to a programmable processor. The term machine-readable signal refers to a signal that is used to provide machine instructions and / or data to a programmable processor.

[0084] To provide for user interaction, the systems and techniques described herein may be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be captured in any form, including acoustic, speech, or tactile input.

[0085] The systems and techniques described herein may be implemented in a computer system with a back-end component (e.g., a data server), a middleware component (e.g., an application server), a front-end component (e.g., a client computer having a graphical interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN (local area network), a WAN (wide area network), and the Internet. In some implementations, the systems and techniques described herein may be implemented on embedded systems where speech recognition and other processing is performed directly on the device.

[0086] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0087] Although several implementations have been described in detail, other variations are possible. For example, while a client application has been described as accessing a delegate, in other implementations the delegate may be used by other applications implemented by one or more processors, such as applications running on one or more servers. Additionally, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve desirable results. Additionally, other operations may be provided in or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method which, when executed by data processing hardware, receiving audio data of an utterance performed by a user and captured by a computing device associated with said user; determining a plurality of phonemes from the audio data corresponding to the utterance; generating one or more candidate transcriptions for the utterance using the plurality of phonemes corresponding to the utterance, each of the one or more candidate transcriptions having a corresponding transcription confidence score; determining a context indicative of a user listening to music through the computing device; selecting a grammar corresponding to a particular user intent to issue a media playback command based on the context indicating that the user is listening to music through the computing device; parsing a candidate transcription of the one or more candidate transcriptions having a highest transcription confidence score using the selected grammar that corresponds to the particular user's intent to issue the media playback command to identify an action to be performed by the computing device; and causing the data processing hardware to perform operations including:

2. The operations further include instructing the computing device to perform the identified action. The method of claim 1.

3. The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device. The method of claim 1.

4. each of said plurality of grammars includes a different specified structure of a plurality of terms; The method according to claim 3.

5. The computing device comprises a mobile phone. The method of claim 1.

6. The computing device comprises a wearable device. The method of claim 1.

7. each of the one or more candidate transcripts comprises a plurality of terms; The method of claim 1.

8. each of the plurality of terms includes a corresponding term confidence score; The method of claim 7.

9. the transcription confidence score indicating the likelihood that the candidate transcription matches the utterance made by the user. The method of claim 1.

10. the data processing hardware is located on the computing device; The method of claim 1.

11. 1. A system comprising: Data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, receiving audio data of an utterance performed by a user and captured by a computing device associated with said user; determining a plurality of phonemes from the audio data corresponding to the utterance; generating one or more candidate transcriptions for the utterance using the plurality of phonemes corresponding to the utterance, each of the one or more candidate transcriptions having a corresponding transcription confidence score; determining a context indicative of a user listening to music through the computing device; selecting a grammar corresponding to a particular user intent to issue a media playback command based on the context indicating that the user is listening to music through the computing device; parsing a candidate transcription of the one or more candidate transcriptions having a highest transcription confidence score using the selected grammar that corresponds to the particular user's intent to issue the media playback command to identify an action to be performed by the computing device; The system causes the data processing hardware to perform operations including:

12. The operations further include instructing the computing device to perform the identified action. The system of claim 11.

13. The operations further include selecting the grammar from a plurality of grammars based on the context of the computing device. The system of claim 11.

14. each of said plurality of grammars includes a different named structure of a plurality of terms; The system of claim 13.

15. The computing device comprises a mobile phone or a wearable device. The system of claim 11.

16. The grammar includes a default grammar. The system of claim 11.

17. each of the one or more candidate transcripts comprises a plurality of terms; The system of claim 11.

18. each of the plurality of terms includes a corresponding term confidence score; 20. The system of claim 17.

19. the transcription confidence score indicating the likelihood that the candidate transcription matches the utterance made by the user. The system of claim 11.

20. the data processing hardware is located on the computing device; The system of claim 11.

Citation Information

Patent Citations

  • Speech recognizing device

    JP1994102896A

  • Developer voice actions system

    WO2017151215A1