Hot word identification and passive assistance

Abstract

Methods, systems, and apparatus for implementing hotword recognition and passive assistance are disclosed, including computer programs encoded on computer storage media. In one aspect, a method includes an action of receiving audio data corresponding to an utterance by a computing device, the computing device being operated in a low power mode and including a display that displays a graphical interface when the computing device is in the low power mode and is configured to exit the low power mode in response to detecting a first hotword. The method further includes determining that the audio data includes a second different hotword. The method further includes obtaining a transcription of the utterance by performing speech recognition on the audio data. The method further includes generating an additional user interface. The method further includes providing an additional graphical interface for output on the display.

Description

Technical Field

[0001] This specification generally relates to automatic speech recognition. Background Art

[0002] The reality of a voice-enabled home or other environment, that is, an environment where a user only needs to speak a query or command out loud and a computer-based system will respond (field) and answer the query and / or cause the command to be executed, has arrived. A voice-enabled environment (e.g., home, workplace, school, etc.) can be implemented using a network of connected microphone devices distributed in various rooms or areas of the environment. Through such a microphone network, a user has the ability to verbally query the system from essentially anywhere in the environment without having a computer or other device in front of him / her or even nearby. For example, when cooking in the kitchen, a user may directly ask the system "how many milliliters in three cups?" and, in response, receive an answer from the system, such as in the form of a synthesized voice output. Alternatively, a user may ask the system questions such as "when does my nearest gas station close?" or, when preparing to leave the house, "should I wear a coat today? "

[0003] In addition, the user can ask the system queries and / or issue commands related to the user's personal information. For example, the user may ask the system "when is my meeting with John?" or command the system "remind me to call John when I get back home." Summary of the invention

[0004] For voice-enabled systems, the way users interact with the system is designed to be primarily, but not exclusively, by means of voice input. Therefore, a system that potentially acquires all utterances made in the surrounding environment (including those not directed to the system) must have a way to identify when any given utterance is directed to the system rather than, for example, to an individual present in the environment. A way to achieve this is to use a hot word, which is reserved as a predetermined word by the user's consent in the environment, and the predetermined word is spoken to attract the attention of the system. In the example environment, the hot word used to attract the attention of the system is the word "OK computer". Therefore, whenever the word "OK computer" is spoken, it is acquired by the microphone and communicated to the system, which can perform speech recognition technology or use audio features and neural networks to determine whether the hot word has been spoken, and if so, wait for subsequent commands or queries. Therefore, the utterance directed to the system adopts the general format [hot word] [query], where the "hot word" in this example is "OK computer" and the "query" can be any question, command, statement, or other request that can be recognized, parsed and acted on by the system alone or in conjunction with the server voice via the network.

[0005] Voice-enabled systems can be configured to respond to more than one hotword. In the case where some voice-enabled systems include an always-on display, the system can provide passive assistance to the user in response to detecting some additional hotwords. The system can display information related to the detected hotwords and any speech that follows on the always-on display in an unobtrusive manner. For example, a user may be having a conversation with a friend about lunch plans. The user's phone may be resting on a table, and a microphone may be able to detect the conversation. During the conversation, a friend may ask the user "Are you free for lunch tomorrow?" The user's phone can detect the question and determine whether the question includes a hotword. In addition to "OK computer", the phone can also be configured to detect hotwords such as "are you free" and "don't forget". In response to receiving audio data of "are you free for lunch tomorrow", the phone recognizes the hotword "are you free" and performs voice recognition on the remaining audio data. Based on recognizing the hot word "are you free" and generating the transcription "for lunch tomorrow", the phone automatically accesses the user's calendar information and displays the relevant calendar information for lunch time the next day on the always-on display. The user is able to look at the phone to determine the user's own availability and quickly answer whether the user is free without having to interact with the phone through any buttons or displays or saying "OK computer" to the phone.

[0006] According to one innovative aspect of the subject matter described in the present application, a method for implementing hotword recognition and passive assistance includes the following actions: receiving audio data corresponding to an utterance by a computing device, the computing device (i) being operated in a low power mode and including a display that displays a graphical interface when the computing device is in the low power mode and (ii) being configured to exit the low power mode in response to detecting a first hot word; determining, by the computing device, that the audio data includes a second, different hot word; obtaining, by the computing device, a transcription of the utterance by performing speech recognition on the audio data in response to determining that the audio data includes the second, different hot word; generating, by the computing device, an additional user interface based on the second, different hot word and the transcription of the utterance; and providing, while the computing device remains in the low power mode, an additional graphical interface for output on the display.

[0007] These and other embodiments can each optionally include one or more of the following features. These actions include: receiving input including key presses by the computing device after providing an additional graphical interface for output on the display; and, after receiving the input including key presses, switching the computing device to a high power mode that consumes more power than the low power mode. These actions include: after switching the computing device to a high power mode that consumes more power than the low power mode and while the display remains active, returning the computing device to the low power mode; and, after returning the computing device to the low power mode, providing a user interface for output on the display. When in high power mode, the computing device fetches data from a network at a first frequency. When in low power mode, the computing device fetches data from the network at a second, lower frequency. The display is a touch-sensitive display. When the computing device is in low power mode, the display is not capable of receiving touch input.

[0008] The display is capable of receiving touch input when the computing device is in a high power mode that consumes more power than in a low power mode. The action includes: identifying an application accessible to the computing device based on a second, different hot word; and providing a transcription of the utterance to the application. The additional user interface is generated based on providing a transcription of the utterance to the application. The action includes receiving, by the computing device, a first hot word model for a first hot word and a second, different hot word model for a second, different hot word. The action of determining that the audio data includes a second, different hot word includes applying the audio data to the second, different hot word model. The additional graphical interface includes a selectable option that, when selected by a user, updates the application. The action includes maintaining the computing device in a low power mode in response to determining that the audio data includes a second, different hot word. The action includes determining, by the computing device, that the speaker of the utterance is not a primary user of the computing device.

[0009] The action of obtaining a transcription of the utterance by performing speech recognition on the audio data is in response to determining that the speaker of the utterance is not a primary user of the computing device. The action includes: receiving, by the computing device, additional audio data corresponding to the additional utterance; determining, by the computing device, that the additional audio data includes a first hot word; and, in response to determining that the audio data includes a second, different hot word, switching the computing device from a low power mode to a high power mode that consumes more power than the low power mode. The action includes determining, by the computing device, that the speaker of the additional utterance is a primary user of the computing device. The action of switching the computing device from a low power mode to a high power mode that consumes more power than the low power mode is in response to determining that the speaker of the additional utterance is a primary user of the computing device.

[0010] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

[0011] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A computing device can be configured to automatically provide information on an always-on display in response to detecting a hot word. By distinguishing a first hot word that is configured to switch the computing device from a low-power state to a high-power state and a second, different hot word that is configured to enable the computing device to perform background processing and output information in a low-power state, the technology described herein provides a mechanism that enables user input to be properly processed in different situations and different uses of the computing device. The always-on display combined with distinguishing the first hot word and the second hot word provides a low-power way to convey information to the user without the user having to actively retrieve the information, which would cause the phone to switch to a mode that consumes more power when the user retrieves the information.

[0012] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An example system 100 is illustrated that performs hotword recognition and provides passive assistance through an always-on display.

[0014] Figure 2 An example system that performs hotword recognition and provides passive assistance is illustrated.

[0015] Figure 3 is a flow chart of an example process for performing hotword identification and providing passive assistance.

[0016] Figure 4 are examples of computing devices and mobile computing devices.

[0017] Like reference numbers and designations throughout the various drawings refer to like elements. DETAILED DESCRIPTION

[0018] Figure 1An example system 100 is illustrated that performs hotword recognition and provides passive assistance through an always-on display. Briefly, and as described in more detail below, a user 105 and a user 110 are having a conversation. The user 105 utters an utterance 115 by asking the user 110, "Are you free for lunch tomorrow?" During the conversation, the computing device 120 of the user 110 may be resting on a table and close enough to detect the utterance 115. Without the user 110 directing a command to the computing device 120, the computing device 120 processes the utterance 115 and displays the calendar information for the user 110 for the next day. The display of the computing device 120 may also be always-on, even when the computing device is in a sleep mode or in a low power mode.

[0019] exist Figure 1 In the example shown in , user 105 and user 110 are discussing their lunch plans for the next day. User 105 may not be directing commands to computing device 120. User 110 may be holding computing device 120 in the user's hand, or computing device 120 may be resting on a nearby table or in the user's shirt pocket. Computing device 120 may be any type of device configured to receive audio data, such as a smart phone, laptop computer, desktop computer, smart speaker, television, smart watch, or any other similar device.

[0020] The computing device 120 includes a display 125 that can be always active. When the phone is in sleep mode, locked, or the user 110 has not interacted with the phone or directed a command to the phone for a period of time, the display 125 can be in a low power state. When in a low power state, the display 125 can show the current date and time, but can be mainly blank or "off". When in a low power state, the display 125 can only display information in a single color such as gray. When in a low power state, the display 125 can display information at a lower pixel resolution than in a high power state. When in a low power state, the display 125 can operate at a reduced brightness, operate at a predetermined brightness, or operate at a maximum brightness lower than the maximum brightness of the display 125 when the device is operating in a high power state. When the computing device 120 receives additional messages or notifications, the computing device 120 can update the display 125. For example, if the computing device 120 receives a new email, the computing device 120 can update the display 125 in the low power state to include an envelope icon.

[0021] The low power state of the display 125 of the computing device 120 may be opposite to the high power state of the display 125 of the computing device 120. The high power state of the display 125 may be a typical state in which the user 110 interacts with the display 125. For example, when the display 125 is in the high power state, the user 110 may browse the Internet, check email, and write text messages. When the display 125 is in the high power state, the user 110 may adjust the brightness of the display 125. In some embodiments, the display 125 may be able to receive touch input across the entire display when in the high power state but not when in the low power state. For example, the display 125 may not be able to receive touch input in the low power state, or may be able to receive touch input only in a limited predefined area.

[0022] To switch the display 125 from the low-power state to the high-power state, the user 110 may provide a password, a biometric identifier, speak a specific hot word (e.g., OK Computer), press a sleep / wake button, or any other similar action. In some embodiments, the hot word may be a single word (e.g., "assistant") or multiple words (e.g., "OK Computer", "are you free", etc.). To switch the display 125 from the high-power state to the low-power state, the user 110 may not interact with the computing device 120 for a specific period of time (e.g., thirty seconds), press a sleep / wake button, or any other similar action.

[0023] In some embodiments, computing device 120 may perform some actions when in high power mode, and not perform those actions when in low power mode to save battery power, network bandwidth, processing power and / or any similar computing resources. For example, when in high power mode, computing device 120 may automatically obtain new messages from the network. When in low power mode, computing device 120 may not automatically obtain new messages from the network. When in high power mode, computing device 120 may automatically update or refresh any application running in the background. When in low power mode, computing device 120 may not update or refresh any application running in the background or not in the background. When in high power mode, computing device 120 may activate GPS sensors for location service applications or other applications. When in low power mode, computing device 120 may deactivate GPS sensors. When in high power mode, computing device 120 may synchronize data stored on computing device 120 with data stored in the cloud or vice versa. When in low power mode, the computing device may not synchronize data stored on the computing device 120 with data stored in the cloud or vice versa. When in high power mode, the computing device 120 may automatically download application updates from the application store. When in low power mode, the computing device 120 may not download application updates from the application store. In some embodiments, the computing device 120 may perform any of the above processes at a lower frequency when in low power mode than when in high power mode. For example, when in high power mode, the computing device 120 may automatically fetch new messages from the network every second. When in low power mode, the computing device may automatically fetch new messages from the network every minute.

[0024] exist Figure 1 In stage A of FIG. 1 , display 125 of computing device 120 is in a low power state. Display 125 displays graphical interface 130 including the current date and time and instructions for entering a password to unlock computing device 120. User 110 may not be interacting with computing device 120, and computing device 120 may be in a dormant state. Computing device 120 may be resting on a table near user 105 and user 110.

[0025] In stage B, user 105 speaks utterance 115. For example, user 105 may speak: “Are you free for lunch tomorrow.” In stage C, computing device 120 detects utterance 115 through a microphone. When computing device 120 receives the utterance, computing device 120 processes audio data 130 corresponding to utterance 115.

[0026] In stage D, computing device 120 compares audio data 130 to one or more hot word models. Computing device 120 may use the hot word models to determine whether the audio data includes one of the hot words without performing speech recognition. For example, computing device 120 may have a hot word model for “are you free” and a hot word model for “ok computer”. Computing device 120 may apply the hot word model to audio data 130 to determine that audio data 130 includes the hot word “are you free”. In some embodiments, computing device 120 may use one hot word model that is trained to detect multiple phrases. In some embodiments, computing device 120 may use multiple hot word models that are each trained on different phrases.

[0027] In stage E, computing device 120 performs speech recognition on the portion of audio data 130 that does not include the hot words. For example, computing device 120 may generate the transcription “for lunch tomorrow” by performing speech recognition on audio data 130 that does not include “are you free”.

[0028] Based on the identified hotwords or transcriptions, or both, computing device 130 may determine a specific action to perform or a specific application from which to access data. For example, the hotword "are you free" may trigger computing device 130 to access a calendar application. The hotword "don't forget to" may trigger the computing device to access a reminder application. The hotword "let's go" may access a carpool or bike-sharing application. In stage F, computing device 120 accesses a calendar application in response to detecting the hotword "are you free."

[0029] In stage G, computing device 120 may use the application accessed in stage F to determine the action to perform. Computing device 120 may identify an action based on the transcription of audio data 130. For example, the transcription may be "for lunch tomorrow". Based on the transcription and the identified hot words, computing device 120 may access a calendar during typical lunch times the next day and identify the schedule of user 110 during those times. As another example, computing device 120 may identify an action based on the transcription "next week". Based on this transcription and the identified hot word "are you free", computing device 120 may access next week's calendar. As another example, computing device 120 may identify the hot word "let's go". Computing device 120 may not recognize any other words spoken after the hot word. In this case, based on the hot word "let's go", the computing device may access a carpooling or bike-sharing application.

[0030] In stage H, computing device 120 generates graphical interface 135 that includes details from the application accessed in stage G. In stage I, computing device 120 displays graphical interface 135 on display 125 while display 125 remains in a low power state. For example, computing device 120 identifies a schedule for user 110's lunch time tomorrow by accessing a calendar. Graphical interface 135 may include information that user 110 is free the next day between 11 a.m. and 2 p.m. As another example, computing device 120 identifies the hot word "are you free" and the transcription "next week." In this case, computing device 120 may identify several different time slots that user 110 is free next week by accessing a calendar application. Computing device 120 generates graphical interface 135 with some free time slots and information indicating that there are additional free time slots. Graphical interface 135 may indicate “July 23 10am-2pm free,” “July 24 1pm-3pm free,” and “additional free time available.” Graphical interface 135 may also indicate that a password is to be entered to unlock computing device 120. In lieu of a password, computing device 120 may be unlocked upon receiving an appropriate biometric identifier. Graphical interface 135 may show a rendering of a day / week / month calendar with busy periods excluded. Event information for each period may be excluded so that private information is not shown on graphical interface 135.

[0031] In some embodiments, computing device 120 may passively authenticate the primary user and adjust the amount of detail that computing device 120 shows on the always-on display. For example, computing device 120 may verify that the user is near computing device 120 by recognizing the user's face in the field of view of the camera. In this case, computing device 120 may provide more details on the always-on display during the time period displayed on the always-on display, such as event information for each calendar appointment. As another example, computing device 120 may not be able to verify that the user is near computing device 120 by recognizing the user's face in the field of view of the camera. In this case, computing device 120 may provide less details on the always-on display during the time period displayed on the always-on display, such as a free / busy identifier for each calendar appointment. As an alternative to or in addition to facial recognition, computing device 120 may determine that the primary user is nearby by recognizing the primary user's voice using speaker verification or authentication. Computing device 120 may estimate the distance between computing device 120 and the primary user by measuring the received volume of voice audio identified as belonging to the primary user.

[0032] In some embodiments, the display 125 may be capable of receiving touch input while in a low-power state. In this case, the computing device 120 may generate a graphical interface 135 that includes buttons that the user can select to initiate additional actions by the computing device 120. For example, the computing device may recognize the hot word "let's go" and access a ride-sharing application. The graphical interface 135 may indicate that the ride-sharing driver is three minutes away and includes a button that the user 110 can select to initiate a ride-sharing request. The user 110 may select the button and the display 125 may transition to a high-power state in which the user 110 can complete the ride-sharing request. In some embodiments, after selecting the button and before the user 110 can complete the ride-sharing request, the user 110 may have to enter a password or provide other identification information.

[0033] While computing device 120 performs stages C through I and displays graphical interface 135, user 110 may not be interacting with computing device 120, either by touch or by voice commands. In stage J, user 110 looks at display 125 of computing device 120 and notices that computing device 120 has automatically updated display 125 to include the user's availability during lunch time the next day.

[0034] Figure 2 An example system 200 is shown that performs hot word recognition and provides passive assistance. System 200 can be any type of computing device configured to receive and process speech audio. For example, system 200 can be similar to Figure 1 The components of system 200 may be implemented in a single computing device or distributed across multiple computing devices. For privacy reasons, it may be beneficial for system 200 to be implemented in a single computing device.

[0035] System 200 includes audio subsystem 202. Audio subsystem 202 may include microphone 204, analog-to-digital converter 206, buffer 208, and various other audio filters. Microphone 204 may be configured to detect sounds in surrounding areas, such as speech. Analog-to-digital converter 206 may be configured to sample audio data detected by microphone 204. Buffer 208 may store sampled audio data for processing by system 200. In some embodiments, audio subsystem 202 may be continuously active. In this case, microphone 204 may detect sounds all the time. Analog-to-digital converter 206 may sample detected audio data all the time. Buffer 208 may store the latest sampled audio data, such as the last ten seconds of sound. If the audio data in buffer 208 is not processed by other components of system 200, buffer 208 may overwrite previous audio data.

[0036] exist Figure 2 In the example shown in , microphone 204 can detect speech corresponding to “Don't forget to buy milk”. Analog-to-digital converter 206 can sample the received audio data, and buffer 208 can store the sampled audio data 212.

[0037] The audio subsystem 202 provides the audio data 212 to the hotword detector 210. The hotword detector 210 is configured to identify hotwords in the audio received by the microphone 204 and / or stored in the buffer 208. The hotword detector 210 may be referred to as a hotword detector, a keyword finder, or a keyword detector. In some embodiments, the hotword detector 210 may be active at any time when the system 200 is powered on. The hotword detector 210 continuously analyzes the audio data stored in the buffer 208. The hotword detector 210 calculates a hotword confidence score, which reflects the possibility that the current audio data in the buffer 208 includes a hotword. In order to calculate the hotword confidence score, the hotword detector 210 may use a hotword model 214. The hotword detector 210 may extract audio features such as filter bank energy or Mel frequency cepstral coefficients from the audio data 212. The hotword detector 210 may use a classification window to process these audio features such as by using a support vector machine or a neural network. In some embodiments, the hotword detector 210 does not perform speech recognition to determine the hotword confidence score. If the hot word confidence score satisfies the hot word confidence score threshold, the hot word detector 210 determines that the audio includes a hot word. For example, if the hot word confidence score is 0.8 and the hot word confidence score threshold is 0.7, the hot word detector 210 determines that the audio 212 includes a hot word.

[0038] In some embodiments, the hot word model 214 includes hot word models for multiple hot words. For example, the hot word model 214 may include hot word models for "are you free", "don't forget", "let's go", "ok computer", and other terms. In some embodiments, a user or another system such as a server can add additional hot word models. For example, a user can indicate to the system 200 that a hot word model for "let's go" is to be added. The system 200 can request several audio samples of the user speaking "let's go". The system 200 can generate hot word models based on different audio samples. In some embodiments, the system can access

[0039] exist Figure 2, the hotword detector 210 determines that the audio data 212 includes the hotword 216 "don'tforget". The hotword detector 210 can identify the portion of the audio data 212 that includes the hotword 216. For example, the hotword detector 210 can identify the portion of the "don't forget to buy milk" audio data that includes the hotword "don't forget" for the speech recognizer 218. The hotword detector 210 can provide timing data to the speech recognizer 218, indicating that the audio data between 0.0 seconds and 0.5 seconds includes the hotword. Alternatively or additionally, the hotword detector 210 can provide memory location information to the speech recognizer 218, indicating that the audio data stored between memory addresses 0x98b89d24 and 0x98b8e35a includes the hotword.

[0040] In some embodiments, the hotword detector 210 may include speaker recognition capabilities. In this case, the hotword detector 210 may identify a specific person who spoke the hotword or someone other than the specific person who spoke the hotword. For example, the hotword detector 210 may determine that the user 105 is likely to have spoken the hotword. The hotword detector 210 may provide data indicating that the user 105 is likely to have spoken the hotword to the speech recognizer 218 and / or the action recognizer 222. As another example, the hotword detector 210 may determine that a user other than the user 110 is likely to have spoken the hotword. The hotword detector 210 may provide data indicating that a user other than the user 110 is likely to have spoken the hotword to the speech recognizer 218 and / or the action recognizer 222. In order to perform speaker recognition, the hotword detector 210 may have previously collected the user's voice data by requesting the user to repeat various phrases. The hotword detector 210 may have used the collected voice samples to train the speaker recognition model. For example, the hotword detector 210 may have used the voice samples spoken by the user 110 to train the speaker recognition model.

[0041] The speech recognizer 218 performs speech recognition on the audio data 212 or on the portion of the audio data 212 that does not include the hot words. The speech recognizer 218 can use the language model and the acoustic model to generate a transcription of the audio data 212 or the portion of the audio data 212 that does not include the hot words. For example, the speech recognizer 218 can perform speech recognition on the portion of the audio data 212 that does not include "don't forget" and generate the transcription 220 "to buy milk".

[0042] In some implementations, the hotword 210 is active if the system 200 is turned on. In this case, the hotword 210 can be implemented in hardware that uses less power than the main processor of the system 200. For example, the hotword 210 can be implemented in a digital signal processor (DSP). In some implementations, the speech recognizer 218 can be implemented in software executed by a processor of the system 200. The speech recognizer 218 and / or the processor of the system 200 can be activated in response to the hotword 210 detecting the hotword 216.

[0043] The speech recognizer 218 provides the hot word 216 and the transcription 220 to the action recognizer 222. The action recognizer 222 is configured to recognize an action from among the actions 224 for the system 200 to perform in response to the hot word 216 and the transcription 220. In some embodiments, the action may be related to the hot word 216. For example, the hot word 216 may be "don't forget". In this case, the hot word 216 "don't forget" may trigger the action recognizer 222 to recognize the action of adding a reminder to a reminder list or calendar application. As another example, the hot word 216 may be "are you free", which may trigger the action recognizer 222 to recognize the action of identifying free and busy times.

[0044] In some implementations, the action recognizer 222 receives an action from the user that is linked to the hot word. For example, the user may specify that in response to the hot word 216 "let's go", nearby shared bike options are checked. In some implementations, the server may specify that in response to the hot word 216 "let's go", nearby shared bike options and shared car options are checked.

[0045] In some embodiments, the action recognizer 222 can determine that the hotword is invalid due to input from the user and / or input from the server. For example, the user can specify that the action recognizer 222 does not respond to the hotword "are you free". The user can input selections for different hotwords and input additional hotwords through a menu or similar graphical interface provided by the system 200 via a display.

[0046] Action recognizer 222 may provide data identifying an action to application recognizer 226. Application recognizer 226 may identify an application from among applications 228 as a candidate application for performing the action recognized by action recognizer 222. Application recognizer 226 may identify an application for accessing additional application data 230 to provide to the user. For example, application recognizer 226 may identify a reminder application as a candidate application for performing an action of adding a reminder to a reminder list. Action 232 may be to add a reminder to the reminder application to "buy milk."

[0047] In some embodiments, applications 228 include applications installed on system 200 and / or applications accessible by system 200, for example, via a network connection. For example, an application installed on system 200 may be a reminder application or a calendar application. An application accessible via a network connection may be a web application. In some embodiments, application data 230 of an application installed on system 200 may be accessed via a network connection.

[0048] In some embodiments, the application identifier 226 identifies candidate applications based on the transcription 220 of the portion of the audio data 212 other than the hot word 216. For example, the hot word 216 may be "let's go" and the transcription 220 may be "to Alice's house". In this case, the action identifier 222 may identify an action to determine the availability of a dockless shared bike. The action identifier 222 may access a contacts application to determine the location of Alice's house. In the event that Alice's house is within a threshold distance from the system 200, the action identifier 222 accesses a shared bike application to determine the availability of nearby bikes. In the event that the location following the "let's go" hot word is outside the threshold distance, then the action identifier 222 may access a ridesharing application to determine the availability of nearby ridesharing vehicles.

[0049] In some implementations, the speaker of hotword 210 may affect the action selected by action recognizer 222 and / or the application selected by application recognizer 226. For example, if the speaker of hotword 216 is someone other than the user of system 200, hotword 216 "are you free" may trigger action recognizer 222 to recognize an action. If the speaker of hotword 216 is user 110, system 200 may refrain from triggering an action in response to hotword 216 "are you free." Return to Figure 1 Because computing device 120 determines that user 110 did not actually utter the hot word “are you free,” computing device 120 identifies the action and application.

[0050] Some hot words may have different actions depending on whether the speaker is the primary user of system 200 (e.g., the owner) or someone other than the primary user of system 200. For example, system 200 may detect "don't forget to call mom". If the speaker is someone other than the primary user, action recognizer 222 may recognize an action to add "call mom" to a reminder list. If the speaker is the primary user, action recognizer 222 may recognize an action to automatically add "call mom" to a reminder list or automatically schedule a calendar appointment for "call mom".

[0051] Action recognizer 222 provides recognized action 232 to user interface generator 234. User interface generator 234 generates graphical interface 236 for display on a display of system 200. System 200 can display graphical interface 236 when the display of system 200 is in a low power state. For example, user interface generator 234 can display graphical interface 236 including button 242 that a user can select to perform action 232 of adding “buy milk” to the reminder list.

[0052] Graphical interface 236 may include date and time portion 238. System 200 may display the current date and time on date and time portion 238 at any time when the display is in a low power state. User interface generator 234 may add additional graphical portion 240 including identified action 232. In some embodiments, additional graphical portion 240 includes button 242. A user may select button 242 to initiate system 200 to perform action 232. For example, a user may select button 242 to add "buy milk" to a reminder list. The user may transition the display to a high power state without selecting button 242. When the display transitions back to a low power state, additional graphical portion 240 and / or button 242 may or may not reappear.

[0053] In some implementations, the graphical interface 236 does not include the button 242. In the case where the purpose of the additional graphical portion 240 is to inform the user of some data that can be accessed by the system 200, the user interface generator 234 may not include the button 242. For example, the additional graphical portion 240 may indicate the user's schedule for lunch the next day. In the case where there is additional information that may not fit in the additional graphical portion 240, the user interface generator 234 may include the button 242 for the user to view the additional information.

[0054] In some implementations, system 200 may include privacy settings that allow a user to configure the level of detail displayed in additional graphical portion 240. A user may wish to adjust the level of detail to prevent the user's calendar information from being displayed on a display of system 200, since the display may always be on even when the display is in a low power state. As an example, a user may configure a calendar to display details of each calendar appointment in additional graphical portion 240. A user may also configure a calendar to only display whether the user is busy or free during the displayed time slots.

[0055] In addition to the above examples, the system 200 may also be configured to respond to the hot words "where's my phone" or "I can't find my phone". In this case, the system 200 may respond only if the system 200 is able to verify that the primary user of the system 200 is speaking the hot word using speaker verification or authentication. In response to the hot word, the system 200 may flash and / or brighten the always-on display, play a sound from the speaker, and / or activate the location module and transmit the location of the system 200 to another device (e.g., an email address and / or phone number specified by the primary user). The system 200 may also be configured to respond to the hot words "what's the weather today?" or "is it sunny today?" In this case, the system 200 may respond by generating an interface including a weather forecast through the user interface generator 234. The system 200 may provide the weather forecast interface to the display of the system 200 for presentation on the always-on display. In some implementations, system 200 may respond to the hotwords "what's the weather today?" or "is it sunny today?" only if system 200 can verify, using speaker verification or authentication, that the primary user of system 200 is speaking the hotword.

[0056] In some embodiments, the system 200 may be configured to detect the hot words "ok, I'll do it" or "will do it". The system 200 may detect these hot words after a speaker other than the primary user says something related to a reminder. The system 200 may update the user interface 236 with any details that follow the hot words, such as a time period. For example, a speaker other than the primary user may say "Don't forget to call mom". The primary user responds "ok, I'll do it tomorrow". The system 200 recognizes the hot words "don't forget", recognizes the speech of "to call mom", and recognizes the action of calling mom. The system 200 recognizes the hot words "ok, I'll do it", recognizes the speech of "tomorrow", and recognizes the time period of tomorrow. The system 200 may generate a user interface 236 indicating not to forget to call mom tomorrow for display on the always-on display. The user interface 236 may also include selectable options 242 as described above.

[0057] Figure 3 An example process 300 for performing hotword recognition and providing passive assistance is illustrated. In general, the process 300 performs speech recognition on audio that includes predefined hotwords. The process 300 outputs results related to the transcription of the audio on the display while the display remains in a low power mode. The process 300 will be described as being performed by a computer system including one or more computers, such as Figure 1 The computing device 120 or Figure 2 Executed by system 200.

[0058] The system is operating in a low power mode and includes a display that displays a graphical interface when the system is in the low power mode and (ii) is configured to exit the low power mode in response to detecting a first hot word and receive audio data corresponding to the speech (310). For example, the system can be configured to exit the low power mode in response to the hot word "OK computer". In this case, the system can brighten the display to indicate that the system is listening for further input from the speaker. In some embodiments, the system may not brighten the display until the user has stopped speaking for a threshold period of time. For example, the speaker can say "OK computer" and pause for two seconds. In this case, the display can be brightened and include a prompt asking the speaker how the system can help. As another example, the speaker can say "OK computer, call Mom" ​​and pause for two seconds. In this case, the display can be brightened and the system can open a phone application to initiate a call to Mom.

[0059] In some embodiments, the display of the system is a touch-sensitive display. When the system is in low power mode, the display may not be able to receive touch input. When the system is in high power mode, the display may be able to receive touch input. In low power mode, the system may be locked and display the date and time on the display. In high power mode, the system may be unlocked and display a home screen or application on the display.

[0060] The system determines that the audio data includes a second, different hot word (320). For example, the system may receive audio data of "are you free". In some embodiments, the system receives hot word models for various hot words that the system is configured to recognize. The system may receive hot word models for "OK ​​computer", "are you free", "don't forget", and other terms and phrases. The system can be configured to recognize hot words without using speech recognition. The system can use a hot word recognizer that operates continuously on the detected audio. The hot word recognizer can apply the hot word model to the detected audio and determine that the system received "are you free".

[0061] In some embodiments, the system remains in low power mode in response to detecting a hot word other than "OK computer". For example, the system remains in low power mode in response to detecting the hot word "are you free". The hot word "OK computer" can be a way for a user to speak directly to the system. In response to receiving "OK computer", the system can attempt to identify additional audio data, which includes commands such as "text Alice that I'll be homesoon", "order a large cheese pizza", or "what is my next appointment". The system executes the identified command and actively initiates the command. The system can send a text, order a pizza, or display the next appointment. Other hot words such as "are you free" and "don't forget" are more likely to occur during conversations between people. These hot words can trigger the system to listen for additional voices following the hot words. The system can passively provide information or request permission for additional actions in response to other hot words and following voices.

[0062] In response to determining that the audio data includes a second hot word, the system obtains a transcription of the utterance by performing speech recognition on the audio data (330). For example, the system performs speech recognition on the audio data following the hot word "are you free" and generates a transcription "for lunch tomorrow". In some embodiments, the system determines that the speaker of the second hot word is not a primary user of the system. The primary user of the system may be the owner of the system (e.g., the owner of a smart phone) or someone who uses the system most of the time. If the speaker is someone other than the primary user, the system activates a speech recognizer and obtains a transcription of the utterance by performing speech recognition on the audio data. The speaker being someone other than the primary user may indicate that the primary user is speaking to another person. If the speaker is the primary user, the system may not obtain a transcription of the speech and may not output any other information on the display.

[0063] The system generates an additional user interface based on the second hot word and the transcription of the utterance (340). For example, the system can generate an additional user interface that indicates a schedule for a user of the system during lunch time the next day. In some embodiments, the system identifies an application based on the second hot word and the transcription. The system can access the application to obtain information to generate the additional user interface. For example, the system can access a calendar application in response to "are you free for lunch tomorrow". As another example, the system can access a reminder application in response to "don't forget to call mom". Each hot word can be linked to an application. The hot word "are you free" triggers the system to access the calendar application. The hot word "don't forget" triggers the system to access the reminder application.

[0064] The system, while remaining in low power mode, provides an additional graphical interface for output on a display (350). For example, the system displays a graphical interface that includes the date, time, and busy / free schedule of the user between 11 a.m. and 2 p.m. the next day. In some embodiments, the additional graphical interface includes a button that the user can select for the system to initiate an action. For example, the additional graphical interface may include a button for adding "call mom" to a reminder list. In this case, the user may be able to select the button to add "call mom" to the reminder list. Selection of the button may also trigger the user to unlock the system. The user presses the button, unlocks the system, and the system updates the reminder list. If the user presses the button and fails to unlock the phone, the button may still be part of the graphical interface.

[0065] In some embodiments, the user can press a button or key on the system. In response to the button or key press, the system can switch to high power mode. The button can be a physical button such as a sleep / wake button. In some embodiments, the button or key press can be a specific touch gesture performed on the display, such as a diagonal sweep or a gesture selected by the user. The user can press the button, perform the gesture again, or wait for a period of time (e.g., ten seconds) to return the system to low power mode. In some embodiments, when the system returns to low power mode, the display can continue to display additional graphical interfaces. For example, the system can continue to display the busy / free time of the user's lunch the next day. In some embodiments, when the system returns to low power mode, the system displays the original graphical interface. In some embodiments, the system can open the application accessed by the system when switching to high power mode to generate an additional user interface. For example, when the system switches to high power mode and the user unlocks the system, the system can open a calendar application.

[0066] In some embodiments, the system may detect that a user speaks a first hot word, such as "OK computer". In this case, the system may switch to a high power mode and wait for commands from the speaker. If the speaker does not say anything else within a threshold time period, the system may return to a low power mode. In some embodiments, the system may respond to the first hot word only if the primary user of the system speaks the first hot word. If a speaker other than the primary user speaks the first hot word, the system may not switch to a high power mode and may ignore any commands spoken by the user after the first hot word. In some embodiments, the system may respond to hot words other than "OK computer" (e.g., "are you free" and "don't forget") only when the system is in low power mode. When the system is in high power mode, the system may not respond to hot words other than "ok computer".

[0067] Figure 4 Examples of computing device 400 and mobile computing device 450 that can be used to implement the techniques described herein are shown. Computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are examples only and are not meant to be limiting.

[0068] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connected to the memory 404 and a plurality of high-speed expansion ports 410, and a low-speed interface 412 connected to the low-speed expansion port 414 and the storage device 406. Each of the processor 402, the memory 404, the storage device 406, the high-speed interface 408, the high-speed expansion port 410, and the low-speed interface 412 is interconnected using various buses and can be installed on a general motherboard or in other suitable ways. The processor 402 can process instructions for execution within the computing device 400, including instructions stored in the memory 404 or stored on the storage device 406, to display graphical information for a GUI on an external input / output device such as a display 416 coupled to the high-speed interface 408. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various types of memories, can be used appropriately. Moreover, multiple computing devices can be connected, each device providing a portion of the necessary operations (for example, as a server group, a group of blade servers, or a multi-processor system).

[0069] Memory 404 stores information within computing device 400. In some implementations, memory 404 is a volatile memory unit or multiple volatile memory units. In some implementations, memory 404 is a non-volatile memory unit or multiple non-volatile memory units. Memory 404 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0070] Storage device 406 can provide mass storage for computing device 400. In some embodiments, storage device 406 can be or include computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, tape devices, flash memory or other similar solid-state memory devices or device arrays, including devices in storage area networks or other configurations. Instructions can be stored in information carriers. When instructions are executed by one or more processing devices (e.g., processor 402), one or more methods, such as those described above, are executed. Instructions can also be stored by one or more memory devices (e.g., memory 404, storage device 406, or memory on processor 402) such as computer-readable or machine-readable media.

[0071] The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low-speed interface 412 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In some embodiments, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 410 that can accept various expansion cards (not shown). In an embodiment, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, for example, through a network adapter.

[0072] As shown in the figures, computing device 400 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 420, or multiple times as a group of such servers. In addition, it can be implemented in a personal computer such as a laptop computer 422. It can also be implemented as part of a rack server system 424. Alternatively, components from computing device 400 can be combined with other components in a mobile device (not shown) such as mobile computing device 450. Each such device can contain one or more of computing device 400 and mobile computing device 450, and the entire system can be composed of multiple computing devices that communicate with each other.

[0073] The mobile computing device 450 includes a processor 452, a memory 464, an input / output device such as a display 454, a communication interface 466, and a transceiver 468, among other components. The mobile computing device 450 may also be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of the processor 452, the memory 464, the display 454, the communication interface 466, and the transceiver 468 are interconnected using various buses, and several components may be mounted on a common motherboard or in other suitable ways.

[0074] Processor 452 may execute instructions within mobile computing device 450, including instructions stored in memory 464. Processor 452 may be implemented as a chipset including separate and multiple analog and digital processor chips. Processor 452 may provide, for example, coordination for other components of mobile computing device 450, such as control of a user interface, applications executed by mobile computing device 450, and wireless communications performed by mobile computing device 450.

[0075] Processor 452 can communicate with a user through control interface 458 and display interface 456 coupled to display 454. Display 454 can be, for example, a TFT (thin film transistor liquid crystal display) display or an OLED (organic light emitting diode) display or other suitable display technology. Display interface 456 may include appropriate circuits for driving display 454 to present graphics and other information to the user. Control interface 458 may receive commands from the user and convert them for submission to processor 452. In addition, external interface 462 may provide communication with processor 452, so as to enable near-area communication of mobile computing device 450 with other devices. For example, external interface 462 may be provided for wired communication in some embodiments, or for wireless communication in other embodiments, and multiple interfaces may also be used.

[0076] Memory 464 stores information within mobile computing device 450. Memory 464 may be implemented as one or more of a computer-readable medium or multiple computer-readable media, a volatile storage unit or multiple volatile storage units, or a non-volatile storage unit or multiple non-volatile storage units. Expansion memory 474 may also be provided and connected to mobile computing device 450 via expansion interface 472, which may include, for example, a SIMM (single in-line memory module) card interface. Expansion memory 474 may provide additional storage space for mobile computing device 450, or may also store applications or other information for mobile computing device 450. Specifically, expansion memory 474 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, expansion memory 474 may be configured as a security module for mobile computing device 450, and may be programmed with instructions that allow secure use of mobile computing device 450. In addition, security applications may be provided via SIMM cards, as well as other information, such as placing identification information on SIMM cards in an inviolable manner.

[0077] The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some embodiments, instructions are stored in an information carrier that, when executed by one or more processing devices (e.g., processor 452), perform one or more methods, such as those described above. The instructions may also be stored by one or more storage devices, such as one or more computer or machine readable media (e.g., memory 464, expansion memory 474, or memory on processor 452). In some embodiments, the instructions may be received, for example, as a propagated signal via transceiver 468 or external interface 462.

[0078] The mobile computing device 450 may communicate wirelessly via a communication interface 466, which may include digital signal processing circuitry, if necessary. The communication interface 466 may provide communications in a variety of modes or protocols, such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Message Service) or MMS messages (Multimedia Message Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), etc. For example, such communications may occur using radio frequencies through a transceiver 468. In addition, short-range communications may occur, such as using Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation and location-related wireless data to the mobile computing device 450, which may be used appropriately by applications running on the mobile computing device 450.

[0079] Mobile computing device 450 may also communicate audibly using audio codec 460, which may receive voice information from a user and convert it into usable digital information. Audio codec 460 may similarly generate audible sound for the user, such as through a speaker, for example, in an earpiece of mobile computing device 450. Such sound may include sound from a voice telephone call, may include recorded sound (e.g., voice messages, music files, etc.), and may also include sound generated by applications running on mobile computing device 450.

[0080] Mobile computing device 450 can be implemented in many different forms, as shown in the drawings. For example, it can be implemented as a cellular phone 480. It can also be implemented as part of a smart phone 482, a personal digital assistant, or other similar mobile devices.

[0081] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor that can be special or general purpose, the at least one programmable processor being coupled to receive data and instructions therefrom and to transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0082] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms machine-readable medium, computer-readable medium refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal for providing machine instructions and / or data to a programmable processor.

[0083] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0084] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (such as an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0085] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0086] Although some embodiments have been described in detail above, other modifications are possible. For example, although the client application is described as accessing the delegate, in other embodiments, the delegate may be employed by other applications implemented by one or more processors, such as applications executed on one or more servers. In addition, the logic flows depicted in the accompanying drawings do not require the specific order or sequential order shown to achieve the desired results. In addition, other actions may be provided from the described flows, or actions may be eliminated, and other components may be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method comprising: receiving, by a computing device, audio data corresponding to an utterance, the computing device (i) being operated in a low power mode and comprising a display that displays a graphical user interface while the computing device is in the low power mode and (ii) configured to exit the low power mode in response to detecting a first hot word; as well as While the computing device remains in the low power mode: determining, by the computing device, that the audio data includes a second hot word different from the first hot word; In response to determining that the audio data includes the second hot word, obtaining, by the computing device, a transcription of the utterance by performing speech recognition on the audio data; generating, by the computing device, an additional graphical user interface based on the second hotword and the transcription of the utterance; as well as providing said additional graphical user interface for output on said display, in: The low power mode consumes less power than a high power mode of the computing device, While in the low power mode, the computing device does not perform at least one action performed by the computing device while in the high power mode, and Each of the first hot word and the second hot word is a predetermined word spoken to attract the attention of the computing device.

2. The method according to claim 1, comprising: receiving, by the computing device, input comprising a key press after providing the additional graphical user interface for output on the display; and After receiving the input comprising a key press, the computing device is switched to the high power mode.

3. The method according to claim 2, comprising: after switching the computing device to the high power mode and while the display remains active, returning the computing device to the low power mode; and After returning the computing device to the low power mode, the graphical user interface is provided for output on the display.

4. The method according to claim 2, wherein: When in the high power mode, the computing device retrieves data from a network at a first frequency, and While in the low power mode, the computing device retrieves data from the network at a second, lower frequency.

5. The method according to claim 1, wherein: the display being a touch-sensitive display, The display is incapable of receiving touch input while the computing device is in the low power mode, and The display is capable of receiving touch input while the computing device is in the high power mode.

6. The method according to claim 1, comprising: identifying an application accessible to the computing device based on the second hot word; and providing said transcription of said utterance to said application, wherein the additional graphical user interface is generated based on providing the transcription of the utterance to the application.

7. The method according to claim 1, comprising: A first hot word model for the first hot word and a second hot word model for the second hot word are received by the computing device, wherein the second hot word model is different from the first hot word model, Wherein, determining that the audio data includes the second hot word includes applying the audio data to the second hot word model.

8. The method according to claim 1, wherein: The additional graphical user interface includes a selectable option that updates the application when selected by a user.

9. The method according to claim 1, comprising: The computing device is maintained in the low power mode in response to determining that the audio data includes the second hot word.

10. The method according to claim 1, comprising: determining, by the computing device, that the speaker of the utterance is not a primary user of the computing device, Wherein obtaining the transcription of the utterance by performing speech recognition on the audio data is in response to determining that a speaker of the utterance is not a primary user of the computing device.

11. The method according to claim 1, comprising: receiving, by the computing device, additional audio data corresponding to additional utterances; Determining, by the computing device, that the additional audio data includes the first hot word; as well as In response to determining that the audio data includes the first hot word, the computing device is switched from the low power mode to the high power mode.

12. The method according to claim 11, comprising: determining, by the computing device, that the speaker of the additional utterance is a primary user of the computing device, Wherein switching the computing device from the low power mode to the high power mode is in response to determining that a speaker of the additional utterance is a primary user of the computing device.

13. A system comprising: One or more computers; and One or more storage devices storing executable instructions that, when executed by the one or more computers, cause the one or more computers to perform operations, the operations comprising: receiving, by a computing device, audio data corresponding to an utterance, the computing device (i) being operated in a low power mode and comprising a display that displays a graphical user interface while the computing device is in the low power mode and (ii) configured to exit the low power mode in response to detecting a first hot word; and While the computing device remains in the low power mode: determining, by the computing device, that the audio data includes a second hot word different from the first hot word; In response to determining that the audio data includes the second hot word, obtaining, by the computing device, a transcription of the utterance by performing speech recognition on the audio data; generating, by the computing device, an additional graphical user interface based on the second hotword and the transcription of the utterance; and providing said additional graphical user interface for output on said display, in: The low power mode consumes less power than a high power mode of the computing device, While in the low power mode, the computing device does not perform at least one action performed by the computing device while in the high power mode, and Each of the first hot word and the second hot word is a predetermined word spoken to attract the attention of the computing device.

14. The system according to claim 13, wherein: The operations include: receiving, by the computing device, input comprising a key press after providing the additional graphical user interface for output on the display; and After receiving the input comprising a key press, the computing device is switched to the high power mode.

15. The system of claim 13, wherein: The operations include: the display being a touch-sensitive display, The display is incapable of receiving touch input while the computing device is in the low power mode, and The display is capable of receiving touch input while the computing device is in the high power mode.

16. The system of claim 13, wherein: The operations include: identifying an application accessible to the computing device based on the second hot word; and providing said transcription of said utterance to said application, wherein the additional graphical user interface is generated based on providing the transcription of the utterance to the application.

17. The system of claim 13, wherein: The operations include: A first hot word model for the first hot word and a second hot word model for the second hot word are received by the computing device, wherein the second hot word model is different from the first hot word model, Wherein, determining that the audio data includes the second hot word includes applying the audio data to the second hot word model.

18. The system of claim 13, wherein: The additional graphical user interface includes a selectable option that updates the application when selected by a user.

19. The system of claim 13, wherein: The operations include: The computing device is maintained in the low power mode in response to determining that the audio data includes the second hot word.

20. A non-transitory computer-readable medium storing software, the software comprising instructions executable by one or more computers, the instructions, when such executed, causing the one or more computers to perform operations comprising: receiving, by a computing device, audio data corresponding to an utterance, the computing device (i) being operated in a low power mode and comprising a display that displays a graphical user interface while the computing device is in the low power mode and (ii) configured to exit the low power mode in response to detecting a first hot word; as well as While the computing device remains in the low power mode: determining, by the computing device, that the audio data includes a second hot word different from the first hot word; In response to determining that the audio data includes the second hot word, obtaining, by the computing device, a transcription of the utterance by performing speech recognition on the audio data; generating, by the computing device, an additional graphical user interface based on the second hotword and the transcription of the utterance; as well as providing said additional graphical user interface for output on said display, in: The low power mode consumes less power than a high power mode of the computing device, While in the low power mode, the computing device does not perform at least one action performed by the computing device while in the high power mode, and Each of the first hot word and the second hot word is a predetermined word spoken to attract the attention of the computing device.

Images