Detecting and / or registering hot commands for triggering response actions by an automated assistant

Abstract

To provide a method, system and computer readable medium for detecting or enrolling new hot commands that can be used to cause an automatic assistant to perform a response action without first having to explicitly trigger them.SOLUTION: An automatic assistant operating on a computing device is transitioned from a limited listening state to a full speech recognition state in response to a trigger event, and while in the full speech recognition state, receives a command spoken by a user to generate a text command, performs speech recognition processing, determines if the text command meets the frequency threshold in the corpus of the text command, and if so, enrolls the data indicating the text command as a hot command. A subsequent utterance of another text command that is semantically consistent with the text command triggers execution of a response action by the automatic assistant without requiring explicit activation.SELECTED DRAWING: Figure 6

Description

[Background technology]

[0001] Humans may engage in human-to-computer dialogue with interactive software applications referred to herein as "automated assistants" (also referred to as "chatbots," "conversational personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," "virtual assistants," etc.). For example, humans (sometimes referred to as "users" when they interact with an automated assistant) may provide commands, queries, and / or requests using free-form natural language input, which may include voice utterances that are converted to text and then processed and / or converted into typed free-form natural language input.

[0002] In many cases, before an automated assistant can interpret and respond to a user's request, it must first be "activated," for example, using a predefined verbal activation phrase, often referred to as a "hot word" or "wake word." Accordingly, many automated assistants operate in what will be referred to herein as a "limited listening state" or "default listening state," in which the automated assistant is always "listening" to audio data sampled by a microphone for a limited (or finite or "default") set of hot words. Any utterances captured in the audio data other than the default set of hot words are ignored. Once the automated assistant is activated with one or more of the default set of hot words, the automated assistant may operate, for at least some time interval after activation, in what will be referred to herein as a "full listening state," in which the automated assistant performs speech-to-text ("STT") processing (also referred to as "speech recognition processing") of the audio data sampled by the microphone to generate text input, which is then semantically processed to determine (and realize) the user's intent.

[0003] Operating an automated assistant in a default listening state provides various benefits. Limiting the number of "listened-for" hot words allows for conservation of power and / or computational resources. For example, an on-device machine learning model can be trained to generate an output indicating when one or more hot words are detected. Implementing such a model can require minimal computational resources and / or power, which is particularly beneficial for assistant devices that are often resource-constrained. Along with these benefits, operating an automated assistant in a limited hot word listening state also presents various challenges. To avoid inadvertent activation of an automated assistant, hot words are generally selected to be words or phrases (e.g., "long-tail" words or phrases) that are not often uttered in everyday conversation. However, there are various scenarios that can be troublesome, requiring a user to utter a long-tail hot word before activating an automated assistant to perform some action. Summary of the Invention [Means for solving the problem]

[0004] Techniques are described herein for detecting and / or registering (or commissioning) new "hot commands" that can be used to cause an automated assistant to perform a responsive action without first having to be explicitly activated. As used herein, a "hot command" refers to one or more words or phrases that, when spoken, cause the automated assistant to respond without first having to be explicitly activated and transitioning to a full listening / responding state in which the automated assistant attempts to respond to any captured utterances.

[0005] In some implementations, speech recognition may be implemented, in whole or at least in part, on-board a client device, such as a stand-alone interactive speaker, which may or may not include other components, such as a display, camera, and / or other sensors. In some such implementations, the automated assistant may perform speech recognition processing on spoken utterances captured at times other than immediately after the automated assistant is launched. These other times may include, for example, whenever a user is detected in proximity to the computing device, whenever user voice is detected and determined not to be originating from another machine, such as a television or radio, etc.

[0006] In other words, in some implementations, a computing device implementing an automated assistant configured with selected aspects of the present disclosure may perform speech recognition processing on more detected utterances than a conventional automated assistant, which, as discussed above, may perform speech recognition processing only on utterances detected after the automated assistant is explicitly invoked. Text fragments generated from this speech recognition processing may be analyzed using the techniques described herein to determine whether these text fragments should trigger a response action by the automated assistant, be registered as hot commands, or be ignored or discarded. In many implementations, even though text fragments are generated for a larger portion of detected utterances than a conventional automated assistant, the techniques described herein may be executed locally on the client device, thereby avoiding transmission of text fragments to a cloud-based system.

[0007] In some implementations, hot commands may be selectively registered in a hot command library based on various “hot command registration criteria.” One hot command registration criterion may be that a text command received after explicit activation of the automated assistant satisfies a frequency threshold in a corpus of text commands. This corpus may be associated with a particular user (e.g., speaker) or a population of users. For example, assume that a particular user activates the automated assistant some threshold number of times to issue a particular command, “Turn off the lights.” Before the threshold is met, this command cannot trigger a response action by the automated assistant without first being explicitly activated. However, once the threshold is met (e.g., when the user activates the automated assistant 10 times to turn off the lights), the text command “Turn off the lights” is registered in the hot command library. Thereafter, whenever the same command is issued by the same user without first activating the automated assistant, the automated assistant still takes a response action, for example, by turning off the lights in the same room.

[0008] In some implementations, particularly where speech recognition processing is performed only after the automated assistant is explicitly invoked, registering the text command in the hot command library can further train the on-device machine learning model described above to detect text commands in the future. In other implementations where the automated assistant performs speech recognition processing (on-board) on other utterances captured at other times, the text fragments generated from each utterance can be compared to the text fragments stored in the hot command library. For example, a match can occur if a given text fragment is sufficiently similar to and / or semantically consistent with a record in the hot command library.

[0009] Sufficient similarity may be found, for example, when there is an exact match between the text fragment and a registered hot command, or when the edit distance between the text fragment and the registered hot command meets some threshold (e.g., is less than some edit distance), or even when the embedding of the text fragment is within some Euclidean distance of the embedding of the hot command. In some implementations, the hot command library may only store embeddings of previously registered text commands. For example, semantic consistency may exist between two commands if corresponding slots of the commands may receive semantically similar values. For example, "set timer for 10 minutes" and "set timer for 1 hour" are semantically consistent because the slot they share is a time interval.

[0010] In other implementations, a text fragment generated from a spoken utterance may be deemed a hot command based on factors other than (or in addition to) its registration in a hot command library. For example, in some implementations, speech recognition processing may be performed on multiple utterances to generate multiple text fragments (e.g., without requiring explicit activation of the automated assistant). Meanwhile, one or more visual sensors may generate visual data that, when analyzed, reveals the speaker's gaze during each utterance. A particular text fragment may be deemed a hot command when the speaker was looking at a computing device on which an automated assistant is at least partially implemented while speaking the particular utterance that resulted in the particular text fragment.

[0011] For example, another signal that can be considered for registration in a hot command library or for determining whether a given text fragment meant a command to an automated assistant is whether one or more smart appliances were operated by a person immediately after the utterance was captured. Suppose a user says "Turn on the lights," but then another user operates a switch to turn on some smart lights. That would suggest that the command "Turn on the lights" should not be registered or interpreted as a hot command. However, in the same scenario, suppose the smart lights are not turned on, but are followed by silence. That may suggest that the command "Turn on the lights" should be interpreted as a hot command, at least in this case, and may also increase the likelihood that the command should be registered as a hot command in the future, especially if this scenario occurs multiple times.

[0012] In some implementations, a method is provided that is executed by one or more processors and includes: operating an automated assistant using one or more of the processors; transitioning the automated assistant from a limited listening state to a full speech recognition state in response to a trigger event; receiving, by the automated assistant, a spoken command from a user while in the full speech recognition state; performing speech recognition processing on the spoken command to generate a text command; determining that the text command satisfies a frequency threshold in a corpus of text commands; and, in response to the determination, registering data indicative of the text command as a hot command, wherein following registration, utterance of another text command that is semantically consistent with the text command triggers performance of a responsive action by the automated assistant without requiring explicit activation of the automated assistant.

[0013] In various implementations, the trigger event may include the detection of one or more default hotwords in audio data captured by a microphone. In various implementations, the corpus of text commands may include a corpus of text commands generated by a user. In various implementations, the corpus of text commands may include a corpus of text commands generated by a population of users, including the user.

[0014] In various implementations, the method may further include applying the text command as an input through a machine learning model to generate an output. The output may indicate a probability that the text command was directed to the automated assistant, and the registration is further responsive to satisfaction of a threshold value according to the probability. In various implementations, the method may further include detecting, based on visual data captured by one or more cameras, that a user's gaze is directed toward a computing device on which the automated assistant is at least partially implemented, and the registration is further responsive to the detection.

[0015] In various implementations, registration may be further responsive to a determination that no additional speech is detected during a predetermined time interval after the spoken command. In various implementations, registration is further responsive to a determination that no responsive action is taken by a person during a predetermined time interval after the spoken command.

[0016] In various implementations, the method may further include caching data indicative of the text command or the text command itself in a local memory of the computing device that at least partially implements the automated assistant in response to the registration. In various implementations, the method may further include triggering a text command following the registration based on the cached data without the text command being explicitly invoked to cache information responsive to the text command, wherein following the trigger, the cached information is output in response to a subsequent invocation of the text command instead of the text command being triggered. In various implementations, following the caching, invocation of the text command may cause the smart appliance to transmit data indicative of the text command directly to the smart appliance via one or both of a local area network (“LAN”) and a personal area network (“PAN”). In various implementations, receipt of the data indicative of the text command may cause the smart appliance to perform a responsive action.

[0017] In various implementations, the data indicative of the text command may include a template including one or more slots for receiving one or more subsequent parameters that are semantically consistent with the one or more parameters provided in the spoken command received from the user.

[0018] In another related aspect, a method implemented using one or more processors may include receiving audio data captured at one or more microphones, the audio data including first and second spoken utterances; performing speech recognition processing on the audio data to generate first and second text fragments corresponding to the first and second spoken utterances; performing a response action based on the first text fragment in response to determining that the first text fragment satisfies one or more hot command criteria; and discarding or ignoring the second text fragment without performing another response action in response to determining that the second text fragment fails to satisfy the one or more hot command criteria.

[0019] In various implementations, the one or more hot command criteria may include registering the text fragment under consideration in a hot command library. In various implementations, the hot command library may include text fragments that, when spoken, trigger the performance of a response action by the automated assistant without requiring detection of one or more default hot words or transitioning the automated assistant to an active listening state.

[0020] In various implementations, the method may further include detecting, based on visual data captured by the one or more cameras, a first gaze of the user while the user is providing the first spoken utterance and a second gaze of the user while the user is providing the second spoken utterance. In various implementations, the one or more hot command criteria may include the gaze being directed toward a computing device on which the automated assistant is at least partially implemented.

[0021] Additionally, some implementations include one or more processors of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause any of the aforementioned methods to be performed. Some implementations also include one or more non-transitory computer-readable storage media having stored thereon computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0022] It should be appreciated that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the presently disclosed subject matter, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the presently disclosed subject matter. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2A] FIG. 10 is a diagram illustrating an example of how an utterance may be processed to activate an automated assistant, according to various implementations. [Figure 2B] FIG. 10 is a diagram illustrating an example of how an utterance may be processed to activate an automated assistant, according to various implementations. [Figure 3A] FIG. 10 is a diagram that schematically illustrates exemplary state machine stages that may be implemented by an automated assistant and / or assistant device configured with selected aspects of the present disclosure, according to various implementations. [Figure 3B] FIG. 10 is a diagram that schematically illustrates exemplary state machine stages that may be implemented by an automated assistant and / or assistant device configured with selected aspects of the present disclosure, according to various implementations. [Figure 3C]FIG. 10 is a diagram that schematically illustrates exemplary state machine stages that may be implemented by an automated assistant and / or assistant device configured with selected aspects of the present disclosure, according to various implementations. [Figure 4A] FIG. 1 illustrates an exemplary interaction between a user and an automated assistant configured with selected aspects of the present disclosure. [Figure 4B] FIG. 1 illustrates an exemplary interaction between a user and an automated assistant configured with selected aspects of the present disclosure. [Figure 5A] 10A-10C illustrate further exemplary interactions between a user and an automated assistant configured with selected aspects of the present disclosure. [Figure 5B] 10A-10C illustrate further exemplary interactions between a user and an automated assistant configured with selected aspects of the present disclosure. [Figure 6] 1 is a flowchart illustrating an exemplary method according to implementations disclosed herein. [Figure 7] 1 is a flow chart illustrating an exemplary method according to implementations disclosed herein. [Figure 8] FIG. 1 illustrates an exemplary architecture of a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0024] 1, an example environment in which the techniques disclosed herein may be implemented is shown. This example environment includes one or more client computing devices 106. Each client device 106 may run a respective instance of an automated assistant client 108, sometimes referred to herein as the "client portion" of the automated assistant. One or more cloud-based automated assistant components 119, sometimes collectively referred to herein as the "server portion" of the automated assistant, may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems), shown generally at 115, communicatively coupled to the client devices 106 via one or more local and / or wide area networks (e.g., the Internet).

[0025] In various implementations, an instance of an automated assistant client 108, through its interaction with one or more cloud-based automated assistant components 119, can form what appears from a user's perspective to be a logical instance of an automated assistant 120 with which the user may engage in a human-computer dialogue. One instance of such an automated assistant 120 is illustrated in FIG. 1 by a dashed line. Thus, it should be understood that each user engaging with an automated assistant client 108 running on a client device 106 effectively engages its own logical instance of the automated assistant 120. For simplicity and brevity, the term "automated assistant" as used herein as "serving" a particular user will refer to the combination of the user and the automated assistant client 108 running on the client device 106 operated by one or more cloud-based automated assistant components 119 (which may be shared among multiple automated assistant clients 108). It should also be understood that in some implementations, the automated assistant 120 can respond to requests from any user, regardless of whether the user is actually being "served" by that particular instance of the automated assistant 120.

[0026] The one or more client devices 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (which may in some cases include a visual sensor), a smart appliance such as a smart television (or a standard television with a network-connected dongle with automated assistant functionality), and / or a user wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality computing device or an augmented reality computing device). Additional and / or alternative client computing devices may be provided. Some client devices 106, such as standalone interactive speakers (or “smart speakers”), may take the form of an assistant device designed primarily to facilitate a dialogue between the user and the automated assistant 120. Some such assistant devices may take the form of a standalone interactive speaker with an attached display, which may or may not be a touchscreen display.

[0027] In some implementations, the client device 106 may include one or more visual sensors 107 having one or more fields of view, although this is not required. The visual sensor 107 may take various forms, such as a digital camera, a passive infrared (“PIR”) sensor, a stereoscopic camera, an RGBd camera, etc. The one or more visual sensors 107 may be used, for example, by the image capture module 111 to capture image frames (still images or video) of the environment in which the client device 106 is deployed. These image frames may then be analyzed, for example, by the visual cue module 1121, to detect user-provided visual cues contained within the image frames. These visual cues may include, but are not limited to, hand gestures, gaze toward specific reference points, facial expressions, predefined movements by the user, etc. These detected visual cues may be used for various purposes, such as activating the automated assistant 120 and / or causing the automated assistant 120 to perform various actions.

[0028] Additionally or alternatively, in some implementations, client device 106 may include one or more proximity sensors 105. Proximity sensors may take various forms, such as a passive infrared ("PIR") sensor, a radio frequency identification ("RFID"), a component that receives a signal emitted from another nearby electronic component (e.g., a Bluetooth signal from a nearby user's client device, a high-frequency or low-frequency sound emitted from the device, etc.). Additionally or alternatively, visual sensor 107 and / or microphone 109 may be used as a proximity sensor by, for example, visually and / or audibly detecting the proximity of a user.

[0029] As described in more detail herein, the automated assistant 120 engages in a human-to-computer dialog session with one or more users via user interface input devices and user interface output devices of one or more client devices 106. In some implementations, the automated assistant 120 may engage in a human-to-computer dialog session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the client devices 106. In some of these implementations, the user interface input is explicitly targeted to the automated assistant 120. For example, a user can verbally provide (e.g., type or speak) a predetermined activation (“hot” or “wake”) phrase, such as “OK, assistant” or “hey, assistant,” to cause the automated assistant 120 to begin actively listening or monitoring typed text. Additionally or alternatively, in some implementations, the automated assistant 120 may be activated based on one or more detected visual cues, alone or in combination with a verbal activation phrase.

[0030] In some implementations, the automated assistant 120 can engage in a human-computer dialog session in response to a user interface input, even when the user interface input is not explicitly targeted to the automated assistant 120. For example, the automated assistant 120 can examine the content of the user interface input and engage in a dialog session in response to the presence of certain terms in the user interface input and / or based on other cues. In many implementations, the automated assistant 120 can utilize speech recognition to convert utterances from a user into text and respond to the text accordingly, for example, by providing search results, general information, and / or by taking one or more responsive actions (e.g., playing media, starting a game, ordering food, etc.). In some implementations, the automated assistant 120 can additionally or alternatively respond to an utterance without converting the utterance to text. For example, the automated assistant 120 can convert the voice input into embeddings, into entity representations (indicating one or more entities present in the voice input), and / or into other “non-text” representations and operate on such non-text representations. Thus, implementations described herein as operating on text converted from voice input may additionally and / or alternatively operate on the voice input directly and / or on other non-textual representations of the voice input.

[0031] The client computing device 106 and the computing devices operating the cloud-based automated assistant component 119 may each include one or more memories for storing data and software applications, one or more processors for accessing data and running applications, and other components that facilitate communication over a network. Operations performed by the client computing device 106 and / or by the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented, for example, as a computer program executing on one or more computers at one or more locations coupled together through a network.

[0032] As described above, in various implementations, client computing device 106 can operate automated assistant client 108, or the "client portion" of automated assistant 120. In various implementations, automated assistant client 108 can include audio capture module 110, the aforementioned image capture module 111, visual cue module 1121, and / or launch module 113. In other implementations, one or more aspects of audio capture module 110, image capture module 111, visual cue module 112, and / or launch module 113 can be implemented separately from automated assistant client 108, for example, by one or more cloud-based automated assistant components 119. For example, in FIG. 1 , there is also a cloud-based visual cue module 1122 that can detect visual cues in image data.

[0033] In various implementations, the audio capture module 110, which may be implemented using some combination of hardware and software, may interface with hardware such as the microphone 109 or other pressure sensor to capture an audio recording of the user's speech. In some implementations, the speech may be at least temporarily stored as audio data in a buffer, such as a ring buffer. Various types of processing may be performed on this audio recording for various purposes. In some implementations, the image capture module 111, which may be implemented using some combination of hardware or software, may be configured to interface with the visual sensor 107 to capture one or more image frames (e.g., digital photographs) corresponding to the field of view of the visual sensor 107.

[0034] In various implementations, the visual cue module 1121 (and / or the cloud-based visual cue module 1122) may be implemented using some combination of hardware or software and may be configured to analyze one or more image frames provided by the image capture module 111 to detect one or more visual cues captured within and / or across the one or more image frames. The visual cue module 1121 may employ various techniques to detect visual cues. For example, the visual cue module 1122 may use one or more artificial intelligence (or machine learning) models trained to generate output indicative of user-provided visual cues detected in the image frames.

[0035] The voice capture module 110 may be configured to capture the user's voice, for example, via the microphone 109, as previously described. Additionally or alternatively, in some implementations, the voice capture module 110 may be further configured to convert the captured audio into text and / or other representations or embeddings, for example, using voice-to-text (“STT”) processing techniques (also referred to herein as “speech recognition processing”). As shown in FIG. 1 , in some implementations, the voice capture module 110 may include an on-board STT module 117A used in addition to or instead of the cloud-based STT module 117 described below. Additionally or alternatively, in some implementations, the voice capture module 110 may be configured to perform text-to-speech (“TTS”) processing to convert text into computer-synthesized voice, for example, using one or more voice synthesizers.

[0036] However, in some cases, client device 106 may be relatively constrained in terms of computational resources (e.g., processing cycles, memory, battery, etc.), and therefore audio capture module 110 local to client device 106 may be configured to convert a finite number of different spoken phrases, particularly phrases that activate automated assistant 120, into text (or other forms, such as lower dimensionality embeddings). Other audio input may be sent to cloud-based automated assistant component 119, which may include cloud-based "TTS" module 116 and / or cloud-based STT module 117.

[0037] In various implementations, the activation module 113 may be configured to determine whether to activate the automated assistant 120 based on, for example, output provided by the audio capture module 110 and / or the visual cue module 1121 (which, in some implementations, may be combined with the image capture module 111 in a single module). For example, the activation module 113 may determine whether a user's utterance qualifies as an activation phrase that should initiate a human-to-computer dialogue session with the automated assistant 120.

[0038] In some implementations, the activation module 113 may analyze data indicative of a user's utterance, such as an audio recording or a vector of features extracted from an audio recording (e.g., embeddings), alone or in conjunction with one or more visual cues detected by the visual cue module 1121. In some implementations, the threshold employed by the activation module 113 to determine whether to activate the automated assistant 120 in response to a voice utterance may be reduced when certain visual cues are also detected. As a result, even when a user provides a voice utterance that is different from, but somewhat phonetically similar to, the appropriate activation phrase "OK, assistant," the utterance may still be accepted as an appropriate activation when detected in conjunction with a visual cue (e.g., a hand wave by the speaker, the speaker gazing directly at the visual sensor 107, etc.).

[0039] In some implementations, one or more on-device activation models stored, for example, in on-device model database 114, may be used by activation module 113 to determine whether an utterance and / or visual cue qualifies as an activation. Such on-device activation models may be trained to detect variants of activation phrases / gestures. For example, in some implementations, on-device activation models (e.g., one or more neural networks) may be trained using training examples that each include an audio recording (or extracted feature vector) of an utterance from a user and one or more image frames and / or data showing detected visual cues captured contemporaneously with the utterance. In some such implementations, on-device activation model 114 may generate output in the form of a probability p that the captured utterance constitutes an activation phrase meant to awaken automated assistant 120.

[0040] In FIG. 1, the on-device model database 114 includes one or more on-device activation models 1141-1144. N In some implementations, default on-device activation model 1141 may be trained to detect one or more default activation phrases or hot words in audio recordings or other data indicative thereof, such as those previously mentioned (e.g., "OK Assistant," "Hey Assistant," etc.). In some such implementations, these models may always be available and effective to transition automated assistant 120 to a full listening state in which any audio recordings captured by voice capture module 110 (at least over some period of time relative to subsequent activations) may be processed using other components of automated assistant 120 (e.g., on client device 106 or by one or more cloud-based automated assistant components 119) as described below.

[0041] Additionally, in some implementations, the on-device model database 114 may include one or more additional "context activation models" 1142-1144. N These context activation models 1142 to 1144 can be stored at least temporarily. N may be used by and / or available (e.g., activated) to the launch module 113 in a particular context. N may be trained to detect one or more context-specific hot words, for example, in an audio recording or other data indicative thereof. N may be selectively downloaded as needed from the hot command engine 128 of FIG. 1 , which, for example, forms part of the cloud-based automated assistant component 119 but may be implemented in whole or in part on the client device 106, as described in more detail below.

[0042] In various implementations, the launch module 113 includes context launch models 1142-1144. N When detecting various dynamic hotwords using the same, the launch module 113 can transition the automated assistant 120 to the full listening state described previously. Additionally or alternatively, regardless of whether the launch module 113 transitions the automated assistant 120 to the general listening state, the launch module 113 can transition the automated assistant 120 to a context-specific state in which one or more context-specific response actions are performed. In many cases, the audio data that triggered the transition of the automated assistant 120 to the context-specific state need not be sent to the cloud. Instead, the one or more context-specific response actions may be performed entirely on the client device 106, which may reduce both the response time and the amount of information sent to the cloud, which may be beneficial from a privacy perspective.

[0043] In some implementations, automated assistant 120, and more particularly, voice capture module 110, may perform STT processing on utterances detected other than simultaneously with activation of automated assistant 120. For example, in some implementations, voice capture module 110 may perform STT processing on all captured utterances, on utterances captured in a particular context, etc. The text generated from this STT processing may then be analyzed by various components described herein, e.g., to activate automated assistant 120, to perform various response actions, etc.

[0044] The cloud-based TTS module 116 may be configured to leverage the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formed by the automated assistant 120) into computer-generated voice output. In some implementations, the TTS module 116 may provide the computer-generated voice output to the client device 106, for example, to be output directly using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automated assistant 120 may be provided to the voice capture module 110, which may then convert the text data into computer-generated voice that is output locally.

[0045] The cloud-based STT module 117 may be configured to leverage the virtually unlimited resources of the cloud to convert audio data captured by the voice capture module 110 into text, which may then be provided to the intent matcher 135. In some implementations, the cloud-based STT module 117 may convert an audio recording of the speech into one or more phonemes and then convert the one or more phonemes into text. Additionally or alternatively, in some implementations, the STT module 117 may employ a state decoding graph. In some implementations, the STT module 117 may generate multiple candidate text interpretations of a user's utterance. In some implementations, the STT module 117 may weight or bias certain candidate text interpretations higher than other candidate text interpretations depending on whether a simultaneously detected visual cue is present.

[0046] Automated assistant 120 (and specifically cloud-based automated assistant component 119) may include intent matcher 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some implementations, one or more of the modules of automated assistant 120 may be omitted, combined, and / or implemented in a component that is separate from automated assistant 120. In some implementations, to protect privacy, one or more of the components of automated assistant 120, such as natural language processor 122, TTS module 116, STT module 117, etc., may be implemented at least in part on client device 106 (e.g., to eliminate the cloud).

[0047] In some implementations, the automated assistant 120 generates response content in response to various inputs generated by a user of one of the client devices 106 during a human-computer dialog session with the automated assistant 120. The automated assistant 120 may provide the response content (e.g., via one or more networks when separate from the user's client device) for presentation to the user as part of the dialog session. For example, the automated assistant 120 may generate response content in response to free-form natural language input provided via the client device 106. As used herein, free-form input is input that is formed by the user and is not constrained by a group of options presented for selection by the user.

[0048] As used herein, a "dialog session" may include a local, self-contained exchange of one or more messages between a user and automated assistant 120 (and possibly other human participants). Automated assistant 120 may distinguish between multiple dialog sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and the client device other than the dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from a standalone voice-activated product and then returns to it later), locking / sleeping the client device between sessions, a change in the client device used to interface with one or more instances of automated assistant 120, etc.

[0049] The intent matcher 135 may be configured to determine the user's intent based on input provided by the user (e.g., voice utterances, visual cues, etc.) and / or based on other signals, such as sensor signals, online signals (e.g., data obtained from a web service), etc. In some implementations, the intent matcher 135 may include the natural language processor 122 and the aforementioned cloud-based visual cue module 1122. In various implementations, the cloud-based visual cue module 1122 may operate similarly to the visual cue module 1121, except that the cloud-based visual cue module 1122 may have more resources at its disposal. Specifically, the cloud-based visual cue module 1122 may detect visual cues that can be used by the intent matcher 135, alone or in combination with other signals, to determine the user's intent.

[0050] The natural language processor 122 can be configured to process natural language input generated by a user via the client device 106 and may generate annotated output (e.g., in text format) for use by one or more other components of the automated assistant 120. For example, the natural language processor 122 may process free-form natural language input generated by a user via one or more user interface input devices of the client device 106. The annotated output generated includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0051] In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information within the natural language input. For example, the natural language processor 122 may include a morphological module capable of separating individual words into morphemes and / or annotating morphemes, for example, with their classes. The natural language processor 122 may also include a portion of a phonetic tagger configured to annotate terms with their grammatical roles. For example, that portion of the phonetic tagger may tag each term with its part of speech, such as “noun,” “verb,” “adjective,” “pronoun,” etc. Also, for example, in some implementations, the natural language processor 122 may additionally and / or alternatively include a dependency parser (not shown) configured to determine syntactic relationships between terms within the natural language input. For example, the dependency parser may determine which terms modify other terms, the subject and verb of a sentence, etc. (e.g., a parse tree) and annotate such dependencies.

[0052] In some implementations, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments with references to people (e.g., including literary characters, famous people, public figures, etc.), organizations, locations (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes that represent known entities (and possibly entity attributes) as well as edges that connect the nodes and represent relationships between the entities. For example, a “banana” node may be connected (e.g., as a child) to a “fruit” node, which may in turn be connected (e.g., as a child) to “products” and / or “food” nodes. As another example, a restaurant called “Hypothetical Cafe” may be represented by a node that also includes attributes such as its address, the types of food served, business hours, contact information, etc. The "Hypothetical Cafe" node may, in some implementations, be connected to one or more other nodes by edges (e.g., representing child-to-parent relationships), such as a "Restaurant" node, a "Business" node, a node representing the town and / or state in which the restaurant is located, etc.

[0053] The entity tagger of natural language processor 122 may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and / or at a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities and / or may optionally communicate with a knowledge graph or other entity database to resolve particular entities.

[0054] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be utilized to resolve the term "over there" to "Hypothetical Cafe" in the natural language input "We really liked it the last time we ate at Hypothetical Cafe."

[0055] In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122. For example, in some implementations, a named entity tagger may rely on annotations from a coreference resolver and / or a dependency parser when annotating all mentions of a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from a dependency parser when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 122 may use related previous input and / or other relevant data outside the particular natural language input to determine one or more annotations.

[0056] The intent matcher 135 may use various techniques to determine the user's intent, for example, based on output from the natural language processor 122 (which may include annotations and terms of the natural language input) and / or based on output from the visual cue modules (e.g., 1121 and / or 1122). In some implementations, the intent matcher 135 may have access to one or more databases (not shown) that contain multiple mappings, for example, between grammars, visual cues, and response actions (or, more generally, intentions). Often, these grammars may be selected and / or learned over time and may represent the most common intents of users. For example, one grammar, “play <artist>,” may be mapped to an intent that triggers a response action that causes music by that <artist> to be played on the user-operated client device 106. Another grammar, “[weather|forecast] today,” may be able to match user queries such as “What's the weather like today?” and “What's the forecast for today?”

[0057] In addition to or instead of grammars (alternatively, sometimes referred to herein as “templates”), in some implementations, the intent matcher 135 may employ one or more trained machine learning models, alone or in combination with one or more grammars and / or visual cues. These trained machine learning models may be stored in one or more databases and trained to identify intents, for example, by embedding data indicative of user utterances and / or any detected user-provided visual cues into a reduced-dimensional space and then determining which other embeddings (and therefore intents) are closest, using techniques such as, for example, Euclidean distance, cosine similarity, etc.

[0058] As seen in the example grammar for "play <artist>," some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or parameters). Slot values ​​can be determined in a variety of ways. A user will often actively provide a slot value. For example, for the grammar "order me <topping> pizza," a user may likely utter the phrase "order me a sausage pizza," in which case the slot <topping> is automatically filled. Additionally or alternatively, if the user does not actively provide a slot value and the user activates a grammar that includes slots to be filled with slot values, the automated assistant 120 can solicit those slot values ​​from the user (e.g., "What type of crust would you like on your pizza?"). In some implementations, slots can be filled with slot values ​​based on visual cues detected by visual cue modules 1121-1122. For example, a user may say something like "order me this many cat bowls" while holding up three fingers to the visual sensor 107 of the client device 106. Or, the user may say something like "Find me more movies like this" while holding the DVD case for a particular movie.

[0059] In some implementations, the automated assistant 120 may serve as an intermediary between a user and one or more third-party computing services 130 (or “third-party agents” or “agents”). These third-party computing services 130 may be independent software processes that receive input and provide response output. Some third-party computing services may take the form of third-party applications that may or may not run on a computing system separate from the computing system running the cloud-based automated assistant component 119, for example. One type of user intent that may be identified by the intent matcher 135 is to engage a third-party computing service 130. For example, the automated assistant 120 may provide access to an application programming interface (“API”) to a service for controlling a smart device. A user may invoke the automated assistant 120 and provide a command such as “I want to turn on the heating.” The intent matcher 135 can map this command to a grammar that engages the third-party service, thereby triggering the automated assistant 120 to turn on the user's heating. The third-party service 130 may provide the automated assistant 120 with a minimal list of slots that must be filled to realize (or "resolve") the command to turn on the heat. In this example, the slots may include the temperature the heat will be set to and the duration the heat will be turned on. The automated assistant 120 may generate and provide to the user (via the client device 106) a natural language output requesting the parameters for the slots.

[0060] The realization module 124 may be configured to receive the predicted / inferred intent output by the intent matcher 135 and associated slot values ​​(whether actively provided or requested by the user) and realize (or resolve) the intent. In various implementations, realization (or "resolution") of the user's intent may, for example, cause the realization module 124 to generate / obtain various realization information (also referred to as "response information" or "resolution information"). As will be described below, the realization information may, in some implementations, be provided to a natural language generator ("NLG" in some figures) 126, which may generate natural language output based on the realization information.

[0061] Because intent can be realized (or "resolved") in various ways, realization (or "resolution") information may take various forms. Suppose a user simply requests information, such as, "Where were the exterior shots in 'The Shining' filmed?" The user's intent may be determined to be a search query, for example, by intent matcher 135. The intent and the contents of the search query may be provided to realization module 124, which may be in communication with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) for responsive information, as shown in FIG. 1 . Realization module 124 may provide data indicative of the search query (e.g., text of the query, a reduced dimensional embedding, etc.) to search module 150. Search module 150 may provide responsive information, such as GPS coordinates, or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon." This responsive information may form part of the realization information generated by realization module 124.

[0062] Additionally or alternatively, the realization module 124 may be configured to receive, e.g., from the intent matcher 135, the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.) and trigger a responsive action. The responsive action may include, e.g., ordering goods / services, starting a timer, setting a reminder, initiating a phone call, playing media, operating a smart appliance, sending a message, etc. In some such implementations, the realization information may include a slot value associated with the realization, an acknowledgment response (which may in some cases be selected from pre-determined responses), etc.

[0063] The natural language generator 126 may be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 126 may be configured to receive as input realization information related to the realization of an intent and generate the natural language output based on the realization information. Additionally or alternatively, the natural language generator 126 may receive information from other sources, such as third-party applications (e.g., requested slots), which the natural language generator 126 may use to compose the natural language output for the user.

[0064] Hot command engine 128 may be configured to selectively register text commands as hot commands, e.g., in database 129, based on various signals. While shown in FIG. 1 as part of cloud-based automated assistant component 119, in various implementations hot command engine 128 and / or database 129 may additionally or alternatively be implemented in whole or in part on one or more client devices 106. Similarly, in some implementations, database 129 containing hot commands registered for a particular user, e.g., the user controlling client device 106, may be maintained in whole or in part on client device 106.

[0065] In various implementations, the hot commands engine 128 may be configured to determine, for example, that a text command generated by the STT 117A / 117 satisfies a frequency threshold within a corpus of text commands (e.g., a corpus of text commands received after the automated assistant 120 was invoked). For example, the text command may appear more than some minimum number of times within the corpus of past text commands, or may appear at some frequency or rate that matches or exceeds some minimum threshold. In response to this determination, the hot commands engine 128 may register data indicative of the text command in the database 129 (and / or a local database of the client device 106). This data indicative of the text command may or may not include the text command itself verbatim, embeddings generated from the hot command, templates or grammars (described below) generated from the hot command, etc. Following registration by the hot command engine 128, the utterance of another text command that matches or is semantically consistent with the text command may trigger the performance of a response action by the automated assistant 120 without requiring explicit invocation of the automated assistant 120.

[0066] In some implementations, hot commands registered by the hot command engine 128 using the techniques described herein may be stored as a grammar or "template" that includes one or more slots. In some cases, these slots may be expressed by wildcards or other similar mechanisms so that they may receive subsequent parameters that are semantically consistent with the parameters provided in the spoken command that gave rise to the hot command for which they are registered. Two different values ​​are "semantically consistent" when they can be used interchangeably in a slot.

[0067] For example, assume that over time, a user invokes the automated assistant 120 and then issues the following commands: "set timer for 5 minutes," "set timer for 20 minutes," and "set timer for 1 hour." The values ​​"5 minutes," "20 minutes," and "1 hour" are semantically consistent and all refer to time intervals. As a result, hot command templates such as "set timer for <time>" can be generated and registered so that the user does not need to invoke the automated assistant 120 before issuing semantically consistent commands in the future.

[0068] Other values ​​besides time intervals may be semantically consistent with each other as well. For example, in the context of controlling smart appliances, the identities of different smart lights (e.g., "Kitchen Light 1," "Living Room Light 4," "Room Light," "Back Porch Light") may be semantically consistent. Similarly, appliances that can be controlled using similar commands may be semantically consistent. For example, the template "Turn up <appliance>" may refer to the volume of an audio system, the thermostat temperature, the oven temperature, an air conditioner, the light intensity of a smart light, etc. Different people may likewise be semantically consistent in the context of hot commands. For example, "Send a message to <person's name>" may be used to send a message to different people depending on whose name is inserted in the slot.

[0069] The automated assistant 120 may be invoked in a variety of ways, depending on the functionality available at the client device 106 and / or in the audio capture module 110. Figures 2A-2B schematically illustrate two example pipelines for invoking the automated assistant 120 to cause the automated assistant 120 to perform a response action. Selected aspects of the present disclosure may be implemented in whole or in part either on the pipelines or on other pipelines that share various characteristics with the pipelines illustrated in Figures 2A-2B.

[0070] 2A shows a pipeline in which the automated assistant 120 relies primarily on the cloud-based STT module 117 to perform speech recognition processing. Starting from the left, an utterance is captured by the voice capture module 110, e.g., as recorded audio data. The voice capture module 110 extracts various audio features from the recorded audio data, such as phonemes, intonation, pitch, and tone. The activation module 113 applies these features as inputs to an activation model 114 to generate a probability p that the captured utterance was meant to activate the automated assistant 120.

[0071] If p fails to satisfy T at 240, the utterance may be ignored / discarded at 242. However, if p satisfies some threshold T at 240 (e.g., p>0.75, p>0.65, p>0.9, etc.), processing may return to the voice capture module 110, which may capture / buffer audio data including spoken commands uttered after, before, or mixed with the previously processed utterance. The voice capture module 110 may hand this command data (e.g., raw audio data, embedding, etc.) to the STT module 117 (cloud-based or on-board client device 106) for speech recognition processing.

[0072] The STT module 117 can generate text commands, which are then provided to the intent matcher 135 (e.g., via the natural language processor 122), which can process the commands as described above. The intent determined by the intent matcher 135 can be provided to the fulfillment module 124. The fulfillment module 124 can fulfill the intent as described above, for example, by performing one or more response actions. As indicated by the interval at the bottom of FIG. 2A , the automated assistant 120 can be in a “limited listening state” in which it responds primarily or exclusively to default hot words or other activation triggers (e.g., hot key presses) until it is successfully launched and the spoken command is captured / buffered by the voice capture module 110. After launch, the automated assistant 120 is in a “full listening state” in which it will perform STT processing on any utterances captured by the voice capture module 110 and attempt to fulfill them.

[0073] 2B illustrates an alternative pipeline in which STT processing is implemented primarily or exclusively on the onboard client device 106. Performing STT processing on-device may offer various technical advantages, such as protecting privacy by transmitting less data to the cloud and, specifically, by reducing latency associated with interpreting spoken commands. Spoken commands processed using the pipeline of FIG. 2B may often be processed much more quickly than spoken commands issued using the pipeline of FIG. 2A. This is particularly true in implementations in which registered hot commands and / or data indicative of registered hot commands are cached locally, e.g., in memory of the client device 106, as described below.

[0074] 2B, speech is captured by voice capture module 110. Voice capture module 110 generates and provides audio data to STT module 117A (onboard client device 106). STT module 117A performs speech recognition processing to generate text data ("TXT" in FIG. 2B) that is provided to downstream components, such as launch module 113 and / or hot command engine 128.

[0075] Whichever downstream component receives the text data from STT module 117A may analyze the text and make a decision at 244. If the analysis reveals that the utterance is a default hot word or phrase meant to activate automated assistant 120, processing may return to voice capture module 110, which may receive and / or provide additional audio data from a buffer, including the spoken command (which may follow, precede, or be intermixed with the initial utterance). Processing may then proceed through components 117, 135, and 124, as previously described.

[0076] However, at 244, the text data may alternatively be determined to contain a hot command. In that case, additional STT processing may be unnecessary. Instead, as shown in FIG. 2B , the hot command may be provided directly to the intent matcher 135, which may generate an intent to process the realization module 124. A text fragment may be considered a hot command based on various factors. In some implementations, a text command may be considered a hot command if it is similar to, e.g., semantically consistent with, a registered hot command.

[0077] Additionally or alternatively, in some implementations, a text command may be applied as an input across a trained machine learning model to generate an output. The output may indicate the probability that the text command was directed to the automated assistant 120. In some implementations, the machine learning model may be trained using free-form commands / queries issued by a user after the user explicitly invokes the automated assistant 120.

[0078] Additionally or alternatively, as indicated by the dashed arrow in FIG. 2B , in some implementations, rather than sending the registered hot command to the intent matcher 135, the registered hot command may already be cached locally on the client device 106, for example, as part of the registration process. In some such implementations, if the registered hot command is subsequently detected in a spoken utterance, a response action may be triggered based on the cached registered hot command, without the registered hot command being processed by the cloud-based automated assistant component 119.

[0079] In some implementations, information responsive to a text command, such as search results, may be cached. The cached information may be output in response to a subsequent invocation of a hot command instead of triggering a text command. For example, assume that a user frequently asks automated assistant 120, "What's the weather like today?" This query may be registered as a hot command. In some such implementations, this query may be automatically submitted to an appropriate search engine, e.g., periodically, randomly, every morning, etc. When the user later asks, "What's the weather like today?" (e.g., without having to first invoke automated assistant 120), the response data may already be cached, and thus automated assistant 120 may output the weather with little or no discernible latency.

[0080] Cached hot commands are not limited to search queries. In some implementations, hot commands for controlling smart appliances may be cached on the onboard client device 106 (e.g., as part of the registration process), e.g., to avoid communication with the cloud-based automated assistant component 119 and / or to reduce latency. Subsequent invocation of such a hot command may cause the smart appliance to transmit data indicative of the hot command directly. As used herein, transmitting data “directly” to a smart appliance may refer to transmitting data to the smart appliance via one or both of a local area network (“LAN”) and / or a personal area network (“PAN”) without exchanging data with a remote component (e.g., 119) via one or more wide area networks (“WAN”).

[0081] For example, some smart appliances, such as smart lighting or smart locks, may communicate wirelessly with a bridge component using technologies such as Bluetooth, mesh networking, etc. The bridge component may be reachable over one or more other types of wireless networks, such as Wi-Fi. In other words, the bridge component acts as a “bridge” between a first wireless communication network (e.g., Bluetooth, mesh) on which the smart appliance resides and a second wireless communication network (e.g., Wi-Fi) on which a user typically operates a client device 106.

[0082] The cached hot command (or data indicative thereof) may be sent directly from the client device 106 to the bridge component, or to the smart appliance itself if no bridge component is present, without exchanging any data with the cloud-based automated assistant component 119. Receipt of data indicative of the cached hot command can cause the smart appliance to perform a responsive action. For example, assume the command "turn on the lights" is registered as a hot command. In some implementations, this registration may include the command data that is actually communicated to the cached lights (or to the bridge component that controls those lights). As a result, when the hot command is later detected, its cached data may be immediately sent over the LAN / PAN to the lights / bridge component with little or no discernible latency.

[0083] With the pipeline of Figure 2B, automated assistant 120 performs STT processing more frequently than with the pipeline of Figure 2A. Thus, rather than having a "limited listening state" as in Figure 2A, in Figure 2B, automated assistant 120 has what is referred to herein as a "selectively responsive state," in which the automated assistant selectively responds to text commands / queries as described above with respect to decision making at 244. When automated assistant 120 is explicitly invoked, for example, using a default hotword, automated assistant 120 can transition to a "fully responsive state."

[0084] In the selectively responsive state, automated assistant 120 selectively responds to text commands based on various factors. One such factor is whether the text command (or a similar text command or matching grammar / template) was previously registered as a hot command. Another factor is whether the speaker's gaze is directed toward a computing device that, at least from the user's perspective, appears to implement at least a portion of automated assistant 120. Other factors are also contemplated herein. In the fully responsive state of the implementation of FIG. 2B , automated assistant 120 may send data indicative of any utterances it detects (e.g., STT text output) to cloud-based automated assistant component 119 for further processing.

[0085] 3A-3C schematically illustrate stages of an example state machine that may be implemented by an automated assistant (e.g., 120) and / or an assistant device (e.g., 106) configured with selected aspects of the present disclosure, according to various implementations. Starting on the left in FIG. 3A, the automated assistant 120 may initiate or default to a first state, which may be the limited listening state of FIG. 2A or the selective response state of FIG. 2B. Upon detecting one or more default hot words ("DIW" in FIGS. 3A-3C), the automated assistant 120 may transition to a second state, which may correspond to the full listening state of FIG. 2A or the full response state of FIG. 2B.

[0086] In the second state, the automated assistant 120 can wait for some free-form input, such as a spoken utterance. Upon detecting the free-form input, the automated assistant 120 can at least temporarily transition to a “command registration analysis” state. In the command registration analysis state, the automated assistant 120 can determine, for example, by the hot command engine 128, based on various factors, whether the free-form input should be registered as a hot command for future use without requiring the automated assistant 120 to transition from the first state to the second state. Although shown sequentially as “general processing,” in some implementations, the command registration analysis and general processing can be performed in parallel, or the command registration analysis can be performed after general processing, for example, to ensure that the command was valid and that the automated assistant 120 was able to respond.

[0087] The most important factor considered during command registration analysis is the frequency with which a command, or semantically consistent commands, are received. For example, the number of times a command, or semantically consistent commands, are received may be compared to a threshold. If the threshold is met, the text command, or a template / grammar assembled from the command and its slots, may be registered as a hot command that is available for future use without the need for the automated assistant 120 to be explicitly invoked. On the other hand, regardless of whether a free-form input is registered as a hot command, the input may be subject to general processing (e.g., by the intent matcher 135 and / or the realization module 124) to perform a response action.

[0088] FIG. 3B shows the state machine of FIG. 3A after a first registered hot command (“EHC1”) has been registered as a result of command registration analysis. Here, when the automated assistant 120 is in the first state, the automated assistant 120 may transition to the second state, as before. However, the automated assistant 120 may also respond to the first registered hot command EHC1 to perform a first response action (“Response Action 1”). As indicated by the dashed arrow, in some implementations, once the first response action is performed, the automated assistant 120 may transition to the second state, for example, so that the user can follow up on the registered hot command with whatever query the user desires. This registration process may be repeated in the future. For example, FIG. 3C shows the state machine of the automated assistant 120 after another registered hot command (“EHC2”) has been registered.

[0089] 4A and 4B illustrate an example of how a human-to-computer dialog session may occur between a user 101 and an instance of an automated assistant (not shown in FIGS. 4A-4B) via a microphone and speaker (shown as a standalone interactive speaker, but this is not meant to be limiting) of a client computing device 406 according to implementations described herein. One or more aspects of the automated assistant 120 may be implemented on the computing device 406 and / or on one or more computing devices in network communication with the computing device 406.

[0090] In FIG. 4A, user 101 invokes automated assistant 120 using the default hot phrase "hey, assistant." This can transition automated assistant 120 from the first state to the second state of FIGS. 3A-3C. User 101 then says "set timer for 5 minutes." This phrase can be processed according to both the command registration analysis and the general processing of FIGS. 3A-3C. For this example, assume that user 101 (and / or, depending on the implementation, other users) have uttered this command, or a semantically compatible command (e.g., "set timer for 10 minutes"), some number of times that meets some minimum hot command registration threshold.

[0091] Automated assistant 120 may initially set a timer and respond by saying, "OK, now...start timer." However, because the command (and / or a semantically consistent command) meets the hot command registration criteria (e.g., the minimum hot command registration threshold previously mentioned), automated assistant 120 registers the command, or more generally, the template or grammar generated from the command and its slots, as a registered hot command. Immediately, the automated assistant notifies user 101, "Since you're issuing this command frequently, I've changed my settings so that you no longer have to wake me up before issuing this command...just say, 'Set the timer for X minutes.'" As a result, as shown in FIG. 4B, user 101 later simply utters the command "Set the timer for 6 minutes," and automated assistant 120 starts the timer without first being explicitly invoked.

[0092] In addition to or instead of frequency of use, in some implementations, other factors and / or signals may be used to determine whether a command should be interpreted as a hot command and / or whether to register the command as a hot command going forward. Additionally or alternatively, in some implementations, repeated utterances of a command can cause the command to be interpreted and / or registered as a hot command. Intuitively, if a user says something like "Turn the lights red," waits, and then repeats the same thing one or more times, it suggests that the user is waiting for the automated assistant 120 to turn the lights red.

[0093] 5A-5B show an example in which user 101's gaze is used to determine whether the user's utterance was meant to be directed to automated assistant 120. Again, user 101 engages automated assistant 120, which runs at least in part on client device 506C. Client device 506C includes visual sensor 507C, which generates visual data that can be analyzed to determine the direction of the user's gaze. Two other client devices, laptop 506A and smart television 506B, are also present and include respective visual sensors 507A and 507B.

[0094] In FIG. 5A , user 101 says, “Turn on the lights,” without first activating automated assistant 120. To determine the direction of the user’s gaze, visual data generated by one or more of visual sensors 507A-C may be analyzed, for example, using a gaze detection algorithm. If user 101 is looking in the direction of one of client devices 506A-C (indicated by dashed arrows in FIG. 5A ), that may be a signal used alone or in combination with other signals to determine whether a command should be responded to by automated assistant 120. In the case of FIG. 5A , automated assistant 120 responds, “OK, turn on the three lights.”

[0095] Contrast that with FIG. 5B, in which a first user 101A says to another user 101B, "Turn on the lights." As indicated by the dashed arrow, the first user 101A's gaze is directed toward the second user 101B. This can again be detected based on visual data gathered by one or more of the client devices 506A-C using visual data captured by one or more of their respective visual sensors 507A-C. As a result, the automated assistant 120 takes no action, based on the assumption that the second user 101B is instructed to turn on the lights.

[0096] In some implementations, the speaker's gaze direction may not be available or ascertainable, for example, due to the speaker's location (e.g., outside the field of view of any visual sensors 507). Or, there may not be sufficient hardware resources, such as visual sensors, to determine the speaker's gaze direction. However, signals from other sensors, such as microphones or proximity sensors, may be analyzed to determine whether multiple people are co-present in the area. As in FIG. 5B , if multiple people are present, it may weigh against interpreting what one of those people potentially says to another of those people as a command directed to the automated assistant 120.

[0097] In some implementations, text data generated from user 101's utterance can be applied as input through the aforementioned machine learning model to generate a probability that the command was directed to automated assistant 120. That probability may not be dispositive on its own—after all, “turn on the lights” is a command that another person in the room might obey—but in combination with user 101's gaze targeting a client device on which automated assistant 120 may at least be perceived to be running, the combined signal may be sufficient to trigger a response by automated assistant 120. Additionally or alternatively, rather than detecting the user's direction of gaze, some implementations may determine whether user 101 is alone. If the answer is yes, it may be inferred that user 101 is addressing automated assistant 120, even without detecting the user's direction of gaze.

[0098] 6 is a flow diagram illustrating an example method 600 according to implementations disclosed herein. For convenience, the operations of the flow diagram are described with reference to a system that performs those operations. The system may include various components of various computer systems, such as one or more components of the automated assistant 120. Furthermore, although the operations of the method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0099] At block 602, the system may operate the automated assistant 120 at least partially on one or more client devices, such as 106, 406, or 506A-C. At block 604, the system may monitor one or more trigger events. For example, the system may monitor audio data captured at one or more microphones for one or more default hot words meant to activate the automated assistant 120. As described above, in some implementations, this may require extracting and applying features of the audio data as input across one or more activation models 114. In other implementations, the system may perform speech recognition processing (also referred to herein as STT processing) on ​​the audio data capturing the speech. The resulting generated text fragment may be analyzed to determine whether the fragment contained a hot word. In some implementations, events other than the detection of a default hot word may activate the automated assistant 120, such as pressing one or more hard or soft keys, making a gesture in front of a visual sensor, or making a specific gesture or movement while holding a mobile phone accessible to the automated assistant 120.

[0100] If one or more trigger events are not detected in block 606, method 600 may return to block 604. However, if the answer in block 606 is yes, then in block 608 the system may transition the automated assistant 120 from a first state, such as a limited listening state (FIG. 2A) or a selective response state (FIG. 2B), to a second state, such as a full speech recognition state (FIG. 2A) or a full response state (FIG. 2B).

[0101] While in the second state, in block 610, the system may receive spoken commands from the user, which may be captured as audio data, for example, by voice capture module 110. In block 612, the system may perform speech recognition processing on the audio data generated from the spoken commands received in block 610 to generate text commands, for example, by cloud-based STT module 117 or local STT module 117A.

[0102] In block 614, the system may determine, for example, by the hot command engine 128, whether the text command meets a frequency threshold in a corpus of text commands (command registration analysis of FIGS. 3A-3C). In various implementations, the corpus of text commands may be a corpus of text commands generated by the speaking user or a corpus of text commands generated by a population of users, including that user. In some implementations, a count associated with the text command may be compared to a frequency threshold in block 616. This count may be determined in various ways, as described below.

[0103] If the threshold is met in block 616, the system may register the data indicating the text command as a hot command in block 618. Following registration of an utterance of another text command that matches or is semantically consistent (e.g., matches a template / grammar) in block 618, the text command may trigger the performance of a responsive action by the automated assistant 120 without requiring explicit invocation of the automated assistant 120.

[0104] In some implementations, only the same user who registered the hot command can then use the hot command to trigger an action by the automated assistant 120 without initiating it. This user can be identified across multiple interaction sessions using, for example, speaker recognition processing, facial recognition processing (e.g., when one or more cameras are present), etc. In other implementations, once a hot command is registered, the hot command can be used by other than the user who registered it, such as other registered users (e.g., family members) or users at large.

[0105] In some implementations, in optional block 620, the system may cache data indicative of the text command, such as the text command itself, at least in part in the local memory of one or more client devices 106 implementing the automated assistant 120. This may provide various benefits regarding latency for the automated assistant 120 when responding to similar commands in the future. For example, in some implementations, the text command may be triggered automatically (e.g., periodically) without the text command being explicitly invoked by a person to cache information responsive to the text command. This may be particularly useful if the text command causes the automated assistant 120 to output an informational response to a query such as "What's the weather outside?" or "What are my plans for today?" The periodically re-cached information may be output directly by the automated assistant 120 in response to subsequent invocations of the text command, instead of the text command being processed remotely, e.g., at a cloud-based automated assistant component 119, to resolve the speaker's request.

[0106] In some implementations, registered hot commands (or data indicative thereof) controlling smart appliances may be cached to improve latency when instructing automated assistant 120 to control these smart appliances in the future. Invocation of such a cached text command can cause the smart appliance to directly transmit the text command or data indicative of the text command, such as proprietary instructions configured to control a proprietary smart appliance, via one or both of the LAN and the PAN. As previously mentioned, receipt of the text command or data indicative thereof can cause the smart appliance to perform a responsive action. All of this can occur without remotely processing the text command, for example, at the cloud-based automated assistant component 119.

[0107] After block 618 (or optional block 620), method 600 may then proceed to block 622, at which point automated assistant 120 may perform a responsive action. If the answer at block 616 is no, method 600 may skip blocks 618-620 and proceed directly to block 622. In some implementations, a count associated with the command may be incremented at this point. In some implementations, the count for a particular text command may be incremented each time the command is spoken to someone or by someone who is a member of a predefined group (e.g., a family member, a registered user group). In some implementations, the count for a particular text command may be accumulated across a collaborative ecosystem of client devices (e.g., smart speakers, smartphones, tablets, assistant devices included in the home). For example, one or more users may have accounts associated with automated assistant 120, and these users may access these accounts on any client device in such an ecosystem. When these cumulative counts meet the frequency threshold of block 616, these particular text commands may be selectively registered as hot commands for the ecosystem, which may effectively make the hot commands widely available in environments where ecosystem client devices are deployed, such as within the home, in vehicles driven by family members, on mobile phones carried by family members, etc.

[0108] Similarly, in some implementations, when text commands (or templates / grammars generated therefrom) are registered as hot commands, they may be registered on an individual user basis, on a multi-user basis (e.g., available to family members and / or users registered with a particular ecosystem of client devices), or even for dissemination across a population. In some implementations in which hot commands are registered on an individual user basis, when a user utters the registered hot command, the automated assistant 120 may first attempt to verify the identity of the speaker and / or match the speaker with a registered user profile. For example, the automated assistant 120 may perform speaker recognition processing (or voice matching), facial recognition processing (if a visual sensor is available), or the like to determine the identity of the speaker or at least match the speaker with a registered voice profile. If the speaker is not recognized, in some implementations, the automated assistant 120 may ignore the registered hot command. On the other hand, if the speaker is recognized, the spoken utterance may be analyzed using techniques described herein to determine whether the automated assistant 120 should take a response action.

[0109] As mentioned above, in some implementations, hot commands are registered for multiple users, e.g., family members or users registered in a defined ecosystem of client devices (e.g., client devices and networked appliances in a home). In some such implementations, any of those registered users may be able to speak the registered hot command to a client device in the ecosystem. As long as the speaker can be matched, e.g., using voice matching or facial recognition, to a registered user profile (or more generally, registered as someone who is enabled to speak commands to automated assistant 120, even if there is no predefined profile), automated assistant 120 can take a responsive action without first being invoked.

[0110] 6, a text command is selectively registered as a hot command based on whether the frequency threshold of block 614 is met. However, this is not meant to be limiting. In other implementations, additional or alternative signals and / or techniques may be used to determine whether a text command should be registered as a hot command and / or whether the text command actually addresses the automated assistant 120.

[0111] For example, in some implementations, the text command may be applied as an input to a machine learning model, e.g., by the activation module 113, to generate an output. The output may indicate a probability that the text command was directed to the automated assistant. In some such implementations, the registration of block 616 is further responsive to the probability conveyed by the machine learning model output satisfying a threshold. Additionally or alternatively, in some implementations, the conditional registration of block 616 may be further conditioned on detecting that the user's gaze is directed toward a computing device on which the automated assistant is at least partially implemented, based on visual data captured by one or more cameras, as shown in FIG. 5A.

[0112] In yet another implementation, the conditional registration of block 616 may be further conditioned on the determination that no additional speech is detected for a predetermined time interval after the spoken command. Suppose a speaker issues a command followed by, for example, 5 to 10 seconds of silence. This may suggest that the speaker is alone and that another human conversation participant would have responded by then, and therefore, that they must be addressing the automated assistant 120. Alternatively, the momentary silence may suggest that someone else present in the room realizes that the speaker was addressing the automated assistant 120, and therefore, they remain silent.

[0113] Additionally or alternatively, in some implementations, the conditional registration of block 616 is further conditioned on a determination that no responsive action has been taken by a person within a predetermined time interval following the spoken command. Suppose a speaker says "Turn on the lights," but another person quickly activates a physical switch to turn on the lights. This suggests that the speaker is addressing another person rather than the automated assistant 120, and therefore the phrase "Turn on the lights" may not be suitable for registration as a hot command, at least at this point. On the other hand, if the speaker issues a command and nothing happens for some predetermined time interval (e.g., 5 seconds, 10 seconds), this suggests that the speaker is actually addressing the automated assistant 120, and therefore the text command "Turn on the lights" should be registered as a hot command.

[0114] FIG. 7 illustrates an exemplary method 700 for practicing selected aspects of the present disclosure according to various implementations, including an implementation as shown in FIG. 2B in which STT / speech recognition processing is performed on utterances captured outside of a full listening state (FIG. 2A) or a full response state (FIG. 2B). For convenience, the operations of the flowchart are described with reference to a system that performs these operations. This system may include various components of various computer systems. Moreover, while the operations of method 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0115] At block 702, the system may receive audio data captured at one or more microphones. The audio data may include multiple spoken utterances captured over time, such as first and second spoken utterances. At block 704, the system may perform speech recognition processing (i.e., STT processing) on ​​the audio data to generate first and second text fragments corresponding to the first and second spoken utterances.

[0116] In block 706, the system may determine that the first text fragment satisfies one or more hot command criteria. These "hot command criteria" may be similar to the "hot command registration criteria" described herein. For example, the hot command criteria may include, but are not limited to, whether the speaker's gaze is directed toward the assistant device while speaking, whether anyone else is present with the speaker, whether the speaker looks at another person rather than at the computing device while speaking, whether the utterance is followed by some predetermined interval of silence and / or inaction by a person, whether the text fragment is spoken some threshold number of times while the automated assistant 120 is in the fully listening (FIG. 2A) or fully responding (FIG. 2B) state, etc.

[0117] At block 708, the system may determine that the second text fragment fails to satisfy one or more hot command criteria. At block 710, in response to the determination of block 706, the system may perform a responsive action based on the first text fragment. At block 712, in response to the determination of block 708, the system may discard or ignore the second text fragment without performing another responsive action.

[0118] In situations where some implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, as well as user actions and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only upon receiving explicit authorization to do so from the associated user.

[0119] For example, a particular user may be provided with control over whether a program or feature collects user information about that user or other users associated with that program or feature. Each user whose personal information is to be collected is presented with one or more options to provide permission or authorization regarding whether information is collected and what portions of the information should be collected, enabling control over the collection of information related to that user. For example, one or more such control options may be provided to the user via a communications network. Additionally, certain data may be treated in one or more ways before the data is stored or used so that personally identifiable information is removed. As one example, a user's identifying information may be treated so that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0120] 8 is a block diagram of an exemplary computing device 810 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the user-controlled resource engine 134, and / or other components may comprise one or more components of the exemplary computing device 810.

[0121] Computing device 810 typically includes at least one processor 814 that communicates with several peripheral devices via a bus subsystem 812. These peripheral devices may include, for example, a storage subsystem 824, including a memory subsystem 825 and a file storage subsystem 826, a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices allow a user to interact with computing device 810. Network interface subsystem 816 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0122] The user interface input devices 822 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 810 or over a communications network.

[0123] The user interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 810 to a user or to another machine or computing device.

[0124] Storage subsystem 824 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 824 may include logic for performing selected aspects of the methods of Figures 6 and 7, as well as for implementing the various components shown in Figure 1.

[0125] These software modules are generally executed by processor 814, alone or in combination with other processors. The memory 825 used in storage subsystem 824 may include several memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution, and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 may provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of some implementations may be stored by file storage subsystem 826 within storage subsystem 824 or other machines accessible by processor 814.

[0126] Bus subsystem 812 provides a mechanism for allowing the various components and subsystems of computing device 810 to communicate with each other as intended. Although bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0127] Computing device 810 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 810 shown in Figure 8 is intended only as a specific example to illustrate some implementations. Many other configurations of computing device 810 are possible, having more or fewer components than the computing device shown in Figure 8.

[0128] While several implementations have been described and illustrated herein, various other means and / or structures may be utilized to perform the functions and / or obtain one or more of the results and / or advantages described herein, and each such variation and / or modification is considered within the scope of the implementations described herein. More generally, it is intended that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications in which the teachings are used. Those skilled in the art will recognize and be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. Accordingly, it should be understood that the foregoing implementations are presented by way of example only, and that, within the scope of the appended claims and their equivalents, implementations may be practiced other than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0129] 101 users 101A First User 101B Another user, second user 105 Proximity Sensor 106 Client Computing Device, Client Device, Cloud-Based or On-Board Client Device 107 Visual sensors, cameras 108 Automated Assistant Client 109 Microphone 110 Audio Capture Module 111 Image Capture Module 112 Visual Cue Module 1121 Visual Cue Module 1122Cloud-based visual cue module,Visual cue module 113 Startup Module 114 On-device model database, on-device launch model 1141 Default on-device boot model 1142~114 N Contextual Launch Model 116 Cloud-based Text-to-Speech (“TTS”) Module 117 Cloud-based STT Module,STT Module,Components 117A Onboard STT Module, STT, STT Module 119 Cloud-Based Automated Assistant Components 120 Automated Assistants 122 Natural Language Processor 124 Realization Modules, Components 126 Natural Language Generator 128 Hot Command Engine, Dynamic Hot Word Engine 129 databases 130 Third Party Computing Services; Third Party Services 134 User-Controlled Resource Engine 135 Intention Matcher, Components 150 Search Modules 406 Client Device 506A Laptop, Client Device 506B Smart Television, Client Device 506C Client Device, Client Device 507 Visual Sensor 507A Visual Sensor 507B Visual Sensor 507C Visual Sensor 600 ways 700 methods 810 Computing Devices 812 Bus Subsystem 814 processor 816 Network Interface Subsystem 820 User Interface Output Device 822 User Interface Input Devices 824 Memory Subsystem 825 Memory Subsystem, Memory 826 File Storage Subsystem 830 Main Random Access Memory (RAM) 832 Read-Only Memory (ROM)

Claims

1. 1. A method implemented using one or more processors, comprising: using one or more of the processors to operate an automated assistant; transitioning the automated assistant from a limited listening state to a full speech recognition state in response to a trigger event; receiving, by the automated assistant while in the full speech recognition state, a spoken command from a user; performing a speech recognition process on the spoken command to generate a text command; applying the text command as an input to a machine learning model to generate an output, the output indicating a probability that the text command was directed to the automated assistant; determining that the text command satisfies a frequency threshold within a corpus of text commands; registering data representing the text command as a hot command based on the determination in response to the probability satisfying a threshold, wherein subsequent to the registration, utterance of another text command that is semantically consistent with the text command triggers performance of a responsive action by the automated assistant without requiring explicit invocation of the automated assistant; A method comprising:

2. The method of claim 1 , wherein the trigger event comprises the detection of one or more default hot words in audio data captured by a microphone.

3. The method of claim 1 , wherein the corpus of text commands comprises a corpus of text commands generated by the user.

4. The method of claim 1 , wherein the corpus of text commands comprises a corpus of text commands generated by a population of users, including the user.

5. 10. The method of claim 1, further comprising detecting, based on visual data captured by one or more cameras, that the user's gaze is directed toward a computing device on which the automated assistant is at least partially implemented, and wherein the registration is further responsive to the detection.

6. The method of claim 1 , wherein the registering step is further responsive to determining that no additional speech has been detected during a predetermined time interval after the spoken command.

7. The method of claim 1 , wherein the registering step is further responsive to a determination that no responsive action has been taken by a person during a predetermined time interval following the spoken command.

8. 10. The method of claim 1, further comprising, in response to the registering step, caching data indicative of the text command or the text command itself in a local memory of a computing device that at least partially implements the automated assistant.

9. 9. The method of claim 8, further comprising: triggering the text command based on the cached data following the registering step without the text command being explicitly invoked to cache information responsive to the text command, wherein following the triggering step, the cached information is output in response to a subsequent invocation of the text command instead of the text command being triggered.

10. 9. The method of claim 8, wherein, following the caching step, initiation of the text command causes the smart appliance to transmit the data indicative of the text command directly over one or both of a local area network ("LAN") and a personal area network ("PAN"), and receipt of the data indicative of the text command causes the smart appliance to perform a responsive action.

11. 2. The method of claim 1, wherein the data indicative of the text command includes a template including one or more slots for receiving one or more subsequent parameters that are semantically consistent with one or more parameters provided in the spoken command received from the user.

12. At least one non-transitory computer-readable medium comprising instructions, the instructions causing one or more processors, in response to execution of the instructions by the one or more processors, to: using one or more of the processors to operate an automated assistant; transitioning the automated assistant from a limited listening state to a full speech recognition state in response to a trigger event; receiving, by the automated assistant while in the full speech recognition state, a spoken command from a user; performing a speech recognition process on the spoken command to generate a text command; an act of applying the text command as an input to a machine learning model to generate an output, the output indicating a probability that the text command was directed to the automated assistant; and determining that the text command satisfies a frequency threshold within a corpus of text commands; and an operation of registering data indicating the text command as a hot command based on the determination in response to the probability satisfying a threshold, wherein subsequent to the registering step, utterance of another text command that is semantically consistent with the text command triggers performance of a responsive action by the automated assistant without requiring explicit activation of the automated assistant. At least one non-transitory computer-readable medium for causing the

13. 13. The at least one non-transitory computer-readable medium of claim 12, wherein the trigger event comprises detection of one or more default hot words in audio data captured by a microphone.

14. 13. The at least one non-transitory computer-readable medium of claim 12, wherein the corpus of text commands comprises a corpus of text commands generated by the user.

15. 13. The at least one non-transitory computer-readable medium of claim 12, wherein the corpus of text commands comprises a corpus of text commands generated by a population of users, including the user.

Images