Hotword-free deployment for automated assistant functionality

Abstract

The invention discloses hot word-free deployment of automated assistant functions. Sensor data from one or more sensor components of a helper device providing an automated helper interface (graphical and / or audible) is processed to determine an occurrence and / or confidence measure of various attributes of a user proximate to the helper device. Whether to deploy each of the one or more functions of the automation assistant is based on the occurrence and / or confidence of one or more of the various attributes. For example, certain processing of at least some of the sensor data may be initiated, such as initiating a previously dormant local processing of at least some of the sensor data and / or initiating a transmission of at least some of the audio data to a remote automation assistant component.

Description

Description of the case This application is a divisional application of Chinese invention patent application No. 201980022744.4, filed on May 2, 2019. Technical Field The present disclosure relates to hotword-free deployment of automated assistant functions. Background Art

[0001] A human may engage in a human-computer dialogue with an interactive software application, referred to herein as an "automated assistant" (also referred to as a "digital agent," "interactive personal assistant," "intelligent personal assistant," "assistant application," "dialog agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., speech) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed. The automated assistant responds to the request by providing responsive user interface output, which may include audible and / or visual user interface output.

[0002] As mentioned above, many automated assistants are configured to interact with them via the said speech. In order to protect user privacy and / or save resources, the user must often invoke the automated assistant in an explicit manner before the automated assistant will fully process the said speech. The explicit invocation of the automated assistant is usually performed in response to certain user interface inputs received at a client device. The client device includes an assistant interface that provides an interface to the user of the client device for interacting with the automated assistant (e.g., receiving input from the user, and providing audible and / or graphical responses) and docking with one or more additional components that implement the automated assistant (e.g., on-device components and / or remote server devices that process user inputs and generate appropriate responses).

[0003] Some user interface inputs that can invoke an automated assistant via a client device include a hardware and / or virtual button at the client device for invoking the automated assistant (e.g., a tap on a hardware button, a selection of a graphical interface element displayed by the client device). Additionally or alternatively, many automated assistants can be invoked in response to one or more specific verbal invocation phrases, also referred to as "hot words / phrases" or "trigger words / phrases." For example, a specific verbal invocation phrase such as "Hey Assistant," "OK Assistant," and / or "Assistant" can be spoken to invoke an automated assistant. Summary of the invention

[0004] Many client devices that facilitate interaction with automated assistants, also referred to herein as "assistant devices," enable users to engage with the automated assistants in a touch-free manner. For example, assistant devices often include a microphone that allows a user to provide verbal speech to invoke the automated assistant and / or otherwise interact with it. In addition or alternatively, the assistant devices described herein integrate and / or communicate with one or more visual components (e.g., cameras, light detection and ranging (LIDAR) components, radar components, etc.) to facilitate touch-free interaction with the automated assistant.

[0005] Embodiments disclosed herein relate to the hotword-free adaptation of one or more functions of an automated assistant. Those embodiments process sensor data from one or more sensor components of an assistant device that provides an automated assistant interface (graphical and / or audible) to determine the occurrence and / or confidence measure of various attributes of a user close to the assistant device. Those embodiments further determine whether to adapt each of the one or more functions based on the occurrence and / or confidence of one or more of the various attributes. For example, those embodiments can determine to initiate certain processing of at least some sensor data, such as initiating local processing of at least some sensor data that was previously dormant and / or initiating transmission of at least some audio data (and / or text recognized locally based on the audio data) to a remote automated assistant component. In addition or alternatively, after such processing and / or transmission of the data has been partially or completely started, the embodiment can determine to stop processing at least some sensor data, for example, by stopping the local processor of at least some sensor data and / or by stopping the transmission of at least some data to a remote automated assistant component. Determining to stop processing and / or transmitting data can occur after the time evolution of the determined confidence of one or more of the various attributes. For example, a determination may be made to deploy one or more functions of an automated assistant based on initial sensor data associated with one or more first attributes. However, when additional sensor data becomes available, such as sensor data associated with one or more additional attributes, a determination may be made to cease deploying one or more functions of the assistant. This may reduce usage of computing resources at an assistant device, as well as usage of a data transmission network between the assistant device and a remotely located computing component, in the event that an initial positive determination as to whether to begin processing and / or transmitting at least some data is subsequently determined to no longer be correct due to a confidence metric evolving over time.

[0006] The various attributes utilized may include, for example, one or more (e.g., all) of the following: (1) a user's gaze directed toward the assistant device (also referred to herein as "directional gaze"); (2) a user's distance from the assistant device; (3) a user's mouth movement (also referred to herein as "mouth movement"); (4) voice activity; (5) a user's mouth movement and voice activity occurring simultaneously; (6) facial recognition; (7) speaker recognition; (8) presence recognition; (9) a user's body posture relative to the assistant device (e.g., facing toward the assistant device); and / or (10) a user's contactless gesture. For example, initiation of certain processing may be responsive to a determination made locally on the assistant device that a user's gaze is directed toward the assistant device, and optionally further based on one or more of the following: a user's distance from the assistant device, detected voice activity, detected mouth movement of the user (optionally occurring simultaneously with voice activity), and / or other detected attributes and optionally corresponding confidence levels. Additional descriptions are provided herein of determining each of these attributes and optionally associated confidence metrics, and utilizing the attributes and multiple confidence metrics in determining whether to deploy one or more assistant functions.

[0007] Typically, various attributes are utilized when determining whether a user intends to interact with an automated assistant via an assistant device. For example, it is determined whether a user intends to interact via the user's verbal input detected by the microphone of the assistant device and / or via the user's contactless gesture input detected by the visual component of the assistant device. The deployment that occurs may include one or more deployments that promote such interactions, such as the initiation of certain local and / or remote processing of sensor data, the transmission of certain sensor data for facilitating the interaction, and / or the provision of human-perceivable prompts for providing feedback about the interaction to the user. The technology disclosed herein enables the interaction between the user and the automated assistant to be initiated and / or guided, without the user having to start such an interaction by speaking a hot word. This enables the user input to be provided by the user to be reduced (at least due to the omission of the hot word), which directly shortens the duration of the interaction and thus saves various network resources and / or remote and / or local processing resources that would otherwise be utilized in the extended interaction. In addition, hot words are often unnatural prescribed phrases that are difficult to speak. This embarrassment can increase the friction of the user-assistant interaction and / or further prolong the interaction.

[0008] As mentioned above, in some embodiments, the deployment that may occur based on the occurrence and / or confidence measure of one or more attributes may include the initiation of certain processing of certain sensor data (e.g., audio data, video, image, etc.), which has not been performed before (i.e., it was previously dormant). For example, before the occurrence and / or confidence of one or more of the various attributes, the automated assistant may only perform limited processing (or no processing) of certain sensor data such as audio data, video / image data, etc.

[0009] For example, before deployment, the automated assistant can optionally process the audio data at a minimum locally (e.g., when monitoring the occurrence of voice activity and / or the occurrence of explicit call phrases), and / or can temporarily buffer the audio data locally. Before deployment, the automated assistant will "discard" the audio data after local processing and / or temporary buffering, and will not cause the audio data to be processed by one or more additional components of the automated assistant. Such additional components may, for example, include a local and / or remote speech-to-text processor, a local and / or remote natural language understanding component, a local and / or remote component that verifies that the speech included in the audio data is intended for an automated assistant (also referred to herein as "semantic filtering") and / or from a certain speaker, such as a speaker matched with a speaker visually identified based on face recognition (also referred to herein as "text-independent speaker identification / recognition"). However, in response to determining the occurrence and / or confidence of one or more of the various attributes, the deployment may include causing the audio data to be processed by one or more additional components. In these and other ways, processing and / or network resources may be reduced by transmitting audio data and / or performing certain processing of audio data only in response to determining that certain conditions have occurred.

[0010] Moreover, for example, before deployment, this automated assistant can optionally process visual data only for a certain purpose locally, such as determining the occurrence of directional gaze, the simultaneous occurrence of mouth movement and voice activity, face recognition and / or corresponding confidence measure. Before such deployment, this automated assistant can avoid certain processing of visual data, such as avoiding any visual data being transmitted to the component of the remote automated assistant and / or avoiding generating responses for any gesture of the user captured in such visual data. However, in response to determining the occurrence and / or its confidence of one or more of the various attributes, such further processing of visual data can be performed. In these and other ways, processing and / or network resources can be reduced by only transmitting visual data and / or performing certain processing of visual data in response to the occurrence of certain conditions.

[0011] In addition, in many embodiments, the occurrence and / or confidence measure of the attribute is determined based on the sensor device of the visual component (camera) of the assistant device, the microphone of the assistant device and / or other sensor components of the assistant device and is performed at the assistant device. In addition, in some of those embodiments, the assistant device determines whether to deploy the function of the automated assistant locally based on the occurrence and / or confidence measure determination. Therefore, in those embodiments, the determination of whether to deploy the function of the automated assistant can be performed locally on the assistant device without any sensor data from the assistant device and / or other data from the assistant device being transmitted to one or more remote automated assistant components through the network. On the contrary, such a transmission can optionally be one of the functions of the deployed automated assistant and can occur only in response to the local determination of deploying such a function. In addition to maintaining its security by preventing the transmission of sensor data and / or other local assistant device data unless certain conditions are met, such an embodiment further saves network resources and resources of remote devices through such local determination. Further, such an embodiment enables a determination to be made quickly without the additional delay that originally existed due to client-server transmission. Such reduced delays can also reduce the overall duration of user-assistant interaction.

[0012] More generally, the various embodiments described herein can provide efficiency in computing resources and communication networks used to implement automated assistants. For example, as will be clear from the description herein, multiple aspects of the embodiments can produce more selective initiation of communications on data networks and a corresponding reduction in data traffic on the network. More selective initiation of network communications, for example, network communications from assistant devices, can further lead to more efficient use of computing resources at remote devices with which the communications are initiated, because various potential communications from assistant devices are filtered before any contact with the remote system is initiated. Similarly, the reduction of data traffic on the network and / or more efficient use of computing resources at assistant devices and / or remote systems can be achieved by determining at the assistant device to stop processing at least some data at the assistant device and / or stop transmitting at least some data over the network. As discussed above, as more sensor data is received and processed at the assistant device, such a determination can be made based on an evolving confidence measure associated with one or more attributes. Improved efficiency in the use of computing resources on data networks and remote systems can result in significant savings in power usage of transmitters and receivers in the network and in memory operations and processing usage at remote systems. As described below, corresponding effects can also be experienced at the assistant device. These effects - especially over time and with the ongoing operation of the automated assistant - allow the devices and systems in the network and in the computing device that include the running assistant to experience significant additional capabilities as a whole. This additional performance can be used for additional communications in the data network, whether or not it is related to the assistant, without the need to expand network performance, for example, through additional or upgraded infrastructure and additional computing operations in the computing device. Other technical improvements will be clear from the description herein.

[0013] In some embodiments, the deployment of the automated assistant that occurs may include the deployment of the rendering of the user interface output. In some of those embodiments, the deployment of the rendering of the user interface output by the assistant device may include the rendering of one or more human-perceptible prompts. The rendering of the one or more human-perceptible prompts may optionally be provided before further deployment of the automated assistant, and may (directly or indirectly) indicate that further deployment is about to occur. Providing human-perceptible prompts in this way can inform the user of the state of his / her interaction with the assistant. This enables the user to confirm the state of the automated assistant so as to ensure that it is processing the said speech and / or contactless gestures from the user if necessary, and also enables the user to redirect his / her gaze and / or provide a clear termination input if the user does not intend such processing. If the user redirects his / her gaze or provides a clear termination input, further deployment of the assistant's functions can be terminated or prevented, thereby saving various resources. In some embodiments, the human-perceptible prompt includes (or is limited to) visual cues from the display of the assistant device, which can prevent any verbal input of the user from being interrupted, thereby making the user's interaction with the assistant more concise.

[0014] As an example, a first human-perceptible cue may be rendered in response to detecting a directional gaze of a user (e.g., detecting a user gaze for at least a threshold duration and / or a threshold percentage of consecutive image frames). The first human-perceptible cue may (directly or indirectly) notify the user that a directional gaze has been detected and that further processing of certain sensor data may be performed (e.g., if the user does not redirect his / her gaze, issue a visual and / or verbal stop command, or otherwise indicate a desire to terminate further processing). The first human-perceptible cue may be rendered prior to further adjustments, such as certain processing of visual and / or sensor data and / or transmission of visual and / or sensor data. A non-limiting example of a first human-perceptible cue is a glow rendered on a display, such as a white glow on all or part of a display (e.g., a white glow around the periphery of a display).

[0015] Continuing with this example, a second human-perceptible prompt can be rendered in response to continuing to detect the directional gaze and, optionally, based on determining the occurrence and / or confidence measure of additional attributes, such as the user being within a threshold distance of the assistant device (e.g., within 4 feet or other threshold as estimated based on image analysis), the presence of voice activity, and / or the simultaneous occurrence of mouth movement and voice activity. The second human-perceptible prompt can (directly or indirectly) notify the user that voice activity has been detected and that further processing of certain sensor data can be performed. The second human-perceptible prompt can be rendered prior to further adjustments, such as certain processing of visual and / or sensor data and / or transmission of visual and / or sensor data. A non-limiting example of a second human-perceptible prompt is a bar rendered on a display, such as a white bar rendered vertically or horizontally across a portion of a display, either completely or partially. Optionally, the width of the bar, the color of the bar, the brightness of the bar and / or other characteristics of the bar can depend on the number of additional attributes and / or the magnitude of their confidence metrics, thereby indicating to the user the degree of confidence in the determined user intent to interact with the assistant, and the corresponding degree of confidence that further processing can be performed.

[0016] Still continuing with the example, a third human-perceptible prompt can be rendered in response to detecting the end of voice activity (e.g., through end point detection local to the assistant device) and / or other conditions. The third human-perceptible prompt can (directly or indirectly) notify the user that further processing of certain sensor data is ongoing (but optionally can still be aborted). Such further processing can include speech-to-text processing of audio data, natural language understanding of the converted text, gesture recognition based on processing of visual data, and / or implementation based on natural language understanding and / or gesture recognition. In addition or alternatively, such further processing may include additional processing to confirm that any speech and / or gestures included in the additional sensor data are intended for the automated assistant and should be implemented. A non-limiting example of a third human-perceptible prompt is a multi-colored bar or arc rendered on a display, such as a multi-colored arc (e.g., a rainbow) rendered on a portion of the display.

[0017] Similarly, one or more further deployments can be performed after the rendering of human-perceptible prompts and in response to the detection of the continuous occurrence of attributes and / or confidence measures and / or the occurrence of additional attributes and / or confidence measures. For example, the further deployment can include certain sensor data generated by one or more sensor components of the client device transmitted by the client device to one or more remote automation assistant components (but the sensor data deployment from the sensor component will not be transmitted before the further deployment). The certain sensor data can, for example, include visual and / or audio data captured after determining the occurrence of directional gaze and / or other attributes, or visual and / or audio data captured before or during such occurrence (for example, such visual and / or audio data can be temporarily buffered at the client device, and only in response to determining the occurrence and / or confidence is used). By providing human-perceptible prompts, users can be warned that further deployment is about to occur, and are provided with the opportunity to prevent the further deployment. For example, in the case where further deployment depends on the user's continuous directional gaze, the user can shift his / her gaze to prevent the further deployment (for example, if the user does not want to interact with the automation assistant and cause the sensor data to be transmitted). In this way, the further deployment, along with the use of network and / or computing resources resulting from the further deployment, can be prevented. Various human-perceptible cues can be provided, such as an audible "ding," an audible "verbal output" (e.g., "Looks like you're talking to the Assistant, look away if you don't want to"), a visual symbol on the assistant device's display screen, an LED of the assistant device lighting up, and the like.

[0018] In some embodiments, in addition or alternatively, the provisioning of the rendering of the user interface output includes: reducing the volume of the audible user interface output rendered by the assistant device, and / or terminating the audible user interface output and / or the video output visually rendered by the assistant device. As an example, assume that a directional gaze and / or other attribute is detected before the user begins to speak a speech directed to the automated assistant. It is further assumed that before the directional gaze and / or other attribute is detected, the assistant device is rendering audible and / or visual content. For example, the automated assistant client of the assistant device may be causing the audible rendering of a song and the visual rendering of a video of the song. In response to detecting the directional gaze and / or other attribute, the automated assistant client may reduce the volume of the audible rendering of the song (while still continuing the audible rendering and the visual rendering of the video at the reduced volume). The reduction in volume can improve the performance of the processing of audio data that captures the speech, such as audio data captured via one or more microphones of the assistant device. For example, speech-to-text processing of audio data may be improved due to reduced volume, voice activity detection (VAD) based on audio data may be improved due to reduced volume, speaker voice diarization based on audio data may be improved due to reduced volume, etc. Improved audio data processing may increase the likelihood that the automated assistant correctly interprets the spoken speech and responds in an appropriate manner. This may result in improved user-assistant interaction and / or mitigate the risk of inappropriate automated assistant responses that cause the user to repeat the spoken speech (and thereby require computing resources to be expended to process the repeated spoken speech and generate and render another response).

[0019] As a variation of the above example, instead of a volume reduction, the adjustment may include suspending the audible rendering of the song (and optionally the video). As another variation of the above example, the adjustment may initially include reducing the volume of the audible rendering of the song, and the adjustment may further include subsequently suspending the audible rendering of the song in response to the occurrence and / or confidence of one or more other attributes.

[0020] In some embodiments, the deployment initiated includes gesture recognition based on visual data processing and / or implementation based on the recognized gesture. For example, in some embodiments, gesture recognition and / or implementation based on the recognized gesture (e.g., generating a response) can depend on the occurrence of directional gaze detected from the same user from which the gesture is recognized. In some of those or other embodiments, the response to the gesture is based on the user's gesture and optionally based on the content rendered by the client device at the time of the gesture. For example, a gesture can be assigned to multiple responsive actions and a single one of the responsive actions can be selected as a response based on the content being rendered by the device at this time. For example, a thumbs-up gesture can be assigned to an "increase volume" action and a "scroll up" action, and which of the two actions is selected can be based on the content being rendered. For example, when audio content and / or audio-visual content is being rendered, the "increase volume" action can be selected, and when no audio or audio-visual content is rendered and content affected by the previous "scroll down" action is being rendered, the "scroll up" action can be selected. As another example, a "stop" gesture can be assigned to a "stop timer alarm" action, a "stop music" action, a "stop call" action, and / or other actions - and which one is selected can depend on what the assistant device and / or other assistant devices are currently rendering. For example, if the assistant device is currently rendering a timer alarm, the "stop timer alarm" action can be selected. More generally, various recognized gestures can be interpreted differently based on what the assistant device at which the gesture is detected is rendering and / or optionally based on what is being rendered at other assistant devices. In addition, recognizing a gesture and / or taking action based on a gesture can optionally depend on determining the occurrence of directional gaze from the gesturing user, determining that the gesturing user is within a threshold distance of the assistant device, and / or other considerations.

[0021] In some embodiments, the response can be generated even independently of any gestures, spoken speech, and touch input by the user, but depends on the content being rendered by the client device (or another client device in the ecosystem of the client device) and one or more of the non-gesture user attributes discussed herein. For example, if the assistant device is currently rendering a timer alarm, the "stop timer alarm" action can be performed in response to the following: the timer alarm is currently being rendered and the user's directional gaze is determined (e.g., directional gaze that reaches at least 1 second or other threshold duration), the user is determined to be within a threshold distance of the assistant device (e.g., within 8 feet or other threshold distance), the user is determined to be moving closer to the assistant device and / or other considerations. Therefore, the determined attributes can enable a given action to be implemented when those determined attributes occur simultaneously with certain conditions at the assistant device, such as the rendering of a certain output by the assistant device (e.g., the rendering of a timer alarm). More generally, various combinations of determined attributes can be interpreted so that the corresponding assistant actions are only performed in some situations, such as situations where certain corresponding content is being rendered by the assistant device that determines the attribute.

[0022] In some embodiments, the client device utilizes a trained machine learning model (e.g., a neural network model) stored locally on the assistant device to at least selectively process at least a portion of the sensor data from the sensor components of the client device (e.g., image frames from a camera of the client device, audio data from a microphone of the device) when determining the occurrence of various attributes and / or their confidence (if any). For example, in response to detecting the presence of one or more users (e.g., via a presence sensor), the client device may utilize the locally stored machine learning model to process at least a portion of the visual data while monitoring the occurrence of directional gaze of the user, determining the distance of the user, determining the simultaneous occurrence of mouth movement and voice activity, determining voice activity, performing facial recognition, and / or determining the occurrence of other attributes. The client device may detect the presence of one or more users using a dedicated presence sensor (e.g., a passive infrared sensor (PIR)), using visual data and a separate machine learning model (e.g., a separate machine learning model trained specifically for human presence detection), and / or using audio data and a separate machine learning model (e.g., a VAD using a VAD machine learning model). In embodiments where processing of visual data and / or audio data upon determining the occurrence of an attribute is dependent upon first detecting the presence of one or more users, power resources may be conserved by discontinuous processing of visual data and / or audio data upon monitoring the occurrence of the attribute. Conversely, in those embodiments, processing may be performed only in response to detecting the presence of one or more users in the environment of the assistant device via one or more lower power technologies.

[0023] In some embodiments where a local machine learning model is used to monitor the occurrence of directional gaze, mouth movement (optionally occurring simultaneously with voice movement), distance, facial recognition, and / or gestures, different models may be used, each of which monitors the occurrence of one or more different attributes. In some variations of those embodiments, one or more "upstream" models (e.g., object detection and classification models) may be used to detect parts of visual data (e.g., images) that may be faces, eyes, mouths, and the like - and those parts processed using corresponding machine learning models. For example, the face and / or eye portions of an image may be detected using an upstream model and processed using a gaze machine learning model. Also, for example, the face and / or mouth portions of an image may be detected using an upstream model and processed using a mouth movement (optionally occurring simultaneously with voice movement) machine learning model. As yet another example, the human portion of an image may be detected using an upstream model and processed using a gesture machine learning model.

[0024] In some embodiments, certain portions of a video and / or image may be filtered / ignored / given less weight when detecting the occurrence of one or more attributes. For example, a television captured in a video and / or image may be ignored to prevent erroneous detection of a person rendered by the television (e.g., a weather forecaster). For example, a portion of an image may be determined to correspond to a television based on a separate object detection / classification machine learning model in response to detecting a certain display frequency in the portion over multiple frames for the portion (i.e., matching the television refresh rate), etc. Such a portion may be ignored in certain techniques described herein to prevent detection of those various attributes from a television or other video display device. As another example, a photo frame may be ignored. These and other techniques may reduce the false positive deployment of automated assistants, which may save various computing and / or network resources that would otherwise be consumed in false positive deployment. Moreover, in various embodiments, once a location of a television, photo frame, etc. is detected, it may optionally continue to be ignored in multiple frames (e.g., while performing intermittent verification until movement of a client device or object is detected, etc.). This may also save various computing resources.

[0025] Some embodiments disclosed herein include one or more computing devices including one or more processors, such as a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU). The one or more processors are operable to execute instructions stored in an associated memory, and the instructions are configured to enable any of the methods described herein to be performed. The computing device may, for example, include a client assistant device having a microphone, at least one display, a visual component, and / or other sensor components. Some embodiments also include one or more non-transitory computer-readable storage media that store computer instructions that can be executed by one or more processors to implement any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0027] Figure 2A and 2B Example process flows are depicted that demonstrate various aspects of the disclosure, in accordance with various implementations.

[0028] Figure 3A An example of an assistant device and a user providing directional gaze and speaking (thereby causing mouth movement) is depicted, and images captured by a camera of the assistant device while the user provides directional gaze and speaks are also depicted.

[0029] Figure 3B1 ,3B2 3B3 provide examples of visually perceptible cues that may each be provided in response to determining the occurrence and / or confidence of a corresponding attribute.

[0030] Figure 4 A flow chart of an example method according to embodiments disclosed herein is depicted.

[0031] Figure 5 A flow chart depicts another example method according to embodiments disclosed herein.

[0032] Figure 6 A flow chart depicts another example method according to embodiments disclosed herein.

[0033] Figure 7 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0034] Figure 1 An example environment in which the techniques disclosed herein may be implemented is illustrated. The example environment includes a client device 101 that at least selectively executes an automation assistant client 110. The term "assistant device" is also used herein to refer to the client device 101 that at least selectively executes the automation assistant client 110. One or more cloud-based automation assistant components 130 may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the client device 101 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 100. The cloud-based automation assistant components 130 may be implemented, for example, via a cluster of high-performance servers.

[0035] In various embodiments, an instance of automated assistant client 110, through its interaction with one or more cloud-based automated assistant components 130, can form what appears to be a logical instance of automated assistant 120 from a user's perspective, with which the user can perform human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction). Figure 1101. Thus, it should be understood that each user that interfaces with an automated assistant client 110 executing on a client device 101 can actually interface with its own logical instance of automated assistant 120. For reasons of clarity and conciseness, the term "automated assistant" as used herein to "serve" a particular user will refer to the combination of the automated assistant 110 executed on the client device 101 operated by the user and optionally one or more cloud-based automated assistant components 130 (which can be shared among multiple automated assistant clients 110). It should also be understood that in some embodiments, the automated assistant 120 can respond to certain requests from any user, regardless of whether the user is actually "served" by that particular instance of the automated assistant 120.

[0036] Client device 101 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a stand-alone interactive speaker (which may include a visual sensor in some cases), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality or augmented reality computing device). Additional and / or alternative client computing devices may be provided. As previously mentioned, client device 101 may optionally take the form of an assistant device that is primarily designed to facilitate interaction between a user and automated assistant 120 (e.g., a stand-alone interactive device with a speaker, a microphone, a camera (and / or other visual components), and a display).

[0037] The client device 101 may be equipped with one or more visual components 107 having one or more fields of view. The visual component 107 may take various forms, such as a monoscopic camera, a stereo camera, a LIDAR component, a radar component, etc. The one or more visual components 107 may be used, for example, by a visual capture module 114 to capture visual frames (e.g., image frames (still images or videos)) of the environment in which the client device 101 is deployed. The client device 101 may also be equipped with one or more microphones 109. The voice capture module 112 may be configured to capture the user's voice and / or other audio data captured via the microphone 109.

[0038] The orchestration engine 115 processes the visual data captured by the visual capture module 114 and / or the audio data captured by the voice capture module 112 in determining the occurrence and / or confidence measures of one or more user attributes. In addition, the orchestration engine 115 determines whether to orchestrate one or more aspects of the automated assistant 120, such as aspects of the automated assistant client 110 and / or aspects of the cloud-based automated assistant components 130, based on the occurrence and / or confidence measures of the attributes. Such orchestration may, for example, include the orchestration of user interface outputs (e.g., audible and / or visual) rendered by the client device 101 and controlled by the automated assistant client 110. In addition or alternatively, such orchestration may, for example, include the orchestration of sensor data processing performed by the client device 101 (e.g., by one or more components of the automated assistant client) and / or by one or more cloud-based automated assistant components 130.

[0039] The client device 101 may also include one or more presence sensors 105 and / or one or more displays 103 (e.g., touch-sensitive displays). The display 103 may be one of the user interface output components in which the user interface output is rendered in various embodiments. The display 103 may further be one of the user interface output components through which the visual portion of the response from the automated assistant client 110 is rendered. The presence sensor may be, for example, a PIR and / or other passive presence sensor. In various embodiments, one or more components and / or functions of the automated assistant client 110 may be activated in response to detecting human presence based on the output from the presence sensor 105. For example, the visual capture module 114 and / or the voice capture module 112 may be activated only in response to the detection of human presence, and may optionally be deactivated in response to no longer detecting human presence. Moreover, for example, the visual component 107 and / or the microphone 109 may be activated only in response to the detection of human presence, and may optionally be deactivated in response to no longer detecting human presence. As yet another example, one or more (e.g., all) modules of the orchestration engine 115 may be activated only in response to detection of human presence, and optionally may be deactivated in response to no longer detecting human presence. In embodiments where initiation of components and / or functionality of the automated assistant client 110 is dependent upon first detecting the presence of one or more users, power resources may be conserved.

[0040] exist Figure 1 , the deployment engine 115 includes a gaze module 1151, a distance module 1152, a mouth / voice module 1153, a face recognition module 1154, a voice activity module 1155, and one or more other modules 1156.

[0041] The deployment engine 115 utilizes determinations and / or confidence measures from one or more modules in modules 1151-1156 when determining whether to deploy one or more aspects of the automated assistant 120. For example, before initiating the transmission of audio data to the cloud-based automated assistant component 130 and / or performing local speech-to-text processing of the audio data, the deployment engine 115 requires (by the gaze module 1151) the detection of the occurrence of directional gaze, and the continued occurrence of directional gaze together with the occurrence of one or more other attributes. The occurrence of the one or more other attributes, for example, includes the user distance (measured by the distance module 1152) being within a threshold distance, the user distance being closer than the distance previously determined for the user (e.g., the user has been closer to the assistant device at least to a threshold degree and / or at least at a threshold rate), the simultaneous occurrence of mouth movement and voice activity (determined by the mouth / voice module 1153), and / or the occurrence of voice activity (determined by the voice module 1155). As another example, prior to initiating transmission of audio data to the cloud-based automated assistant component 130 and / or performing local speech-to-text processing of the audio data, the orchestration engine 115 can require detection of: the occurrence of directional gaze, and the continued occurrence of directional gaze along with a requirement that one or more criteria be met regarding: user distance, co-occurrence of mouth movement and voice activity, and / or the occurrence of voice activity. For example, the one or more other criteria include consideration of: distance, and a confidence measure for the simultaneous occurrence of mouth movement and voice activity. For example, the orchestration engine 115 may: initiate processing when the distance is 6 feet and the confidence measure is 0.9; initiate processing when the distance is 3 feet and the confidence measure is 0.7; and suppress initiation of processing when the distance is 6 feet and the confidence measure is 0.7. Moreover, for example, the deployment engine 115 may: initiate processing when the distance of the user relative to the assistant device is determined to be closer to the assistant device than one or more previously determined distances and the confidence metric is 0.9; initiate processing when the distance and one or more previously determined distances indicate that the user is moving closer to the assistant device at least at a threshold rate and the confidence metric is 0.7; and suppress initiation of processing when the distance and one or more previously determined distances indicate that the user is moving away from the assistant device and the confidence metric is 0.7. When considering whether to initiate further processing based on the presence of one or more attributes and / or associated confidence metrics, the deployment engine 115 may utilize one or more rules and / or trained machine learning models. For example, the machine learning model may be trained based on training examples, each of which includes training example inputs having various attributes and / or confidence metrics, and corresponding labels (supervised or semi-supervised) indicating whether further processing should be performed.

[0042] The gaze module 1151 processes the visual data from the visual capture module 114 to determine the occurrence of a directional gaze, and optionally a confidence measure of the directional gaze. Figure 2B ), in some embodiments, the gaze module 1151 can process only those regions based on the detection of human and / or face regions of the visual data using an upstream detection and classification model, which can be one of the other modules 1156. The gaze module 1151 can use one or more gaze machine learning models 1161 to detect directional gaze. Such a machine learning model can be, for example, a neural network model, such as a convolutional neural network (CNN) model. The training of such a CNN model can be based on training examples, which include a visual frame (e.g., an image) as a training example input and an indication of whether the image includes a directional gaze as a training example output. For example, the training example output can be a single value indicating whether there is a directional gaze. For example, the single value can be "0" when there is no directional gaze, the single value can be "1" when there is a gaze directly pointing to the sensor capturing the image or within 5 degrees (or other range, which can depend on the size of the display 103), the single value can be "0.75" when there is a gaze within 5-10 degrees of the sensor capturing the image, and so on.

[0043] In some of those and / or other embodiments, the gaze module 1151 determines the directional gaze only when the directional gaze is detected with at least a threshold probability and / or the directional gaze is detected to reach at least a threshold duration. For example, a stream of image frames can be processed using a CNN model, and processing each frame will produce a corresponding probability that the frame includes directional gaze. The gaze module 1151 can determine that there is directional gaze only when at least X% of the image frame sequence (corresponding to the threshold duration) has a corresponding probability that meets the threshold. For example, assuming that X% is 60%, the probability threshold is 0.7, and the threshold duration is 0.5 seconds. It is further assumed that 10 image frames correspond to 0.5 seconds. If the image frame is processed and generates a probability [0.75, 0.85, 0.5, 0.4, 0.9, 0.85, 0.89, 0.6, 0.85], directional gaze can be detected because 70% of the frames indicate directional gaze with a probability greater than 0.7. In these and other ways, directional gaze can be detected even when the user briefly shifts his / her gaze direction.Additional and / or alternative machine learning models (e.g., RNN models) and / or techniques can be used to detect directional gaze that occurs for at least a threshold duration.

[0044] As described herein, in various embodiments, hotword-free interaction with the automated assistant 120 can be dependent on determining the occurrence of a user's directional gaze, and optionally based on determining that the directional gaze continues to occur throughout the user input (e.g., verbal input and / or contactless gesture input). For example, in various embodiments, the deployment engine 115 will only initiate certain processing of the visual data and / or audio data that captures the user input if the user's directional gaze is determined and continues throughout the user input (e.g., until an end point is determined).

[0045] For example, transmitting visual data and / or audio data to a cloud-based assistant component 130 in order to generate a response based on visual data and / or audio data may depend on continued directional gaze. In addition, a human-perceivable prompt may be provided to notify the user that directional gaze is being detected (optionally together with other attributes), thereby providing the user with the ability to divert his / her eyes to prevent certain processing from being initiated or to terminate certain processing (if already initiated). In these and other ways, unnecessary processing and / or transmission may be quickly terminated or prevented from occurring. Also as described herein, in various embodiments, one or more other modules of the deployment engine 115 may process corresponding sensor data and / or may limit processing to sensor data corresponding to directional gaze only when the gaze module 1151 detects directional gaze. Those embodiments also result in various efficiencies at the client device 101 as described herein.

[0046] The distance module 1152 processes the visual data and / or other sensor data from the visual capture module 114 to determine an estimated distance of the user providing the directional gaze (as determined by the gaze module 1151). As described herein (e.g., Figure 2B ), in some embodiments, the distance module 1152 may process only human and / or facial regions of the visual data based on detection of those regions using upstream detection and classification models. Further, in some additional or alternative embodiments, the distance module 1152 may process the visual data only in response to the gaze module 1151 detecting the occurrence of a directional gaze. For example, the distance module 1152 may optionally process visual data corresponding only to human and / or facial regions of a user determined by the gaze module 1151 to have a directional gaze. By using the distance module 1152 for only selective processing (e.g., only in response to the gaze module 1151 detecting the occurrence of a directional gaze) and / or processing only certain regions of the visual data (e.g., only those regions corresponding to portions that include a directional gaze), resources of the client device 101 may be saved.

[0047] The distance module 1152 may optionally use one or more distance machine learning modules 1162 to detect the distance. Such a machine learning model may be, for example, a neural network model, such as a convolutional neural network (CNN) model. The training of such a CNN model may be based on training examples, which include visual frames (e.g., images including only the face and / or body area of ​​a single user) as training example inputs and values ​​indicating the distance of the user as training example outputs. For example, the training example output may be a one-hot vector, wherein the values ​​of the vector each correspond to a different distance (or range of distances), and the "hot" heat indicates the true calibration (ground truth) distance (or range of distances). In other embodiments, the distance module 1152 may optionally determine the distance without directly utilizing the distance machine learning model 1162. For example, the distance module 1152 may be provided with visual data that includes only the facial area of ​​the user determined to correspond to a directional gaze based on the output from the upstream model and / or the gaze module 1151. Further, the distance module 1152 can estimate the distance based on the distance between the eyes of the user in the visual data (the position of the eyes can optionally be indicated by an upstream model), the head size of the user in the visual data, and / or other attributes of the user in the visual data.

[0048] The mouth / speech module 1153 can determine the occurrence of mouth movements and / or the simultaneous occurrence of mouth movements and speech activity using one or more mouth / speech machine learning models 1163. In some embodiments, the mouth / speech module 1153 only determines the occurrence of mouth movements, and optionally one or more confidence measures for the mouth movements.

[0049] In some embodiments, in addition or alternatively, the mouth / speech module 1153 determines the co-occurrence of mouth movements and speech activity and optionally one or more corresponding confidence measures. In some of those embodiments, the determined co-occurrence of mouth movements and speech activity is the co-occurrence of mouth movements and speech activity from the moving mouth. In some of those embodiments, the mouth / speech machine learning model 1163 utilized can be, for example, a neural network model, such as an RNN model including one or more memory layers. For example, the machine learning model can be trained to process visual data from the visual capture module 114 (e.g., visual data including a user's facial area) and audio data from the voice capture module 112 to generate a probability (confidence measure) of whether the user captured in the visual data is actually speaking.

[0050] Training of such a machine learning model may be based on training examples that include as training example inputs a sequence of visual frames (e.g., a sequence of images that only include a user's facial region) and a sequence of audio data frames. The sequence of audio data frames may overlap in time with the sequence of visual frames, but the sequence of visual frames may optionally have a duration different from (e.g., a shorter duration) than the sequence of audio data frames. The training example may further include as a training example output an indication of whether the sequence includes mouth movement combined with speech activity from a moving mouth. In other words, an indication of whether the user whose face is captured by the sequence of images is speaking. For example, the training example output may be a single value indicating whether the user whose face is captured by the sequence of images is speaking. For example, the single value may be "0" in the case of no and "1" in the case of yes. Note that such a model trained on both visual data and overlapping audio data can distinguish between: (1) speech activity that occurs simultaneously with and is a result of mouth movement; and (2) speech activity that is not from mouth movement but happens to occur simultaneously with non-speech mouth movement. For example, training examples may include training examples that have training example inputs that include non-speech mouth movements in visual data and voice activity in audio data (e.g., voice activity from another user, from a television, and / or from other sources), and those training examples include training example outputs that indicate that the user whose face is captured by the image sequence is not speaking. Training on such training examples can enable the trained model to be used to distinguish between: (1) voice activity that occurs simultaneously with and is a result of mouth movement; and (2) voice activity that is not from mouth movement but happens to occur simultaneously with non-speech mouth movement. This can reduce the occurrence of false positives due to extraneous voice activity (e.g., from a television) and / or other sources that happens to occur simultaneously with mouth movement from a user (e.g., a user determined by gaze module 1151 to have directional gaze), thereby reducing the occurrence of unnecessary deployment of various computationally and / or network-intensive assistant functions by deployment engine 115.

[0051] As described herein (e.g., Figure 2B) In some embodiments, the mouth / voice module 1153 may process only human and / or facial regions of the visual data based on detection of those regions using upstream detection and classification models. Further, in some additional or alternative embodiments, the mouth / voice module 1153 may process visual data only in response to the gaze module 1151 detecting the occurrence of a directional gaze. For example, the mouth / voice module 1153 may optionally process visual data corresponding only to human and / or facial regions of a user determined by the gaze module 1151 to have a directional gaze. By using the mouth / voice module 1153 to perform only selective processing (e.g., only in response to the gaze module 1151 detecting the occurrence of a directional gaze) and / or processing only certain regions of the visual data (e.g., only those regions corresponding to portions that include a directional gaze), resources of the client device 101 may be saved.

[0052] In various embodiments, the mouth / speech module 1153 may determine the occurrence of a mouth movement only when the mouth movement is detected with at least a threshold probability and / or when the mouth movement is detected for at least a threshold duration. For example, a sequence-to-sequence RNN model in the mouth / speech model 1163 may be used to process a stream of image frames and audio data frames, and processing each frame may produce a corresponding probability of simultaneous occurrence of mouth movement and speech activity (from the moving mouth) (e.g., which may be based on previous frames due to the cyclic nature of the network). The mouth / speech module 1153 may determine that there is simultaneous occurrence of mouth movement and speech activity only when at least X% of the frame sequence (corresponding to the threshold duration) has a corresponding probability that satisfies the threshold. Additional and / or alternative machine learning models and / or techniques may be used to detect mouth movement and / or simultaneous occurrence of mouth movement and speech activity.

[0053] The facial recognition module 1154 may use one or more facial recognition machine learning models 1164 to determine the presence of a recognized face and / or determine an account identifier associated with the recognized face. Figure 2B), in some embodiments, the facial recognition module 1154 may process only human and / or facial regions of the visual data based on detection of those regions using upstream detection and classification models. Further, in some additional or alternative embodiments, the facial recognition module 1154 may process the visual data only in response to the gaze module 1151 detecting the occurrence of a directional gaze. For example, the facial recognition module 1154 may optionally process visual data corresponding only to human and / or facial regions of a user determined by the gaze module 1151 to have a directional gaze. By using the facial recognition module 1154 to perform only selective processing (e.g., only in response to the gaze module 1151 detecting the occurrence of a directional gaze) and / or processing only certain regions of the visual data (e.g., only those regions corresponding to portions including the directional gaze), resources of the client device 101 may be saved.

[0054] The face recognition module 1154 may optionally use one or more face recognition machine learning models 1164. For example, such a machine learning model may be, for example, a neural network model, such as a CNN model and / or an RNN model. In various embodiments, the image from the visual capture module 114 is processed using the face recognition machine learning model 1164 to generate one or more face embedding maps, and the face embedding maps are compared with the locally stored embedding maps of one or more user accounts / profiles registered with the client device 101. In some embodiments, the deployment of one or more functions by the deployment engine may depend on the face recognition module determining that the user providing the directional gaze is the recognized user (e.g., a user registered with the automated assistant client 110). In some additional or alternative embodiments, the account identifier associated with the recognized face may be compared with the account identifier determined based on text-independent speaker recognition, which is based on processing of audio data capturing the said speech. In some of those embodiments, the processing and / or provision of a response to the said speech may depend on a comparison indicating the same account identifier. In these and other ways, it can be verified that the spoken speech was provided by the same user who also provided the directional gaze (because facial recognition can be performed on the user providing the directional gaze). This can prevent inadvertently generating and / or providing responses to spoken speech from a user (or other source) other than the user providing the directional gaze. This text-independent speaker identification can be performed by the TI-SID module 142 (described herein), which can be implemented at the cloud-based automated assistant component 130, or alternatively implemented at the client device 101 itself.

[0055] The voice activity module 1155 can use one or more voice activity machine learning modules 1165 to determine the occurrence of voice activity and, optionally, the corresponding confidence measure of the voice activity. The voice activity module 1155 can use a machine learning model to process the audio data from the voice capture module 112 to generate a corresponding probability that the audio data includes human voice. In some embodiments, the machine learning model 1165 includes an RNN trained based on training examples, each of which includes a corresponding sequence of audio frames as input to the training example and a corresponding label indicating whether human voice is included in the sequence of audio frames as output of the training example. In some embodiments, the voice activity module 1155 can process audio data only in response to the gaze module 1151 detecting the occurrence of a directional gaze and / or in response to other determinations and / or occurrences from other modules of the orchestration engine 115.

[0056] Other modules 1156 may optionally utilize one or more other machine learning models 1166 to determine the occurrence of one or more other attributes. As a non-limiting example, other modules 1156 may include a gesture module that processes visual data from vision module 114 and uses one or more gesture models to determine the occurrence of one or more gestures (e.g., any of one or more predefined gestures) and optionally a label or other identifier of the gesture. In some embodiments, such a gesture module may process only those areas based on the detection of human and / or facial areas of the visual data using upstream detection and classification models. Further, in some additional or alternative embodiments, the gesture module may process visual data only in response to the gaze module 1151 detecting the occurrence of directional gaze. For example, the gesture module may optionally process visual data corresponding only to human and / or facial areas of a user determined by the gaze module 1151 to have directional gaze. Resources of the client device 101 can be saved by using the gesture module for only selective processing (e.g., only in response to the gaze module 1151 detecting the occurrence of a directional gaze) and / or processing only certain areas of the visual data (e.g., only those areas corresponding to the portion including the directional gaze).

[0057] As a non-limiting example of orchestration sensor data processing performed by the orchestration engine 115, visual data and / or audio data captured at the client device 101 may be processed and / or temporarily buffered only locally at the client device 101 (i.e., not transmitted to the cloud-based automated assistant component 130) prior to determining the occurrence of one or more attributes by a module of the orchestration engine 115. However, in response to determining the occurrence of an attribute and / or a corresponding confidence metric, such processing may be orchestrated by causing the audio data and / or visual data (e.g., most recently buffered data and / or data received after the determination) to be transmitted to the cloud-based automated assistant component 130 for further processing of the orchestration. Thus, there may be no need for the user to speak an explicit invocation phrase (e.g., "OK Assistant") in order for the user's spoken speech and / or the user's contactless gesture to be fully processed by the automated assistant 120, and the response content generated by the automated assistant 120 and rendered to the user.

[0058] For example, instead of the user having to say "OK Assistant, what's today's?" In an embodiment, the user may use the client device 101 to obtain today's forecast by saying "OK Assistant, turn up the heat" instead of looking at the client device 101 and speaking "what's today's forecast" only while looking at the client device 101 or in close temporal proximity (e.g., within a time threshold before and / or after). Data corresponding to the spoken speech "what's today's" (e.g., audio data capturing the spoken speech, or a textual or other semantic transformation thereof) may be transmitted by the client device 101 to the cloud-based automated assistant component 130 in response to detecting the directional gaze and one or more other conditions. The other conditions may include, for example, the simultaneous occurrence of mouth movement and speech activity during the directional gaze (caused by speaking all or part of "what's today's weather forecast"), and may optionally be based on the distance of the user during the directional gaze and / or speaking. In another example, rather than the user needing to say "OK Assistant, turn up the heat", the user may use the client device 101 to speak "what's today's weather forecast" instead of "OK Assistant, turn up the heat". Instead of saying “OK Assistant, open the garage door” to increase the temperature in his / her home via a connected thermostat, the user may look at client device 101 and say “turn up the heat” only while looking at client device 101 or close in time (e.g., within a time threshold before and / or after). In another example, rather than the user needing to say “OK Assistant, open the garage door” to open his / her garage, the user may look at client device 101 and say “open the garage door” only while looking at client device 101 or close in time (e.g., within a time threshold before and / or after).

[0059] In some embodiments, the voice capture module 112 may be further configured to convert the captured audio into text and / or other representations or embedding mappings, for example, using speech-to-text (“STT”) processing techniques. Further, a separate module of the automated assistant client 110 may be configured to perform local natural language processing on the voice representation, and optionally generate responses based on the natural language processing locally. In those embodiments, local voice processing, local natural language processing, and / or local response generation may be functions deployed by the deployment engine 115. For example, local voice processing, local natural language processing, and / or local response generation may be dormant until the deployment engine 115 determines that one or more attributes occur. However, since the client device 101 may be relatively limited in terms of computing resources (e.g., processor cycles, memory, battery, etc.), other voice inputs may be sent to a cloud-based automated assistant component 130, which may include a cloud-based STT module 132.

[0060] The cloud-based STT module 132 can be configured to utilize the nearly unlimited resources of the cloud to convert the audio data captured by the voice capture module 112 into text, which can then be provided to the natural language understanding module 135. In some embodiments, the cloud-based STT module 132 can convert the audio recording of the voice into one or more phonemes, and then convert the one or more phonemes into text. In addition or alternatively, in some embodiments, the STT module 132 can employ a state decoding diagram. In some embodiments, the STT module 132 can generate multiple candidate text interpretations of the user's speech and select a given interpretation from the candidates using one or more techniques.

[0061] The cloud-based TTS module 131 can be configured to utilize the nearly unlimited resources of the cloud to convert text data (e.g., natural language responses planned by the automated assistant) into computer-generated voice output. In some embodiments, the TTS module can provide the computer-generated voice output to the client device 101 for direct output, for example, using one or more speakers. In other embodiments, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the client device 101, and the local TTS module of the client device 101 can then convert the text data into computer-generated voice that is output locally.

[0062] The automated assistant 120 (e.g., the cloud-based automated assistant component 130) may include an intent understanding module 135, the TTS module 131 described above, and the STT module 132 described above, as well as other components described in more detail herein. In some embodiments, one or more of the modules and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. In some embodiments, one or more of the components of the automated assistant 120, such as the intent understanding module 135, the TTS module 131, the STT module 132, etc., may be implemented at least partially on the client device 101 (e.g., in combination with or without a cloud-based implementation).

[0063] The natural language processor 133 of the intent understanding module 135 processes the natural language input generated by the user via the client device 101, and can generate annotated output (e.g., in text form) for use by one or more other components of the automated assistant 120. For example, the natural language processor 133 can process free-form input in natural language generated by the user via one or more user interface input devices of the client device 101. The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0064] In some embodiments, the natural language processor 133 is configured to recognize and annotate various types of grammatical information in natural language input. For example, the natural language processor 133 may include a morphological module that can split individual words into morphemes and / or annotate the morphemes, such as using their categories. The natural language processor 133 may also include a part-of-speech tagger that is configured to annotate terms using their grammatical roles. Moreover, for example, in some embodiments, in addition and / or alternatively, the natural language processor 133 may include a dependency parser (not shown) that is configured to determine syntactic relationships between terms in a natural language such as.

[0065] In some embodiments, in addition and / or alternatively, the natural language processor 133 may include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, in addition and / or alternatively, the natural language processor 133 includes a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more context clues. In some embodiments, one or more components of the natural language processor 133 may rely on annotations from one or more other components of the natural language processor 133.

[0066] The intent understanding module 135 may also include an intent matcher 134 configured to determine the intent of the user interacting with the automated assistant 120. Figure 1 133, but in other embodiments, the intent matcher 134 may be an integral part of the natural language processor 133 (or more generally, a pipeline including the natural language processor 133). In some embodiments, the natural language processor 133 and the intent matcher 134 may together form the above-mentioned intent understanding module 135.

[0067] Intent matcher 134 may determine the user's intent using a variety of techniques, such as based on output from natural language processor 133 (which may include annotations and terms of natural language input), based on user touch input at a touch-sensitive display of client device 101, and / or based on touch-free gestures and / or other visual cues detected in visual data. In some embodiments, intent matcher 134 may access one or more databases (not shown) that include, for example, multiple mappings between grammars and responsive actions (or more generally, intents), visual cues and responsive actions, and / or touch inputs and responsive actions. For example, the grammars included in the mappings may be selected and / or learned over time and may represent common intents of users. For example, a grammar "play <artist>(play <artist>)" can be mapped to a call that makes <artist>The intent of a responsive action of causing music to be played on the client device 101 operated by the user. Another syntax "[weather|forecast] today" may be able to match to user queries such as "what's the weather today" and "what's the forecast for today?" As another example, the mapping of visual cues to actions may include "general" mappings and / or user-specific mappings that are applicable to multiple users (e.g., all users). Some examples of mappings of visual cues to actions include mappings for gestures. For example, a "wave" gesture may be mapped to an action that causes customized content (customized for the user providing the gesture) to be rendered to the user, a "thumbs-up" gesture may be mapped to a "play music" action; and a "high five" gesture may be mapped to a "routine" of automated assistant actions to be performed, such as starting a smart coffee machine, starting certain smart lights, and rendering a news overview in an audible manner. As described herein, some gesture-to-action mappings may include multiple actions mapped to a given gesture. In some of those embodiments, intent matcher 134 may select one of the plurality of actions for a given gesture based at least in part on content rendered by client device 101 at the time of the given gesture.

[0068] In addition to or as an alternative to grammars, in some embodiments, intent matcher 134 can employ one or more trained machine learning models alone or in combination with one or more grammars, visual cues, and / or touch inputs. These trained machine learning models can also be stored in one or more databases and can be trained to recognize intent, for example by embedding data indicative of user speech and / or any detected user-provided visual cues into a reduced dimensionality space and then determining which other embedding mappings (and therefore intents) are closest, for example using techniques such as Euclidean distance, cosine similarity, etc.

[0069] As in the above "play <artist>As you can see in the examples, some grammars have slots that can be filled with slot values ​​(or "parameters") (e.g. <artist>). Slot values ​​can be determined in a variety of ways. Users will often proactively provide slot values. For example, for the syntax "Order me a <topping>pizza", the user might say the phrase "order me a sausage pizza", in which case the slot <topping>Additionally or alternatively, if the user invokes a syntax that includes slots to be filled with slot values, and the user does not proactively provide the slot values, the automated assistant 120 can solicit those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?"). In some embodiments, slots can be filled with slot values ​​based on visual cues that are detected based on visual data captured by the visual component 107. For example, a user can say something like "Order me this many cat bowls" while holding up three fingers to the visual component 107 of the client device 101. Or, a user can say something like "Find me more movies like this" while picking up a DVD box of a particular movie.

[0070] In some implementations, the automated assistant 120 can facilitate (or "broker") transactions between a user and an agent, which can be an independent software process that receives input and provides responsive output. Some agents can take the form of third-party applications that may or may not operate on a computing system separate from those that operate, for example, the cloud-based automated assistant component 130. One user intent that may be identified by the intent matcher 134 is a desire to engage a third-party application. For example, the automated assistant 120 can provide access to an application programming interface (API) for a pizza delivery service. A user can invoke the automated assistant 120 and provide a command such as "I'd like to order a pizza." The intent matcher 134 can map the command to a syntax that triggers the automated assistant 120 to engage with a third-party pizza delivery service. The third-party pizza delivery service can provide the automated assistant 120 with a minimum list of slots that need to be filled in order to fulfill the pizza delivery order. The automated assistant 120 can generate a natural language output soliciting parameters for the slots and provide it to the user (via the client device 101).

[0071] The implementation module 138 can be configured to receive the predicted / estimated intent output by the intent matcher 134, and the associated slot value (provided proactively by the user or solicited from the user) and implement (or "solve") the intent. In various embodiments, the implementation (or "resolve") of the user's intent may result in various implementation information (also referred to as "responsiveness" information or data) being generated / obtained by, for example, the implementation module 138.

[0072] The implementation information may take various forms, as intent may be implemented in various ways. Assume that a user requests simple information, such as "Where were the outdoor shots of 'The Shining' filmed?". The user's intent may be determined as a search query by intent matcher 134, for example. The intent and content of the search query may be provided to implementation module 138, such as Figure 1 As depicted in , the implementation module 138 can communicate with one or more search modules 150 configured to search a document corpus and / or other data sources (e.g., a knowledge graph, etc.) for responsiveness information. The implementation module 138 can provide data indicating a search query (e.g., the text of the query, a dimensionality reduction embedding map, etc.) to the search module 150. The search module 150 can provide responsiveness information, such as GPS coordinates or other more specific information, such as "Timerline Lodge, Mt. Hood, Oregon". This responsiveness information can form part of the implementation information generated by the implementation module 138.

[0073] Additionally or alternatively, the implementation module 138 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from the intent understanding module 135, and trigger a responsive action. The responsive action may include, for example, ordering goods / services, timing a timer, setting a reminder, initiating a phone call, playing media, sending a message, initiating a routine of multiple actions, etc. In some such embodiments, the implementation information may include a slot value associated with the implementation, a confirmation response (which in some cases may be selected from predetermined responses), etc.

[0074] In addition or alternatively, the implementation module 138 can be configured to infer the user's intent (e.g., based on time of day, past interactions, etc.) and obtain responsive information for those intents. For example, the implementation module 138 can be configured to obtain a daily calendar summary for the user, a weather forecast for the user, and / or other content for the user. The implementation module 138 can further cause such content to be "pushed" for graphical and / or audible rendering to the user. For example, the rendering of such content can be a deployment in response to the deployment engine 115 detecting the occurrence of a directional gaze and / or one or more other attributes.

[0075] The natural language generator 136 may be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 136 may be configured to receive as input implementation information associated with the implementation of the intent, and to generate natural language output based on the implementation information. Additionally or alternatively, the natural language generator 136 may receive information from other sources, such as third-party applications, which the natural language generator 136 may use to compile natural language output and / or other content for rendering to a user via the automated assistant client 110.

[0076] The semantic filtering module 140 can determine whether the spoken speech and / or detected gestures are intended for the assistant 120. The implementation module 138 can utilize the output from the semantic filtering module 140 in determining whether to generate and / or provide a response in response to the user input. For example, the semantic filtering module 140 can provide a confidence metric that indicates the likelihood that the spoken speech and / or detected gestures are intended for the assistant 120, and the implementation module 138 can determine whether to generate and / or provide a response based on the confidence metric from the semantic filtering module 140. The confidence metric from the semantic filtering module 140 can optionally be considered in conjunction with one or more other confidence metrics, such as metrics from the orchestration engine 115 and / or the TI-SID module 142. Although the semantic filtering module 140 is not limited to the confidence metric from the semantic filtering module 140, the implementation module 138 can utilize the output from the semantic filtering module 140 in determining whether to generate and / or provide a response in response to the user input. Figure 1 1 is illustrated as being implemented by the cloud-based automated assistant component 130, but in various embodiments, it may additionally or alternatively be implemented as one of the modules of the orchestration engine 115 and utilized in determining whether to orchestrate one or more assistant functions.

[0077] In some embodiments, the semantic filtering module 140 utilizes one or more semantic machine learning models 141 in determining whether the spoken speech and / or detected gesture is intended for the assistant 120. In some of those embodiments, the semantic machine learning model 141 can process audio data corresponding to the spoken speech, text output from the STT module 132, and / or output from the intent understanding module 135 when determining the confidence metric that the spoken speech and / or detected gesture is intended for the assistant 120. The semantic machine learning model 141 can be trained using supervised or semi-supervised training examples. For example, an instance in which a user speaks to an automated assistant can be labeled with a first label (e.g., "1"), and an instance in which the user speaks to other people can be labeled with a second label (e.g., "0"). In general, the confidence metric should be low for spoken speech that has atypical (or non-occurring) corresponding text and / or NLU output for assistant input. For example, if the text does not conform to any pattern for assistant input, the confidence metric should be low. Additionally or alternatively, if the NLU output will not result in an assistant response, or will result in a "punt" response or just a "personality" response, then the confidence metric should be low. Furthermore, the audio data itself can indicate whether the input is intended as assistant input, as users often speak with different speech attributes (e.g., inflection, tone, intonation) when speaking to an assistant device than when speaking to another person.

[0078] The TI-SID module 142 processes the audio data containing the speech using the TI-SID machine learning model 143 to determine whether the speech is from the recognized voice and / or to determine the account identifier associated with the recognized voice. Such recognition is "text-independent" because it can be performed on various voice inputs, rather than being limited to certain voice inputs, such as those including hot words. In some embodiments, the TI-SID module 142 determines the account identifier associated with the recognized voice and compares it with the account identifier determined by the face recognition module 1154. In some of those embodiments, the TI-SID module 142 provides an indication of whether the account identifier is the same for face and voice recognition, and prevents and / or suspends certain processing and / or provision of responses when they do not match. For example, the TI-SID module 142 can prevent the implementation module 138 from generating and / or providing a response when they do not match. In these and other ways, it can be verified that the speech is provided by the same user who also provides directional gaze (because facial recognition can be performed on the user who provides directional gaze). This can prevent inadvertently generating and / or providing responses to spoken speech from a user (or other source) other than the user providing the directional gaze. Figure 1 1 as being implemented at the cloud-based automated assistant component 130, but the TI-SID module 142 may alternatively be implemented at the client device 101 as part of the provisioning engine 115 and utilized in determining whether to provision one or more assistant functions.

[0079] Reference now Figure 2A and 2B , depicts example process flows illustrating various aspects of the present disclosure according to various implementations.

[0080] First turn Figure 2A , the visual capture module 114 provides visual frames to the gaze module 1151, the distance module 1152, the face recognition module 1154, the mouth / voice module 1152, and the other modules 1156. In some embodiments, the visual capture module 114 provides a real-time stream of visual frames to one or more of those modules. In some additional or alternative embodiments, the visual capture module 114 begins providing visual frames to at least some of the modules in response to a signal from a separate presence sensor 105 indicating that a person is present in the environment with the client device 101. In some of those embodiments, the visual frames are initially provided only to the gaze module 1151 and / or other modules, and are subsequently provided to additional modules only in response to the gaze module 1151 determining that a directional gaze has occurred.

[0081] The voice capture module 112 provides audio data to the voice activity module 1144, the mouth / speech module 1153, and the other modules 1156. In some embodiments, the voice capture module 112 provides a real-time stream of audio data to one or more of those modules. In some additional or alternative embodiments, the voice capture module 112 begins providing visual frames to at least some of the modules in response to a signal from a separate presence sensor 105 indicating that a person is present in the environment with the client device 101. In some additional or alternative embodiments, audio data is provided to the various blocks only in response to the gaze module 1151 determining the occurrence of a directional gaze.

[0082] Each of modules 1151-1156 is based at least in part on the visual frame and / or audio data and optionally utilizes one or more corresponding models ( Figure 2A The corresponding attributes and / or confidence measures are provided to the deployment engine 115 to determine whether one or more deployments 201 should be performed. As described herein, the deployment engine 115 can initiate some deployments in response to the occurrence and / or confidence measures of only some attributes, and initiate other deployments in response to the occurrence and / or confidence measures of additional attributes (optionally together with the continued occurrence of other attributes). For example, the deployment engine 115 can cause a first human-perceptible prompt to be rendered only in response to directional gaze. Moreover, for example, the deployment engine 115 can cause another human-perceptible prompt to be rendered and / or perform certain processing and / or transmission of sensor data in response to the continued detection of directional gaze and the simultaneous occurrence and / or occurrence of voice activity based on distance, mouth movement and voice activity.

[0083] In some embodiments and Figure 2B As shown in , an upstream detection and classification module 1157 can optionally be provided on the client device 101 as part of the orchestration engine 115. The detection and classification module 1157 can process the visual frames from the visual capture module 114, determine one or more regions of those visual frames using one or more detection and classification modules 1167, and the one or more detection and classification modules 1167 classify the respective regions of each visual frame accordingly. For example, the detection and classification module 1157 can classify human regions (if any) corresponding to humans in each visual frame, and provide such human regions or indications of such human regions to one or more of the modules 1151-1154 for each visual frame. Also, for example, the detection and classification module 1157 can classify regions (if any) corresponding to multiple face regions in each visual frame, and provide such face regions or indications of such face regions to one or more of the modules 1151-1154 for each visual frame. Also, for example, the detection and classification module 1157 may classify the regions (if any) corresponding to eye regions of a human being in each visual frame and provide such eye regions or indications of such eye regions to one or more of the modules 1151-1154 for each visual frame. Further, the detection and classification module 1157 may optionally provide such regions only for persons detected as providing directional gaze (as determined by the gaze module 1151).

[0084] As an example, the gaze module 1151 can utilize the provided area to process only the corresponding portion of each visual frame. For example, the gaze module 1151 can "crop" the visual frame and resize it so that only those portions including human areas or face areas or eye areas are processed. In some of those embodiments, the gaze machine learning model 1161 can be trained based on the "cropped" visual frames, and the resizing can be sized to conform to the input size of such a model. In some additional or alternative embodiments, the gaze module 1151 can utilize the provided area to skip processing of some visual frames altogether (e.g., those indicated as not including human and / or face areas). In still other embodiments, the gaze module 1151 can utilize the provided area as an attention mechanism (e.g., as a separate attention input to the gaze machine learning model 1161) to focus the processing of each visual frame.

[0085] As another example, the distance module 1152 can utilize the indication of eye regions to efficiently determine the distance between those eye regions for estimating the distance of the corresponding user. As described herein, the provided eye regions can optionally be provided only for users who have provided directional gaze as determined by the gaze module 1151.

[0086] As yet another example, the mouth / speech module 1153 can utilize the provided regions to process only the corresponding portions of each visual frame. For example, the mouth module 1153 can "crop" the visual frame and resize it to process only those portions that include a human region or a face region or a mouth region. In some of those embodiments, the mouth / speech module machine learning model 1163 can be trained based on the "cropped" visual frame, and the resizing can be sized to conform to the input size of such a model. In some additional or alternative embodiments, the mouth / speech module 1153 can utilize the provided regions to skip processing of some visual frames (e.g., those that are indicated as not including a human and / or face region) altogether. In yet other embodiments, the mouth / speech module 1153 can utilize the provided regions as an attention mechanism (e.g., as a separate attention input for the gaze machine learning model 1161) to focus the processing of each visual frame.

[0087] In some embodiments, the detection and classification model 1157 may additionally or alternatively provide information to the other condition modules 1156 ( Figure 2B The gesture module 1156 may provide an indication of a certain area (not shown in the figure for simplicity) for use by other condition modules 1156. For example, when the detection of gestures is an additional condition for deploying an automated assistant, the gesture module may use body regions when detecting gestures using a corresponding gesture machine learning model. Moreover, for example, the posture of the user's body relative to the assistant device may be considered when determining whether to deploy an automated assistant function. In some of those embodiments, the body region may be utilized when determining the posture of the body by processing such body region using one or more posture determination techniques.

[0088] In some embodiments, in addition or alternatively, the detection and classification model 1157 can provide indications of areas classified as television or other video display sources to one or more of the modules 1151-1154. In some of those embodiments, the modules can crop those areas out of the processed visual frames, focus away from those areas, and / or otherwise ignore those areas in detection or reduce the chance that detection will be based on such areas. In these and other ways, false positive deployment of automated assistants can be mitigated.

[0089] Figure 3 depicts Figure 1 3, the example client device is represented as 101A, the example display 103 is represented as 103A, and the example visual component 107 is represented as 107A. The client device 101 may further include a speaker and a microphone. FIG3 also depicts a user 301 providing a gaze directed to the camera 107A. FIG3 also depicts an example image 360 ​​captured by the camera 107A when the user provides a directional gaze. It can be seen that the user and the television behind the user (and therefore not visible in the perspective view of FIG3) are captured in the image 360.

[0090] In image 360, a bounding box 362 is provided and represents the image in which the image may be (e.g., by Figure 2B 3 ) as corresponding to a face. In some embodiments, one or more modules of the orchestration engine 115 operating on the client device 101A may process only (or focus attention on) portions of the bounding box 362 in the image when detecting simultaneous occurrences of gaze, distance, mouth, and voice activity and / or when performing facial recognition based on the portions of the bounding box 362 in the image being indicated as corresponding to a face. Although only a single image is depicted in FIG. 3 , in various embodiments, determining the occurrence of various attributes may be based on a sequence of images as described herein.

[0091] In image 360, bounding box 363 is also provided and represents an area of ​​the image that may be determined to correspond to a video display and where false positives of visual cues may occur. For example, a television may render a video showing one or more individuals talking, looking toward a camera, etc., any of which may be misinterpreted as the occurrence of mouth movement and / or directional gaze. In some embodiments, Figure 2B The detection and classification module 1157 can determine such areas (e.g., based on detecting TV classification) and / or such areas can be determined based on an analysis of the image 360 ​​and previous images to determine that the area has a display frequency corresponding to the display frequency of the video display (e.g., approximately 60 Hz, 120 Hz, and / or other typical video display frequencies). In some embodiments, one or more modules can crop the area out of the processed video frame, focus away from those areas, and / or otherwise ignore those areas in the detection or reduce the chance that the detection will be based on such areas. In these and other ways, the deployment of false positives by the automated assistant can be mitigated.

[0092] Figure 3B1 , 3B2 3B1 and 3B2 provide examples of visually perceptible cues, each of which may be provided in response to determining the occurrence and / or confidence of a corresponding attribute. Figure 3B1 , 3B2 The visually perceptible prompts of and 3B1 are one non-limiting example of orchestrations that can be initiated by the orchestration engine 115 alone or in conjunction with other orchestrations such as audio and / or visual data processing and / or transmission.

[0093] exist Figure 3B1 , a first visually perceptible prompt is rendered by display 103A and indicated by shadow 171A. The visually perceptible prompt can be a white (or other color) light around the periphery of display 103A, and can optionally be displayed together with other content rendered by display 103A (e.g., surrounding pictures, weather forecasts, upcoming events, etc.). In some embodiments, the first visually perceptible prompt is rendered by the deployment engine 115 in response to the gaze module 1151 detecting the directional gaze of the user 301. The first visually perceptible prompt can (directly or indirectly) notify the user 301 that the directional gaze has been detected and further processing of certain sensor data can be performed. If the user 301 does not want to perform further processing, the user can redirect his / her gaze, which will cause the gaze module 1151 to no longer detect the gaze, and as a result, the deployment engine 115 stops rendering the first visually perceptible prompt.

[0094] exist Figure 3B2 , a second visually perceptible cue is rendered by display 103A and indicated by shadow 171B. The second visually perceptible cue can be a white (or other color) glow rendered horizontally across a portion of display 103A, and can optionally be rendered on top of other content rendered by display 103A. Optionally, the second visually perceptible cue can be partially transparent so that other content rendered by display 103A can continue to be viewed. In some embodiments, the second visually perceptible cue is rendered by the deployment engine 115 in response to the gaze module 1151 continuing to detect the directional gaze of the user 301 and optionally based on the occurrence and / or confidence measure of determining additional attributes, such as the user is within a threshold distance of the assistant device 101A (e.g., within 4 feet, or other thresholds as estimated based on image analysis), the presence of voice activity, and / or the simultaneous occurrence of mouth movement and voice activity. The second visually perceptible cue can (directly or indirectly) notify the user 301 that voice activity has been detected and that further processing of certain sensor data can be performed. The second visually perceptible cue can be rendered prior to further deployment such as certain processing of visual and / or sensor data and / or transmission of visual and / or sensor data. If the user 301 does not want further processing, the user can redirect his / her gaze, which will cause the gaze module 1151 to no longer detect the gaze, and as a result, the deployment engine 115 stops rendering the second visually perceptible cue. Optionally, the width of the bar, the color of the bar, the brightness of the bar and / or other characteristics of the bar can depend on the number of additional attributes and / or the magnitude of their confidence measures, thereby indicating to the user the determined degree of confidence that the user intends to interact with the assistant and the corresponding degree of confidence that further processing can be performed.

[0095] exist Figure 3B3 , a third visually perceptible prompt is rendered by display 103A and indicated by shadow 171C. The third visually perceptible prompt can be a multi-color (or other color) arc rendered across a portion of display 103A, and can optionally be rendered on top of other content rendered by display 103A. Optionally, the third visually perceptible prompt can be partially transparent so that other content rendered by display 103A can continue to be viewed. In some embodiments, the deployment engine 115 causes the third visually perceptible prompt to be rendered in response to continued detection of directional gaze and detection of the end of voice activity (e.g., through local end point detection of assistant device 101A) and / or other conditions. The third visually perceptible prompt can (directly or indirectly) notify the user that further processing of certain sensor data is being performed (but optionally can still be aborted). Such further processing can include speech-to-text processing of audio data, natural language understanding of the converted text, gesture recognition based on processing of visual data, and / or implementation based on natural language understanding and / or gesture recognition. Additionally or alternatively, such further processing may include additional processing to confirm that any speech and / or gestures included in the further sensor data are intended for the automated assistant and should be carried out.

[0096] Figure 4 , 5 6 each illustrate a flowchart that illustrates an example method that may be performed according to the embodiments disclosed herein. Although illustrated as separate flowcharts, it is to be understood that a single automated assistant may perform multiple (eg, all) of these methods. For convenience, Figure 4-6 The operations in the flowchart of are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a computer system (e.g., a client device and / or a remote computing system) that implements the automated assistant 120. In addition, although the operations of the method are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0097] First turn Figure 4 In method 400, at box 402, the system receives visual data based on output from a visual component and audio data based on output from a microphone. In some embodiments, the visual component and / or the microphone can be integrated with a client device including an assistant client (also referred to as a "assistant device" herein). In some embodiments, the visual component and / or the microphone can be separated from the client device but communicate with it. For example, the visual component can include an independent smart camera that communicates wired and / or wirelessly with the client device including the assistant client.

[0098] At box 404, the system processes the visual data and / or audio data to determine the occurrence and / or confidence of various attributes. The processing of box 404 can be performed on multiple frames of visual data and / or audio data, such as a stream of visual data and a stream of audio data. In some embodiments, the processing of box 404 can be initiated in response to an output from a presence sensor indicating the presence of a person. In various embodiments, various separate modules perform separate processing, each determining whether a corresponding attribute has occurred and / or a confidence measure for the corresponding attribute. One or more of the separate modules can utilize a corresponding trained machine learning model in the processing. Non-limiting examples of modules are Figure 1 As described herein, one or more of the individual modules may perform processing in parallel with other modules. Also as described herein, initiation of processing by one or more of the individual modules may optionally depend on one or more other modules first determining the occurrence of an attribute (e.g., directional gaze).

[0099] Block 406 and / or block 410 may be performed during (eg, in parallel with) the processing of block 404 and each rely on attributes and / or confidence metrics determined during such processing.

[0100] At block 406, the system determines whether to provision the user interface output based on the determination made during processing at block 404. If not, the system does not provision the user interface output, but continues to execute block 406. If yes, the system proceeds to block 408 and provisions the user interface output, but optionally continues to execute block 406 to determine whether to further provision the user interface output.

[0101] As an example of block 406, the system may determine that the user interface output is initially deployed in response to the processing of block 404 indicating the occurrence of a directional gaze. For example, at block 408, the system may render a first human-perceptible prompt. Continuing with this example, the system may continue to execute block 406. If continued processing of block 404 indicates that the directional gaze is no longer occurring, the system may continue to block 408 and remove the first human-perceptible prompt (and not render any other human-perceptible prompts). On the other hand, if continued processing of block 404 indicates that the directional gaze continues and also indicates that one or more other attributes occur, the system may continue to block 408 and provide a second human-perceptible prompt.

[0102] As an example of box 408, the system can initiate certain previously dormant local processing of audio data and / or visual data in response to the processing of box 404 indicating the occurrence of directional gaze, based on the distance of the user indicated by the processing of box 404 and / or optionally based on one or more other conditions. Such conditions may include, for example: the processing of box 404 indicates the simultaneous occurrence of voice activity, mouth movement and voice activity, and / or indicates the end point of voice activity. As another example of box 408, the system can determine to transmit audio data and / or visual data to a remote assistant component in response to the processing of box 404 indicating the occurrence of directional gaze and based on the distance of the user indicated by the processing of box 404.

[0103] Blocks 402, 404, 406, 408, and / or 410 may optionally continue until one or more conditions are met. Such conditions may include, for example, that human presence is no longer detected, that method 400 stops based on user input requesting a stop, and / or other conditions.

[0104] Now turn to Figure 5 , another example method 500 is illustrated and is a specific implementation of the method 400 .

[0105] At block 502, the system receives visual data based on output from a visual component and audio data based on output from a microphone. Block 502 may be associated with Figure 4 Block 402 of shares one or more aspects.

[0106] At block 504, the system processes the visual data and / or audio data to determine the occurrence and / or confidence of various attributes. The processing at block 504 may be performed on multiple frames of the visual data and / or audio data, such as on a stream of visual data and a stream of audio data. In some implementations, the processing at block 504 may be initiated in response to output from a presence sensor indicating human presence. The processing at block 504 may be performed in conjunction with Figure 4 Block 404 of shares one or more aspects.

[0107] Blocks 506 , 508 , and / or 510 may be performed during (eg, in parallel with) the processing of block 504 , and each rely on attributes and / or confidence metrics determined during such processing.

[0108] At block 506, the system determines whether the occurrence of a directional gaze has been determined (e.g., with at least a threshold confidence measure) based on the processing of block 504. If so, the system proceeds to block 510 and optionally also renders a first human-perceptible cue at block 508. If not, the system continues to execute block 506.

[0109] In box 510, the system determines whether the directional gaze continues and whether one or more other attributes occur. For example, the other attributes may include: the user (providing the directional gaze) is within a threshold distance of the assistant device, determining the occurrence of voice activity, and / or determining the simultaneous occurrence of voice activity and mouth movement (of the user providing the directional gaze). In some embodiments, in box 510, the system proceeds to "yes" if the directional gaze continues and one or more of the other attributes meet certain constraints (for example, the distance meets a threshold (or at least based on the magnitude of the distance) and / or the voice activity and / or the simultaneous occurrence of voice activity and mouth movement has a confidence measure of at least a threshold).

[0110] If the decision at block 510 is "no," the system returns to block 506. If the decision at block 510 is "yes," the system proceeds to block 518 or block 514, and optionally also renders a second human-perceptible prompt at block 512. If the system proceeds to block 518 after a "yes" determination at block 510, the system may perform further local processing of the visual data and / or audio data (i.e., further processing relative to the processing already performed at block 504), and / or may transmit the visual data and / or audio data (and / or a locally determined transformation thereof - such as text converted from the audio data) to a remote automated assistant component for further processing.

[0111] If the system proceeds to box 514 after a "yes" determination at box 510, the system waits for a local end point indicating completion of the user input at box 514. For example, the end point can be based on an analysis of audio data and can indicate that the current verbal input is complete. As another example, in addition or alternatively, the end point can be based on an analysis of visual data and can indicate that the current gesture input is complete. Various end point techniques can be utilized when making such an end point determination. If the local end point indicates that the current user input is complete, the system can proceed to box 516 and render a third perceptible prompt and / or can proceed to box 518. If proceeding from box 514 to box 518, the system can perform further local processing of the visual data and / or audio data, and / or can transmit the visual data and / or audio data to a remote automated assistant component for further processing. For example, if the system does not proceed to block 518 after a "yes" determination at block 510 and transmit visual data and / or audio data at block 518, the system may proceed to block 518 and transmit visual data and / or audio data after a "yes" determination at block 514. Also, for example, if the system does not proceed to block 518 after a "yes" determination at block 510, but only performs further local processing of the visual data and / or audio data, the system may proceed to block 518 and transmit visual data and / or audio data after a "yes" determination at block 514.

[0112] In optional box 520, the system performs further processing when determining whether to provide a response to the verbal input and / or gesture input captured in the visual data and / or audio data. In some embodiments, box 520 can be implemented at a remote automated assistant component, and the other boxes of method 500 can be executed locally on the assistant device by the automated assistant client. This further processing can include, for example: semantic filtering described herein and / or determining whether text-independent speaker recognition indicates that the verbal input matches the user indicated by the facial recognition of the user providing the directional gaze. For example, if semantic filtering indicates that the input may not be intended for the automated assistant, and / or if the TI-SID indicates that the verbal input does not match the user indicated by the facial recognition, response generation can be terminated and / or the response is prevented from being transmitted to the client device and / or rendered by it.

[0113] Now turn to Figure 6 In method 600 , at block 602 , the system receives visual data based on output from a visual component, such as a visual component integrated with an assistant device.

[0114] At block 604, the system processes the visual data to determine the occurrence and / or confidence of the user's directional gaze and / or the user's contactless gesture. The processing of block 604 can be performed on multiple frames of visual data, such as a stream of visual data. In some embodiments, the processing of block 604 can be initiated in response to output from a presence sensor indicating human presence.

[0115] Block 606 and / or block 610 may be performed during (eg, in parallel with) the processing of block 604 , and both are dependent upon the processing of block 604 .

[0116] At block 606, the system determines whether an occurrence of a directional gaze of the user has been determined based on processing the visual data of block 604. If not, the system continues to execute block 606. If yes, the system continues to block 610 and optionally deploys a user interface output (e.g., renders a human-perceptible cue) at block 608.

[0117] At block 610, the system determines whether a gesture by the user (the same user who provided the directional gaze) has been determined based on processing the visual data of block 604, and determines a label or other identifier for the gesture. In some implementations, separate processing may be performed at block 604 when determining whether a directional gaze has occurred and when determining whether a gesture has been determined. If the determination at block 610 is "no," the system returns to block 606 to determine whether the directional gaze continues (and optionally also whether to continue the user interface output deployment of block 608).

[0118] If the result of the determination at block 610 is "yes", the system proceeds to block 612 and generates a response based on the gesture and optionally based on the content rendered by the client device at the time of the gesture. In some embodiments, in order to proceed to block 612, the system also determines that the distance of the user (providing the directional gaze and gesture) meets the threshold based on the processing of the visual data at block 604.

[0119] In some embodiments, box 612 includes determining that the gesture is assigned to multiple responsive actions. For example, the gesture-to-action mapping may indicate that the gesture is assigned to multiple candidate responsive actions. In those embodiments, box 612 further includes selecting a single responsive action from the multiple responsive actions based on the content rendered by the client device at the time of the gesture, and generating a response so that the selected single responsive action is performed. For example, a left swipe (i.e., moving the hand horizontally to the left) gesture may be assigned to a "next track" action and a "lower smart thermostat temperature" action, and which of the two actions is selected may be based on the content being rendered. For example, the "next track" action may be selected when rendering audio content, while the "lower smart thermostat temperature" action may be selected when no audio or audio-visual content is being rendered and a thermostat control is being rendered (e.g., in response to a previously spoken speech "what is the current thermostat temperature").

[0120] The system then implements the response at block 614. This may include interfacing with one or more third-party systems (e.g., to implement a "lower smart thermostat temperature" action and / or a "next track" action) and / or controlling an assistant device (e.g., to implement a "lower volume" action).

[0121] Figure 7 710, which may optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant components, and / or other components may include one or more components of the example computing device 710.

[0122] The computing device 710 typically includes at least one processor 714 that communicates with a plurality of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724—for example, including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with the computing device 710. The network interface subsystem 716 provides an interface with an external network and is coupled to corresponding interface devices in other computing devices.

[0123] The user interface input devices 722 may include a keyboard, a pointing device such as a mouse, trackball, touch pad or graphic tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the term "input device" is used to include all possible types of devices and ways to input information into the computing device 710 or over a computer network.

[0124] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display via an audio output device. Typically, the term "output device" is used to include devices and modes of all possible types of information output from computing device 710 to a user or another machine or computing device.

[0125] The storage subsystem 724 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 724 may include an executable Figure 4 , 5 and / or selected aspects of the method of 6 and methods for implementing Figure 1 , 2A , 2B and 3A.

[0126] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include multiple memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functions of a certain embodiment may be stored in the storage subsystem 724 by the file storage subsystem 726, or in other machines accessible to the processor 714.

[0127] The bus subsystem 712 provides a mechanism for the various components and subsystems of the computing device 710 to communicate with each other as desired. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0128] Computing device 710 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 7 The description of the computing device 710 depicted in FIG. 7 is intended only as a specific example for illustrating some embodiments. Many other configurations of the computing device 710 are possible, which have more Figure 7 The computing devices depicted may have greater or fewer components.

[0129] In the case where the system described herein collects or otherwise monitors personal information about a user or can make use of personal information and / or monitored information, the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current location), or whether and / or how to receive content that may be more relevant to the user from a content server. Moreover, certain data may be processed in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as processed to the level of a city, zip code, or state) when location information is obtained, so that the user's specific location cannot be determined. Therefore, the user can control how to collect and / or use information about the user. For example, the user can choose to cancel the assistant device's use of the visual component 107 and / or use of data from the visual component 107 when enabling hotword-free interaction with the automated assistant.

[0130] In some embodiments, a method implemented by one or more processors is provided, and includes: receiving a stream of image frames based on the output of one or more cameras from the client device at a client device. The method further includes processing the image frames of the stream at the client device using at least one trained machine learning model stored locally at the client device to detect the occurrence of a user gaze directed to the client device. The method further includes determining the distance of the user relative to the client device at the client device. Determining the distance of the user relative to the client device can be based on one or more of the image frames and / or additional sensor data from additional sensors of the client device. The method further includes determining at the client device to perform certain processing of audio data detected by one or more microphones of the client device based on detecting the occurrence of the user's gaze and based on the distance of the user relative to the client device. The method further includes initiating certain processing of the audio data at the client device in response to determining to perform certain processing of the audio data.

[0131] These and other implementations disclosed herein may optionally include one or more of the following features.

[0132] In some implementations, initiating certain processing of the audio data may include: initiating local automatic speech recognition of the audio data at the client device; initiating transmission of the audio data to a remote server associated with the automated assistant; and / or initiating transmission of recognized text from the local automatic speech recognition to the remote server.

[0133] In some implementations, at least a portion of the audio data is buffered audio data that is buffered in a temporary buffer local to the client device prior to detecting a gaze of the user.

[0134] In some embodiments, the method further includes, before initiating certain processing of the audio data: rendering at least one human-perceptible cue via an output component of the client device. In some versions of those embodiments, the at least one human-perceptible cue includes a first queue rendered in response to detecting an occurrence of a user gaze directed toward the client device.

[0135] In some embodiments, processing the image frames using the at least one trained machine learning model to detect the occurrence of user gaze directed toward one or more cameras of the client device includes: processing the sequence of image frames using the at least one trained machine learning model to determine, for each image frame in the sequence, whether the user's gaze is directed toward the client device; and detecting the occurrence of user gaze directed toward the client device based on the number of image frames in the sequence for which the user's gaze is determined to be directed toward the one or more cameras.

[0136] In some embodiments, the method further includes, prior to initiating certain processing of audio data detected via one or more microphones of the client device: at the client device: detecting voice activity based on local processing of at least a portion of the audio data; and / or detecting a co-occurrence of a mouth movement of the user and voice activity based on local processing of one or more of the image frames and at least a portion of the audio data. In some of those embodiments, initiating certain processing of the audio data is further responsive to detecting voice activity and one or both of a co-occurrence of a mouth movement of the user and voice activity. In some versions of those embodiments, detecting an occurrence of the user's gaze is performed at a first time, and detecting voice activity and one or both of a co-occurrence of a mouth movement of the user and voice activity is performed at a second time after the first time. In some of those versions, the method further includes: before initiating certain processing of the audio data and before the second time: rendering a first human-perceptible cue via an output component of the client device in response to detecting an occurrence of a user gaze directed toward one or more cameras of the client device; and before initiating certain processing of the audio data and after rendering the first human-perceptible cue: rendering a second human-perceptible cue via the output component or an additional output component of the client device in response to detecting one or both of voice activity and a simultaneous occurrence of a mouth movement and voice activity of the user. The first human-perceptible cue may, for example, include a first visual cue rendered via a display of the client device, and the second human-perceptible cue may, for example, include a second visual cue rendered via a display of the client device. Further, in some of those versions, the method may further include, in response to initiating certain processing of the audio data and after the second time: rendering a third human-perceptible cue via the output component or the additional output component of the client device.

[0137] In some embodiments, determining the distance of the user relative to the client device is based on one or more of the image frames. In some of those embodiments, determining the distance of the user is based on an eye distance between the eyes of the user detected in one or more of the determined image frames.

[0138] In some implementations, determining to perform certain processing on the audio data based on detecting the gaze of the user and based on the distance of the user relative to the client device includes: determining to perform the certain processing based on the distance of the user satisfying a threshold distance.

[0139] In some embodiments, determining to perform certain processing of the audio data based on detecting the user's gaze and based on the distance of the user relative to the client device includes: determining to perform certain processing of the audio data based on a magnitude of the user's distance and based on a gaze confidence measure of the user's gaze. The gaze confidence measure can be generated based on processing of image frames of the stream using the at least one trained machine learning model.

[0140] In some embodiments, the method further includes determining that the user is a recognized user based on processing one or more of the image frames locally at the client device. In some of those embodiments, determining to perform certain processing of the audio data is further based on determining that the user is a recognized user.

[0141] In some embodiments, certain processing of the audio data includes automatic speech recognition of the audio data to generate recognized speech. In some of those embodiments, the method further includes: determining an assistant request metric indicating a probability that the recognized speech is a request for an automated assistant based at least in part on the recognized speech; and determining whether to render a response to the recognized speech via the client device based at least in part on the assistant request metric. In some of those versions, determining whether to render a response to the recognized speech is further based on: a distance of the user; whether the user is the recognized user; and / or a gaze confidence metric of a user's gaze.

[0142] In some embodiments, a method implemented by one or more processors of a client device is provided, and includes: receiving a stream of image frames based on output from one or more cameras of the client device; processing the image frames of the stream at the client device using at least one trained machine learning model stored locally at the client device to detect an occurrence of a user gaze directed at the client device. The method further includes rendering a first human-perceptible queue at a display of the client device in response to detecting an occurrence of a user gaze directed at the client device. The method further includes performing one or more of the following operations at the client device while rendering the first human-perceptible queue: detecting locally processed voice activity based on at least a portion of audio data captured by one or more microphones of the client device; detecting a co-occurrence of a mouth movement of the user and the voice activity based on local processing of one or more of the image frames and at least a portion of the audio data; and detecting a gesture of the user based on local processing of one or more of the image frames. The method further includes replacing the rendering of the first human-perceptible queue with rendering of a second human-perceptible queue at the display of the client device in response to continuing to detect the occurrence of the gaze and detecting one or more of the voice activity, the co-occurrence of the mouth movement of the user and the voice activity, and the gesture of the user. The method further includes, after rendering the second human-perceptible queue: initiating certain additional processing of one or more of the audio data and / or the image frames at the client device.

[0143] These and other implementations disclosed herein may optionally include one or more of the following features.

[0144] In some embodiments, the method further includes, in response to initiating certain additional processing of one or more of the audio data and / or image frames: replacing rendering of the second human-perceptible queue with rendering of a third human-perceptible queue at a display of the client device.

[0145] In some implementations, initiating some additional processing of one or more of the audio data and / or image frames includes initiating transmission of the audio data and / or the image data to a remote server associated with the automated assistant.

[0146] In some embodiments, the method further includes determining one or more confidence metrics associated with one or more of: detecting the occurrence of the gaze, detecting the voice activity, detecting the co-occurrence of the mouth movement and the voice activity, and detecting the gesture. In some of those embodiments, the size of the second human-perceptible queue is based on the one or more confidence metrics, and / or initiating certain additional processing of one or more of the audio data and / or image frames is based on the one or more confidence metrics satisfying one or more thresholds.

[0147] In some embodiments, a method implemented by one or more processors is provided, and includes: receiving a stream of image frames based on the output of one or more cameras from the client device at a client device. The method further includes processing the image frames of the stream at the client device using at least one trained machine learning model stored locally on the client device to detect the occurrence of a user gaze directed at the client device. The method further includes generating a response to a gesture of the user captured by one or more of the image frames of the stream based on detecting the occurrence of the user gaze. The method further includes generating a response to the gesture of the user. Generating the response may include determining the gesture of the user based on processing one or more of the image frames of the stream, and generating the response based on the gesture of the user and based on the content rendered by the client device at the time of the gesture. The method further includes implementing the response at the client device.

[0148] These and other implementations disclosed herein may optionally include one or more of the following features.

[0149] In some embodiments, generating the response based on the user's gesture and based on the content rendered by the client device at the time of the gesture includes: determining that the gesture is assigned to multiple responsive actions; selecting a single responsive action from the multiple responsive actions based on the content rendered by the client device at the time of the gesture; and generating the response so that the selected single responsive action is performed.

[0150] In some embodiments, the method further includes determining, at the client device, a distance of the user relative to the client device, and determining to generate a response to the gesture of the user is further based on the magnitude of the user's distance. In some of those embodiments, determining to generate a response to the gesture of the user based on the distance of the user includes determining to generate the response based on the distance of the user satisfying a distance threshold.

[0151] In some embodiments, the method further includes determining that the user is a recognized user based on processing one or more of the image frames locally at the client device. In some of those embodiments, determining to generate a response to the gesture for the user is further based on determining that the user is the recognized user, and optionally further based on determining that the same recognized user initiated the provision of the content rendered by the client device.

[0152] In some embodiments, a method implemented by one or more processors is provided, and includes: receiving at a client device a stream of image frames based on outputs from one or more cameras of the client device and audio data detected by one or more microphones of the client device. The method further includes processing the image frames and the audio data at the client device to determine the co-occurrence of: a mouth movement of a user captured by one or more of the image frames, and a voice activity of the user. The method further includes performing at the client device and based on determining the co-occurrence of the mouth movement of the user and the voice activity of the user: certain processing of the audio data, and / or rendering of at least one human-perceptible prompt via an output component of the client device. The method further includes initiating certain processing of the audio data and / or rendering of the at least one human-perceptible prompt at the client device. The initiation is responsive to determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt.

[0153] These and other implementations disclosed herein may optionally include one or more of the following features.

[0154] In some embodiments, certain processing of the audio data is initiated in response to determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt, and initiating certain processing of the audio data includes: initiating local automatic speech recognition of the audio data at the client device; initiating transmission of the audio data to a remote server associated with the automated assistant; and / or initiating transmission of recognized text from the local automated speech recognition to the remote server.

[0155] In some embodiments, processing the image frame and the audio data at the client device to determine the co-occurrence of the user's mouth movement and the user's voice activity includes: processing the image frame and the audio data using a locally stored machine learning model. The locally stored machine learning model can be trained to distinguish between: voice activity that occurs simultaneously with and as a result of a mouth movement; and voice activity that does not result from the mouth movement but occurs simultaneously with the mouth movement.

[0156] In some embodiments, the method further includes determining, at the client device, a distance of the user relative to the client device, and determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt is further based on the distance of the user relative to the client device. In some versions of those embodiments, determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt further based on the distance of the user relative to the client device includes: determining that the distance of the user relative to the client device satisfies a threshold. In some additional or alternative versions, determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt further based on the distance of the user relative to the client device includes: determining that the distance of the user relative to the client device is closer to the client device than one or more previously determined distances of the user relative to the client device.

[0157] In some embodiments, the method further includes determining, at the client device and based on one or more of the image frames, that the user's gaze is directed toward the client device. In some of those embodiments, determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt is further based on determining that the user's gaze is directed toward the client device.

[0158] In some embodiments, the method further includes determining, at the client device and based on one or more of the image frames, that a body posture of the user is directed toward the client device. In some of those embodiments, determining to perform certain processing of the audio data and / or rendering of the at least one human-perceptible prompt is further based on determining that the body posture of the user is directed toward the client device.< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method for facilitating hotword-free interaction between a user and an automated assistant, the method being implemented by one or more processors of a client device, and comprising: receiving a stream of image frames based on output from one or more cameras of the client device; At the client device, the image frames in the stream are processed using at least one trained machine learning model stored locally on the client device to detect occurrences of: a user's gaze directed toward the client device; responsive to detecting said occurrence of said gaze of said user directed toward said client device, rendering a first human-perceptible queue at a display of said client device; While rendering the first human-perceivable queue: Detecting, at the client device, one or more of the following: voice activity based on locally processed audio data of at least a portion of the audio data captured by one or more microphones of the client device; a concurrence of the user's mouth movement and the speech activity based on local processing of one or more of the image frames and at least a portion of the audio data; as well as a gesture of the user based on local processing of one or more of the image frames; In response to continuing to detect the occurrence of the gaze, and detecting one or more of the voice activity, the simultaneous occurrence of the mouth movement of the user and the voice activity, and the gesture of the user: replacing, at the display of the client device, rendering of the first human-perceptible queue with rendering of a second human-perceptible queue; After rendering the second human-perceivable queue: Some additional processing of the audio data and / or one or more of the image frames is initiated at the client device.

2. The method according to claim 1, further comprising: In response to initiating the certain additional processing of the audio data and / or one or more of the image frames: Rendering of the second human-perceptible queue is replaced, at the display of the client device, with rendering of a third human-perceptible queue.

3. The method according to claim 1, wherein: Initiating the certain additional processing of the audio data and / or one or more of the image frames comprises: Transmission of the audio data and / or the image frames is initiated to a remote server associated with the automated assistant.

4. The method according to claim 1, further comprising: One or more confidence measures are determined, each associated with one or more of: detecting the occurrence of the gaze, detecting the voice activity, detecting the co-occurrence of the mouth movement and the voice activity, and detecting the gesture.

5. The method according to claim 4, wherein: Initiating the certain additional processing of the audio data and / or one or more of the image frames is based on the one or more confidence measures satisfying one or more thresholds.

6. The method according to claim 4 or claim 5, wherein: The size of the second human-perceptible queue is based on one or more of the confidence metrics.

7. A method for facilitating hotword-free interaction between a user and an automated assistant, the method being implemented by one or more processors, and comprising: Received at the client device: a stream of image frames based on the output of one or more cameras of the client device, and audio data detected by one or more microphones of the client device; The image frames and the audio data are processed at the client device to determine a concurrence of: a user's mouth movement captured by one or more of the image frames, and the user's voice activity; At the client device and based on determining the co-occurrence of the mouth movement of the user and the voice activity of the user, determine to perform one or both of the following: some processing of the audio data, and rendering at least one human-perceptible cue via an output component of the client device; as well as In response to determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue, the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue is initiated at the client device.

8. The method according to claim 7, wherein: In response to determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue, initiating the certain processing of the audio data, and wherein initiating the certain processing of the audio data comprises one or more of: initiating local automatic speech recognition of the audio data at the client device, initiating transmission of the audio data to a remote server associated with the automated assistant, and Transmission of recognized text from the local automatic speech recognition is initiated to the remote server.

9. The method according to claim 7, wherein: Processing the image frames and the audio data at the client device to determine the co-occurrence of the mouth movement of the user and the voice activity of the user includes: Both the image frames and the audio data are processed using a locally stored machine learning model, the machine learning model being trained to distinguish between: speech activity that occurs simultaneously with and is a result of mouth movement; and speech activity that does not result from the mouth movement but occurs simultaneously with the mouth movement.

10. The method according to claim 7, further comprising: Determining, at the client device, a distance of the user relative to the client device, wherein determining the distance of the user relative to the client device is based on one or both of the following: one or more of the image frames, and additional sensor data from additional sensors of the client device; Wherein determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue is further based on the distance of the user relative to the client device.

11. The method according to claim 10, wherein: Determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible audio data further based on the distance of the user relative to the client device comprises: It is determined that the distance of the user relative to the client device satisfies a threshold.

12. The method according to claim 10, wherein: Determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible audio data further based on the distance of the user relative to the client device comprises: The distance of the user relative to the client device is determined to be closer to the client device than one or more previously determined distances of the user relative to the client device.

13. The method according to claim 7, further comprising: determining, at the client device and based on one or more of the image frames, that the user's gaze is directed toward the client device; Wherein, determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue is further based on determining that the gaze of the user is directed toward the client device.

14. The method according to any one of claims 7 to 13, further comprising: determining, at the client device and based on one or more of the image frames, that a body gesture of the user is directed toward the client device; Wherein, determining to perform the certain processing of the audio data and / or the rendering of the at least one human-perceptible cue is further based on determining that the body gesture of the user is directed toward the client device.

15. A client device, comprising: at least one visual component; At least one microphone; at least one display; One or more processors that execute locally stored instructions to implement the method according to any one of claims 1 to 14.

16. A computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 14.

17. A system comprising one or more processors for executing the method according to any one of claims 1 to 14.

Images