Invoking automated assistant function(s) based on detected gesture and gaze

Abstract

To invoke one or more previously dormant functions of an automated assistant in response to detecting, based on processing of vision data from one or more vision components: (1) a particular gesture (e.g., of one or more "invocation gestures") of a user; and / or (2) that a gaze of the user is directed at an assistant device that provides a (graphical and / or audible) automated assistant interface of the automated assistant.SOLUTION: For example, the previously dormant function(s) can be invoked in response to detecting the particular gesture, detecting that the gaze of the user is directed at the assistant device for at least a threshold amount of time, and optionally detecting that the particular gesture and the directed gaze of the user co-occur or occur within a threshold temporal proximity of one another.SELECTED DRAWING: Figure 4A

Description

[Technical field]

[0001] The present invention relates to invoking functions of an automated assistant based on detected gestures and gaze. [Background technology]

[0002] A person may engage in human-computer interaction using an interactive software application (also referred to as a "digital agent," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.) referred to herein as an "automated assistant." For example, a person (which may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., speech), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the request by providing a responsive user interface output, which may include audible and / or visual user interface output.

[0003] As mentioned above, many automated assistants are configured to be interacted with by verbal utterances. To protect user privacy and / or conserve resources, a user must often explicitly invoke the automated assistant before the automated assistant can fully process the verbal utterance. Explicit invocation of an automated assistant generally occurs in response to a particular user interface input being received at a client device. The client device includes an assistant interface that provides a user of the client device with an interface for interfacing with the automated assistant (e.g., receiving input from the user and providing an audible and / or graphical response) and interfaces with one or more additional components that implement the automated assistant (e.g., a remote server device that processes the user input and generates an appropriate response).

[0004] Some user interface inputs that can invoke an automated assistant via a client device include hardware and / or virtual buttons on the client device (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device) to invoke the automated assistant. Many automated assistants may additionally or alternatively be invoked in response to one or more spoken invocation phrases, also known as "hot words / phrases" or "trigger words / phrases." For example, spoken invocation phrases such as "Hey, Assistant," "Okay, Assistant," and / or "Assistant" may be spoken to invoke the automated assistant. Summary of the Invention [Means for solving the problem]

[0005] As described above, many client devices that facilitate interaction with an automated assistant, also referred to herein as "assistant devices," allow a user to engage in contactless interaction with the automated assistant. For example, an assistant device often includes a microphone that allows a user to provide audio utterances to invoke and / or otherwise interact with the automated assistant. The assistant devices described herein can additionally or alternatively incorporate and / or communicate with one or more vision components (e.g., a camera, a Light Detection and Ranging (LIDAR) component, a radar component, etc.) to facilitate contactless interaction with the automated assistant.

[0006] Implementations disclosed herein relate to invoking one or more previously dormant functions of an automated assistant in response to (1) detecting a particular gesture of a user (e.g., one or more "invocation gestures") and / or (2) detecting that the user's gaze is directed toward an assistant device that provides an automated assistant interface (graphical and / or audible) of the automated assistant, based on processing of visual data from one or more visual components. For example, a previously dormant function may be invoked in response to detecting a particular gesture, detecting that the user's gaze is directed toward the assistant device for at least a threshold amount of time, and optionally detecting that the particular gesture and the user's directed gaze occur simultaneously or within a threshold temporal proximity of each other (e.g., within 0.5 seconds, within 1.0 seconds, or other threshold temporal proximity). For example, a previously dormant function may be invoked in response to detecting a particular gesture (e.g., a "thumbs up" gesture and / or a "waving" gesture) and in response to detecting a directed gaze of at least 0.5 seconds duration that occurred simultaneously with the particular gesture or within 1.0 seconds of the particular gesture.

[0007] In some versions of those implementations, one or more previously dormant functions may be invoked in response to detecting the gesture and the directed gaze only. For example, in response to detecting the gesture and the directed gaze only, certain sensor data generated by one or more sensor components may be transmitted by the client device to one or more remote automated assistant components (but prior to detecting the gesture and the directed gaze, no sensor data from the sensor components was transmitted). The certain sensor data may include, for example, visual and / or audio data captured after detecting the gesture and the directed gaze, and / or buffered visual and / or audio data captured during the execution of the gesture and / or during the directed gaze. In this manner, the transmission of data to the remote computing component over the data network is performed selectively and in a timely manner. This provides for efficient use of the network capacity as well as the computing resources and other hardware resources involved in transmitting and receiving data over the network. The increased efficiency of the use of the computing resources of the data network and the remote system may lead to significant savings in terms of power usage by transmitters and receivers in the network as well as memory operation and processor usage in the remote system. Corresponding effects may also be experienced at the client device. These effects, particularly over time, allow significant additional capacity to be experienced in the network and across computing devices, including the devices and systems that run the assistant, throughout the ongoing operation of the automated assistant, which may be used for further communications in the data network, whether or not related to the assistant, without the need to expand network capacity, for example, through additional or updated infrastructure and additional computing operations on the computing devices.Other technical improvements will become apparent from the following discussion.

[0008] In some other versions of those implementations, one or more previously dormant functions may be invoked in response to detecting the gesture and directed gaze and detecting the occurrence of one or more other conditions. The occurrence of the one or more other conditions may include, for example, detecting voice activity (e.g., voice activity including any voice activity, voice activity of the user giving the gesture and directed gaze, voice activity of an authorized user, and a spoken call phrase) in temporal proximity to the detected gesture and directed gaze based on audio data, detecting user mouth movements that coincide with or in temporal proximity to the detected gesture and directed gaze based on visual data, detecting that the user is an authorized user based on audio data and / or visual data, and / or detecting other conditions. For example, buffered audio data may be transmitted by the client device to one or more remote automated assistant components in response to detecting the gesture and directed gaze and in response to detecting voice activity in at least a portion of the buffered audio data (e.g., using a voice activity detector (VAD) module). This has advantages corresponding to those explained above.

[0009] In some implementations disclosed herein, previously dormant functionality of the automated assistant that is invoked in response to detecting a gesture and a directed gaze may include specific processing of specific sensor data (e.g., audio data, video, images, etc.) and / or rendering of specific content (e.g., graphical and / or audible). For example, prior to an invocation based on detecting a gesture and a directed gaze, the automated assistant may perform only limited processing (or no processing at all) of specific sensor data, such as audio data, video / image data, etc. For example, prior to an invocation, the automated assistant may process some sensor data locally upon monitoring an explicit invocation, but after local processing, "discard" the data without having the data processed by one or more additional components implementing the automated assistant (e.g., a remote server device that processes user input and generates an appropriate response). However, in response to an invocation, such data may be processed by the additional components. In these and other ways, processing and / or network resources may be reduced by only transmitting specific sensor data and / or performing specific processing of specific sensor data in response to an invocation.

[0010] Also, for example, prior to an explicit invocation, the automated assistant may render only limited content (e.g., graphically) (or may not render any content at all). However, in response to an invocation, the automated assistant may render other content, such as content tailored to the user who invoked the automated assistant. For example, prior to an explicit invocation, no content may be rendered graphically by a display screen controlled by the automated assistant, or only limited low power content may be rendered (e.g., only the current time on a small portion of the display screen). However, in response to an invocation, the automated assistant may cause additional and optionally higher power content to be rendered graphically by the display screen and / or audibly by a speaker, such as a weather forecast, a summary of daily events, and / or other content that may be displayed brighter on the display screen and / or occupy a larger portion of the display screen. In these and other ways, power consumption may be reduced by displaying low power content (or no content) prior to an invocation and only displaying higher power content in response to an invocation.

[0011] In some implementations, when monitoring for specific gestures and gazes directed at the client device, a trained machine learning model (e.g., a neural network model) stored locally on the client device is utilized by the client device to at least selectively process at least a portion of the visual data from the client device's vision component (e.g., image frames from the client device's camera). For example, in response to detecting the presence of one or more users, the client device may utilize the locally stored machine learning model when monitoring for specific gestures and gazes directed at the client device to process at least a portion of the visual data for at least a certain duration (e.g., at least for a threshold duration and / or until the presence is no longer detected). The client device can detect the presence of one or more users using a dedicated presence detection sensor (e.g., a passive infrared sensor (PIR)), using visual data and a separate machine learning model (e.g., a separate machine learning model trained only for detecting the presence of a person), and / or using audio data and a separate machine learning model (e.g., a VAD using a VAD machine learning model). In implementations where the processing of visual data when monitoring a particular gesture is contingent on first detecting the presence of one or more users, power resources may be conserved by non-continuous processing of visual data when monitoring a gesture and / or directed gaze. Rather, in those implementations, the processing of visual data when monitoring a gesture and / or directed gaze may occur only in response to detecting the presence of one or more users in the assistant device's environment via one or more low power techniques.

[0012] In some implementations where local machine learning models are utilized in monitoring specific gestures and directed gazes, at least one gesture detection machine learning model is utilized in monitoring gestures and a separate gaze detection machine learning model is utilized in monitoring gazes. In some versions of those implementations, one or more "upstream" models (e.g., object detection and classification models) can be utilized to detect parts of the visual data (e.g., an image) that are likely to be faces, parts that are likely to be eyes, parts that are likely to be arms / body, etc., which are processed using the respective machine learning models. For example, face and / or eye parts of an image may be detected using an upstream model and processed using a gaze detection machine learning model. Also, for example, arm and / or body parts of an image may be detected using an upstream model and processed using a gesture detection machine learning model. As yet another example, human parts of an image may be detected using an upstream model and processed using both a gaze detection machine learning model and a gesture detection machine learning model.

[0013] Optionally, the gaze detection machine learning model may process higher resolution visual data (e.g., images) than the gesture detection machine learning model. This may allow for more efficient utilization of the gesture detection machine learning model by processing lower resolution images. Additionally, the gaze detection machine learning model may optionally be utilized to process portions of an image only after the gesture detection machine learning model has been utilized to detect possible gestures (or vice versa). This may also lead to computational efficiencies by not utilizing both models to continuously process image data.

[0014] In some implementations, facial matching, eye matching, voice matching, and / or other techniques may be utilized to identify a particular user profile associated with the gesture and / or directed gaze, and content rendered by the automated assistant application of the client device that is tailored to the particular user profile. Rendering the tailored content may be one of the functions of the automated assistant that is invoked in response to detecting the gesture and directed gaze. Optionally, the identification of the particular user profile occurs only after the directed gaze and gesture are detected. In some implementations, as described above, the occurrence of one or more additional conditions may also be required for the detection of the invocation - the additional conditions are added to the detection of the gaze and / or gesture. For example, in some implementations, the additional conditions may include identifying (e.g., using facial matching, voice matching, and / or other techniques) that the user providing the gesture and directed gaze is associated with an authorized user profile for the client device.

[0015] In some implementations, certain portions of a video / image may be filtered out / ignored / weighted lower in detecting gestures and / or gazes. For example, a captured television in a video / image may be ignored to prevent false detections as a result of a person (e.g., a weather forecaster) rendered by the television. For example, a portion of an image may be determined to correspond to a television based on a separate object detection / classification machine learning model, such as by detecting a particular display frequency (i.e., matching the television's refresh rate) in the portion across multiple frames for the portion. Such portions may be ignored in the gesture and / or directed gaze detection techniques described herein to prevent detection of gestures and / or directed gazes from a television or other video display device. As another example, a photo frame may be ignored. These and other techniques may mitigate falsely detected invocations of an automated assistant, which may save various computational and / or network resources that would otherwise be consumed in falsely detected invocations. Also, in various implementations, once the position of a TV, photo frame, etc. is detected, it can optionally continue to be ignored for multiple frames (e.g., while intermittently validating, until motion of the client device or object is detected, etc.), which can also save various computational resources.

[0016] The above description is provided as a general overview of various implementations disclosed herein, various of which and additional implementations are described in more detail herein.

[0017] In some implementations, a method for facilitating contactless interaction between one or more users and an automated assistant is executed by one or more processors of a client device. The method includes receiving a stream of image frames based on output from one or more cameras of the client device. The method further includes processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a user's invocation gesture captured by at least one of the image frames and the user's gaze directed toward the client device. The method further includes detecting based on the monitoring the occurrence of both the invocation gesture and the gaze. The method further includes activating at least one function of the automated assistant in response to detecting the occurrence of both the invocation gesture and the gaze.

[0018] These and other implementations of the technology may include one or more of the following features.

[0019] In some implementations, at least one function of the automated assistant that is activated in response to detecting the occurrence of both an invocation gesture and a gaze includes transmitting audio data captured by one or more microphones of the client device to a remote server associated with the automated assistant.

[0020] In some implementations, at least one function activated in response to detecting the occurrence of both the invocation gesture and the gaze additionally or alternatively includes transmitting an additional image frame to a remote server associated with the automated assistant, the additional image frame being received after detecting the occurrence of both the invocation gesture and the gaze based on output from one or more of the cameras.

[0021] In some implementations, the at least one function activated in response to detecting the occurrence of both the invocation gesture and the gaze additionally or alternatively includes processing of buffered audio data at the client device, the buffered audio data being stored in a memory of the client device and captured by one or more microphones of the client device, and the processing of the buffered audio data including one or both of an invocation phrase detection process and an automatic speech recognition. In some versions of those implementations, the processing of the buffered audio data includes an automatic speech recognition process, and the automatic speech recognition includes a speech-to-text process. In some additional or alternative versions of those implementations, the processing of the buffered audio data includes an invocation phrase detection process, and the method further includes performing one or both of the following in response to the invocation phrase detection process detecting the presence of an invocation phrase in the buffered audio data: transmitting further audio data captured by one or more microphones of the client device to a remote server associated with the automated assistant, and transmitting additional image frames to a remote server associated with the automated assistant, the additional image frames being received after detecting the occurrence of both the invocation gesture and the gaze based on output from one or more of the cameras.

[0022] In some implementations, the step of processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a call gesture and a gaze includes using a first trained machine learning model to monitor the occurrence of a call gesture and using a second trained machine learning model to monitor the user's gaze directed at the client device. In some versions of those implementations, using the second trained machine learning model to monitor the user's gaze directed at the client device occurs only in response to detecting the occurrence of a call gesture using the first trained machine learning model. In some of those versions, or in other versions, using the first trained machine learning model to monitor the occurrence of a call gesture includes processing a first resolution version of the image frames using the first machine learning model, and using the second trained machine learning model to monitor the user's gaze includes processing a second resolution version of the image frames using the second machine learning model.

[0023] In some implementations, the method further includes receiving a stream of audio data frames based on output from one or more microphones of the client device, processing the stream of audio data frames using at least one trained invocation phrase detection machine learning model stored locally on the client device to monitor occurrences of verbal invocation phrases, and detecting occurrences of verbal invocation phrases based on monitoring occurrences of verbal invocation phrases. In some of those implementations, activating at least one function of the automation assistant is responsive to detecting occurrences of the verbal invocation phrase in temporal proximity to both the invocation gesture and the gaze. In some versions of those implementations, the at least one function that is activated includes one or both of transmitting additional audio data frames captured by one or more microphones of the client device to a remote server associated with the automation assistant and transmitting one or more additional image frames from one or more of the cameras to a remote server associated with the automation assistant.

[0024] In some implementations, the step of processing image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a calling gesture and a gaze includes processing the image frames using a first trained machine learning model to predict regions of the image frames that include a human face, and processing regions of the image frames using a second trained machine learning model trained to detect a user's gaze.

[0025] In some implementations, processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor for the occurrence of both a calling gesture and a gaze includes determining that a region of the image frame corresponds to an electronic display, and ignoring the region when monitoring for the occurrence of both a calling gesture and a gaze in response to determining that the region corresponds to an electronic display. In some of these implementations, determining that the region of the image frame corresponds to an electronic display is based on detecting a display frequency within the region of the image frame that corresponds to a display frequency of the electronic display.

[0026] In some implementations, the method further includes detecting that a person is present in an environment of the client device based on a signal from the presence detection sensor, and causing one or more cameras to provide a stream of image frames in response to detecting that a person is present in the environment.

[0027] In some implementations, a client device is provided that includes at least one vision component, at least one microphone, one or more processors, and a memory operably coupled to the one or more processors. The memory stores instructions that cause one or more of the processors to perform the following operations in response to execution of the instructions by one or more of the processors: receiving a stream of vision data based on an output from the vision component of the client device, processing the vision data using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a user's invocation gesture captured by the vision data and a user's gaze directed toward the client device, detecting based on the monitoring the occurrence of both the invocation gesture and the gaze, and in response to detecting the occurrence of both the invocation gesture and the gaze, transmitting by the client device to one or more remote automated assistant components one or both of additional vision data based on the output from the vision component and audio data based on the output from the microphone of the client device. The operations may further include optionally receiving response content in response to the transmitting, and rendering the response content by one or more user interface output devices of the client device.

[0028] In some implementations, a system is provided that includes at least one vision component and one or more processors that receive a stream of vision data based on an output from the vision component, wherein the one or more of the processors are configured to process the vision data using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a user's invocation gesture captured by the vision data and a user's gaze directed toward the client device, detect based on the monitoring the occurrence of both the invocation gesture and the gaze, and activate at least one function of an automated assistant in response to detecting the occurrence of both the invocation gesture and the gaze.

[0029] In addition, some implementations include one or more processors of one or more computing devices operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the methods described above. Some implementations also include one or more non-transitory computer-readable recording media storing computer instructions that may be executed by the one or more processors to perform any of the methods described above.

[0030] It should be understood that all combinations of the above concepts, and additional concepts described in more detail herein, are contemplated as part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2A] 1A-1D depict exemplary process flows demonstrating various aspects of the present disclosure according to various implementations. [Figure 2B] 1A-1D depict exemplary process flows demonstrating various aspects of the present disclosure according to various implementations. [Figure 2C] 1A-1D depict exemplary process flows demonstrating various aspects of the present disclosure according to various implementations. [Diagram 3] FIG. 1 shows an example of an assistant device and a user providing gestures and directed gaze, and also shows an image captured by the assistant device's camera as the user is providing the gestures and directed gaze. [Figure 4A] 1 is a flow diagram illustrating an example method according to implementations disclosed herein. [Figure 4B] 4B is a flow diagram illustrating specific examples of particular blocks of the example method of FIG. 4A. [Diagram 5] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] 1 illustrates an exemplary environment in which the techniques disclosed herein may be implemented. The exemplary environment includes one or more client computing devices 106. Each client device 106 may run a respective instance of an automation assistant client 110. One or more cloud-based automation assistant components 130 may be implemented in one or more computing systems (collectively referred to as "cloud" computing systems) communicatively coupled to the client devices 106 via one or more local area and / or wide area networks (e.g., the Internet) generally shown at 114. The cloud-based automation assistant components 130 may be implemented, for example, by a cluster of high performance servers.

[0033] In various implementations, an instance of an automation assistant client 110, through its interaction with one or more cloud-based automation assistant components 130, may form what appears from a user's perspective to be a logical instance of an automation assistant 120 with which the user may engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction). One instance of such an automation assistant 120 is shown within a dashed line in FIG. 1 . Thus, it should be understood that each user who interacts with an automation assistant client 110 executing on a client device 106 may actually interact with that user's own logical instance of the automation assistant 120. For brevity and simplicity, the term "automation assistant" as used herein as "providing a service" to a particular user refers to a combination of an automation assistant client 110 executing on a client device 106 operated by the user and, optionally, one or more cloud-based automation assistant components 130 (which may be shared among multiple automation assistant clients 110). It should also be understood that in some implementations, the automation assistant 120 may respond to requests from any user regardless of whether the user is actually "serviced" by that particular instance of the automation assistant 120.

[0034] The one or more client devices 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (which may possibly include a visual sensor), a smart appliance such as a smart TV (or a regular TV with a dongle connected to a network with automation assistant capabilities), and / or a user wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. As mentioned above, some client devices 106 may take the form of an assistant device designed primarily to facilitate interaction between the user and the automation assistant 120 (e.g., a standalone interactive device with a speaker and a display).

[0035] The client device 106 may include one or more vision components 107 having one or more fields of view. The vision components 107 may take various forms, such as a monographic camera, a stereographic camera, a LIDAR component, a radar component, etc. The one or more vision components 107 may be used by the visual capture module 114 to capture, for example, a vision frame (e.g., an image frame (still or video)) of an environment in which the client device 106 is located. These vision frames may then be at least selectively analyzed by the gaze and gesture module 116 of the call engine 115 to monitor, for example, the occurrence of a particular gesture (of one or more candidate gestures) of the user captured by the vision frame and / or a directed gaze from the user (i.e., a gaze directed at the client device 106). The gaze and gesture module 116 may utilize one or more trained machine learning models 117 in monitoring the occurrence of the particular gesture and / or the directed gaze.

[0036] In response to detection of a particular gesture and directed gaze (and optionally in response to detection of one or more other conditions by other condition module 118), invocation engine 115 can invoke one or more previously dormant functions of automation assistant 120. Such dormant functions may include, for example, processing of particular sensor data (e.g., audio data, video, images, etc.) and / or rendering of particular content (e.g., graphical and / or audible).

[0037] As one non-limiting example, prior to detection of a particular gesture and directed gaze, visual and / or audio data captured at the client device 106 may be processed and / or temporarily buffered only locally at the client device 106 (i.e., without transmission to the cloud-based automated assistant component 130). However, in response to detection of a particular gesture and directed gaze, audio and / or visual data (e.g., recently buffered data and / or data received after detection) may be transmitted to the cloud-based automated assistant component 130 for further processing. For example, detection of a particular gesture and directed gaze may cause the user's verbal utterance to be fully processed by the automated assistant 120, eliminating the need for the user to say an explicit invocation phrase (e.g., "Okay, Assistant") to have responsive content generated by the automated assistant 120 and rendered to the user.

[0038] For example, rather than a user having to say, "Okay, Assistant. What's the forecast for today" to get today's forecast, the user might instead make a particular gesture, look at the client device 106, and simply say, "What's the forecast for today" while making the gesture and / or looking at the client device 106 or in close temporal proximity (e.g., within a threshold number of hours before and / or after) making the gesture and / or looking at the client device 106. Data corresponding to the verbal utterance "What's the forecast for today" (e.g., audio data capturing the verbal utterance or a textual or other semantic conversion thereof) may be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and directed gaze and in response to the verbal utterance being received during and / or in close temporal proximity to the gesture and directed gaze. In another example, rather than a user having to say "OK, Assistant, turn up the heat" to increase the temperature in the user's home via a connected thermostat, the user might instead make a particular gesture, look at the client device 106, and simply say "turn up the heat" while making the gesture and / or looking at the client device 106 or in close temporal proximity (e.g., within a threshold time before and / or after) making the gesture and / or looking at the client device 106. Data corresponding to the verbal utterance "turn up the heat" (e.g., audio data capturing the verbal utterance or a textual or other semantic transformation thereof) may be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and directed gaze and in response to the verbal utterance being received during and / or in close temporal proximity to the gesture and directed gaze.In another example, rather than a user having to say "OK, Assistant, open the garage door" to open the user's garage, the user may instead make a particular gesture, look at the client device 106, and simply say "open the garage door" while making the gesture and / or looking at the client device 106 or in close temporal proximity (e.g., within a threshold time before and / or after) to making the gesture and / or looking at the client device 106. Data corresponding to the verbal utterance "open the garage door" (e.g., audio data capturing the verbal utterance or a textual or other semantic transformation thereof) may be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and the directed gaze and in response to the verbal utterance being received during and / or in close temporal proximity to the gesture and the directed gaze. In some implementations, the transmission of the data by the client device 106 may be further conditioned on the occurrence of one or more additional conditions by the other conditions module 118 determining the occurrence of one or more additional conditions. For example, the transmission of the data may be further based on local voice activity detection processing of the audio data performed by the other condition module 118 indicating that voice activity is present in the audio data. Also, for example, the transmission of the data may additionally or alternatively be based on determining by the other condition module 118 that the audio data corresponds to a user who provided a gesture and a directed gaze. For example, a direction of the user (relative to the client device 106) may be determined based on visual data, and the transmission of the data may be further based on determining by the other condition module 118 that verbal speech in the audio data comes from the same direction (e.g., using beamforming and / or other techniques).Also, for example, a user profile of the user can be determined based on the visual data (e.g., using facial recognition), and the transmission of the data may be further based on determining by the other condition module 118 that the verbal utterances in the audio data have voice characteristics that match the user profile. As yet another example, the transmission of the data may additionally or alternatively be based on determining by the other condition module 118 based on the visual data that the user's mouth movements occurred simultaneously with or within a threshold amount of time of the detected gesture and / or directed gaze of the user. The other condition module 118 may optionally utilize one or more other machine learning models 119 in determining that the other condition exists. Further description of the gaze and gesture module 116 and the implementation of the other condition module 118 are provided herein (e.g., with reference to Figures 2A-2C).

[0039] The client computing device 106 and the computing devices running the cloud-based automation assistant component 130 may each include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components for facilitating communication over a network. Operations performed by the client computing device 106 and / or by the automation assistant 120 may be distributed across multiple computer systems. The automation assistant 120 may be implemented as a computer program running on one or more computers at one or more locations coupled together, for example through a network.

[0040] As mentioned above, in various implementations, the client computing device 106 may operate the automation assistant client 110. In some of those various implementations, the automation assistant client 110 may include a voice capture module 112, the above-mentioned visual capture module 114, and a call engine 115, which may include a gaze and gesture module 116 and optionally other condition modules 118. In other implementations, one or more aspects of the voice capture module 112, the visual capture module 114, and / or the call engine 115 may be implemented separately from the automation assistant client 110, for example, by one or more cloud-based automation assistant components 130.

[0041] The audio capture module 112, which in various implementations may be implemented using any combination of hardware and software, may interface with hardware such as a microphone 109 or other pressure sensor to capture an audio recording of the user's verbal utterances. Various types of processing may be performed on this audio recording for various purposes, as described below. The visual capture module 114, which in various implementations may be implemented using any combination of hardware and software, may be configured to interface with the visual component 107 to capture one or more visual frames (e.g., digital images) corresponding to any adaptable field of view of the visual sensor 107.

[0042] The voice capture module 112 may be configured to capture the user's voice, for example by microphone 109, as described above. Additionally or alternatively, in some implementations, the voice capture module 112 may be further configured to convert the captured audio into text and / or other representations or embeddings, for example using speech-to-text ("STT") processing techniques. However, because the client device 106 may be relatively constrained in terms of computational resources (e.g., processor cycles, memory, battery, etc.), the voice capture module 112 local to the client device 106 may be configured to convert a finite number of different spoken phrases, such as phrases that invoke the automated assistant 120, into text (or other forms, such as lower dimensional embeddings). Other voice inputs may be sent to the cloud-based automated assistant component 130, which may include a cloud-based STT module 132.

[0043] The cloud-based TTS module 131 may be configured to utilize the virtually limitless resources of the cloud to convert text data (e.g., natural language responses produced by the automation assistant 120) into computer-generated speech output. In some implementations, the TTS module 131 may provide the computer-generated speech output to the client device 106 for direct output, for example, using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automation assistant 120 may be provided to the client device 106, and then a local TTS module of the client device 106 may convert the text data into computer-generated speech that is output locally.

[0044] The cloud-based STT module 132 may be configured to utilize the virtually unlimited resources of the cloud to convert voice data captured by the voice capture module 112 into text, which may then be provided to the natural language understanding module 135. In some implementations, the cloud-based STT module 132 may convert an audio recording of the voice into one or more phonemes and then convert the one or more phonemes into text. Additionally or alternatively, in some implementations, the STT module 132 may use a state decoding graph. In some implementations, the STT module 132 may generate multiple candidate text interpretations of the user's utterance and select a given interpretation from the candidates using one or more techniques.

[0045] The automation assistant 120 (and in particular the cloud-based automation assistant component 130) may include an intent understanding module 135, the TTS module 131 described above, the STT module 132 described above, and other components described in more detail herein. In some implementations, one or more of the modules and / or modules of the automation assistant 120 may be omitted, combined, and / or implemented in a component that is separate from the automation assistant 120. In some implementations, one or more of the components of the automation assistant 120, such as the intent understanding module 135, the TTS module 131, the STT module 132, etc., may be implemented at least partially on the client device 106 (e.g., in combination with or to the exclusion of a cloud-based implementation).

[0046] In some implementations, the automated assistant 120 generates various content for audible and / or graphical rendering to the user via the client device 106. For example, the automated assistant 120 may generate content such as a weather forecast, a daily schedule, and the like, and may have the content rendered in response to detecting gestures and / or directed gaze from the user as described herein. Also, for example, the automated assistant 120 may generate content in response to a user's free-form natural language input provided via the client device 106, in response to a user's gestures detected by visual data from the visual component 107 of the client device, and the like. As used herein, a free-form input is an input made by a user and not constrained to a set of options presented for selection by the user. A free-form input can be, for example, a typed input and / or a spoken input.

[0047] The natural language processor 133 of the intent understanding module 135 may process natural language input generated by a user via the client device 106 and generate annotated output (e.g., in text format) for use by one or more other components of the automated assistant 120. For example, the natural language processor 133 may process free-form natural language input generated by a user via one or more user interface input devices of the client device 106. The annotated output generated includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0048] In some implementations, the natural language processor 133 is configured to identify and annotate various types of grammatical information in the natural language input. For example, the natural language processor 133 may include a morphology module that may split individual words into morphemes and / or annotate the morphemes, for example, with their morpheme classes. The natural language processor 133 may also include a part-of-speech tagger configured to annotate words with their grammatical roles. Also, for example, in some implementations, the natural language processor 133 may additionally and / or alternatively include a dependency parser (not shown) configured to determine syntactic relationships between words in the natural language input.

[0049] In some implementations, the natural language processor 133 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate mentions of entities in one or more segments, such as references to people (including, e.g., literary characters, celebrities, notable people, etc.), organizations, places (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as a knowledge graph (not shown), and the entity tagger of the natural language processor 133 may utilize such databases for tagging entities.

[0050] In some implementations, the natural language processor 133 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be utilized to resolve the word "there" in the natural language input "I liked Hypothetical Cafe last time we ate there" to "Hypothetical Cafe."

[0051] In some implementations, one or more components of the natural language processor 133 may rely on annotations from one or more other components of the natural language processor 133. For example, in some implementations, a designated entity tagger may rely on annotations from a coreference resolver and / or a dependency parser in annotating all references to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from a dependency parser in clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 133 may use relevant prior input and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0052] The intent understanding module 135 may further include an intent matcher 134 configured to determine the intent of a user engaging in an interaction with the automated assistant 120. Although shown separately from the natural language processor 133 in FIG. 1, in other implementations the intent matcher 134 may be an integral part of the natural language processor 133 (or of a pipeline that includes the natural language processor 133 more broadly). In some implementations, the natural language processor 133 and the intent matcher 134 may collectively form the intent understanding module 135 described above.

[0053] The intent matcher 134 may use various techniques to determine the user's intent, for example, based on output from the natural language processor 133 (which may include annotations and terms of the natural language input), based on the user's touch input on a touch-sensitive display of the client device 106, and / or based on gestures and / or other visual cues detected in the visual data. In some implementations, the intent matcher 134 may have access to one or more databases (not shown) that include multiple mappings, for example, between grammars and response actions (or intents more broadly), between visual cues and response actions, and / or between touch input and response actions. For example, the grammars included in the mappings can be selected and / or learned over time and may represent common intents of users. For example, one grammar "play <artist>"but, <artist>A visual cue may be mapped to an intent that invokes a response action that causes music by "Weather|Forecast" to be played on the user-operated client device 106. Another grammar, "[weather|forecast] today" may be matched to user queries such as "what's the weather today" and "what's the forecast for today?". As another example, the mapping of visual cues to actions may include "blanket" mappings that may apply to multiple users (e.g., all users) and / or user-specific mappings. Some examples of visual cue to action mappings include mappings for gestures. For example, a "wave" gesture can be mapped to an action that causes tailored content (tailored to the user giving the gesture) to be rendered to the user, a "thumbs up" gesture can be mapped to a "play music" action, and a "high five" gesture can be mapped to "routines" of automated assistant actions to be performed such as turning on a smart coffee maker, turning on certain smart lights, and audibly rendering news summaries.

[0054] In addition to or instead of grammars, in some implementations the intent matcher 134 may use one or more trained machine learning models, alone or in combination with one or more grammars, visual cues, and / or touch inputs. These trained machine learning models may also be stored in one or more databases and trained to identify intents, for example, by embedding data indicative of a user's speech and / or any detected user-given visual cues into a reduced-dimensional space and then determining which other embeddings (and therefore intents) are closest, using techniques such as Euclidean distance, cosine similarity, etc.

[0055] On "play <artist>As can be seen in the example grammar for ", some grammars include slots that may be filled by slot values ​​(or "parameters") (e.g., <artist>) Slot values ​​may be determined in a variety of ways. In many cases, the user proactively provides slot values. For example, the grammar "Order me a <topping>For "order me a sausage pizza," the user may be likely to say the phrase "order me a sausage pizza," in which case the slot <topping>are automatically filled. Additionally or alternatively, if the user invokes a grammar that includes slots that are filled by slot values ​​without the user proactively providing the slot values, the automated assistant 120 may prompt the user for those slot values ​​(e.g., "what type of crust do you want on your pizza?"). In some implementations, the slots may be filled by slot values ​​based on visual cues detected based on visual data captured by the vision component 107. For example, a user may say something like "Order me this many cat bowls" while holding up three fingers toward the vision component 107 of the client device 106. Or, a user may say something like "Find me more movies like this" while holding up a DVD case for a particular movie.

[0056] In some implementations, the automation assistant 120 may facilitate (e.g., "mediate") transactions between a user and an agent, which may be an independent software process that receives input and provides response output. Some agents may take the form of a third-party application that may or may not run on a computing system that is separate from the computing system that runs the cloud-based automation assistant component 130, for example. One type of user intent that may be identified by the intent matcher 134 is to engage a third-party application. For example, the automation assistant 120 may provide access to an application programming interface ("API") to a pizza delivery service. A user may invoke the automation assistant 120 and provide a command such as "I'd like to order a pizza". The intent matcher 134 may map this command to a grammar that triggers the automation assistant 120 to engage with the third-party pizza delivery service. The third-party pizza delivery service may provide the automation assistant 120 with a minimal list of slots that need to be filled to fulfill the pizza delivery order. The automation assistant 120 may generate and provide to the user (via the client device 106) natural language output requesting parameters for the slot.

[0057] The fulfillment module 138 may be configured to receive the predicted / inferred intent output by the intent matcher 134 and the associated slot values ​​(whether proactively provided by the user or requested by the user) and fulfill (or "resolve") the intent. In various implementations, the fulfillment (or "resolution") of the user's intent may cause various fulfillment information (also referred to as "response" information or data) to be generated / obtained, for example, by the fulfillment module 138.

[0058] The performance information may take various forms as intent may be fulfilled in various ways. Assume a user requests straightforward information such as "Where were the outdoor shots of 'The Shining' filmed?" The user's intent may be determined to be a search query, for example, by the intent matcher 134. The intent and content of the search query may be provided to the performance module 138, which may communicate with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., knowledge graphs, etc.) for response information, as shown in FIG. 1. The performance module 138 may provide data indicative of the search query (e.g., text of the query, reduced dimensional embedding, etc.) to the search module 150. The search module 150 may provide response information, such as GPS coordinates or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon". This response information may form part of the performance information generated by the performance module 138.

[0059] Additionally or alternatively, the fulfillment module 138 may be configured to receive, e.g., from the intent understanding module 135, the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.) and trigger a responsive action. The responsive action may include, e.g., ordering a product / service, starting a timer, setting a reminder, initiating a call, playing media, sending a message, initiating a routine of multiple actions, etc. In some such implementations, the fulfillment information may include a slot value associated with the fulfillment, an acknowledgment response (which may possibly be selected from predefined responses), etc.

[0060] Additionally or alternatively, the fulfillment module 138 may be configured to infer the user's intents (e.g., based on time of day, past interactions, etc.) and obtain response information regarding those intents. For example, the fulfillment module 138 may be configured to obtain a daily schedule summary for the user, a weather forecast for the user, and / or other content for the user. Further, the fulfillment module 138 may cause such content to be "pushed" to be rendered graphically and / or audibly to the user. For example, rendering such content may be a dormant function that is invoked in response to the invocation engine 115 detecting the occurrence of a particular gesture and directed gaze.

[0061] The natural language generator 136 may be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 136 may be configured to receive as input performance information related to the performance of an intent and generate the natural language output based on the performance information. Additionally or alternatively, the natural language generator 136 may receive information from other sources, such as third-party applications that the natural language generator 136 may use to configure the natural language output for the user.

[0062] 2A, 2B, and 2C, various examples are shown of how the gaze and gesture module 116 can detect specific gestures and / or directed gazes and how the invocation engine 115 can responsively invoke one or more previously dormant automated assistant functions.

[0063] Turning first to FIG. 2A , the visual capture module 114 provides visual frames to the gaze and gesture module 116. In some implementations, the visual capture module 114 provides a real-time stream of visual frames to the gaze and gesture module 116. In some of those implementations, the visual capture module 114 begins providing visual frames in response to a signal from a separate presence detection sensor 105 indicating the presence of a person in an environment having the client device 106. For example, the presence detection sensor 105 can be a PIR sensor and can provide a signal to the visual capture module 114 in response to detecting the presence of a person. The visual capture module 114 may refrain from providing any visual frames to the gaze and gesture module 116 unless the presence of a person is detected. In other implementations in which the visual capture module 114 only selectively provides visual frames to the gaze and gesture module 116, additional and / or alternative cues may be utilized to initiate such provision. For example, the presence of a person may be detected based on audio data from the voice capture module 112, based on analysis of a visual frame by one or more other components, and / or based on other signals.

[0064] The gaze and gesture module 116 processes the visual frames using one or more machine learning models 117 to monitor the occurrence of both directed gaze and specific gestures. When both directed gaze and specific gestures are detected, the gaze and gesture module 116 provides an indication of the detection of the gaze and gesture to the invocation engine 115.

[0065] In FIG. 2A , the visual frames and / or audio data (provided by the voice capture module 112) are also provided to the other condition module 118. The other condition module 118 processes the provided data, optionally using one or more other machine learning models 119 to monitor the occurrence of one or more other conditions. For example, the other condition may be detecting any voice activity based on the audio data, detecting the presence of a spoken call phrase in the audio data, detecting voice activity that is from the user's direction or position based on the audio data, detecting that the user is an authorized user based on the visual frames and / or audio data, detecting the user's mouth movement (given gestures and directed gaze) based on the visual frames, etc. When the other condition is detected, the other condition module 118 provides an indication of the occurrence of the other condition to the call engine 115.

[0066] When the invocation engine 115 receives indications of directed gaze and gesture and temporally proximate indications of other conditions, the invocation engine 115 causes invocation 101 of the dormant function. For example, invocation 101 of the dormant function may include one or more of activating a display screen of the client device 106, causing content to be visually and / or audibly rendered by the client device 106, causing visual frames and / or audio data to be transmitted by the client device 106 to one or more cloud-based automated assistant components 130, etc.

[0067] In some implementations, as described in more detail in connection with Figures 2B and 2C, the gaze and gesture module 116 can use one or more first machine learning models 117 to detect directed gaze and one or more second machine learning models 117 to detect gestures.

[0068] In some other implementations, the gaze and gesture module 116 may utilize an end-to-end machine learning model that may be utilized to accept a visual frame (or features thereof) as input and generate an output (based on processing the input on the model) indicating whether a particular gesture and directed gaze have occurred. Such a machine learning model may be, for example, a neural network model, such as a recurrent neural network (RNN) model that includes one or more memory layers (e.g., a long short-term memory (LSTM) layer). Training of such an RNN model may be based on training examples that include a sequence of visual frames (e.g., a video) as training example input and an indication of whether the sequence includes both a gesture and a directed gaze as training example output. For example, the training example output may be a single value that indicates whether both a gesture and a directed gaze are present. As another example, the training example output may include a first value indicating whether a directed gaze is present and N additional values ​​each indicating whether a corresponding one of the N gestures is included (thereby enabling training of the model to predict a corresponding probability for each of the N distinct gestures). As yet another example, the training example output may include a first value indicating whether a directed gaze is present and a second value indicating whether any of one or more specific gestures are present (thereby enabling training of the model to predict a corresponding probability for whether any of the gestures is included).

[0069] In implementations where the model is trained to predict a corresponding probability for each of N distinct gestures, the gaze and gesture module 116 can optionally provide an indication to the invocation engine 115 of which of the N gestures occurred. Additionally, the invocation 101 of a dormant function by the invocation engine 115 may depend on which of the N distinct gestures occurred. For example, for a "wave" gesture, the invocation engine 115 can cause certain content to be rendered on the client device's display screen, for a "thumbs up" gesture, the invocation engine 115 can cause audio data and / or visual frames to be sent to the cloud-based automated assistant component 130, and for a "high five" gesture, the invocation engine 115 can cause a "routine" of automated assistant actions to be performed, such as turning on a smart coffee maker, turning on certain smart lights, and audibly rendering a news roundup.

[0070] 2B illustrates an example in which the gesture and gaze detection module 116 includes a gesture module 116A that utilizes a gesture machine learning model 117A in monitoring the occurrence of a gesture, and a gaze module 116B that utilizes a gaze machine learning model 117B in monitoring the occurrence of a directed gaze. Other condition modules 118 are not shown in FIG. 2B for simplicity, but may also be optionally utilized in combination with the gesture modules 116A and 117B in a manner similar to that described in connection with FIG. 2A.

[0071] 2B, visual capture module 114 provides a visual frame. A lower resolution version of the visual frame is provided to gesture module 116A, and a higher resolution version of the visual frame is stored in buffer 104. The lower resolution version is lower resolution than the higher resolution version. The higher resolution version can be uncompressed or less compressed than the lower resolution version. Buffer 104 can be a first-in, first-out buffer and can temporarily store the most recent duration of the higher resolution visual frame.

[0072] In the example of FIG. 2B, the gesture module 116A can process lower resolution visual frames when monitoring for the presence of a gesture, and the gaze module 116B can be inactive until the gesture module 116A detects the occurrence of a gesture. When the occurrence of a gesture is detected, the gesture module 116A can provide an indication of the detection of the gesture to the gaze module 116B and the call engine 115. The gaze module 116B can be activated in response to receiving the indication of the detection of the gesture and can retrieve the buffered higher resolution visual frames from the buffer 104 and utilize the buffered higher resolution visual frames (and optionally even higher resolution visual frames) in determining whether a directed gaze exists. In this manner, the gaze module 116B is only selectively activated, thereby saving computational resources that would otherwise be consumed by additional processing of the higher resolution visual frames by the gaze module 116B.

[0073] The gesture module 116A can use one or more gesture machine learning models 117A to detect specific gestures. Such machine learning models can be neural network models, such as RNN models, including one or more memory layers. Training of such RNN models can be based on training examples that include a sequence of visual frames (e.g., a video) as a training example input and an indication of whether the sequence includes one or more specific gestures as a training example output. For example, the training example output can be a single value indicating whether a single specific gesture is present. For example, the single value can be "0" when a single specific gesture is not present and "1" when a single specific gesture is present. In some of these examples, multiple gesture machine learning models 117A, each tailored to a different single specific gesture, are utilized. As another example, the training example output can include N values, each indicating whether a corresponding one of N gestures is included (thereby enabling training of a model to predict a corresponding probability for each of the N distinct gestures). In implementations in which the model is trained to predict a corresponding probability for each of N distinct gestures, the gesture module 116A can optionally provide an indication of which of the N gestures occurred to the invocation engine 115. Additionally, invocation of a dormant function by the invocation engine 115 may depend on which of the N distinct gestures occurred.

[0074] The gaze module 116B can use one or more gaze machine learning models 117B to detect directed gaze. Such machine learning models can be neural network models, such as, for example, convolutional neural network (CNN) models. Training of such CNN models can be based on training examples that include visual frames (e.g., images) as training example inputs and an indication of whether the images include directed gaze as training example outputs. For example, the training example output can be a single value indicating whether directed gaze is present. For example, the single value can be "0" when directed gaze is absent, "1" when directed gaze is present that is directed directly at the sensor capturing the image or within 5 degrees of the sensor capturing the image, "0.75" when directed gaze is present within 5-10 degrees of the sensor capturing the image, etc.

[0075] In some of these and / or other implementations, the gaze module 116B determines a directed gaze only when a directed gaze is detected with at least a threshold probability and / or for at least a threshold duration. For example, a stream of image frames can be processed using a CNN model, and processing of each frame may result in a corresponding probability that the frame contains a directed gaze. The gaze module may determine that a directed gaze exists only if at least X% of a sequence of image frames (corresponding to the threshold duration) have a corresponding probability that meets the threshold. For example, assume that X% is 60%, the probability threshold is 0.7, and the threshold duration is 0.5 seconds. Further assume that 10 image frames correspond to 0.5 seconds. If image frames are processed to yield a probability of [0.75, 0.85, 0.5, 0.4, 0.9, 0.95, 0.85, 0.89, 0.6, 0.85], then a directed gaze may be detected because 70% of the frames exhibited a directed gaze with a probability greater than 0.7. In these and other methods, a directed gaze may be detected even if the user briefly deviates from the direction of the user's gaze. Additional and / or alternative machine learning models (e.g., RNN models) and / or techniques may be utilized to detect directed gaze that occurs for at least a threshold duration.

[0076] 2C illustrates another example in which the gesture and gaze detection module 116 includes a gesture module 116A that utilizes a gesture machine learning model 117A in monitoring the occurrence of a gesture, and a gaze module 116B that utilizes a gaze machine learning model 117B in monitoring the occurrence of a directed gaze. Other condition modules 118 are not shown in FIG. 2C for simplicity, but may also be optionally utilized in combination with the gesture modules 116A and 117B in a manner similar to that described in connection with FIG. 2A. Also, the buffer 104 and higher / lower resolution visual frames of FIG. 2B are not shown in FIG. 2C for simplicity, but similar techniques may be implemented in FIG. 2C (e.g., a higher resolution portion of the visual frame may be provided to the gaze module 116B).

[0077] In FIG. 2C, the visual capture module 114 provides the visual frames to the detection and classification module 116C. The detection and classification module 116C utilizes an object detection and classification machine learning model 117C to classify various regions of each visual frame. For example, the detection and classification module 116C can classify the person regions (if any) of each visual frame that correspond to a person and provide an indication of such person regions for each visual frame to the gesture module 116A and the gaze module 116B. Also, for example, the detection and classification module 116C can classify the regions (if any) of each visual frame that correspond to the person's body regions (e.g., arms and torso) and provide an indication of such regions for each visual frame to the gesture module 116A. Also, for example, the detection and classification module 116C can classify the regions (if any) of each visual frame that correspond to the person's face regions and provide an indication of such regions for each visual frame to the gaze module 116B.

[0078] In some implementations, the gesture module 116A can utilize the provided region to process only the corresponding portion of each visual frame. For example, the gesture module 116A can "crop" and resize the visual frames to process only the portion that includes the person and / or body region. In some of those implementations, the gesture machine learning model 117A can be trained based on the visual frames that are "cropped," and the resizing can be to a size that matches the dimensions of the input of such model. In some additional or alternative implementations, the gesture module 116A can utilize the provided region to skip processing some visual frames all together (e.g., visual frames that are indicated as not including the person and / or body region). In still other implementations, the gesture module 116A can utilize the provided region as an attention mechanism to focus the processing of each visual frame (e.g., as a separate attention input to the gesture machine learning model 117A).

[0079] Similarly, in some implementations, the gaze module 116B can utilize the provided region to process only the corresponding portion of each visual frame. For example, the gaze module 116B can "crop" and resize the visual frames to process only the portion that includes the human and / or face region. In some of those implementations, the gaze machine learning model 117B can be trained based on the visual frames that are "cropped," and the resizing can be to a size that matches the dimensions of the input of such model. In some additional or alternative implementations, the gaze module 116B can utilize the provided region to skip processing some visual frames all together (e.g., visual frames that are indicated as not including human and / or face regions). In still other implementations, the gaze module 116B can utilize the provided region as an attention mechanism to focus the processing of each visual frame (e.g., as a separate attention input to the gaze machine learning model 117B).

[0080] In some implementations, the detection and classification model 116C may additionally or alternatively provide an indication of the particular region to the other condition module 118 (not shown in FIG. 2C for simplicity) for use by the other condition module 118. For example, the face region may be used by the other condition module 118 in detecting mouth movements utilizing a corresponding mouth movement machine learning model when mouth movements are an additional condition for invoking an inactive feature.

[0081] In some implementations, the detection and classification model 116C can additionally or alternatively provide an indication to the gesture module 116A and the gaze module 116B of regions that are classified as TV or other video display sources. In some of those implementations, the modules 116A and 116B can crop those regions from the processed visual frame, focus outside of those regions, and / or otherwise ignore those regions in detection or make it less likely that detection is based on such regions. In these and other ways, falsely detected invocation of dormant functions may be mitigated.

[0082] Figure 3 shows an example of the client device 106 and vision component 107 of Figure 1. In Figure 3, the example client device is shown as 106A and further includes a speaker and a display. In Figure 3, the example vision component is shown as 107A and is a camera. Figure 3 also shows a user 301 providing a hand movement gesture (indicated by the "movement" line near the user's right hand) and a gaze directed toward the camera 107A.

[0083] 3 also shows an example of an assistant device and a user providing a gesture and a directed gaze, and also shows an example image 360 ​​captured by camera 107A as the user provides the gesture and the directed gaze. It can be seen that the user and a television behind the user (and therefore not visible in the perspective view of FIG. 3) are captured in image 360.

[0084] A bounding box 361 is provided within image 360 ​​to represent a region of the image that may be determined (e.g., by detection and classification module 116C of FIG. 2C) to correspond to a person. In some implementations, a gesture detection module operating on client device 106A may process (or focus on) only that portion of the image when monitoring for a particular gesture based on that portion being indicated as corresponding to a person.

[0085] A bounding box 362 is also provided within image 360, representing a region of the image that may be determined (e.g., by detection and classification module 116C of FIG. 2C) to correspond to a face. In some implementations, a gaze detection module operating on client device 106A may process (or focus on) only that portion of the image when monitoring for directed gaze based on that portion being indicated as corresponding to a face. Although only a single image is shown in FIG. 3, in various implementations directed gaze detection and / or gesture detection may be based on a sequence of images as described herein.

[0086] A bounding box 363 is also provided within image 360 ​​to represent an area of ​​the image that may be determined to correspond to a video display and may cause a false positive of a visual cue. For example, a television may render a moving image showing one or more people making a gesture, looking at the camera, etc., any of which may be misinterpreted as an occurrence of a gesture and / or directed gaze. In some implementations, the detection and classification module 116C of FIG. 2C may determine such an area (e.g., based on detecting a classification of a TV) and / or such an area may be determined based on an analysis of image 360 ​​and preceding images to determine that the area has a display frequency corresponding to a display frequency of a video display (e.g., approximately 60 Hz, 120 Hz, and / or other typical video display frequencies). In some implementations, the gaze detection module and / or gesture module may crop the area from the processed visual frame, look outside of the area, and / or otherwise ignore the area in the detection or make it less likely that the detection is based on such an area. In these and other ways, falsely detected invocations of dormant functions may be mitigated.

[0087] FIG. 4A is a flow diagram illustrating an example method 400 according to implementations disclosed herein. FIG. 4B is a flow diagram illustrating an example implementation of blocks 402, 404, and 406 of FIG. 4A. For convenience, the operations of the flow diagrams of FIG. 4A and FIG. 4B are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a computing system (e.g., a client device and / or a remote computing system) that implements the automation assistant 120. Furthermore, although the operations of the method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0088] In block 402, the system receives visual data based on the output from the visual component. In some implementations, the visual component may be integrated into a client device, including an assistant client. In some implementations, the visual component is separate from the client device, but may communicate with the client device. For example, the visual component may include a standalone smart camera that communicates wired and / or wirelessly with a client device, including an assistant client.

[0089] In block 404, the system processes the visual data using at least one machine learning model to monitor the occurrence of both gestures and directed gaze.

[0090] In block 406, the system determines whether both a gesture and a gaze are detected based on monitoring block 404. If not, the system returns to block 402 to receive additional visual data and perform another iteration of blocks 404 and 406. In some implementations, the system determines that both a gesture and a gaze are detected based on detecting that the gesture and the directed gaze occur simultaneously or within a threshold temporal proximity of each other. In some additional or alternative implementations, the system determines that both a gesture and a gaze are detected based on detecting that the gesture is at least a threshold duration (e.g., a "wave" for at least duration X or a "thumbs up" for at least duration X) and / or that the directed gaze is at least a threshold duration (which may optionally be the same or different than the threshold duration used for the gesture duration).

[0091] In an iteration of block 406, if the system determines that both a gesture and a gaze have been detected based on the monitoring of block 404, the system optionally proceeds to block 408 (or directly to block 410 if block 408 is not included).

[0092] In optional block 408, the system determines whether one or more other conditions are met. If the one or more other conditions are not met, the system returns to block 402 to receive additional visual data and perform another iteration of blocks 404, 406, and 408. If the one or more other conditions are met, the system proceeds to block 410. The system may determine whether one or more other conditions are met using the visual data, audio data, and / or other sensor or non-sensor data received in block 402. Various other conditions may be considered by the system, such as conditions explicitly described herein.

[0093] In block 410, the system activates one or more inactive automated assistant functions. The system can activate various inactive automated assistant functions, such as the inactive automated assistant functions explicitly described herein. In some implementations, different types of gestures can be monitored in block 404, and which inactive functions are activated in block 410 may depend on the particular type of gesture detected in the monitoring of block 404.

[0094] In block 412, the system monitors for deactivation conditions for the automated assistant functionality activated in block 410. Deactivation conditions may include, for example, a timeout, at least a threshold duration of detected verbal input and / or lack of detected directed gaze, an explicit stop command (verbal, gestured, or touch-entered), and / or other conditions.

[0095] In block 414, the system determines whether a deactivation condition is detected based on the monitoring of block 412. If a deactivation condition is not detected, the system returns to block 412 and continues monitoring for a deactivation condition. If a deactivation condition is detected, the system may deactivate the functionality activated in block 410 and returns to block 402 to again receive visual data and again monitor for the occurrence of both an invocation gesture and a gaze.

[0096] As one example of blocks 412 and 414 where the activated functionality includes streaming audio data to one or more cloud-based automated assistant components, the system may stop streaming in response to detecting a lack of voice activity (e.g., using a VAD) for at least a threshold duration, in response to an explicit stop command, or in response to detecting that the user's gaze has not been directed at the client device for at least a threshold duration (by continued gaze monitoring).

[0097] Turning now to Figure 4B, an example of an implementation of blocks 402, 404, and 406 of Figure 4A is shown. Figure 4B shows an example where gesture detection and directed gaze detection are performed utilizing separate models, and directed gaze monitoring is only responsive to first detecting a gesture. In Figure 4B, block 402A is a specific example of block 402 of Figure 4A, blocks 404A and 404B are specific examples of block 404 of Figure 4A, and blocks 406A and 406B are specific examples of block 406 of Figure 4A.

[0098] In block 402A, the system receives and buffers visual data.

[0099] In block 404A, the system processes the visual data using a gesture machine learning model to monitor the occurrence of a gesture. In some implementations, block 404A includes a sub-block 404A1 in which the system processes a portion of the visual data using a gesture machine learning model based on detecting that the portion corresponds to a person and / or body region.

[0100] In block 406A, the system determines whether a gesture was detected based on the monitoring of block 404 A. If a gesture was not detected, the system returns to block 402A to receive and buffer additional visual data and perform another iteration of block 404A.

[0101] In an iteration of block 406A, if the system determines that a gesture was detected based on the monitoring of block 404A, the system proceeds to block 404B.

[0102] In block 404B, the system processes the buffered and / or additional visual data using a gaze machine learning model to monitor the occurrence of directed gaze.

[0103] In some implementations, block 404B includes a sub-block 404B1 in which the system processes a portion of the visual data using a gaze machine learning model based on detecting that the portion corresponds to a human and / or facial region.

[0104] In block 406B, the system determines whether a directed gaze is detected based on the monitoring of block 404B. If a directed gaze is not detected, the system returns to block 402A to receive and buffer additional visual data and perform another iteration of block 404A.

[0105] If, in an iteration of block 406B, the system determines that a gesture was detected based on the monitoring of block 404B, the system proceeds to blocks 408 or 410 of FIG. 4A.

[0106] Various examples of activating a dormant assistant function in response to detecting both a specific gesture and a directed gaze are described herein. However, in various implementations, a dormant assistant function may be activated in response to detecting only one of a specific gesture and a directed gaze, optionally in combination with one or more other conditions, such as those described herein. For example, in some of these various implementations, a dormant assistant function may be activated in response to detecting a user's directed gaze for at least a threshold duration together with other concurrent conditions, such as the user's mouth movement. Also, for example, in some of these various implementations, a dormant assistant function may be activated in response to detecting a user's gesture together with other concurrent and / or temporally proximate conditions, such as detected voice activity.

[0107] 5 is a block diagram of an example computing device 510 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices, the user-controlled resource module 130, and / or other components may include one or more components of the example computing device 510.

[0108] Generally, the computing device 510 includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices of other computing devices.

[0109] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 510 or a communications network.

[0110] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display such as by an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computing device 510 to a user or to another machine or computing device.

[0111] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of the methods of Figures 4A and 4B and for implementing various components shown in Figures 1, 2A-2C, and 3.

[0112] These software modules are generally executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include several memories including a main random access memory (RAM) 530 for storing instructions and data during execution of the program, and a read-only memory (ROM) 532 in which certain instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 526 in the storage subsystem 524 or other machines that may be accessed by the processor 514.

[0113] The bus subsystem 512 provides a mechanism for allowing the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0114] The computing device 510 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 510 shown in Figure 5 is intended only as a specific example intended to illustrate some implementations. Many other configurations of the computing device 510 having more or fewer components than the computing device shown in Figure 5 are possible.

[0115] In situations where the systems described herein may collect or otherwise monitor personal information about a user or utilize personal information and / or monitored information, the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how content that may be more relevant to the user should be received from a content server. Certain data may also be processed in one or more ways before being stored or used, such that personally identifiable information is removed. For example, the user's identity may be processed such that personally identifiable information cannot be determined about the user, or if geographic location information is obtained, the user's geographic location may be generalized (such as to the level of a city, zip code, or state) and thus the user's specific geographic location cannot be determined. Thus, the user may be able to control how information is collected and / or used about the user. For example, in some implementations, a user may disengage from an assistant device that uses vision component 107 and / or uses visual data from vision component 107 in monitoring the occurrence of gestures and / or directed gaze. [Explanation of symbols]

[0116] 104 Buffer 105 Presence detection sensor 106 Client Computing Devices 106A Client Device 107 Vision components, vision sensors 107A Camera 109 Microphone 110 Automation Assistant Client 112 Audio Capture Module 114 Visual Capture Module 115 Call Engine 116 Gaze and Gesture Module, Gesture and Gaze Detection Module 116A Gesture Module 116B Gaze Module 116C Detection and Classification Module, Detection and Classification Model 117 Machine Learning Models 117A Gesture Machine Learning Model 117B Staring Machine Learning Model 117C Object detection and classification machine learning models 118 Other Conditional Modules 119 Other Machine Learning Models 120 Automation Assistant 130 Cloud-based automation assistant components, resource modules 131 Cloud-based TTS module 132 Cloud-based STT module 133 Natural Language Processors 134 Intention Matcher 135 Natural language understanding module, intent understanding module 136 Natural Language Generator 138 Execution Module 150 Search Module 301 users 360 Images 361 Bounding Box 362 Bounding Box 400 ways 510 Computing Devices 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Storage Subsystem 525 Memory Subsystem, Memory 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 Read-Only Memory (ROM)< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. 1. A method implemented by one or more processors of a client device that facilitates contactless interaction between one or more users and an automated assistant, comprising: receiving a stream of image frames based on output from one or more cameras of the client device; using at least one trained machine learning model stored locally on the client device; a user's calling gesture captured by at least one of the image frames; and the user's gaze being directed at the client device; processing the image frames of the stream to monitor the occurrence of both Based on the monitoring step, the calling gesture; The gaze and detecting the occurrence of both in response to detecting the occurrence of both the calling gesture and the gaze, activating at least one function of the automated assistant; A method comprising:

2. the at least one function of the automated assistant that is activated in response to detecting the occurrence of both the invocation gesture and the gaze, Transmitting audio data captured by one or more microphones of the client device to a remote server associated with the automated assistant.

2. The method of claim 1, comprising:

3. the at least one function activated in response to detecting the occurrence of both the invocation gesture and the gaze, 3. The method of claim 1 or 2, further comprising transmitting an additional image frame to a remote server associated with the automation assistant, the additional image frame being received based on output from one or more of the cameras and after detecting the occurrence of both the invocation gesture and the gaze.

4. the at least one function activated in response to detecting the occurrence of both the invocation gesture and the gaze, processing of buffered audio data at the client device, the buffered audio data being stored in a memory of the client device and captured by one or more microphones of the client device, the processing of the buffered audio data comprising: Call phrase detection process; and Automatic Speech Recognition The method of any one of claims 1 to 3, comprising one or both of the following:

5. The method of claim 4 , wherein the processing of the buffered audio data comprises automatic speech recognition, the automatic speech recognition comprising speech-to-text processing.

6. and wherein the processing of the buffered audio data includes a call phrase detection process, the method comprising: In response to the call phrase detection process detecting the presence of a call phrase in the buffered audio data, transmitting further audio data captured by the one or more microphones of the client device to a remote server associated with the automated assistant; transmitting an additional image frame to a remote server associated with the automation assistant, the additional image frame being received after detecting the occurrence of both the invocation gesture and the gaze based on output from one or more of the cameras; and performing one or both of the steps of: The method according to claim 4 or 5.

7. processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both the invocation gesture and the gaze; using a first trained machine learning model to monitor occurrences of the calling gesture; using a second trained machine learning model to monitor the gaze of the user directed at the client device; 7. The method of claim 1 , comprising:

8. 8. The method of claim 7, wherein using the second trained machine learning model to monitor the user's gaze directed at the client device occurs only in response to detecting the occurrence of the calling gesture using the first trained machine learning model.

9. 9. The method of claim 7 or 8, wherein using the first trained machine learning model to monitor the occurrence of the calling gesture includes processing a first resolution version of the image frame using the first machine learning model, and using the second trained machine learning model to monitor the gaze of the user includes processing a second resolution version of the image frame using the second machine learning model.

10. receiving a stream of audio data frames based on output from one or more microphones of the client device; processing the audio data frames of the stream using at least one trained invocation phrase detection machine learning model stored locally on the client device to monitor occurrences of spoken invocation phrases; detecting the occurrence of the spoken call phrase based on the monitoring of the occurrence of the spoken call phrase; Further comprising: and activating the at least one function of the automated assistant in response to detecting the occurrence of the spoken invocation phrase in temporal proximity to both the invocation gesture and the gaze. The method of claim 1.

11. The at least one function to be activated is transmitting additional audio data frames captured by the one or more microphones of the client device to a remote server associated with the automation assistant; and Transmitting one or more additional image frames from one or more of the cameras to the remote server associated with the automation assistant.

11. The method of claim 10, comprising one or both of:

12. processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both the invocation gesture and the gaze; processing the image frames using a first trained machine learning model to predict regions of the image frames that contain human faces; processing the region of the image frame using a second trained machine learning model trained to detect the gaze of the user; 2. The method of claim 1, comprising:

13. processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both the invocation gesture and the gaze; determining that a region of the image frame corresponds to an electronic display; responsive to determining that the region corresponds to the electronic display, ignoring the region when monitoring the occurrence of both the calling gesture and the gaze.

2. The method of claim 1, comprising:

14. 14. The method of claim 13, wherein determining that the region of the image frame corresponds to the electronic display is based on detecting a display frequency within the region of the image frame that corresponds to a frequency of an electronic display.

15. detecting the presence of a person in an environment of the client device based on a signal from a presence detection sensor; causing the one or more cameras to provide the stream of image frames in response to detecting the presence of the person in the environment; 15. The method of any one of claims 1 to 14, further comprising:

16. At least one visual component; At least one microphone; one or more processors; a memory operatively coupled to the one or more processors, the memory providing to the one or more processors in response to execution of instructions by the one or more processors: receiving a stream of visual data based on output from the visual component of a client device; using at least one trained machine learning model stored locally on the client device; a calling gesture of a user captured by said visual data; and the user's gaze being directed at the client device; processing the visual data to monitor the occurrence of both Based on the monitoring, the calling gesture; The gaze and and detecting the occurrence of both in response to detecting the occurrence of both the calling gesture and the gaze, by the client device to one or more remote automation assistant components, additional visual data based on output from the vision component; and Audio data based on the output from the microphone of the client device. and a memory storing the instructions for performing operations comprising: A client device comprising:

17. The operation, receiving a response content in response to said transmitting; rendering the response content via one or more user interface output devices of the client device; The client device of claim 16 further comprising:

18. 1. A system comprising: At least one visual component; one or more processors receiving a stream of visual data based on output from the vision component, one or more of the processors: using at least one trained machine learning model stored locally on the client device; a calling gesture of a user captured by said visual data; and the user's gaze being directed at the client device; processing the visual data to monitor the occurrence of both Based on the monitoring, the calling gesture; The gaze and and detecting the occurrence of both in response to detecting the occurrence of both the calling gesture and the gaze, Activating at least one feature of the Automation Assistant; one or more processors configured to A system comprising:

19. 16. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15.

20. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 15.

21. 16. A system comprising one or more processors for carrying out the method of any one of claims 1 to 15.

Images