Automated assistant function invoking based on detected gestures and gaze

Abstract

The invention relates to invoking automated assistant functionality based on detected gestures and gaze. Based on processing of visual data from the one or more visual components, one or more previously dormant functions of the automated assistant are invoked in response to detecting: (1) detecting a particular gesture of the user (e.g., a particular gesture of the one or more "invoking gestures"); and / or (2) detecting that the user's gaze is directed towards an assistant device for providing an automated assistant interface (graphical and / or audible) of the automated assistant. For example, a previously dormant function can be invoked in response to detecting a particular gesture, detecting that a user's gaze is directed towards an assistant device for at least a threshold amount of time, and optionally, the particular gesture and the directional gaze of the user occur concurrently or within a threshold temporal proximity to each other.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880094289.4, filed on May 4, 2018. Background Art

[0003] A human may engage in a human-computer dialogue with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed. The automated assistant responds to the request by providing responsive user interface output, which can include audible and / or visual user interface output.

[0004] As mentioned above, many automated assistants are configured to be interacted with via spoken utterances. In order to protect user privacy and / or conserve resources, the user must often explicitly invoke the automated assistant before the automated assistant will fully process the spoken utterance. The explicit invocation of the automated assistant typically occurs in response to receiving certain user interface inputs at a client device. The client device includes an assistant interface that provides an interface to a user of the client device for interfacing with the automated assistant (e.g., receiving inputs from the user and providing audible responses and / or graphical responses) and that interfaces with one or more additional components that implement the automated assistant (e.g., a remote server device for processing user inputs and generating appropriate responses).

[0005] Some user interface inputs that can invoke an automated assistant via a client device include a hardware button and / or a virtual button at the client device for invoking the automated assistant (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device). Many automated assistants can additionally or alternatively be invoked in response to one or more verbal invocation phrases, which are also referred to as "hot words / phrases" or "trigger words / phrases." For example, a verbal invocation phrase such as "Hey Assistant," "OK Assistant," and / or "Assistant" can be spoken to invoke an automated assistant. Summary of the invention

[0006] As mentioned above, many client devices for facilitating interaction with automated assistants—also referred to herein as "assistant devices"—enable users to engage in contactless interaction with the automated assistant. For example, assistant devices often include microphones that allow users to provide verbal utterances to invoke and / or otherwise interact with the automated assistant. The assistant devices described herein can additionally or alternatively incorporate and / or communicate with one or more visual components (e.g., cameras, light detection and ranging (LIDAR) components, radar components, etc.) to facilitate contactless interaction with the automated assistant.

[0007] Embodiments disclosed herein relate to invoking one or more previously dormant functions of an automated assistant in response to the following detections, based on processing of visual data from one or more visual components: (1) detecting a particular gesture of a user (e.g., a particular gesture in one or more "invoke gestures"); and / or (2) detecting a gaze of a user directed toward an assistant device that provides an automated assistant interface (graphical and / or audible) of the automated assistant. For example, a previously dormant function can be invoked in response to detecting a particular gesture, detecting that the user's gaze is directed toward the assistant device for at least a threshold amount of time, and optionally the user's particular gesture and directional gaze occurring simultaneously or within a threshold time proximity of each other (e.g., within 0.5 seconds, within 1.0 seconds, or within other threshold time proximity). For example, a previously dormant function can be invoked in response to detecting a particular gesture (e.g., a "thumbs-up" gesture and / or a "wave" gesture) and in response to detecting a directional gaze having a duration of at least 0.5 seconds and occurring simultaneously with the particular gesture or within 1.0 seconds of the particular gesture.

[0008] In some versions of those embodiments, one or more previously dormant functions can be called in response to the detection of the gesture and directional gaze alone. For example, in response to the detection of the gesture and directional gaze alone, certain sensor data generated by one or more sensor components can be transmitted by the client device to one or more remote automated assistant components (however, sensor data from the sensor components are not transmitted before the gesture and directional gaze are detected). The certain sensor data can include, for example, visual and / or audio data captured after the gesture and directional gaze are detected and / or buffered visual and / or audio data captured during the execution of the gesture and / or during the directional gaze. In this way, data is transmitted to the remote computing component through the data network selectively and at the appropriate time. This provides efficient use of network capacity and the computing and other hardware resources involved in sending and receiving data through the network. Improvements in the efficiency of the use of computing resources and data networks on the remote system can lead to significant savings in terms of the use of power by the transmitters and receivers in the network and in terms of memory operations and processor use at the remote system. The corresponding effect can also be experienced at the client device. These effects, particularly over time and with ongoing operation of the automated assistant, allow significant additional capacity to be experienced in the network and throughout the computing device (including the devices and systems running the assistant). This additional capacity can be used for further communications in the data network (whether assistant-related or not, without having to expand network capabilities, e.g., through additional or updated infrastructure), as well as for additional computing operations in the computing device. Other technical improvements will become apparent from the following discussion.

[0009] In some other versions of those embodiments, one or more previously dormant functions can be called in response to detecting a gesture and directional gaze and detecting the occurrence of one or more other conditions. The occurrence of the one or more other conditions can include, for example: based on audio data, detecting voice activity close in time to the detected gesture and directional gaze (e.g., any voice activity, voice activity of the user providing the gesture and directional gaze, voice activity of the authorized user, voice activity including a verbal call phrase); based on visual data, detecting the user's mouth movement that occurs simultaneously with or close in time to the detected gesture and directional gaze; based on audio data and / or visual data, detecting that the user is an authorized user; and / or detecting other conditions. For example, the client device can transmit the buffered audio data to one or more remote automated assistant components in response to detecting the gesture and directional gaze and in response to detecting voice activity in at least a portion of the buffered audio data (e.g., by using a voice activity detector (VAD) module). This has advantages corresponding to the advantages described above.

[0010] In some embodiments disclosed herein, previously dormant functionality of an automated assistant that is invoked in response to detection of gestures and directional gaze can include some processing of certain sensor data (e.g., audio data, video, images, etc.) and / or rendering of certain content (e.g., graphically and / or audibly). For example, before invocation based on detection of gestures and directional gaze, the automated assistant can perform only limited processing (or no processing) on ​​certain sensor data such as audio data, video / image data, etc. For example, before invocation, the automated assistant can process some sensor data locally while monitoring explicit invocations, but will "discard" the data after local processing and without having the data processed by one or more additional components (e.g., a remote server device for processing user input and generating appropriate responses) that implement the automated assistant. However, in response to invocation, such data can be processed by additional components. In these and other ways, processing and / or network resources can be reduced by transmitting certain sensor data and / or performing certain processing on the certain sensor data only in response to invocations.

[0011] In addition, for example, before an explicit call, the automated assistant can render (e.g., graphically) only limited content (or no content). However, in response to the call, the automated assistant can render other content, such as content customized for the user who called the automated assistant. For example, no content may be graphically rendered via a display screen controlled by the automated assistant before the explicit call, or only limited lower power consumption content may be rendered (e.g., only the current time is rendered on a small portion of the display screen). However, in response to the call, the automated assistant can additionally and optionally cause higher power consumption content to be graphically rendered via the display screen and / or audibly rendered via a speaker, such as a weather forecast, a daily event summary, and / or other content that may be displayed more brightly on the display screen and / or occupy a larger portion of the display screen. In these and other ways, power consumption can be reduced by displaying lower power consumption content (or no content) before the call and displaying higher power consumption content only in response to the call.

[0012] In some embodiments, a trained machine learning model (e.g., a neural network model) stored locally on the client device is utilized by the client device to at least selectively process at least a portion of visual data from a visual component of the client device (e.g., image frames from a camera of the client device) while monitoring for a particular gesture and while monitoring for a gaze directed toward the client device. For example, in response to detecting the presence of one or more users, the client device can utilize the locally stored machine learning model to process at least a portion of the visual data while monitoring for a particular gesture and directed gaze for at least a duration (e.g., for at least a threshold duration and / or until presence is no longer detected). The client device can detect the presence of one or more users using a dedicated presence sensor (e.g., a passive infrared sensor (PIR)), using visual data and an additional machine learning model (e.g., an additional machine learning model trained only for human presence detection), and / or using audio data and an additional machine learning model (e.g., a VAD using a VAD machine learning model). In embodiments where processing of visual data while monitoring for a particular gesture is dependent on first detecting the presence of one or more users, power resources can be conserved by discontinuously processing visual data while monitoring for gestures and / or directed gaze. In contrast, in those embodiments, processing of visual data while monitoring gestures and / or directional gaze can occur solely in response to detecting the presence of one or more users in the environment of the assistant device via one or more lower power techniques.

[0013] In some embodiments where local machine learning models are utilized when monitoring for specific gestures and directional gaze, at least one gesture detection machine learning model is utilized when monitoring for gestures, and an additional gaze detection machine learning model is utilized when monitoring for gaze. In some versions of those embodiments, one or more "upstream" models (e.g., object detection and classification models) can be utilized to detect portions of visual data (e.g., an image) that are likely to be faces, likely to be eyes, likely to be arms / body, etc., and those portions processed using corresponding machine learning models. For example, the face and / or eye portions of an image can be detected using an upstream model and processed using a gaze detection machine learning model. Additionally, for example, the arms and / or body portions of an image can be detected using an upstream model and processed using the gesture detection machine learning model. As yet another example, the human portion of an image can be detected using an upstream model and processed using both the gaze detection machine learning model and the gesture detection machine learning model.

[0014] Optionally, the gaze detection machine learning model can process higher resolution visual data (e.g., images) than the gesture detection machine learning model. This can enable utilization of the gesture detection machine learning model to be more efficient by processing lower resolution images. Additionally, optionally, the gaze detection machine learning model can be utilized to process portions of an image only after the gesture detection machine learning model has been used to detect a probable gesture (or vice versa). This can also result in computational efficiencies by not utilizing both models to process image data continuously.

[0015] In some embodiments, face matching, eye matching, voice matching, and / or other techniques can be used to identify a specific user profile associated with the gesture and / or directional gaze, as well as content rendered by an automated assistant application of a client device that is customized for the specific user profile. The rendering of the customized content can be one of the functions of the automated assistant that is invoked in response to detecting the gesture and directional gaze. Optionally, the identification of a specific user profile occurs only after the directional gaze and gesture have been detected. In some embodiments, and as mentioned above, one or more additional conditions may also be required for the invocation detection to occur—wherein the additional conditions are in addition to the gaze and / or gesture detection. For example, in some embodiments the additional conditions can include recognizing that the user providing the gesture and directional gaze is associated with an authorized user profile for the client device (e.g., by using face matching, voice matching, and / or other techniques).

[0016] In some embodiments, it is possible to filter out / ignore certain parts of the video / image or weight the parts less heavily when gestures and / or gazes are detected. For example, it is possible to ignore the television captured in the video / image to prevent false detection as a result of a person (e.g., a weatherman) rendered by the television. For example, in response to a plurality of frames throughout a portion of the image detecting a certain display frequency in the portion (i.e., it matches the television refresh rate), etc., it is possible to determine the portion as corresponding to the television based on another object detection / classification machine learning model. Such a portion can be ignored in the gesture and / or directional gaze detection technology described herein to prevent gestures and / or directional gazes from being detected from a television or other video display device. As another example, the picture frame can be ignored. These and other technologies can mitigate false positive calls of automated assistants, which can save various computing and / or network resources that would otherwise be consumed in false positive calls. In addition, in various embodiments, once the location of the television, picture frame, etc. is detected, it can be optionally ignored over multiple frames (e.g., while intermittently verifying, until the movement of the client device or object is detected, etc.). This can also save various computing resources.

[0017] The above description is provided as an overview of various embodiments disclosed herein. These various embodiments as well as additional embodiments are described in more detail herein.

[0018] In some embodiments, a method is performed by one or more processors of a client device, the method facilitating contactless interaction between one or more users and an automated assistant. The method includes receiving a stream of image frames based on the output of one or more cameras of the client device. The method also includes processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both: a user's invocation gesture captured by at least one of the image frames, and a user's gaze directed at the client device. The method also includes detecting the occurrence of both: an invocation gesture, and a gaze based on monitoring. The method also includes, in response to detecting the occurrence of both the invocation gesture and the gaze: causing at least one function of the automated assistant to be activated.

[0019] These and other implementations of the technology can include one or more of the following features.

[0020] In some implementations, at least one function of the automated assistant that is activated in response to detecting the occurrence of both the invocation gesture and the gaze includes transmitting audio data captured via one or more microphones of the client device to a remote server associated with the automated assistant.

[0021] In some embodiments, at least one functionality activated in response to detecting the occurrence of both an invocation gesture and a gaze additionally or alternatively includes transmitting additional image frames to a remote server associated with the automated assistant, the additional image frames being based on output from one or more of the cameras and being received after detecting the occurrence of both the invocation gesture and the gaze.

[0022] In some embodiments, at least one function activated in response to detecting the occurrence of both an invocation gesture and a gaze additionally or alternatively includes: processing buffered audio data at a client device, wherein the buffered audio data is stored in a memory at the client device and is captured via one or more microphones of the client device, and wherein processing the buffered audio data includes one or both of the following: invocation phrase detection processing and automatic speech recognition. In some versions of those embodiments, processing the buffered audio data includes automatic speech recognition, and the automatic speech recognition includes speech-to-text processing. In some additional or alternative versions of those embodiments, processing the buffered audio data includes invocation phrase detection processing, and the method further includes: in response to the invocation phrase detection processing detecting the presence of an invocation phrase in the buffered audio data, performing one or both of the following: transmitting additional audio data captured via one or more microphones of the client device to a remote server associated with the automated assistant; and transmitting additional image frames to a remote server associated with the automated assistant, wherein the additional image frames are based on output from one or more of the cameras and are received after detecting the occurrence of both an invocation gesture and a gaze.

[0023] In some embodiments, processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both the invocation gesture and the gaze includes: monitoring the occurrence of the invocation gesture using a first trained machine learning model; and monitoring the user's gaze directed toward the client device using a second trained machine learning model. In some versions of those embodiments, monitoring the user's gaze directed toward the client device using the second trained machine learning model occurs only in response to detecting the occurrence of the invocation gesture using the first trained machine learning model. In some of those versions, or in other versions, monitoring the occurrence of the invocation gesture using the first trained machine learning model includes processing a first resolution version of the image frame using the first machine learning model; and monitoring the user's gaze using the second trained machine learning model includes processing a second resolution version of the image frame using the second machine learning model.

[0024] In some embodiments, the method further includes: receiving a stream of audio data frames based on output from one or more microphones of the client device; processing the audio data frames of the stream using at least one trained call phrase detection machine learning model stored locally on the client device to monitor for occurrences of spoken call phrases; and detecting the occurrence of spoken call phrases based on monitoring for the occurrences of spoken call phrases. In some of those embodiments, causing at least one function of the automated assistant to be activated is in response to detecting the occurrence of the spoken call phrase in temporal proximity to both the call gesture and the gaze. In some versions of those embodiments, at least one function that is activated includes one or both of the following: transmitting additional audio data frames captured via one or more microphones of the client device to a remote server associated with the automated assistant; and transmitting one or more additional image frames from one or more of the cameras to a remote server associated with the automated assistant.

[0025] In some embodiments, processing image frames of the stream using at least one trained machine learning model stored locally on a client device to monitor the occurrence of both the invocation gesture and the gaze includes: processing the image frames using a first trained machine learning model to predict an area of ​​the image frames that includes a face; and processing the area of ​​the image frames using a second trained machine learning model that is trained to detect the user's gaze.

[0026] In some embodiments, processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor for occurrences of both the invocation gesture and the gaze includes: determining that a region of the image frame corresponds to an electronic display; and in response to determining that the region corresponds to the electronic display, disregarding the region when monitoring for occurrences of both the invocation gesture and the gaze. In some of those embodiments, determining that the region of the image frame corresponds to the electronic display is based on detecting a display frequency in the region of the image frame that corresponds to a display frequency of the electronic display.

[0027] In some implementations, the method further includes detecting human presence in the environment of the client device based on a signal from a presence sensor; and causing one or more cameras to provide a stream of image frames in response to detecting human presence in the environment.

[0028] In some embodiments, a client device is provided, the client device comprising at least one visual component, at least one microphone, one or more processors, and a memory operably coupled to the one or more processors. The memory stores instructions, which, in response to the execution of the instructions by one or more processors, cause one or more of the processors to perform the following operations: receiving a stream of visual data based on the output of the visual component from the client device; using at least one trained machine learning model stored locally on the client device to process the visual data to monitor the occurrence of the following two: a user's call gesture captured by the visual data, and the user's gaze directed to the client device; based on the monitoring, detecting the occurrence of the following two: a call gesture, and gaze; and in response to detecting the occurrence of both the call gesture and gaze: transmitting one or two of the following items by the client device to one or more remote automated assistant components: additional visual data based on the output from the visual component, and audio data based on the output of the microphone from the client device. These operations can optionally also include: receiving response content in response to the transmission; and rendering the response content via one or more user interface output devices of the client device.

[0029] In some embodiments, a system is provided, the system comprising at least one visual component and one or more processors that receive a stream of visual data based on an output from the visual component. One or more of the processors are configured to: process the visual data using at least one trained machine learning model stored locally on a client device to monitor the occurrence of: a user's invocation gesture captured by the visual data, and the user's gaze directed at the client device; based on the monitoring, detect the occurrence of: an invocation gesture, and gaze; and in response to detecting the occurrence of both the invocation gesture and gaze: cause at least one function of an automated assistant to be activated.

[0030] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the foregoing methods.

[0031] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0033] Figure 2A , Figure 2B and Figure 2C Example process flows demonstrating various aspects of the present disclosure are depicted, according to various implementations.

[0034] Figure 3 An example of an assistant device and a user providing gestures and directional gaze is depicted, and images captured by a camera of the assistant device while the user is providing the gestures and directional gaze are also depicted.

[0035] Figure 4A Depicted is a flow chart illustrating an example method according to implementations disclosed herein.

[0036] Figure 4B Describing the diagram Figure 4A A flowchart of a specific example of certain blocks of an example method.

[0037] Figure 5 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0038] Figure 1 An example environment in which the techniques disclosed herein may be implemented is illustrated. The example environment includes one or more client computing devices 106. Each client device 106 may execute a respective instance of an automated assistant client 110. One or more cloud-based automated assistant components 130 can be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the client devices 106 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 114. The cloud-based automated assistant components 130 can be implemented, for example, via a cluster of high-performance servers.

[0039] In various embodiments, an instance of automated assistant client 110, through its interaction with one or more cloud-based automated assistant components 130, may form a logical instance of automated assistant 120 that appears to the user as a user with whom the user can engage in human-computer interaction (e.g., verbal interaction, gesture-based interaction, and / or touch-based interaction). Such an instance of automated assistant 120 is Figure 1 106. It will be appreciated, therefore, that each user engaging with an automated assistant client 110 executing on a client device 106 may actually engage with his or her own logical instance of an automated assistant 120. For the sake of brevity and simplicity, the term "automated assistant" as used herein to "serve" a particular user will refer to the combination of an automated assistant client 110 executing on a client device 106 operated by a user and optionally one or more cloud-based automated assistant components 130 (which may be shared among multiple automated assistant clients 110). It will also be appreciated that in some embodiments, an automated assistant 120 may respond to requests from any user, regardless of whether the user is actually "served" by that particular instance of an automated assistant 120.

[0040] The one or more client devices 106 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which may include a visual sensor in some cases), a smart appliance such as a smart TV (or a standard TV equipped with a networked dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. As previously noted, some client devices 106 may take the form of assistant devices that are primarily designed to facilitate interaction between a user and an automated assistant 120 (e.g., a stand-alone interactive device with a speaker and a display).

[0041] The client device 106 can be equipped with one or more visual components 107 (e.g., visual sensors) having one or more fields of view. The visual component 107 can take various forms, such as a thematic camera, a stereo camera, a LIDAR component, a radar component, etc. The one or more visual components 107 can be used, for example, by a visual capture module 114 to capture visual frames (e.g., image frames (still images or videos)) of the environment in which the client device 106 is deployed. These visual frames can then be at least selectively analyzed, for example, by a gaze and gesture module 116 of a call engine 115 to monitor the occurrence of the following: a specific gesture (of one or more candidate gestures) of the user captured by the visual frame and / or a directional gaze from the user (i.e., a gaze directed toward the client device 106). The gaze and gesture module 116 can utilize one or more trained machine learning models 117 when monitoring the occurrence of specific gestures and / or directional gaze.

[0042] In response to detecting the particular gesture and directional gaze (and optionally in response to detecting one or more other conditions by other condition module 118), invocation engine 115 can invoke one or more previously dormant functions of automated assistant 120. Such dormant functions can include, for example, processing of certain sensor data (e.g., audio data, video, images, etc.) and / or rendering of certain content (e.g., graphically and / or audibly).

[0043] As a non-limiting example, prior to detecting the particular gesture and directional gaze, visual data and / or audio data captured at the client device 106 can be processed only locally and / or temporarily buffered at the client device 106 (i.e., without being transmitted to the cloud-based automated assistant component 130). However, in response to detecting the particular gesture and directional gaze, the audio data and / or visual data (e.g., most recently buffered data and / or data received after detection) can be transmitted to the cloud-based automated assistant component 130 for further processing. For example, detection of the particular gesture and directional gaze can eliminate the need for the user to speak an explicit invocation phrase (e.g., "OK Assistant") in order for the user's spoken utterance to be fully processed by the automated assistant 120, and for the automated assistant 120 to generate response content and render it to the user.

[0044] For example, rather than the user needing to speak “OK Assistant, What's today's forecast” to obtain today's forecast, the user can instead: perform a particular gesture, look at the client device 106, and speak “What's today's forecast” only during or in temporal proximity to (e.g., within a time threshold before and / or after) performing the gesture and / or looking at the client device 106. Data corresponding to the spoken utterance “What's today's forecast” (e.g., audio data capturing the spoken utterance or a textual or semantic transformation thereof) can be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and directional gaze and in response to receiving the spoken utterance during and / or in temporal proximity to the gesture and directional gaze. In another example, rather than the user needing to speak “OK Assistant, turn up the heat” to increase the temperature of his / her home via a connected thermostat, the user can instead: perform a particular gesture, look at the client device 106, and speak “turn up the heat” only during or temporally proximate to (e.g., within a time threshold before and / or after) performing the gesture and / or looking at the client device 106. Data corresponding to the spoken utterance “turn up the heat” (e.g., audio data capturing the spoken utterance or a textual or other semantic transformation thereof) can be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and directional gaze and in response to receiving the spoken utterance during and / or temporally proximate to the gesture and directional gaze. In another example, rather than the user needing to say “OK Assistant, open the garage door” to open his / her garage, the user can instead: perform a particular gesture, look at the client device 106, and say “open the garage door” only during or in temporal proximity to (e.g., within a time threshold before and / or after) performing the gesture and / or looking at the client device 106.Data corresponding to the spoken utterance "open the garage door" (e.g., audio data capturing the spoken utterance or text or other semantic data thereof) can be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the gesture and directional gaze and in response to receiving the spoken utterance during and / or in temporal proximity to the gesture and directional gaze. In some embodiments, the transmission of the data by the client device 106 may be further dependent on the occurrence of one or more additional conditions determined by the other condition module 118. For example, the transmission of the data can be further based on local voice activity detection processing of the audio data performed by the other condition module 118 to indicate the presence of voice activity in the audio data. Also, for example, the transmission of the data can be additionally or alternatively further based on a determination by the other condition module 118 that the audio data corresponds to a user who provided the gesture and directional gaze. For example, the direction of the user (relative to the client device 106) can be determined based on visual data, and the transmission of the data can be further based on a determination by the other condition module 118 that the spoken utterance in the audio data comes from the same direction (e.g., using beamforming and / or other techniques). Additionally, for example, a user profile for the user can be determined based on visual data (e.g., using facial recognition), and the transmission of the data can be further based on a determination by the other condition module 118 that spoken utterances in the audio data have voice characteristics that match the user profile. As yet another example, the transmission of the data can additionally or alternatively be further based on a determination by the other condition module 118 based on visual data that a mouth movement of the user occurs simultaneously with a detected gesture and / or directional gaze of the user, or occurs for a threshold amount of time of the detected gesture and / or directional gaze. The other condition module 118 can optionally utilize one or more other machine learning models 119 when determining that other conditions exist. Additional descriptions of implementations of the gaze and gesture module 116 and other condition modules 118 are provided herein (e.g., references). Figures 2A-2C ).

[0045] Each of the client computing device 106 and the computing device for operating the cloud-based automated assistant component 130 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components for facilitating communication over a network. The operations performed by the client computing device 106 and / or by the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented as a computer program running on one or more computers coupled to each other over a network in one or more locations, for example.

[0046] As noted above, in various embodiments, the client computing device 106 can operate an automated assistant client 110. In some of those various embodiments, the automated assistant client 110 can include a voice capture module 112, the aforementioned visual capture module 114, and an invocation engine 115, which can include the gaze and gesture module 116 and optionally the other conditions module 118. In other embodiments, one or more aspects of the voice capture module 112, the visual capture module 114, and / or the invocation engine 115 can be implemented separately from the automated assistant client 110, for example, by one or more cloud-based automated assistant components 130.

[0047] In various embodiments, the voice capture module 112, which can be implemented using any combination of hardware and software, can interface with hardware such as a microphone 109 or other pressure sensor to capture an audio recording of the user's spoken words. As will be described below, various types of processing can be performed on this audio recording for various purposes. In various embodiments, the visual capture module 114, which can be implemented using any combination of hardware or software, can be configured to interface with the visual component 107 to capture one or more visual frames (e.g., digital images) corresponding to the optionally adaptive field of view of the visual sensor.

[0048] As previously mentioned, the voice capture module 112 can be configured to capture the user's voice, for example, via the microphone 109. Additionally or alternatively, in some embodiments, the voice capture module 112 can be further configured to convert the captured audio to text and / or other representations or embeddings, for example, using speech-to-text ("STT") processing techniques. However, because the client device 106 may be relatively constrained in computing resources (e.g., processor cycles, memory, battery, etc.), the voice capture module 112 local to the client device 106 can be configured to convert a limited number of different spoken phrases—such as phrases that invoke the automated assistant 120—to text (or other forms, such as lower dimensional embeddings). Other voice inputs can be sent to a cloud-based automated assistant component 130, which can include a cloud-based STT module 132.

[0049] The cloud-based TTS module 131 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some embodiments, the TTS module 131 can provide the computer-generated speech output to the client device 106 for direct output, for example, using one or more speakers. In other embodiments, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the client device 106, and the local TTS module of the client device 106 can then convert the text data into computer-generated speech that is output locally.

[0050] The cloud-based STT module 132 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the speech capture module 112 into text, which can then be provided to the intent understanding module 135. In some embodiments, the cloud-based STT module 132 can convert the audio recording of the speech into one or more phonemes, and then convert the one or more phonemes into text. Additionally or alternatively, in some embodiments, the STT module 132 can employ a state decoding graph. In some embodiments, the STT module 132 can generate multiple candidate text interpretations of the user's utterance and utilize one or more techniques to select a given interpretation from the candidates.

[0051] Automated assistant 120 (and in particular cloud-based automated assistant component 130) may include intent understanding module 135, the aforementioned TTS module 131, the aforementioned STT module 132, and other components described in greater detail herein. In some implementations, modules of automated assistant 120 and / or one or more of the modules may be omitted, combined, and / or implemented in components separate from automated assistant 120. In some implementations, one or more of the components of automated assistant 120, such as intent understanding module 135, TTS module 131, STT module 132, and the like, may be implemented at least in part on client device 106 (e.g., in conjunction with or excluding a cloud-based implementation).

[0052] In some embodiments, the automated assistant 120 generates various content to be audibly and / or graphically rendered to the user via the client device 106. For example, the automated assistant 120 can generate content such as a weather forecast, a daily schedule, etc., and can render the content in response to detecting a gesture and / or directional gaze from the user as described herein. In addition, for example, the automated assistant 120 can generate content in response to a user's free-form natural language input provided via the client device 106, in response to a user's gesture detected via visual data from the visual component 107 of the client device, and the like. As used herein, a free-form input is an input formulated by a user and is not limited to a set of options presented for selection by the user. The free-form input can be, for example, typed input and / or verbal input.

[0053] Natural language processor 133 of intent understanding module 135 processes natural language input generated by a user via client device 106 and may generate annotated output (e.g., in text form) for use by one or more other components of automated assistant 120. For example, natural language processor 133 may process natural language free-form input generated by a user via one or more user interface input devices of client device 106. The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0054] In some embodiments, the natural language processor 133 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the natural language processor 133 may include a lexical module that can separate individual words into morphemes and / or annotate the morphemes, for example, by their class. The natural language processor 133 may also include a portion of a part-of-speech tagger that is configured to annotate terms by their grammatical roles. In addition, for example, in some embodiments the natural language processor 133 may additionally and / or optionally include a dependency parser (not shown) that is configured to determine syntactic relationships between terms in the natural language input.

[0055] In some embodiments, the natural language processor 133 may additionally and / or optionally include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as in a knowledge graph (not shown), and the entity tagger of the natural language processor 133 can leverage such databases in entity tagging.

[0056] In some implementations, the natural language processor 133 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more context clues. For example, a coreference resolver may be utilized to resolve the term "there" in "I liked Hypothetical Café last time we ate there" to "Hypothetical Café".

[0057] In some implementations, one or more components of the natural language processor 133 can rely on annotations from one or more other components of the natural language processor 133. For example, in some implementations, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. Additionally, for example, in some implementations, a coreference resolver can rely on annotations from a dependency resolver when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 133 can use relevant prior inputs and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0058] Intent understanding module 135 may also include an intent matcher 134 configured to determine the intent of the user engaging in the interaction with automated assistant 120. Figure 1 134 , but in other embodiments, the intent matcher 134 may be an integral part of the natural language processor 133 (or more generally, a pipeline including the natural language processor 133). In some embodiments, the natural language processor 133 and the intent matcher 134 may collectively form the aforementioned intent understanding module 135.

[0059] Intent matcher 134 may use a variety of techniques to determine the user's intent, such as based on output from natural language processor 133 (which may include annotations and terms of natural language input), based on user touch input at a touch-sensitive display of client device 106, and / or based on gestures and / or other visual cues detected in visual data. In some embodiments, intent matcher 134 may be able to access one or more databases (not shown) that include, for example, multiple mappings between grammars and response actions (or more generally, intents), visual cues and response actions, and / or between touch inputs and response actions. For example, the grammars included in the mappings may be selected and / or learned over time and may represent common intents of users. For example, a grammar "play<artist>[play <artist>]" may be mapped to an intent that invokes a response action that causes music by <artist> to be played on a client device 106 operated by the user. Another syntax, "[weather|forecast] today," may be able to match user queries such as "what's the weather today" and "what's the forecast for today?" As another example, mappings of visual cues to actions can include "general" mappings that apply to multiple users (e.g., all users) and / or user-specific mappings. Some examples of mappings of visual cues to actions include mappings for gestures. For example, a "wave" can be mapped to an action that causes customized content (customized for the user providing the gesture) to be rendered to the user, a "thumbs-up" gesture can be mapped to an action of "play music"; and a "high five" gesture can be mapped to a "routine" for an automated assistant action to perform, such as turning on a smart coffee maker, turning on certain smart lights, and audibly rendering a news digest.

[0060] In addition to or in lieu of a grammar, in some embodiments, intent matcher 134 can employ one or more trained machine learning models, alone or in combination with one or more grammars, visual cues, and / or touch inputs. These trained machine learning models can also be stored in one or more databases and can be trained to recognize intent, for example, by embedding data representing user utterances and / or any detected user-provided visual cues into a reduced dimensionality space and then determining which other embeddings (and therefore, intents) are closest, for example, using techniques such as Euclidean distance, cosine similarity, etc.

[0061] As above "play <artist>As you can see in the example grammars, some grammars have slots that can be filled with slot values ​​(or "parameters") (for example, <artist>). Slot values ​​can be determined in a variety of ways. Users often actively provide slot values. For example, for the syntax "Order me a <topping>pizza", the user might very well say the phrase "Order me a sausage pizza", in which case the slot <topping>Additionally or alternatively, if a user invokes a syntax that includes slots to be filled with slot values, without the user actively providing the slot values, automated assistant 120 can solicit those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?" In some implementations, slots can be filled with slot values ​​based on visual cues that are detected based on visual data captured by vision component 107. For example, a user can say something like "Order me this many cat bowls" while holding up three fingers to a visual sensor of client device 106. Or, a user can say something like "Find me more movies like this" while holding up a particular movie DVD case.

[0062] In some embodiments, the automated assistant 120 can facilitate (or "mediate") transactions between a user and an agent, which can be an independent software process that receives input and provides responsive output. Some agents can take the form of third-party applications that may or may not operate on a computing system separate from the computing system that operates, for example, the cloud-based automated assistant component 130. One user intent that can be identified by the intent matcher 134 is to engage a third-party application. For example, the automated assistant 120 can provide access to an application programming interface ("API") to a pizza delivery service. A user can invoke the automated assistant 120 and provide a command such as "I'd like to order a pizza". The intent matcher 134 can map this command to a grammar that triggers the automated assistant 120 to engage with a third-party pizza delivery service. The third-party pizza delivery service can provide the automated assistant 120 with a minimum list of slots that need to be filled in order to complete a pizza delivery order. The automated assistant 120 can generate and provide (via the client device 106) to the user a natural language output that solicits parameters for the slot.

[0063] The fulfillment module 138 can be configured to receive the predicted / estimated intent and associated slot value (whether actively provided by the user or solicited from the user) output by the intent matcher 134 and fulfill (or "resolve") the intent. In various embodiments, the fulfillment (or "resolve") of the user's intent can cause various fulfillment information (also referred to as "response" information or data) to be generated / obtained, for example, by the fulfillment module 138.

[0064] Because intent can be fulfilled in a variety of ways, fulfillment information can take a variety of forms. Assume that a user requests pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" The user's intent can be determined as a search query, for example, by intent matcher 134. The intent and content of the search query can be provided to fulfillment module 138, which can be used as a search query. Figure 1 As depicted in , the search module 138 may communicate with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) to obtain response information. The fulfillment module 138 may provide data indicative of the search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to the search module 150. The search module 150 may provide response information, such as GPS coordinates or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon". This response information may form part of the fulfillment information generated by the fulfillment module 138.

[0065] Additionally or alternatively, fulfillment module 138 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from intent understanding module 135, and trigger a response action. The response action may include, for example, ordering goods / services, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, initiating a routine of multiple actions, etc. In some such embodiments, fulfillment information may include slot values ​​associated with the fulfillment, confirmation responses (which in some cases may be selected from predetermined responses), etc.

[0066] Additionally or alternatively, fulfillment module 138 can be configured to infer the user's intent (e.g., based on time of day, past interactions, etc.) and obtain responsive information to those intents. For example, fulfillment module 138 can be configured to obtain a daily schedule summary for the user, a weather forecast for the user, and / or other content for the user. Fulfillment module 138 can further cause such content to be "pushed" for graphical and / or audible rendering to the user. For example, rendering of such content can be a dormant functionality invoked in response to invocation engine 115 detecting the occurrence of a particular gesture and directional gaze.

[0067] The natural language generator 136 can be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 136 can be configured to receive as input fulfillment information associated with the fulfillment of an intent, and to generate natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 136 can receive information from other sources, such as third-party applications, that it can use to compose natural language output for the user.

[0068] Reference now Figure 2A , Figure 2B and Figure 2C , depicts various examples of how gaze and gesture module 116 can detect particular gestures and / or directional gaze and how invocation engine 115 can invoke one or more previously dormant automated assistant functions in response.

[0069] initial turn Figure 2A , the visual capture module 114 provides visual frames to the gaze and gesture module 116. In some embodiments, the visual capture module 114 provides a real-time stream of visual frames to the gaze and gesture module 116. In some of those embodiments, the visual capture module 114 begins to provide visual frames in response to a signal from a separate presence sensor 105 indicating that humans are present in the environment together with the client device 106. For example, the presence sensor 105 can be a PIR sensor and can provide a signal to the visual capture module 114 in response to detecting human presence. Unless human presence is detected, the visual capture module 114 can avoid providing any visual frames to the gaze and gesture module 116. In other embodiments in which the visual capture module 114 selectively provides visual frames only to the gaze and gesture module 116, additional and / or alternative clues can be used to initiate such provision. For example, human presence can be detected based on audio data from the voice capture module 112, based on analysis of visual frames by one or more other components, and / or other signals.

[0070] The gaze and gesture module 116 processes the visual frames using one or more machine learning models 117 to monitor the occurrence of both directional gaze and specific gestures. When both directional gaze and specific gestures are detected, the gaze and gesture module 116 provides an indication to the call engine 115 that both gaze and gestures are detected.

[0071] exist Figure 2A In the process, the visual frame and / or the audio data (provided by the voice capture module 112) are also provided to the other condition module 118. The other condition module 118 optionally uses one or more other machine learning models 119 to process the provided data to monitor the occurrence of one or more other conditions. For example, the other condition can be to detect any voice activity based on the audio data; detect the presence of a spoken call phrase in the audio data; detect voice activity from the direction or position of the user based on the audio data; detect that the user is an authorized user based on the visual frame and / or the audio data; detect the mouth movement of the user (who provides gestures and directional gaze) based on the visual frame; and so on. When other conditions are detected, the other condition module 118 provides an indication of the occurrence of the other condition to the call engine 115.

[0072] When the invocation engine 115 receives indications of directed gaze and gestures, as well as temporally proximate indications of other conditions, the invocation engine 115 causes invocation of the dormant functionality 101. For example, the invocation of the dormant functionality 101 can include one or more of: activating a display screen of the client device 106; causing content to be visually and / or audibly rendered by the client device 106; causing visual frames and / or audio data to be transmitted by the client device 106 to one or more cloud-based automated assistant components 130; and the like.

[0073] In some embodiments, and as with respect to Figure 2B and Figure 2C Described in more detail, the gaze and gesture module 116 can use one or more first machine learning models 117 for detecting directional gaze and one or more second machine learning models 117 for detecting gestures.

[0074] In some other embodiments, the gaze and gesture module 116 can utilize an end-to-end machine learning model that accepts a visual frame (or features thereof) as an input and can be utilized to generate (based on processing of the input throughout the model) an output indicating whether a particular gesture and directional gaze have occurred. Such a machine learning model can be, for example, a neural network model, such as a recurrent neural network (RNN) model including one or more memory layers (e.g., a long short-term memory (LSTM) layer). The training of such an RNN model can be based on training examples that include a sequence of visual frames (e.g., videos) as training example inputs and an indication of whether the sequence as training example outputs includes both gestures and directional gaze. For example, the training example output can be a single value indicating whether both gestures and directional gaze are present. As another example, the training example output can include a first value indicating whether there is a directional gaze and N additional values ​​each indicating whether a corresponding one of N gestures is included (so that the training of the model can predict the corresponding probability of each of the N separate gestures). As yet another example, the training example output can include a first value indicating whether a directional gaze is present and a second value indicating whether any of one or more specific gestures are present (such that training of the model can predict probabilities corresponding to whether any gesture is included).

[0075] In an embodiment where a model is trained to predict a corresponding probability for each of N separate gestures, the gaze and gesture module 116 can optionally provide an indication to the invocation engine 115 of which of the N gestures occurred. In addition, the invocation of the dormant function 101 by the invocation engine 115 may depend on which of the N separate gestures occurred. For example, for a "wave" gesture, the invocation engine 115 can cause certain content to be rendered on a display screen of a client device; for a "thumbs-up" gesture, the invocation engine 115 can cause audio data and / or visual frames to be transmitted to a cloud-based automated assistant component 130; and for a "high five" gesture, the invocation engine 115 can cause a "routine" of automated assistant actions to be executed, such as turning on a smart coffee machine, turning on certain smart lights, and audibly rendering a news summary.

[0076] Figure 2B An example is shown in which the gaze and gesture module 116 includes a gesture module 116A that utilizes a gesture machine learning model 117A when monitoring for the occurrence of gestures and includes a gaze module 116B that utilizes a gaze machine learning model 117B when monitoring for the occurrence of directional gaze. Figure 2B The other condition module 118 is not shown in FIG. 1 , but can also be optionally used as in Figure 2A A similar approach is described for utilizing the other condition module 118 in conjunction with the gesture modules 116A and 117B.

[0077] exist Figure 2B , visual capture module 114 provides a visual frame. A lower resolution version of the visual frame is provided to gesture module 116A, and a higher resolution version of the visual frame is stored in buffer 104. The lower resolution version has a lower resolution relative to the higher resolution version. The higher resolution version may be uncompressed, or less compressed, relative to the lower resolution version. Buffer 104 may be a first-in, first-out buffer and may be capable of temporarily storing the most recent duration of the higher resolution visual frame.

[0078] exist Figure 2B In an example of , gesture module 116A can process lower resolution visual frames while monitoring for the presence of a gesture, and gaze module 116B can be inactive until gesture module 116A detects the occurrence of a gesture. When the occurrence of a gesture is detected, gesture module 116A can provide an indication of the detection of the gesture to gaze module 116B and to call engine 115. Gaze module 116B can be activated in response to receiving an indication that a gesture has been detected, extract a buffered higher resolution visual frame from buffer 104, and utilize the buffered higher resolution visual frame (and optionally additional higher resolution visual frames) in determining whether there is a directional gaze. In this manner, gaze module 116B is only selectively activated, thereby saving computing resources that would otherwise be consumed by additional processing of higher resolution visual frames by gaze module 116B.

[0079] The gesture module 116A can use one or more gesture machine learning models 117A to detect specific gestures. Such a machine learning model can be, for example, a neural network model, such as an RNN model including one or more memory layers. The training of such an RNN model can be based on training examples, which include a sequence of visual frames (e.g., videos) as training example inputs and an indication of whether the sequence includes one or more specific gestures as training example outputs. For example, the training example output can be a single value indicating whether a single specific gesture exists. For example, the single value can be "0" when the single specific gesture does not exist and can be "1" when the single specific gesture exists. In some of those examples, multiple gesture machine learning models 117A are utilized, each of which is customized for a different single specific gesture. As another example, the training example output can include N values, each of which indicates whether a corresponding gesture in N gestures is included (so that the training of the model can predict the corresponding probability of each of N separate gestures). In embodiments where the model is trained to predict the corresponding probability of each of N separate gestures, gesture module 116A can optionally provide an indication of which of the N gestures occurred to invocation engine 115. Additionally, invocation of the dormant function by invocation engine 115 may depend on which of the N separate gestures occurred.

[0080] The gaze module 116B can use one or more gaze machine learning models 117A for detecting directional gaze. Such a machine learning model can be, for example, a neural network model, such as a convolutional neural network (CNN) model. The training of such a CNN model can be based on training examples that include visual frames (e.g., images) as training example inputs and an indication of whether the image includes directional gaze as a training example output. For example, the training example output can be a single value indicating whether there is directional gaze. For example, the single value can be "0" when there is no directional gaze, "1" when there is a gaze directly pointing at the sensor capturing the image or within 5 degrees of the sensor capturing the image, or "0.75" when there is a gaze pointing within 5-10 degrees of the sensor capturing the image, and so on.

[0081] In some of those and / or other embodiments, the gaze module 116B determines directional gaze only when the directional gaze is detected with at least a threshold probability and / or within at least a threshold duration. For example, a stream of image frames can be processed using a CNN model, and processing each frame can result in the corresponding probability that the frame includes directional gaze. The gaze module can determine that there is directional gaze only when at least X% of the sequence of image frames (which corresponds to the threshold duration) has a corresponding probability that meets the threshold. For example, assuming that X% is 60%, the probability threshold is 0.7, and the threshold duration is 0.5 seconds. It is further assumed that 10 image frames correspond to 0.5 seconds. If the image frame is processed to generate a probability of [0.75, 0.85, 0.5, 0.4, 0.9, 0.95, 0.85, 0.89, 0.6, 0.85], directional gaze can be detected because 70% of the frames indicate directional gaze with a probability greater than 0.7. In these and other ways, directional gaze can be detected even when a user briefly shifts his / her gaze direction.Additional and / or alternative machine learning models (e.g., RNN models) and / or techniques can be utilized to detect directional gaze that occurs for at least a threshold duration.

[0082] Figure 2C Another example is shown in which the gaze and gesture module 116 includes a gesture module 116A that utilizes a gesture machine learning model 117A when monitoring for the occurrence of gestures and includes a gaze module 116B that utilizes a gaze machine learning model 117B when monitoring for the occurrence of directional gaze. Figure 2B Other condition modules 118 are not shown in FIG. 1 , but can also be optionally used as in Figure 2A The other condition module 118 is utilized in conjunction with the gesture modules 116A and 117B in a similar manner as described. Figure 2C Not shown Figure 2B buffer 104 and higher resolution / lower resolution visual frames, but can be Figure 2C Similar techniques may be implemented in (e.g., being able to provide a higher resolution portion of the visual frame to gaze module 116B).

[0083] exist Figure 2C , the visual capture module 114 provides the visual frames to the detection and classification module 116C. The detection and classification module 116C uses the object detection and classification machine learning model 117C to classify the various regions of each visual frame. For example, the detection and classification module 116C can classify the human regions (if any) corresponding to humans for each visual frame and provide an indication of such human regions to the gesture module 116A and / or the gaze module 116B for each visual frame. In addition, for example, the detection and classification module 116C can classify the regions (if any) corresponding to the body regions (e.g., arms and torso) of humans for each visual frame and provide an indication of such regions to the gesture module 116A for each visual frame. In addition, for example, the detection and classification module 116C can classify the regions (if any) corresponding to the facial regions of humans for each visual frame and provide an indication of such regions to the gaze module 116B for each visual frame.

[0084] In some embodiments, the gesture module 116A can utilize the provided area to process only the corresponding portion of each visual frame. For example, the gesture module 116A can "crop" the visual frame and resize it to process only those portions that include humans and / or body regions. In some of those embodiments, the gesture machine learning model 117A can be trained based on the "cropped" visual frames, and the resizing can be to a size that conforms to the input size of such a model. In some additional or alternative embodiments, the gesture module 116A can utilize the provided area to skip processing of some visual frames (e.g., visual frames that are indicated as not including humans and / or body regions) altogether. In still other embodiments, the gesture module 116A can utilize the provided area as an attention mechanism (e.g., as an additional attention input to the gesture machine learning model 117A) to focus the processing of each visual frame.

[0085] Likewise, in some embodiments, the gaze module 116B can utilize the provided regions to process only the corresponding portions of each visual frame. For example, the gaze module 116B can "crop" the visual frames and resize them to process only those portions that include human and / or facial regions. In some of those embodiments, the gaze machine learning model 117B can be trained based on the "cropped" visual frames, and the resizing can be to a size that conforms to the input size of such a model. In some additional or alternative embodiments, the gaze module 116B can utilize the provided regions to skip processing of some visual frames (e.g., visual frames that are indicated as not including human and / or facial regions) altogether. In still other embodiments, the gaze module 116B can utilize the provided regions as an attention mechanism (e.g., as an additional attention input to the gaze machine learning model 117B) to focus the processing of each visual frame.

[0086] In some embodiments, the detection and classification module 116C can additionally or alternatively provide other condition modules 118 (for simplicity in Figure 2C 1 (not shown) provides indications of certain regions for use by other condition module 118. For example, when mouth movement is an additional condition for invoking an inactive function, other condition module 118 can use facial regions when detecting mouth movement using a corresponding mouth movement machine learning model.

[0087] In some embodiments, the detection and classification module 116C can additionally or alternatively provide indications of areas classified as TV or other video display sources to the gesture module 116A and gaze module 116B. In some of those embodiments, the modules 116A and 116B can crop those areas from the processed visual frame, focus outside of those areas, and / or otherwise ignore those areas in detection or reduce the chance that detection will be based on such areas. In these and other ways, false positive calls to the sleep function can be mitigated.

[0088] Figure 3 Depiction Figure 1 An example of a client device 106 and a visual component 107. Figure 3 In , the example client device is represented as 106A and also includes a speaker and a display. Figure 3 , an example vision component is represented as 107A and is a camera. Figure 3 Also depicted is user 301, providing a gesture of hand movement (indicated by the “Move” line near the user's right hand) and a gaze directed toward camera 107A.

[0089] Figure 3 Also depicted is an example of an assistant device and a user providing gestures and directional gaze, and also depicts an example image 360 ​​captured by camera 107A while the user is providing gestures and directional gaze. It can be seen that the user and the person behind the user (and therefore in the Figure 3 A television (not visible in the perspective view) is captured in the image 360.

[0090] In the image 360, a bounding box 361 is provided, and the bounding box 361 represents the image that can be determined (for example, by Figure 2C In some implementations, based on the portion being indicated as corresponding to a human, the gesture detection module operating on the client device 106A can process only that portion of the image (or focus on that portion) when monitoring for a particular gesture.

[0091] In the image 360, a bounding box 362 is also provided, and the bounding box 362 represents the image that can be determined (e.g., by Figure 2C In some embodiments, based on the portion being indicated as corresponding to a face, a gaze detection module operating on client device 106A can process only that portion of the image (or focus on that portion) when monitoring for directional gaze. Figure 3 Only a single image is depicted in , but in various embodiments directional gaze detection and / or gesture detection can be based on a sequence of images as described herein.

[0092] In image 360, a bounding box 363 is also provided and represents an area of ​​the image that can be determined to correspond to a video display and that may produce a false positive visual clue. For example, a television may render a video showing one or more people making gestures, looking at a camera, etc., any of which may be misinterpreted as gestures and / or directional gaze occurring. In some embodiments, Figure 2C The detection and classification module 116C can determine such an area (e.g., based on detecting TV classifications), and / or can determine such an area based on an analysis of image 360 ​​and previous images to determine that the area has a display frequency corresponding to the display frequency of the video display (e.g., approximately 60 Hz, 120 Hz, and / or other typical video display frequencies). In some embodiments, the gaze detection module and / or gesture module can crop the area from the processed visual frame, focus attention outside of the area, and / or otherwise ignore the area in detection or reduce the chance that the detection will be based on such an area. In these and other ways, false positive calls to the sleep function can be mitigated.

[0093] Figure 4A is a flow chart illustrating an example method 400 according to implementations disclosed herein. Figure 4B It is a graphic Figure 4A For convenience, reference is made to the system description for performing the operations described. Figure 4A and Figure 4B The operations of the flowchart of method 400 are shown in detail. This system may include various components of various computer systems, such as one or more components of a computing system (e.g., a client device and / or a remote computing system) that implements automated assistant 120. In addition, although the operations of method 400 are shown in a particular order, this is not limiting. One or more operations may be reordered, omitted, or added.

[0094] At box 402, the system receives visual data based on the output from the vision component. In some embodiments, the vision component can be integrated with a client device including an assistant client. In some embodiments, the vision component can be separate from the client device, but communicate with the client device. For example, the vision component can include a standalone smart camera that communicates wired and / or wirelessly with the client device including the assistant client.

[0095] At block 404, the system processes the visual data using at least one machine learning model to monitor for the occurrence of both: gestures and directional gaze.

[0096] At box 406, the system determines whether both a gesture and a gaze have been detected based on the monitoring of box 404. If not, the system proceeds back to box 402, receives additional visual data, and performs another iteration of boxes 404 and 406. In some embodiments, the system determines that both a gesture and a gaze have been detected based on detecting that the gesture and the directional gaze occur simultaneously or occur within a threshold time proximity of each other. In some additional or alternative embodiments, the system determines that both a gesture and a gaze have been detected based on detecting that the gesture has at least a threshold duration (e.g., "wave" for at least X duration or "thumbs up" for at least X duration) and / or the directional gaze has at least a threshold duration (which can be a threshold duration that is the same or different from the threshold duration optionally used for the gesture duration).

[0097] If, upon iteration of block 406 , the system determines that both a gesture and a gaze have been detected based on monitoring of block 404 , the system optionally proceeds to block 408 (or, when block 408 is not included, directly to block 410 ).

[0098] At optional block 408, the system determines whether one or more other conditions are met. If not, the system proceeds back to block 402, receives additional visual data, and performs another iteration of blocks 404, 406, and 408. If so, the system proceeds to block 410. The system can use the visual data, audio data, and / or other sensor or non-sensor data received at block 402 to determine whether one or more other conditions are met. The system can consider a variety of other conditions, such as those explicitly described herein.

[0099] At block 410, the system activates one or more inactive automated assistant functions. The system can activate various inactive automated assistant functions, such as the automated assistant functions explicitly described herein. In some implementations, different types of gestures can be monitored in block 404, and which inactive functions are activated in block 410 may depend on the specific type of gesture detected in the monitoring of block 404.

[0100] At block 412, the system monitors a deactivation condition for the automated assistant function activated at block 410. The deactivation condition can include, for example, a timeout, a lack of detected verbal input and / or a detected directional gaze of at least a threshold duration, an explicit stop command (verbal, gesture, or touch input), and / or other conditions.

[0101] At block 414, the system determines whether a deactivation condition has been detected based on the monitoring of block 412. If not, the system proceeds back to block 412 and continues to monitor for deactivation conditions. If so, the system can deactivate the function activated at block 410 and proceed back to block 402 to receive visual data again and monitor again for the occurrence of both the invocation gesture and gaze.

[0102] As an example of blocks 412 and 414, where the activated functionality includes streaming audio data to one or more cloud-based automated assistant components, the system is capable of stopping streaming in response to detecting a lack of voice activity for at least a threshold duration (e.g., using VAD), in response to an explicit stop command, or in response to detecting (through continuous gaze monitoring) that the user's gaze has not been directed toward the client device for at least a threshold duration.

[0103] Now go to Figure 4B , diagram Figure 4A An example of an implementation of blocks 402, 404, and 406 of FIG. Figure 4B An example is shown in which gesture detection and directional gaze detection are performed using separate models, and in which monitoring directional gaze occurs only in response to first detecting a gesture. Figure 4B , box 402A is Figure 4A Specific example of block 402, blocks 404A and 404B are Figure 4A A specific example of block 404 of FIG. 406A and block 406B is Figure 4A A specific example of block 406 of FIG.

[0104] At block 402A, the system receives and buffers visual data.

[0105] At block 404A, the system processes the visual data using a gesture machine learning model to monitor for the occurrence of a gesture. In some implementations, block 404A includes subblock 404A1, wherein the system processes a portion of the visual data using a gesture machine learning model based on detecting that the portion corresponds to a human and / or body region.

[0106] At block 406A, the system determines whether a gesture has been detected based on the monitoring of block 404A. If not, the system proceeds back to block 402A, receives and buffers additional visual data, and performs another iteration of block 404A.

[0107] If, upon iteration of block 406A, the system determines that a gesture has been detected based on monitoring of block 404A, the system proceeds to block 404B.

[0108] At block 404B, the system processes the buffered and / or additional visual data using a gaze machine learning model to monitor for occurrences of directional gaze.

[0109] In some implementations, block 404B includes subblock 404B1 , where the system processes a portion of the visual data using a gaze machine learning model based on detecting that the portion corresponds to a human and / or facial region.

[0110] At block 406B, the system determines whether a directional gaze has been detected based on the monitoring of block 406 A. If not, the system proceeds back to block 402A, receives and buffers additional visual data, and performs another iteration of block 404A.

[0111] If, at the iteration of block 406B, the system determines based on the monitoring of block 404B that a gesture has been detected, the system proceeds to Figure 4A 408 or 410 of the embodiment.

[0112] Various examples of activating a dormant assistant function in response to detecting both a specific gesture and a directional gaze are described herein. However, in various embodiments, a dormant assistant function can be activated in response to detecting only one of the following: a specific gesture and a directional gaze, optionally in combination with one or more other conditions such as those described herein. For example, in some of those various embodiments, a dormant assistant function can be activated in response to detecting a user's directional gaze of at least a threshold duration and other conditions occurring simultaneously, such as movement of the user's mouth. Additionally, for example, in some of those various embodiments, a dormant assistant function can be activated in response to detecting a user's gesture and other conditions occurring simultaneously and / or close in time, such as detected voice activity.

[0113] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing devices, user-controlled resource modules, and / or other components can include one or more components of the example computing device 510.

[0114] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0115] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or onto a communication network.

[0116] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include devices and methods of all possible types of information output to a user or to another machine or computing device from computing device 510.

[0117] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include an executable Figure 4A and Figure 4B Selected aspects of the method and implementation Figure 1 , Figures 2A-2C and Figure 3 The logic of the various components is depicted in .

[0118] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory subsystem 525 used in the storage subsystem 524 can include many memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules for implementing the functionality of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible to the processor 514.

[0119] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0120] Computing device 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 510 may have more Figure 5 The computing devices depicted may have greater or fewer components.

[0121] In the case where the system described herein collects or otherwise monitors personal information about a user or can utilize personal and / or monitored information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social actions or activities, occupation, user's preferences, or user's current geographic location) or to control whether and / or how to receive content that may be more relevant to the user from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information cannot be determined for the user, or the geographic location of the user may be generalized (such as to a city, zip code, or state level) when geographic location information is obtained so that the specific geographic location of the user cannot be determined. Therefore, the user may control how information is collected and / or used about the user. For example, in some embodiments, the user may opt out of using the visual component 107 and / or an assistant device that uses visual data from the visual component 107 when monitoring gestures and / or directional gaze.< / topping> < / topping> < / artist> < / artist> < / artist>

Claims

1. A client device, comprising: at least one visual component; At least one microphone; one or more processors; a memory operatively coupled to the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more of the processors, cause one or more of the processors to: receiving a stream of visual data based on output of the visual component from the client device; receiving a stream of audio data based on output from the microphone of the client device; Based on processing the visual data, determining: the user's gaze is directed toward the client device, and a user profile of the user; determining a spoken utterance included in the audio data based on processing the audio data: corresponds in time to the gaze, and having voice characteristics that match the user profile determined based on processing the visual data; and In response to determining the gaze of the user and dependent on determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the visual data: At least one dormant function of the automated assistant is activated.

2. The client device according to claim 1, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the visual data comprises: Data is transmitted from the client device to a remote server associated with the automated assistant.

3. The client device according to claim 1, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the visual data further comprises: Content customized to the user profile is graphically rendered.

4. The client device according to claim 1, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the visual data comprises: Automatic speech processing of the audio data.

5. The client device according to claim 1, wherein: Determining the user profile of the user based on processing the visual data includes performing facial recognition based on processing the visual data.

6. The client device according to claim 1, wherein: Determining that the gaze of the user is directed toward the client device based on processing the visual data includes processing the visual data using a trained gaze machine learning model stored locally on the client device.

7. The client device according to claim 1, further comprising: determining that the user profile is authorized for use with the client device; Wherein causing the at least one dormant function of the automated assistant to be activated is further dependent upon determining that the user profile is authorized for use with the client device.

8. A method implemented by one or more processors of a client device, the method facilitating contactless interaction between one or more users and an automated assistant, the method comprising: processing image frames captured by a camera of the client device; Determining based on processing the image frame: the user's gaze is directed toward the client device, and a user profile of the user; processing audio data captured by one or more microphones of the client device; determining a spoken utterance included in the audio data based on processing the audio data: corresponds in time to the gaze, and having voice characteristics that match the user profile determined based on processing the image frame; as well as In response to determining the gaze of the user and dependent on determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the image frames: At least one dormant function of the automated assistant is activated.

9. The method according to claim 8, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the image frames comprises: Data is transmitted from the client device to a remote server associated with the automated assistant.

10. The method according to claim 9, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the image frames comprises: Automatic speech processing of the audio data.

11. The method according to claim 8, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the image frames comprises: Content customized to the user profile is graphically rendered.

12. The method according to claim 8, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to determining the gaze of the user and dependent upon determining that the spoken utterance corresponds in time to the gaze and has voice characteristics that match the user profile determined based on processing the image frames comprises: Automatic speech processing of the audio data.

13. The method according to claim 8, wherein: Determining the user profile of the user based on processing the image frames includes performing facial recognition based on processing at least one of the image frames.

14. The method according to claim 13, wherein: Determining that the gaze of the user is directed toward the client device based on processing the image frame includes processing the image frame using a trained gaze machine learning model stored locally on the client device.

15. The method according to claim 8, further comprising: determining that the user profile is authorized for use with the client device; Wherein causing the at least one dormant function of the automated assistant to be activated is further dependent upon determining that the user profile is authorized for use with the client device.

16. A client device comprising: Visual components; Presence sensor; One or more processors, wherein one or more of the processors are configured to: detecting the presence of a human in an environment of the presence sensor based on a signal from the presence sensor; In response to detecting the presence of the human in the environment: activating the vision component to provide a stream of visual data based on output from the vision component; The visual data is processed using at least one trained machine learning model stored locally on the client device to monitor for occurrences of: a user's invocation gesture captured by the visual data, and a gaze of the user directed toward the client device; Based on the monitoring, the occurrence of the following two is detected: the invocation gesture, and said gaze; and In response to detecting occurrence of both the invocation gesture and the gaze: At least one dormant function of the automated assistant is activated.

17. The client device of claim 16, wherein: The at least one dormant functionality of the automated assistant that is caused to activate in response to detecting occurrence of both the invocation gesture and the gaze includes: Data is transmitted from the client device to a remote server associated with the automated assistant.

18. The client device of claim 16, wherein: The at least one dormant functionality of the automated assistant caused to activate in response to detecting occurrence of both the invocation gesture and the gaze further comprises: Automatic speech processing of the audio data.

Images