Adapting an automated assistant based on detected mouth movements and / or gaze

Abstract

To provide an adaptation to an automated assistant based on a detected mouth motion and / or a gaze.SOLUTION: An automated assistant is adapted based on detection that a motion of the mouth of a user and / or a gaze of the user is being directed to an assistant device that provides a (graphical and / or audible) automated assistant interface of an automated assistant. Detection of the mouth motion and / or the directive gaze may be based on processing of visual data from one or more visual components associated with the assistant device, such as a camera incorporated in the assistant device. The detected mouth motion may be a motion indicating that the user (to which the mouth belongs) is talking.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Humans may engage in human-computer dialogs with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using oral natural language input (i.e., utterances), which in some cases may be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the request by providing responsive user interface output, which may include audible and / or visual user interface output. Summary of the Invention [Means for solving the problem]

[0002] Many client devices that facilitate interaction with automated assistants, also referred to herein as "assistant devices," allow users to engage in touch-free interaction with the automated assistant. For example, assistant devices often include a microphone that enables users to provide voiced utterances to invoke the automated assistant and / or otherwise interact with the assistant. The assistant devices described herein can additionally or alternatively incorporate and / or communicate with one or more visual components (e.g., a camera, a light detection and ranging (LIDAR) component, a radar component, etc.) to facilitate touch-free interaction with the automated assistant.

[0003] Implementations disclosed herein relate to adapting an automated assistant based on detecting (1) a user's mouth movements (also referred to herein as "mouth movements") and / or (2) the user's gaze directed toward an assistant device (also referred to herein as "directed gaze"), where the assistant device provides an automated assistant interface (graphical and / or audible) for the automated assistant. Detection of mouth movements and / or directed gaze may be based on processing visual data from one or more visual components associated with the assistant device, such as a camera integrated into the assistant device or a camera separate from (but in communication with) the client device. The detected mouth movements may be movements indicative of the user (to which the mouth belongs) speaking. This is in contrast to user mouth movements, which may result from the user shaking their head, stepping their foot left / right, etc. As explained below, implementations described herein may result in efficiencies in the computing resources and communication networks used to implement the automated assistant. For example, as will become apparent from the discussion below, aspects of implementations may result in more selective initiation of communications over a data network and a corresponding reduction in data traffic over the network. More selective initiation of network communications, e.g., from a client device, may further lead to more efficient use of computing resources at the remote system where the communications are initiated, since any possible communications from the client device are filtered out before any contact with the remote system is initiated. Increased efficiency in the use of data network and computing resources on the remote system may lead to significant savings in power usage by transmitters and receivers in the network, as well as in memory operations and processing utilization at the remote system. Corresponding effects may also be experienced at client devices, as described below.These effects, particularly over time and over the ongoing operation of the automated assistant, cause the network and, as a whole, the computing devices, including the devices and systems running the assistant, to experience significant additional capacity. This additional capacity can be used for further communications in the data network, whether assistant-related or not, without having to expand network capacity through, for example, additional or updated infrastructure and additional computing operations in the computing devices. Other technical improvements will become apparent from the description below.

[0004] As an example, the automated assistant can be adapted in response to detecting a user's mouth movement (optionally of a threshold duration), the user's gaze being directed at the assistant device (optionally by the same or a different threshold duration), and, optionally, detecting that the mouth movement and the user's directed gaze co-occur or occur within a threshold time proximity of each other (e.g., within 0.5 seconds, within 1.0 seconds, or other threshold time proximity). For example, the automated assistant can be adapted in response to detecting a mouth movement of at least 0.3 seconds duration and a directed gaze that is at least 0.5 seconds duration and co-occurs with the mouth movement or occurred within 0.5 seconds of the mouth movement.

[0005] In some implementations, the automated assistant may be adapted in response to detecting only lip movements and a directed gaze. In some other implementations, the automated assistant may be adapted in response to detecting lip movements and a directed gaze and detecting the occurrence of one or more other conditions. The occurrence of one or more other conditions may include, for example, detecting voice activity (e.g., any voice activity, voice activity of a user providing lip movements and a directed gaze, voice activity of an authorized user, voice activity including a verbal invocation phrase) that is temporally proximate to the detected lip movements and directed gaze based on audio data, detecting a user gesture (e.g., a "wave," "thumbs up," "high five") that co-occurs with or is temporally proximate to the detected lip movements and directed gaze based on visual data, detecting that the user is an authorized user based on audio data and / or visual data, and / or detecting other conditions.

[0006] In some implementations disclosed herein, adaptation of the automated assistant in response to detecting mouth movements and directional gaze can include adapting the rendering of user interface output by the assistant device. In some of these implementations, adapting the rendering of user interface output includes reducing the volume of audible user interface output rendered by the assistant device and / or stopping the audible user interface output and / or video output visually rendered by the assistant device.

[0007] As an example, assume that when a user begins speaking an utterance directed at an automated assistant, the user's mouth movements are detected, and the user's directional gaze is detected, co-occurring with the detected mouth movements. Further, assume that prior to and during the detection of the mouth movements and directional gaze, the assistant device is rendering audible and / or visual content. For example, an automated assistant client of the assistant device may be causing an audible rendering of a song and a visual rendering of a video that accompanies the song. In response to detecting the mouth movements and directional gaze, the automated assistant client may reduce the volume of the audible rendering of the song (while still continuing the audible rendering at the reduced volume and the visual rendering of the video). The volume reduction can improve performance of processing of audio data capturing oral utterances, such as audio data captured by one or more microphones of the assistant device. For example, voice-to-text processing of the audio data can be improved as a result of the volume reduction, voice activity detection (VAD) based on the audio data can be improved as a result of the volume reduction, speaker diarization based on the audio data can be improved as a result of the volume reduction, etc. Improved processing of audio data can increase the likelihood that the automated assistant will properly interpret the verbal utterance and respond in an appropriate manner, resulting in improved user-assistant interaction and / or reducing the risk of an inappropriate automated assistant response that may cause the user to repeat the verbal utterance (which would require computational resources to be expended in processing the repeated verbal utterance and generating and rendering another response).

[0008] As a variation of the above example, the adaptation may include ceasing the audible rendering of the song (and optionally the video) instead of reducing the volume. As a further variation of the above example, the adaptation may initially include reducing the volume of the audible rendering of the song, and the adaptation may further include subsequent ceasing of the audible rendering of the song in response to the occurrence of one or more other conditions. For example, the volume reduction may occur in response to detecting only mouth movement and directional gaze, and the ceasing may occur in response to later detection of the occurrence of voice activity based on processing of the audio data.

[0009] In some implementations, the adaptation of the rendering of the user interface output by the assistant device may additionally or alternatively include rendering a human-perceptible cue. The rendering of the human-perceptible cue can optionally occur prior to further adapting the automated assistant and can indicate (directly or indirectly) that further adaptation is about to occur. For example, the rendering of the human-perceptible cue may occur in response to initially detecting a lip movement and a directed gaze, and further adaptation may occur in response to detecting a continued lip movement and / or a continued directed gaze. Continuing with this example, the further adaptation may include transmitting, by the client device, to one or more remote automated assistant components, specific sensor data generated by one or more sensor components of the client device (no sensor data from the sensor components was transmitted prior to the further adaptation). The specific sensor data may include, for example, visual and / or audio data captured after detecting a lip movement and a directed gaze, and / or buffered visual and / or audio data captured during the performance of the lip movement and / or the directed gaze. By providing a human-perceptible cue, the user can be warned of the impending further adaptation and be given an opportunity to prevent the further adaptation. For example, if the further adaptation is conditional on the user's continued directed gaze, the user can simply avert their gaze to prevent the further adaptation (e.g., if the user does not intend to interact with the automated assistant and have it transmit sensor data). In this way, the further adaptation can be prevented, along with the use of network and / or computational resources that would result from the further adaptation. Various human-perceptible cues can be provided, such as an audible "ringing sound," an audible "verbal output" (e.g., "It looks like you're talking to your assistant, look away if you're not"), a visual symbol on the display screen of the assistant device, illumination of a light-emitting diode on the assistant device, etc.

[0010] In some implementations, adapting the rendering of the user interface output by the assistant device may additionally or alternatively include tailoring rendered content to a user corresponding to the detected lip movements and directional gaze. Tailoring the rendered content may include determining the user's distance relative to the assistant device and rendering the content based on the distance. For example, audible content may be rendered at a volume based on the user's distance corresponding to the detected lip movements and directional gaze. Also, for example, image content may be rendered at a size based on the user's distance corresponding to the detected lip movements and directional gaze. As yet another example, content may be generated based on distance. For example, more detailed content may be generated when the distance is relatively close to the client device, and less detailed content may be generated when the distance is relatively far from the client device. As a specific example, in response to the verbal utterance "What's the weather like?", a one-day weather forecast may be generated at a relatively close distance, and a three-day weather forecast may be generated at a relatively far distance. The user distance may be determined in response to the user corresponding to detected mouth movements and directed gaze, which may indicate the user is verbally engaging with the automated assistant. This may be useful in situations where multiple users (at multiple distances) are captured in the visual data, as tailoring the rendered content to the user distance corresponding to the detected mouth movements and directed gaze allows the rendered content to be tailored to users who are actively engaged in a dialog with the automated assistant.

[0011] In some implementations disclosed herein, and as described above, the adaptation of the automated assistant that occurs in response to detecting mouth movements and directional gaze may additionally and / or alternatively include adaptation of sensor data processing, such as processing of audio data and / or visual data.

[0012] In some of these implementations, the adaptation may include initiating specific processing of particular sensor data (e.g., audio data, video, images, etc.) in response to detecting lip movements and directed gaze (where specific processing was not previously performed). For example, prior to detecting lip movements and directed gaze, the automated assistant may perform only limited processing (or no processing) of particular sensor data, such as audio data, video / image data, etc. For example, prior to such detection, the automated assistant may process audio data locally when monitoring for explicit invocation phrases, but after local processing and without having the audio data processed by one or more additional components implementing the automated assistant (e.g., a remote server device that processes user input and generates appropriate responses), the data will be “discarded.” However, in response to detecting lip movements and directed gaze (and optionally the occurrence of one or more other conditions), such data may be processed by additional components. In these and other ways, processing and / or network resources may be reduced by only transmitting and / or performing specific processing of particular sensor data in response to detecting lip movements and directed gaze.

[0013] In some additional or alternative implementations described herein, adapting the processing of sensor data may include adapting local and / or remote processing based on a determined location of a user from which mouth movements and directional gaze are detected. The user's location may be relative to the client device and may be determined, for example, based on portions of visual data determined to correspond to the user. Processing audio data based on the user's location may include, for example, isolating portions of audio data corresponding to verbal utterances and / or removing background noise from the audio data. Such processing may rely on the determined location and beamforming and / or other techniques in isolating portions of audio data and / or removing background noise from the audio data. This may improve processing of audio data in environments with significant background noise, multiple speakers speaking simultaneously, etc.

[0014] In some implementations, when monitoring mouth movements and directed gazes toward the client device, a trained machine learning model (e.g., a neural network model) stored locally on the client device is used by the client device to at least selectively process at least some portions of visual data from the client device's visual components (e.g., image frames from the client device's camera). For example, in response to detecting the presence of one or more users, the client device may process at least one or more portions of the visual data using the locally stored machine learning model when monitoring mouth movements and directed gazes for at least a certain duration (e.g., at least a certain threshold duration and / or until presence is no longer detected). The client device may detect the presence of one or more users using a dedicated presence sensor (e.g., a passive infrared sensor (PIR)), using visual data and a separate machine learning model (e.g., a separate machine learning model trained only for human presence detection), and / or using audio data and a separate machine learning model (e.g., a VAD using a VAD machine learning model). In implementations in which processing of visual data when monitoring mouth movements and / or directed gaze is contingent on first detecting the presence of one or more users, power resources can be conserved through non-continuous processing of visual data when monitoring mouth movements and / or directed gaze. Instead, in those implementations, processing of visual data when monitoring mouth movements and / or directed gaze can occur only in response to detecting the presence of one or more users in the environment of the assistant device through one or more low-power consumption techniques.

[0015] In some implementations in which local machine learning models are used to monitor mouth movements and directed gaze, at least one mouth movement machine learning model is used to monitor mouth movements and a separate gaze machine learning model is used to monitor directed gaze. In some versions of these implementations, one or more “upstream” models (e.g., object detection and classification models) are used to detect face-like, eye-like, mouth-like, etc., portions of visual data (e.g., images), which can be processed using the respective machine learning models. For example, face and / or eye portions of an image can be detected using an upstream model and processed using a gaze machine learning model. Also, for example, face and / or mouth portions of an image can be detected using an upstream model and processed using a mouth movement machine learning model. As yet another example, human portions of an image can be detected using an upstream model and processed using both a gaze detection machine learning model and a mouth movement machine learning model.

[0016] In some implementations, face matching, eye matching, voice matching, and / or other techniques may be used to identify a particular user profile associated with lip movements and / or directed gaze, and content rendered by an automated assistant application on a client device that is tailored to the particular user profile. Rendering the tailored content may be all or part of adapting the automated assistant in response to detecting lip movements and directed gaze. Optionally, identifying a particular user profile occurs only after lip movements and directed gaze are detected. In some implementations, and as described above, adapting the automated assistant may require the occurrence of one or more additional conditions, where the additional conditions are in addition to gaze and / or lip movement detection. For example, in some implementations, the additional conditions may include identifying (e.g., using face matching, voice matching, and / or other techniques) that the user providing the lip movements and directed gaze is associated with a user profile that is authorized for the client device.

[0017] In some implementations, some portions of a video / image may be filtered out / ignored / weighted relatively lightly when detecting lip movements and / or gaze. For example, a television captured in a video / image may be ignored to prevent false detections resulting from a person being rendered by a television (e.g., a weather forecaster). For example, a portion of an image may be determined to correspond to a television based on a separate object detection / classification machine learning model, such as in response to detecting a particular display frequency at that portion (i.e., matching the television refresh rate) across multiple frames for that portion. Such portions may be ignored in the lip movement and / or directional gaze detection techniques described herein to prevent detection of lip movements and / or directional gaze from a television or other video display device. As another example, picture frames may be ignored. These and other techniques can mitigate false positive adaptation of an automated assistant, thereby saving various computational and / or network resources that would otherwise be consumed in false positive adaptation. Also, in various implementations, once a location of a TV, picture frame, etc. is detected, it may optionally continue to be ignored for multiple frames (e.g., while intermittently verifying, until motion of the client device or object is detected, etc.), again saving various computational resources.

[0018] The above description is provided as a general overview of various implementations disclosed herein, which, as well as additional implementations, are described in more detail herein.

[0019] In some implementations, a method implemented by one or more processors of a client device is provided that facilitates touch-free interaction between one or more users and an automated assistant. The method includes receiving a stream of image frames based on output from one or more cameras of the client device. The method further includes processing the image frames of the stream with at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a user's gaze and a user's mouth movements directed toward the one or more cameras of the client device. The method further includes detecting the occurrence of both a user's gaze and a user's mouth movements based on the monitoring. The method further includes adapting one or both of a rendering of a user interface output of the client device and a processing of audio data by the client device in response to detecting the occurrence of both a user's gaze and a user's mouth movements.

[0020] These and other implementations of the techniques described herein may include one or more of the following features.

[0021] In some implementations, adapting the rendering of the user interface output of the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movements. In some of those implementations, adapting the rendering of the user interface output of the client device includes lowering a volume of audible user interface output rendered by the client device. In some versions of those implementations, the method further includes performing voice activity detection of audio data that corresponds in time with the user's mouth movements and determining the occurrence of voice activity based on the voice activity detection of the audio data that corresponds in time with the user's mouth movements. In those versions, lowering the volume of the audible user interface output rendered by the client device is further responsive to determining the occurrence of voice activity, where the occurrence of voice activity is based on the audio data that corresponds in time with the user's mouth movements.

[0022] In some implementations in which adapting the rendering of the user interface output of the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movements, adapting the rendering of the user interface output includes ceasing the rendering of the audible user interface output rendered by the client device. In some of these implementations, the method further includes performing voice activity detection of audio data that corresponds in time with the user's mouth movements and determining the occurrence of voice activity based on the voice activity detection of the audio data that corresponds in time with the user's mouth movements. In these implementations, ceasing the rendering of the audible user interface output rendered by the client device is further responsive to determining the occurrence of voice activity, where the occurrence of voice activity is based on the audio data that corresponds in time with the user's mouth movements.

[0023] In some implementations, adapting the rendering of the user interface output of the client device includes rendering a human-perceptible cue, adapting the audio data processing by the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movement, and adapting the audio data processing by the client device includes initiating local automatic speech recognition at the client device, which is further responsive to detecting that the user's gaze continues to be directed toward one or more cameras of the client device following the rendering of the cue.

[0024] In some implementations, adapting the rendering of the user interface output of the client device includes rendering a human-perceptible cue, adapting the audio data processing by the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movement, and adapting the audio data processing by the client device includes initiating transmission of audio data captured by one or more microphones of the client device to a remote server associated with the automated assistant, and initiating the transmission of the audio data to the remote server is further responsive to detecting that the user's gaze remains directed toward one or more cameras of the client device following the rendering of the cue.

[0025] In some implementations, adapting audio data processing by the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movement. In some of those implementations, adapting audio data processing by the client device includes initiating transmission of audio data captured by one or more microphones of the client device to a remote server associated with the automated assistant. In some versions of those implementations, the method further includes performing a voice activity analysis of particular audio data included in or preceding the audio data that corresponds in time with the user's mouth movement, and determining the occurrence of voice activity based on the voice activity analysis of the particular audio data that corresponds in time with the user's mouth movement. In those versions, initiating transmission of the audio data is further responsive to determining the occurrence of voice activity, and the occurrence of voice activity is based on the audio data that corresponds in time with the user's mouth movement.

[0026] In some implementations in which adapting audio data processing by the client device is performed in response to detecting the occurrence of both a user's gaze and a user's mouth movement, adapting the audio data processing includes determining a user's position relative to the client device based on one or more of the image frames and using the user's position when processing audio data captured by one or more microphones of the client device. In some versions of these implementations, using the user's position when processing audio data captured by one or more microphones of the client device includes using the position when isolating portions of the audio data that correspond to the user's oral utterances. In some additional or alternative versions of these implementations, using the user's position when processing audio data captured by one or more microphones of the client device includes using the position when removing background noise from the audio data.

[0027] In some implementations, processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor the occurrence of both a user's gaze and a user's mouth movements includes using a first trained machine learning model to monitor the occurrence of a user's gaze and using a second trained machine learning model to monitor the occurrence of a user's mouth movements.

[0028] In some implementations, the method further includes detecting the presence of a human in an environment of the client device based on a signal from the presence sensor, and causing one or more cameras to provide a stream of image frames in response to detecting the presence of a human in the environment.

[0029] In some implementations, a client device is provided that includes at least one visual component, at least one microphone, one or more processors, and a memory operatively coupled to the one or more processors. The memory stores instructions that, in response to execution of the instructions by one or more of the processors, cause the one or more processors to perform operations of receiving a stream of visual data based on output from the visual components of the client device, processing the stream of visual data with at least one trained machine learning model stored locally on the client device to monitor occurrences of both a user's gaze and a user's mouth movements directed toward the visual components of the client device, detecting, based on the monitoring, occurrences of both the user's gaze and the user's mouth movements, and adapting rendering of a user interface output of the client device in response to detecting the occurrences of both the user's gaze and the user's mouth movements.

[0030] In some implementations, a system is provided that includes at least one visual component, one or more microphones, and one or more processors that receive a stream of visual data based on output from the visual component, wherein one or more of the processors are configured to process the stream of visual data using at least one trained machine learning model to monitor occurrences of both a user's gaze and a user's mouth movements directed toward the visual component, detect occurrences of both the user's gaze and the user's mouth movements based on the monitoring, and, in response to detecting the occurrences of both the user's gaze and the user's mouth movements, both adapt rendering of a user interface output of a client device and adapt processing of audio data captured by the one or more microphones.

[0031] Additionally, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the above-described methods. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the above-described methods.

[0032] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter listed at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2A] 1 is an exemplary process flow demonstrating various aspects of the present disclosure, according to various implementations. [Figure 2B] 1 is an exemplary process flow demonstrating various aspects of the present disclosure, according to various implementations. [Figure 3] FIG. 1 shows an example of an assistant device and a user providing directed gaze and speech (thereby causing mouth movements), and also shows images captured by the assistant device's camera as the user provides directed gaze and speech. [Figure 4A] 1 is a flowchart illustrating an example method according to implementations disclosed herein. [Figure 4B] 4B is a flowchart illustrating specific examples of some blocks of the exemplary method of FIG. 4A. [Figure 4C] 4B is a flowchart illustrating specific examples of some blocks of the exemplary method of FIG. 4A. [Figure 4D] 4B is a flowchart illustrating specific examples of some blocks of the exemplary method of FIG. 4A. [Figure 4E] 4B is a flowchart illustrating specific examples of some blocks of the exemplary method of FIG. 4A. [Figure 5] FIG. 1 illustrates an exemplary architecture of a computing device. DETAILED DESCRIPTION OF THE INVENTION

[0034] 1 illustrates an exemplary environment in which the techniques disclosed herein may be implemented. The exemplary environment includes one or more client computing devices 106. Each client device 106 may run a respective instance of an automation assistant client 110. One or more cloud-based automation assistant components 130 may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) communicatively coupled to the client devices 106 via one or more local and / or wide area networks (e.g., the Internet), generally indicated at 114. The cloud-based automation assistant components 130 may be implemented, for example, by a cluster of high-performance servers.

[0035] In various implementations, an instance of an automation assistant client 110, in interaction with one or more cloud-based automation assistant components 130, can form what appears from a user's perspective to be a logical instance of an automation assistant 120 with which the user may engage in human-computer interactions (e.g., verbal interactions, gesture-based interactions, and / or touch-based interactions). One instance of such an automation assistant 120 is shown in dashed lines in FIG. 1 . Thus, it should be understood that each user interacting with an automation assistant client 110 executing on a client device 106 may actually be interacting with their own logical instance of the automation assistant 120. For brevity and simplicity, the term “automation assistant,” as used herein as “serving” a particular user, refers to the combination of an automation assistant client 110 executing on a client device 106 operated by the user and, optionally, one or more cloud-based automation assistant components 130 (which may be shared among multiple automation assistant clients 110). It should also be understood that in some implementations, the automation assistant 120 may respond to requests from any user, regardless of whether the user is actually "served" by that particular instance of the automation assistant 120.

[0036] The one or more client devices 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a smart appliance such as a standalone interactive speaker (which in some cases may include a visual sensor), a smart television (or a standard television equipped with a network-connected dongle with automation assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's eyeglasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. As mentioned above, some client devices 106 may take the form of an assistant device that is primarily designed to facilitate interaction between the user and the automation assistant 120 (e.g., a standalone interactive device with a speaker and display).

[0037] The client device 106 may be equipped with one or more vision components 107 having one or more fields of view. The vision components 107 may take various forms, such as a monographic camera, a stereographic camera, a LIDAR component, a radar component, etc. The one or more vision components 107 may be used, for example, by a visual capture module 114 to capture visual frames (e.g., image frames (still images or video)) of the environment in which the client device 106 is deployed. These visual frames may then be at least selectively analyzed, for example, by a gaze and mouth module 116 of the adaptation engine 115 to monitor for the occurrence of a user's mouth movements (e.g., mouth movements indicating that the user is speaking) and / or a directed gaze from the user (e.g., a gaze directed toward the client device 106) captured by the visual frames. The gaze and mouth module 116 may use one or more trained machine learning models 117 in monitoring for the occurrence of mouth movements and / or directed gaze.

[0038] In response to detecting the mouth movements and directional gaze (and optionally in response to detecting one or more other conditions by other condition module 118), adaptation engine 115 can adapt one or more aspects of automated assistant 120, such as aspects of automated assistant client 110 and / or aspects of cloud-based automated assistant component 130. Such adaptation may include, for example, adaptation of user interface output (e.g., audible and / or visual) rendered by client device 106 and controlled by automated assistant client 110. Such adaptation may additionally or alternatively include, for example, adaptation of sensor data processing by client device 106 (e.g., by one or more components of the automated assistant client) and / or by one or more cloud-based automated assistant components 130.

[0039] As one non-limiting example of adapting sensor data processing, prior to detection of lip movements and directed gaze, visual and / or audio data captured at client device 106 may be processed and / or temporarily buffered only locally at client device 106 (i.e., without transmission to cloud-based automated assistant component 130). However, in response to detection of lip movements and directed gaze, such processing may be adapted by triggering transmission of audio and / or visual data (e.g., recently buffered data and / or data received after the detection) to cloud-based automated assistant component 130 for further processing. For example, detection of lip movements and directed gaze may eliminate the need for a user to speak an explicit invocation phrase (e.g., “OK assistant”) to have their verbal utterances fully processed by automated assistant 120 and responsive content generated and rendered by automated assistant 120 to the user.

[0040] For example, rather than a user having to say, "OK Assistant, what's the forecast for today" to get today's forecast, the user can instead simply look at the client device 106 and say, "What's the forecast for today" while looking at the client device 106 or nearby in time (e.g., within a time threshold before and / or after). Data corresponding to the verbal utterance of "What's the forecast for today" (e.g., audio data capturing the verbal utterance, or a text or other semantic transformation thereof) can be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the mouth movements (caused by speaking all or some portion of "What's the forecast for today") and the directed gaze, and in response to receiving the verbal utterance during and / or nearby in time with the mouth movements and the directed gaze.

[0041] In another example, rather than a user having to say "OK Assistant, turn up the heat" to increase the temperature in their home with a connected thermostat, the user instead need only look at the client device 106 and say "turn up the heat" while looking at the client device 106 or nearby in time (e.g., within a time threshold before and / or after). Data corresponding to the verbal utterance "turn up the heat" (e.g., audio data capturing the verbal utterance, or a text or other semantic transformation thereof) can be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the mouth movements (triggered by speaking all or some portion of "turn up the heat") and the directed gaze, and in response to receiving the verbal utterance during and / or nearby in time with the mouth movements and the directed gaze.

[0042] In another example, rather than a user having to say "OK Assistant, open the garage door" to open the garage door at their home, the user may instead simply look at the client device 106 and say "open the garage door" while looking at the client device 106 or nearby in time (e.g., within a time threshold before and / or after). Data corresponding to the verbal utterance "open the garage door" (e.g., audio data capturing the verbal utterance, or a text or other semantic transformation thereof) may be transmitted by the client device 106 to the cloud-based automated assistant component 130 in response to detecting the mouth movements (caused by speaking all or some portion of "open the garage door") and the directed gaze, and in response to receiving the verbal utterance during and / or nearby in time with the mouth movements and the directed gaze.

[0043] In some implementations, the transmission of data by the client device 106 may be further conditioned on the other condition module 118 determining the occurrence of one or more additional conditions. For example, the transmission of data may be further based on local voice activity detection processing of the audio data performed by the other condition module 118 indicating the presence of voice activity in the audio data. Also, for example, the transmission of data may additionally or alternatively be further based on the other condition module 118 determining that the audio data corresponds to a user providing a gesture and directional gaze. For example, the user's direction (relative to the client device 106) may be determined based on visual data, and the transmission of data may be further based on the other condition module 118 determining (e.g., using beamforming and / or other techniques) that oral utterances in the audio data come from the same direction. Also, for example, the user profile of the user may be determined based on visual data (e.g., using facial recognition), and the transmission of data may be further based on the other condition module 118 determining that oral utterances in the audio data have voice characteristics that match the user profile. As yet another example, the transmission of the data may additionally or alternatively be based on the other condition module 118 determining, based on the visual data, that a user gesture (e.g., any of one or more candidate invocation gestures) co-occurred with the user's mouth movement and / or directed gaze or occurred within a threshold amount of time of the detected mouth movement and / or directed gaze. The other condition module 118 may optionally use one or more other machine learning models 119 in determining that other conditions exist. Additional descriptions of implementations of the gaze and mouth module 116 and of the other condition module 118 are provided herein (e.g., with reference to FIGS. 2A-2B). Furthermore, additional descriptions of implementations that adapt an automated assistant based on detected mouth movement and / or gaze are provided herein (e.g., with reference to FIGS. 4B-4D).

[0044] Each of the client computing devices 106 and computing devices operating the cloud-based automation assistant component 130 may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. Operations performed by the client computing device 106 and / or by the automation assistant 120 may be distributed across multiple computer systems. The automation assistant 120 may be implemented, for example, as a computer program running on one or more computers at one or more locations coupled with each other through a network.

[0045] As described above, in various implementations, client computing device 106 may operate automated assistant client 110. In some of these various implementations, automated assistant client 110 may include speech capture module 112, visual capture module 114 described above, and adaptation engine 115, which may include gaze and mouth module 116 and optionally other condition module 118. In other implementations, one or more aspects of speech capture module 112, visual capture module 114, and / or adaptation engine 115 may be implemented separately from automated assistant client 110, for example, by one or more cloud-based automated assistant components 130.

[0046] In various implementations, the speech capture module 112 may be implemented using any combination of hardware and software and may interface with hardware such as a microphone 109 or other pressure sensor to capture an audio recording of the user's verbal utterances. Various types of processing may be performed on this audio recording for various purposes, as described below. In various implementations, the visual capture module 114 may be implemented using any combination of hardware or software and may be configured to interface with the vision component 107 to capture one or more visual frames (e.g., digital images) corresponding to an optionally adaptable field of view of the vision sensor 107.

[0047] The speech capture module 112 may be configured to capture the user's speech, for example, via the microphone 109, as described above. Additionally or alternatively, in some implementations, the speech capture module 112 may be further configured to convert the captured audio into text and / or other representations or embeddings, for example, using speech-to-text (“STT”) processing techniques. However, because the client device 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the speech capture module 112 local to the client device 106 may be configured to convert a finite number of different spoken phrases, such as phrases that invoke the automated assistant 120, into text (or other forms, such as lower-dimensional embeddings). Other speech input may be sent to the cloud-based automated assistant component 130, which may include a cloud-based STT module 132.

[0048] The cloud-based TTS module 131 may be configured to leverage the nearly limitless resources of the cloud to convert text data (e.g., natural language responses formulated by the automation assistant 120) into computer-generated speech output. In some implementations, the TTS module 131 may provide the computer-generated speech output to the client device 106, for example, to be output directly using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automation assistant 120 may be provided to the client device 106, and a local TTS module on the client device 106 may then convert the text data into computer-generated speech that is output locally.

[0049] The cloud-based STT module 132 may be configured to leverage the nearly limitless resources of the cloud to convert audio data captured by the speech capture module 112 into text, which may then be provided to the natural language understanding module 135. In some implementations, the cloud-based STT module 132 may convert an audio recording of speech into one or more phonemes and then convert the one or more phonemes into text. Additionally or alternatively, in some implementations, the STT module 132 may utilize a state decoding graph. In some implementations, the STT module 132 may generate multiple candidate text interpretations of a user's utterance and select a given translation from the candidates using one or more techniques.

[0050] The automation assistant 120 (and in particular the cloud-based automation assistant component 130) may include an intent understanding module 135, the TTS module 131 described above, the STT module 132 described above, and other components described in more detail herein. In some implementations, one or more of the modules and / or modules of the automation assistant 120 may be omitted, combined, and / or implemented in a component separate from the automation assistant 120. In some implementations, one or more of the components of the automation assistant 120, such as the intent understanding module 135, the TTS module 131, the STT module 132, etc., may be implemented at least in part on the client device 106 (e.g., in combination with or exclusive of a cloud-based implementation).

[0051] In some implementations, the automated assistant 120 generates various content for audible and / or graphical rendering to the user via the client device 106. For example, the automated assistant 120 may generate content such as a weather forecast, a daily schedule, etc., and may cause the content to be rendered in response to detecting lip movements and / or directional gaze from the user, as described herein. Also, for example, the automated assistant 120 may generate content in response to a user's free-form natural language input provided via the client device 106, in response to a user's gestures detected via visual data from the client device's visual component 107, etc. As used herein, free-form input is input that is formulated by a user and is not limited to a group of options presented for selection by the user. Free-form input may be, for example, typed input and / or spoken input.

[0052] The natural language processor 133 of the intent understanding module 135 may process natural language input generated by a user via the client device 106 and generate annotated output (e.g., in the form of text) for use by one or more other components of the automated assistant 120. For example, the natural language processor 133 may process natural language free-form input generated by a user via one or more user interface input devices of the client device 106. The generated annotated output includes one or more annotations of the natural language input and one or more (e.g., all) of the terms of the natural language input.

[0053] In some implementations, the natural language processor 133 is configured to identify and annotate various types of grammatical information in the natural language input. For example, the natural language processor 133 may include a morpheme module that can separate individual words into morphemes and / or annotate the morphemes, for example, with their classes. The natural language processor 133 may also include a part-of-speech tagger configured to annotate words with their grammatical roles. Also, for example, in some implementations, the natural language processor 133 may additionally and / or alternatively include a dependency parser (not shown) configured to determine syntactic relationships between words in the natural language input.

[0054] In some implementations, the natural language processor 133 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not shown), and the entity tagger of the natural language processor 133 may use such databases in entity tagging.

[0055] In some implementations, the natural language processor 133 may additionally and / or alternatively include a co-reference determiner (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a co-reference determiner may be used to resolve the term "there" to "Hypothetical Cafe" in the natural language input "I ate at Hypothetical Cafe the other day and loved it."

[0056] In some implementations, one or more components of natural language processor 133 may rely on annotations from one or more other components of natural language processor 133. For example, in some implementations, a named entity tagger may rely on annotations from a cross-reference determiner and / or a dependency parser when annotating all references to a particular entity. Also, for example, in some implementations, a cross-reference determiner may rely on annotations from a dependency parser when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of natural language processor 133 may use other, relevant previous input and / or other associated data of the particular natural language input to determine one or more annotations.

[0057] The intent understanding module 135 may also include an intent matcher 134 configured to determine the intent of a user involved in an interaction with the automated assistant 120. Although shown separately from the natural language processor 133 in FIG. 1 , in other implementations the intent matcher 134 may be an integral part of the natural language processor 133 (or more generally of a pipeline that includes the natural language processor 133). In some implementations, the natural language processor 133 and the intent matcher 134 may collectively form the intent understanding module 135 described above.

[0058] The intent matcher 134 can use various techniques to determine the user's intent, for example, based on output from the natural language processor 133 (which may include annotations and terminology of the natural language input), based on user touch input at a touch-sensitive display of the client device 106, and / or based on gestures and / or other visual cues detected in the visual data. In some implementations, the intent matcher 134 may have access to one or more databases (not shown) that include multiple mappings between, for example, grammars and responsive actions (or more generally, intentions), visual cues and responsive actions, and / or touch input and responsive actions. For example, the grammars included in the mappings can be selected and / or learned over time and can represent common user intents. For example, one grammar, "play <artist>," can be mapped to an intent that invokes a responsive action that causes music by <artist> to be played on the client device 106 operated by the user. Another grammar, "today [weather|forecast]," can match user queries such as "What's the weather like today?" and "What's the forecast for today?" As another example, visual cue-to-action mappings may include "global" mappings applicable to multiple users (e.g., all users) and / or user-specific mappings. Some examples of visual cue-to-action mappings include mappings for gestures. For example, a "shake" gesture may be mapped to an action that causes tailored content (tailored to the user giving the gesture) to be rendered to the user, a "thumbs up" gesture may be mapped to a "play music" action, and a "high five" gesture may be mapped to a "routine" of automated assistant actions to be performed, such as turning on a smart coffee maker, turning on several smart lights, and audibly rendering a news summary.

[0059] In addition to or instead of grammars, in some implementations, the intent matcher 134 can utilize one or more trained machine learning models, alone or in combination with one or more grammars, visual cues, and / or touch inputs. These trained machine learning models may be stored in one or more databases and can be trained to identify intents, for example, by embedding data indicative of a user's utterances and / or any detected user-provided visual cues into a reduced-dimensionality space and then determining which other embeddings (and therefore intents) are closest, using techniques such as, for example, Euclidean distance, cosine similarity, etc.

[0060] As seen in the example grammar "play <artist>" above, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"). Slot values ​​can be determined in a variety of ways. Often, a user actively provides a slot value. For example, for the grammar "order <topping> pizza," the user may likely speak the phrase "order a sausage pizza," in which case the slot <topping> is automatically filled. Additionally or alternatively, if a user invokes a grammar that includes slots to be filled with slot values ​​without the user actively providing them, the automation assistant 120 may prompt the user for those slot values ​​(e.g., "What type of crust do you want on your pizza?"). In some implementations, slots may be filled with slot values ​​based on visual cues detected based on visual data captured by the visual component 107. For example, a user may utter something like "order a lot of these cat dishes" while holding up three fingers against the visual component 107 of the client device 106. Or, the user may grab a DVD case for a particular movie and say something like, "Find me more movies like this."

[0061] In some implementations, the automation assistant 120 may facilitate (or “intermediate”) transactions between a user and an agent, which may be an independent software process that receives input and provides responsive output. Some agents may take the form of a third-party application, for example, that may or may not run on a computing system separate from that operating the cloud-based automation assistant component 130. One type of user intent that may be identified by the intent matcher 134 is to engage with a third-party application. For example, the automation assistant 120 may provide access to an application programming interface (“API”) to a pizza delivery service. A user may invoke the automation assistant 120 and provide a command such as “I would like to order a pizza.” The intent matcher 134 may map this command to a grammar that triggers the automation assistant 120 to engage the third-party pizza delivery service. The third-party pizza delivery service may provide the automation assistant 120 with a minimal list of slots that need to be filled to fulfill the pizza delivery order. The automation assistant 120 may generate and provide to the user (via the client device 106) a natural language output requesting parameters for the slots.

[0062] The execution module 138 may be configured to receive the predicted / inferred intent and associated slot value (whether actively provided by the user or requested of the user) output by the intent matcher 134 and execute (or "resolve") the intent. In various implementations, the execution (or "resolve") of the user's intent may, for example, cause the execution module 138 to generate / obtain various execution information (also referred to as "responsive" information or data).

[0063] Because intent can be fulfilled in various ways, performance information can take various forms. Suppose a user requests pure information, such as, "Where were the outdoor scenes in 'The Shining' filmed?" The user's intent can be determined to be a search query, for example, by the intent matcher 134. The intent and content of the search query can be provided to the fulfillment module 138, which can communicate with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) for responsiveness information, as shown in FIG. 1 . The fulfillment module 138 can provide data indicative of the search query (e.g., the text of the query, a dimensionality-reduced embedding, etc.) to the search module 150. The search module 150 can provide responsiveness information, such as GPS coordinates, or other more explicit information, such as "Timberline Lodge, Mount Hood, Oregon." This responsiveness information can form part of the performance information generated by the fulfillment module 138.

[0064] Additionally or alternatively, the fulfillment module 138 may be configured to receive, for example, from the intent understanding module 135, the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.) and trigger responsive actions. The responsive actions may include, for example, ordering a product / service, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, initiating a multi-action routine, etc. In some such implementations, the fulfillment information may include a slot value associated with the fulfillment, an acknowledgment response (which in some cases may be selected from predetermined responses), etc.

[0065] Additionally or alternatively, the fulfillment module 138 may be configured to infer a user's intent (e.g., based on time of day, past interactions, etc.) and obtain responsiveness information about those intents. For example, the fulfillment module 138 may be configured to obtain a daily calendar summary for the user, a weather forecast for the user, and / or other content for the user. The fulfillment module 138 may further cause such content to be "pushed" for graphical and / or audible rendering to the user. For example, the rendering of such content may be adaptive in response to the adaptation engine 115 detecting the occurrence of mouth movements and directional gaze.

[0066] The natural language generator 136 may be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 136 may be configured to receive as input performance information associated with the performance of an intent and to generate the natural language output based on the performance information. Additionally or alternatively, the natural language generator 136 may receive information from other sources, such as third-party applications, and may use that information to assemble the natural language output for the user.

[0067] 2A and 2B, various examples are shown of how the gaze and mouth module 116 can detect specific mouth movements and / or directional gaze, and how the adaptation engine 115 can adapt the automated assistant accordingly.

[0068] 2A , the visual capture module 114 provides visual frames to the gaze and mouth module 116. In some implementations, the visual capture module 114 provides a real-time stream of visual frames to the gaze and mouth module 116. In some of those implementations, the visual capture module 114 begins providing visual frames in response to a signal from a separate presence sensor 105 indicating the presence of a human in the environment with the client device 106. For example, the presence sensor 105 may be a PIR sensor and may provide a signal to the visual capture module 114 in response to detecting the presence of a human. The visual capture module 114 may refrain from providing any visual frames to the gaze and mouth module 116 unless a human presence is detected. In other implementations in which the visual capture module 114 only selectively provides visual frames to the gaze and mouth module 116, additional and / or alternative cues may be used to initiate such provision. For example, human presence may be detected based on analysis of the visual frame by one or more other components, audio data from the speech capture module 112, and / or other signals.

[0069] The gaze and mouth module 116 processes the visual frames using one or more machine learning models 117 to monitor the occurrence of both directed gaze and mouth movement. If both directed gaze and mouth movement are detected, the gaze and mouth module 116 provides gaze and mouth movement detection instructions to the adaptation engine 115.

[0070] 2A , the visual frames and / or audio data (provided by the speech capture module 112) are also provided to the other condition module 118. The other condition module 118 processes the provided data, optionally using one or more other machine learning models 119, to monitor for the occurrence of one or more other conditions. For example, the other conditions may be detecting any voice activity based on the audio data, detecting the presence of a spoken call phrase in the audio data, detecting voice activity from the direction or position of the user based on the audio data, detecting that the user is an authorized user based on the visual frames and / or audio data, detecting user gestures (provided with mouth movements and directional gaze) based on the visual frames, etc. When the other condition is detected, the other condition module 118 provides an indication of the occurrence of the other condition to the adaptation engine 115.

[0071] When the adaptation engine 115 receives the directional gaze and mouth movement indications, as well as the temporally proximate indications of other conditions, the adaptation engine 115 causes adaptation 101 of the automated assistant 120. For example, adaptation 101 may include adapting the rendering of the user interface output of the client device 106 and / or adapting audio data processing by the client device 106.

[0072] In some implementations, and as described in more detail with respect to FIG. 2B , the gaze and mouth module 116 can use one or more first machine learning models 117 for detecting directional gaze and one or more second machine learning models 117 for detecting mouth movements. In some other implementations, the gaze and mouth module 116 can use an end-to-end machine learning model that can accept visual frames (or their features) as input and generate an output (based on processing the input across the model) indicating whether mouth movements and directional gaze occurred. Such a machine learning model can be, for example, a neural network model such as a recurrent neural network (RNN) model that includes one or more memory layers (e.g., long short-term memory (LSTM) layers). Training of such an RNN model can be based on training examples that include, as training example input, a sequence of visual frames (e.g., video) and, as training example output, an indication of whether the sequence includes both mouth movements and directional gaze. For example, the training example output can include one or more values ​​that each indicate whether both mouth movements and directional gaze are present. For example, the training example output may include a sequence of values, each of the values ​​in the sequence for a corresponding one of the visual frames.

[0073] 2B shows an example in which the mouth movement and gaze detection module 116 includes a mouth module 116A that uses a mouth machine learning model 117A when monitoring the occurrence of mouth movement, and a gaze module 116B that uses a gaze machine learning model 117B when monitoring the occurrence of directional gaze. The other condition module 118 is not shown in FIG. 2B for simplicity, but may optionally be used in combination with the mouth module 116A and the gaze module 116B, similar to that described with respect to FIG. 2A.

[0074] 2B , the visual capture module 114 provides the visual frames to the detection and classification module 116C. The detection and classification module 116C uses the object detection and classification machine learning model 117C to classify various regions of each visual frame. For example, the detection and classification module 116C may classify human regions (if any) in each visual frame that correspond to humans and provide an indication of such human regions to the mouth module 116A and the gaze module 116B for each visual frame. Also, for example, the detection and classification module 116C may classify regions (if any) in each visual frame that correspond to face regions and provide an indication of such regions to the mouth module 116A and the gaze module 116B for each visual frame. Also, for example, the detection and classification module 116C may classify regions (if any) in each visual frame that correspond to human eye regions and provide an indication of such regions to the gaze module 116B for each visual frame. As yet another example, the detection and classification module 116C may classify the regions (if any) of each visual frame that correspond to human mouth regions and provide an indication of such regions to the mouth module 116A for each visual frame.

[0075] In some implementations, the mouth module 116A can use the given region to process only the corresponding portion of each visual frame. For example, the mouth module 116A can “crop” and resize the visual frames to process only the portion containing the human region, or the face region, or the mouth region. In some of these implementations, the mouth machine learning model 117A can be trained based on the “cropped” visual frames, and the resizing can be up to a size according to the input dimensions of such a model. In some additional or alternative implementations, the mouth module 116A can use the given region to skip processing some visual frames altogether (e.g., those indicated as not containing a human and / or face region). In still other implementations, the mouth module 116A can use the given region as an attention mechanism (e.g., as a separate attention input to the mouth machine learning model 117A) to focus the processing of each visual frame.

[0076] Similarly, in some implementations, the attention module 116B can use the given region to process only the corresponding portion of each visual frame. For example, the attention module 116B can “crop” and resize the visual frames to process only the portion containing the human region, or the face region, or the eye region. In some of these implementations, the attention machine learning model 117B can be trained based on the “cropped” visual frames, and the resizing can be up to a size according to the input dimensions of such a model. In some additional or alternative implementations, the attention module 116B can use the given region to skip processing some visual frames altogether (e.g., those indicated as not containing human and / or face regions). In still other implementations, the attention module 116B can use the given region as an attention mechanism (e.g., as a separate attention input to the attention machine learning model 117B) to focus the processing of each visual frame.

[0077] In some implementations, the detection and classification module 116C can additionally or alternatively provide an indication of a particular region to the other condition module 118 (not shown in FIG. 2B for simplicity) for use by the other condition module 118. For example, when detecting a gesture is an additional condition for adapting an automated assistant, the body region can be used by the other condition module 118 in detecting the gesture using a corresponding gesture machine learning model.

[0078] In some implementations, the detection and classification module 116C can additionally or alternatively provide the mouth module 116A and the gaze module 116B with indications of regions that should be classified as TV or other video display sources. In some of these implementations, the modules 116A and 116B can crop those regions from the processed visual frame, distract attention from those regions, and / or otherwise ignore those regions in detection or reduce the chance of detection being based on such regions. In these and other ways, false positive adaptation of the automated assistant can be mitigated.

[0079] As described above, the mouth module 116A can use one or more mouth machine learning models 117A to detect mouth movements. Such machine learning models may be, for example, neural network models such as RNN models including one or more memory layers. Training of such RNN models may be based on training examples that include, as training example inputs, sequences of visual frames (e.g., videos) and, as training example outputs, indications of whether the sequence includes one or more specific mouth movements. For example, the training example output may be a single value indicating whether any mouth movements indicating that the corresponding user is speaking are present. For example, the single value may be “0” when no mouth movements are present and “1” when mouth movements indicating that the user is speaking are present. For example, the training example output may include one or more values, each indicating whether any mouth movements indicating that the corresponding user is speaking are present. For example, the training example output may include a sequence of values, each for a corresponding one of the visual frames (e.g., when the model being trained is a sequence-sequence model). For example, the value may be “0” when no mouth movements are present and “1” when mouth movements indicating that the user is speaking are present. As another example, the training example output may include a single value for a sequence of visual frames, where the single value indicates whether any mouth movements are present that indicate the corresponding user is speaking.

[0080] In some of these and / or other implementations, the mouth module 116A determines mouth activity only when mouth activity is detected with at least a threshold probability and / or for at least a threshold duration. For example, a stream of image frames can be processed using a sequence-sequence RNN model, and processing each frame can yield a corresponding probability that mouth activity is occurring (which may be based on previous frames due to the recurrent nature of the network). The mouth activity module can determine that there is mouth activity only if at least X% of the sequence of image frames (corresponding to the threshold duration) have a corresponding probability of satisfying the threshold. For example, assume that X% is 60%, the probability threshold is 0.6, and the threshold duration is 0.25 seconds. Further, assume that five image frames correspond to 0.5 seconds. If the image frames are processed to generate a probability of [0.75, 0.85, 0.5, 0.7, 0.9], mouth activity can be detected because 80% of the frames exhibited mouth activity with a probability greater than 0.7. Additional and / or alternative machine learning models and / or techniques may be used to detect mouth movements.

[0081] The gaze module 116B can use one or more gaze machine learning models 117B to detect directed gaze. Such machine learning models can be, for example, neural network models such as convolutional neural network (CNN) models. Training of such CNN models can be based on training examples that include visual frames (e.g., images) as training example inputs and an indication of whether the images contain directed gaze as training example outputs. For example, the training example output can be a single value indicating whether directed gaze is present. For example, the single value can be "0" when no directed gaze is present, "1" when gaze is directed directly at or within 5 degrees of the sensor capturing the image, "0.75" when gaze is directed within 5-10 degrees of the sensor capturing the image, etc.

[0082] In some of these and / or other implementations, the gaze module 116B determines a directional gaze only when a directional gaze with at least a threshold probability and / or of at least a threshold duration is detected. For example, a stream of image frames can be processed using a CNN model, and processing each frame can yield a corresponding probability that the frame contains a directional gaze. The gaze module can determine that there is a directional gaze only if at least X% of the sequence of image frames (corresponding to the threshold duration) have a corresponding probability of satisfying the threshold. For example, assume that X% is 60%, the probability threshold is 0.7, and the threshold duration is 0.5 seconds. Further, assume that 10 image frames correspond to 0.5 seconds. If image frames are processed to produce probabilities of [0.75, 0.85, 0.5, 0.4, 0.9, 0.95, 0.85, 0.89, 0.6, 0.85], then 70% of the frames exhibited directional gaze with a probability greater than 0.7, and thus directional gaze can be detected. In these and other ways, directional gaze can be detected even when a user temporarily averts their gaze direction. Additional and / or alternative machine learning models (e.g., RNN models) and / or techniques can be used to detect directional gaze that occurs for at least a threshold duration.

[0083] Figure 3 shows an example of client device 106 and visual component 107 of Figure 1. In Figure 3, the exemplary client device is labeled 106A and further includes a speaker and a display. In Figure 3, the exemplary visual component is labeled 107A and is a camera. Figure 3 also shows user 301 speaking (not shown) and providing a gaze directed toward camera 107A. Figure 3 also shows an exemplary image 360 ​​captured by camera 107A as the user speaks and provides a directional gaze. The user can be seen captured in image 360, as well as a television behind the user (and therefore not visible in the perspective view of Figure 3).

[0084] In image 360, a bounding box 362 is provided to represent a region of the image that may be determined (e.g., by detection and classification module 116C of FIG. 2B) to correspond to a face. In some implementations, a gaze and lip module operating on client device 106A need only process (or focus attention on) a portion of the image when monitoring directional gaze and lip movement based on that portion being indicated as corresponding to a face. Although only a single image is shown in FIG. 3, in various implementations, directional gaze detection and / or lip movement detection may be based on a sequence of images, as described herein.

[0085] A bounding box 363 is also provided in image 360 ​​to represent a region of the image that may be determined to correspond to a video display and that may result in a false visual cue. For example, a television may render video that shows one or more individuals speaking, looking into the camera, etc., all of which may be mistaken for the appearance of lip movements and / or directional gaze. In some implementations, the detection and classification module 116C of FIG. 2B can determine such a region (e.g., based on detecting a TV classification) and / or can determine such a region based on an analysis of image 360 ​​and preceding images to determine that the region has a display frequency that corresponds to the display frequency of a video display (e.g., approximately 60 Hz, 120 Hz, and / or other typical video display frequencies). In some implementations, the lip movement module and / or gaze module can crop the region from the processed visual frame, distract attention from the region, and / or otherwise ignore the region in detection or reduce the chance of detection being based on such a region. In these and other ways, misjudgment of automated assistant adaptation can be mitigated.

[0086] FIG. 4A is a flowchart illustrating an example method 400 according to an implementation disclosed herein. FIGS. 4B, 4C, 4D, and 4E each illustrate a flowchart illustrating a different example implementation of block 410 of FIG. 4A. For convenience, the operations of the flowcharts of FIGS. 4A-4D are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of a computing system (e.g., a client device and / or a remote computing system) that implements the automated assistant 120. Moreover, while the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0087] In block 402, the system receives visual data based on output from the visual component. In some implementations, the visual component may be integrated with a client device including an assistant client. In some implementations, the visual component may be separate from but communicate with the client device. For example, the visual component may include a standalone smart camera that communicates wired and / or wirelessly with a client device including an assistant client.

[0088] In block 404, the system processes the visual data using at least one machine learning model to monitor the occurrence of both mouth movements and directed gaze.

[0089] In block 406, the system determines whether both mouth movement and directed gaze have been detected based on the monitoring of block 404. If not, the system proceeds back to block 402 to receive additional visual data and iteratively perform blocks 404 and 406 again. In some implementations, the system determines that both mouth movement and directed gaze have been detected based on detecting that the mouth movement and directed gaze co-occur or occur within a threshold time proximity of each other. In some additional or alternative implementations, the system determines that both mouth movement and directed gaze have been detected based on detecting that the mouth movement lasts for at least a threshold duration and / or the directed gaze lasts for at least a threshold duration (which may optionally be the same as or different from that used for the mouth movement duration). As described herein, the system may use one or more machine learning models when performing block 406.

[0090] In the iteration of block 406, if the system determines, based on the monitoring of block 404, that both mouth movement and eye gaze have been detected, the system optionally proceeds to block 408 (or directly to block 410 if block 408 is not included).

[0091] In optional block 408, the system determines whether one or more other conditions are satisfied. If not, the system proceeds back to block 402 to receive additional visual data and again iteratively perform blocks 404, 406, and 408. If satisfied, the system proceeds to block 410. The system can use the visual data, audio data, and / or other sensor or non-sensor data received in block 402 to determine whether one or more other conditions are satisfied. Various other conditions may be considered by the system, such as those explicitly described herein.

[0092] In block 410, the system adapts the automated assistant. The system can adapt the automated assistant in various ways, such as one or more of those described below with respect to Figures 4B-4D.

[0093] 4B-4D, various example implementations of block 410 of FIG. 4A are shown. Turning first to FIG. 4B, an initial iteration 410B1 of example block 410 is performed in response to "yes" determinations at blocks 406 and 408 of FIG. 4A. In iteration 410B1, the system provides an audible and / or visual cue prior to providing sensor data (e.g., visual and / or audio data) for remote processing. The rendering of the cue may indicate (directly or indirectly) that remote processing of the audio and / or visual data is about to occur. Various human-perceptible cues may be rendered, such as an audible "ringing" and / or the illumination of a light-emitting diode on the assistant device.

[0094] After iteration 410B1, further iterations of blocks 402, 404, and 406 (FIG. 4A) are performed, with further “yes” decisions being made at blocks 406 and 408, as shown in FIG. 4B. The further “yes” decisions indicate that the user is speaking and maintaining their gaze at the client device. In response to the further “yes” decisions, an additional iteration 410B2 of example block 410 is performed. In iteration 410B2, the system transmits audio and / or visual data to a remote component for remote processing. If no further “yes” decisions are made, further adaptations of iteration 410B2 do not occur. For example, if the user has averted their gaze and / or stopped speaking, a “no” decision will be made and further adaptations of iteration 410B2 will be prevented.

[0095] Turning now to FIG. 4C , example block 410, block 410C1, is implemented in response to “yes” decisions at blocks 406 and 408 of FIG. 4A . In block 410C1, if “yes” decisions are made at blocks 406 and 408, the system reduces the volume of the audible output being rendered. In some implementations, further example blocks 410C2 and optionally 410C3 of block 410 are also implemented. In block 410C2, the system begins specific processing of the audio data. Block 410C2 can be implemented in response to “yes” decisions at blocks 406 and 408 of FIG. 4A and may include specific processing of buffered and / or streaming audio data. The specific processing may include, for example, voice activity detection, call phrase detection, and / or other processing. In block 410C3, if the specific processing of block 410C2 satisfies one or more conditions, the system stops the audible output. For example, if the voice activity detection of FIG. 410C2 indicates that a user's oral utterances are present in the processed audio data and / or if the processing indicates that oral utterances are present and emanating from a direction of the user corresponding to mouth movements and directional gaze, the system may stop the audible processing.

[0096] 4D , example block 410, block 410D1, is performed in response to “yes” determinations at blocks 406 and 408 of FIG. 4A. In block 410D1, the system stops the audible output being rendered in response to “yes” determinations at blocks 406 and 408. In some implementations, a further example block, block 410D2, of block 410 is also performed. In block 410D2, the system begins specific processing of the audio data. Block 410D2 can be performed in response to “yes” determinations at blocks 406 and 408 of FIG. 4A and may include specific processing of buffered and / or streaming audio data. The specific processing may include, for example, voice activity detection, call phrase detection, transmission of the audio data to a remote component for further processing, and / or other processing.

[0097] Turning now to FIG. 4E, example block 410, block 410E, is implemented in response to "yes" determinations at blocks 406 and 408 of FIG. 4A. In block 410E, the system uses the determined location of the user when processing the audio data in response to the "yes" determinations made at blocks 406 and 408. In some implementations, a further example block, block 410D2, of block 410 is also implemented. The user's location may be relative to the client device and can be determined by the system based, for example, on portions of the visual data determined to correspond to the user. The user's location may be determined and used in processing based on the user's correspondence to detected mouth movements and directional gaze (which may indicate the user is attempting to audibly interact with the system). Processing the audio data based on the user's location may include, for example, isolating portions of the audio data corresponding to oral utterances and / or removing background noise from the audio data. Such processing may rely on determined locations and beamforming and / or other techniques in isolating portions of the audio data and / or removing background noise from the audio data.

[0098] 4A-4E are shown separately, it should be noted that in various implementations, multiple exemplary blocks of FIG. 4A-4E may be performed in response to detecting the occurrence of mouth movement and directional gaze. As an example, in response to "yes" determinations at blocks 406 and 408 of FIG. 4A, blocks 410C1 and 410C2 of FIG. 4C may be performed, and block 410C2 may include processing of block 410E of FIG. 4E. As another example, in response to an initial "yes" determination at blocks 406 and 408 of FIG. 4A, block 410B1 of FIG. 4B may be performed, and in response to an additional "yes" determination, block 410B2 may be performed, and a remote component may perform processing of block 410E of FIG. 4E.

[0099] Various examples of adapting an automated assistant in response to detecting both mouth movements and directed gaze are described herein. However, in various implementations, the automated assistant may be adapted in response to detecting only one of mouth movements and directed gaze, optionally in combination with one or more other conditions, such as those described herein. For example, in some of these various implementations, the automated assistant may be adapted in response to detecting a user's directed gaze for at least a threshold duration along with other co-occurring conditions, such as a user gesture. Also, for example, in some of these various implementations, the automated assistant may be adapted in response to detecting a user's mouth movements along with other co-occurring and / or temporally proximate conditions, such as a user gesture and / or detected voice activity.

[0100] 5 is a block diagram of an example computing device 510 that may optionally be used to implement one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the cloud-based automation assistant component 130, and / or other components may include one or more components of the example computing device 510.

[0101] Computing device 510 typically includes at least one processor 514 that communicates with several peripheral devices via a bus subsystem 512. These peripheral devices may include, for example, a storage subsystem 524 including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices enable user interaction with computing device 510. Network interface subsystem 516 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

[0102] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or graphics tablet, a scanner, a touchscreen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods of inputting information into the computing device 510 or over a communications network.

[0103] The user interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 510 to a user or to another machine or computing device.

[0104] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of the methods of Figures 4A-4D and for implementing various components shown in Figures 1, 2A-2B, and 3.

[0105] These software modules are generally executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include several memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution, and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive, along with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 526 in the storage subsystem 524 or on other machines accessible by the processor 514.

[0106] The bus subsystem 512 provides a mechanism for allowing the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0107] The computing device 510 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 510 shown in Figure 5 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of the computing device 510 may have more or fewer components than the computing device shown in Figure 5.

[0108] In situations where the systems described herein may collect or otherwise monitor personal information about users or utilize personal and / or monitored information, users may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Also, certain data may be treated in one or more ways such that personally identifying information is removed before it is stored or used. For example, a user's identity may be treated such that personally identifying information about the user cannot be determined, or a user's geographic location may be generalized (e.g., to the city, zip code, or state level) such that geographic location information is obtained such that the user's specific geographic location cannot be determined. Thus, users may have control over how information is collected about them and / or used. For example, in some implementations, the user may not select an assistant device using visual component 107 and / or using visual data from visual component 107 when monitoring the occurrence of mouth movements and / or directional gaze. [Explanation of symbols]

[0109] 105 Presence Sensor 106 Client Computing Device, Client Device 106A Client Device 107 Visual components, visual sensors 107A Camera 109 Microphone 110 Automation Assistant Client 112 Speech Capture Module 114 Local and / or Wide Area Network, Visual Capture Module 115 Adaptive Engine 116 Gaze and Mouth Module 116A Port Module, Module 116B gaze module, module 116C Detection and Classification Module 117 Trained Machine Learning Models, Machine Learning Models 117A Machine Learning Model 117B Attention Machine Learning Model 117C Object Detection and Classification Machine Learning Models, Detection and Classification Machine Learning Models 118 Other Condition Module 119 Machine Learning Models 120 Automation Assistant 130 Cloud-Based Automated Assistant Components 131 Cloud-based TTS module, TTS module 132 Cloud-based STT module, STT module 133 Natural Language Processor 134 Intent Matcher 135 Natural language understanding module, intent understanding module 136 Natural Language Generator 138 Execution Module 150 Search Modules 510 Computing Devices 512 Bus Subsystem 514 processor 516 Network Interface Subsystem 520 User Interface Output Device 522 User Interface Input Devices 524 Memory Subsystem 525 Memory Subsystem, Memory 526 File Storage Subsystem 530 Main Random Access Memory (RAM) 532 read-only memory (ROM)

Claims

1. 1. A method for facilitating touch-free interaction between a user and an automated assistant, the method being implemented by one or more processors of a client device, the method comprising: receiving a stream of image frames based on output from one or more cameras of the client device; processing the image frames of the stream using at least one trained machine learning model stored locally on the client device to monitor for the occurrence of the user's gaze directed towards the client device; initially detecting an occurrence of the user's gaze based on the monitoring; in response to initially detecting the occurrence of the gaze of the user; Rendering a human-perceptible cue; determining whether there is a continued occurrence of the user's gaze based on the monitoring and subsequent to rendering the human-perceptible cue; In response to determining that there is a continued occurrence of the user's gaze, transmitting sensor data from one or more sensors of the client device to one or more remote automated assistant components; A method comprising:

2. 2. The method of claim 1, wherein the sensor data transmitted to the one or more remote automated assistant components includes the image frame or an additional image frame based on additional output from the one or more cameras.

3. The method of claim 1 , wherein the sensor data transmitted to the one or more remote automated assistant components includes audio data based on output from one or more microphones of the client device.

4. 10. The method of claim 1, wherein the sensor data transmitted to the one or more remote automated assistant components comprises buffered sensor data that is buffered prior to detecting a subsequent continued occurrence of both the user's gaze and the user's mouth movement.

5. In response to determining that there is no continued occurrence of the gaze of the user, Preventing transmission of the sensor data to the one or more remote automation assistant components. The method of claim 1 further comprising:

6. the human-perceptible cue comprises an audible output, and the step of rendering the human-perceptible cue comprises: The method of claim 1 , further comprising rendering the audible output through a speaker of the client device.

7. The method of claim 6 , wherein the audible output comprises a spoken output from the automated assistant.

8. the human-perceptible cue comprises a visual output, and the step of rendering the human-perceptible cue comprises: The method of claim 1 , comprising rendering the visual output via a visual display of the client device.

9. The method of claim 8 , wherein the visual output is a symbol.

10. at least one visual component; at least one microphone; one or more processors; 10. A client device comprising the one or more processors and a memory operatively coupled to the one or more processors, the memory storing instructions that, in response to execution of the instructions by one or more of the processors, cause the one or more of the processors to perform the method of any one of claims 1 to 9.

11. A computer program comprising instructions which, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 9.

12. A computer-readable storage medium containing instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of claims 1 to 9.

Images