Dynamically determining whether to perform candidate automated assistant actions determined from spoken utterances

Abstract

Implementations perform automatic speech recognition (ASR) on audio data detected via a microphone of an assistant device independently of any explicit assistant call input to generate ASR text predicting spoken utterances captured in the audio data. The ASR text is processed and a candidate automated assistant action corresponding to the command, if any, is generated. For each of any candidate automated assistant actions, it is determined whether (a) causes automatic performance of the automated assistant action in response to the spoken utterance, or alternatively, (b) suppresses any automatic performance of the automated assistant action in response to the spoken utterance. This determination may be made based on processing both: (i) action features of candidate automated assistant actions; and (ii) environmental characteristics, each environmental characteristic reflecting a corresponding current value of a corresponding dynamic state of an environment of the assistant device.

Description

Background Art

[0001] A human may engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using verbal natural language input (i.e., utterances), which in some cases may be converted to text and then processed. The automated assistant responds to the request by providing responsive user interface output, which may include audible and / or visual user interface output. The automated assistant may additionally and / or alternatively respond to the request by controlling other computing devices such as smart devices (e.g., smart lights, smart TVs).

[0002] As mentioned above, many automated assistants are configured to interact via spoken utterances. To preserve user privacy and / or conserve resources, the user typically must explicitly invoke the automated assistant, which then processes the spoken utterances entirely. The explicit invocation of the automated assistant typically occurs in response to receiving some kind of user interface input at a client device. The client device includes an assistant interface that provides an interface to a user of the client device for interfacing with the automated assistant (e.g., receiving input from the user, and providing audible and / or graphical responses), and interfaces with one or more additional components that implement the automated assistant (e.g., on-device components and / or remote server devices that process the user input and generate appropriate responses).

[0003] Some user interface inputs that have been used to explicitly invoke an automated assistant via a client device include hardware and / or virtual buttons at the client device for invoking the automated assistant (e.g., tapping a hardware button, selecting a graphical interface element displayed by the client device). Many automated assistants can additionally or alternatively be explicitly invoked in response to one or more specific verbal invocation phrases, which are also referred to as "hot words / phrases" or "trigger words / phrases." For example, specific verbal invocation phrases such as "Hey Assistant," "OK Assistant," and / or "Assistant" can be spoken to invoke the automated assistant. Some automated assistants can additionally or alternatively be explicitly invoked in response to detecting, via the assistant device's camera and / or other visual components, that the invoking user maintains a gaze toward the assistant device, is within a certain positioning and / or orientation range relative to the assistant device, and / or has provided a certain touchless gesture. Summary of the invention

[0004] Implementations disclosed herein enable a user to bypass explicit invocation of an automated assistant, but still provide a spoken utterance that embodies an automated assistant command, and cause the automated assistant to cause the execution of an automated assistant action corresponding to the assistant command. For example, an implementation may enable a user to say "what's the thermostat's set point" near an assistant device (i.e., a client device that at least partially implements the automated assistant), and in response, the automated assistant will cause the current set point of the smart thermostat to be obtained, and will provide an audible and / or visual response that communicates the current set point. This may occur in response to the user saying "what's the thermostat's setpoint" even though the user did not provide any explicit invocation input associated with the spoken utterance. For example, this may occur even though the user did not provide any specific invocation phrase before or after the spoken utterance, did not provide any input to a button of the assistant device, did not provide any touchless invocation gestures, or maintained a gaze directed toward the camera (if any) of the assistant device, and did not provide any other explicit invocation input.

[0005] More specifically, in implementations disclosed herein, automatic speech recognition (ASR) is performed on audio data detected via a microphone of an assistant device (i.e., a client device that (at least partially) implements an automated assistant) to generate ASR text that predicts the user's spoken utterances captured in the audio data. As noted above, ASR is performed despite the user not providing any explicit invocation input associated with the spoken utterances. For example, with the user's prior explicit approval, on-device ASR (i.e., ASR performed locally at the assistant device) may be performed continuously or selectively. For example, it may occur selectively, such as during a certain day and / or time, when the user's presence is detected, when any voice activity is detected, and / or when other conditions are met.

[0006] Further, in these implementations, the ASR text is processed and candidate automated assistant actions corresponding to the command (if any) are generated. For example, the ASR text can be processed locally at the assistant device and the automated assistant actions can be generated using the assistant device's local natural language understanding (NLU) engine. Likewise, this operation is performed even though the user does not provide any explicit invocation input associated with the spoken utterance.

[0007] Still further, for each candidate automated assistant action in any candidate automated assistant action, a determination is made whether to (a) cause automatic execution of the automated assistant action in response to the spoken utterance, or, alternatively, (b) suppress any automatic execution of the automated assistant action in response to the spoken utterance. As described in detail herein, such a determination can be made based on processing both: (i) action features of the candidate automated assistant action; and (ii) environmental features, each environmental feature reflecting a corresponding current value of a corresponding dynamic state of the environment of the assistant device. Likewise, such a determination is performed despite the fact that the user does not provide any explicit invocation input associated with the spoken utterance. Finally, when it is determined that (a) causes automatic execution of the automated assistant action in response to the spoken utterance, such automatic execution is caused to be performed. Likewise, automatic execution occurs despite the fact that the user does not provide any explicit invocation input associated with the spoken utterance.

[0008] Thus, the implementations disclosed herein enable user interaction with an automated assistant to occur without requiring the user to provide any explicit invocation input. This enables reduced user input to be provided by the user, which directly reduces the duration of the interaction and thereby conserves various network resources and / or remote and / or local processing resources that would otherwise be utilized in a prolonged interaction. Furthermore, hot words are often unnaturally prescribed phrases that are awkward to say, and other invocation inputs are equally awkward to perform. This awkwardness may increase the friction of the user-assistant interaction and / or further prolong the interaction.

[0009] While various technical benefits are achieved by interactions with an automated assistant that occur independently of explicit invocation input, there are disadvantages that may result from false positive events in which (a) automatic execution of a candidate automated assistant action is incorrectly determined to be caused in response to a spoken utterance and / or from false negative events in which (b) any automatic execution of a candidate automated assistant action is suppressed in response to a spoken utterance.

[0010] A false positive event may occur in the following circumstances: a user provides a spoken utterance including a valid assistant command and the corresponding assistant action is automatically executed, although the user does not intend the automatic execution of the corresponding assistant action. For example, the user may instead speak the spoken utterance to another concurrent user during a conversation, rather than speaking the spoken utterance to the automated assistant. Additionally or alternatively, a false positive event may occur in the following circumstances: a user provides a spoken utterance including a spoken command corresponding to a plurality of distinct assistant actions, and an unintended action in these actions is executed instead of an intended action in these actions. For example, the spoken utterance "stock performance" may correspond to a first assistant action that renders non-personalized stock information (such as daily quotes for various stock indices), and may also correspond to a second assistant action that renders personalized stock information (such as requesting daily quotes for the user's brokerage account). The first assistant action may be automatically executed in response to the spoken utterance, although the second assistant action is an action intended to be executed in response to the spoken utterance. In the case of a false positive event, the assistant action is executed unnecessarily, resulting in a wasteful utilization of assistant device resources, remote server resources, and / or network resources.

[0011] A false negative event may occur in the following situation: a user provides a verbal utterance that includes a valid assistant command and the corresponding assistant action is not automatically executed, despite the user's intent for automatic execution of the corresponding assistant action. In the case of a false negative event, the user's interaction with the automated assistant is prolonged, and the user may need to provide additional user input (e.g., speak a hot word or provide other explicit invocation input) to actually cause execution of the corresponding assistant action. Furthermore, false negative and false positive events may undermine user confidence in interacting with the automated assistant without explicit invocation input, thereby causing the user to choose to disable such capabilities and prevent the realization of their technical benefits.

[0012] In view of these and other considerations, various implementations disclosed herein attempt to reduce false positive events and / or false negative events. In doing so, these various implementations—when determining whether to (a) cause automatic execution of a candidate automated assistant action in response to a spoken utterance, or, alternatively, (b) suppress any automatic execution of a candidate automated assistant action in response to a spoken utterance—process both: (i) action features of the automated assistant action; and (ii) environmental features. By considering both action features and environmental features as described herein, the accuracy of such determinations is improved, thereby reducing false positive events and / or false negative events. Each of the environmental features reflects a corresponding current value of a corresponding dynamic state of the environment of the assistant device. The action features of the automated assistant action may include a confidence measure for the automated assistant action (e.g., generated by an NLU engine and reflecting the confidence that the automated assistant action does correspond to the ASR text on which it is generated) and / or at least one semantic category of the automated assistant action (e.g., a semantic category covering the intent of the action and other intents).

[0013] As a specific example, a trained machine learning model can be used to process each of the action features and the environmental features of a candidate automated assistant action to generate an output. Based on the output, a determination can be made as to whether to (a) cause automatic execution of the candidate automated assistant action or, alternatively, (b) suppress execution of the candidate automated assistant action. For example, the output can be a probability, and such a determination can be based on the magnitude of the probability. For example, if the probability meets a threshold and / or is a higher probability than other candidate automated assistant actions (if any) considered for the same spoken utterance, it can be determined that only (a) causes automatic execution.

[0014] As another specific example, a human-created rule can be identified based on being indexed in association with a semantic category, which can be one of the action features of a candidate automated assistant action. In other words, the rule can be a rule created for a semantic category (e.g., a first rule for a "media control action" semantic category, a second rule for a "control of an IoT device" semantic category, etc.). Further, environmental features and optionally confidence metrics (which can be action features of a candidate automated assistant action) can be applied to the rule to determine whether the rule is satisfied (in which case the automated assistant action is automatically executed). As a specific example, a given rule may be indexed in association with a "media control action" semantic category and may indicate that the rule is satisfied only when a certain set of one or more conditions exists, such as: (a) the confidence metric action feature > 0.6, and the environmental features indicate that there are 2 or fewer people, and that the corresponding spoken utterance is toward the assistant device; (b) the confidence metric action feature > 0.8, and the environmental features indicate that there are 2 or fewer people, and that the corresponding spoken utterance is away from the assistant device; (c) the confidence metric action feature > 0.85, and the environmental features indicate that there are more than 2 people; and / or other sets of conditions.

[0015] In implementations utilizing rules, the rules may include rules created by a developer of an automated assistant and deployed for use across a large number of assistant devices, and / or may include rules created and / or customized by a corresponding user for the corresponding user's assistant device.

[0016] In an implementation utilizing a trained machine learning model, the trained machine learning model can be trained in various ways. For example, it can be trained based on training instances, each training instance including a training instance input including a corresponding action feature (corresponding to a candidate automated assistant action) and a corresponding environmental feature, and each training instance can include a corresponding training instance output, which is a label indicating whether automatic execution of the candidate automated assistant action should occur in view of the environmental feature and the action feature. For example, if automatic execution should not occur, the training instance output can be "0", and if automatic execution should occur, the training instance output can be "1".

[0017] Such training instances can be generated in various ways. For example, positive training instances or negative training instances can be generated based on corresponding supervisory labels assigned by human readers when consulting environmental features and action features of assistant actions (and / or text corresponding to assistant actions). As another example, positive training instances can be generated in a semi-supervised manner by using environmental features and action features of assistant actions performed in response to verbal utterances after explicit assistant calls. For example, if a user explicitly calls an automated assistant using a hot word, and then provides verbal utterances corresponding to automated assistant actions, positive training instances can be automatically generated using the action features of automated assistant actions and the environmental features when providing verbal utterances. As yet another example, positive training instances or negative training instances can be generated based on the output specified by the rule when action features and / or environmental features are applied to human-created rules. In other words, rules can be applied to generate training instances, and these training instances can be used to train machine learning models, so that machine learning models can efficiently learn rules, while robustness is also higher than the robustness achieved by the rules.

[0018] As mentioned above, each of the (ii) environmental features reflects a corresponding current value of a corresponding dynamic state of the assistant device's environment. The environmental features may be generated based on processing data from the assistant device corresponding to the spoken utterance detected in the audio data, and / or data from other electronic devices in the same environment as the assistant device that detected the corresponding spoken utterance. Various environmental features may be generated, such as time features, spoken utterance source features, person quantity features, user activity features, environmental location features, human-to-device features, ambient light features, and / or other environmental features.

[0019] The time feature indicates one or more current time conditions, such as a time of day, a day of the week, a month of the year, and / or other time conditions. The time feature can be generated by the assistant device, for example, using a local clock and / or a local calendar of the assistant device. As a specific example, a time feature can be generated that indicates a time of day at a discrete level, such as one of "morning", "afternoon", or "evening", or one of "daylight" or "dark". As another specific example, a time feature can be generated that indicates a time of day at a discrete level, such as one of "weekday" or "weekend".

[0020] The spoken utterance source feature indicates an origin location and / or an origin direction of the spoken utterance. The origin location and / or the origin direction can optionally be relative to an assistant device that detected the corresponding spoken utterance (such as the only assistant device that detected the corresponding spoken utterance and / or the assistant device that won assistant arbitration for the corresponding spoken utterance). The origin location can indicate the location of the user who uttered the spoken utterance, and the origin direction can indicate whether the user was facing the assistant device when the spoken utterance was uttered.

[0021] The spoken utterance source feature can be generated by the assistant device, for example, using visual data from a visual component of the assistant device (e.g., a camera) and / or using audio data detected by a microphone of the assistant device (such as audio data corresponding to the spoken utterance). For example, the image from the camera can be processed locally to generate the positioning of the speaking user relative to the assistant device at a discretized level, such as "close", "medium", or "far". For example, the size of the speaking user in the image can be used to generate the positioning of the user. As another example, the image can be processed locally to generate the origin direction of the speaking user relative to the assistant device at a discretized level, such as "toward" or "away". For example, if the face of the speaking user is visible in the image, "toward" can be generated, and if the face of the speaking user is not visible in the image, "away" can be generated. As another example, the assistant device may include a microphone array, and beamforming technology may be used on the audio data corresponding to the spoken utterance to determine the positioning of the speaking user relative to the assistant device at a discretized level, such as "close", "medium", or "far". Typically, beamforming techniques can use the known distances between the microphones of the array and the time differences in detecting the spoken utterance audio data at the microphones of the array to predict the positioning of the speaking user relative to the assistant device. As yet another example, the audio data corresponding to the spoken utterance can be processed to determine the degree of reverberation present in such audio data, and based on the degree, the origin direction of the speaking user relative to the assistant device, such as "toward" or "away", can be determined at a discretized level. For example, if the reverberation level is less than a threshold, "toward" can be generated, and if the reverberation level is greater than the threshold, "away" can be generated. A greater degree of reverberation can indicate that the speaking user is facing away from the assistant device when speaking the spoken utterance, because in this case, the sound from the spoken utterance will be reflected from more surfaces before reaching the microphone of the assistant device.

[0022] The number of people feature indicates the number of people in the environment of the assistant device where the spoken utterance is detected. The number of people feature may indicate an exact number (1, 2, 3, 4, etc.), or may be discretized, such as one of "1", "2-4", or "5 or more". In some implementations, the number of people feature may additionally indicate whether the people in the environment have registered / registered with the assistant device in the environment. For example, the number of people feature may be discretized, such as one of "1", "more than 1 and all registered", or "more than 1 and at least one unregistered".

[0023] The number of people feature can be generated by the assistant device, for example, using visual data from a visual component (e.g., a camera) of the assistant device and / or using audio data detected by a microphone of the assistant device. For example, images captured from a camera before (e.g., within the past 1 minute or other threshold) and / or during a spoken utterance can be processed to determine the number of unique people captured in the image. For example, the number of unique people can be determined corresponding to the number of unique faces captured in the processed image. It can be determined, for example, that a captured face is unique relative to other captured faces by comparing the facial embedding of a given captured face with the embeddings of other captured faces. For example, the cosine distance of the facial embedding to the embeddings of other captured faces can be determined, and if the cosine distances are all greater than a threshold distance, it is determined that the given captured face is unique. A portion of an image in which a face is captured can be processed using a facial recognition machine learning model or other image encoding machine learning model to generate a facial embedding for the captured face. In an implementation of generating a person count feature indicating whether a person in an environment is registered, a facial embedding generated for a unique face may be compared to a locally stored registered embedding, and based on the comparison, it may be determined whether the unique face is registered or unregistered. For example, if a cosine distance between the facial embedding of the unique face and the locally stored registered embedding is less than a threshold distance, it may be determined that the unique face is registered.

[0024] As another example, audio data detected before (e.g., within the past 1 minute or other threshold) and / or during a spoken utterance may be processed to determine the number of unique human voices captured in the audio data. For example, the number of unique persons may be determined corresponding to the number of unique human voices detected based on the processed audio data. As an example, each of the segments of the audio data containing voice activity may be processed using a text-independent speaker identification (TISID) model or other speech coding model to generate corresponding voice embeddings for the audio data. The corresponding voice embeddings may then be clustered based on the distances of the corresponding voice embeddings in the embedding space. For example, the voice embeddings may be clustered together based on two or more voice embeddings being close to each other in terms of distance in the embedding space, thereby indicating that the corresponding utterances on which they are based are all from the same human speaker. Further, a given cluster may be determined to be unique relative to an additional cluster based on the fact that the embeddings of the given cluster and the additional cluster are at least a threshold distance from each other in the embedding space. The number of unique persons may then be determined based on the number of unique clusters. In an implementation of generating a feature indicating whether a person in an environment is registered, a voice embedding generated for a unique voice may be compared with a locally stored registered embedding, and based on the comparison, it may be determined whether the unique voice is registered or unregistered. For example, if a cosine distance between the voice embedding for the unique face and the locally stored registered embedding is less than a threshold distance, it may be determined that the unique voice is registered.

[0025] In some implementations, when determining the number of unique human voices, the candidate human voices detected in the segment of the audio data can be filtered out from the number of unique human voices included based on determining that the corresponding segment of the audio data indicates that the candidate human voice originates from the speaker component of the electronic device rather than from a human speaker present in the environment. For example, if a television is turned on near the assistant device, the candidate human voice can be detected based on the output from the speaker component of the television (e.g., when an actor is speaking during the rendering of a television program on the television). In some of these implementations, it can be determined that the candidate human voice originates from the speaker component based on an analysis of the frequency of the segment of the audio data corresponding to the segment spoken by the candidate human voice. For example, if all or a threshold percentage (e.g., 80%, 90%, or other threshold percentage) of the frequencies are all within a certain range of each other, all below a certain upper threshold, and / or all above a certain lower threshold, it can be determined that the candidate human voice originates from the speaker component. For example, if all or a threshold percentage of the frequencies are within 3,000 Hz of each other and / or all below 3,500 Hz, it can be determined that the candidate human voice originates from the speaker component. These and / or other frequency analysis techniques may be utilized because the frequencies generated by the speaker assembly will be more constrained and / or otherwise objectively different than the frequencies generated by human speakers that are actually present simultaneously in the environment. Thus, in these and other ways, speech from televisions and / or other non-existent human speakers is not erroneously included in the number of people feature, thereby improving the usefulness of the number of people feature in the determinations disclosed herein.

[0026] The user activity feature indicates one or more activities in which the user is currently engaged. The user activity feature can be generated by the assistant device, for example, using local sensor data of the assistant device and / or calendar entries stored locally by the assistant device. As a specific example, a user activity feature indicating "preparing for dinner" can be generated based on sensing of ambient sounds indicating that dinner is being prepared (e.g., banging pots, stirring sounds, etc.). As another specific example, a user activity feature indicating "party" can be generated based on calendar entries indicating that a party is currently being held and / or based on sensing of ambient sounds indicating a party. As another specific example, a user activity feature of "exercising" can be generated based on accelerometer and / or other sensor readings from the assistant device and / or an additional assistant device worn by the user.

[0027] The environmental location feature indicates one or more semantic classifications of the environment. The environmental location feature can be optionally generated, for example, based on location data from the assistant device and with reference to one or more databases that associate the location data with semantic classifications. For example, an environmental location feature can be generated that indicates a semantic classification at a discretization level, such as one of "private location" or "public location", or one of "home" or "away from home", or at a more fine-grained discretization level, such as one of "train station", "restaurant", "shopping mall", "home", "work", etc.

[0028] The human-to-device feature indicates the physical context between a mobile, battery-powered assistant device (e.g., mobile phone, watch) and the user. For example, the human-to-device feature may indicate whether the mobile assistant device is "in a pocket of the user", "being held by the user but at a distance", "being held by the user close to the user's face", or "is placed on a surface".

[0029] Various example environmental features and their corresponding example discretizations have been described in detail above. However, additional or alternative environmental features and / or discretizations may be utilized in various implementations disclosed herein.

[0030] It is noted that various implementations disclosed herein can be used to reduce the time required to obtain a response / fulfillment from an automated assistant. This is particularly because such implementations can eliminate the need for a user to provide an explicit invocation to the automated assistant (such as by speaking a hot word / phrase or performing a specific user input) before speaking a command or query. Further, in certain implementations, the automated assistant can receive, understand, and in some cases respond to / fulfill commands or queries without communicating with a server, thereby further reducing the time it can provide a response / fulfillment.

[0031] The implementation disclosed herein relates to and / or can be used in conjunction with an assistant device, which includes at least one or more microphones and an automated assistant application. The automated assistant application can be installed "on top" of the operating system of the assistant device, and / or the automated assistant application itself can form part (or the whole) of the operating system of the assistant device. The automated assistant application includes and / or can access the on-device ASR, on-device NLU and / or on-device execution. For example, the device ASR can be performed using an on-device speech recognition module, which uses an end-to-end speech recognition machine learning model stored locally at the assistant device to process audio data (detected by the microphone). The on-device ASR generates ASR text recognized for spoken words (if any) present in the audio data. In addition, for example, the on-device natural language understanding (NLU) module can be used to perform the on-device NLU, which processes the recognized ASR text generated using the on-device speech recognition and optionally the context data to generate NLU data. The NLU data may include the intent corresponding to the spoken words and optionally the parameters (e.g., slot values) for the intent. The NLU data may also optionally include a corresponding confidence value for each of the intents (and optional slot values) that reflects the likelihood that the intent (and optional slot value) is intended by the corresponding ASR text (as determined by the NLU module). On-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine an action to be taken to resolve the intent (and optionally parameters for the intent) of the spoken utterance. This may include determining a local and / or remote response (e.g., answer) to the spoken utterance, an interaction with a locally installed application to be performed based on the spoken utterance, a command to be sent to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other resolution actions to be performed based on the spoken utterance. The on-device fulfillment may then initiate local and / or remote execution / execution of the determined actions to resolve the spoken utterance.

[0032] In various implementations, remote ASR, remote NLU, and / or remote fulfillment may be at least selectively utilized. For example, the recognized ASR text may be at least selectively sent to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, the recognized ASR text may be optionally sent for remote execution in parallel with on-device execution or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device ASR, on-device NLU, on-device fulfillment, and / or on-device execution may optionally be prioritized at least due to the improved data security and / or reduced latency they provide when resolving spoken utterances (due to the fact that no client-server round trip is required to resolve spoken utterances). Further, on-device functionality may be the only functionality available in situations where there is no network connection or limited network connection.

[0033] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of these implementations and other implementations are described in more detail below.

[0034] Some implementations disclosed herein include one or more computing devices that include one or more processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or a tensor processing unit (TPU). One or more of the processors are operable to execute instructions stored in an associated memory, and the instructions are configured to cause the execution of any of the methods described herein. The computing device may include, for example, a client assistant device having a microphone, at least one display, and / or other sensor components. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions that can be executed by one or more processors to perform any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0036] Figure 2A , Figure 2B , Figure 2C and Figure 2D Each depicts an example process flow demonstrating various aspects of the disclosure, according to various implementations.

[0037] Figure 3 A flowchart illustrating an example method according to implementations disclosed herein is depicted.

[0038] Figure 4 Depicted Figure 3An example of an optional sub-box of box 308.

[0039] Figure 5 An example architecture for a computing device is illustrated. DETAILED DESCRIPTION

[0040] Figure 1 An example environment is shown in which the techniques disclosed herein may be implemented. The example environment includes a client device 101 that selectively executes at least an automated assistant client 110. The term "assistant device" is also used herein to refer to a client device 101 that selectively executes at least an automated assistant client 110. One or more cloud-based automated assistant components 130 may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the client device 101 via one or more local area networks and / or wide area networks (e.g., the Internet) indicated generally at 100. The cloud-based automated assistant components 130 may be implemented, for example, via a cluster of high-performance servers.

[0041] In various implementations, an instance of automated assistant client 110, by virtue of its interaction with one or more cloud-based automated assistant components 130, may form what appears to be a logical instance of automated assistant 120 from the user's perspective, through which the user may engage in human-computer interaction (e.g., verbal interaction). Figure 1 An example of such an automated assistant 120 is depicted in dashed lines in FIG. In some implementations and / or in some human-computer interactions, the automated assistant client 110 may perform some or all automated assistant actions locally without any interaction with the cloud-based automated assistant component 130.

[0042] Client device 101 may be, for example, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which may include a visual sensor in some cases), a smart appliance such as a smart TV (or a standard TV equipped with a network-connected dongle with automated assistant capabilities), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. As previously noted, client device 101 may optionally take the form of an assistant device that is primarily designed to facilitate interaction between a user and automated assistant 120 (e.g., a stand-alone interactive device with a speaker, microphone, camera (and / or other visual components), and display).

[0043] The client device 101 may be equipped with one or more visual components 107 having one or more fields of view. The visual component 107 may take various forms, such as a monocular camera, a stereo camera, a lidar component, a radar component, etc. The one or more visual components 107 may be used to capture visual frames (e.g., image frames (still images or videos)) of the environment in which the client device 101 is deployed. The client device 101 may also be equipped with one or more microphones 109.

[0044] Client device 101 may also include one or more presence sensors 105 and / or one or more displays 103 (e.g., touch-sensitive displays). In various implementations, display 103 may be one of the user interface output components through which user interface output is rendered. Display 103 may further be one of the user interface output components through which the visual portion of the response from automated assistant client 110 is rendered. Client device 101 may also include one or more speaker components (not shown) through which the audible portion of the response from automated assistant client 110 is rendered. Presence sensor 105 may include, for example, a PIR and / or other passive presence sensor.

[0045] Automated assistant client 110 is illustrated as including an on-device VAD engine, an on-device ASR engine 122, an on-device NLU engine 123, an on-device semantic classification engine 125, an on-device environmental feature engine 126, an on-device determination engine 127, and an on-device fulfillment engine 128. In various implementations, automated assistant client 110 may include additional engines, which are not shown or described herein for the sake of brevity. Further, in some additional or alternative implementations, one or more on-device engines, such as on-device VAD engine 121, may be omitted.

[0046] The VAD engine 121 may at least selectively process audio data captured via the microphone 109 to detect whether the processed audio data includes voice activity. For example, the VAD engine 121 may process the audio data using the VAD model 151 to generate an output indicating a probability of capturing voice activity in the processed audio data. If the probability satisfies a threshold, the VAD engine 121 may determine that voice activity is included in the processed audio data.

[0047] The on-device ASR engine 122 processes the audio data captured via the microphone 109 to generate ASR text, which is a predictive text recognition of any spoken utterance included in the audio data. In generating the ASR text, the on-device ASR engine 122 may process the audio data using one or more on-device ASR models 152. The on-device ASR model may be a neural network model, such as a transformer model or other neural network model that enables real-time streaming generation of ASR text.

[0048] As described herein, the on-device ASR engine 122 may process audio data captured via the microphone 109, at least selectively, independently of any explicit user invocation of the automated assistant client 110. For example, the on-device ASR engine 122 may continuously process the audio data captured via the microphone 109. As another example, the on-device ASR engine 122 may process the audio data in response to the presence sensor 105 and / or the visual component 107 indicating that one or more humans are present near the client device 101. For example, the on-device ASR engine 122 may process the audio data in response to detecting human presence at any time within the past thirty seconds or other threshold time period. As another example, the on-device ASR engine 122 may process the audio data in response to the VAD engine 121 indicating the presence of voice activity. For example, if the VAD engine 121 detects the presence of voice activity, the buffered audio data in which the voice activity was detected may be provided to the on-device ASR engine 122 for processing, along with subsequent streamlined audio data for at least a certain amount of time after the detection.

[0049] The on-device NLU engine 123 processes the ASR text generated by the on-device ASR engine 122 to generate candidate automated assistant actions corresponding to the ASR text, and optionally generates corresponding confidence metrics for each of the automated assistant actions. When generating candidate automated assistant actions and / or confidence metrics, the on-device NLU engine 123 can utilize one or more NLU models 153. The NLU model 153 may include a machine learning model that can be used to process the text and generate predicted intents, and / or optionally generate parameters (e.g., slot values) corresponding to the text of the intent. The intent and optional parameters can jointly define the candidate automated assistant actions. The NLU model 153 can additionally or alternatively include a grammar that includes a mapping between text phrases (and / or regular expressions or other representations of corresponding text phrases) and automated assistant actions corresponding to the text phrases. For example, "what's on my calendar" can be grammatically mapped to an assistant action of "render calendar entries for today's date". The grammatical mappings may also be stored in association with confidence metrics, each confidence metric reflecting the confidence that a corresponding assistant action is a correct action for the corresponding text phrase (and / or regular expression) of the mapping.

[0050] The on-device semantic classification engine 125 selects at least one corresponding semantic category for each candidate automated assistant action in the candidate automated assistant actions generated by the NLU engine 123. For a given candidate automated assistant action, the NLU engine 123 may select a subset of semantic categories from a superset of candidate semantic categories. In some implementations, the candidate semantic categories of the superset include or are limited to genus categories that each cover multiple distinct intents. For example, the "control of IoT device" semantic category may cover multiple distinct, finer-grained intents, such as "turn on", "turn off", "dim", "adjust", etc. As another example, the "render potentially sensitive content" semantic category may cover multiple distinct intents, such as "render email", "render calendar entry", "place call", etc. In selecting at least one semantic category for a candidate assistant action, the on-device semantic classification engine 125 may utilize a semantic model 152. The semantic model 155 may include, for example, a mapping between intents and corresponding semantic categories. For example, each of the plurality of fine-grained IoT device control intents may be mapped to a "control of IoT device" semantic category. The semantic model 155 may additionally or alternatively include a machine learning classifier that may be used to process the intents and / or parameters of the candidate automated assistant actions and generate an output indicating one or more semantic categories that should be selected for the candidate automated assistant actions.

[0051] The on-device environment feature engine 126 generates environment features based on processing data from the client device 101 and / or from other devices that are in network communication with the client device 101 and are in the same environment as the client device 101. The on-device environment feature engine 126 can generate various, such as time features, verbal speech source features, number of people features, user activity features, environmental location features, human-to-device features, ambient light features, and / or other environmental features. In some implementations, the on-device environment feature engine 126 generates updated environment features at regular intervals or irregular intervals. In some implementations, the on-device environment feature engine 126 generates updated environment features in response to the ASR engine 122 generating recognized ASR text and / or the NLU engine 123 generating candidate automated assistant actions.

[0052] When generating environmental features, the on-device environmental feature engine 126 can utilize the environmental feature model 156. For example, the environmental feature model 156 can include the TISID model described herein, and the TISID model can be used by the on-device environmental feature engine 126 when generating a number of people feature. As another example, the model 156 can include a model that can process sensor data from a sensor of the client device 101 to predict user activity, and such a model can be used by the on-device environmental feature engine 126 when generating a user activity feature.

[0053] For each candidate automated assistant action determined by the on-device NLU engine 123 for the ASR text of the spoken utterance, the on-device determination engine 127 determines whether to (a) cause automatic execution of the candidate automated assistant action in response to the spoken utterance, or, alternatively, (b) suppress any automatic execution of the candidate automated assistant action in response to the spoken utterance. The on-device determination engine 127 makes this determination based on both (i) action features of the automated assistant action (e.g., a confidence measure determined by the on-device NLU engine 123 and / or at least one semantic category determined by the on-device semantic classification engine 125), and (ii) environmental features determined by the on-device environmental features engine 126. Further, in making the determination, the on-device determination engine 127 may utilize one or more determination models 157. For example, the determination model 157 may include a trained machine learning model, and the trained machine learning model may be used to process each of the action features and environmental features of the candidate automated assistant action to generate an output, and make a determination based on the output. As another example, the determination model may additionally or alternatively include rules, and action features of the candidate automated assistant actions as well as environmental features may be applied to the rules in making the determination.

[0054] The on-device fulfillment engine 128 can be used to cause the automatic execution of the candidate automated assistant action when the on-device determination engine 127 determines to cause the automatic execution of the candidate automated assistant action. For example, the on-device fulfillment engine 128 can initiate the on-device execution and / or remote execution of the candidate automated assistant action. For example, when the candidate automated assistant action is a control of an IoT device, the on-device fulfillment engine 128 can send the corresponding control command directly to the IoT device through the local network, or send the command to a remote server, which will then send the corresponding control command to the IoT device. In addition, for example, the on-device fulfillment engine 128 can interact with the cloud-based automated assistant component 130 to obtain a response to the candidate automated assistant action, and then cause an auditory and / or visual rendering of the response.

[0055] The cloud-based automated assistant component 130 may include one or more engines that supplement any on-device engine counterparts of the automated assistant client 110 and / or are provided in place of any local engine counterparts. For example, the cloud-based automated assistant component may include a cloud-based ASR, NLU, and / or fulfillment engine. For example, the cloud-based automated assistant component 130 may include a cloud-based fulfillment engine that may be utilized in place of the on-device fulfillment engine 128, or when the on-device fulfillment engine 128 is not able to perform automated assistant actions locally. In addition, for example, the cloud-based automated assistant component 130 may include a cloud-based NLU engine that may be utilized in place of the on-device NLU engine 123, or when the on-device NLU engine 123 is not able to generate any candidate automated assistant actions locally based on ASR text.

[0056] Figure 2A , Figure 2B , Figure 2C and Figure 2D Each depicts an example process flow demonstrating various aspects of the disclosure, according to various implementations.

[0057] First go to Figure 2A , audio data 191A is detected via microphone 109 of client device 101 and processed by on-device ASR engine 122, optionally in response to VAD engine 121 processing the audio data and determining that the audio data includes voice activity. Audio data 191A captures the spoken sentence "what's on my calendar". On-device ASR engine 122 processes audio data 191A to generate ASR text 192A for "what's on my calendar".

[0058] The on-device NLU engine 123 processes the ASR text 192A to generate a single candidate automated assistant action 193A of "render today's calendar entries", along with a confidence measure of 0.9. The on-device semantic classification engine 125 processes the candidate automated assistant action 193A to generate a semantic category 194A of "render potentially sensitive content".

[0059] The on-device environment feature engine 115 also generates current environment features 195A, which include a personnel quantity feature of "2", a time feature of "weeknight", a user activity feature of "cooking", and a verbal discourse source feature of "directed toward".

[0060] On-device determination engine 127 processes context features 195A and confidence measure (0.9) and / or semantic category 194A of candidate automated assistant action 193A to generate output 196 A. On-device determination engine 127 performs processing using, for example, machine learning models and / or rules as described herein.

[0061] On-device determination engine 127 determines to cause automatic execution of candidate automated assistant action 193A based on output 196A. For example, output 196A may indicate that a rule is satisfied and / or that the machine learning output satisfies a threshold. Accordingly, on-device determination engine 127 provides command 197A to fulfillment engine 128, causing fulfillment engine 128 to automatically cause execution of candidate automated assistant action 193A.

[0062] Now turn to Figure 2B , audio data 191B is detected via microphone 109 of client device 101 and processed by on-device ASR engine 122, which is optionally responsive to VAD engine 121 processing the audio data and determining that the audio data includes voice activity. Audio data 191B captures the spoken utterance "what's on my calendar" that is consistent with Figure 2A The on-device ASR engine 122 processes the audio data 191B to generate ASR text 192B of “what's on my calendar”, which is the same as the spoken utterance of Figure 2A The same as ASR text 192A.

[0063] The on-device NLU engine 123 processes the ASR text 192B to generate a single candidate automated assistant action 193B of "render today's calendar entries", along with a confidence measure of 0.9, which is consistent with Figure 2A The on-device semantic classification engine 125 processes candidate automated assistant action 193B to generate a semantic category 194B of “render potentially sensitive content” that is identical to candidate automated assistant action 193A. Figure 2A Same as semantic category 194A.

[0064] The on-device environment characteristic engine 115 also generates a current environment characteristic 195B, which is Figure 2A For example, environment feature 195B includes a number of people feature of "> 5", a time feature of "weekend night", a user activity feature of "party", and a verbal speech source feature of "directed away from". Similarly, the values ​​of these features in environment feature 195B are different from those in environment feature 195A. Figure 2A The environmental feature 195A is different from its counterpart.

[0065] On-device determination engine 127 processes context features 195B and confidence measure (0.9) and / or semantic category 194B of candidate automated assistant action 193B to generate output 196B, which is similar to Figure 2A The output 196A is different. The on-device determination engine 127 performs processing using, for example, the machine learning models and / or rules described herein.

[0066] On-device determination engine 127 determines to suppress automatic execution of candidate automated assistant action 193B based on output 196B. For example, output 196B may indicate that no rule is satisfied and / or the machine learning output fails to satisfy a threshold. Therefore, on-device determination engine 127 does not provide any commands to fulfillment engine 128, and more specifically, does not take further action in response to the spoken utterance captured in audio data 191B.

[0067] therefore, Figure 2A and Figure 2B Collectively, it is illustrated that an action corresponding to the spoken utterance of "what's on my calendar" will be automatically performed under some environmental conditions, but not automatically performed under other environmental conditions.

[0068] Now turn to Figure 2C , audio data 191C is detected via microphone 109 of client device 101 and processed by on-device ASR engine 122, which is optionally in response to VAD engine 121 processing the audio data and determining that the audio data includes voice activity. Audio data 191C captures the spoken word "dim". On-device ASR engine 122 processes audio data 191C to generate ASR text 192C for "dim".

[0069] The on-device NLU engine 123 processes the ASR text 192C to generate two candidate automated assistant actions and corresponding confidence measures. That is, a first candidate action 193C1 of “dim assistant device screen”, together with a confidence measure of 0.8; and a second candidate action 193C2 of “dim lights”, together with a confidence measure of 0.9. The on-device semantic classification engine 125 processes the candidate automated assistant action 193C1 to generate a semantic category 194C1 of “control of client device output rendering”. The on-device semantic classification engine 125 also processes the candidate automated assistant action 193C2 to generate a semantic category 194C2 of “control of IoT device”.

[0070] The on-device environmental feature engine 115 also generates current environmental features 195C, which include spoken utterance source features of "faraway" and "directed away", a time feature of "night", and an ambient light sensor feature of "bright".

[0071] On-device determination engine 127 processes environmental features 195C and the confidence measure (0.8) and / or semantic category 194C1 of first candidate automated assistant action 193C1 to generate output 196C1. On-device determination engine 127 also processes environmental features 195C and the confidence measure (0.7) and / or semantic category 194C2 of second candidate automated assistant action 193C2 to generate output 196C2. On-device determination engine 127 performs processing using, for example, machine learning models and / or rules as described herein.

[0072] On-device determination engine 127 determines to cause automatic execution of first candidate automated assistant action 193C1, and to suppress automatic execution of second candidate automated assistant action 193C2, based on outputs 196C1 and 196C2. For example, output 196C1 may indicate that a rule is satisfied and / or a machine learning output satisfies a threshold value—while output 196C2 may fail to indicate that any rule is satisfied and / or a machine learning output satisfies a threshold value. For example, the machine learning output in output 196C1 may indicate a higher probability than the machine learning output in output 196C2. Therefore, on-device determination engine 127 provides command 197C to fulfillment engine 128, so that fulfillment engine 128 automatically causes execution of first candidate automated assistant action 193C1.

[0073] Now turn to Figure 2D , audio data 191D is detected via microphone 109 of client device 101 and processed by on-device ASR engine 122, which is optionally in response to VAD engine 121 processing the audio data and determining that the audio data includes voice activity. Audio data 191D captures the spoken word "dim", which is consistent with Figure 2C The on-device ASR engine 122 processes the audio data 191D to generate ASR text 192D for “dim”, which is identical to the spoken word. Figure 2C The ASR text is the same as 192C.

[0074] The on-device NLU engine 123 processes the ASR text 192D to generate two candidate automated assistant actions and corresponding confidence measures. That is, a first candidate action 193D1 of “dim assistant device screen”, with a confidence measure of 0.8; and a second candidate action 193D2 of “dim lights”, with a confidence measure of 0.9. The first candidate action 193D1 and the second candidate action 193D2 and the corresponding confidence measures are compared with the corresponding confidence measures in the Figure 2C The counterparts in (193C1 and 193C2) are the same.

[0075] On-device semantic classification engine 125 processes candidate automated assistant action 193D1 to generate a semantic category 194D1 of “control of client device output rendering” that is consistent with Figure 2C The on-device semantic classification engine 125 also processes the candidate automated assistant action 193D2 to generate a semantic category 194D2 of “control of IoT device”, which is the same as Figure 2C The semantic category is the same as 194C2.

[0076] The on-device environment characteristic engine 115 also generates a current environment characteristic 195D, which is Figure 2C Environmental feature 195D includes the spoken utterance source features of “close” and “directed toward”, the time feature of “day”, and the ambient light sensor feature of “medium”. Similarly, the values ​​of these features in environmental feature 195D are different from those in Figure 2C The environmental characteristics of 195C are different from their counterparts.

[0077] On-device determination engine 127 processes environmental features 195D and the confidence measure (0.8) and / or semantic category 194D1 of first candidate automated assistant action 193D1 to generate output 196D1. On-device determination engine 127 also processes environmental features 195D and the confidence measure (0.7) and / or semantic category 194D2 of second candidate automated assistant action 193D2 to generate output 196D2. On-device determination engine 127 performs processing using, for example, machine learning models and / or rules as described herein.

[0078] On-device determination engine 127 determines to cause automatic execution of second candidate automated assistant action 193D2 based on outputs 196D1 and 196D2, and to suppress execution of first candidate automated assistant action 193D1. For example, output 196D2 may indicate that a rule is satisfied and / or the machine learning output satisfies a threshold value—while output 196D1 may fail to indicate that any rule is satisfied and / or the machine learning output satisfies a threshold value. For example, the machine learning output in output 196D2 may indicate a higher probability than the machine learning output in output 196D1. Therefore, on-device determination engine 127 provides command 197D to fulfillment engine 128, so that fulfillment engine 128 automatically causes execution of second candidate automated assistant action 193D1.

[0079] therefore, Figure 2C and Figure 2D It is collectively illustrated that under some environmental conditions, a first action corresponding to the spoken utterance of "dim" will be automatically performed, while a second action corresponding to "dim" will be suppressed, while under other environmental conditions, the second action will be automatically performed, while the first action is suppressed. Although not illustrated, it is noted that there may be other environmental conditions where both the first action and the second action will be suppressed, and no action is automatically performed in response to the spoken utterance of "dim." For example, when these environmental conditions exist, the output generated by the determination engine 127 for each of the first action and the second action may fail to indicate that any rule is satisfied and / or fail to indicate that the machine learning output satisfies a threshold.

[0080] Figure 3 A flowchart illustrating an example method 300 according to implementations disclosed herein is depicted. For convenience, the operations of the method 300 are described with reference to a system performing the operations. This system may include various components of various computer systems, such as a client device (e.g., Figure 1 In addition, although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0081] At block 302, the system performs ASR on the audio data independently of any explicit assistant invocation to generate ASR text that predicts spoken utterances captured in the audio data. For example, the audio data may be detected via a microphone of the assistant device and may be processed by an on-device ASR engine of the assistant device to generate ASR text.

[0082] At block 304, the system processes the ASR text (from block 302) to generate candidate automated assistant actions that each correspond to the ASR text. For example, the system can use an on-device NLU engine to process the ASR text to generate one or more candidate automated assistant actions that each correspond to the ASR text. The system can utilize one or more grammars and / or one or more machine learning models in generating the candidate automated assistant actions. Note that for some ASR text, the system will not be able to generate any candidate automated assistant actions that correspond to the ASR text. These situations are addressed in block 305 below.

[0083] Box 304 optionally includes subbox 304A, in which the system generates a corresponding confidence metric for each candidate automated assistant action in the candidate automated assistant actions. For example, the NLU engine can generate a confidence metric for each candidate automated assistant action in the candidate automated assistant actions. For example, when a machine learning model is used to process ASR text and predict candidate automated assistant actions, the machine learning model can also generate a confidence metric for the prediction, and the confidence metric is used for the candidate automated assistant actions. In addition, for example, when a mapping of a grammar is used to process ASR text and predict candidate automated assistant actions, the mapping can have an associated confidence metric stored therewith, and the confidence metric is used for the candidate automated assistant actions.

[0084] At block 305, the system determines whether any candidate automated assistant actions were generated at block 304. If so, the system proceeds to block 306. If not, the system returns to block 302 to perform ASR on further audio data that captures the spoken utterance. For example, if the ASR text is a single word "patent," no candidate automated assistant actions may be determined for the ASR text at block 304. Therefore, at block 305, the system may decide to return to block 302.

[0085] At optional block 306, the system selects at least one corresponding semantic category for the candidate automated assistant action for each of the candidate automated assistant actions determined at block 304. The at least one corresponding semantic category selected for the candidate automated assistant action can be a subset of the candidate semantic categories. Further, the at least one corresponding semantic category can optionally be a generic category covering a plurality of distinct intents, including the intent of the candidate automated assistant action.

[0086] Optional block 306 optionally includes subblock 306A, in which the system selects a semantic category for the candidate automated assistant action based on a mapping of the intent of the candidate automated assistant action to the semantic category. For example, the semantic category can be mapped to multiple intents, including the intent of the candidate automated assistant action.

[0087] At block 308, the system generates environmental features based on processing data from a device of the environment in which the audio data of block 302 was captured. For example, the data may be from at least the device via which the audio data of block 302 was captured. The environmental features each reflect a corresponding current value of a corresponding dynamic state of the environment. Various environmental features may be generated, such as time features, spoken utterance source features, number of people features, user activity features, environmental location features, human-to-device features, ambient light features, and / or other environmental features. In some implementations, block 308 includes Figure 4 One or more of sub-blocks 308A-308F illustrated in and described in more detail below.

[0088] At block 310, the system generates a corresponding output for each of the candidate automated assistant actions determined at block 304 based on processing the context features generated in block 308, and a confidence measure for the action (optionally generated in sub-block 304A) and / or at least one semantic category for the action (optionally generated in block 306). In some implementations, block 310 includes sub-block 310A and / or sub-block 310B.

[0089] At sub-box 310A, the system uses a trained machine learning model to process environmental features and action features when generating an output for a candidate automated assistant action. For example, the system can use a trained machine learning model to process environmental features and action features to generate an output indicating a probability. The trained machine learning model can be trained in a variety of ways. For example, it can be trained based on training instances, each training instance including a training instance input, the training instance input including a corresponding action feature (corresponding to a candidate automated assistant action) and a corresponding environmental feature, and each training instance can include a corresponding training instance output, the corresponding training instance output being a label indicating whether automatic execution of the candidate automated assistant action should occur in view of the environmental features and the action features. For example, if automatic execution should not occur, the training instance output can be "0", and if automatic execution should occur, the training instance output can be "1". Such training instances can be generated in various ways. For example, a positive training instance or a negative training instance can be generated based on a corresponding supervisory label assigned by a human reviewer when reviewing environmental features and action features of an assistant action (and / or text corresponding to an assistant action). As another example, positive training instances can be generated in a semi-supervised manner by using environmental features and action features of assistant actions performed in response to spoken utterances following explicit assistant invocations. As yet another example, positive or negative training instances can be generated based on outputs specified by a human-created rule when action features and / or environmental features are applied to the rule.

[0090] At sub-box 310B, the system uses one or more rules to process environmental features and action features when generating output for candidate automated assistant actions. For example, the system can use rules to process environmental features and action features to generate output indicating whether any rule in the rule is satisfied. For example, the rule can be identified by being indexed in association with the semantic category generated in optional box 306 of the candidate automated assistant action based on human-created rules. Further, the environmental features generated at box 308 and the confidence measure for the candidate automated assistant action generated in optional sub-box 304A can be applied to the rule. As a specific example, a given rule can be indexed in association with the "obtain general information" semantic category, and can indicate that the rule is satisfied only when there is a certain one or more condition sets, such as: (a) confidence measure action feature> 0.9, and environmental features indicate that there are 2 or more people, and indicate that the corresponding spoken utterance is toward the assistant device; and (b) confidence measure action feature> 0.8, and environmental features indicate that there is only 1 person; and / or other condition sets.

[0091] At block 312, the system determines whether to (a) cause the automatic execution of one of the candidate automated assistant actions, or (b) suppress the automatic execution of all candidate automated assistant actions based on the output of block 310. For example, if the machine learning outputs for all candidate automated assistant actions fail to meet the threshold, and / or if the rule outputs for all candidate automated assistant actions fail to indicate that the rule is satisfied, the system can determine to suppress. As another example, if there is only a single candidate automated assistant action, the ML output of the single candidate automated assistant action meets the threshold and / or the single candidate automated assistant action meets the rule, it can be determined to cause the automatic execution of the single candidate automated assistant action. As yet another example, if there are multiple candidate automated assistant actions, the ML outputs of the multiple candidate automated assistant actions meet the threshold, it can be determined to cause the automatic execution of the candidate automated assistant action whose ML output has the highest probability. As yet another example, if there are multiple candidate automated assistant actions, the rule outputs of the multiple candidate automated assistant actions indicate that the rule is satisfied, it can be determined to cause the automatic execution of the candidate automated assistant action with the highest confidence measure (generated in optional block 304A).

[0092] Now go to Figure 4 , exemplifies Figure 3 Optional sub-block of block 308 of method 300. Figure 3 When performing block 308 of method 300, one or more of the sub-blocks may be optionally performed. Figure 4 Each sub-box in the sub-box describes the generation of the corresponding environmental features.

[0093] At subframe 308A, the system generates a time feature indicating the current time condition. The system can, for example, generate the time feature using a local clock and / or a local calendar. As a specific example, a time feature can be generated that indicates the time of day with a discretization level, such as one of "weekday morning", "weekend morning", "weekday afternoon", "weekend afternoon", "weekend evening" or "weekend evening".

[0094] At sub-block 308B, the system generates an environmental location feature that indicates a semantic classification of the environment. The system can generate the environmental location feature based on, for example, location data from an assistant device where the audio is detected and optionally with reference to one or more databases that associate location data with semantic classifications. For example, an environmental location feature can be generated that indicates a semantic classification at a discretized level, such as one of "private location" or "public location".

[0095] At subblock 308C, the system generates a spoken utterance feature indicating an origin location and / or an origin direction of the spoken utterance. The origin location may indicate a location of a user who uttered the spoken utterance (e.g., relative to a location of an assistant device at which the audio data was detected), while the origin direction may indicate whether the user was facing toward the assistant device when the spoken utterance was uttered.

[0096] Sub-block 308C may optionally include further sub-block 308C1 and / or further sub-block 308C2.

[0097] At further sub-block 308C1, the system generates a spoken utterance feature based on the visual data. For example, the system processes the image from the assistant device's camera to generate a position of the speaking user relative to the assistant device at a discretized level, such as "close", "medium", or "far". As another example, the system processes the image to additionally or alternatively generate an origin direction of the speaking user relative to the assistant device at a discretized level, such as "toward" or "away".

[0098] At further sub-box 308C2, the system generates spoken utterance features based on the audio data and using multiple microphones and beamforming and / or reverberation level. For example, when using multiple microphones and beamforming, the system can use the known distances between the multiple microphones of the assistant device and the time differences in detecting spoken utterances in the audio data at the microphones to predict the location of the speaking user relative to the assistant device. In addition, for example, when using the reverberation level, the system can process the audio data to determine the level of reverberation present in the audio data and the origin direction of the speaking user relative to the assistant device, the origin direction being determined based on the level.

[0099] At sub-block 308D, the system generates a user activity feature that indicates one or more activities in which the user of the environment is currently engaged. Sub-block 308D may optionally include further sub-block 308D1 and / or further sub-block 308D2. In further sub-block 308D1, the system generates the user activity feature based on visual data. At further sub-block 308D2, the system generates the user activity feature using audio data and / or other non-visual sensor data. For example, the system may generate a user activity feature indicating "preparing for dinner" based on sensing of ambient sounds indicating preparation for dinner.

[0100] At subblock 308E, the system generates a people count feature that indicates the number of people in the environment and, optionally, the number of registered people in the environment and / or the number of unregistered people in the environment. A person may be registered with the assistant device if the person has an account linked to the assistant device and / or if the person has stored the person's facial and / or voice embeddings locally at the assistant device as registered embeddings.

[0101] Sub-block 308E may optionally include further sub-block 308E1 and / or further sub-block 308E2.

[0102] In further sub-box 308E1, the system generates a number of people feature based on the visual data. For example, the system can process images captured from a camera of the assistant device before (e.g., within the past 2 minutes or other threshold) and / or during a spoken utterance to determine the number of unique people captured in the image. For example, the number of unique people can be determined corresponding to the number of unique faces captured in the processed image. It can be determined, for example, by comparing the facial embedding of a given captured face with the embeddings of other captured faces to determine whether the captured face is unique relative to other captured faces. In an implementation of generating a number of people feature indicating whether a person in an environment is registered, the facial embedding generated for a unique face can be compared with a locally stored registered embedding, and based on the comparison, it can be determined whether the unique face is registered or unregistered.

[0103] At further sub-block 308E2, the system generates a number of people feature based on the audio data and / or other non-visual sensor data. Optionally, further sub-block 308E2 includes: yet further sub-block 308E2A, wherein the system filters out speech from the speaker assembly; and / or yet further sub-block 308E2B, wherein the system generates a number of people feature based on speaker embedding from TISID.

[0104] For example, at sub-frame 308E2, the system may process audio data detected before (e.g., within the past 5 minutes or other threshold) and / or during a spoken utterance to determine the number of unique human voices captured in the audio data. For example, the number of unique personnel may be determined corresponding to the number of unique human voices detected based on the processed audio data. As an example, the system may use TISID to process each segment of a segment containing voice activity of the audio data at a further sub-frame 308E2A to generate a corresponding voice embedding for the audio data. The system may then cluster the corresponding voice embedding based on the distance of the corresponding voice embedding in the embedding space. The system may then determine the number of unique personnel based on the number of unique clusters. In an implementation of generating a personnel quantity feature indicating whether personnel in an environment are registered, the voice embedding generated for the unique voice may be compared with the registered embedding stored locally, and based on the comparison, it may be determined whether the unique voice is registered or unregistered.

[0105] In some implementations, when determining the number of unique human voices, the system may filter out the candidate human voices detected in the corresponding segment of the audio data from the number of unique human voices included in a further sub-frame 308E2B. The system may perform such filtering based on determining that the segment of the audio data indicates that the candidate human voice originates from a speaker component of the electronic device, rather than from a human speaker present in the environment. In some of these implementations, the system may determine that the candidate human voice originates from the speaker component based on an analysis of the frequency of the segment of the audio data corresponding to the segment spoken by the candidate human voice. For example, if all or a threshold percentage (e.g., 80%, 90%, or other threshold percentage) of the frequencies are all within a certain range of each other, all below a certain upper threshold, and / or all above a certain lower threshold, the system may determine that the candidate human voice originates from the speaker component.

[0106] At sub-block 308F, the system generates a person-and-device signature that indicates a physical context between a mobile, battery-powered assistant device (eg, cell phone, watch) and the user.

[0107] At subblock 308G, the system generates additional and / or alternative features that each indicate a corresponding current value of a corresponding state of the environment.

[0108] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automated assistant component, and / or other components can include one or more components of the example computing device 510.

[0109] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0110] The user interface input device 522 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system), a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 510 or onto a communication network.

[0111] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and methods of all possible types of information output from the computing device 510 to a user or another machine or computing device.

[0112] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include software to perform selected aspects of the method of FIG. 2 and implement Figure 1 and FIG. 2A to FIG. 2D The logic of the various components depicted in .

[0113] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored by the file storage subsystem 526 in the storage subsystem 524, or stored in other machines accessible to the processor 514.

[0114] Bus subsystem 512 provides a mechanism for the various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0115] Computing device 510 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Figure 5 The description of the depicted computing device 510 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computing device 510 are possible, and these configurations are similar to those of the embodiment of the present invention. Figure 5 Computing devices are depicted as having more or fewer components than those shown.

[0116] In situations where the systems described herein collect or otherwise monitor personal information about a user or can make use of the personal information and / or monitored information, the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or, in the case of geographic location information, the user's geographic location may be generalized (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may control how information about the user is collected and / or used.

[0117] In some implementations, a method is provided, the method comprising: performing ASR on audio data independently of any explicit call to an automated assistant to generate an ASR text that predicts a user's spoken utterance. The audio data is detected via one or more microphones of a client device in an environment and captures the user's spoken utterance. The method further comprises: generating a candidate automated assistant action corresponding to the ASR text and a confidence measure for the candidate automated assistant action based on processing the ASR text. The method further comprises: generating one or more environmental features, each environmental feature reflecting a corresponding current value of a corresponding dynamic state of the environment. Generating one or more environmental features is based on processing data from a client device and / or from one or more additional client devices in the environment. The method further comprises: determining whether to cause the automatic execution of a candidate automated assistant action in response to spoken utterance. Determining whether to cause the automatic execution of a candidate automated assistant action is based on processing both: a confidence measure for a candidate automated assistant action, and one or more environmental features. The method further comprises, in response to determining to cause the automatic execution of a candidate automated assistant action: causing the automatic execution of a candidate automated assistant action in response to spoken utterance. The method further includes, in response to not determining to cause automatic performance of the candidate automated assistant action: suppressing any automatic performance of the candidate automated assistant action in response to the spoken utterance.

[0118] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0119] In some implementations, the method further includes: selecting a semantic category of a candidate automated assistant action from a plurality of candidate semantic categories based on the candidate automated assistant action. The semantic category may be a genus category covering a plurality of disparate intentions, the plurality of disparate intentions including the intention of the candidate automated assistant action, and determining whether to cause the automatic execution of the candidate automated assistant action is further based on processing the semantic category. In some versions of these implementations, determining whether to cause the automatic execution of the candidate automated assistant action includes: using a trained machine learning (ML) model to process a confidence metric, one or more environmental features, and a semantic category to generate an ML output, and determining whether to cause the automatic execution of the candidate automated assistant action based on the ML output. In some of these versions, the ML output is a probability, and determining whether to cause the automatic execution of the candidate automated assistant action based on the ML output includes: comparing the probability with a threshold. In some additional or alternative versions, determining whether to cause the automatic execution of the candidate automated assistant action includes: identifying the rule based on the rule being indexed in association with the semantic category, and determining whether to cause the automatic execution of the candidate automated assistant action based on applying the confidence metric and one or more environmental features to the rule. In some additional or alternative versions, the candidate automated assistant actions are generated based on performing natural language processing on the ASR text and include an intent and one or more slot values ​​for one or more corresponding slots of the intent.

[0120] In some implementations, determining whether to cause automatic performance of a candidate automated assistant action includes: processing a confidence metric and one or more environmental features using a trained machine learning (ML) model to generate an ML output, and determining whether to cause automatic performance of the candidate automated assistant action based on the ML output.

[0121] In some implementations, one or more environmental features include a number of people feature that indicates the number of people in the environment. In some versions of these implementations, generating the number of people feature includes: processing audio data and / or additional audio data (e.g., detected via a microphone of a client device before the audio data is detected) to determine the number of unique human voices captured in the audio data and / or additional audio data; and generating a number of people features based on the number of unique human voices. In some of these versions, processing the audio data and / or additional audio data to determine the number of unique human voices captured in the audio data and / or additional audio data includes: determining that given audio data corresponding to a candidate human voice indicates that the candidate human voice originates from a speaker component rather than from a human speaker present in the environment; and in response to determining that the given audio data indicates that the candidate human voice originates from the speaker component, filtering the candidate human voice from the number of unique human voices included. As an example, determining that the given audio data indicates that the candidate human voice originates from the speaker component may include: determining that the frequencies of the given audio data are all within a given frequency range indicating that it originates from the speaker component. In some additional or alternative versions, processing the audio data and / or the additional audio data to determine the number of unique human voices includes: processing the audio data and / or the additional audio data using a text-independent speaker recognition model to generate corresponding speaker embeddings; clustering the corresponding speaker embeddings; and determining the number of unique human voices based on the number of clusters obtained by clustering. In some additional or alternative versions, determining the number of unique human voices based on the number of clusters obtained by clustering includes: in response to determining that a given audio data indication for generating a corresponding speaker embedding of a given cluster in the clustering originates from a speaker component rather than from a human speaker present in the environment, filtering the cluster from being included in the number of unique human voices.

[0122] In some implementations, the one or more environmental features include a temporal feature indicating one or more current temporal conditions, a spoken utterance source feature indicating an origin location and / or origin direction of a spoken utterance, a person number feature indicating the number of people in the environment, a user activity feature indicating one or more activities in which a user is currently engaged, and / or an environmental location feature indicating one or more semantic classifications of the environment.

[0123] In some implementations, the client device is a battery-powered mobile device, and the one or more environmental features include a human-to-device feature indicating a physical context between the client device and the user. For example, the physical context between the client device and the user indicated by the human-to-device feature may be in a pocket of the user or held and near the face of the user.

[0124] In some implementations, a method is provided, and the method includes: performing ASR on audio data independently of any explicit call to an automated assistant to generate ASR text that predicts a user's spoken utterance. The audio data is detected via one or more microphones of a client device in an environment and captures the user's spoken utterance. The method further includes: generating a candidate automated assistant action corresponding to the ASR text based on processing the ASR text. The method further includes: selecting a semantic category of a candidate automated assistant action from a plurality of candidate semantic categories based on the candidate automated assistant action. A semantic category is a genus category covering a plurality of distinct intents, the plurality of distinct intents including the intent of the candidate automated assistant action. The method further includes: generating one or more environmental features, each environmental feature reflecting a corresponding current value of a corresponding dynamic state of the environment. Generating one or more environmental features is based on processing data from a client device and / or from one or more additional client devices in the environment. The method further includes: determining whether to cause the automatic execution of a candidate automated assistant action in response to a spoken utterance. Determining whether to cause the automatic execution of a candidate automated assistant action is based on processing both: a semantic category of a candidate automated assistant action and one or more environmental features. The method further includes, in response to determining to cause automatic performance of the candidate automated assistant action: causing automatic performance of the candidate automated assistant action in response to the spoken utterance. The method further includes, in response to not determining to cause automatic performance of the candidate automated assistant action: suppressing any performance of the candidate automated assistant action in response to the spoken utterance.

[0125] In some implementations, a method is provided, the method comprising: performing ASR on audio data independently of any explicit call to an automated assistant to generate ASR text that predicts a user's spoken utterance. The audio data is detected via one or more microphones of a client device in an environment and captures the user's spoken utterance. The method further comprises: generating a first candidate automated assistant action corresponding to the ASR text and a second candidate automated assistant action corresponding to the ASR text based on processing the ASR text. For example, the first intention of the first candidate automated assistant action may be different from the second intention of the second candidate automated assistant action. The method further comprises: generating one or more environmental features, each environmental feature reflecting a corresponding current value of a corresponding dynamic state of the environment. Generating one or more environmental features is based on processing data from a client device and / or from one or more additional client devices in the environment. The method further comprises: determining whether the first candidate automated assistant action or the second automated assistant action is automatically executed in response to the spoken utterance. Determining whether to cause automatic execution of a first candidate automated assistant action or a second automated assistant action in response to a spoken utterance includes: generating a first output based on processing one or more environmental features and one or more first features of the first automated assistant action; generating a second output based on processing one or more environmental features and one or more second features of the second automated assistant action; and selecting one of the first candidate action and the second candidate action based on comparing the first output to the second output. The method further includes: causing automatic execution of the selected one of the first candidate action and the second candidate action in response to the spoken utterance.

[0126] In some implementations, one or more first features include a first confidence measure for a first automation and / or a first semantic category for a first automated assistant action, and one or more second features include a second confidence measure for a second automated assistant action and / or a second semantic category for a second automated assistant action.

Claims

1. A method implemented by one or more processors, the method comprising: performing automatic speech recognition (ASR) on audio data, independent of any explicit invocation of the automated assistant, to generate ASR text that predicts spoken utterances of a user, wherein the audio data is detected via one or more microphones of a client device in an environment and captures the spoken utterances of the user; Based on processing the ASR text: candidate automated assistant actions corresponding to the ASR text, and a confidence measure for the candidate automated assistant action; generating one or more environmental characteristics, each environmental characteristic reflecting a corresponding current value of a corresponding dynamic state of the environment, wherein generating the one or more environmental characteristics is based on processing data from the client device and / or from one or more additional client devices in the environment; Determining whether to cause automatic performance of the candidate automated assistant action in response to the spoken utterance, wherein determining whether to cause automatic performance of the candidate automated assistant action is based on processing both: the confidence measure for the candidate automated assistant action, and the one or more environmental characteristics; and In response to determining to cause automatic performance of the candidate automated assistant action: causing automatic performance of the candidate automated assistant action in response to the spoken utterance; In response to not determining causing automatic performance of the candidate automated assistant action: Any automatic performance of the candidate automated assistant action is suppressed in response to the spoken utterance.

2. The method of claim 1, further comprising: selecting a semantic category for the candidate automated assistant action from a plurality of candidate semantic categories based on the candidate automated assistant action, wherein the semantic category is a generic category covering a plurality of distinct intents, the plurality of distinct intents including the intent of the candidate automated assistant action; Wherein determining whether to cause automatic execution of the candidate automated assistant action is further based on processing the semantic category.

3. The method of claim 2, wherein determining whether to cause automatic execution of the candidate automated assistant action comprises: Processing the confidence measure, the one or more environmental features, and the semantic category using a trained machine learning (ML) model to generate an ML output; as well as A determination is made based on the ML output whether to cause automatic performance of the candidate automated assistant action.

4. The method of claim 3, wherein the ML output is a probability, and wherein determining whether to cause automatic execution of the candidate automated assistant action based on the ML output comprises: The probability is compared to a threshold.

5. The method of any one of claims 2, wherein determining whether to cause automatic execution of the candidate automated assistant action comprises: identifying the rule based on the rule being indexed in association with the semantic category; as well as A determination is made whether to cause automatic performance of the candidate automated assistant action based on applying the confidence metric and the one or more environmental characteristics to the rule.

6. The method of any preceding claim, wherein the candidate automated assistant actions are generated based on performing natural language processing on the ASR text and include an intent and one or more slot values ​​for one or more corresponding slots of the intent.

7. The method of claim 1, wherein determining whether to cause automatic execution of the candidate automated assistant action comprises: Processing the confidence metric and the one or more environmental features using a trained machine learning (ML) model to generate an ML output; as well as A determination is made based on the ML output whether to cause automatic performance of the candidate automated assistant action.

8. The method of any preceding claim, wherein the one or more environment characteristics include a people quantity characteristic indicating the quantity of people in the environment.

9. The method of claim 8, wherein generating the personnel quantity feature comprises: processing the audio data and / or additional audio data to determine a number of unique human voices captured in the audio data and / or the additional audio data, wherein the additional audio data is detected via the one or more microphones of the client device prior to detecting the audio data; as well as The number of people feature is generated based on the number of unique human voices.

10. The method of claim 9, wherein processing the audio data and / or the additional audio data to determine the number of unique human voices captured in the audio data and / or the additional audio data comprises: determining that given audio data corresponding to a candidate human voice indicates that the candidate human voice originates from a speaker assembly rather than from a human speaker present in the environment; as well as In response to determining that the given audio data indicates that the candidate human voice originates from the speaker assembly, the candidate human voice is filtered out from the number of unique human voices included.

11. The method of claim 10, wherein determining that the given audio data indicates that the candidate human voice originated from the speaker assembly comprises: It is determined that the frequencies of the given audio data are all within a given frequency range indicating origination from a speaker assembly.

12. The method of claim 9, wherein processing the audio data and / or the additional audio data to determine the number of unique human voices comprises: Processing the audio data and / or the additional audio data using a text-independent speaker recognition model to generate corresponding speaker embeddings; clustering the corresponding speaker embeddings; as well as The number of unique human voices is determined based on a number of clusters resulting from the clustering.

13. The method of claim 12, wherein determining the number of unique human voices based on the number of clusters resulting from the clustering comprises: In response to determining that a given audio data indication used to generate the corresponding speaker embedding of a given cluster in the clusters originates from a speaker assembly rather than from a human speaker present in the environment, filtering the cluster from inclusion in the number of unique human voices.

14. A method as claimed in any preceding claim, wherein the one or more environmental characteristics include one or more of the following: a time characteristic indicating one or more current time conditions; a spoken utterance source feature, the spoken utterance source feature indicating an origin position and / or an origin direction of the spoken utterance; a personnel quantity feature, the personnel quantity feature indicating the quantity of personnel in the environment; User activity characteristics, the user activity characteristics indicating one or more activities in which the user is currently engaged; or An environmental location feature, wherein the environmental location feature indicates one or more semantic classifications of the environment.

15. A method as claimed in any preceding claim, wherein the one or more environmental characteristics include two or more of the following: a time characteristic indicating one or more current time conditions; a spoken utterance source feature, the spoken utterance source feature indicating an origin position and / or an origin direction of the spoken utterance; a personnel quantity feature, the personnel quantity feature indicating the quantity of personnel in the environment; User activity characteristics, the user activity characteristics indicating one or more activities in which the user is currently engaged; or An environmental location feature, wherein the environmental location feature indicates one or more semantic classifications of the environment.

16. A method as claimed in any preceding claim, wherein the client device is a battery-powered mobile device, and wherein the one or more environmental characteristics include a human-to-device characteristic, the human-to-device characteristic indicating a physical context between the client device and the user.

17. The method of claim 16, wherein the physical context between the client device and the user indicated by the human-to-device feature is in the user's pocket or held near the user's face.

18. A method implemented by one or more processors, the method comprising: performing automatic speech recognition (ASR) on audio data, independent of any explicit invocation of the automated assistant, to generate ASR text that predicts spoken utterances of a user, wherein the audio data is detected via one or more microphones of a client device in an environment and captures the spoken utterances of the user; Based on processing the ASR text: candidate automated assistant actions corresponding to the ASR text; selecting a semantic category for the candidate automated assistant action from a plurality of candidate semantic categories based on the candidate automated assistant action, wherein the semantic category is a generic category covering a plurality of distinct intents, the plurality of distinct intents including the intent of the candidate automated assistant action; generating one or more environmental characteristics, each environmental characteristic reflecting a corresponding current value of a corresponding dynamic state of the environment, wherein generating the one or more environmental characteristics is based on processing data from the client device and / or from one or more additional client devices in the environment; Determining whether to cause automatic performance of the candidate automated assistant action in response to the spoken utterance, wherein determining whether to cause automatic performance of the candidate automated assistant action is based on processing both: the semantic category of the candidate automated assistant action, and the one or more environmental characteristics; and In response to determining to cause automatic performance of the candidate automated assistant action: causing automatic performance of the candidate automated assistant action in response to the spoken utterance; In response to not determining causing automatic performance of the candidate automated assistant action: Any performance of the candidate automated assistant action is suppressed in response to the spoken utterance.

19. A method implemented by one or more processors, the method comprising: performing automatic speech recognition (ASR) on audio data, independent of any explicit invocation of the automated assistant, to generate ASR text that predicts spoken utterances of a user, wherein the audio data is detected via one or more microphones of a client device in an environment and captures the spoken utterances of the user; Based on processing the ASR text: a first candidate automated assistant action corresponding to the ASR text; a second candidate automated assistant action corresponding to the ASR text; generating one or more environmental characteristics, each environmental characteristic reflecting a corresponding current value of a corresponding dynamic state of the environment, wherein generating the one or more environmental characteristics is based on processing data from the client device and / or from one or more additional client devices in the environment; determining whether to cause automatic performance of the first candidate automated assistant action or the second automated assistant action in response to the spoken utterance, wherein determining whether to cause automatic performance of the first candidate automated assistant action or the second automated assistant action in response to the spoken utterance comprises: generating a first output based on processing the one or more environmental characteristics together with one or more first characteristics of the first automated assistant action, generating a second output based on processing the one or more environmental characteristics together with one or more second characteristics of the second automated assistant action, and selecting one of the first candidate action and the second candidate action based on comparing the first output to the second output; as well as Automatic performance of a selected one of the first candidate action and the second candidate action is caused in response to the spoken utterance.

20. The method of claim 19, wherein the one or more first features include a first confidence measure for the first automated assistant action and / or a first semantic category for the first automated assistant action; and The one or more second features include a second confidence measure for the second automated assistant action and / or a second semantic category for the second automated assistant action.

21. An assistant device, comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations according to any one of claims 1 to 20.

22. A computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 20.

Images