Contextual suppression of assistant command

Abstract

To provide a method for suppressing execution of an assistant command based on contextual audio data.SOLUTION: In some embodiments, an audio data stream is processed for determining a portion of audio data corresponding to a specific word and / or a phrase (for example, a warm word) associated with an assistant command using a warm word model, a preamble portion of audio data (for example, preceding the warm word) and / or a post-amble portion of the audio data (for example, following the warm word) is processed for generating an ASR output using an automatic speech recognition (ASR) model, and based on processing of the ASR output, it is determined whether or not a user has intended that the assistant command is to be executed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application relates to contextual suppression of assistant commands. [Background technology]

[0002] Humans may participate in human-computer dialogs using interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). For example, a human (who may be referred to as a “user,” when interacting with an automated assistant) may provide the automated assistant with verbal natural language input (i.e., spoken utterances), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. The automated assistant generally responds to the verbal utterances by providing responsive user interface output (e.g., audible and / or visual user interface output), by controlling a smart device, and / or by performing other actions.

[0003] Automated assistants typically rely on a pipeline of components in interpreting and responding to verbal utterances. For example, an automatic speech recognition (ASR) engine can process audio data corresponding to a user's verbal utterance to generate ASR output, such as a transcription of the verbal utterance (i.e., a sequence of terms and / or other tokens). Additionally, a natural language understanding (NLU) engine can process the ASR output to generate NLU output, such as the user's intent in providing the verbal utterance and, optionally, slot values for parameters associated with the intent. Additionally, a fulfillment engine can be used to process the NLU output and generate fulfillment output, such as a structured request for obtaining response content to the verbal utterance.

[0004] In some cases, this pipeline of components can be bypassed. For example, some machine learning (ML) models (also called “warm word models”) can be trained to detect specific words and / or phrases (also called “warm words”) that map directly to fulfillment outputs. For example, if a user is listening to music and provides the verbal utterance “Turn up the volume,” one or more of these ML models can process the audio data and generate a structured request to turn up the volume on the device playing the music without generating any ASR and / or NLU outputs. However, these warm words are typically common words and / or phrases that may occur in everyday conversation. As a result, assistant commands associated with these warm words may be unintentionally triggered by the user, thereby wasting computational resources. Furthermore, in some of these cases, the user may have to cancel these assistant commands, thereby wasting further computational resources. Summary of the Invention [Means for solving the problem]

[0005] Implementations disclosed herein are directed to contextually inhibiting the execution of assistant commands associated with one or more specific words and / or phrases (e.g., warm words) based on the identity of a user who provided contextual audio data and / or a verbal utterance including one or more of the specific words and / or phrases. Some implementations process a stream of audio data to determine portions of the audio data corresponding to the specific words and / or phrases associated with the assistant commands using one or more warm word models. Some of these implementations use an automatic speech recognition (ASR) model to process a preamble portion of the audio data (e.g., a portion of the audio data preceding a portion of the audio data corresponding to one or more of the specific words and / or phrases) and / or a postamble portion of the audio data (e.g., a portion of the audio data following a portion of the audio data corresponding to one or more of the specific words and / or phrases) to generate an ASR output. Furthermore, some of these implementations determine, based on processing the ASR output, whether a user who provided the verbal utterance captured in the audio data intended the assistant command associated with one or more of the specific words or phrases to be executed. Additional or alternative implementations can process the stream of audio data using a speaker identification (SID) model to determine whether the audio data is sufficient to identify a user who provided the verbal utterance captured in the stream of audio data, or to determine whether the user who provided the verbal utterance is authorized to cause execution of an assistant command.

[0006] For example, assume that a call is received at a user's client device. Further, assume that the user of the client device provides the verbal utterance "answer" in response to the call being received at the client device. In this example, assuming that "answer" is a warm word associated with an assistant command that causes an automated assistant implemented at least partially within the client device to answer the call, the automated assistant can answer the call on behalf of the user based on detecting the occurrence of the warm word "answer" using one or more warm word models. In contrast, assume that in response to the call being received at the client device, the user of the client device provides the verbal utterance "I don't want to answer that." In this example, the automated assistant can process the contextual audio data to determine that the call should not be answered on behalf of the user, even though the automated assistant detected the occurrence of the warm word "answer" using one or more of the warm word models.

[0007] In some implementations, one or more automated assistant components that utilize one or more of the warm word models can be activated in response to detecting the occurrence of a warm word activation event. Warm word activation events can include, for example, a phone call received at a client device, a text message received at a client device, an email received at a client device, an alarm or timer sounding at a client device, media playing at the client device or additional client devices in the client device's environment, a notification received at a client device, the location of the client device, a software application accessible at the client device, and / or other events associated with the client device to which a user can provide verbal utterances to control the client device or additional client devices communicating with the client device. In particular, some of these warm word activation events are individual events (e.g., a notification received at a client device, a text message received at a client device, an email received at a client device, an alarm or timer sounding at a client device, etc.), while others of these warm word activation events are continuous events (e.g., a software application accessible at the client device). Furthermore, it should be noted that these warm word activation events are not mutually exclusive. In other words, multiple warm word activation events may be detected at a given time, such that one or more automated assistant components are actively monitoring warm words based on multiple disparate warm word activation events.

[0008] The warm word models described herein can include multiple heterogeneous warm word models (e.g., audio keyword classification models) trained to detect one or more specific words and / or phrases (e.g., warm words) that, when detected, cause the automated assistant to execute an assistant command associated with one or more of the specific words and / or phrases. In some implementations, a given warm word model can be trained to detect a specific subset of words and / or phrases associated with a given warm word activation event. For example, assume that music is playing on a client device or another client device (e.g., a smart speaker) that communicates with the client device. In this example, a given warm word model may be trained to detect a subset of specific words and / or phrases that cause assistant commands associated with controlling music to be executed, such as a "pause" warm word that may pause music, a "resume" warm word that may resume music after pausing, a "volume up" warm word that may increase the volume of music, a "volume down" warm word that may decrease the volume of music, a "next" warm word that may skip music to the next song, and other specific words and / or phrases associated with music. Thus, in this example, in response to determining that music is being played on the client device or another client device in communication with the client device, one or more of the currently paused assistant functions utilizing the given warm word model may be activated.

[0009] In additional or alternative implementations, multiple warm word models may be trained to detect subsets of words and / or phrases associated with a given warm word activation event. Continuing with the example above, a first warm word model may be trained to detect one or more first specific words and / or phrases, such as the “pause” warm word and the “resume” warm word, a second warm word model may be trained to detect one or more second specific words and / or phrases, such as the “volume up” warm word and the “volume down” warm word, and a third warm word model may be trained to detect one or more third specific words and / or phrases, such as those related to the “next” warm word and other specific words and / or phrases associated with a music warm word activation event. Thus, in this example, one or more of the currently dormant assistant functions utilizing at least the first warm word model, the second warm word model, and the third warm word model may be activated in response to determining that music is playing on the client device or another client device in communication with the client device.

[0010] In some implementations, in response to one or more of the particular words and / or phrases being detected in the stream of audio data, a preamble portion of the audio data and / or a postamble portion of the audio data can be processed to determine whether the user who provided the verbal utterance actually intended to execute an assistant command associated with the detected one or more of the particular words and / or phrases. For example, assume again that a call is received at a user's client device, and assume that the user of the client device provides the verbal utterance "I don't want to answer that" in response to receiving the call at the client device. In this example, the preamble portion of the audio data (e.g., responding "I don't want to") can be obtained from the audio buffer of the client device and processed using an ASR model to generate an ASR output. Further, the ASR output can be processed using an NLU model to generate an NLU output. In this example, the ASR output and / or the NLU output indicates that the user did not intend to "answer," which would cause the automated assistant to answer the call on the user's behalf. In some versions of those implementations, a portion of the audio data corresponding to one or more particular words and / or phrases (e.g., corresponding to "respond") may additionally or alternatively be processed together with a preamble portion of the audio data to generate the ASR output and the NLU output. In some versions of those implementations, a postamble portion of the audio data (e.g., corresponding to "that") may additionally or alternatively be processed together with the preamble portion of the audio data and / or portions of the audio data corresponding to one or more of the particular words and / or phrases to generate the ASR output and the NLU output.

[0011] In some versions of these implementations, in response to detecting additional voice activity in addition to one or more of the specific words and / or phrases, one or more currently dormant automated assistant components utilizing an ASR model and / or an NLU model may be activated. For example, a stream of audio data may be processed using a voice activity detection (VAD) model to determine whether any additional voice activity is present. In implementations in which additional voice activity is detected, one or more of the currently dormant automated assistant components utilizing an ASR model and / or an NLU model may be activated to process a preamble portion of the audio data, a portion of the audio data corresponding to one or more of the specific words and / or phrases, and / or a postamble portion of the audio data. In implementations in which additional voice activity is not detected, one or more of the currently dormant automated assistant components utilizing an ASR model and / or an NLU model may remain dormant, and assistant commands associated with the specific words and / or phrases may be executed.

[0012] In various implementations, the identity of the user who provided the verbal utterance must be detected before execution of the assistant command. For example, an assistant command associated with a particular warm word activation event, a particular device, and / or a particular software application may require that the user's identity be verified. For example, in an implementation in which an assistant command causes an automated assistant to respond to an electronic communication (e.g., a phone call, a text message, an email, a social media message, and / or other electronic communication), the user who provided the verbal utterance may need to be verified before the automated assistant can respond to the electronic communication. Also, for example, in an implementation in which an assistant command causes an automated assistant to control a particular smart device (e.g., a smart lock, a smart oven, a smart garage door, etc.), the user who provided the verbal utterance may need to be verified before the automated assistant can control the particular smart device. However, some assistant commands may not require that the user's identity be verified before execution of the assistant command. For example, in an implementation in which an assistant command causes an automated assistant to control media (e.g., music, video, television programs, and / or other media), the user who provided the utterance may not need to be verified before the automated assistant responds to the electronic communication. Also, for example, in an implementation in which an assistant command causes an automated assistant to control other specific smart devices (e.g., smart lights, smart clocks, etc.), the user who provided the verbal utterance may not need to be verified before the automated assistant controls the specific smart device.

[0013] As described above, a stream of audio data may be processed using a SID model to verify the identity of a user who provided a verbal utterance. In some versions of these implementations, assuming one or more previously generated text-dependent (TD) speaker embeddings are accessible at a client device for one or more specific words and / or phrases for a user, portions of the audio data corresponding to one or more of the specific words and / or phrases may be processed using a TD SID model to generate a TD speaker embedding. The generated TD speaker embedding may be effectively compared to one or more of the previously generated TD speaker embeddings for the user accessible at the client device to determine whether the verbal utterance is from an authorized user. In particular, one or more specific words and / or phrases in a TD SID may be constrained to one or more invocation phrases configured to invoke an automated assistant (e.g., hot words and / or trigger words such as “Hey Assistant,” “OK Assistant,” and / or “Assistant”) or one or more warm words described herein.

[0014] In additional or alternative versions of those implementations, assuming one or more previously generated text-independent (TI) speaker embeddings are accessible at a client device for the user, audio data corresponding to an oral utterance in the stream of audio data may be processed using a TI SID model to generate a TI speaker embedding. The generated TI speaker embedding may be effectively compared to one or more of the previously generated TI speaker embeddings for the user accessible at the client device to determine whether the oral utterance is from an authorized user. In particular, in a TD SID, the one or more previously generated TI speaker embeddings of the TI may be constrained to one or more invocation phrases configured to invoke an automated assistant (e.g., hot words and / or trigger words such as “Hey Assistant,” “OK Assistant,” and / or “Assistant”) or one or more warm words described herein.

[0015] Thus, in some versions of these implementations, the length of an oral utterance captured in a stream of audio data may be compared to a length threshold to determine whether the utterance is of sufficient length to perform a TI SID. If the length of the oral utterance is sufficient to perform a TI SID, the audio data corresponding to the oral utterance may be processed using a TI SID model and / or a TD SID model in an attempt to verify the user's identity. However, even if the length of the oral utterance is insufficient to perform a TI SID, the audio data corresponding to the oral utterance may still be processed using a TD SID model in an attempt to verify the user's identity. Assuming the length of the oral utterance is insufficient to perform a TI SID (and / or the TI SID is insufficient to verify the user's identity), and assuming the TD SID is insufficient to verify the user's identity, a prompt including example utterances may be provided for presentation to the user, where the example utterance is long enough (or long enough) to perform a TI SID or corresponds to particular terms and / or phrases for which one or more previously generated TD speaker embeddings are available. Additionally or alternatively, the prompt may require the user to provide other forms of biometric identification (e.g., facial print matching and / or fingerprint identification) in order to have the assistant command executed.

[0016] By using the techniques described herein, one or more technical advantages may be achieved. As one non-limiting example, the techniques described herein enable an automated assistant to determine whether a user actually intended to utter a warm word that would trigger the execution of an assistant command based on processing contextual audio data for the warm word. As a result, false detection of warm words may be reduced, thereby saving computational and / or network resources at the client device when the automated assistant utilizes one or more networks to cause the assistant command to be executed. Furthermore, because the user of the client device does not need to take action to modify or cancel any assistant command, the amount of user input may be reduced, thereby saving computational resources at the client device. Furthermore, various automated assistant functions described herein can remain dormant until needed, as described herein, thereby saving computational resources at the client device. As another non-limiting example, the techniques described herein enable an automated assistant to verify the identity of a user providing a verbal utterance that includes a warm word using an SID and / or other techniques. As a result, only authorized users can cause a particular assistant command to be executed, thereby improving data security for data associated with the user of the client device.

[0017] The above description is provided as a summary of only some implementations disclosed herein. These and other implementations are described in more detail herein.

[0018] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram of an exemplary environment demonstrating various aspects of the present disclosure in which implementations disclosed herein may be implemented. [Figure 2] 2A-2C illustrate exemplary process flows demonstrating various aspects of the present disclosure using various components of FIG. 1 according to various implementations. [Figure 3] 10 is a flowchart illustrating an example method for determining whether to cause an assistant command to be executed based on contextual audio data, according to various implementations. [Figure 4A] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on contextual audio data, according to various implementations. [Figure 4B] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on contextual audio data, according to various implementations. [Figure 4C] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on contextual audio data, according to various implementations. [Figure 5] 10 is a flowchart illustrating an example method for determining whether to cause an assistant command to be executed based on speaker identification, according to various implementations. [Figure 6A] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on speaker identification, according to various implementations. [Figure 6B] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on speaker identification, according to various implementations. [Figure 6C] 10A-10C illustrate various non-limiting examples of determining whether to execute an assistant command based on speaker identification, according to various implementations. [Figure 7]FIG. 1 illustrates an exemplary architecture of a computing device according to various implementations. DETAILED DESCRIPTION OF THE INVENTION

[0020] Referring now to FIG. 1 , a block diagram of an example environment is shown that demonstrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. The example environment includes a client device 110 and a warm word system 180. In some implementations, the warm word system 180 may be implemented locally at the client device 110. In additional or alternative implementations, as shown in FIG. 1 , the warm word system 180 may be implemented remotely from the client device 110. In these implementations, the client device 110 and the warm word system 180 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs,” including Wi-Fi LANs, mesh networks, Bluetooth, near field communications, etc.) or wide area networks (“WANs,” including the Internet).

[0021] Client device 110 may be, for example, one or more of a desktop computer, a laptop computer, a tablet, a mobile phone, a vehicle computing device (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker (optionally having a display), a smart appliance such as a smart television, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality or augmented reality computing device). Additional and / or alternative client devices may also be provided.

[0022] Client device 110 can execute automated assistant client 114. An instance of automated assistant client 114 can be an application separate from (e.g., installed “on”) the operating system of client device 110, or alternatively, can be implemented directly by the operating system of client device 110. Automated assistant client 114 can interact with warm word system 180 implemented locally at client device 110, or can interact with warm word system 180 over one or more of networks 199 as shown in FIG. 1. Automated assistant client 114 (and optionally, by interacting with other remote systems (e.g., servers)) can form what, from a user's perspective, appears to be a logical instance of automated assistant 115 with which the user can engage in a human-to-computer dialogue. An instance of automated assistant 115 is shown in FIG. 1 and is surrounded by a dashed line that includes automated assistant client 114 of client device 110 and warm word system 180. Thus, it should be understood that a user participating in an automated assistant client 114 running on a client device 110 may in effect participate in the user's own logical instance of the automated assistant 115 (or a logical instance of the automated assistant 115 shared among a household or other group of users). For brevity or simplicity, automated assistant 115 as used herein refers to an automated assistant client 114 running on a client device 110 and / or on one or more servers that may implement warm word system 180.

[0023] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may include one or more microphones that capture audio data, such as audio data corresponding to a user's verbal speech or other sounds in the client device's environment. Additionally or alternatively, client device 110 may include one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected within the field of view of one or more of the visual components. Additionally or alternatively, client device 110 may include one or more touch-sensitive components (e.g., a keyboard and mouse, a stylus, a touchscreen, a touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110.

[0024] In various implementations, client device 110 may include a rendering engine 112 configured to provide content for audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may include one or more speakers that enable content to be provided for audible presentation to a user via client device 110. Additionally or alternatively, client device 110 may include a display or projector that enables content to be provided for visual presentation to a user via client device 110.

[0025] In various implementations, client device 110 may include one or more presence sensors 113 configured to provide a signal indicating a detected presence, particularly human presence, with approval from a corresponding user. In some of those implementations, automated assistant 115 may identify client device 110 (or another computing device associated with the user of client device 110) to fulfill the verbal utterance based at least in part on the user's presence at client device 110 (or at another computing device associated with the user of client device 110). The verbal utterance may be fulfilled by rendering response content at client device 110 and / or other computing devices associated with the user of client device 110 (e.g., via rendering engine 112), by causing client device 110 and / or other computing devices associated with the user of client device 110 to control, and / or by causing client device 110 and / or other computing devices associated with the user of client device 110 to perform any other action that fulfills the verbal utterance. As described herein, the automated assistant 115 can utilize data determined based on the presence sensor 113 when determining which client devices 110 (or other computing devices) the user is near or has recently been near, and provide corresponding commands only to the client devices 110 (or those other computing devices).In some additional or alternative implementations, the automated assistant 115 may utilize data determined based on the presence sensor 113 when determining whether any user (any user or a specific user) is currently in proximity to the client device 110 (or other computing device), and may optionally suppress the provision of data to and / or from the client device 110 (or other computing device) based on the user's proximity to the client device 110 (or other computing device).

[0026] The presence sensor 113 may take a variety of forms. For example, the client device 110 may utilize one or more of the user interface input components described above with respect to the user input engine 111 to detect the presence of a user. Additionally or alternatively, the client device 110 may include other types of light-based presence sensors 113, such as a passive infrared ("PIR") sensor that measures infrared ("IR") light emitted from objects within its field of view.

[0027] Additionally or alternatively, in some implementations, the presence sensor 113 may be configured to detect other phenomena related to the presence of a human or a device. For example, in some implementations, the client device 110 may include a presence sensor 113 that detects, for example, other computing devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by a user and / or various types of wireless signals (e.g., radio, ultrasonic, electromagnetic, etc. waves) emitted by the other computing devices. For example, the client device 110 may be configured to emit waves imperceptible to humans, such as ultrasonic or infrared waves, that can be detected by the other computing devices (e.g., via an ultrasonic / infrared receiver, such as an ultrasound-enabled microphone).

[0028] Additionally or alternatively, client device 110 may emit other types of imperceptible waves, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), that can be detected by other computing devices carried / operated by the user (e.g., mobile devices, wearable computing devices, etc.) and used to determine the user's specific location. In some implementations, GPS and / or Wi-Fi triangulation may be used to detect a person's location, for example, based on GPS and / or Wi-Fi signals to and / or from client device 110. In other implementations, other wireless signal characteristics, such as time of flight, signal strength, etc., may be used by client device 110, singly or collectively, to determine the location of a specific person based on signals emitted by other computing devices carried / operated by the user.

[0029] Additionally or alternatively, in some implementations, client device 110 may perform speaker identification (SID) to recognize a user from their voice (e.g., as described with respect to user verification engine 160). In some implementations, the speaker's movements may then be determined by client device 110's presence sensor 113 (and optionally client device 110's GPS sensor, Soli chip, and / or accelerometer). In some implementations, based on the movements so determined, the user's location may be predicted, and this location may be assumed to be the user's location when any content is caused to be rendered on client device 110 and / or other computing devices, based at least in part on the proximity of client device 110 and / or other computing devices to the user's location. In some implementations, the user may simply be assumed to be at the location where the user last engaged with automated assistant 115, especially if not much time has passed since the last engagement.

[0030] Additionally, client device 110 and / or warm word system 180 may include one or more memories for storing data and / or software applications 198, one or more processors for accessing data and executing software applications 198, and / or other components that facilitate communication over one or more of networks 199. In some implementations, one or more of the software applications 198 may be installed locally at client device 110, while in other implementations, one or more of the software applications 198 may be hosted remotely (e.g., by one or more servers) and accessible by client device 110 via one or more of networks 199. Operations performed by client device 110, other computing devices, and / or automated assistant 115 may be distributed across multiple computer systems. The automated assistant 115 may be implemented, for example, as a computer program running on the client device 110 and / or on one or more computers at one or more locations coupled to each other via a network (e.g., network 199 of FIG. 1).

[0031] In some implementations, the operations performed by automated assistant 115 may be implemented locally at client device 110 via automated assistant client 114. As shown in FIG. 1, automated assistant client 114 may include automatic speech recognition (ASR) engine 120A1, natural language understanding (NLU) engine 130A1, and text-to-speech (TTS) engine 140A1. In some implementations, the operations performed by automated assistant 115 may be distributed across multiple computer systems, such as when warm word system 180 is implemented remotely from client device 110 as shown in FIG. 1. In these implementations, automated assistant 115 may additionally or alternatively utilize ASR engine 120A2, NLU engine 130A2, and TTS engine 140A2 of warm word system 180.

[0032] Each of these engines may be configured to perform one or more functions. For example, ASR engine 120A1 and / or 120A2 may capture oral utterances and process audio data generated by the microphone of client device 110 using an ASR model (e.g., a recurrent neural network (RNN) model, a Transformer model, and / or any other type of ML model capable of performing ASR) stored in machine learning (ML) model database 115A to generate an ASR output. Additionally, NLU engine 130A1 and / or 130A2 may process the ASR output using an NLU model (e.g., a long short-term memory (LSTM), a gated recurrent unit (GRU), or other ML model capable of performing NLU) and / or grammar-based rules stored in ML model database 115A to generate an NLU output. Additionally, TTS engines 140A1 and / or 140A2 can process text data (e.g., text formulated by automated assistant 115) using TTS models stored in ML model database 115A to generate synthetic speech audio data, including computer-generated synthetic speech.

[0033] In various implementations, the ASR output may include, for example, one or more speech hypotheses (e.g., term hypotheses and / or transcript hypotheses) predicted to correspond to the user's spoken utterance captured in the audio data, one or more corresponding predicted values (e.g., probabilities, log-likelihoods, and / or other values) for each of the one or more speech hypotheses, a plurality of phonemes predicted to correspond to the user's spoken utterance captured in the audio data, and / or other ASR outputs. In some versions of these implementations, ASR engine 120A1 and / or 120A2 may select one or more of the speech hypotheses as recognized text corresponding to the spoken utterance (e.g., based on the corresponding predicted values).

[0034] In various implementations, the NLU output may include annotated recognized text, for example, including one or more annotations of the recognized text for one or more (e.g., all) of the terms in the recognized text. For example, NLU engines 130A1 and / or 130A2 may include a portion of a part-of-speech tagger (not shown) configured to annotate terms with their grammatical roles. Additionally or alternatively, NLU engines 130A1 and / or 130A2 may include an entity tagger (not shown) configured to annotate entity references within one or more segments of the recognized text, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, places (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and, in some cases, entity attributes) as well as edges connecting the nodes and representing relationships between the entities. The entity tagger may annotate references to entities at a high level of granularity (e.g., enabling identification of all references to an entity class such as people) and / or at a lower level of granularity (e.g., enabling identification of all references to a particular entity such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities and / or may optionally communicate with a knowledge graph or other entity database to resolve particular entities. Additionally or alternatively, NLU engines 130A1 and / or 130A2 may include a coreference resolver (not shown) configured to group or “cluster” references to the same entity based on one or more contextual cues.For example, a coreference resolver may be utilized to resolve the term “them” to “buy theater tickets” in a natural language input of “buy them” based on a mention of “theater tickets” in a client device notification rendered immediately prior to receiving the “buy them” input. In some implementations, one or more components of NLU engines 130A1 and / or 130A2 may rely on annotations from one or more other components of NLU engines 130A1 and / or 130A2. For example, in some implementations, an entity tagger may rely on annotations from a coreference resolver when annotating all references to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from an entity tagger when clustering references to the same entity.

[0035] 1 is described with respect to a single client device with a single user, it should be understood that this is for purposes of example and not meant to be limiting. For example, one or more additional client devices of the user may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form an ecosystem of devices that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., a group of users, a household).

[0036] As described herein, the automated assistant 115 can determine whether to cause an assistant command to be executed in response to detecting a warm word in the stream of audio data based on the contextual audio data and / or the identity of the speaker who provided the verbal utterance captured in the stream of audio data. In making this determination, the automated assistant can utilize the warm word engine 150, the user verification engine 160, and / or the acoustic engine 170. In various implementations, as shown in FIG. 1, the warm word engine 150 can include a warm word activation event engine 151, a warm word detection engine 152, and a warm word verification engine 153.

[0037] In some implementations, the warm word activation event engine 151 can monitor for the occurrence of warm word activation events. Warm word activation events can include, for example, a phone call received at the client device 110, a text message received at the client device 110, an email received at the client device 110, an alarm sounding at the client device 110, a timer sounding at the client device 110, media playing at the client device 110 or additional client devices in the client device 110's environment, a notification received at the client device 110, the location of the client device 110 (e.g., the user at home, at work, etc.), a software application among the one or more software applications 198 accessible at the client device 110, and / or other events associated with the client device 110 in which the user can provide a verbal utterance to cause the client device 110 or additional client devices in communication with the client device 110 to be controlled based on the verbal utterance. In some versions of those implementations, in response to detecting the occurrence of a warm word activation event, the warm word activation event engine 151 may activate one or more currently dormant automated assistant functions that utilize warm word models stored in the ML model database 115A (e.g., a low-power digital signal processor (DSP), central processing unit (CPU), etc.). The warm word models may be trained to detect one or more particular words or phrases (also referred to herein as "warm words") that are each associated with an assistant command.One or more particular words or phrases may be associated with assistant commands such that, when detected, automated assistant 115 can cause a particular action to be performed without explicitly invoking automated assistant 115 via an invocation word or phrase (e.g., “Assistant,” “Hey Assistant,” etc.), an invocation gesture, or other means for invoking automated assistant 115. In other words, one or more particular words or phrases may be mapped to a particular intent, particular fulfillment data, and / or a particular software application among one or more software applications 198 accessible at client device 110 such that automated assistant 115 can bypass utilization of various assistant components (e.g., ASR engines 120A1 and / or 120A2, NLU engines 130A1 and / or 130A2, and / or other automated assistant components) to cause execution of an assistant command associated with one or more particular words or phrases.

[0038] In some further versions of these implementations, currently dormant automated assistant functions utilizing a warm word model can remain active until one or more conditions are met. The one or more conditions can include, for example, the lapse of a threshold duration, the detection of a specific word or phrase, the determination that a warm word activation event has ended, and / or other conditions described herein. For example, assume that the warm word activation event includes a timer ringing at the client device 110. In this example, one or more of the currently dormant automated assistant functions utilizing a warm word model can remain active until the user provides a warm word to silence the timer or until the user manually stops the timer. As another example, assume that the warm word activation event includes the receipt of a notification at the client device 110. In this example, one or more of the currently dormant automated assistant functions utilizing a warm word model can remain active until the user provides a warm word to interact with the notification, until the user manually closes the notification, or until a threshold duration (e.g., 30 seconds, 60 seconds, etc.) has elapsed. As yet another example, assume that the warm word activation event includes receiving a phone call at the client device 110. In this example, one or more of the currently dormant automated assistant functions utilizing the warm word model can remain active until the user provides a warm word to interact with the call or until the phone stops ringing. Thus, once activated, the currently dormant automated assistant functions utilizing the warm word model described herein can be activated for a static or dynamic duration based on the detected warm word activation event and / or how the user of the client device 110 responds to the warm word activation event.

[0039] In additional or alternative implementations, one or more of the automated assistant functions that utilize the warm word model can be active at all times. In other words, some of the automated assistant functions that utilize the warm word model are always active regardless of whether a warm word activation event is detected by the warm word activation event engine 151. For example, one or more of the automated assistant functions that utilize a warm word model to detect specific words or phrases associated with a particular query, such as "what's the weather" or "what's on my calendar," specific words or phrases associated with a particular assistant routine, such as a "good morning" routine or a "good night" routine, specific words or phrases associated with a particular device, such as "turn on the lights" or "turn on the [room / light identifier] lights," specific words or phrases associated with a particular software application, whether launched or not, of one or more software applications 198 accessible at the client device 110, such as "open first application," "launch first application," "directions to 123 Main St.," when a navigation application is launched or not, or "cheese" when a camera application is launched, and / or other specific words or phrases.

[0040] The warm word models described herein can include multiple heterogeneous warm word models (e.g., audio keyword classification models) trained to detect one or more specific words or phrases (e.g., warm words) that, when detected, cause the automated assistant 115 to execute an assistant command associated with one or more of the specific words or phrases. In some implementations, a given warm word model can be trained to detect a specific subset of words or phrases associated with a given warm word activation event. For example, assume that music is playing on client device 110 or another client device in communication with client device 110. In this example, a given warm word model may be trained to detect a subset of specific words or phrases that cause assistant commands associated with controlling music to be executed, such as a "pause" warm word that may pause music, a "resume" warm word that may resume music after pausing, a "volume up" warm word that may increase the volume of music, a "volume down" warm word that may decrease the volume of music, a "next" warm word that may skip music to the next song, and other specific words or phrases associated with music. Thus, in this example, in response to determining that music is being played at client device 110 or another client device communicating with client device 110, one or more currently dormant assistant functions utilizing the given warm word model may be activated. Thus, at a single time instance, one or more warm word models may be active at client device 110.

[0041] In additional or alternative implementations, multiple warm word models may be trained to detect subsets of words or phrases associated with a given warm word activation event. Continuing with the above example, a first warm word model may be trained to detect one or more first specific words or phrases, such as the “pause” warm word and the “resume” warm word, a second warm word model may be trained to detect one or more second specific words or phrases, such as the “volume up” warm word and the “volume down” warm word, and a third warm word model may be trained to detect one or more third specific words or phrases, such as those related to the “next” warm word and other specific words or phrases associated with music. Thus, in this example, one or more of the currently dormant assistant functions utilizing at least the first warm word model, the second warm word model, and the third warm word model may be activated in response to determining that music is playing on client device 110 or another client device in communication with client device 110.

[0042] The warm word models described herein may be trained based on multiple training instances. Each of the multiple training instances may include a training instance input and a training instance output. The training instance input may include audio data, portions of the audio data corresponding to one or more of the specific words or phrases that the given warm word model is trained to detect, and the training instance output may include a ground truth output, which includes an indication of whether the audio data of the training instance input includes one or more of the specific words or phrases that the given warm word model is trained to detect. For example, when training a given warm word model in the above example described with respect to music playback, the training instance input for the given training instance may include audio data corresponding to “turn the volume up,” and the training instance output for the given training instance may include ground truth outputs indicating that the audio data includes one or more of the particular words or phrases (e.g., “volume up”). The training instance inputs may be applied as inputs to the given warm word model to generate predicted outputs (e.g., values such as probabilities, log-likelihoods, binary values, or other values) indicating the likelihood that the training instance input includes one or more of the particular words or phrases. The predicted outputs may be compared with the ground truth outputs of the training instance inputs (e.g., values indicating that the audio data of the training instance input actually includes one or more of the particular words or phrases, such as a value of “1” in this example) to generate one or more losses, and the given warm word model may be updated based on one or more of the losses.In this example, a given training instance may be considered a positive training instance because the training instance input includes a portion of the audio data that the given warm word model is trained to detect. Further, in this example, the given warm word model may optionally be additionally trained based on negative training instances, or training instances having training instance input corresponding to audio data that does not include one or more of the particular words or phrases that the given warm word model is trained to detect.

[0043] For example, when training a given warm word model in the above example described with respect to music playback, the training instance input for the given training instance may include audio data corresponding to "turn on the lights," and the training instance output for the given training instance may include ground truth outputs indicating that the audio data does not include one or more of the specific words or phrases (e.g., "volume up"). The training instance input may be applied as an input to the given warm word model to generate a predicted output (e.g., a value such as a probability, log-likelihood, binary value, or other value) indicating the likelihood that the training instance input includes one or more of the specific words or phrases. The predicted output may be compared with the ground truth output of the training instance input (e.g., a value indicating that the audio data of the training instance input actually includes one or more of the specific words or phrases, such as a value of "0" in this example) to generate one or more losses, and the given warm word model may be updated based on one or more of the losses. In this example, the given training instance may be considered a negative training instance because the training instance input does not include any portion of the audio data that the given warm word model is trained to detect (e.g., "volume up" or any other audio corresponding to controlling music).

[0044] In some implementations, assuming one or more of the automated assistant functions utilizing the warm word model are active, the warm word detection engine 152 can process a stream of audio data generated by the client device's microphone using the warm word model to monitor the occurrence of one or more specific words or phrases. For example, assume that a text message is received at the client device 110. Further assume that the warm word activation event engine 151 detects that a text message has been received at the client device 110 and activates one or more automated assistant components utilizing one or more warm word models to detect specific words or phrases (e.g., "reply," "dismiss," "show me," etc.) related to the warm word activation event in the text message being received at the client device 110 until one or more of the conditions are met. In this example, further assume that the user of the client device 110 provides the verbal utterance, "reply that I'll be there in five minutes." Thus, when processing the stream of audio data, warm word detection engine 152 may detect that a portion of the audio data corresponds to an occurrence of the warm word “reply.” In some implementations, warm word detection engine 152 may, in response to detecting the occurrence of the warm word, activate one or more additional currently active automated assistant components (e.g., ASR engines 120A1 and / or 120A2, NLU engines 130A1 and / or 130A2, and / or other automated assistant components) that utilize the ASR model and / or the NLU model.

[0045] As described in more detail below (e.g., with respect to FIGS. 2, 3, and 4A-4C), ASR engines 120A1 and / or 120A2 can use ASR models stored in ML model database 115A to process portions of audio data corresponding to occurrences of warm words, preamble portions of audio data preceding the portions of audio data corresponding to occurrences of warm words (if any), and / or postamble portions of audio data following the portions of audio data corresponding to occurrences of warm words (if any)) to generate ASR output. Further, NLU engines 130A1 and / or 130A2 can use NLU models and / or grammar-based rules stored in ML model database 115A to process the ASR output to generate NLU output. In some versions of these implementations, ASR engines 120A1 and / or 120A2 can retrieve one or more of those portions (e.g., preamble portions of audio data (if any) and / or portions of audio data corresponding to occurrences of warm words) from an audio buffer of client device 110. In various implementations, the ASR output may be biased toward particular terms and / or phrases associated with the warm word model. For example, if the warm word activation event corresponds to a media event (e.g., a user of client device 110 is listening to music on client device 110), the ASR output may be biased toward music-related terms and / or phrases, such as artists, songs, and music genres.

[0046] For example, the automated assistant 115 may process the stream of audio data using the end-pointing model stored in the ML model database 115A to generate multiple timestamps for verbal utterances captured in the stream of audio data. The multiple timestamps may include at least a first timestamp associated with a first time when the user began providing the verbal utterance, a second timestamp associated with a second time subsequent to the first time when the user began providing one or more specific words or phrases included in the verbal utterance, a third timestamp associated with a third time subsequent to the second time when the user finished providing one or more specific words or phrases included in the verbal utterance, and a fourth timestamp associated with a fourth time subsequent to the third time when the user finished providing the verbal utterance. In this example, the preamble portion of the audio data includes any audio data corresponding to the verbal utterance between the first and second timestamps, the portion of the audio data includes any audio data corresponding to the verbal utterance between the second and third timestamps, and the postamble portion of the audio data includes any audio data corresponding to the verbal utterance between the third and fourth timestamps. It should be noted that this example is provided for illustrative purposes and is not intended to be limiting. In some cases, such as when the verbal utterance includes only one or more of particular words or phrases, the multiple timestamps may include at least a first timestamp associated with a first time when the user began providing the verbal utterance and a second timestamp associated with a second time following the first time when the user finished providing the verbal utterance. In this example, a preamble portion of the audio data or a postamble portion of the audio data may not be present.

[0047] In some implementations, warm word verification engine 153 can process the ASR output to determine whether the user intended one or more particular words or phrases to cause the execution of an assistant command. For example, warm word verification engine 153 can cause NLU engines 130A1 and / or 130A2 to process the ASR output using an NLU model stored in ML model database 115A or grammar-based rules to generate an NLU output. Based on the ASR output and / or the NLU output, warm word verification engine 153 can determine whether the user intended one or more particular words or phrases to cause the execution of an assistant command. Continuing with the above example, responding to a warm word (e.g., capturing the verbal utterance, "reply that I'll be there in five minutes") Continuing with the above example, in response to detecting an occurrence of a warm word (e.g., “reply”) in the stream of audio data, ASR engines 120A1 and / or 120A2 and / or NLU engines 130A1 and / or 130A2 may be activated and utilized to process the stream of audio data to generate an ASR output for the stream of audio data (e.g., capturing the verbal utterance of “reply that I'll be there in five minutes”). Further, NLU engines 130A1 and / or 130A2 may process the ASR output to generate an NLU output using an NLU model stored in ML model database 115A. In this example, warm word verification engine 153 may determine that “reply” was intended by the user to reply to a text message received at client device 110 with the reply message “I'll be there in five minutes.”

[0048] In contrast to the above example, assume that the user of client device 110 instead provides the verbal utterance "I'll reply to that later" when a text message is received at client device 110. In this example, warm word activation event engine 151 can detect the occurrence of a warm word activation event (e.g., a text message is received at the client device) and activate one or more of the currently dormant automated assistant functions that utilize the warm word models. Furthermore, when warm word detection engine 152 processes the stream of audio data using one or more of the warm word models activated for the warm word activation event, it can detect the occurrence of one or more particular words or phrases (e.g., "reply") in the portion of the audio data in the same or similar manner as described above. However, in this example, automated assistant 115 can determine, based on processing the postamble portion of the audio data (e.g., the postamble portion corresponding to "to that later") using ASR engines 120A1 and / or 120A2 and / or NLU engines 130A1 and / or 130A2, that the user did not intend "reply" to be the specific word or phrase that would cause the automated assistant to reply to the received text message. Thus, in this example, automated assistant 115 can refrain from causing execution of an assistant command to reply to the text message, even though the user provided a verbal utterance that includes one or more specific words or phrases (e.g., "reply") that would typically cause an assistant command to be executed.

[0049] In additional or alternative implementations, the automated assistant 115 may execute an assistant command only if the user verification engine 160 determines that the identity of the user who provided the verbal utterance has been verified and that the user who provided the verbal utterance is authorized to cause the execution of the assistant command. In some versions of these implementations, only a subset of assistant commands may require that the identity of the user who provided the verbal utterance has been verified and that the user who provided the verbal utterance is authorized to cause the execution of the assistant command. For example, assistant commands associated with a call warm word activation event, a text message warm word activation event, and an email warm word activation event may require that the identity of the user who provided the verbal utterance has been verified and that the user who provided the verbal utterance is authorized to cause the execution of the assistant command. However, assistant commands associated with a media warm word activation event may not require that the identity of the user who provided the verbal utterance has been verified and that the user who provided the verbal utterance is authorized to cause the execution of the assistant command. The identity of the user who provided the verbal utterance may be verified using various techniques.

[0050] In some versions of these implementations, user verification engine 160 can process the stream of audio data to generate a speaker embedding for the user who provided the oral utterance using a speaker identification (SID) model stored in ML model database 115A. The generated speaker embedding can be compared in the embedding space to one or more previously generated speaker embeddings for the user of client device 110 (e.g., embeddings associated with the user's user profile). For example, if a distance metric determined based on a comparison between the generated speaker embedding and one or more previously generated speaker embeddings for the user in the embedding space meets a threshold, the user can be verified as the user who provided the oral utterance. The one or more previously generated speaker embeddings for the user can be generated based on output generated based on processing one or more instances of audio data including oral utterances from the user (e.g., when initially configuring client device 110). For example, the one or more previously generated speaker embeddings can be based on an average (or other combination) of multiple different embeddings, each generated based on processing a corresponding instance of audio data including a corresponding oral utterance from the user of the client device. In some versions of those implementations, the SID model may be a text-independent (TI) SID model, and in other implementations, the SID model may be a text-dependent (TD) SID model.

[0051] In a TD SID model, one or more previously generated speaker embeddings of a user are generated based on verbal utterances that include only one or more specific words or phrases. Furthermore, during use, a user must speak one or more specific words or phrases for one or more TD SID speaker embeddings to be generated using the TD SID model, which can be effectively compared to one or more previously generated TD speaker embeddings for the user to determine whether the verbal utterances are from an authorized user (e.g., the user of client device 110 or another user associated with client device 110). For example, the one or more specific words or phrases in TD speaker recognition may be constrained to one or more invocation phrases configured to invoke an automated assistant (e.g., "Hey Assistant," "OK Assistant," and / or "Assistant"), or one or more warm words described herein. In contrast, in TI SID, verbal utterances processed using the TI SID model are not constrained to one or more specific words or phrases. In other words, audio data based on virtually any spoken utterance may be processed using the TI SID to generate a TI SID speaker embedding that can be effectively compared to one or more previously generated TI speaker embeddings for the user to determine whether the spoken utterance is from an authorized user (e.g., the user of the client device 110 or another user associated with the client device 110). Furthermore, in various implementations, the one or more previously generated TI speaker embeddings of the user utilized in the TI SID are generated based on spoken utterances that include heterogeneous words and / or phrases, and are not limited to call words and / or phrases, warm words, and / or any other particular spoken utterances.

[0052] In additional or alternative versions of these implementations, other biometric information may be utilized to verify the identity of the user who provided the verbal utterance. The biometric information utilized to verify the identity of the user may include, for example, facial print identification, fingerprint identification, and / or other types of biometric information. For example, the visual component of the presence sensor 113 of the client device 110 may capture visual data including the face of the user who provided the verbal utterance. The user verification engine 160 may process the visual data using a facial identification (FID) model stored in the ML model database 115A to generate a facial print of the user who provided the verbal utterance. The facial print of the user who provided the verbal utterance may correspond to an embedding that, in the embedding space, may be compared with one or more previously generated embeddings corresponding to the facial print of the user of the client device to determine whether the user who provided the verbal utterance is authorized to cause execution of an assistant command. As another example, a fingerprint sensor of presence sensor 113 can receive touch input of a user who provided the verbal utterance and compare the detected fingerprint to known fingerprints of the user of client device 110 to determine whether the user who provided the verbal utterance is authorized to cause execution of an assistant command.

[0053] In additional or alternative implementations, acoustic engine 170 can process the audio data to generate one or more acoustic characteristics associated with the audio data. In these implementations, automated assistant 115 can additionally or alternatively consider one or more of the acoustic characteristics associated with the audio data when determining whether to execute an assistant command in response to detecting a warm word in a stream of audio data. Acoustic engine 170 can utilize various techniques to generate the one or more acoustic characteristics associated with the audio data.

[0054] In some versions of these implementations, acoustic engine 170 may use a voice activity detection (VAD) model stored in ML model database 115A to process the audio data to determine whether a preamble portion of the audio data preceding a portion of the audio data corresponding to one or more particular words or phrases and / or a postamble portion of the audio data following a portion of the audio data corresponding to one or more particular words or phrases is present. In some further versions of these implementations, ASR engine 120A1 and / or 120A2 and / or NLU engine 130A1 and / or 130A2 may be activated only in response to determining that a preamble portion of the audio data and / or a postamble portion of the audio data is present. For example, assume that a user of client device 110 provided the verbal utterance “view” when a notification was received at client device 110. In this example, the warm word activation event engine 151 can detect the occurrence of a warm word activation event (e.g., a notification being received at a client device) and activate one or more of the currently dormant automated assistant functions that utilize the warm word models. Additionally, the warm word detection engine 152 can detect the occurrence of one or more particular words or phrases (e.g., "view") in the portion of the audio data in the same or similar manner as described above when processing the stream of audio data using one or more of the warm word models activated for the warm word activation event.However, acoustic engine 170 may determine, based on processing the stream of audio data using the VAD model, that a preamble portion of the audio data (e.g., preceding "view" in the stream of audio data) or a postamble portion of the audio data (e.g., following "view" in the stream of audio data) is not present. As a result, ASR engine 120A1 and / or 120A2 and / or NLU engine 130A1 and / or 130A2 may not be activated, and automated assistant 115 may have a notification provided for presentation to the user (and optionally, based on determining that the user is authorized to view the notification).

[0055] In additional or alternative versions of these implementations, the acoustic engine 170 can process the audio data to determine the language of the spoken utterances captured in the audio data using a language identification model stored in the ML model database 115A. In some versions of these implementations, the language of the spoken utterances can provide an indication of whether the preamble portion of the audio data or the postamble portion of the audio data provides contextual information that informs the automated assistant 115 whether to trigger an assistant command associated with one or more specific words or phrases corresponding to the portion of the audio data, or to refrain from executing an assistant command associated with one or more specific words or phrases corresponding to the portion of the audio data. For example, in English, certain expressions or phrases indicating refrain from triggering an assistant command tend to precede portions of the audio data (e.g., in the preamble portion of the audio data) that correspond to one or more specific words or phrases (e.g., "won't answer," "can't view," etc.). However, in other languages (e.g., German), specific terms or phrases indicating to refrain from triggering an assistant command tend to follow portions of the audio data (e.g., within the postamble portion of the audio data) that correspond to one or more of the specific words or phrases.

[0056] In additional or alternative implementations, the language of the spoken utterance can provide an indication of which warm word model and / or which ASR model should be utilized to further process the stream of audio data. For example, if the output generated across the language identification model indicates that the user who provided the spoken utterance captured in the stream of audio data is speaking English, the warm word model and / or ASR model subsequently utilized by the automated assistant component may be specialized for English. However, if the output generated across the language identification model indicates that the user who provided the spoken utterance captured in the stream of audio data is speaking German (or another language), the warm word model and / or ASR model subsequently utilized by the automated assistant component may be specialized for German (or another language). In additional or alternative implementations, warm word models trained to detect one or more specific words and / or phrases in various languages may be utilized by one or more active automated assistant components, and which ASR model should subsequently be utilized by one or more of the automated assistant components to further process the stream of audio data may be determined based on the detected language or languages of the specific words and / or phrases. For example, if the output generated across the warm word models indicates that the detected warm words are provided in Spanish, the ASR model subsequently utilized by the automated assistant component may be specialized for Spanish. However, if the output generated across the warm word models indicates that the detected warm words are provided in English (or another language), the ASR model subsequently utilized by the automated assistant component may be specialized for English (or another language).

[0057] In additional or alternative versions of these implementations, the acoustic engine 170 can process the audio data to determine one or more prosodic features of the spoken speech captured in the stream of audio data using a prosodic feature classification model stored in the ML model database 115A. The prosodic features of the spoken speech captured in the stream of audio data can include, for example, one or more of intonation, tone, stress, rhythm, tempo, and pauses. One or more of these prosodic features can inform the automated assistant 115 whether to trigger an assistant command associated with one or more specific words or phrases corresponding to the portion of the audio data or to refrain from executing an assistant command associated with one or more specific words or phrases corresponding to the portion of the audio data. For example, if the spoken speech captured in the audio data does not include any pauses and has additional stress placed in one or more specific words or phrases, the one or more prosodic features of the spoken speech can inform the automated assistant 115 that it is more likely that the user intended the assistant command to be executed. In contrast, if the spoken utterance captured in the audio data includes pauses and uncertain tones, one or more of the prosodic characteristics of the spoken utterance may inform the automated assistant 115 that it is more likely that the user did not intend the assistant command to be executed.

[0058] In additional or alternative implementations, the automated assistant 115 can analyze instances of sensor data generated by one or more of the presence sensors 113. The automated assistant 115 considers the instances of sensor data generated by one or more of the presence sensors 113 when determining whether to cause an assistant command to be executed in response to detecting a warm word in the stream of audio data. For example, if one or more visual components of the client device 110 indicate that the gaze of the user who provided the verbal utterance is not directed toward the client device 110, the visual data can inform the automated assistant 115 that it is more likely that the user did not intend to cause the assistant command to be executed. Additionally or alternatively, if one or more accelerometers of the client device 110 indicate that the client device 110 is moving (and optionally coupled with the user's lack of gaze), the visual data can inform the automated assistant 115 that it is more likely that the user did not intend to cause the assistant command to be executed.

[0059] 2, an example process flow is shown demonstrating various aspects of the present disclosure using various components of FIG. 1. The warm word activation event engine 151 may monitor for the occurrence of a warm word activation event 251. In response to detecting the occurrence of a warm word event, the warm word detection engine 152 may begin processing a stream of audio data 201A generated by one or more microphones 201 of the client device 110 using a warm word model associated with the warm word activation event 251 that is activated and stored in the ML model database 115A. As shown in block 299A, in response to determining that the audio data does not correspond to one or more warm words, the warm word detection engine 152 may continue to monitor for the occurrence of one or more warm words until one or more conditions are met.

[0060] However, as also shown in block 299A, in response to determining that the portion of the audio data corresponds to one or more warm words, warm word detection engine 152 may cause preamble portion 201B of the audio data that precedes the portion of the audio data that corresponds to one or more of the warm words (if any) and / or postamble portion 201C of the audio data that follows the portion of the audio data that corresponds to one or more of the warm words (if any). Further, in response to determining that the portion of the audio data corresponds to one or more of the warm words, warm word detection engine 152 may activate ASR engines 120A1 and / or 120A2 and / or NLU engines 130A1 and / or 130A2. ASR engine 120A1 and / or 120A2 can process preamble portion 201B of audio data and / or postamble portion 201C of audio data (and optionally portions of the audio data corresponding to one or more of the warm words) to generate ASR output 220 (e.g., recognized for any utterances of the user included in the audio data) using an ASR model stored in ML model database 115A. Further, NLU engine 130A1 and / or 130A2 can process ASR output 220 to generate NLU output 230 (e.g., with recognized annotations for any utterances of the user included in the audio data) using an NLU model stored in ML model database 115A. Warm word verification engine 153 can determine whether the user actually intended for automated assistant 115 to cause execution of an assistant command based on ASR output 220 and / or NLU output 230.

[0061] For example, assume that audio data 201A captures the spoken utterance "will you turn the volume up" while listening to music via client device 110 or an additional client device (e.g., a smart speaker) in communication with client device 110. In this example, warm word activation event 251 can correspond to a media event, and the warm word "volume up" can be detected in the audio data capturing the spoken utterance using one or more warm word models associated with the media event. Further, ASR engine 120A1 and / or 120A2 can process preamble portion 201B of the audio data corresponding to "will you turn the" and / or the portion of the audio data corresponding to "volume up" using the ASR models to generate recognized text corresponding to "will you turn the" and / or "volume up" (e.g., as ASR output 220). Further, NLU engines 130A1 and / or 130A2 can process the recognized text corresponding to "will you turn the" and / or "volume up" using an NLU model to generate annotated recognized text (e.g., as NLU output 230) that applies various annotations to one or more terms in the recognized text. In this example, as shown in block 299B, automated assistant 115 can determine, based on the recognized text and / or the annotated recognized text, that assistant command 215A to increase the volume of the playing music should be executed (e.g., by increasing the volume at client device 110 or sending an increase volume request to an additional client device in communication with client device 110) to cause assistant command 215 to be executed.

[0062] In contrast, assume that audio data 201A captures the spoken utterance of "please don't turn the volume up" while listening to music via client device 110 or an additional client device (e.g., a smart speaker) in communication with client device 110. In this example, warm word activation event 251 can correspond to a media event, and the warm word "volume up" can be detected in the audio data capturing the spoken utterance using one or more warm word models associated with the media event. Further, ASR engine 120A1 and / or 120A2 can process preamble portion 201B of the audio data corresponding to "please don't turn the" and / or the portion of the audio data corresponding to "volume up" using the ASR models to generate recognized text corresponding to "please don't turn the" and / or "volume up" (e.g., as ASR output 220). Additionally, NLU engines 130A1 and / or 130A2 can process the recognized text corresponding to "please don't turn the" and / or "volume up" using an NLU model to generate annotated recognized text (e.g., as NLU output 230) that applies various annotations to one or more terms in the recognized text. In this example, as shown in block 299B, automated assistant 115 can determine, based on the recognized text and / or the annotated recognized text, that an assistant command to change the volume of the playing music should not be executed and refrain from executing the assistant command.

[0063] In some implementations, whether automated assistant 115 determines to cause an assistant command to be executed can be based on whether the identity of the user who provided the verbal utterance has been verified and whether the identified user is authorized to cause the execution of the assistant command (e.g., determined using user verification engine 160 as described above with respect to FIGS. 1, 5, and 6A-6C). For example, in an implementation in which the warm word activation event corresponds to an electronic communication (e.g., a phone call, a text message, an email, a social media message, etc.) being received at client device 110, user identification may need to be performed before automated assistant 115 causes the assistant command to be executed. As another example, in an implementation in which the warm word activation event corresponds to a particular device being controlled (e.g., a smart appliance such as a smart lock, a smart oven, a smart garage door opener, etc.), user identification may need to be performed before automated assistant 115 causes the assistant command to be executed.

[0064] While a particular implementation is described with respect to the process flow of FIG. 2 , it should be understood that this is for purposes of example and not meant to be limiting. For example, assume that audio data 201A captures the spoken utterance of "show me directions home." In this example, detected warm word activation event 251 may correspond to a navigation application being accessible at client device 110. Accordingly, one or more warm word models associated with the navigation event may continuously monitor the occurrence of one or more navigation-related warm words. Additionally, the automated assistant may utilize additional or alternative signals in determining whether to execute an assistant command (e.g., as described with respect to acoustic engine 170).

[0065] By using the techniques described herein, one or more technical advantages may be achieved. As one non-limiting example, the techniques described herein enable an automated assistant to determine whether a user actually intended to utter a warm word that would trigger the execution of an assistant command based on processing contextual audio data for the warm word. As a result, the amount of false positives of warm words may be reduced, thereby saving computational and / or network resources at the client device when the automated assistant utilizes one or more networks to execute the assistant command. Furthermore, because the user of the client device does not need to take action to modify or cancel any assistant commands, the amount of user input may be reduced, thereby saving computational resources at the client device. Furthermore, various automated assistant functions described herein can remain dormant until needed, as described herein, thereby saving computational resources at the client device. As another non-limiting example, the techniques described herein enable an automated assistant to verify the identity of a user providing a spoken utterance that includes a warm word using an SID and / or other techniques. As a result, only authorized users can cause certain assistant commands to be executed, thereby increasing data security for data associated with the user of the client device.

[0066] Referring now to FIG. 3 , a flowchart illustrating an example method 300 for determining whether to execute an assistant command based on contextual audio data is shown. For convenience, operations of method 300 are described with reference to a system that performs the operations. This system of method 300 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 of FIGS. 1 , 4A-4C, and 6A-6C, and / or computing device 710 of FIG. 7, one or more servers, and / or other computing devices). Furthermore, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0067] In block 352, the system monitors for the occurrence of warm word activation events. Warm word activation events can include, for example, a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, media playing at the client device or additional client devices in the client device's environment, a notification received at the client device, the location of the client device, a software application accessible at the client device, and / or other events associated with the client device to which a user can provide verbal utterances to control the client device or additional client devices communicating with the client device. Notably, some of these warm word activation events are individual events (e.g., a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, etc.), while others of these warm word activation events are continuous events (e.g., a software application accessible at the client device). Furthermore, it should be noted that these warm word activation events are not mutually exclusive. In other words, the system may detect multiple warm word activation events at a given time instance. If, in an iteration of block 352, the system does not detect the occurrence of a warm word activation event, the system may continue to monitor for the occurrence of a warm word activation event at block 352. If, in an iteration of block 352, the system detects the occurrence of a warm word activation event, the system may proceed to block 354.

[0068] In block 354, the system processes the stream of audio data to monitor the occurrence of one or more particular words and / or phrases (e.g., warm words) using one or more warm word models. In response to detecting the occurrence of a warm word activation event in block 352, the system can trigger one or more automated assistant components that utilize one or more warm word models. The one or more warm word models utilized to process the stream of audio data can be associated with the warm word activation event detected in block 352. For example, if the warm word activation event detected in block 352 corresponds to a phone call being received at the client device, one or more warm word models associated with the phone call event can be utilized to process the stream of audio data. As another example, if the warm word activation event detected in block 352 corresponds to media being played on the client device, one or more warm word models associated with the media event can be utilized to process the stream of audio data.

[0069] In block 356, the system determines whether one or more occurrences of specific words and / or phrases are detected in the stream of audio data. The system may continue monitoring for one or more occurrences of the specific words and / or phrases until one or more conditions are met. The one or more conditions may include, for example, the passage of a threshold duration, detecting one or more specific words or phrases, determining that a warm word activation event has ended, and / or other conditions. If, in a repetition of block 356, the system does not detect one or more occurrences of the specific words and / or phrases, the system may return to block 354 and continue processing the stream of audio data and monitoring for one or more occurrences of the specific words and / or phrases in block 356 until one or more of the conditions are met. If, in a repetition of block 356, the system detects one or more occurrences of the specific words and / or phrases, the system may proceed to block 358.

[0070] In block 358, the system determines whether there is additional voice activity in addition to the warm word (e.g., preceding or following the warm word). The system can use a VAD model to process the stream of audio data to determine whether the stream of audio data captures any additional audio data in addition to the portion of the audio data corresponding to one or more of the specific words and / or phrases. In other words, the system can use the VAD model to determine whether the user provided any additional utterances in addition to one or more of the specific words and / or phrases, such that the audio data corresponding to the additional utterances can be used as contextual audio data (e.g., the preamble portion of the audio data and / or the postamble portion of the audio data) in interpreting one or more of the specific words and / or phrases. If, in a repetition of block 358, the system determines that there is no additional voice activity in addition to the warm word, the system proceeds to block 366 and causes the automated assistant to execute an assistant command associated with one or more of the specific words and / or phrases. Notably, when proceeding to block 366, the system may bypass the ASR and / or NLU. If, in any iteration of block 358, the system determines that additional voice activity is present in addition to the warm word, the system proceeds to block 360. In some implementations, the operations of block 358 may be omitted and the system may proceed directly to block 360.

[0071] In block 360, the system uses the ASR model to process the preamble portion of the audio data and / or the postamble portion of the audio data to generate an ASR output. Additionally or alternatively, the system can use the ASR model to process portions of the audio data corresponding to warm words along with the preamble portion of the audio data and / or the postamble portion of the audio data (e.g., as a single instance of the audio data) in generating the ASR output. One or more automated assistant components utilizing the ASR model may be activated in response to detecting one or more occurrences of particular words and / or phrases in block 356 and / or in response to determining that additional voice activity is present in block 358. In various implementations, the ASR output may be biased toward one or more of the particular words and / or phrases associated with the warm word model. In block 362, the system uses the NLU model to process the ASR output to generate an NLU output. Similar to what is described above with respect to block 360, one or more automated assistant components utilizing an NLU model may be activated in response to detecting one or more occurrences of particular words and / or phrases in block 356 and / or in response to determining that there is additional voice activity in block 358.

[0072] In block 364, the system determines whether the user who provided the verbal utterance containing the warm word and captured in the stream of audio data intended the assistant command associated with the warm word detected in block 356 to be executed by the automated assistant. The system can determine whether the user intended the assistant command to be executed by the automated assistant based on the ASR output generated in block 360 and / or the NLU output generated in block 362. If, in the iteration of block 364, the system determines that the user intended the assistant command to be executed by the automated assistant, the system proceeds to block 366. In block 366, the system causes the automated assistant to execute the assistant command associated with the warm word detected in block 356. If, in the iteration of block 364, the system determines that the user did not intend the assistant command to be executed by the automated assistant, the system proceeds to block 368. In block 368, the system refrains from causing the automated assistant to execute the assistant command associated with the warm word detected in block 356.

[0073] 4A, 4B, and 4C, various non-limiting examples are shown that utilize the method 300 of FIG. 3 to determine whether to execute an assistant command based on contextual audio data. The automated assistant may be implemented at least in part in the client device 110 (e.g., the automated assistant 115 described with reference to FIG. 1). The automated assistant may utilize the warm word system 180 (e.g., the warm word system 180 described with reference to FIG. 1) to determine whether to execute an assistant command based on contextual audio data (e.g., as described with reference to the method 300 of FIG. 3).

[0074] 4A, 4B, and 4C may include various user interface components including, for example, a microphone that generates audio data based on oral speech and / or other audible input, a speaker that audibly renders synthesized speech and / or other audible output, and a display 190 that receives touch input and / or visually renders transcription and / or other visual output. Additionally, the display 190 of the client device 110 includes various system interface elements (e.g., hardware and / or software interface elements) that may be interacted with by a user of the client device 110 to cause the client device 110 to perform one or more actions (e.g., as described with respect to FIGS. 6B and 6C). While the client device 110 shown in FIGS. 4A-4C is a mobile device, it should be understood that this is for purposes of example and not meant to be limiting.

[0075] 4A , assume that user 101 associated with client device 110 is co-located with additional user 102 in environment 400. Further assume that a call is received at client device 110. In this example, the automated assistant can determine that the call being received at client device 110 corresponds to a warm word activation event. As a result, the automated assistant can activate one or more currently dormant automated assistant functions that utilize one or more warm word models associated with the call event. The one or more warm word models associated with the call event can be utilized to process a stream of audio data generated by the microphone of client device 110 to monitor occurrences of specific words and / or phrases associated with the call event (e.g., “answer,” “decline,” “screen call,” “send to voicemail”) that, when detected, cause the automated assistant to perform a specific action associated with the specific words and / or phrases.

[0076] For example, as shown in FIG. 4A , assume that user 101 of client device 110 provides verbal utterance 452A of “Answer” while client device 110 is ringing. In this example, when the automated assistant processes a stream of audio data generated by the microphone of client device 110 using one or more warm word models associated with the call event, it can determine that the audio data corresponds to the warm word “Answer,” which, when detected, causes the automated assistant to answer the call on behalf of user 101. Further, the automated assistant can have the stream of audio data processed using a VAD model to determine whether any additional voice activity is present. In this example, the output generated using the VAD model would indicate the absence of additional voice activity, and consequently, the absence of contextual audio data to be processed to determine whether user 101 actually intended verbal utterance 452A to answer the call. As a result, the automated assistant will cause the call to be answered at client device 110.

[0077] In contrast, as shown in FIG. 4B , assume that while client device 110 is ringing, user 101 of client device 110 provides verbal utterance 452B of "I won't answer it now, I'll call them back later." In this example, when the automated assistant processes a stream of audio data generated by the microphone of client device 110 using one or more warm word models associated with the call event, it can determine that a portion of the audio data corresponds to the warm word "Answer," which, when detected, causes the automated assistant to answer the call on behalf of user 101. Furthermore, the automated assistant can have the stream of audio data processed using a VAD model to determine whether any additional voice activity is present. In this example, the output generated using the VAD model would indicate the presence of additional voice activity, and consequently, the presence of contextual audio data that should be processed to determine whether user 101 actually intended verbal utterance 452B to answer the call. As a result, the automated assistant may retrieve the preamble portion of the audio data and / or the postamble portion of the audio data.

[0078] In this example, the preamble portion of the audio data corresponds to "I won't" in spoken utterance 452B. The automated assistant may have the preamble portion of the audio data processed using an ASR model to generate an ASR output for the preamble portion of the audio data. Further, the automated assistant may have the ASR output processed using an NLU model to generate an NLU output for the preamble portion of the audio data. Based on the ASR output and / or the NLU output for the preamble portion of the audio data, the automated assistant may determine whether user 101 actually intended spoken utterance 452B to answer the call. For example, based on the ASR output and / or the NLU output, the automated assistant may determine that "won't" negates the warm word "Answer." Thus, in this example, the automated assistant would refrain from answering the call at client device 110 based on processing the preamble portion of the audio data. In various implementations, one or more negative warm word models may be additionally or alternatively utilized in addition to the warm word models described herein. In these implementations, the one or more negative warm word models may be utilized to detect other words and / or phrases (e.g., "won't," "can't," "don't," etc.) proximate to the warm word that negate the warm word. Furthermore, if one or more of these negative warm word models detect other words and / or phrases proximate to the warm word, one or more automated assistant components utilizing the ASR model and / or NLU model may remain dormant.

[0079] In some implementations, assuming the automated assistant cannot determine whether user 101 actually intended verbal utterance 452B to answer the call based on processing the preamble portion of the audio data, the automated assistant can further cause the postamble portion of the audio data to be processed using an ASR model to generate an ASR output for the postamble portion of the audio data (e.g., corresponding to verbal utterance 452B, "I'll call them back later"). Further, the automated assistant can cause the ASR output to be processed using an NLU model to generate an NLU output for the postamble portion of the audio data. The automated assistant can determine whether user 101 actually intended verbal utterance 452B to answer the call based on the ASR output and / or the NLU output for the postamble portion of the audio data. For example, the automated assistant may determine, based on the ASR output and / or the NLU output, that "I'll call them back later" indicates that the user 101 does not intend for the call to be answered at this time instance, such that the postamble portion of the audio data negates the warm word "Answer." Thus, in this example, the automated assistant will refrain from answering the call at the client device 110 based on further processing the postamble portion of the audio data.

[0080] As another example, as shown in FIG. 4C , assume that while the client device 110 is ringing, an additional user 102 co-located in the environment of the user 101 associated with the client device 110 provides the spoken utterance 452C of “Are you going to answer that?” In this example, when the automated assistant processes a stream of audio data generated by the microphone of the client device 110 using one or more warm word models associated with the call event, it can determine that the audio data corresponds to the warm word “Answer,” which, when detected, causes the automated assistant to answer the call on behalf of the user 101. Further, the automated assistant can have the stream of audio data processed to determine whether any additional voice activity is present. In this example, the output generated using the VAD model would indicate the presence of additional voice activity, and consequently, the presence of contextual audio data to be processed. However, in contrast to the examples of FIGS. 4A and 4B , the spoken utterance 452C is provided by the additional user 102, not by the user 101 associated with the client device 110.

[0081] 5 and 6A-6C, the identity of the user who provided the verbal utterance 452A, 452B, and / or 452C may need to be verified before the automated assistant causes the assistant command to be executed. Further, the automated assistant may additionally or alternatively need to determine that the identified user is authorized to cause the execution of the assistant command. These assistant commands for which the automated assistant needs to verify the identity of the user who provided the verbal utterance and / or determine whether the user is authorized may be assistant commands associated with a particular warm word model, assistant commands associated with a particular software application accessible at client device 110, assistant commands associated with a particular device to be controlled based on the assistant command, and / or other subsets of assistant commands.

[0082] 4C , the automated assistant may still have the preamble portion of the audio data and / or the postamble portion of the audio data obtained. In other words, the automated assistant may not attempt to verify the identity of the additional user 102 who provided the verbal utterance 452C if the utterance is not intended to cause an assistant command to be executed. In this example, the preamble portion of the audio data corresponds to the verbal utterance 452C, "Are you going to." The automated assistant may have the preamble portion of the audio data processed in the same or similar manner as described above (and optionally in combination with portions of the audio data corresponding to one or more of the particular terms or phrases and / or the postamble portion of the audio data (e.g., as a single instance of the audio data or multiple disparate instances of the audio data)) to generate ASR and NLU outputs for the preamble portion of the audio data. Further, the automated assistant can determine, based on the ASR output and / or NLU output for the preamble portion of the audio data, whether the additional user 102 actually intended the verbal utterance 452C to answer the call. For example, the automated assistant can determine that "Are you going to," based on the ASR output and / or NLU output, corresponds to a question posed by the additional user 102 to the user 101 associated with the client device 110 and does not intend to answer the call. Thus, in this example, the automated assistant would refrain from answering the call at the client device 110 based on processing the preamble portion of the audio data (and optionally, further processing the postamble audio data corresponding to "that" in verbal utterance 452C).

[0083] 4A-4C are described with respect to particular warm words that cause particular assistant commands to be executed, it should be understood that this is for purposes of example and not meant to be limiting. Rather, the techniques described with respect to Figures 4A-4C can be utilized to detect any warm word activation event, detect any warm word using the warm word model associated with the warm word activation event, and cause any assistant command to be executed.

[0084] Referring now to FIG. 5, a flowchart illustrating an example method 500 for determining whether to execute an assistant command based on speaker identification is shown. For convenience, the operations of method 500 are described with reference to a system that performs the operations. This system of method 500 includes one or more processors, memory, and / or other components of a computing device (e.g., client device 110 of FIGS. 1, 4A-4C, and 6A-6C, and / or computing device 710 of FIG. 7, one or more servers, and / or other computing devices). Furthermore, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0085] In block 552, the system monitors for the occurrence of a warm word activation event. If in the iteration of block 552, the system does not detect the occurrence of a warm word activation event, the system may continue to monitor for the occurrence of a warm word activation event in block 552. If in the iteration of block 552, the system detects the occurrence of a warm word activation event, the system may proceed to block 554. In block 554, the system processes the stream of audio data to monitor for the occurrence of one or more particular words and / or phrases (e.g., warm words) using one or more warm word models. In block 556, the system determines whether one or more occurrences of the particular words and / or phrases have been detected in the stream of audio data. If in the iteration of block 556, the system does not detect the occurrence of one or more of the particular words and / or phrases, the system may return to block 554 and continue processing the stream of audio data and monitoring for the occurrence of one or more of the particular words and / or phrases in block 556 until one or more of the conditions are met. If, in the iteration of block 556, the system detects one or more occurrences of particular words and / or phrases, the system proceeds to block 558. The operations of blocks 552, 554, and 556 may be performed in the same or similar manner as described with respect to blocks 352, 354, and 356, respectively, of method 300 of FIG.

[0086] In block 558, the system determines whether to verify the identity of a user who provided a verbal utterance captured in the stream of audio data that includes one or more of the particular words and / or phrases. The system can determine whether to verify the identity of a user who provided the verbal utterance based on whether one or more of the particular words and / or phrases are associated with a particular warm word model, a particular software application accessible on the user's client device, a particular device to be controlled based on the Assistant command, and / or other subset of the Assistant command. For example, in an implementation in which an Assistant command causes media playing on a client device or a smart light to be controlled, the user's identity may not need to be verified. However, in an implementation in which an Assistant command causes a reply to an incoming electronic communication (e.g., a call, a text message, an email, a social media message, etc.) or a smart lock to be controlled, the user's identity may need to be verified. If, in a repetition of block 558, the system determines that the identity of a user who provided the verbal utterance does not need to be verified, the system can proceed to block 358 of FIG. 3 and continue executing method 300 of FIG. 2. If, in the iteration of block 558 , the system determines that the identity of the user who provided the verbal utterance needs to be verified, the system can proceed to block 560 .

[0087] At block 560, the system determines whether the audio data contained within the stream of audio data is sufficient to perform TI SID. The system may determine whether the audio data contained within the stream of audio data is sufficient to perform TI SID by comparing the length of the verbal utterance captured within the audio data to a threshold length. The system may determine that the audio data is sufficient to perform TI SID if the verbal utterance meets the threshold length. In the iteration of block 560, if the system determines that the audio data is sufficient to perform TI SID, the system may proceed to block 562. At block 562, the system processes the audio data to verify the identity of the user using the TI SID model. At block 566, from block 562, the system determines whether the identity of the user has been verified. For example, the system may process the audio data using the TI SID model to generate a TI speaker embedding for the user who provided the verbal utterance. Further, the system may compare the TI speaker embedding for the user who provided the verbal utterance to one or more previously generated TI speaker embeddings for one or more users of the client device in the embedding space. Further, the system can determine whether the TI speaker embedding corresponds to one or more of the previously generated TI speaker embeddings based on the distance metric determined based on the comparison. If, in the iteration of block 566, from block 562, the system determines that the distance metric meets the distance threshold, the identity of the user who provided the verbal utterance can be verified as the user associated with the client device, and, assuming the identified user is authorized to cause execution of assistant commands, in block 568 the system can cause the automated assistant to execute the assistant command.However, if, in the iteration of block 566, from block 562 the system determines that the distance metric does not meet the distance threshold, then in block 570 the system generates and provides a prompt for further verification of the user's identity (e.g., requesting that the user who provided the verbal utterance provide faceprint or fingerprint identification) in order to have the automated assistant execute the assistant command.

[0088] If, in the iteration of block 560, the system determines that the audio data is not sufficient to perform TI SID, the system may proceed to block 564. In block 564, the system processes the audio data to verify the user's identity using the TD SID model. In block 566, from block 562, the system determines whether the user's identity has been verified. For example, the system may process the audio data using the TD SID model to generate a TD speaker embedding for the user who provided the oral utterance. Further, the system may compare the TD speaker embedding for the user who provided the oral utterance with one or more previously generated TI speaker embeddings for one or more users of the client device in the embedding space. In particular, the TD speaker embedding corresponds to a speaker embedding for one or more particular words and / or phrases (e.g., portions of the audio data corresponding to the warm words verified in block 556). Further, the system may determine whether the TD speaker embedding corresponds to one or more previously generated TD speaker embeddings for one or more of the particular words and / or phrases based on a distance metric determined based on the comparison. In the iteration of block 566, if the system determines from block 564 that the distance metric meets the distance threshold, the identity of the user who provided the verbal utterance may be verified as a user associated with the client device, and assuming the identified user is authorized to cause execution of the assistant command, in block 568 the system may cause the automated assistant to execute the assistant command.However, if, in the iteration of block 566, from block 564, the system determines that the distance metric does not meet the distance threshold, then in block 570, the system may generate and provide a prompt for additional verification of the user's identity (e.g., requesting the user who provided the verbal utterance to provide an example utterance for TI SID verification or TD SID verification, and / or requesting the user who provided the verbal utterance to provide faceprint or fingerprint identification) in order to have the automated assistant execute the assistant command.

[0089] In particular, while FIG. 5 is described with respect to separately executing the TI SID and the TD SID to verify the identity of the user who provided the verbal utterance, it should be understood that this is for purposes of example and not meant to be limiting. For example, in implementations where the verbal utterance is long enough to perform the TI SID, both the TI SID and the TD SID may be performed to verify the identity of the user who provided the verbal utterance. As another example, in various implementations, only the TD SID may be performed if it is determined that one or more of the previously generated TD speaker embeddings correspond to one or more particular words and / or phrases captured within the verbal utterance. Furthermore, these techniques may be utilized to verify portions of verbal utterances captured within a stream of audio data provided by the same user. For example, assume that a stream of audio data captures a first verbal utterance from a first user of "don't" and a second verbal utterance from a second user of "answer." Further assume that a second user is associated with the client device and is an authorized response, and a TD speaker embedding for the second user for the verbal utterance of "answer" has previously been generated and stored at the client device. In this case, the TD speaker embedding generated using the TD SID model can indicate that the second user is a verified user authorized to trigger the execution of an assistant command to answer an incoming call at the client device. Furthermore, multiple TI speaker embeddings for the verbal utterances "don't" and "answer" can indicate that different users provided the verbal utterances captured in the stream of audio data. Thus, in this example, the system can cause an automated assistant to answer the incoming call.

[0090] 6A, 6B, and 6C, various non-limiting examples are shown that utilize method 500 of FIG. 5 to determine whether to execute an assistant command based on speaker identification. The automated assistant may be implemented at least in part in client device 110 (e.g., automated assistant 115 described with respect to FIG. 1). The automated assistant may utilize warm word system 180 (e.g., warm word system 180 described with respect to FIG. 1) to determine whether to execute an assistant command based on contextual audio data (e.g., as described with respect to method 500 of FIG. 5).

[0091] 6A-6C may include various user interface components, including, for example, a microphone that generates audio data based on oral speech and / or other audible input, a speaker that audibly renders synthesized speech and / or other audible output, and a display 190 that receives touch input and / or visually renders transcription and / or other visual output. Additionally, as shown in Figures 6B and 6C, display 190 of client device 110 includes various system interface elements 191, 192, and 193 (e.g., hardware and / or software interface elements) that may be interacted with by a user of client device 110 to cause client device 110 to perform one or more actions. The display 190 of the client device 110 allows the user to interact with the content rendered on the display 190 by touch input (e.g., by pointing user input at the display 190 or a portion thereof (e.g., a text entry box 194 or other portion of the display 190)) and / or by verbal input (e.g., by selecting a microphone interface element 195 or by simply speaking without necessarily selecting the microphone interface element 195) (i.e., the automated assistant may monitor one or more particular terms or phrases, gestures, gaze, mouth movements, lip movements, and / or other conditions that activate verbal input at the client device 110). While the client device 110 shown in FIGS. 6A-6C is a mobile device, it should be understood that this is for purposes of example and not meant to be limiting.

[0092] 6A , assume that user 101 associated with client device 110 is co-located with additional user 102 in environment 600. Further assume that a call is received at client device 110. In this example, the automated assistant may determine that the call being received at client device 110 corresponds to a warm word activation event. As a result, the automated assistant may activate one or more currently dormant automated assistant functions that utilize one or more warm word models associated with the call event. The one or more warm word models associated with the call event may be utilized to process a stream of audio data generated by the microphone of client device 110 to monitor occurrences of specific words and / or phrases associated with the call event (e.g., “answer,” “decline,” “screen call,” “send to voicemail”) that, when detected, cause the automated assistant to perform a specific action associated with the specific words and / or phrases.

[0093] For example, as shown in FIG. 6A , assume that an additional user 102 provides a spoken utterance 652A of “Answer” while the client device 110 is ringing. Notably, in contrast to FIG. 4A , the spoken utterance 652A of “Answer” is provided by the additional user 102 and not by the user 101 associated with the client device 110. In this example, when the automated assistant processes the stream of audio data generated by the microphone of the client device 110 using one or more warm word models associated with the call event, it can determine that the audio data corresponds to the warm word of “Answer,” which, when detected, causes the automated assistant to answer the call. Further, assume that the identity of the user (e.g., the additional user 102) who provided the spoken utterance 652A needs to be verified in order to cause the automated assistant to answer the call.

[0094] In this example, the automated assistant can determine whether the verbal utterance 652A is long enough to perform a TI SID to verify the identity of the user who provided the verbal utterance. In this example, we further assume that the verbal utterance 652A is not long enough to perform a TI SID. However, the automated assistant can attempt to verify the identity of the user who provided the verbal utterance 652A using a TD SID (and assume that one or more previously generated TD speaker embeddings for the "Answer" exist, regardless of whether a TI SID is performed). For example, the automated assistant can use the TD SID model to process portions of the audio data corresponding to one or more particular words and / or phrases to generate a TD speaker embedding for the "Answer." Furthermore, the automated assistant can compare the TD speaker embedding for the "Answer" to one or more previously generated speaker embeddings for the "Answer" in the embedding space. Further, the automated assistant can determine whether the user who provided the spoken utterance 652A is user 101 of client device 110 based on the distance metric determined based on the comparison. In this example, the automated assistant determines that the TD speaker embedding for "Answer" does not match one or more previously generated speaker embeddings for "Answer." Therefore, because the automated assistant did not verify the identity of the user (e.g., additional user 102) who provided the spoken utterance 652A, the automated assistant may refrain from executing an assistant command to answer the call on behalf of user 101.

[0095] However, in various implementations, the automated assistant can generate a prompt to be provided for presentation to user 101 (e.g., visually and / or audibly) that requests the user to provide additional verification to have the assistant command executed. For example, with particular reference to FIG. 6B , the automated assistant can generate prompt 652B of “please say [example utterance] to verify your identity and answer the incoming call” and provide the prompt for visual presentation to the user via automated assistant application 198A accessible at client device 110. Thus, user 101 of client device 110 can provide additional user input in response to the prompt to verify whether user 101 actually intended for the automated assistant to answer the call. In this example, the automated assistant can request that user 101 provide an example utterance long enough to perform a TI SID on the example utterance, or an utterance corresponding to another particular term or phrase for which client device 110 has access to one or more previously generated TD speaker embeddings. Assuming that the user 101 of the client device 110 did not provide a verbal utterance to cause the automated assistant to answer the call (e.g., in the example of FIG. 6A), the user 101 may ignore or dismiss the prompt 652B. However, assuming that the user 101 of the client device 110 provided an utterance to cause the automated assistant to answer the call (e.g., in the example of FIG. 4A), the user 101 may provide an exemplary utterance.

[0096] As another example, with particular reference to FIG. 6C , the automated assistant may generate prompt 652C, “Please provide face identification or fingerprint identification to answer the incoming call,” and provide the prompt for visual presentation to the user via automated assistant application 198A accessible at client device 110. Accordingly, user 101 of client device 110 may provide additional user input in response to the prompt to verify whether user 101 actually intended for the automated assistant to answer the call. In this example, the automated assistant may request that user 101 provide this biometric identification if it is unable to identify the identity of the user who provided the verbal utterance. In some cases, this biometric information may only be requested in response to determining that the TD SID and TD SID are insufficient to identify the user who provided the verbal utterance. Assuming that user 101 of client device 110 did not provide a verbal utterance to have the automated assistant answer the call (e.g., in the example of FIG. 6A ), user 101 may ignore or dismiss prompt 652C. However, assuming that the user 101 of the client device 110 provides an utterance to cause the automated assistant to answer the call (e.g., in the example of FIG. 4A), the user 101 may provide biometric information.

[0097] While prompts 652B and 652C shown in Figures 6B and 6C are provided for presentation to user 101 via automated assistant application 198A accessible at client device 110, it should be understood that this is for purposes of example and not meant to be limiting. For example, promotional advertisements 652B and 652C may additionally or alternatively be rendered audibly for presentation to the user via speakers of client device 110 and visually rendered as notifications (e.g., pop-up notifications, banner notifications, etc.) for presentation to the user via display 190 of client device 110. Furthermore, while Figures 6A-6C are described with respect to particular warm words that cause particular assistant commands to be executed, it should be understood that this is for purposes of example and not meant to be limiting. Rather, the techniques described with respect to Figures 6A-6C may be utilized to detect any warm word activation event, detect any warm word using a warm word model associated with the warm word activation event, and cause any assistant command to be executed once the identity of the user providing the verbal utterance is verified as an authorized user.

[0098] 7, shown is a block diagram of an example computing device 710 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device, cloud-based automated assistant component, and / or other component may comprise one or more components of the example computing device 710.

[0099] Computing device 710 typically includes at least one processor 714 that communicates with several peripheral devices via a bus subsystem 712. These peripheral devices may include, for example, a storage subsystem 724, including a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with computing device 710. Network interface subsystem 716 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0100] The user interface input devices 722 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, an input device such as a scanner, a touch screen integrated into a display, a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 710 or a communications network.

[0101] The user interface output devices 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to encompass all possible types of devices and methods for outputting information from the computing device 710 to a user or to another machine or computing device.

[0102] Storage subsystem 724 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic for performing selected aspects of the methods disclosed herein and for implementing the various components shown in FIGS. 1 and 2.

[0103] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used within the storage subsystem 724 may include several memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution, and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation may be stored by the file storage subsystem 726 within the storage subsystem 724 or in other machines accessible by the processor 714.

[0104] The bus subsystem 712 provides a mechanism for allowing the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem 712 may use multiple buses.

[0105] The computing device 710 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 710 shown in Figure 7 is intended only as a specific example for purposes of describing some implementations. Many other configurations of the computing device 710 can have more or fewer components than the computing device shown in Figure 7.

[0106] In situations where the systems described herein may collect or otherwise monitor personal information about users or utilize personal and / or monitored information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from content servers that may be more relevant to the user. Also, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identification information may be processed so that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized where the geographic location information is obtained (e.g., to the city, zip code, or state level) so that the user's specific geographic location cannot be identified. Thus, users may control how information is collected and / or used about them.

[0107] In some implementations, a method implemented by one or more processors is provided, comprising: processing a stream of audio data using a warm word model to monitor occurrences of one or more specific words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, each of the one or more specific words or phrases being associated with an assistant command; processing a preamble portion of the audio data and / or a postamble portion of the audio data using an automatic speech recognition (ASR) model to generate an ASR output in response to determining that a portion of the audio data corresponds to one or more of the specific words or phrases; and determining, based on processing the ASR output, whether the user intended the one or more specific words or phrases to cause execution of the assistant command. The preamble portion of the audio data precedes the portion of the audio data corresponding to the one or more specific words or phrases, and the postamble portion of the audio data follows the portion of the audio data corresponding to the one or more specific words or phrases. The method further includes, in response to determining that the user did not intend the one or more particular words or phrases to cause execution of the assistant command associated with the one or more of the particular words or phrases, refraining from causing the automated assistant to execute the assistant command associated with the one or more of the particular words or phrases, and, in response to determining that the user did intend the one or more particular words or phrases to cause execution of the assistant command associated with the one or more of the particular words or phrases, causing the automated assistant to execute the assistant command associated with the one or more of the particular words or phrases.

[0108] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0109] In some implementations, the method may further include detecting the occurrence of a warm word activation event and activating one or more currently dormant automated assistant functions that utilize the warm word model in response to detecting the occurrence of the warm word activation event. Processing the stream of audio data using the warm word model to monitor the occurrence of one or more specific words or phrases may respond to activating one or more currently dormant automated assistant functions that utilize the warm word model. In some versions of these implementations, the warm word activation event may include one or more of a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm sounding at the client device, a timer sounding at the client device, media playing at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

[0110] In some implementations, determining whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with the one or more particular words or phrases based on processing the ASR output may include processing the ASR output using a natural language understanding (NLU) model to generate an NLU output, and determining, based on the NLU output, whether the user intended the one or more particular words or phrases to cause execution of an assistant command. The ASR output may be generated based on a preamble portion of the audio data but not based on a postamble portion of the audio data. In some versions of these implementations, the method may further include, in response to determining that the NLU output is insufficient to determine whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases, processing a postamble portion of the audio data using an ASR model to generate an additional ASR output, and determining, based on processing the additional ASR output, whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases.

[0111] In some implementations, determining whether the user intended one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases based on processing the ASR output may include processing the ASR output using a natural language understanding (NLU) model to generate an NLU output, and determining, based on the NLU output, whether the user intended the one or more particular words or phrases to cause execution of the assistant command. The ASR output may be generated based on both a preamble portion of the audio data and a postamble portion of the audio data. In some versions of these implementations, the method may further include, in response to determining that the NLU output is insufficient to determine whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases, processing an additional postamble portion of the audio data using an ASR model to generate an additional ASR output, and determining, based on processing the additional ASR output, whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases. An additional postamble portion of the audio data may follow the postamble portion of the audio data.

[0112] In some implementations, the method may further include processing the stream of audio data using the end-pointing model to generate multiple timestamps for oral utterances captured within the stream of audio data and including one or more specific words or phrases. In some versions of these implementations, the multiple timestamps may include at least a first timestamp associated with a first time when the user began providing the oral utterance, a second timestamp associated with a second time subsequent to the first time when the user began providing one or more specific words or phrases included within the oral utterance, a third timestamp associated with a third time subsequent to the second time when the user finished providing one or more specific words or phrases included within the oral utterance, and a fourth timestamp associated with a fourth time subsequent to the third time when the user finished providing the oral utterance. In some further versions of these implementations, the preamble portion of the audio data may include any audio data corresponding to the oral utterance between the first timestamp and the second timestamp. In additional or alternative versions of these implementations, the postamble portion of the audio data may include any audio data corresponding to the oral utterance between the third timestamp and the fourth timestamp.

[0113] In some implementations, the method may further include activating one or more currently dormant automated assistant functions that utilize the ASR model in response to determining that the spoken utterance includes one or more of the particular words or phrases.

[0114] In some implementations, the method may further include processing portions of the audio data corresponding to one or more particular words or phrases, along with the preamble and / or postamble portion audio data of the audio data, using the ASR model to generate an ASR output.

[0115] In some implementations, the method may further include processing the stream of audio data to monitor for occurrences of additional voice activity using a voice activity detection (VAD) model before the portion of the audio data corresponds to one or more of the particular words or phrases and / or after the portion of the audio data corresponds to one or more of the particular words or phrases, and activating one or more currently dormant automated assistant functions that utilize the ASR model in response to detecting the occurrences of the additional voice activity. In some versions of these implementations, the method may further include causing the automated assistant to execute assistant commands associated with one or more of the particular words or phrases in response to not detecting the occurrences of the additional voice activity.

[0116] In some implementations, the method may further include processing the audio data to determine one or more prosodic characteristics of the oral utterance using an acoustic model. Determining whether the user intended the one or more particular words or phrases to cause execution of an assistant command associated with one or more of the particular words or phrases may be further based on the one or more prosodic characteristics of the oral utterance.

[0117] In some implementations, the method may further include receiving an instance of sensor data from one or more sensors of the client device. Determining whether the one or more particular words or phrases meant that the user intended to cause execution of an assistant command associated with one or more of the particular words or phrases may be further based on the instance of sensor data. In some versions of these implementations, the instance of sensor data may include one or more of accelerometer data generated by an accelerometer of the client device, image data generated by one or more camera components of the client device, or motion data generated by one or more motion sensors of the client device.

[0118] In some implementations, the method may further include obtaining a preamble portion of the audio data from an audio buffer of the client device and / or obtaining a postamble portion of the audio data from the stream of audio data.

[0119] In some implementations, a method is provided, implemented by one or more processors, comprising: processing a stream of audio data using a warm word model to monitor occurrences of one or more particular words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, each of the one or more particular words or phrases being associated with an assistant command; and, in response to determining that a portion of the audio data corresponds to one or more of the particular words or phrases and in response to determining to verify the identity of a user who provided an oral utterance corresponding to the audio data, using a speaker identification (SID) model to identify the user who provided the oral utterance. The method includes processing the audio data to determine whether the data is sufficient; causing the automated assistant to execute an assistant command associated with one or more of the specific words or phrases in response to determining that the audio data is sufficient to identify a user who provided the verbal utterance and in response to determining that the user who provided the verbal utterance is authorized to cause execution of an assistant command associated with one or more of the specific words or phrases; generating a prompt requesting further verification of the identity of the user who provided the verbal utterance in response to determining that the audio data is insufficient to identify the user who provided the verbal utterance; and causing the prompt to be provided for presentation at the client device.

[0120] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0121] In some implementations, the method may further include detecting the occurrence of a warm word activation event and activating one or more currently dormant automated assistant functions that utilize the warm word model in response to detecting the occurrence of the warm word activation event. Processing the stream of audio data using the warm word model to monitor the occurrence of one or more specific words or phrases may respond to activating one or more currently dormant automated assistant functions that utilize the warm word model. In some versions of these implementations, the warm word activation event may include one or more of a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm sounding at the client device, a timer sounding at the client device, media playing at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

[0122] In some implementations, the method may further include, in response to determining that the portion of the audio data corresponds to one or more of the particular words or phrases, and in response to determining not to verify the identity of the user who provided the verbal utterance corresponding to the audio data, causing the automated assistant to execute an assistant command associated with the one or more of the particular words or phrases.

[0123] In some implementations, processing the audio data to determine whether the audio data is sufficient to identify the identity of the user who provided the oral utterance includes processing a portion of the audio data corresponding to one or more of the particular words or phrases to generate a TD speaker embedding using a text-dependent (TD) SID model as the SID model. Determining whether the audio data is sufficient to identify the user who provided the oral utterance may include comparing, in the embedding space, the TD speaker embedding to one or more stored TD speaker embeddings stored locally at the client device, and determining whether the portion of the audio data is sufficient to verify the identity of the user who provided the oral utterance based on comparing the TD speaker embedding to the one or more stored TD speaker embeddings.

[0124] In some versions of those implementations, processing the portion of the audio data corresponding to one or more of the particular words or phrases to generate a TD speaker embedding using the TD SID model may include determining whether one or more of the stored TD speaker embeddings correspond to the particular words or phrases. Processing the portion of the audio data corresponding to one or more of the particular words or phrases to generate a TD speaker embedding using the TD SID model is responsive to determining that one or more of the stored TD speaker embeddings correspond to the particular words or phrases.

[0125] In additional or alternative versions of these implementations, generating a prompt requesting further verification of the identity of the user who provided the verbal utterance may include generating a prompt requesting the user to provide the additional verbal utterance, the additional verbal utterance being of sufficient length to execute a TI SID for use in verifying the identity of the user who provided the verbal utterance.

[0126] In additional or alternative versions of those implementations, the method may further include, in response to determining that the oral utterance is long enough to perform a TI SID, processing the stream of audio data to generate a TI speaker embedding using the TI SID model as the SID model. Determining whether the audio data is sufficient to identify a user who provided the oral utterance may include comparing, in embedding space, the TI speaker embedding to one or more stored TI speaker embeddings stored locally at the client device, and determining whether the stream of audio data is sufficient to verify the identity of the user who provided the oral utterance based on comparing the TI speaker embedding to the one or more stored TI speaker embeddings. In some further versions of those implementations, generating a prompt requesting additional verification of the identity of the user who provided the oral utterance may include generating a prompt requesting additional biometric information to verify the user's identity. Requesting additional biometric information to verify the user's identity may include requesting one or more of a fingerprint identification or a faceprint fingerprint identification.

[0127] In some implementations, the method may further include receiving additional user input in response to providing the prompt for presentation at the client device, and determining, in response to receiving the additional user input, whether the additional user input is sufficient to identify a user who provided the verbal utterance. In some versions of these implementations, the method may further include causing the automated assistant to execute a corresponding assistant command in response to determining that the additional user input is sufficient to identify a user who provided the verbal utterance.

[0128] Additionally, some implementations include one or more processors (e.g., central processing unit (CPU), graphics processing unit (GPU), and / or tensor processing unit (TPU)) of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the aforementioned methods. Some implementations also include computer program products that include instructions executable by the one or more processors to perform any of the aforementioned methods. [Explanation of symbols]

[0129] 101 users 102 additional users 110 client devices 111 User Input Engine 112 Rendering Engine 113 Presence Sensor 114 Automated Assistant Client 115 Automated Assistants 115A Machine Learning (ML) Model Database, ML Model Database 120A1 Automatic Speech Recognition (ASR) Engine, ASR Engine 120A2 ASR engine 130A1 Natural Language Understanding (NLU) Engine, NLU Engine 130A2 NLU engine 140A1 Text-to-Speech (TTS) Engine, TTS Engine 140A2 TTS engine 150 Warm Word Engine 151 Warm Word Activation Event Engine 152 Warm Word Detection Engine 153 Warm Words Verification Engine 160 User Verification Engine 170 Acoustic Engine 180 Warm Word System 190 displays 194 Text Entry Box 198 Software Applications 198A Automated Assistant Application 199 Network 201 Microphone 201A Audio Data 201B Preamble part of audio data 201C Postamble part of audio data 220 ASR output 230 NLU output 251 Warm Word Activation Event 400 Environment 452A Oral Speech 452B Oral Speech 452C Oral Speech 600 Environment 652A Oral Speech 652B Prompt, promotional advertisement 652C Prompt, Promotional Advertisement 710 Computing Devices 712 Bus Subsystem 714 processor 716 Network Interface Subsystem 720 User Interface Output Device 722 User Interface Input Devices 724 Memory Subsystem 725 Memory Subsystem, Memory 726 File Storage Subsystem 730 Main Random Access Memory (RAM) 732 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, the method comprising: using a warm word model to process a stream of audio data to monitor occurrences of one or more particular words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, each of the one or more particular words or phrases being associated with an assistant command; in response to determining that a portion of the audio data corresponds to one or more of the particular words or phrases; processing a preamble portion of the audio data and / or a postamble portion of the audio data using an automatic speech recognition (ASR) model to generate an ASR output; the preamble portion of the audio data precedes the portion of the audio data corresponding to the one or more particular words or phrases; the postamble portion of the audio data follows the portion of the audio data corresponding to the one or more particular words or phrases; Steps and determining whether the user intended the one or more particular words or phrases to cause execution of the assistant command based on processing the ASR output; In response to determining that the user did not intend the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases, refraining from causing the automated assistant to execute the assistant commands associated with one or more of the particular words or phrases; In response to determining that the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases, causing the automated assistant to execute the assistant command associated with one or more of the specific words or phrases; A method comprising:

2. detecting an occurrence of a warm word activation event; activating one or more currently dormant automated assistant functions that utilize the warm word model in response to detecting the occurrence of the warm word activation event; further comprising processing the stream of audio data using the warm word model to monitor the occurrence of the one or more particular words or phrases in response to activating the one or more currently dormant automated assistant functions that utilize the warm word model; The method of claim 1.

3. 3. The method of claim 2, wherein the warm word activation event includes one or more of a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm sounding at the client device, a timer sounding at the client device, media playing at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

4. determining, based on processing the ASR output, whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with the one or more particular words or phrases; processing the ASR output using a natural language understanding (NLU) model to generate an NLU output, the ASR output being generated based on the preamble portion of the audio data but not based on the postamble portion of the audio data; determining, based on the NLU output, whether the user intended the one or more particular words or phrases to cause execution of the assistant command; 4. The method of claim 1, comprising:

5. In response to determining that the NLU output is insufficient to determine whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases, processing the postamble portion of the audio data using the ASR model to generate additional ASR output; determining, based on processing the additional ASR output, whether the one or more particular words or phrases were intended by the user to cause execution of the assistant command associated with one or more of the particular words or phrases; 5. The method of claim 4, further comprising:

6. determining, based on processing the ASR output, whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases; processing the ASR output using a natural language understanding (NLU) model to generate an NLU output, the ASR output being generated based on both the preamble portion of the audio data and the postamble portion of the audio data; determining, based on the NLU output, whether the user intended the one or more particular words or phrases to cause execution of the assistant command; 4. The method of claim 1, comprising:

7. In response to determining that the NLU output is insufficient to determine whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases, processing an additional postamble portion of the audio data using the ASR model to generate an additional ASR output, the additional postamble portion of the audio data following the postamble portion of the audio data; determining, based on processing the additional ASR output, whether the one or more particular words or phrases were intended by the user to cause execution of the assistant command associated with one or more of the particular words or phrases; 7. The method of claim 6, further comprising:

8. processing the stream of audio data using an end-pointing model to generate a plurality of timestamps for spoken utterances captured within the stream of audio data that include the one or more particular words or phrases; 8. The method of claim 1, further comprising:

9. 9. The method of claim 8, wherein the plurality of timestamps include at least a first timestamp associated with a first time when the user began providing the oral utterance, a second timestamp associated with a second time following the first time when the user began providing the one or more specific words or phrases included in the oral utterance, a third timestamp associated with a third time following the second time when the user finished providing the one or more specific words or phrases included in the oral utterance, and a fourth timestamp associated with a fourth time following the third time when the user finished providing the oral utterance.

10. 10. The method of claim 9, wherein the preamble portion of the audio data includes any audio data corresponding to the spoken utterance between the first timestamp and the second timestamp.

11. 11. The method of claim 9 or 10, wherein the postamble portion of the audio data includes any audio data corresponding to the spoken utterance between the third timestamp and the fourth timestamp.

12. activating one or more currently dormant automated assistant functions that utilize the ASR model in response to determining that the spoken utterance includes one or more of the particular words or phrases.

12. The method of any one of claims 1 to 11, further comprising:

13. using the ASR model to process the portion of the audio data corresponding to the one or more particular words or phrases, along with the preamble portion of the audio data and / or the postamble portion of the audio data, to generate the ASR output.

13. The method of any one of claims 1 to 12, further comprising:

14. processing the stream of audio data to monitor for occurrences of additional voice activity using a voice activity detection (VAD) model before the portion of the audio data corresponds to one or more of the particular words or phrases and / or after the portion of the audio data corresponds to one or more of the particular words or phrases; in response to detecting the occurrence of the additional voice activity; activating one or more currently dormant automated assistant functions that utilize the ASR model; 14. The method of any one of claims 1 to 13, further comprising:

15. in response to not detecting the occurrence of the additional voice activity. causing the automated assistant to execute the assistant commands associated with one or more of the specific words or phrases.

15. The method of claim 14, further comprising:

16. processing the audio data to determine one or more prosodic features of the spoken utterance using an acoustic model; further comprising 16. The method of claim 1, wherein determining whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases is further based on the one or more prosodic characteristics of the oral utterance.

17. receiving instances of sensor data from one or more sensors of the client device; further comprising 17. The method of claim 1, wherein determining whether the user intended the one or more particular words or phrases to cause execution of the assistant command associated with one or more of the particular words or phrases is further based on the instance of the sensor data.

18. 20. The method of claim 17, wherein the instances of sensor data include one or more of accelerometer data generated by an accelerometer of the client device, image data generated by one or more camera components of the client device, and motion data generated by one or more motion sensors of the client device.

19. retrieving the preamble portion of the audio data from an audio buffer of the client device; and / or obtaining the postamble portion of the audio data from the stream of audio data.

19. The method of any one of claims 1 to 18, further comprising:

20. 1. A method implemented by one or more processors, the method comprising: using a warm word model to process a stream of audio data to monitor occurrences of one or more particular words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, each of the one or more particular words or phrases being associated with an assistant command; in response to determining that the portion of the audio data corresponds to one or more of the particular words or phrases, and in response to determining to verify the identity of the user who provided the oral utterance corresponding to the audio data; processing the audio data to determine whether the audio data is sufficient to identify the user who provided the verbal utterance using a speaker identification (SID) model; in response to determining that the audio data is sufficient to identify the user who provided the verbal utterance, and in response to determining that the identified user who provided the verbal utterance is authorized to cause execution of the assistant command associated with one or more of the particular words or phrases, causing an automated assistant to execute the assistant commands associated with one or more of the specific words or phrases; In response to determining that the audio data is insufficient to identify the user who provided the verbal utterance, generating a prompt requesting further verification of the identity of the user who provided the verbal utterance; providing the prompt for presentation at the client device; A method comprising:

21. detecting an occurrence of a warm word activation event; activating one or more currently dormant automated assistant functions that utilize the warm word model in response to detecting the occurrence of the warm word activation event; further comprising 21. The method of claim 20, wherein processing the stream of audio data using the warm word model to monitor the occurrences of the one or more particular words or phrases is responsive to activating the one or more currently dormant automated assistant functions that utilize the warm word model.

22. 22. The method of claim 21, wherein the warm word activation event includes one or more of a phone call received at the client device, a text message received at the client device, an email received at the client device, an alarm sounding at the client device, a timer sounding at the client device, media playing at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

23. in response to determining that the portion of the audio data corresponds to one or more of the particular words or phrases, and in response to determining not to verify the identity of the user who provided the oral utterance corresponding to the audio data, causing an automated assistant to execute the assistant commands associated with one or more of the particular words or phrases.

23. The method of any one of claims 20 to 22, further comprising:

24. processing the audio data to determine whether the audio data is sufficient to identify the identity of the user who provided the verbal utterance; processing the portions of the audio data corresponding to one or more of the particular words or phrases to generate TD speaker embeddings using a text-dependent (TD) SID model as the SID model. Including, determining whether the audio data is sufficient to identify the user who provided the verbal utterance, comparing the TD speaker embedding in an embedding space to one or more stored TD speaker embeddings stored locally at the client device; determining whether the portion of the audio data is sufficient to verify the identity of the user who provided the oral utterance based on comparing the TD speaker embedding to the one or more stored TD speaker embeddings; 24. The method of any one of claims 20 to 23, comprising:

25. processing the portions of the audio data corresponding to one or more of the particular words or phrases to generate the TD speaker embeddings using the TD SID model; determining whether one or more of the stored TD speaker embeddings correspond to the particular word or phrase; Including, 25. The method of claim 24, wherein processing the portion of the audio data corresponding to one or more of the particular words or phrases using the TD SID model to generate the TD speaker embeddings is responsive to determining that one or more of the stored TD speaker embeddings correspond to the particular words or phrases.

26. generating the prompt requesting the further verification of the identity of the user who provided the verbal utterance, generating a prompt requesting that the user provide an additional verbal utterance, the additional verbal utterance being of sufficient length to implement a TI SID for use in verifying the identity of the user who provided the verbal utterance; 25. The method of claim 24, comprising:

27. In response to determining that the oral utterance is of sufficient length to perform a TI SID, processing the stream of audio data to generate TI speaker embeddings using a TI SID model as the SID model; further comprising determining whether the audio data is sufficient to identify the user who provided the verbal utterance, comparing the TI speaker embedding in an embedding space to one or more stored TI speaker embeddings stored locally at the client device; determining whether the stream of audio data is sufficient to verify the identity of the user who provided the oral utterance based on comparing the TI speaker embedding to the one or more stored TI speaker embeddings; 25. The method of claim 24, comprising:

28. generating the prompt requesting the further verification of the identity of the user who provided the verbal utterance, generating the prompt requesting additional biometric information to verify the identity of the user, wherein requesting the additional biometric information to verify the identity of the user includes requesting one or more of a fingerprint identification or a faceprint fingerprint identification; 28. The method of claim 27, comprising:

29. receiving additional user input in response to providing the prompt for presentation at the client device; In response to receiving the additional user input, determining whether the additional user input is sufficient to identify the user who provided the verbal utterance; 29. The method of any one of claims 20 to 28, further comprising:

30. In response to determining that the additional user input is sufficient to identify the user who provided the verbal utterance, causing the automated assistant to execute the corresponding assistant command.

30. The method of claim 29, further comprising:

31. at least one processor; a memory storing instructions that, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 30; A system comprising:

32. 31. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform operations corresponding to any one of claims 1 to 30.

Images