Assistant Command Field Suppression

By detecting warm words in contextual audio data and combining ASR, NLU, and SID models to verify user intent and identity, the problems of false warm word triggering and resource waste are solved, achieving more efficient resource utilization and secure assistant command execution.

CN116783650BActive Publication Date: 2026-11-17GOOGLE LLC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202180089319.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-05-17
Filing Date
2021-12-14
Publication Date
2026-11-17
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

In existing technologies, the use of warm word models can easily lead to wasted computing resources and false triggers. Users need to frequently cancel assistant commands, and user identity cannot be effectively verified, resulting in low security and low resource utilization efficiency.

Method used

By processing contextual audio data, the system uses a warm word model to detect specific words and phrases, combines ASR and NLU models to verify user intent, and utilizes a SID model to verify user identity, executing assistant commands only under the instructions of authorized users.

Benefits of technology

It reduces false positives triggered by warm words, saves computing resources, improves the efficiency of user input and data security, and ensures that only authorized users can execute critical assistant commands.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116783650B_ABST
    Figure CN116783650B_ABST
Patent Text Reader

Abstract

Some implementations use a warm word model to process the stream of audio data to determine portions of audio data corresponding to specific words and / or phrases (e.g., warm words) associated with an assistant command, use an automatic speech recognition (ASR) model to process the preceding portions of the audio data (e.g., before the warm word) and / or the following portions of the audio data (e.g., after the warm word) to generate ASR output, and determine whether the user intends to execute the assistant command based on the processed ASR output.
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] Humans can engage in human-computer dialogue using interactive software applications, referred to herein as “automatic assistants” (also known as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). For example, humans (who may be referred to as “users” when interacting with an automatic assistant) can provide spoken natural language input (i.e., spoken words) to the automatic assistant—in some cases, spoken natural language input can be converted into text and then processed, and / or provided as text (e.g., typed) natural language input. Automatic assistants typically respond to spoken words by providing responsive user interface outputs (e.g., auditory and / or visual user interface outputs), controlling intelligent devices, and / or performing other actions.

[0002] Automated assistants typically rely on a suite of components to interpret and respond to spoken utterance. For example, an Automatic Speech Recognition (ASR) engine can process audio data corresponding to a user's spoken utterance to generate ASR outputs, such as transcriptions of the spoken utterance (i.e., sequences of terms and / or other commands). Furthermore, a Natural Language Understanding (NLU) engine can process the ASR output to generate NLU outputs, such as the user's intention when providing the spoken utterance and the time-slot values ​​of parameters optionally associated with that intention. Additionally, a performance engine can be used to process the NLU output and generate performance outputs, such as obtaining a structured request for a response to the spoken utterance.

[0003] In some cases, this set of components can be bypassed. For example, some machine learning (ML) models (also known as "warm word models") can be trained to detect specific words and / or phrases (also known as "warm words") that directly map to the output. For instance, if a user is listening to music and provides the spoken phrase "turn up the volume," one or more of these ML models can process the audio data and generate a structured request to make the device playing the music turn up the volume, without generating any ASR output and / or NLU output. However, these warm words are often common words and / or phrases that are likely to appear in everyday speech. Therefore, assistant commands associated with these warm words may be unintentionally triggered by the user, wasting computational resources. Furthermore, in some of these cases, the user may have to retract some of these assistant commands, wasting even more computational resources. Summary of the Invention

[0004] The embodiments disclosed herein aim to suppress the execution (or enforcement) of an assistant command associated with one or more specific words and / or phrases in a context, based on contextual audio data and / or the identity of the user providing spoken utterances including one or more specific words and / or phrases (e.g., warm words). Some embodiments use one or more warm word models to process the stream of audio data to determine the portions of audio data corresponding to the specific words and / or phrases associated with the assistant command. Some of these embodiments use an automatic speech recognition (ASR) model to process the preceding portions of the audio data (e.g., portions of audio data preceding the portions of audio data corresponding to one or more specific words and / or phrases) and / or the following portions of the audio data (e.g., portions of audio data following the portions of audio data corresponding to one or more specific words and / or phrases) to generate ASR output. Furthermore, some of these embodiments determine whether the user providing the spoken utterances captured in the audio data intends to execute an assistant command associated with one or more specific words or phrases based on the processing of the ASR output. Additional or alternative implementations may use a speaker recognition (SID) model to process the audio data stream to determine whether the audio data is sufficient to identify the user providing the spoken words captured in the audio data stream, and to determine whether the user providing the spoken words is authorized to execute assistant commands.

[0005] For example, suppose a phone call is received at a user's client device. Further suppose the user on the client device responds to the received phone call by uttering the verbal phrase "answer." In this example, and suppose "answer" is a warm word associated with an assistant command that causes an automated assistant, implemented at least partially on the client device, to answer the phone call, the automated assistant could answer the phone call on behalf of the user based on detecting the presence of the warm word "answer" using one or more warm word models. Conversely, suppose the user on the client device responds to the received phone call by instead uttering the verbal phrase "I don't want to answer it." In this example, although the presence of the warm word "answer" is detected using one or more of the warm word models, the automated assistant could process contextual audio data to determine that the automated assistant should not answer the phone on behalf of the user.

[0006] In some implementations, one or more automated assistant components utilizing one or more of the warm word model may be activated in response to the detection of a warm word activation event. Warm word activation events may include, for example, a telephone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, media played at the client device or an attached client device in the client device's environment, a notification received at the client device, the client device's location, a software application accessible at the client device, and / or other events associated with the client device in which a user can provide verbal commands to control the client device or an attached client device communicating with the client device. It is noteworthy that some of these warm word activation events are discrete events (e.g., a telephone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, etc.), while some other warm word activation events are continuous events (e.g., a software application accessible at the client device). Furthermore, it should be noted that these warm word activation events are not mutually exclusive. In other words, multiple warm word activation events can be detected at a given moment, enabling one or more automatic assistant components to proactively monitor warm words based on multiple different warm word activation events.

[0007] The warm word model described herein may include multiple different warm word models (e.g., audio keyword classification models) trained to detect one or more specific words and / or phrases (e.g., warm words), which, when detected, cause an automated assistant to execute an assistant command associated with one or more of the specific words and / or phrases. In some implementations, a given warm word model may be trained to detect a specific subset of words and / or phrases associated with a given warm word activation event. For example, suppose music is playing on a client device or another client device (e.g., a smart speaker) communicating with the client device. In this example, a given warm word model may be trained to detect a subset of specific words and / or phrases that cause the execution of an assistant command associated with controlling the music, such as a "pause" warm word that may pause the music, a "continue" warm word that may resume the music after a pause, a "turn up" warm word that may turn up the music volume, a "turn down" warm word that may turn down the music volume, a "next" warm word that may skip to the next song, and other specific words and / or phrases associated with the music. Therefore, in this example, in response to determining that music is playing on a client device or another client device communicating with the client device, one or more of the currently dormant assistant functions utilizing a given warm word model can be activated.

[0008] In additional or alternative implementations, multiple warm word models can be trained to detect subsets of words and / or phrases associated with a given warm word activation event. Continuing the example above, a first warm word model can be trained to detect one or more first specific words and / or phrases, such as the "pause" warm word and the "continue" warm word; a second warm word model can be trained to detect one or more second specific words and / or phrases, such as the "turn up the volume" warm word or the "turn down the volume" warm word; a third warm word model can be trained to detect one or more third specific words and / or phrases, such as the "next" warm word; and other specific words and / or phrases associated with a music warm word activation event. Thus, in this example, in response to determining that music is playing on a client device or another client device communicating with the client device, one or more of the currently dormant assistant functions utilizing at least the first, second, and third warm word models can be activated.

[0009] In some implementations, and in response to the detection of one or more specific words and / or phrases in the audio data stream, the preamble and / or suffix of the audio data can be processed to determine whether the user providing the spoken words actually intends to execute an assistant command associated with one or more of the detected specific words and / or phrases. For example, again assuming a phone call is received at a user's client device, and assuming the user at the client device provides the spoken words "I don't want to answer it" in response to the received phone call. In this example, the preamble of the audio data (e.g., corresponding to "I don't want to") can be obtained from the audio buffer of the client device and can be processed using an ASR model to generate an ASR output. Furthermore, the ASR output can be processed using an NLU model to generate an NLU output. In this example, the ASR output and / or NLU output indicate that the user does not intend to "answer" so that the automated assistant can answer the phone on the user's behalf. In some versions of these implementations, the preamble of the audio data can be combined with, additionally or alternatively, the portion of the audio data corresponding to one or more specific words and / or phrases (e.g., corresponding to "answer") to generate both the ASR output and the NLU output. In some versions of these implementations, the preceding portion of the audio data and / or portions of the audio data corresponding to one or more specific words and / or phrases may be combined, and the following portion of the audio data (e.g., corresponding to "it") may be processed additionally or alternatively to generate ASR output and NLU output.

[0010] In some versions of these implementations, one or more currently dormant automated assistant components utilizing an ASR model and / or an NLU model may be activated in response to the detection of additional voice activity other than one or more of a specific word and / or phrase. For example, a Voice Activity Detection (VAD) model may be used to process the audio data stream to determine the presence of any additional voice activity. In implementations where additional voice activity is detected, one or more currently dormant automated assistant components utilizing an ASR model and / or an NLU model may be activated to process the preceding portion of the audio data, the portion of the audio data corresponding to one or more of the specific word and / or phrase, and / or the following portion of the audio data. In implementations where no additional voice activity is detected, one or more currently dormant automated assistant components utilizing an ASR model and / or an NLU model may remain dormant, and assistant commands associated with one or more of the specific word and / or phrase may be executed.

[0011] In various implementations, the identity of the user providing the spoken words must be verified before executing the assistant command. For example, assistant commands associated with specific warm word activation events, specific devices, and / or specific software applications may require user identity verification. For instance, in implementations where the assistant command causes the automated assistant to respond to electronic communications (e.g., phone calls, text messages, emails, social media messages, and / or other electronic communications), user verification of the spoken words may be required before the automated assistant responds to the electronic communication. Furthermore, in implementations where the assistant command causes the automated assistant to control specific smart devices (e.g., smart locks, smart ovens, smart garage doors, etc.), user verification of the spoken words may be required before the automated assistant controls the specific smart device. However, some assistant commands may not require user identity verification before executing the assistant command. For example, in implementations where the assistant command causes the automated assistant to control media (e.g., music, videos, television programs, and / or other media), user verification of the spoken words may not be required before the automated assistant responds to electronic communications. Furthermore, in implementations where the assistant command causes the automated assistant to control other specific smart devices (e.g., smart locks, smart ovens, smart garage doors, etc.), user verification of the spoken words may not be required before the automated assistant controls the specific smart device.

[0012] As described above, a SID model can be used to process the audio data stream to verify the identity of the user providing the spoken utterance. In some versions of these implementations, and assuming that one or more previously generated text-related (TD) speaker embeddings are accessible to the user on the client device for one or more specific words and / or phrases, a TD SID model can be used to process portions of the audio data corresponding to one or more specific words and / or phrases to generate TD speaker embeddings. The generated TD speaker embeddings can be effectively compared with one or more previously generated TD speaker embeddings accessible to the user on the client device to determine whether the spoken utterance comes from an authorized user. It is worth noting that one or more specific words and / or phrases in the TD SID can be limited to one or more invocation phrases configured to invoke an automated assistant (e.g., hot words and / or trigger words, such as, for example, “Hey assistant,” “OK assistant,” and / or “assistant”) or one or more warm words described herein.

[0013] In some additional or alternative versions of these implementations, and assuming that one or more previously generated text-free (TI) speaker embeddings are available to the user at the client device, the TI SID model can be used to process audio data corresponding to spoken utterances in the audio data stream to generate TI speaker embeddings. The generated TI speaker embeddings can be effectively compared with one or more previously generated TI speaker embeddings available to the user at the client device to determine whether the spoken utterances originated from an authorized user. It is noteworthy that in TD SID, one or more previously generated TI speaker embeddings are not limited to one or more invoking phrases (e.g., hot words and / or trigger words, such as, for example, “Hey Assistant,” “OK Assistant,” and / or “Assistant”) or one or more warm words described herein that are configured to invoke an automated assistant.

[0014] Therefore, in some versions of these implementations, the length of the spoken words captured in the audio data stream can be compared to a length threshold to determine whether the length of the spoken words is sufficient to perform TI SID. If the length of the spoken words is sufficient to perform TI SID, the audio data corresponding to the spoken words can be processed using a TI SID model and / or a TD SID model to attempt to verify the user's identity. However, even if the length of the spoken words is insufficient to perform TI SID, the audio data corresponding to the spoken words can still be processed using a TD SID model to attempt to verify the user's identity. Assuming that the length of the spoken words is insufficient to perform TI SID (and / or that TI SID is insufficient to verify the user's identity), and assuming that TD SID is insufficient to verify the user's identity, a prompt including an example utterance can be presented to the user, where the example utterance is of sufficient length (or long enough) to perform TI SID, or corresponds to one or more previously generated TD speaker embeddings that can be used for specific terms and / or phrases. Additionally or alternatively, the prompt can request the user to provide other forms of biometric identification (e.g., facial recognition and / or fingerprint recognition) to enable the execution of assistant commands.

[0015] By using the techniques described herein, one or more technical advantages can be achieved. As a non-limiting example, the techniques described herein enable automated assistants to determine, based on contextual audio data processing warm words, whether a user actually intends for the warm words to appear so that an assistant command can be executed. Therefore, the number of false positives for warm words can be reduced, thereby saving computational and / or network resources at the client device when the automated assistant utilizes one or more networks to execute assistant commands. Furthermore, since the user on the client device does not need to take action to correct or cancel any assistant commands, the amount of user input can be reduced, thus saving computational resources at the client device. Additionally, the various automated assistant functions described herein can remain dormant until required as described herein, thereby saving computational resources at the client device. As another non-limiting example, the techniques described herein enable automated assistants to use SID and / or other techniques to verify the identity of the user providing spoken utterances including warm words. Therefore, only authorized users can execute certain assistant commands, thereby improving data security associated with the user on the client device.

[0016] The above description is provided only as an overview of some embodiments disclosed herein. These and other embodiments are described in further detail herein.

[0017] It should be understood that all combinations of the foregoing and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0018] Figure 1 A block diagram depicts an example environment in which various aspects of this disclosure may be implemented, and in which embodiments disclosed herein may be practiced.

[0019] Figure 2 Describing the use according to various implementation methods Figure 1 The various components demonstrate example process flows for various aspects of this disclosure.

[0020] Figure 3 A flowchart illustrating example methods for determining whether to execute an assistant command based on contextual audio data, according to various implementations, is provided.

[0021] Figure 4A , Figure 4B and Figure 4C Various non-limiting examples are described, based on various implementations, of determining whether to execute an assistant command using contextual audio data.

[0022] Figure 5 A flowchart illustrating example methods for determining whether to execute an assistant command based on speaker recognition, according to various implementations, is shown.

[0023] Figure 6A , Figure 6B and Figure 6C Various non-limiting examples of speaker recognition-based determination of whether to execute an assistant command are described according to various implementation methods.

[0024] Figure 7 Example architectures of computing devices according to various implementations are described. Detailed Implementation

[0025] Turn now Figure 1 This diagram depicts an example environment illustrating various aspects of this disclosure and in which embodiments disclosed herein may be implemented. The example environment includes a client device 110 and a word warming system 180. In some embodiments, the word warming system 180 may be implemented locally on the client device 110. In additional or alternative embodiments, the word warming system 180 may be implemented remotely from the client device 110, such as... Figure 1 As shown. In these embodiments, the client device 110 and the word system 180 can communicate and couple with each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi LANs, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet).

[0026] Client device 110 may be one or more of the following: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart device such as a smart TV, and / or user wearable device including a computing device (e.g., user watch with computing device, user glasses with computing device, virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0027] Client device 110 may implement an automated assistant client 114. An instance of the automated assistant client 114 may be an application separate from the operating system of client device 110 (e.g., installed "on top" of the operating system), or may alternatively be implemented directly by the operating system within client device 110. The automated assistant client 114 may interact with a word system 180 implemented locally on client device 110, or via, for example... Figure 1 One or more networks 199, as shown, interact with the warm word system 180. The automated assistant client 114 (and optionally through its interaction with other remote systems (e.g., servers)) can form what appears from the user's perspective to be a logical instance of the automated assistant 115 with which the user can engage in human-computer dialogue. An instance of the automated assistant 115 in... Figure 1 The diagram is depicted and surrounded by dashed lines including the Auto Assistant client 114 (client device 110) and the Warm Word system 180. Therefore, it should be understood that a user engaging with the Auto Assistant client 114 implemented on client device 110 can actually engage with his or her own logical instance of Auto Assistant 115 (or a logical instance of Auto Assistant 115 shared within a family or other user group). For brevity, Auto Assistant 115 as used herein will refer to the Auto Assistant client 114 implemented on client device 110 and / or on one or more servers where the Warm Word system 180 may be implemented.

[0028] In various embodiments, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of client device 110. Additionally or alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or motions (e.g., gestures) detected in the field of view of one or more visual components. Additionally or alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., a keyboard and mouse, a stylus, a touchscreen, a touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110.

[0029] In various embodiments, client device 110 may include rendering engine 112, configured to provide content for auditory and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers, enabling the provision of auditory content to a user via client device 110. Additionally or alternatively, client device 110 may be equipped with a display or projector, enabling the provision of visual content to a user via client device 110.

[0030] In various embodiments, client device 110 may include one or more presence sensors 113, configured to provide signals indicating a detected presence (particularly human presence) upon approval by the corresponding user. In some of these embodiments, the automated assistant 115 may recognize client device 110 (or another computing device associated with the user of client device 110) at least in part based on the presence of the user at client device 110 (or at another computing device associated with the user of client device 110) to satisfy a spoken utterance. The spoken utterance may be satisfied by rendering response content at client device 110 and / or the other computing device associated with the user of client device 110 (e.g., via rendering engine 112), by controlling client device 110 or the other computing device associated with the user of client device 110, and / or by causing client device 110 and / or the other computing device associated with the user of client device 110 to perform any other action. As described herein, the automated assistant 115 may use data determined by the presence sensor 113 to determine the location of the client device 110 (or other computing device) based on where the user is or is currently located, and may only provide corresponding commands to the client device 110 (or those other computing devices). In some additional or alternative embodiments, the automated assistant 115 may use data determined by the presence sensor 113 to determine whether any user (any user or a specific user) is currently near the client device 110 (or other computing device), and may optionally suppress the provision of data to and / or from the client device 110 (or other computing device) based on the user who is near the client device 110 (or other computing device).

[0031] The presence sensor 113 can take various forms. For example, the client device 110 can utilize one or more of the user interface input components described above with respect to the user input engine 111 to detect the presence of a user. Additionally or alternatively, the client device 110 can be equipped with other types of light-based presence sensors 113, such as passive infrared (“PIR”) sensors that measure infrared (“IR”) light radiated from objects within its field of view.

[0032] Additionally or alternatively, in some embodiments, the presence sensor 113 may be configured to detect other phenomena associated with human presence or device presence. For example, in some embodiments, the client device 110 may be equipped with the presence sensor 113, which detects various types of wireless signals (e.g., waves such as radio waves, ultrasonic waves, electromagnetic waves, etc.) emitted by other computing devices (e.g., mobile devices, wearable computing devices, etc.) and / or other computing devices carried / operated by the user. For example, the client device 110 may be configured to emit waves imperceptible to humans, such as ultrasonic waves or infrared waves, which can be detected by other computing devices (e.g., via an ultrasonic / infrared receiver such as a microphone with ultrasonic capabilities).

[0033] Additionally or alternatively, client device 110 may emit other types of waves imperceptible to humans, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which can be detected by other computing devices carried / operated by the user (e.g., mobile devices, wearable computing devices, etc.) and used to determine the user's specific location. In some embodiments, GPS and / or Wi-Fi triangulation may be used, for example, based on GPS and / or Wi-Fi signals to / from client device 110 to detect a person's location. In other embodiments, client device 110 may use, alone or in combination, other wireless signal characteristics such as time-of-flight and signal strength to determine the location of a specific person based on signals emitted by other computing devices carried / operated by the user.

[0034] Additionally or alternatively, in some embodiments, client device 110 may perform speaker recognition (SID) to identify the user based on the user's voice (e.g., as described with respect to user verification engine 160). In some embodiments, the speaker's movement may then be determined, for example, by the presence sensor 113 of client device 110 (and optionally by a GPS sensor, Soli chip, and / or accelerometer of client device 110). In some embodiments, based on this detected movement, the user's location may be predicted, and this location may be assumed to be the user's location when the proximity of client device 110 and / or other computing devices to the user's location makes it possible to render any content at client device 110 and / or other computing devices. In some embodiments, it may be simply assumed that the user is in the location where he or she last engaged with the automated assistant 115, particularly where not much time has passed since the last engagement.

[0035] Furthermore, client device 110 and / or the communication system 180 may include one or more memories for storing data and / or software applications 198, one or more processors for accessing data and implementing software applications 198, and / or other components that facilitate communication via one or more of networks 199. In some embodiments, one or more of the software applications 198 may be locally installed at client device 110, while in other embodiments, one or more of the software applications 198 may be remotely hosted (e.g., by one or more servers) and accessible by client device 110 via one or more of networks 199. The operations performed by client device 110, other computing devices, and / or automated assistant 115 may be distributed across multiple computer systems. Automated assistant 115 may be implemented, for example, on client device 110 and / or via a network (e.g., Figure 1 A network (199) is a computer program running on one or more computers in one or more locations that are coupled to each other.

[0036] In some implementations, the operations performed by the automated assistant 115 can be implemented locally on the client device 110 via the automated assistant client 114. For example... Figure 1 As shown, the automated assistant client 114 may include an automatic speech recognition (ASR) engine 120A1, a natural language understanding (NLU) engine 130A1, and a text-to-speech (TTS) engine 140A1. In some embodiments, the operations performed by the automated assistant 115 may be distributed across multiple computer systems, such as when... Figure 1 The description refers to the implementation of the warm word system 180 remotely from the client device 110. In these embodiments, the automatic assistant 115 may additionally or alternatively utilize the ASR engine 120A2, NLU engine 130A2, and TTS engine 140A2 of the warm word system 180.

[0037] Each of these engines can be configured to perform one or more functions. For example, ASR engines 120A1 and / or 120A2 can use ASR models (e.g., recurrent neural network (RNN) models, transformer models, and / or any other type of ML model capable of performing ASR) stored in the machine learning (ML) model database 115A to process audio data captured from spoken utterances and generated by the microphone of client device 110 to generate ASR output. Furthermore, NLU engines 130A1 and / or 130A2 can use NLU models (e.g., long short-term memory (LSTM), gated recurrent units (GRU), and / or any other type of RNN or other ML model capable of performing NLU) and / or syntax-based rules stored in the ML model database 115A to process ASR output to generate NLU output. Additionally, TTS engines 140A1 and / or 140A2 can use TTS models stored in the ML model database 115A to process text data (e.g., text defined by the automation assistant 115) to generate synthetic speech audio data including computer-generated synthesized speech.

[0038] In various implementations, the ASR output may include, for example, one or more speech hypotheses (e.g., terminology hypotheses and / or transcription hypotheses) predicted to correspond to spoken utterances of a user captured in the audio data, one or more corresponding predicted values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the one or more speech hypotheses, multiple phonemes predicted to correspond to spoken utterances of a user captured in the audio data, and / or other ASR outputs. In some versions of these implementations, ASR engines 120A1 and / or 120A2 may select one or more speech hypotheses as the recognized text corresponding to the spoken utterances (e.g., based on the corresponding predicted values).

[0039] In various implementations, the NLU output may include, for example, annotated identified text, which includes one or more (e.g., all) annotations of the identified text's terms. For example, NLU engines 130A1 and / or 130A2 may include a part-of-speech tagger (not shown) configured to annotate terms with their grammatical roles. Additionally or alternatively, NLU engines 130A1 and / or 130A2 may include an entity tagger (not shown) configured to annotate entity references in one or more segments of the identified text, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, places (real and imagined), etc. In some implementations, entity data may be stored in one or more databases, such as in a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting nodes and representing relationships between entities. Entity taggers can annotate references to entities at a high level of granularity (e.g., to enable the identification of all references to an entity class such as a person) and / or a lower level of granularity (e.g., to enable the identification of all references to a specific entity such as a particular person). Entity taggers can rely on the content of the natural language input to parse specific entities, and / or can optionally communicate with a knowledge graph or other entity database to parse specific entities. Additionally or alternatively, NLU engines 130A1 and / or 130A2 may include a coreference parser (not shown) configured to group or “cluster” references to the same entity based on one or more contextual cues. For example, based on “theater tickets” mentioned in a client device notification rendered immediately before the input “buy them” is received, the coreference parser can be used to resolve the term “they” to “buy theater tickets” in the natural language input “buy them”. In some implementations, one or more components of NLU engines 130A1 and / or 130A2 may depend on annotations from one or more other components of NLU engines 130A1 and / or 130A2. For example, in some implementations, the entity tagger may depend on annotations from the coreference resolver to annotate all references to a particular entity. Furthermore, for example, in some implementations, the coreference resolver may depend on annotations from the entity tagger to cluster references to the same entity.

[0040] Although it describes a single client device with a single user Figure 1However, it should be understood that this is for illustrative purposes and does not imply limitation. For example, one or more additional client devices of a user may also implement the techniques described herein. For example, a user's client device 110, one or more additional client devices, and / or any other computing devices may form a device ecosystem that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be used by multiple users in a shared setup (e.g., a group of users, a household).

[0041] As described herein, the automated assistant 115 can determine whether to execute an assistant command in response to the detection of a warm word in the audio data stream, based on contextual audio data and / or the identity of the speaker providing the spoken utterance captured in the audio data stream. In making this determination, the automated assistant may utilize a warm word engine 150, a user verification engine 160, and / or an acoustic engine 170. In various embodiments and as described herein… Figure 1 As described, the warm word engine 150 may include a warm word activation event engine 151, a warm word detection engine 152, and a warm word verification engine 153.

[0042] In some implementations, the warm word activation event engine 151 can monitor the occurrence of warm word activation events. Warm word activation events may include, for example, a telephone call received at client device 110, a text message received at client device 110, an email received at client device 110, an alarm sounded at client device 110, a timer sounded at client device 110, media played at client device 110 or an attached client device in the environment of client device 110, a notification received at client device 110, the location of client device 110 (e.g., the user is at home, at work, etc.), one or more software applications 198 accessible at client device 110, and / or other events associated with client device 110 in which the user can provide verbal utterances to control client device 110 or an attached client device communicating with client device 110 based on the verbal utterances. In some versions of these implementations, and in response to the detection of a warm word activation event, the warm word activation event engine 151 can cause one or more currently dormant automatic assistant functions (e.g., low-power digital signal processors (DSPs), central processing units (CPUs), etc.) to be activated using a warm word model stored in the ML model database 115A. The warm word model can be trained to detect one or more specific words or phrases (also referred to herein as "warm words"), each of which is associated with an assistant command. One or more specific words or phrases can be associated with an assistant command such that, when detected, the automatic assistant 115 can cause the execution of a specific action without explicitly invoking the automatic assistant 115 via a calling word or phrase (e.g., "assistant," "hey assistant," etc.), a calling gesture, or other means used to invoke the automatic assistant 115. In other words, one or more specific words or phrases can be mapped to specific intentions, specific execution data, and / or specific software applications of one or more software applications accessible at client device 110, enabling the automatic assistant 115 to bypass the use of various assistant components (e.g., ASR engines 120A1 and / or 120A2, NLU engines 130A1 and / or 130A1, and / or other automatic assistant components) to execute assistant commands associated with one or more specific words or phrases.

[0043] In some further versions of these implementations, one or more of the currently dormant auto-assistant functions utilizing the warm word model can remain active until one or more conditions are met. One or more conditions may include, for example, the end of a threshold duration, detection of one or more specific words or phrases, determination that the warm word activation event has ended, and / or other conditions described herein. For example, suppose the warm word activation event includes a timer ringing at client device 110. In this example, one or more of the currently dormant auto-assistant functions utilizing the warm word model can remain active until the user provides a warm word to stop the timer from ringing, and / or until the user manually stops the timer. As another example, suppose the warm word activation event includes a notification received at client device 110. In this example, one or more of the currently dormant auto-assistant functions utilizing the warm word model can remain active until the user provides a warm word to interact with the notification, until the user manually cancels the notification, and / or until a threshold duration (e.g., 30 seconds, 60 seconds, etc.) ends. As yet another example, suppose the warm word activation event includes a telephone call received at client device 110. In this example, one or more of the currently dormant auto-assistant functions utilizing the warm word model can remain active until the user provides a warm word to interact with the phone call, and / or until the phone call stops ringing. Therefore, it should be understood that once activated, one or more of the currently dormant auto-assistant functions utilizing the warm word model described herein can be activated for a static or dynamic duration based on the detected warm word activation event and / or the manner in which the user of client device 110 responds to the warm word activation event.

[0044] In additional or alternative implementations, one or more automatic assistant functions utilizing the warm word model may always be active. In other words, some of the automatic assistant functions utilizing the warm word model are always active regardless of whether the warm word activation event engine 151 detects a warm word activation event. For example, the warm word model may be used to detect one or more of the following automatic assistant functions: specific words or phrases associated with a specific query, such as “How’s the weather?”, “What’s on my calendar?”, etc.; specific words or phrases associated with a specific assistant routine, such as “Good morning” routine, “Good evening” routine, etc.; specific words or phrases associated with a specific device, such as “Turn on the lights”, “Turn on [indoor lights / indicator lights]”, etc.; specific words or phrases associated with a specific software application that is in a running or non-running state among one or more software applications 198 accessible at the client device 110, such as “Open first application”, “Launch first application”, “Directions to Main Street 123” when the navigation application is running or non-running, “Cheese” when the camera application is running; and / or other specific words or phrases.

[0045] The warm word model described herein may include multiple different warm word models (e.g., audio keyword classification models) trained to detect one or more specific words or phrases (e.g., warm words), which, when detected, cause the automatic assistant 115 to execute an assistant command associated with one or more of the specific words or phrases. In some implementations, a given warm word model may be trained to detect a specific subset of words or phrases associated with a given warm word activation event. For example, suppose music is playing at client device 110 or another client device communicating with client device 110. In this example, a given warm word model may be trained to detect a subset of specific words or phrases that cause the execution of an assistant command associated with controlling the music, such as a "pause" warm word that may pause the music, a "continue" warm word that may resume the music after a pause, a "turn up" warm word that may turn up the music volume, a "turn down" warm word that may turn down the music volume, a "next" warm word that may skip to the next song, and other specific words or phrases associated with the music. Therefore, in this example, in response to determining that music is being played on client device 110 or another client device communicating with client device 110, one or more currently dormant assistant functions utilizing a given warm word model can be activated. Thus, at a single moment, one or more warm word models can be active at client device 110.

[0046] In additional or alternative implementations, multiple warm word models can be trained to detect a subset of words or phrases associated with a given warm word activation event. Continuing the example above, a first warm word model can be trained to detect one or more first specific words or phrases, such as the warm words "pause" and "continue," a second warm word model can be trained to detect one or more second specific words or phrases, such as the warm words "turn up the volume" or "turn down the volume," a third warm word model can be trained to detect one or more third specific words or phrases, such as the warm word "next," and other specific words or phrases associated with music. Thus, in this example, in response to determining that music is playing on client device 110 or another client device communicating with client device 110, one or more of the currently dormant assistant functions utilizing at least the first, second, and third warm word models can be activated.

[0047] The warm word model described in this paper can be trained based on multiple training instances. Each of the multiple training instances can include training instance input and training instance output. The training instance input can include audio data, where portions of the audio data correspond to one or more of the specific words or phrases that the given warm word model is being trained to detect, and the training instance output can include a baseline truth output, where the baseline truth output includes an indication of whether the audio data of the training instance input includes one or more of the specific words or phrases that the given warm word model is being trained to detect. For example, when training a given warm word model in the above example describing music playback, for a given training instance, the training instance input can include audio data corresponding to "turn up the volume," and for a given training instance, the training instance output can include a baseline truth output indicating that the audio data includes one or more of the specific words or phrases (e.g., "turn up the volume"). The training instance input can be applied as input to a given warm word model to generate a predicted output (e.g., a value, such as probability, log-likelihood, binary value, or other value) indicating the probability that the training instance input includes one or more of the specific words or phrases. The predicted output can be compared with the baseline ground truth output of the training instance input (e.g., a value indicating that the audio data of the training instance input actually includes one or more of a specific word or phrase, such as the value "1" in this example) to generate one or more losses, and the given warm word model can be updated based on one or more of these losses. In this example, a given training instance can be considered a positive training instance because the training instance input includes a portion of the audio data that the given warm word model is being trained to detect. Furthermore, in this example, the given warm word model can optionally be trained based on negative training instances, or on training instances with inputs that correspond to audio data that does not include one or more of the specific words or phrases that the given warm word model is being trained to detect.

[0048] For example, in the above example describing music playback, when training a given warm word model, for a given training instance, the training instance input could include audio data corresponding to “turn on the lights,” and for a given training instance, the training instance output could include a baseline ground truth output indicating whether the audio data includes or excludes one or more of a specific word or phrase (e.g., “turn up the volume”). The training instance input could be applied as input to the given warm word model to generate a predicted output (e.g., a value such as probability, log-likelihood, binary value, or other value) indicating the probability that the training instance input includes one or more of a specific word or phrase. The predicted output could be compared with the baseline ground truth output of the training instance input (e.g., a value indicating that the audio data of the training instance input actually includes one or more of a specific word or phrase, such as the value “1” in this example) to generate one or more losses, and the given warm word model could be updated based on one or more of the losses. In this example, the given training instance could be considered a negative training instance because the training instance input does not include any part of the audio data that the given warm word model is being trained to detect (e.g., “turn up the volume” or any other audio corresponding to controlling music).

[0049] In some implementations, and assuming that one or more of the automated assistant functions utilizing warm word models are active, the warm word detection engine 152 can use warm word models to process the stream of audio data generated by the microphone of the client device to detect the occurrence of one or more specific words or phrases. For example, suppose a text message is received at client device 110. Further suppose that the warm word activation event engine 151 detects that a text message is being received at client device 110 and activates one or more automated assistant components, which utilize one or more warm word models to detect specific words or phrases (e.g., “reply,” “ignore,” “show me,” etc.) of the warm word activation event of the text message being received at client device 110, until one or more conditions are met. In this example, further suppose that the user of client device 110 provides the spoken phrase “Reply: I’ll be there in five minutes.” Therefore, when processing the audio data stream, the warm word detection engine 152 can detect the occurrence of a portion of the audio data corresponding to the warm word “reply.” In some implementations, the warm word detection engine 152 can activate one or more additional currently active automated assistant components (e.g., ASR engine 120A1 and / or 120A2, NLU engine 130A1 and / or 130A2, and / or other automated assistant components) that utilize the ASR model and / or NLU model in response to the detection of a warm word.

[0050] As described in more detail below (e.g., regarding...) Figure 2 , 3(and 4A to 4C), ASR engines 120A1 and / or 120A2 can use ASR models stored in ML model database 115A to process portions of audio data corresponding to the occurrence of warm words, pre-sequence portions (if any) of audio data preceding the portions of audio data corresponding to the occurrence of warm words, and / or post-sequence portions (if any) of audio data following the portions of audio data corresponding to the occurrence of warm words, to generate ASR output. Furthermore, NLU engines 130A1 and / or 130A2 can use NLU models stored in ML model database 115A and / or syntax-based rules to process the ASR output to generate NLU output. In some versions of these implementations, ASR engines 120A1 and / or 120A2 can obtain one or more of these portions (e.g., pre-sequence portions of audio data (if any) and / or portions of audio data corresponding to the occurrence of warm words) from the audio buffer of client device 110. In various implementations, the ASR output may be biased towards specific terms and / or phrases associated with the warm word model. For example, if the warm word activation event corresponds to a media event (e.g., a user on client device 110 is listening to music on client device 110), the ASR output may be biased towards music-related terms and / or phrases, such as artist, song, genre, etc.

[0051] For example, the automated assistant 115 can use endpoint models stored in an ML model database 115A to process the audio data stream to generate multiple timestamps of spoken words captured in the audio data stream. The multiple timestamps may include at least: a first timestamp associated with a first time when the user begins to provide spoken words; a second timestamp associated with a second time after the first time when the user begins to provide one or more specific words or phrases included in the spoken words; a third timestamp associated with a third time after the second time when the user finishes providing one or more specific words or phrases included in the spoken words; and a fourth timestamp associated with a fourth time after the third time when the user finishes providing spoken words. In this example, the preceding portion of the audio data includes any audio data corresponding to spoken words between the first and second timestamps, the portion of the audio data includes any audio data corresponding to spoken words between the second and third timestamps, and the following portion of the audio data includes any audio data corresponding to spoken words between the third and fourth timestamps. It should be noted that this example is provided for illustrative purposes and is not intended to be limiting. For example, in some cases, such as when spoken words consist of only one or more specific words or phrases, multiple timestamps may include at least a first timestamp associated with the first moment when the user begins to deliver the spoken words, and a second timestamp associated with a second moment after the first moment when the user finishes delivering the spoken words. In this example, there may be no preceding or following portion of the audio data.

[0052] In some implementations, the warm word verification engine 153 can process the ASR output to determine whether the user intends to execute one or more specific words or phrases for the assistant command. For example, the warm word verification engine 153 can cause the NLU engines 130A1 and / or 130A2 to process the ASR output using an NLU model stored in the ML model database 115A or syntax-based rules to generate NLU output. The warm word verification engine 153 can determine whether the user intends to execute one or more specific words or phrases for the assistant command based on the ASR output and / or the NLU output. Continuing the example above, in response to a warm word (e.g., capturing the spoken phrase “Reply: I will be there in five minutes”), the ASR engines 120A1 and / or 120A2 and / or the NLU engines 130A1 and / or 130A2 can be activated and used to process the audio data stream to generate the ASR output of the audio data stream (e.g., capturing the spoken phrase “Reply: I will be there in five minutes”). Furthermore, NLU engines 130A1 and / or 130A2 can use NLU models stored in the ML model database 115A to process ASR outputs to generate NLU outputs. In this example, warm word verification engine 153 can determine that "reply" is a text message received at client device 110 that the user intends to reply with a reply message of "I will be there in five minutes".

[0053] Contrary to the example above, suppose that when a text message is received at client device 110, the user of client device 110 instead provides the spoken phrase "I'll reply later." In this example, warm word activation event engine 151 can detect the occurrence of a warm word activation event (e.g., the text message received at the client device) and cause one or more of the currently dormant auto-assistant functions utilizing the warm word model to be activated. Furthermore, warm word detection engine 152, when processing the stream of audio data using one or more of the warm word models activated for the warm word activation event, can detect the occurrence of one or more specific words or phrases (e.g., "reply") in portions of the audio data in the same or similar manner as described above. However, in this example, auto-assistant 115 can determine that the user's lack of intention to "reply" is based on the specific words or phrases that cause the auto-assistant to reply to the received text message, processed by subsequent portions of the audio data using ASR engines 120A1 and / or 120A2 and / or NLU engines 130A1 and / or 130A2. Therefore, in this example, even if the spoken words provided by the user include one or more specific words or phrases that would normally cause the execution of an assistant command (e.g., “reply”), the automatic assistant 115 can avoid executing the assistant command to reply to the text message.

[0054] In additional or alternative implementations, the automated assistant 115 may execute the assistant command only if the user authentication engine 160 determines that the identity of the user providing the spoken word has been verified and that the user providing the spoken word is authorized to execute the assistant command. In some versions of these implementations, only a subset of the assistant commands may require verification of the identity of the user providing the spoken word and authorization for the user to execute the assistant command. For example, assistant commands associated with telephone call warm word activation events, text message warm word activation events, and email warm word activation events may require verification of the identity of the user providing the spoken word and authorization for the user to execute the assistant command. However, assistant commands associated with media warm word activation events may not require verification of the identity of the user providing the spoken word and authorization for the user to execute the assistant command. Various techniques can be used to verify the identity of the user providing the spoken word.

[0055] In some versions of these implementations, the user verification engine 160 may use a speaker recognition (SID) model stored in the ML model database 115A to process the stream of audio data to generate speaker embeddings of the user providing spoken utterances. The generated speaker embeddings can be compared in the embedding space with one or more previously generated speaker embeddings of the user of the client device 110 (e.g., embeddings associated with the user's profile). For example, if a distance metric determined based on the comparison between the generated speaker embedding in the embedding space and one or more previously generated speaker embeddings for the user satisfies a threshold, the user can be verified as the user providing spoken utterances. One or more previously generated speaker embeddings of the user can be generated based on output generated from processing one or more instances of audio data, including spoken utterances from the user (e.g., when the client device 110 is initially configured). For example, one or more previously generated speaker embeddings can be based on the average (or other combination) of multiple distinct embeddings, each of which is generated based on processing a corresponding instance of audio data including corresponding spoken utterances from the user of the client device. In some versions of these implementations, the SID model may be a text-independent (TI) SID model, while in other implementations, the SID model may be a text-dependent (TD) SID model.

[0056] In TD SID, one or more previously generated speaker embeddings of a user are generated based on spoken utterances that include only one or more specific words or phrases. Furthermore, in use, the user must utter one or more specific words or phrases from one or more TD SID speaker embeddings to be generated using the TD SID model. These one or more TD SID speaker embeddings can be effectively compared with one or more previously generated TD speaker embeddings of the user to determine whether the spoken utterance comes from an authorized user (e.g., the user of client device 110 or another user associated with client device 110). For example, one or more specific words or phrases in TD speaker identification can be limited to one or more invocation phrases configured to invoke an automated assistant (e.g., hot words and / or trigger words, such as, for example, “Hey assistant,” “OK assistant,” and / or “assistant”) or one or more warm words as described herein. In contrast, in TI SID, spoken utterances processed using the TI SID model are not limited to one or more specific words or phrases. In other words, the TI SID model can be used to process audio data based on virtually any spoken utterance to generate TI SID speaker embeddings. These TI SID speaker embeddings can be effectively compared with one or more previously generated TI speaker embeddings of the user to determine whether the spoken utterance comes from an authorized user (e.g., the user of client device 110 or another user associated with client device 110). Furthermore, in various implementations, one or more previously generated TI speaker embeddings of the user used in the TISID are generated based on spoken utterances that include different words and / or phrases and are not limited to call words and / or phrases, warm words and / or any other specific spoken utterances.

[0057] In additional or alternative versions of these implementations, other biometric information may be utilized to verify the identity of the user providing the spoken words. Biometric information used to verify user identity may include, for example, facial recognition, fingerprint recognition, and / or other types of biometric information. For example, the visual component of the presence sensor 113 of the client device 110 may capture visual data including the face of the user providing the spoken words. The user verification engine 160 may process the visual data using a facial recognition (FID) model stored in the ML model database 115A to generate the facial fingerprint of the user providing the spoken words. The facial fingerprint of the user providing the spoken words may correspond to an embedding, which may be compared in an embedding space with one or more previously generated embeddings corresponding to the facial fingerprint of the user of the client device to determine whether the user providing the spoken words is authorized to execute an assistant command. As another example, the fingerprint sensor in the presence sensor 113 may receive touch input from the user providing the spoken words and compare the detected fingerprint with a known fingerprint of the user of the client device 110 to determine whether the user providing the spoken words is authorized to execute an assistant command.

[0058] In additional or alternative implementations, the acoustic engine 170 may process audio data to generate one or more acoustic characteristics associated with the audio data. In these implementations, the automated assistant 115 may additionally or alternatively consider one or more of the acoustic characteristics associated with the audio data to determine whether to execute an assistant command in response to the detection of a warm word in the audio data stream. The acoustic engine 170 may utilize various techniques to generate one or more acoustic characteristics associated with the audio data.

[0059] In some versions of these implementations, the acoustic engine 170 may use a speech activity detection (VAD) model stored in the ML model database 115A to process audio data to determine whether a preceding portion of audio data exists before a portion of audio data corresponding to one or more specific words or phrases and / or whether a following portion of audio data exists after a portion of audio data corresponding to one or more specific words or phrases. In some further versions of these implementations, the ASR engine 120A1 and / or 120A2 and / or the NLU engine 130A1 and / or 130A2 may be activated solely in response to determining the presence of a preceding portion and / or a following portion of audio data. For example, suppose that when a notification is received at client device 110, the user of client device 110 provides the spoken utterance “view”. In this example, the warm word activation event engine 151 may detect the occurrence of a warm word activation event (e.g., a notification received at client device) and cause one or more of the currently dormant automatic assistant functions utilizing the warm word model to be activated. Furthermore, when the warm word detection engine 152 processes the audio data stream using one or more warm word models activated for a warm word activation event, it can detect the occurrence of one or more specific words or phrases (e.g., "view") in portions of the audio data in the same or similar manner as described above. However, the acoustic engine 170 can determine, based on processing the audio data stream using a VAD model, the absence of a preceding portion of the audio data (e.g., before "view" in the audio data stream) or a following portion of the audio data (e.g., after "view" in the audio data stream). As a result, the ASR engines 120A1 and / or 120A2 and / or the NLU engines 130A1 and / or 130A2 may not be activated, and the automatic assistant 115 may provide a notification for presentation to the user (and optionally based on determining that the user is authorized to view the notification).

[0060] In additional or alternative versions of these implementations, the acoustic engine 170 may use a language recognition model stored in the ML model database 115A to process the audio data to determine the language of the spoken utterance captured in the audio data. In some versions of these implementations, the language of the spoken utterance may provide an indication of which portion of the audio data—either the preceding or following portion—provides contextual information to inform the automation assistant 115 whether to induce or avoid inducing one or more associated assistant commands corresponding to a particular word or phrase in the audio data portion. For example, in English, certain terms or phrases instructing against inducing an assistant command tend to precede (e.g., in the preceding portion of the audio data) the portion corresponding to one or more of a particular word or phrase (e.g., “don’t answer,” “don’t look,” etc.). However, in other languages ​​(e.g., German), certain terms or phrases instructing against inducing an assistant command tend to follow (e.g., in the following portion of the audio data) the portion corresponding to one or more of a particular word or phrase.

[0061] In additional or alternative implementations, the language of the spoken utterance can provide an indication of which warm word model(s) and / or which ASR model(s) should be used to further process the audio data stream. For example, assuming the output of the cross-language recognition model indicates that the user of the spoken utterance captured in the audio data stream is speaking English, the warm word model(s) and / or ASR model subsequently used by the automation assistant component could be English-specific. Then, assuming the output of the cross-language recognition model indicates that the user of the spoken utterance captured in the audio data stream is speaking German (or another language), the warm word model(s) and / or ASR model subsequently used by the automation assistant component could be German (or other language-specific). In additional or alternative implementations, one or more active automation assistant components can use warm word models trained to detect one or more specific words and / or phrases in different languages, and can determine which ASR model should subsequently be used by one or more automation assistant components to further process the audio data stream based on the language of one or more of the detected specific words and / or phrases. For example, assuming the output of the cross-warm word model indicates that the detected warm word is provided in Spanish, the ASR model subsequently used by the automation assistant component could be Spanish-specific. However, assuming that the output generated by the cross-warm word model indicates that the detected warm words are provided in English (or another language), the ASR model subsequently used by the automatic assistant component can be specific to English (or the other language).

[0062] In additional or alternative versions of these implementations, the acoustic engine 170 may use a prosodic feature classification model stored in the ML model database 115A to process the audio data to determine one or more prosodic features of spoken utterances captured in the audio data stream. The prosodic features of spoken utterances captured in the audio data stream may include one or more of, for example, intonation, pitch, stress, melody, rhythm, and pauses. One or more of these prosodic attributes may inform the automation assistant 115 whether to execute an assistant command associated with one or more specific words or phrases corresponding to a portion of the audio data, and / or avoid executing an assistant command associated with one or more specific words or phrases corresponding to a portion of the audio data. For example, if the spoken utterances captured in the audio data do not include any pauses and additional stress is applied to one or more specific words or phrases, one or more of the prosodic attributes of the spoken utterances may inform the automation assistant 115 that the user is more likely to intend to execute an assistant command. Conversely, if the spoken words captured in the audio data do indeed include pauses and indeterminate tones, one or more of the prosodic properties of the spoken words can inform the automated assistant 115 that the user is more likely not to execute the assistant's commands.

[0063] In additional or alternative implementations, the automation assistant 115 may analyze instances of sensor data generated by one or more of the presence sensors 113. The automation assistant 115 considers instances of sensor data generated by the one or more presence sensors 113 when determining whether to execute an assistant command in response to the detection of a warm word in the audio data stream. For example, if one or more visual components of the client device 110 indicate that the gaze of the user providing the spoken words is not directed at the client device 110, visual data may inform the automation assistant 115 that the user is more likely not to intend to execute the assistant command. Additionally or alternatively, if one or more accelerometers of the client device 110 indicate that the client device 110 is moving (and optionally in conjunction with the absence of the user's gaze), visual data may inform the automation assistant 115 that the user is more likely not to intend to execute the assistant command.

[0064] Turn now Figure 2 It describes the use Figure 1The various components illustrate example process flows of various aspects of this disclosure. The warm word activation event engine 151 can detect the occurrence of a warm word activation event 251. In response to detecting the occurrence of a warm word event, the warm word detection engine 152 can be activated and begin processing the stream of audio data 201A generated by one or more microphones 201 of the client device 110 using the warm word model associated with the warm word activation event 251 stored in the ML model database 115A. As shown in box 299A, in response to determining that the audio data does not correspond to one or more warm words, the warm word detection engine 152 can continue to monitor the occurrence of one or more warm words until one or more conditions are met.

[0065] However, as shown in box 299A, in response to determining that a portion of the audio data does indeed correspond to one or more warm words, the warm word detection engine 152 may enable the acquisition of a pre-context portion 201B (if any) of the audio data preceding the portion corresponding to one or more warm words and / or a post-context portion 201C (if any) of the audio data following the portion corresponding to one or more warm words. Furthermore, also in response to determining that a portion of the audio data does indeed correspond to one or more warm words, the warm word detection engine 152 may activate the ASR engines 120A1 and / or 120A2 and / or the NLU engines 130A1 and / or 130A2. The ASR engines 120A1 and / or 120A2 may use an ASR model stored in the ML model database 115A to process the pre-context portion 201B and / or the post-context portion 201C of the audio data (and optionally the portion corresponding to one or more warm words) to generate an ASR output 220 (e.g., for any speech recognition of the user included in the audio data). Furthermore, NLU engines 130A1 and / or 130A2 can use NLU models stored in the ML model database 115A to process ASR output 220 to generate NLU output 230 (e.g., annotations for any speech recognition of the user included in the audio data). Warm word verification engine 153 can determine, based on ASR output 220 and / or NLU output 230, whether the user actually intends for the automated assistant 115 to execute an assistant command.

[0066] For example, suppose that while listening to music, audio data 201A captures the spoken phrase “Can you turn up the volume?” via client device 110 or an additional client device (e.g., a smart speaker) communicating with client device 110. In this example, warm word activation event 251 may correspond to a media event, and the warm word “turn up the volume” can be detected in the audio data capturing the spoken phrase using one or more warm word models associated with the media event. Furthermore, ASR engines 120A1 and / or 120A2 can use ASR models to process the preceding portion 201B of the audio data corresponding to “can you turn it up” and / or the portion of the audio data corresponding to “turn up the volume” to generate recognized text corresponding to “can you turn it up” and / or “turn up the volume” (e.g., as ASR output 220). Furthermore, NLU engines 130A1 and / or 130A2 can use an NLU model to process the identified text corresponding to “you can turn it up” and / or “louder” to generate annotated identified text (e.g., as NLU output 230), which annotates one or more terms of the identified text with various annotations. In this example, and as shown in box 299B, the automatic assistant 115 can determine, based on the identified text and / or the annotated identified text, that an assistant command 215A should be executed to adjust the volume of the music being played, and cause the assistant command 215A to be executed (e.g., by turning up the volume at client device 110 and / or transmitting a request to turn up the volume to an additional client device communicating with client device 110).

[0067] Conversely, suppose that while listening to music, audio data 201A captures the spoken phrase "Please don't turn up the volume" via client device 110 or an additional client device (e.g., a smart speaker) communicating with client device 110. In this example, warm word activation event 251 may correspond to a media event, and the warm word "turn up the volume" can be detected in the audio data capturing the spoken phrase using one or more warm word models associated with the media event. Furthermore, ASR engines 120A1 and / or 120A2 can use ASR models to process the preceding portion 201B of the audio data corresponding to "Please don't turn up the volume" and / or the portion of the audio data corresponding to "turn up the volume" to generate recognized text corresponding to "Please don't turn up the volume" and / or "turn up the volume" (e.g., as ASR output 220). Furthermore, NLU engines 130A1 and / or 130A2 can use an NLU model to process the identified text corresponding to "Please do not adjust" and / or "High volume" to generate annotated identified text (e.g., as NLU output 230), which annotates one or more terms of the identified text with various annotations. In this example, and as shown in box 299B, the automatic assistant 115 can determine, based on the identified text and / or the annotated identified text, that an assistant command to adjust the volume of the music being played should not be executed, and avoid executing the assistant command. In some of these examples, audio data 201A and / or any other data generated based on audio data 201A can be discarded, as shown in 215B.

[0068] In some implementations, whether the automated assistant 115 determines to execute an assistant command may be based on whether the identity of the user providing the spoken utterance has been verified, and whether the identified user is authorized to execute the assistant command (e.g., using information such as regarding...). Figure 1 , 5 (As determined by the user authentication engine 160 described in 6A to 6C). For example, in an implementation where the warm word activation event corresponds to electronic communication received at client device 110 (e.g., telephone call, text message, email, social media message, etc.), it may be necessary to authenticate the user before the automated assistant 115 executes the assistant command. As another example, in an implementation where the warm word activation event corresponds to some controlled device (e.g., smart lock, smart appliance such as a smart oven, smart garage door opener, etc.), it may be necessary to authenticate the user before the automated assistant 115 executes the assistant command.

[0069] Despite about Figure 2The processing flow described herein is for illustrative purposes only and is not intended to be limiting. For example, suppose audio data 201A captures the spoken phrase “show me the way home.” In this example, the detected warm word activation event 251 could correspond to a navigation application accessible at client device 110. Therefore, one or more warm word models associated with navigation events could continuously monitor the occurrence of one or more navigation-related warm words. Furthermore, the automated assistant could utilize additional or alternative signals to determine whether to execute an assistant command (e.g., as described with respect to acoustic engine 170).

[0070] By using the techniques described herein, one or more technical advantages can be achieved. As a non-limiting example, the techniques described herein enable automated assistants to determine, based on contextual audio data processing warm words, whether a user actually intends for the warm words to appear so that an assistant command can be executed. Therefore, the number of false positives for warm words can be reduced, thereby saving computational and / or network resources at the client device when the automated assistant utilizes one or more networks to execute assistant commands. Furthermore, since the user on the client device does not need to take action to correct or cancel any assistant commands, the amount of user input can be reduced, thus saving computational resources at the client device. Additionally, the various automated assistant functions described herein can remain dormant until required as described herein, thereby saving computational resources at the client device. As another non-limiting example, the techniques described herein enable automated assistants to use SID and / or other techniques to verify the identity of the user providing spoken utterances including warm words. Therefore, only authorized users can execute certain assistant commands, thereby improving data security associated with the user on the client device.

[0071] Turn now Figure 3 A flowchart illustrating an example method 300 for determining whether to execute an assistant command based on ambient audio data is depicted. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes a computing device (e.g., Figure 1 , 4A Client devices 110 and / or 4C and 6A to 6C Figure 7 The computing device 710, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 300 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, and / or added.

[0072] At box 352, the system monitors for the occurrence of warm word activation events. Warm word activation events can include, for example, a telephone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, media played at the client device or an attached client device in the client device's environment, a notification received at the client device, the client device's location, a software application accessible at the client device, and / or other events associated with the client device, where a user can provide verbal commands to control the client device or an attached client device communicating with it. It is worth noting that some of these warm word activation events are discrete events (e.g., a telephone call received at the client device, a text message received at the client device, an email received at the client device, an alarm or timer sounding at the client device, etc.), while some other warm word activation events are consecutive events (e.g., a software application accessible at the client device). Furthermore, it should be noted that these warm word activation events are not mutually exclusive. In other words, the system can detect multiple warm word activation events at a given time instance. If the system does not detect the occurrence of a warm word activation event during the iteration of box 352, the system can continue to monitor for the occurrence of warm word activation events at box 352. If the system detects the occurrence of a warm word activation event during the iteration of box 352, the system can proceed to box 354.

[0073] At box 354, the system uses one or more warm word models to process the audio data stream to monitor the occurrence of one or more specific words and / or phrases (e.g., warm words). In response to the detection of a warm word activation event at box 352, the system can cause one or more automation assistant components to utilize one or more warm word models. The one or more warm word models used to process the audio data stream can be associated with the warm word activation event detected at box 352. For example, if the warm word activation event detected at box 352 corresponds to a telephone call received at a client device, the audio data stream can be processed using one or more warm word models associated with the telephone call event. As another example, if the warm word activation event detected at box 352 corresponds to media played at a client device, the audio data stream can be processed using one or more warm word models associated with the media event.

[0074] At box 356, the system determines whether the occurrence of one or more specific words and / or phrases is detected in the audio data stream. The system may continue to monitor the occurrence of one or more specific words and / or phrases until one or more conditions are met. One or more conditions may include, for example, the end of a threshold duration, the detection of one or more specific words or phrases, the determination that a warm word activation event has ended, and / or other conditions. If the system does not detect the occurrence of one or more specific words and / or phrases in the iteration of box 356, the system may return to box 354 and continue processing the audio data stream and monitoring the occurrence of one or more specific words and / or phrases in box 356 until one or more conditions are met. If the system does detect the occurrence of one or more specific words and / or phrases in the iteration of box 356, the system proceeds to box 358.

[0075] At box 358, the system determines whether there is additional voice activity besides the warm word (e.g., before or after the warm word). The system can use a VAD model to process the audio data stream to determine whether the audio data stream captures any additional audio data besides the portion of audio data corresponding to one or more of the specific words and / or phrases. In other words, the system can utilize the VAD model to determine whether the user has provided any additional speech besides one or more of the specific words and / or phrases, such that the audio data corresponding to the additional speech can be used as contextual audio data (e.g., the preceding and / or following portions of the audio data) when interpreting one or more of the specific words and / or phrases. If, in the iteration of box 358, the system determines that there is no additional voice activity besides the warm word, the system proceeds to box 366 and causes the automated assistant to execute the assistant command associated with one or more of the specific words and / or phrases. It is worth noting that the system can bypass ASR and / or NLU when proceeding to box 366. If, in the iteration of box 358, the system determines that there is additional voice activity besides the warm word, the system proceeds to box 360. In some implementations, the operation of box 358 can be omitted, and the system can proceed directly to box 360.

[0076] At box 360, the system uses an ASR model to process the preceding and / or following portions of the audio data to generate ASR output. Additionally or alternatively, the system may combine the preceding and / or following portions of the audio data (e.g., as a single instance of audio data) to process portions of the audio data corresponding to warm words using an ASR model to generate ASR output. One or more automated assistant components utilizing the ASR model may be activated in response to the detection of the occurrence of one or more of a specific word and / or phrase at box 356 and / or in response to the determination of additional voice activity at box 358. In various embodiments, the ASR output may be biased towards one or more of the specific words and / or phrases associated with the warm word model. At box 362, the system uses an NLU model to process the ASR output to generate NLU output. Similar to what has been described above regarding box 360, one or more automated assistant components utilizing the NLU model may be activated in response to the detection of the occurrence of one or more of a specific word and / or phrase at box 356 and / or in response to the determination of additional voice activity at box 358.

[0077] At box 364, the system determines whether the user, who provided spoken utterances including warm words and captured in the audio data stream, intends for the automated assistant to execute an assistant command associated with the warm words detected at box 356. The system may determine this intention based on the ASR output generated at box 360 and / or the NLU output generated at box 362. If, in the iteration of box 364, the system determines that the user intends for the automated assistant to execute an assistant command, the system proceeds to box 366. At box 366, the system causes the automated assistant to execute the assistant command associated with the warm words detected at box 356. If, in the iteration of box 364, the system determines that the user does not intend for the automated assistant to execute an assistant command, the system proceeds to box 368. At box 368, the system avoids causing the automated assistant to execute the assistant command associated with the warm words detected at box 356.

[0078] Turn now Figure 4A , 4B And 4C, describing the use of Figure 3 Method 300 provides various non-limiting examples of determining whether to execute an assistant command based on ambient audio data. The automated assistant may be implemented at least partially at the client device 110 (e.g., regarding...). Figure 1 The described automatic assistant 115). The automatic assistant can utilize the warm word system 180 (e.g., about). Figure 1 The described warm word system) determines whether to execute assistant commands (e.g., regarding) based on contextual audio data. Figure 3 Method 300 describes.

[0079] Figure 4A, 4B The client device 110 described in 4C may include various user interface components, including, for example, a microphone for generating audio data based on spoken words and / or other auditory input, a speaker for auditory rendering of synthesized speech and / or other auditory output, and a display 190 for receiving touch input and / or visual rendering of transcription and / or other visual output. Furthermore, the display 190 of the client device 110 includes various system interface elements (e.g., hardware and / or software interface elements) that a user of the client device 110 can interact with to cause the client device 110 to perform one or more actions (e.g., regarding...). Figure 6B and 6C As described). Although Figures 4A to 4C The client device 110 depicted is a mobile device, but it should be understood that this is for illustrative purposes and does not imply limitation.

[0080] For details, please refer to the following: Figure 4A Assume that user 101 and additional user 102 associated with client device 110 are both located in environment 400. Further assume that a telephone call is received at client device 110. In this example, the auto-assistant can determine that the telephone call received at client device 110 corresponds to a warm word activation event. Therefore, the auto-assistant can activate one or more currently dormant auto-assistant functions utilizing one or more warm word models associated with the telephone call event. One or more warm word models associated with the telephone call event can be used to process the stream of audio data generated by the microphone of client device 110 to monitor the occurrence of specific words and / or phrases associated with the telephone call event (e.g., “answer,” “reject,” “screen call,” “send to voicemail,” etc.), and when a specific word and / or phrase is detected, cause the auto-assistant to perform a specific action associated with that specific word and / or phrase.

[0081] For example, and as Figure 4AAs shown, suppose that when client device 110 is ringing, user 101 of client device 110 provides the verbal utterance 452A of "answer". In this example, when processing the stream of audio data generated by the microphone of client device 110 using one or more warm word models associated with the phone call event, the automated assistant can determine that the audio data corresponds to the warm word "answer", and when the warm word is detected, cause the automated assistant to answer the phone call on behalf of user 101. Furthermore, the automated assistant can use a VAD model to process the stream of audio data to determine if there is any additional voice activity. In this example, the output generated using the VAD model will indicate that there is no additional voice activity, and therefore no contextual audio data to be processed to determine whether user 101 actually intends to verbally utter 452A to have the phone call answered. Therefore, the automated assistant will cause the phone call to be answered at client device 110.

[0082] On the contrary, and as Figure 4B As shown, suppose that while client device 110 is ringing, user 101 of client device 110 provides the spoken words 452B, "I won't answer now, I'll call them back later." In this example, when processing the stream of audio data generated by the microphone of client device 110 using one or more warm word models associated with the phone call event, the automated assistant can determine that a portion of the audio data corresponds to the warm word "answer," and when this warm word is detected, cause the automated assistant to answer the phone call on behalf of user 101. Furthermore, the automated assistant can use a VAD model to process the stream of audio data to determine if there is any additional voice activity. In this example, the output generated using the VAD model will indicate the presence of additional voice activity, and therefore, there is contextual audio data to be processed to determine whether user 101 actually intends to use the spoken words 452B to have the phone call answered. Therefore, the automated assistant will obtain the preceding and / or following portions of the audio data.

[0083] In this example, the preceding portion of the audio data corresponds to the spoken word "I don't" in utterance 452B. The automated assistant can process the preceding portion of the audio data using an ASR model to generate an ASR output for the preceding portion of the audio data. Furthermore, the automated assistant can process the ASR output using an NLU model to generate an NLU output for the preceding portion of the audio data. The automated assistant can determine whether user 101 actually intends for the spoken word 452B to be answered based on the ASR output and / or NLU output of the preceding portion of the audio data. For example, the automated assistant can determine based on the ASR output and / or NLU output that "no" negates the warm word "answer". Therefore, in this example, the automated assistant will avoid answering the phone call at client device 110 based on the processing of the preceding portion of the audio data. In various embodiments, one or more negative warm word models may be used additionally or alternatively in addition to the warm word models described herein. In these embodiments, one or more negative warm word models may be used to detect other words and / or phrases (e.g., "no", "cannot", "don't") that are very close to the warm word of the negative warm word. Furthermore, if one or more of these negative warm word models detect other words and / or phrases that are very close to the warm word, one or more automatic assistant components utilizing the ASR model and / or NLU model can remain dormant.

[0084] In some implementations, and assuming that the automated assistant cannot determine whether user 101 actually intends to answer the phone call based on the preceding portion of the processed audio data, the automated assistant can further process the following portion of the audio data using an ASR model to generate an ASR output for the following portion of the audio data (e.g., "call them back later" corresponding to the spoken words 452B). Additionally, the automated assistant can process the ASR output using an NLU model to generate an NLU output for the following portion of the audio data. The automated assistant can determine whether user 101 actually intends to answer the phone call based on the ASR output and / or NLU output of the following portion of the audio data. For example, the automated assistant can determine that "call them back later" indicates that user 101 does not intend to answer the call at this moment, causing the following portion of the audio data to negate the warm word "answer" based on the ASR output and / or NLU output. Therefore, in this example, the automated assistant will avoid answering the phone call at the client device 110 based on further processing of the following portion of the audio data.

[0085] As another example, and as Figure 4CAs shown, assuming that when client device 110 is ringing, an additional user 102, co-located in the environment of user 101 associated with client device 110, provides the spoken phrase 452C, "Do you want to answer the phone?". In this example, when processing the stream of audio data generated by the microphone of client device 110 using one or more warm word models associated with the phone call event, the automated assistant can determine that the audio data corresponds to the warm word "answer," and when the warm word is detected, cause the automated assistant to answer the phone call on behalf of user 101. Furthermore, the automated assistant can use a VAD model to process the stream of audio data to determine if any additional voice activity exists. In this example, the output generated using the VAD model will indicate the presence of additional voice activity, and therefore, the existence of contextual audio data to be processed. However, compared to... Figure 4A and 4B In contrast, the spoken words 452C are provided by the attached user 102 rather than by the user 101 associated with the client device 110.

[0086] In additional or alternative implementations, and as per [the relevant provisions] Figure 5 and Figures 6A to 6C As described, before the automated assistant executes the assistant command, it may be necessary to verify the identity of the user providing the spoken words 452A, 452B, and / or 452C. Furthermore, the automated assistant may additionally or alternatively need to determine whether the identified user is authorized to execute the assistant command. These assistant commands that the automated assistant needs to verify the identity of the user providing the spoken words and / or determine whether the user is authorized may be assistant commands associated with a specific word model, assistant commands associated with a specific software application accessible at client device 110, assistant commands associated with a specific device to be controlled based on the assistant command, and / or other subsets of assistant commands.

[0087] Nevertheless, and regarding Figure 4CThe automated assistant can still obtain the preceding and / or following portions of the audio data. In other words, if the spoken words are not intended to cause the assistant to execute a command, the automated assistant may not attempt to verify the identity of the additional user 102 who provided the spoken words 452C. In this example, the preceding portion of the audio data corresponds to “you want” in the spoken words 452C. The automated assistant can process the preceding portion of the audio data in the same or similar manner as described above to generate ASR and NLU outputs for the preceding portion of the audio data (and optionally combine portions of the audio data corresponding to one or more of a particular term or phrase and / or the following portion of the audio data (e.g., as a single instance of audio data or multiple different instances of audio data)). Furthermore, the automated assistant can determine, based on the ASR and / or NLU outputs of the preceding portion of the audio data, whether the additional user 102 actually intends for the spoken words 452C to be answered. For example, the automated assistant can determine, based on the ASR and / or NLU outputs, that “you want” corresponds to a question posed by the additional user 102 to the user 101 associated with the client device 110, and is not intended to be answered. Therefore, in this example, the automatic assistant will avoid answering the telephone call at the client device 110 based on the processing of the preceding part of the audio data (and optionally further processing of the subsequent audio data corresponding to the spoken words "telephone" in the 452C).

[0088] Although specific warm words were used to describe making certain assistant commands execute. Figures 4A to 4C However, it should be understood that this is for illustrative purposes and not intended to be limiting. Conversely, it should be understood that regarding Figures 4A to 4C The described technique can be used to detect any warm word activation event, detect any warm word using a warm word model associated with the warm word activation event, and enable any assistant command to be executed.

[0089] Turn now Figure 5 A flowchart illustrating an example method 500 for determining whether to execute an assistant command based on speaker recognition is provided. For convenience, the operation of method 500 is described with reference to a system that performs the operation. This system of method 500 includes a computing device (e.g., Figure 1 , 4A Client devices 110 and / or 4C and 6A to 6C Figure 7 The computing device 710, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 500 are shown in a specific order, this does not imply limitation. One or more operations may be reordered, omitted, and / or added.

[0090] At box 552, the system monitors for the occurrence of warm word activation events. If the system does not detect the occurrence of a warm word activation event during the iteration of box 552, it can continue monitoring for warm word activation events at box 552. If the system detects the occurrence of a warm word activation event during the iteration of box 552, it can proceed to box 554. At box 554, the system processes the audio data stream using one or more warm word models to monitor for the occurrence of one or more specific words and / or phrases (e.g., warm words). At box 356, the system determines whether the occurrence of one or more specific words and / or phrases is detected in the audio data stream. If the system does not detect the occurrence of one or more specific words and / or phrases during the iteration of box 556, it can return to box 554 and continue processing the audio data stream and monitoring for the occurrence of one or more specific words and / or phrases at box 556 until one or more conditions are met. If the system does detect the occurrence of one or more specific words and / or phrases during the iteration of box 556, it proceeds to box 558. The operations in boxes 552, 554, and 556 can be respectively related to Figure 3 The same or similar methods are performed as described in boxes 352, 354 and 356 of method 300.

[0091] At box 558, the system determines whether to verify the identity of the user providing spoken utterances, including one or more specific words and / or phrases, captured in the audio data stream. The system may determine whether to verify the identity of the user providing the spoken utterances based on one or more specific words and / or phrases associated with a specific warm word model, a specific software application accessible on the user's client device, a specific device to be controlled based on an assistant command, and / or other subset of the assistant commands. For example, in an implementation where the assistant command causes media to play on the client device or controls a smart light, user identity verification may not be required. However, in an implementation where the assistant command causes a response to an incoming electronic communication (e.g., a phone call, text message, email, social media message, etc.) or controls a smart lock, user identity verification may be required. If, in the iteration of box 558, the system determines that user identity verification is not required, the system can proceed to... Figure 3 Box 358 and continue execution Figure 2 Method 300. If, in the iteration of box 558, the system determines that it needs to verify the identity of the user providing the spoken utterance, the system can proceed to box 560.

[0092] At box 560, the system determines whether the audio data included in the audio data stream is sufficient to perform a TI SID. The system can determine this by comparing the length of the spoken utterance captured in the audio data stream to a threshold length. If the spoken utterance meets the threshold length, the system can determine that the audio data is sufficient to perform a TI SID. If, in the iteration of box 560, the system determines that the audio data is sufficient to perform a TI SID, the system can proceed to box 562. At box 562, the system processes the audio data using a TI SID model to verify the user's identity. At box 566, starting from box 562, the system determines whether the user's identity has been verified. For example, the system can process the audio data using a TI SID model to generate a TI speaker embedding for the user providing the spoken utterance. Furthermore, the system can compare the TI speaker embedding for the user providing the spoken utterance in the embedding space with one or more previously generated TI speaker embeddings for one or more users of the client device. Additionally, the system can determine whether the TI speaker embedding corresponds to one or more of the previously generated TI speaker embeddings based on a distance metric determined by the comparison. If, in the iteration of box 566, starting from box 562, the system determines that the distance metric meets the distance threshold, the user providing the spoken words can be authenticated as a user associated with the client device, and assuming the identified user is authorized to execute assistant commands, the system can enable the automated assistant to execute the assistant commands at box 568. However, if, in the iteration of box 566, starting from box 562, the system determines that the distance metric does not meet the distance threshold, the system can generate and provide a prompt at box 570 for additional verification of the user's identity (e.g., requesting the user providing the spoken words to provide facial or fingerprint recognition) to enable the automated assistant to execute assistant commands.

[0093] If, during the iteration at box 560, the system determines that the audio data is insufficient to perform TI SID, the system can proceed to box 564. At box 564, the system processes the audio data using a TD SID model to verify the user's identity. At box 566, starting from box 562, the system determines whether the user's identity has been verified. For example, the system can process the audio data using a TD SID model to generate a TD speaker embedding for the user providing the spoken utterance. Furthermore, the system can compare the TD speaker embedding for the user providing the spoken utterance in the embedding space with one or more previously generated TI speaker embeddings for one or more users of the client device. Notably, the TD speaker embedding corresponds to the speaker embedding for one or more specific words and / or phrases (e.g., a portion of the audio data corresponding to the warm word detected at box 556). Additionally, the system can determine whether the TD speaker embedding corresponds to one or more previously generated TD speaker embeddings for one or more of the specific words and / or phrases based on a distance metric determined by the comparison. If, in the iteration of box 566, starting from box 564, the system determines that the distance metric meets the distance threshold, the user providing the verbal utterance can be authenticated as a user associated with the client device, and assuming the identified user is authorized to execute the assistant command, the system can enable the automated assistant to execute the assistant command at box 568. However, if, in the iteration of box 566, starting from box 564, the system determines that the distance metric does not meet the distance threshold, the system can generate and provide prompts at box 570 for additional verification of the user's identity (e.g., an example utterance requesting the user providing the verbal utterance to provide TI SID verification or TD SID verification, and / or requesting the user providing the verbal utterance to provide facial recognition or fingerprint recognition) to enable the automated assistant to execute the assistant command.

[0094] It is worth noting that, although the description of separately executing TI SID and TD SID to verify the identity of the user providing the verbal message is provided... Figure 5However, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, in implementations where the length of the spoken utterance is sufficient to perform TI SID, both TI SID and TD SID can be performed to verify the identity of the user providing the spoken utterance. As another example, in various implementations, TD SID can only be performed if it is determined that one or more of the previously generated TD speaker embeddings correspond to one or more specific words and / or phrases captured in the spoken utterance. Furthermore, these techniques can be used to verify portions of spoken utterance captured in a stream of audio data provided by the same user. For example, suppose the stream of audio data captures a first spoken utterance “no” from a first user and a second spoken utterance “answer” from a second user. Further suppose the second user is associated with a client device and is authorized to answer, and a TD speaker embedding for the second user’s “answer” spoken utterance was previously generated and stored at the client device. In this case, the TD speaker embedding generated using the TD SID model can indicate that the second user is an experienced user authorized to execute assistant commands to answer incoming telephone calls at the client device. Furthermore, multiple TI speaker embeddings for the spoken words "no" and "answer" can instruct different users to provide spoken words captured in the audio data stream. Therefore, in this example, the system could enable an automated assistant to answer an incoming phone call.

[0095] Turn now Figure 6A , 6B And 6C, describing the use of Figure 5 Method 500 provides various non-limiting examples of methods based on speaker recognition to determine whether to execute an assistant command. The automated assistant may be implemented at least partially at the client device 110 (e.g., regarding...). Figure 1 The described automatic assistant 115). The automatic assistant can utilize the warm word system 180 (e.g., about). Figure 1 The described warm word system) determines whether to execute assistant commands (e.g., regarding) based on contextual audio data. Figure 5 Method 500 describes.

[0096] Similar to Figures 4A to 4C The client device described in the document, Figures 6A to 6C The client device 110 depicted may include various user interface components, including, for example, a microphone for generating audio data based on spoken words and / or other auditory input, a speaker for auditory rendering of synthesized speech and / or other auditory output, and a display 190 for receiving touch input and / or visual rendering of transcription and / or other visual output. Furthermore, and as... Figure 6B and 6CAs shown, the display 190 of client device 110 includes various system interface elements 191, 192, and 193 (e.g., hardware and / or software interface elements) that a user of the client device can interact with to cause client device 110 to perform one or more actions. The display 190 of client device 110 enables the user to interact with content rendered on display 190 via touch input (e.g., by directing user input to display 190 or a portion thereof (e.g., to a text input box 194 or another portion of display 190)) and / or via verbal input (e.g., by selecting microphone interface element 195—or simply by speaking without having to select microphone interface element 195 (i.e., the automated assistant can monitor one or more specific terms or phrases, gaze posture, mouth movements, lip movements, and / or other conditions to activate verbal input at client device 110)). Although Figures 6A-6C The client device 110 depicted is a mobile device, but it should be understood that this is for illustrative purposes and does not imply limitation.

[0097] For details, please refer to the following: Figure 6A Assume that user 101 and additional user 102, associated with client device 110, are both located in environment 600. Further assume that a telephone call is received at client device 110. In this example, the auto-assistant can determine that the telephone call received at client device 110 corresponds to a warm word activation event. Therefore, the auto-assistant can activate one or more currently dormant auto-assistant functions utilizing one or more warm word models associated with the telephone call event. One or more warm word models associated with the telephone call event can be used to process the stream of audio data generated by the microphone of client device 110 to monitor the occurrence of specific words and / or phrases associated with the telephone call event (e.g., “answer,” “reject,” “screen call,” “send to voicemail,” etc.), and when a specific word and / or phrase is detected, cause the auto-assistant to perform a specific action associated with that specific word and / or phrase.

[0098] For example, such as Figure 6A As shown, assuming that when the client device 110 is ringing, the attached user 102 provides the verbal phrase 652A to "answer". It is worth noting that, with... Figure 4AIn contrast, the spoken word "answer" 652A is provided by the additional user 102, not by the user 101 associated with the client device 110. In this example, when processing the stream of audio data generated by the microphone of the client device 110 using one or more warm word models associated with a phone call event, the automated assistant can determine that the audio data corresponds to the warm word "answer," and upon detecting the warm word, cause the automated assistant to answer the phone call. Further assuming that the identity of the user providing the spoken word 652A (e.g., additional user 102) needs to be verified for the automated assistant to answer the call.

[0099] In this example, the automated assistant can determine whether the length of spoken utterance 652A is sufficient to perform TI SID to verify the identity of the user providing the spoken utterance. In this example, it is further assumed that the length of spoken utterance 652A is insufficient to perform TI SID. However, the automated assistant can attempt to use TD SID to verify the identity of the user providing spoken utterance 652A (and assume the existence of one or more previously generated TD speaker embeddings for "answering," regardless of whether TI SID is performed). For example, the automated assistant can use a TD SID model to process portions of audio data corresponding to one or more specific words and / or phrases to generate a TD speaker embedding for "answering." Furthermore, the automated assistant can compare the TD speaker embedding for "answering" with one or more previously generated speaker embeddings for "answering" in the embedding space. Additionally, the automated assistant can determine whether the user providing spoken utterance 652A is user 101 of client device 110 based on a distance metric determined based on the comparison. In this example, the automated assistant can determine that the TD speaker embedding for "answering" does not match one or more previously generated speaker embeddings for "answering." Therefore, since the automated assistant does not verify the identity of the user providing the verbal utterance 652A (e.g., additional user 102), the automated assistant can avoid executing assistant commands to answer phone calls on behalf of user 101.

[0100] However, in various implementations, the automated assistant may generate prompts to be presented to the user 101 (e.g., visual and / or auditory), requesting the user to provide additional verification for the assistant command to be executed. See, for example, specific references. Figure 6BThe automated assistant can generate a prompt 652B stating "Please say [example utterance] to verify your identity and answer the call," and make the prompt available for visual presentation to the user via the automated assistant application 198A accessible at client device 110. Therefore, user 101 of client device 110 can respond to the prompt by providing additional user input to verify whether user 101 actually intends to have the automated assistant answer the call. In this example, the automated assistant can request user 101 to provide an example utterance, which is long enough to perform a TI SID on the example utterance, or corresponds to another specific term or phrase that client device 110 can access one or more previously generated TD speaker embeddings. It is assumed that user 101 of client device 110 does not provide a spoken utterance to have the automated assistant answer the call (e.g., in...). Figure 6A In the example (e.g.), user 101 can ignore or disregard prompt 652B. However, assuming that user 101 of client device 110 does provide verbal instructions to enable the automated assistant to answer phone calls (e.g., in...), Figure 4A In the example), user 101 can provide example utterances.

[0101] As another example, and for more details please refer to Figure 6C The automated assistant can generate a prompt 652C stating "Please provide facial or fingerprint recognition to answer the call," and make the prompt visually present to the user via the automated assistant application 198A accessible at client device 110. Therefore, user 101 of client device 110 can provide additional user input in response to the prompt to verify whether user 101 actually intends to have the automated assistant answer the call. In this example, if the identity of the user providing the verbal utterance cannot be identified, the automated assistant can request this biometric identification from user 101. In some cases, this biometric information may be requested simply in response to determining that the TD SID and TD SID are insufficient to identify the user providing the verbal utterance. Assume that user 101 of client device 110 has not provided verbal utterance to have the automated assistant answer the call (e.g., in...). Figure 6A In the example (e.g.), user 101 can ignore or disregard prompt 652C. However, assuming that user 101 of client device 110 does provide verbal instructions to enable the automated assistant to answer phone calls (e.g., in...), Figure 4A In the example (in which user 101 can provide biometric information), user 101 can provide biometric information.

[0102] although Figure 6B and 6CThe prompts 652B and 652C depicted are provided for presentation to user 101 via an automated assistant application 198A accessible at client device 110; however, it should be understood that this is for illustrative purposes and does not imply limitation. For example, prompts 652B and 652C may additionally or alternatively be auditorily rendered to be presented to the user via the speaker of client device 110, and visually rendered as notifications (e.g., pop-up notifications, banner notifications, etc.) to be presented to the user via the display 190 of client device 110. Furthermore, although specific terms are described regarding the execution of particular assistant commands... Figures 6A to 6C However, it should be understood that this is for illustrative purposes and not intended to be limiting. Conversely, it should be understood that regarding Figures 6A to 6C The described technique can be used to detect any warm word activation event, detect any warm word using a warm word model associated with the warm word activation event, and execute any assistant command when the user providing the spoken words is verified as an authorized user.

[0103] Turn now Figure 7 This diagram depicts a block diagram of an example computing device 710, which may optionally be used to perform one or more aspects of the techniques described herein. In some embodiments, one or more of a client device, a cloud-based automated assistant component, and / or other components may include one or more components of the example computing device 710.

[0104] Computing device 710 typically includes at least one processor 714 that communicates with a plurality of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724, which includes, for example, a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow users to interact with computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0105] User interface input device 722 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into a display, audio input devices such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to include all possible types of means and methods for inputting information into computing device 710 or a communication network.

[0106] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visible images. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of means and methods for outputting information from computing device 710 to a user or another machine or computing device.

[0107] Storage subsystem 724 stores the programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include selected aspects that perform the methods disclosed herein and implementations. Figure 1 and 2 The logic of the various components shown.

[0108] These software modules are typically implemented by processor 714 alone or in combination with other processors. The memory 725 used in storage subsystem 724 may include multiple memories, including main random access memory (RAM) 730 for storing instructions and data during program execution and read-only memory (ROM) 732 for storing fixed instructions. File storage subsystem 726 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 726 in storage subsystem 724 and / or in other machines accessible to processor 714.

[0109] The bus subsystem 712 provides a mechanism for enabling the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is schematically shown as a single bus, alternative implementations of the bus subsystem 712 may use multiple buses.

[0110] The computing device 710 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 7 The description of the computing device 710 depicted herein is intended only as a specific example to illustrate some embodiments. Many other configurations of the computing device 710 may have... Figure 7 The computing device depicted in the document has more or fewer components.

[0111] In situations where the systems described herein collect or otherwise monitor personal information about users, or may use personal information and / or monitored information, users may be given the opportunity to control whether a program or feature collects user information (e.g., information about a user's social networks, social behaviors or activities, occupation, user interests, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, data may be processed in one or more ways before being stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed so that personally identifiable information cannot be determined for the user, or a user's geographic location may be generalized (such as city, zip code, or state level) where geographic location information is obtained, making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or how that information is used.

[0112] In some implementations, a method implemented by one or more processors is provided, comprising: processing a stream of audio data using a warm word model to monitor the occurrence of one or more specific words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, and each of the one or more specific words or phrases being associated with an assistant command; in response to determining that a portion of the audio data corresponds to one or more of the specific words or phrases: processing a preceding portion and / or a following portion of the audio data using an automatic speech recognition (ASR) model to generate an ASR output; and determining, based on the processed ASR output, whether the user intends the one or more specific words or phrases to cause the assistant command to be executed. The preceding portion of the audio data precedes the portion of the audio data corresponding to the one or more specific words or phrases, and the following portion of the audio data follows the portion of the audio data corresponding to the one or more specific words or phrases. The method further comprises: in response to determining that the user does not intend the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: preventing the automatic assistant from executing the assistant command associated with one or more of the specific words or phrases; and in response to determining that the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: causing the automatic assistant to execute the assistant command associated with one or more of the specific words or phrases.

[0113] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0114] In some implementations, the method may further include detecting the occurrence of a warm word activation event; and, in response to detecting the occurrence of a warm word activation event, activating one or more currently dormant automatic assistant functions utilizing a warm word model. Processing a stream of audio data using a warm word model to monitor the occurrence of one or more specific words or phrases may be in response to activating one or more currently dormant automatic assistant functions utilizing a warm word model. In some versions of these implementations, a warm word activation event may include one or more of the following: a telephone call received at a client device, a text message received at a client device, an email received at a client device, an alarm sounded at a client device, a timer sounded at a client device, media played at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

[0115] In some implementations, determining whether a user intends to execute an assistant command associated with one or more specific words or phrases based on the processed ASR output includes: processing the ASR output using a Natural Language Understanding (NLU) model to generate NLU output; and determining whether the user intends to execute an assistant command based on the NLU output. The ASR output is generated based on the preceding portion of the audio data, rather than the following portion. In some versions of these implementations, the method may further include: in response to determining that the NLU output is insufficient to determine that the user intends to execute an assistant command associated with one or more specific words or phrases: processing the following portion of the audio data using an ASR model to generate additional ASR output; and determining whether the user intends to execute an assistant command associated with one or more specific words or phrases based on the processed additional ASR output.

[0116] In some implementations, determining whether a user intends to execute an assistant command associated with one or more specific words or phrases based on processing the ASR output may include: processing the ASR output using a Natural Language Understanding (NLU) model to generate NLU output; and determining whether the user intends to execute an assistant command based on the NLU output. The ASR output is generated based on both the preceding and following portions of the audio data. In some versions of these implementations, the method may further include: in response to determining that the NLU output is insufficient to determine that the user intends to execute an assistant command associated with one or more specific words or phrases; processing an additional following portion of the audio data using an ASR model to generate additional ASR output; and determining whether the user intends to execute an assistant command associated with one or more specific words or phrases based on processing the additional ASR output. The additional following portion of the audio data may be after the following portion of the audio data.

[0117] In some versions of these embodiments, the method may further include: processing the audio data stream using an endpoint model to generate multiple timestamps of spoken utterances captured in the audio data stream and including one or more specific words or phrases. In some versions of these embodiments, the multiple timestamps may include at least: a first timestamp associated with a first time when the user begins to provide the spoken utterance; a second timestamp associated with a second time after the first time when the user begins to provide one or more specific words or phrases included in the spoken utterance; a third timestamp associated with a third time after the second time when the user finishes providing one or more specific words or phrases included in the spoken utterance; and a fourth timestamp associated with a fourth time after the third time when the user finishes providing the spoken utterance. In some further versions of these embodiments, a pre-context portion of the audio data may include any audio data corresponding to the spoken utterances between the first and second timestamps. In other or alternative versions of these embodiments, a post-context portion of the audio data may include any audio data corresponding to the spoken utterances between the third and fourth timestamps.

[0118] In some implementations, the method may further include: in response to determining that the spoken utterance includes one or more of a specific word or phrase, activating one or more currently dormant automated assistant functions that utilize an ASR model.

[0119] In some implementations, the method may further include: combining preceding and / or following portions of audio data, using an ASR model to process portions of audio data corresponding to one or more specific words or phrases to generate ASR output.

[0120] In some implementations, the method may further include: processing the stream of audio data using a Voice Activity Detection (VAD) model to monitor the presence of additional voice activity before and / or after portions of the audio data correspond to one or more specific words or phrases; and in response to detecting the presence of additional voice activity: activating one or more currently dormant automated assistant functions utilizing the ASR model. In some versions of these implementations, the method may further include: in response to not detecting the presence of additional voice activity: causing the automated assistant to execute an assistant command associated with one or more specific words or phrases.

[0121] In some implementations, the method may further include processing audio data using an acoustic model to determine one or more prosodic features of the spoken utterance. Determining whether a user intends to execute an assistant command associated with one or more specific words or phrases may further be based on one or more prosodic features of the spoken utterance.

[0122] In some implementations, the method may further include instances of receiving sensor data from one or more sensors of the client device. Determining whether the user intends to use one or more specific words or phrases to cause the execution of an assistant command associated with one or more of those specific words or phrases may further be based on instances of sensor data. In some versions of these implementations, instances of sensor data may include one or more of the following: accelerometer data generated by the accelerometer of the client device, image data generated by one or more camera components of the client device, and motion data generated by one or more motion sensors of the client device.

[0123] In some implementations, the method may further include: obtaining a preceding portion of audio data from an audio buffer of a client device; and / or obtaining a following portion of audio data from a stream of audio data.

[0124] In some implementations, a method implemented by one or more processors is provided, comprising: processing a stream of audio data using a warm word model to monitor the occurrence of one or more specific words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, and each of the one or more specific words or phrases being associated with an assistant command; in response to determining that a portion of the audio data corresponds to one or more of the specific words or phrases, and in response to determining that the identity of a user providing spoken utterances corresponding to the audio data is verified: processing the audio data using a speaker recognition (SID) model to determine whether the audio data is sufficient to identify the user providing the spoken utterances; in response to determining that the audio data is sufficient to identify the user providing the spoken utterances, and in response to determining that the identified user providing the spoken utterances is authorized to execute an assistant command associated with one or more of the specific words or phrases: causing an automated assistant to execute an assistant command associated with one or more of the specific words or phrases; and in response to determining that the audio data is insufficient to identify the user providing the spoken utterances: generating a prompt requesting additional verification of the identity of the user providing the spoken utterances; and causing the prompt to be provided for presentation at a client device.

[0125] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features.

[0126] In some implementations, the method may further include: detecting the occurrence of a warm word activation event; and, in response to detecting the occurrence of a warm word activation event, activating one or more currently dormant automatic assistant functions utilizing a warm word model. Processing a stream of audio data using a warm word model to monitor the occurrence of one or more specific words or phrases may be in response to activating one or more currently dormant automatic assistant functions utilizing a warm word model. In some versions of these implementations, a warm word activation event may include one or more of the following: a telephone call received at a client device, a text message received at a client device, an email received at a client device, an alarm sounded at a client device, a timer sounded at a client device, media played at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

[0127] In some implementations, the method may further include: in response to determining that a portion of the audio data corresponds to one or more of a specific word or phrase, and in response to determining that the identity of the user providing the spoken utterance corresponding to the audio data is not verified: causing the automated assistant to execute an assistant command associated with one or more of the specific word or phrase.

[0128] In some implementations, processing audio data to determine whether the audio data is sufficient to verify the identity of the user providing the spoken utterance includes: using a text-related (TD) SID model as the SID model, processing portions of the audio data corresponding to one or more specific words or phrases to generate TD speaker embeddings. Determining whether the audio data is sufficient to identify the user providing the spoken utterance may include: comparing the TD speaker embeddings in the embedding space with one or more stored TD speaker embeddings locally stored at a client device; and determining whether a portion of the audio data is sufficient to verify the identity of the user providing the spoken utterance based on the comparison of the TD speaker embeddings with one or more stored TD speaker embeddings.

[0129] In some versions of these implementations, processing portions of audio data corresponding to one or more specific words or phrases using a TD SID model to generate TD speaker embeddings may include determining whether one or more of the stored TD speaker embeddings correspond to the specific word or phrase. Processing portions of audio data corresponding to one or more specific words or phrases using a TD SID model to generate TD speaker embeddings is in response to determining that one or more of the stored TD speaker embeddings correspond to the specific word or phrase.

[0130] In other or alternative versions of these implementations, generating a prompt requesting additional verification of the identity of the user providing the spoken words may include: generating a prompt requesting the user to provide additional spoken words of sufficient length to perform TI SID verification of the identity of the user providing the spoken words.

[0131] In other or alternative versions of these implementations, the method may further include: in response to determining that the length of the spoken utterance is sufficient to perform TI SID: processing the stream of audio data using a TI SID model as the SID model to generate a TI speaker embedding. Determining whether the audio data is sufficient to identify the user providing the spoken utterance may include: comparing the TI speaker embedding with one or more stored TI speaker embeddings locally stored at a client device in the embedding space; and determining whether the stream of audio data is sufficient to verify the identity of the user providing the spoken utterance based on the comparison of the TI speaker embedding with one or more stored TI speaker embeddings. In some further versions of these implementations, generating a prompt requesting additional verification of the identity of the user providing the spoken utterance may include: generating a prompt requesting additional biometric information to verify the user's identity. Requesting additional biometric information to verify the user's identity may include requesting one or more of the following: fingerprint verification or facial recognition verification.

[0132] In some embodiments, the method may further include: receiving additional user input in response to a prompt being provided for presentation at a client device; and in response to receiving the additional user input: determining whether the additional user input is sufficient to identify the user providing the spoken words. In some versions of these embodiments, the method may further include: in response to determining that the additional user input is sufficient to identify the user providing the spoken words: causing the automated assistant to execute a corresponding assistant command.

[0133] Additionally, some embodiments include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above. Some embodiments also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

Claims

1. A method implemented by one or more processors, the method comprising: The warm word model is used to process the stream of audio data to monitor the occurrence of one or more specific words or phrases, the stream of audio data being generated by one or more microphones of a user's client device, and each of the one or more specific words or phrases being associated with an assistant command; In response to determining that a portion of the audio data corresponds to one or more of the specific word or phrase: The audio data stream is processed using a Voice Activity Detection (VAD) model to detect the occurrence of additional voice activity before and / or after the portion of the audio data corresponding to one or more of the specific word or phrase. In response to the absence of detected additional voice activity, the automated assistant executes the assistant command associated with one or more of the specific word or phrase; In response to the detection of the additional voice activity, the Automatic Speech Recognition (ASR) engine is activated; The ASR engine uses an ASR model to process the preceding and / or following portions of the audio data to generate ASR output. Wherein, the preceding portion of the audio data precedes the portion of the audio data corresponding to the one or more specific words or phrases, and Wherein, the subsequent portion of the audio data follows the portion of the audio data corresponding to the one or more specific words or phrases; and Based on processing the ASR output, it is determined whether the user intends for one or more specific words or phrases to cause the assistant command to be executed; In response to determining that the user does not intend for the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: To prevent the automated assistant from executing the assistant commands associated with one or more of the specific word or phrase; and In response to determining that the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: The automated assistant executes the assistant command associated with one or more of the specific word or phrase.

2. The method according to claim 1, further comprising: Detect the occurrence of warm word activation events; as well as In response to the detection of the warm word activation event, one or more currently dormant automatic assistant functions utilizing the warm word model are activated. The use of the warm word model to process the audio data stream to monitor the occurrence of the one or more specific words or phrases is in response to activating the one or more currently dormant automatic assistant functions that utilize the warm word model.

3. The method of claim 2, wherein, The warm word activation event includes one or more of the following: a telephone call received at the client device, a text message received at the client device, an email received at the client device, an alarm sounded at the client device, a timer sounded at the client device, media played at the client device or an additional client device in the client device's environment, a notification received at the client device, the location of the client device, or a software application accessible at the client device.

4. The method of claim 1, wherein, Determining whether the user intends the one or more specific words or phrases to cause the assistant command associated with those words or phrases to be executed based on processing the ASR output includes: The ASR output is processed using a Natural Language Understanding (NLU) model to generate an NLU output, wherein the ASR output is generated based on the preceding portion of the audio data rather than the following portion of the audio data; and The NLU output is used to determine whether the user intends the one or more specific words or phrases to cause the assistant command to be executed.

5. The method of claim 4, further comprising: In response to determining that the NLU output is insufficient to determine whether the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: The ASR model is used to process the subsequent portion of the audio data to generate additional ASR output; and Based on processing the additional ASR output, determine whether the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed.

6. The method of claim 1, wherein, Determining whether the user intends the one or more specific words or phrases to cause the execution of the assistant command associated with one or more of those specific words or phrases based on processing the ASR output includes: The ASR output is processed using a Natural Language Understanding (NLU) model to generate an NLU output, wherein the ASR output is generated based on both the preceding and following portions of the audio data; and The NLU output is used to determine whether the user intends the one or more specific words or phrases to cause the assistant command to be executed.

7. The method of claim 6, further comprising: In response to determining that the NLU output is insufficient to determine whether the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed: The ASR model is used to process an additional post-sequence portion of the audio data to generate an additional ASR output, wherein the additional post-sequence portion of the audio data follows the post-sequence portion of the audio data; and Based on processing the additional ASR output, determine whether the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed.

8. The method of claim 1, further comprising: The audio data stream is processed using an endpoint model to generate multiple timestamps of spoken utterances captured in the audio data stream and including one or more specific words or phrases.

9. The method of claim 8, wherein, The plurality of timestamps includes at least: a first timestamp associated with a first time when the user begins to deliver the spoken utterance, a second timestamp associated with a second time after the first time when the user begins to deliver the one or more specific words or phrases included in the spoken utterance, a third timestamp associated with a third time after the second time when the user finishes delivering the one or more specific words or phrases included in the spoken utterance, and a fourth timestamp associated with a fourth time after the third time when the user finishes delivering the spoken utterance.

10. The method according to claim 9, wherein, The preceding portion of the audio data includes any audio data corresponding to the spoken words between the first timestamp and the second timestamp.

11. The method according to claim 10, wherein, The subsequent portion of the audio data includes any audio data corresponding to the spoken words between the third and fourth timestamps.

12. The method of claim 8, further comprising: In response to determining that the spoken utterance includes one or more of the specific words or phrases, one or more currently dormant automatic assistant functions utilizing the ASR model are activated.

13. The method of claim 1, further comprising: The ASR model is used to process portions of the audio data corresponding to one or more specific words or phrases by combining the preceding and / or following portions of the audio data to generate the ASR output.

14. The method of claim 1, further comprising: In response to the detection of the additional voice activity: Activate one or more currently dormant automatic assistant functions that utilize the ASR model.

15. The method of claim 8, further comprising: The audio data is processed using an acoustic model to determine one or more prosodic features of the spoken utterance. Determining whether the user intends the one or more specific words or phrases to cause the assistant command associated with one or more of the specific words or phrases to be executed is further based on the one or more prosodic characteristics of the spoken utterance.

16. The method of claim 1, further comprising: An example of receiving sensor data from one or more sensors of the client device. The determination of whether the user intends the one or more specific words or phrases to cause the execution of the assistant command associated with one or more of the specific words or phrases is a further example based on the sensor data.

17. The method according to claim 16, wherein, The examples of the sensor data include one or more of the following: accelerometer data generated by the accelerometer of the client device, image data generated by one or more camera components of the client device, and motion data generated by one or more motion sensors of the client device.

18. The method according to any one of claims 1 to 17, further comprising: The preceding portion of the audio data is obtained from the audio buffer of the client device; and / or The subsequent portion of the audio data is obtained from the audio data stream.

19. A system for suppressing assistant commands in a context, comprising: At least one processor; as well as A memory for storing instructions, which, when executed, cause the at least one processor to perform the method corresponding to any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause at least one processor to perform a method corresponding to any one of claims 1 to 18.

Citation Information

Patent Citations

  • Multi-level speech recognition

    CN105393302A

  • Interactive intelligent voice home control device and control method based on open source hardware

    CN110265012A

  • Preventing of audio attacks

    US20180158453A1

  • Contextual hotwords

    US20180182390A1