Dynamic and / or context-specific hotworks for invoking automatic assistants

Abstract

The invention relates to dynamic and / or context-specific hotworks for invoking an automatic assistant. Techniques are described herein that enable invocation of an automatic assistant using "dynamic" or "context-specific" hotworks. In various implementations, an automatic assistant in a default listening state may be performed at least in part on one or more computing devices of a user. When in a default listening state, default hotworks may be monitored for audio data captured by one or more microphones. A default hotword is detected, causing the automated assistant to transition to a speech recognition state. Sensor signals generated by hardware sensors integrated with a computing device may be detected and analyzed to determine attributes of a user. Based on the analysis, the automatic assistant may transition to an enhanced listening state in which enhanced hotworks may be monitored for the audio data. Detecting the enhanced hotworks triggers the automatic assistant to perform a response action without detecting the default hotworks.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880094453.1, filed on August 21, 2018. Background Art

[0003] People may participate in human-machine conversations with what is referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," "virtual assistants," etc.). For example, humans (who may be referred to as "users" when they interact with automated assistants) may provide commands, queries, and / or requests (collectively referred to herein as "queries") using free-form natural language input, where the free-form natural language input may include spoken utterances that are converted to text and then processed and / or typed free-form natural language input.

[0004] In many cases, before an automated assistant can interpret and respond to a user's request, it must first be "invoked," for example, using a predefined verbal invocation phrase, often referred to as a "hot word" or "wake-up word." Therefore, many automated assistants operate in what is referred to herein as a "limited hot word listening state" or "default listening state," in which they always "listen" to audio data sampled by a microphone for a limited (or restricted, or "default") set of hot words. Any utterances captured in the audio data other than the default set of hot words will be ignored. Once the automated assistant has been invoked with one or more hot words in the default set of hot words, it can operate in what is referred to herein as a "speech recognition state," in which, for at least some time intervals after the invocation, the automated assistant performs speech-to-text ("STT") processing on the audio data sampled by the microphone to generate text input, which is then semantically processed to determine the user's intent (and to implement that intent).

[0005] Operating an automated assistant in a default listening state provides a variety of benefits. Limiting the number of hot words that are "listened to" can save power and / or computing resources. For example, an on-device machine learning model can be trained to generate an output that indicates when one or more hot words are detected. Implementing such a model may require only minimal computing resources and / or power, which is particularly useful for assistant devices that are often resource-constrained. Storing such trained models locally on the client device also provides advantages related to privacy. For example, most users do not want STT processing to be performed on everything they say within the hearing capabilities of the computing device running the automated assistant. Additionally, the on-device model also prevents data indicative of user utterances that are not intended to be processed by the automated assistant from being provided to a semantic processor that typically runs at least partially on the cloud.

[0006] In addition to these advantages, operating an automated assistant in a limited hotword listening state also presents various challenges. In order to avoid inadvertently invoking an automated assistant, hotwords are typically selected as words or phrases that are not often spoken in daily conversations (e.g., "long-tail" words or phrases). However, there are various situations in which it may be troublesome to require a user to say a long-tail hotword before invoking the automated assistant to perform certain operations. Some automated assistants may provide an option for a "continuous listening" mode after the user speaks a command, so that the user does not need to "re-awaken" the automated assistant with a hotword before executing subsequent commands. However, switching the automated assistant to a continuous listening mode means that the automated assistant may be performing much more STT processing for much more utterances, which may waste power and / or computing resources. In addition, and as described above, most users prefer that only utterances addressed to the automated assistant be processed by STT. Summary of the invention

[0007] Techniques for enabling an automated assistant to use "dynamic" hot words are described herein. In various cases, an automated assistant configured with selected aspects of the present disclosure may listen more intelligently to context-specific hot words, sometimes embodied in a set of hot words referred to herein as "enhanced." In various embodiments, the automated assistant may listen to these context-specific hot words in addition to or in lieu of the default hot words used to invoke the automated assistant. In other words, in various embodiments, an automated assistant configured with selected aspects of the present disclosure may, in certain circumstances, at least temporarily, expand or change its vocabulary of hot words.

[0008] In various embodiments, dynamic hotwords may be selectively enabled in various circumstances, such as when the user is sufficiently close to the assistant device, or when the user is identified based on the user's appearance, voice, and / or other identifying characteristics. These other identifying characteristics may include, for example, an RFID badge, a visual marker, a uniform, a piece of flair, the user's stature (which may indicate that the user is an adult), physical impairments that may affect the user's ability to communicate verbally and / or via conventional user input (e.g., a mouse, a touch screen, etc.), and the like.

[0009] As an example, in some embodiments, if a user is detected via a proximity sensor, e.g., within a predetermined proximity of a computing device that can be used to engage the automated assistant, one or more additional hot words can be activated to enable the nearby user to more easily invoke the automated assistant. In some embodiments, the closer the user is to the assistant device, the more dynamic hot words can be activated. For example, a user detected within three meters of the assistant device can activate a first set of dynamic hot words. If a user is detected at a closer proximity, e.g., within one meter, additional or alternative hot words can be activated.

[0010] As another example, in some implementations, one or more dynamic hot words associated with (e.g., customized to) a user, such as a hot word manually selected or entered by the user, can be downloaded and / or "activated." The user can then speak one or more of these custom hot words to invoke the automated assistant without having to speak the default set of hot words that people typically use to invoke the automated assistant.

[0011] In some embodiments, dynamic hot words can be activated based on some combination of proximity and recognition. This may be particularly beneficial in noisy and / or crowded environments. For example, in such an environment, any number of other people may also be trying to interact with the automated assistant using default hot words that may be relatively standardized. In order to avoid accidental activation of the automated assistant by others, when the user is close enough to the automated assistant (e.g., holding her smartphone or wearing her smartwatch, for example, an engaged buckle or fastener may be detected and the activation of the enhanced hot word set may be triggered) and recognized, the automated assistant may stop listening to the standard or default hot words, and may instead listen to custom hot words tailored to a specific user. Alternatively, in an alternative solution, the automated assistant may continue to listen to the default hot words, but may raise the confidence threshold required for calls based on these default hot words, while activating or lowering the threshold associated with the custom hot words.

[0012] As another example, in some embodiments, a dynamic hot word can be associated with a group of people (e.g., employees, gender, age range, shared visual characteristics, etc.), and can be activated if one or more of these people are detected and / or identified. For example, a group of people can share visual characteristics. In some embodiments, when one or more hardware sensors of the assistant device detect one or more of these visual characteristics, one or more dynamic hot words can be activated, which are otherwise unavailable to non-group members.

[0013] Hot words may be detected by or on behalf of the automated assistant in a variety of ways. In some implementations, a machine learning model (such as a neural network) may be trained to detect ordered or unordered sequences of one or more hot words in an audio data stream. In some such implementations, a separate machine learning model may be trained for each applicable hot word (or "hot phrase" containing multiple hot words).

[0014] In some embodiments, machine learning models trained to detect these dynamic hot words can be downloaded as needed. For example, suppose a particular user is identified based on, for example, facial recognition processing performed on one or more images of the user captured or voice recognition processing performed on audio data generated from the user's voice. If these models are not yet available on the device, they can be downloaded from the cloud based on the user's online profile.

[0015] In some implementations, to improve the user experience and reduce latency, the response actions intended to be performed by the automated assistant when a context-specific hotword is detected can be pre-cached on the device where the user engages with the automated assistant. This way, once a context-specific hotword is detected, the automated assistant can take immediate action. This is in contrast to a situation where the automated assistant may first perform a round-trip network communication with one or more computing systems (e.g., the cloud) to satisfy the user's request.

[0016] The technology described herein results in a variety of technical advantages. Expanding or changing the vocabulary that can be used to invoke an automated assistant in certain situations (at least temporarily) can reduce the need to first invoke the automated assistant before causing the automated assistant to perform some context-dependent action (e.g., stop a timer, pause music, etc.). Some existing assistant devices make it easy to pause media playback or stop a timer by allowing the user to simply tap an active portion of the device surface (e.g., a capacitive touchpad or display) without having to first invoke the automated assistant. However, users with physical disabilities and / or users who are occupied with other matters (e.g., cooking, driving, etc.) may not be able to easily touch the device. Therefore, the technology described herein enables those users to more easily and quickly cause the automated assistant to perform some responsive actions, such as stopping a timer, without having to first invoke it.

[0017] In addition, as described herein, in some embodiments, the automated assistant can proactively download content in response to context-specific hot words. For example, assume that the user tends to repeat specific requests, such as "what's the weather forecast today?", "what's on my schedule?", etc. Information in response to these requests can be preemptively downloaded and cached in memory, and these commands (or portions thereof) can be used to activate specific dynamic hot words. Therefore, when the one or more hot words are spoken, the automated assistant is able to provide response information more quickly than when the automated assistant must first perform a contact with one or more remote resources through one or more networks to obtain response information. This may also be beneficial when the assistant device is in a vehicle that may be driving in / out of an area where a data network is available. By preemptively downloading and caching content in response to certain context-specific hot words (e.g., when the vehicle is in a data coverage area), the data is available if the user requests the data when traveling outside the data coverage area.

[0018] As yet another example, the techniques described herein may enable a user to trigger a response action without comprehensive speech-to-text ("STT") processing. For example, when certain context-calling models are activated and context-specific hot words are detected, a response action may be triggered based on the output of these models without converting the user's utterance to text using STT processing. This may save computing resources on the client device, and / or may avoid round-trip communications with a cloud infrastructure for STT and / or semantic processing of the user's utterance, which saves network resources. Moreover, avoiding round-trip transmissions may improve latency and avoid sending at least some data to a cloud infrastructure, which may be advantageous and / or desirable from a user privacy perspective.

[0019] In some embodiments, a method performed by one or more processors is provided, the method comprising: executing an automated assistant in a default listening state, wherein the automated assistant is executed at least in part on one or more computing devices operated by a user; while in the default listening state, monitoring audio data captured by one or more microphones for one or more hot words in a default set of one or more hot words, wherein detection of the one or more hot words in the default set will trigger the automated assistant to transition from the default listening state to a speech recognition state; detecting one or more sensor signals generated by one or more hardware sensors integrated with the one or more computing devices; analyzing the one or more sensor signals to determine attributes of the user; based on the above analysis, transitioning the automated assistant from the default listening state to an enhanced listening state; and while in the enhanced listening state, monitoring audio data captured by the one or more microphones for one or more hot words in an enhanced set of one or more hot words, wherein detection of the one or more hot words in the enhanced set will trigger the automated assistant to perform a response action without detecting the one or more hot words in the default set, and wherein the one or more hot words in the enhanced set are not in the default set.

[0020] These and other implementations of the technology disclosed herein may optionally include one or more of the following features.

[0021] In various implementations, the one or more hardware sensors may include a proximity sensor, and the attribute of the user may include: detecting the user via the proximity sensor, or detecting the user via the proximity sensor within a predetermined distance of the one or more computing devices.

[0022] In various embodiments, the one or more hardware sensors may include a camera. In various embodiments, the attributes of the user may include: detecting the user by the camera, or detecting the user by the camera within a predetermined distance of one or more computing devices. In various embodiments, the analysis may include facial recognition processing. In various embodiments, the attributes of the user may include the identity of the user. In various embodiments, the attributes of the user may include the user's membership in a group.

[0023] In various embodiments, the one or more hardware sensors may include one or more microphones. In various embodiments, the one or more attributes may include audibly detecting a user based on audio data generated by the one or more microphones. In various embodiments, the analysis may include speech recognition processing. In various embodiments, the attributes of the user may include the identity of the user. In various embodiments, the attributes of the user may include the user's membership in a group.

[0024] In various implementations, the responsive action may include transitioning the automated assistant to a speech recognition state. In various implementations, the responsive action may include the automated assistant performing a task requested by the user using one or more hot words in the enhanced set.

[0025] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media, which store computer instructions that can be executed by one or more processors to perform any of the foregoing methods.

[0026] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the beginning of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0028] Figure 2 Depicted are examples of state machines that may be implemented by an automated assistant configured with selected aspects of the present disclosure, according to various implementations.

[0029] Figure 3A , Figure 3B , Figure 4A , Figure 4B and Figure 5 Additional example situations are depicted in which the disclosed techniques may be employed.

[0030] Figure 6 A flow chart illustrating an example method according to implementations disclosed herein is depicted.

[0031] Figure 7 A flow chart illustrating an example method according to implementations disclosed herein is depicted.

[0032] Figure 8 An example architecture for a computing device is shown. DETAILED DESCRIPTION

[0033] Now go to Figure 1, showing an example environment in which the techniques disclosed herein may be implemented. The example environment includes one or more client computing devices 106. Each client device 106 may execute a respective instance of an automated assistant client 108, which may also be referred to herein as the "client portion" of the automated assistant. One or more cloud-based automated assistant components 119 (which may also be referred to herein as the "server portion" of the automated assistant) may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems), wherein the one or more computing systems are communicatively coupled to the client devices 106 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 114.

[0034] In various embodiments, an instance of automated assistant client 108, through its interaction with one or more cloud-based automated assistant components 119, can form what appears to be a logical instance of automated assistant 120 from the user's perspective, and the user can participate in a human-computer dialogue with automated assistant 120. Such an instance of automated assistant 120 is Figure 1 108 is shown as a dashed line in . It should be understood, therefore, that each user who engages with an automated assistant client 108 executing on a client device 106 may actually engage with his or her own logical instance of automated assistant 120. For the sake of brevity and simplicity, the "automated assistant (term)" used herein to "serve" a particular user will refer to the combination of an automated assistant client 108 executing on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which one or more cloud-based automated assistant components 119 may be shared among multiple automated assistant clients 108). It should also be understood that in some embodiments, automated assistant 120 may respond to requests from any user, regardless of whether the user is "served" by a particular instance of the automated assistant 120.

[0035] The one or more client devices 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (which in some cases may include a visual sensor), a smart device such as a smart TV (or a standard TV equipped with a network dongle with automated assistant functionality), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. Some client devices 106, such as a stand-alone interactive speaker (or "smart speaker"), may take the form of an assistant device that is primarily designed to facilitate a conversation between a user and an automated assistant 120. Some such assistant devices may take the form of a stand-alone interactive speaker with a display, which may or may not be a touch screen.

[0036] In some embodiments, the client device 106 may be equipped with one or more visual sensors 107 having one or more fields of view, but this is not required. The one or more visual sensors 107 may take various forms, such as digital cameras, passive infrared ("PIR") sensors, stereo cameras, RGBd cameras, and the like. The one or more visual sensors 107 may be used, for example, by an image capture module 111 to capture image frames (still images or videos) of the environment in which the client device 106 is deployed. These image frames may then be analyzed, for example, by a visual cue module 1121 to detect user-provided visual cues contained in the image frames. These visual cues may include, but are not limited to, gestures, gazes at specific reference points, facial expressions, predefined movements of the user, and the like. These detected visual cues may be used for various purposes, such as invoking the automated assistant 120 and / or causing the automated assistant 120 to take various actions.

[0037] Additionally or alternatively, in some implementations, the client device 106 may include one or more proximity sensors 105. The one or more proximity sensors may take various forms, such as a passive infrared ("PIR") sensor, a radio frequency identification ("RFID"), a component that receives a signal transmitted from another nearby electronic component (e.g., a Bluetooth signal from a nearby user's client device, a high or low frequency sound transmitted from the device, etc.), etc. Additionally or alternatively, the visual sensor 107 and / or the microphone 109 may also be used as a proximity sensor, for example, by visually and / or audibly detecting the user's proximity.

[0038] As described in more detail herein, the automated assistant 120 participates in a human-computer conversation session with one or more users via user interface input and output devices of one or more client devices 106. In some embodiments, the automated assistant 120 may participate in a human-computer conversation session with a user in response to a user interface input provided by a user via one or more user interface input devices of one of the client devices 106. In some of those embodiments, the user interface input is explicitly directed to the automated assistant 120. For example, a user may verbally provide (e.g., type, speak) a predetermined invocation phrase, such as "OK, Assistant" or "Hey, Assistant", to cause the automated assistant 120 to actively begin listening to or monitoring typed text. Additionally or alternatively, in some embodiments, the automated assistant 120 may be invoked based on one or more detected visual cues, alone or in combination with a verbal invocation phrase.

[0039] In some embodiments, even when the user interface input is not explicitly directed to the automated assistant 120, the automated assistant 120 may participate in a human-computer dialogue session in response to the user interface input. For example, the automated assistant 120 may examine the content of the user interface input and participate in a dialogue session in response to the presence of certain terms in the user interface input and / or based on other prompts. In many embodiments, the automated assistant 120 may utilize speech recognition to convert the speech from the user into text, and thus respond to the text, for example, by providing search results, general information, and / or taking one or more responsive actions (e.g., playing media, starting a game, ordering food, etc.). In some embodiments, the automated assistant 120 may additionally or alternatively respond to the speech without converting the speech into text. For example, the automated assistant 120 may convert the speech input into an embedding, into an entity representation (indicating one or more entities present in the speech input), and / or other "non-text" representations, and operate on such non-text representations. Therefore, the embodiments described herein as operating based on text converted from speech input may additionally and / or alternatively directly operate on the speech input and / or other non-text representations of the speech input.

[0040] Each of the client computing device 106 and the computing device operating the cloud-based automated assistant component 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by the client computing device 106 and / or the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented as a computer program running on one or more computers in one or more locations coupled to each other over a network, for example.

[0041] As described above, in various embodiments, client computing device 106 may operate automated assistant client 108 or the “client portion” of automated assistant 120. In various embodiments, automated assistant client 108 may include voice capture module 110, image capture module 111 described above, visual prompt module 1121, and / or invocation module 113. In other embodiments, one or more aspects of voice capture module 110, image capture module 111, visual prompt module 112, and / or invocation module 113 may be implemented separately from automated assistant client 108, for example, via one or more cloud-based automated assistant components 119. For example, in Figure 1 In the example, there is also a cloud-based visual cue module 1122 that can detect visual cues in the image data.

[0042] In various embodiments, the voice capture module 110, which can be implemented using any combination of hardware and software, can interface with hardware such as a microphone 109 or other pressure sensor to capture an audio recording of the user's speech. Various types of processing can be performed on this audio recording for various purposes. In some embodiments, the image capture module 111, which can be implemented using any combination of hardware or software, can be configured to interface with the camera 107 to capture one or more image frames (e.g., digital photos) corresponding to the field of view of the visual sensor 107.

[0043] In various embodiments, the visual cue module 1121 (and / or cloud-based visual cue module 1122) may be implemented using any combination of hardware or software and may be configured to analyze one or more image frames provided by the image capture module 111 to detect one or more visual cues captured in and / or across the one or more image frames. The visual cue module 1121 may employ a variety of techniques to detect visual cues. For example, the visual cue module 1122 may use one or more artificial intelligence (or machine learning) models that are trained to generate outputs indicative of user-provided visual cues detected in the image frames.

[0044] The voice capture module 110 can be configured to capture the user's voice, for example, via the microphone 109, as described above. Additionally or alternatively, in some embodiments, the voice capture module 110 can be further configured to convert the captured audio into text and / or other representations or embeddings, for example, using speech-to-text ("STT") processing technology. Additionally or alternatively, in some embodiments, the voice capture module 110 can be configured to convert the text into computer-synthesized speech, for example, using one or more speech synthesizers. However, in some cases, because the client device 106 may be relatively limited in computing resources (e.g., processor cycles, memory, battery, etc.), the voice capture module 110 local to the client device 106 can be configured to convert a limited number of different spoken phrases (especially phrases that invoke the automated assistant 120) into text (or other forms, such as lower-dimensional embeddings). Other voice inputs can be sent to a cloud-based automated assistant component 119, which can include a cloud-based text-to-speech conversion ("TTS") module 116 and / or a cloud-based STT module 117.

[0045] In various embodiments, the invocation module 113 may be configured to determine whether to invoke the automated assistant 120, for example, based on output provided by the voice capture module 110 and / or the visual cue module 1121 (which may be combined with the image capture module 111 in a single module in some embodiments). For example, the invocation module 113 may determine whether the user's utterance meets the invocation phrase that should initiate a human-computer dialogue session with the automated assistant 120. In some embodiments, the invocation module 113 may analyze data indicating the user's utterance, such as an audio recording or a feature vector (e.g., an embedding) extracted from the audio recording, alone or in combination with one or more visual cues detected by the visual cue module 1121. In some embodiments, the threshold used by the invocation module 113 to determine whether to invoke the automated assistant 120 in response to the voiced utterance may be lowered when a particular visual cue is also detected. Therefore, even when the user provides a spoken utterance that is different from the correct invocation phrase "OK assistant" but phonetically similar to a certain extent, the utterance can still be accepted as the correct invocation phrase when detected in conjunction with visual cues (e.g., the speaker waving, the speaker looking directly at the visual sensor 107, etc.).

[0046] In some embodiments, the invocation module 113 can use one or more on-device invocation models stored, for example, in the on-device model database 114 to determine whether the utterance and / or visual cue qualifies as an invocation. Such on-device invocation models can be trained to detect variations in invocation phrases / gestures. For example, in some embodiments, an on-device invocation model (e.g., one or more neural networks) can be trained using training examples, each training example comprising an audio recording (or extracted feature vectors) of an utterance from a user and data indicating one or more image frames and / or detected visual cues captured simultaneously with the utterance.

[0047] exist Figure 1 In the embodiment, the on-device model database 114 can store one or more on-device call models 1141-114 N In some embodiments, default on-device invocation model 1141 can be trained to detect one or more default invocation phrases or hot words, such as those previously mentioned (e.g., "OK, Assistant," "Hey, Assistant," etc.), in an audio recording or other data indicative thereof. In some such embodiments, these models can be always available and can be used to transition automated assistant 120 to a general listening state, in which any audio recording captured by voice capture module 110 (at least for a period of time after the invocation) can be processed using other components of automated assistant 120 as described below (e.g., on client device 106 or through one or more cloud-based automated assistant components 119).

[0048] Additionally, in some embodiments, the on-device model database 114 may at least temporarily store one or more additional “context call models” 1142-114 N These scenarios call models 1142-114 N The contextual invocation model 1142-1144 may be used and / or made available (eg, activated) by the invocation module 113 in a particular context. N The context call model 1142-1144 may be trained to detect one or more context specific hot words, for example, in an audio recording or other data indicating the same. N It may be selectively downloaded as needed from, for example, a dynamic hotword engine 128 forming part of a cloud-based automated assistant component 119, as will be described in more detail below.

[0049] In various embodiments, when the calling module 113 uses the context to call the models 1142-114 NWhen various dynamic hot words are detected, it can convert the automated assistant 120 to the general listening state described above. Additionally or alternatively, the call module 113 can convert the automated assistant 120 to a context-specific state, wherein one or more context-specific response actions are performed with or without converting the automated assistant 120 to the general listening state. In many cases, the audio data that triggers the automated assistant 120 to transition to a context-specific state may not be transmitted to the cloud. Instead, one or more context-specific response actions can be performed entirely on the client device 106, which can reduce response time and the amount of information transmitted to the cloud, which can be beneficial from a privacy perspective.

[0050] The cloud-based TTS module 116 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some implementations, the TTS module 116 can provide the computer-generated speech output to the client device 106 for direct output, for example, using one or more speakers. In other implementations, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the speech capture module 110, which can then convert the text data into computer-generated speech that is output locally.

[0051] The cloud-based STT module 117 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the voice capture module 110 into text, which can then be provided to the intent matcher 135. In some embodiments, the cloud-based STT module 117 can convert the audio recording of the voice into one or more phonemes, and then convert the one or more phonemes into text. Additionally or alternatively, in some embodiments, the STT module 117 can employ a state decoding diagram. In some embodiments, the STT module 117 can generate multiple candidate text interpretations of the user's utterance. In some embodiments, the STT module 117 can weight or bias a particular candidate text interpretation higher than other candidate text interpretations depending on whether there are visual cues detected at the same time.

[0052] The automated assistant 120 (particularly the cloud-based automated assistant component 119) may include an intent matcher 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some embodiments, one or more modules and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. In some embodiments, to protect privacy, one or more components of the automated assistant 120 (such as the natural language processor 122, the TTS module 116, the STT module 117, etc.) may be implemented at least partially on the client device 106 (e.g., excluded from the cloud).

[0053] In some implementations, the automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the client devices 106 during a human-machine conversation session with the automated assistant 120. The automated assistant 120 may provide the responsive content (e.g., over one or more networks when separate from the user's client device) for presentation to the user as part of the conversation session. For example, the automated assistant 120 may generate responsive content in response to free-form natural language input provided via the client device 106. As used herein, free-form input is input formulated by a user and is not limited to a set of options presented to the user for selection.

[0054] As used herein, a "conversation session" may include a logically self-consistent conversation of one or more messages between a user and automated assistant 120 (and, in some cases, other human participants). Automated assistant 120 may distinguish between multiple conversation sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than the conversation between the user and the automated assistant (e.g., the user switches apps for a while, the user leaves, and then later returns to a standalone voice-activated product), locking / hibernation of client devices between sessions, and changes in client devices used to interface with one or more instances of automated assistant 120, etc.

[0055] The intent matcher 135 may be configured to determine the user's intent based on input provided by the user (e.g., vocal speech, visual cues, etc.) and / or based on other signals (e.g., sensor signals and online signals (e.g., data obtained from a Web service), etc.). In some embodiments, the intent matcher 135 may include a natural language processor 122 and the cloud-based visual cue module 1122 described above. In various embodiments, the cloud-based visual cue module 1122 may operate similarly to the visual cue module 1121, except that the cloud-based visual cue module 1122 may have more resources at its disposal. In particular, the cloud-based visual cue module 1122 may detect visual cues that may be used by the intent matcher 135 alone or in combination with other signals to determine the user's intent.

[0056] Natural language processor 122 may be configured to process natural language input generated by a user via client device 106, and may generate annotation output (e.g., in text form) for use by one or more other components of automated assistant 120. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 106. The generated annotation output includes one or more annotations of the natural language input and / or one or more (e.g., all) terms in the natural language input.

[0057] In some embodiments, the natural language processor 122 is configured to recognize and annotate various types of grammatical information in natural language input. For example, the natural language processor 122 may include a morphological module that can separate individual words into morphemes and / or annotate these morphemes, for example, with the category of morphemes. The natural language processor 122 may also include a part-of-speech tagger that is configured to annotate words with the grammatical role of the words. For example, the part-of-speech tagger can mark each word with the part of speech of the word (such as "noun", "verb", "adjective", "pronoun", etc.). In addition, for example, in some embodiments, the natural language processor 122 may additionally and / or alternatively include a dependency parser (not shown) configured to determine the syntactic relationship between words in the natural language input. For example, the dependency parser can determine which words modify other words and the subject and verb of the sentence, etc. (e.g., parse tree), and can annotate such dependencies.

[0058] In some embodiments, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, and locations (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some embodiments, a knowledge graph may include nodes representing known entities (in some cases, entity attributes) and edges connecting nodes and representing relationships between entities. For example, a "banana" node may be connected (e.g., as a child) to a "fruit" node, which may in turn be connected (e.g., as a child) to a "produce" and / or "food" node. As another example, a restaurant called a "Hypothetical Café" may be represented by a node that also includes attributes such as its address, the type of food served, hours, contact information, etc. In some implementations, the "Hypothetical Café" node may be connected to one or more other nodes (e.g., a "restaurant" node, a "business" node, and a node representing the city and / or state where the restaurant is located, etc.) via edges (e.g., representing a child-parent relationship).

[0059] The entity tagger of the natural language processor 122 may annotate references to entities at a higher level of granularity (e.g., to enable identification of all references to a class of entities such as persons) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve the specific entity.

[0060] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not shown) that is configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be used to resolve the term "there" in the natural language input "I liked Hypothetical Café last time we ate there" to "Hypothetical Café".

[0061] In some embodiments, one or more components of the natural language processor 122 can rely on annotations from one or more other components of the natural language processor 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver to annotate all mentions of a particular entity. Similarly, for example, in some embodiments, a coreference resolver may rely on annotations from a dependency resolver to cluster references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the natural language processor 122 can determine one or more annotations using relevant previous inputs and / or other relevant data outside of the particular natural language input.

[0062] The intent matcher 135 can use various techniques to determine the user's intent, for example, based on the output from the natural language processor 122 (which can include annotations and words of the natural language input) and / or based on the output from the visual prompt module (e.g., 1121 and / or 1122). In some embodiments, the intent matcher 135 can access one or more databases (not shown) that include, for example, multiple mappings between grammars, visual prompts, and response actions (or more generally, intents). In many cases, these grammars may be selected and / or learned over time and may represent the most common intents of the user. For example, a grammar "play <artist>(play <artist>)" can be mapped to an intent that invokes the following response action, which causes the <artist>The music of (<artist>) is played on the client device 106 operated by the user. Another syntax "[weather|forecast] today" can match the user query, such as "what's the weather today" and "what's the forecast for today?"

[0063] In addition to or in lieu of grammars, in some embodiments, intent matcher 135 may employ one or more trained machine learning models, either alone or in combination with one or more grammars and / or visual cues. These trained machine learning models may also be stored in one or more databases and may be trained to recognize intent by, for example, embedding data indicative of user utterances and / or any detected user-provided visual cues into a reduced dimensionality space and then determining which other embeddings (and therefore, intents) are closest, for example using techniques such as Euclidean distance, cosine similarity, etc.

[0064] As in "play <artist>As seen in the example grammars, some grammars have slots that can be filled with slot values ​​(or "parameters") (e.g. <artist>). Slot values ​​can be determined in a variety of ways. Users typically provide slot values ​​on their own initiative. For example, for the syntax "Order me a <topping>pizza (order for me <topping>pizza), the user might say "order me a sausage pizza", in which case the autofill slot <topping>Additionally or alternatively, if the user invokes a syntax that includes slots to be filled with slot values, without the user actively providing the slot values, automated assistant 120 can request those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?"). In some implementations, automated assistant 120 can request the slot values ​​based on visual prompt module 112. 1-2 The detected visual cue fills the slot with the slot value. For example, the user can say something like "Order me this many cat bowls" while holding up three fingers toward the visual sensor 107 of the client device 106. Or, the user can say something like "Find me more movies like this" while holding up a DVD box of a particular movie.

[0065] In some embodiments, the automated assistant 120 can act as an intermediary between a user and one or more third-party computing services 130 (or "third party agent" or "agents"). These third-party computing services 130 can be independent software processes that receive input and provide responsive output. Some third-party computing services can take the form of third-party applications that may or may not be running on a computing system separate from the computing system on which the cloud-based automated assistant component 119 is running. One user intent that the intent matcher 135 can recognize is to engage with a third-party computing service 130. For example, the automated assistant 120 can provide access to an application programming interface ("API") to a service for controlling smart devices. A user can invoke the automated assistant 120 and provide a command such as "I'd like to turn the heating on." The intent matcher 135 can map this command to a grammar that triggers the automated assistant 120 to engage with the third-party service to turn on the user's heating. Third-party service 130 may provide automated assistant 120 with a list of minimum slots that need to be filled in order to implement (or "resolve") a command to turn on the heat. In this example, the slots may include the temperature to which the heat is to be set and the duration for which the heat is to be turned on. Automated assistant 120 may generate and provide to the user (via client device 106) natural language output requesting the parameters of the slots.

[0066] The fulfillment module 124 can be configured to receive the predicted / estimated intent and associated slot value (whether actively provided by the user or requested from the user) output by the intent matcher 135 and fulfill (or "resolve") the intent. In various embodiments, the fulfillment (or "resolution") of the user's intent can, for example, generate / obtain various fulfillment information (also referred to as "response" information or "resolution information") through the fulfillment module 124. As described below, in some embodiments, the fulfillment information can be provided to a natural language generator (shown as "NLG" in some figures) 126, which can generate a natural language output based on the fulfillment information.

[0067] Fulfillment (or "resolution") information can take a variety of forms, as intent can be fulfilled (or "resolved") in a variety of ways. Assuming a user asks for pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" the user's intent can be determined as a search query, for example, by intent matcher 135. The intent and content of the search query can be provided to fulfillment module 124, such as Figure 1 As shown in , the fulfillment module can communicate with one or more search modules 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) to obtain response information. The fulfillment module 124 can provide data indicating the search query (e.g., the text of the query, a reduced dimension embedding, etc.) to the search module 150. The search module 150 can provide: response information, such as GPS coordinates; or other more specific information, such as "Timberline Lodge, Mt. Hood, Oregon." The response information can form part of the fulfillment information generated by the fulfillment module 124.

[0068] Additionally or alternatively, fulfillment module 124 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from intent matcher 135, and trigger a response action. The response action may include, for example, ordering goods / services, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such embodiments, the fulfillment information may include slot values ​​associated with the fulfillment, a confirmation response (which in some cases may be selected from predetermined responses), etc.

[0069] The natural language generator 126 can be configured to generate and / or select natural language output (e.g., words / phrases designed to mimic human speech) based on data obtained from various sources. In some embodiments, the natural language generator 126 can be configured to receive as input fulfillment information associated with the fulfillment of an intent and generate natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 126 can receive information (e.g., required slots) from other sources such as third-party applications, which can be used to compose natural language output for a user.

[0070] Figure 2 Schematically depicts an example state machine according to various embodiments that may be implemented by an automated assistant (e.g., 120) and / or an assistant device (e.g., 106) configured with selected aspects of the present disclosure. At the upper left is a "default inactive state" in which the automated assistant 120 may reside when not being engaged by a user. In the default inactive state, one or more microphones of one or more client devices (106) may be activated and audio data captured thereby may be analyzed using the techniques described herein. In response to detecting one or more default call words (e.g., by the call module 113 and / or the visual cue module 112 based on the default call model 1141, for example Figure 2 If an utterance other than the default hotword is heard (e.g., “DIW” in the voice prompt, also referred to herein as a “hotword”) (e.g., “OK, Assistant” or “Hey, Assistant”), automated assistant 120 may transition to a “General Listening State”. Utterances other than the default hotword (e.g., ambient dialogue, etc.) may be ignored and not processed.

[0071] In the general listening state, the automated assistant 120 can capture the audio data spoken after the default call word and transition to the "General Processing" state. In the general processing state, the automated assistant 120 can process the data indicating the audio input, as previously described with reference to Figure 1 The above includes STT processing, natural language processing, intent matching, fulfillment, etc. Once the processing is completed, the automated assistant 120 may transition back to the default inactive state. If no audio input is received after the default call word is detected, the timeout ( Figure 2 ) can transition the automated assistant 120 from the general listening state back to a default inactive state, e.g., so that subsequent utterances not intended for processing by the automated assistant are not captured or processed.

[0072] As previously mentioned, the techniques described herein facilitate context-specific hotwords that can be activated and detected to transition the automated assistant 120 to various different states, such as a general listening state, or other context-specific states in which the automated assistant 120 performs various actions. In some implementations, the vocabulary of invocation words that can be spoken to transition the automated assistant 120 from a default inactive state to a general listening state can be expanded, at least temporarily (e.g., for a limited amount of time, until the context is no longer applicable, etc.), in certain circumstances.

[0073] For example, in Figure 2 In some embodiments, the first context-specific signal CS1 can transition the automated assistant 120 from the default inactive state to the first context-specific listening state "CSLS1". In CSLS1, the automated assistant 120 can listen for a default invocation word ("DIW") and a first context-specific hotword ("C1 hotword"). If either is detected, the automated assistant 120 can transition to the general listening state as described above. Thus, in the first context-specific listening state, the vocabulary of hotwords that transition the automated assistant 120 to the general listening state is expanded to include the default invocation word and the first context-specific hotword. Moreover, in some embodiments, if a sufficient amount of time has passed without an activated hotword being detected while the automated assistant 120 is in the first context-specific listening state, a timeout ("TO") can transition the automated assistant 120 back to the default inactive state.

[0074] Additionally or alternatively, in some embodiments, in certain contexts, automated assistant 120 may be able to transition to a general listening state, for example, using an expanded vocabulary of hot words, or to a context-specific state in which one or more context-specific actions may be performed. Figure 2 In response to a second context signal ("CS2"), automated assistant 120 may transition from a default inactive state to a second context-specific listening state "CSLS2". In this second context-specific listening state, automated assistant 120 may, for example, detect one or more default call words and / or, in some cases, detect one or more second context-specific hotwords ("C2hotword(s)"). A (C2 Hot Words A )") to transition to a general listening state, wherein the one or more second context-specific hot words effectively expand the vocabulary capable of transitioning the automated assistant 120 to the general listening state.

[0075] Additionally or alternatively, automated assistant 120 may transition from the second context-specific state ("CSLS2") to one or more states in which, for example, automated assistant 120 responds to one or more additional second context-specific hotwords ("C2hotword(s)"). B (C2 Hot Words B )") to perform one or more second context-specific response actions ("2 nd (second context-specific resp. act.)”). Example response actions are described below. In some embodiments, a specific second context-specific hotword can be mapped to a specific second context-specific response action, but this is not required. Although not described in detail for clarity Figure 2 , but in some implementations, after performing these one or more second context-specific response actions, automated assistant 120 can transition back to a default, inactive state.

[0076] In some implementations, in certain contexts, automated assistant 120 may no longer listen for default hotwords. Instead, automated assistant 120 may only listen for context-specific hotwords and perform responsive actions. For example, in Figure 2 In response to the Mth context signal ("CS M ”), the automated assistant 120 can switch from the default inactive state to the Mth context-specific listening state (M is a positive integer) “CSLSM”. In this state, the automated assistant 120 can listen to the Mth context-specific hotword (“C M In response to detecting one or more Mth context-specific hot words, automated assistant 120 can perform one or more Mth context-specific response actions ("Mth context-specific resp. act.").

[0077] In various implementations, automated assistant 120 can activate context-specific hotwords in various ways. For example, and with reference to Figure 1 and Figure 2 In some embodiments, when switching to a specific context, the automated assistant 120 may download one or more context-specific machine learning models or classifiers (e.g., 1142, 1143, ..., 1144) from the dynamic hotword engine 128, for example. N ), such as neural networks, hidden Markov models, etc., which are pre-trained to detect hot words to be activated in this specific context. For example, assume that in a specific context, the vocabulary that converts the automatic assistant 120 from a default inactive state to a general listening state is expanded to include the word "howdy". In various embodiments, the automatic assistant 120 can obtain a classifier, for example from a database 129 available to the dynamic hot word engine 128, which is trained to generate an output indicating whether the word "howdy" is detected. In various embodiments, the classifier can be binary (for example, if a hot word is detected, output "1", otherwise, output "0"), or a probability can be generated. If the probability meets a certain confidence threshold, the hot word can be detected.

[0078] Additionally or alternatively, in some embodiments, one or more on-device models 114 may take the form of a dynamic hot word classifier / machine learning model (e.g., a neural network, a hidden Markov model, etc.) that can be dynamically adjusted to generate an output for one or more predetermined phonemes and another output for other phonemes. Suppose the hot word "howdy" is to be activated. In various embodiments, the dynamic hot word classifier can be adjusted, for example, by changing one or more parameters and / or providing dynamic input with or embedded in the audio data to "listen" for the phonemes "how" and "dee". When those phonemes are detected in the audio input, the dynamic hot word classifier can generate an output that triggers the automated assistant 120 to take a response action (e.g., switching to a general listening state, performing certain context-specific response actions, etc.). Other phonemes may generate outputs that are ignored or omitted. Additionally or alternatively, the dynamic hot word classifier may generate outputs only in response to activated phonemes, while other phonemes may not generate outputs at all.

[0079] Figure 3A and Figure 3B 1 illustrates how a user 101 can communicate with an automated assistant instance ( Figure 3A An example of a human-machine dialog session occurs between a client device 306 (not shown in FIG. B) via a microphone and speaker (depicted as a separate interactive speaker, but this is not meant to be limiting) of a client computing device 306. One or more aspects of automated assistant 120 may be implemented on computing device 306 and / or on one or more computing devices in network communication with computing device 306. Client device 306 includes proximity sensor 305 as described above, which may take the form of a camera, microphone, PIR sensor, wireless communication component, etc.

[0080] exist Figure 3A , client device 306 may determine that user 101 is located at a distance D1 from client device 306 based on the signal generated by proximity sensor 305. Distance D1 may be greater than a predetermined distance (e.g., a threshold). As a result, automated assistant 120 may remain in a default listening state in which it may only be invoked (at least verbally) using one or more hotwords from a default set of hotwords. Figure 3A In, for example, user 101 provides natural language input "Hey assistant, set a timer for five minutes". "Hey assistant" may include one or more default hot words that together form a valid "invocation phrase" that can be used to invoke the automated assistant 120. Therefore, the automated assistant 120 replies "OK. Timer starting...now" and starts a five-minute timer.

[0081] On the contrary, Figure 3B In the example, client device 306 can detect that user 101 is now at a distance D2 from the client device based on the signal generated by proximity sensor 305. Distance D2 can be less than a predetermined distance (or threshold). Figure 3A In other words, determining that the distance D2 of user 101 can constitute a context signal (for example, Figure 2 ), which contextual signal causes automated assistant 120 to transition to a context-specific state in which additional or alternative hot words are activated and available to transition automated assistant 120 from a default listening state to an enhanced listening state.

[0082] When in the enhanced listening state, user 101 can use one or more hot words from the enhanced hot word set to invoke automated assistant 120 in addition to or instead of the default hot word set. Figure 3B In the example above, user 101 does not need to start his utterance with "Hey Assistant". Instead, the user simply says "set a timer for five minutes". Words such as "set" or phrases such as "set a timer for" may include hot words that are now part of the enhanced set of hot words that can be used to invoke automated assistant 120. As a result, automated assistant 120 replies "OK. Timer starting...now" and starts the five minute timer.

[0083] In this example, automated assistant 120 transitions from a default listening state to a context-specific state in which additional / alternative hotwords may be used to invoke automated assistant 120. However, automated assistant 120 may transition to additional or alternative states in response to such events. For example, in some embodiments, automated assistant 120 may transition to another context-specific state when user 101 is detected to be sufficiently close to an assistant device (e.g., D2). For example, automated assistant 120 may transition to a context-specific state in which full speech recognition is not available (at least without first using a default hotword to invoke automated assistant 120), but in which other context-specific commands are available.

[0084] For example, assume that user 101 is in a kitchen preparing dinner according to an online recipe that has been accessed through automated assistant 120 operating at least in part at client device 306. Further assume that client device 306 is located in a cooking area, so that user 101 performs steps of the recipe within D2 of client device 306. In some embodiments, the fact that user 101 has been detected within D2 of client device 306 can cause automated assistant 120 to transition to a context-specific state in which at least some commands are available without first invoking automated assistant 120 using a default hotword. For example, user 101 can say "next" to proceed to the next step of a recipe, or say "back" or "repeat last step" to hear the previous step, "start a timer", "stop a timer", etc.

[0085] In some embodiments, other actions may also be available in this context. For example, in some embodiments, multiple assistant devices deployed around a household (e.g., forming a coordinated ecosystem of client devices) may be associated with specific rooms respectively. When someone is close enough to a device, the association with a specific room may cause different dynamic hot words to become active. For example, when user 101 first sets up client device 306 in the kitchen, user 101 can specify client device 306 as being located in the kitchen, for example, by selecting kitchen on a menu. This can activate a kitchen-specific state machine of automated assistant 120, which activates certain kitchen-centric hot words when the user is close enough to the client device. Therefore, when user 101 is within D2 of client device 306 in the kitchen, certain kitchen-related commands, such as "set a timer", "go to the next step", commands for operating smart kitchen appliances, etc., may become active without user 101 first having to invoke automated assistant 120 using a default hot word.

[0086] In this proximity example, user 101 must be detected within D2 of client device 306, but this is not meant to be limiting. In various embodiments, it may be sufficient for user 101 to be detected only by proximity sensor 305. For example, a PIR sensor may detect the presence of a user only when the user is relatively close.

[0087] In addition to or in lieu of proximity, in some implementations, the user's identity and / or membership in a group may be determined and used to activate one or more dynamic hotwords. Figure 4A Figure B depicts an example of this. User ( Figure 4A 101A, Figure 4B 101B in the figure) again engages with the automated assistant 120 running at least in part on the client device 406. The client device 406 includes a sensor 470 that generates a signal that can be used to determine the identity of the user. Thus, the sensor 470 can be one or more cameras configured to generate visual data, where the visual data can be analyzed using facial recognition processing, or even using an object recognition process that can recognize visual features (such as uniforms, visual markings (e.g., badges or QR codes on shirts)) to identify the user (or the user is a member of the group). Additionally or alternatively, the sensor 470 can be one or more microphones configured to perform voice recognition (sometimes referred to as "speaker recognition") on one or more utterances of the user 101. As yet another option, the sensor 470 can be a wireless receiver (e.g., Bluetooth, Wi-Fi, ZigBee, Z-wave, ultrasonic, RFID, etc.) that receives wireless signals from a device carried by the user (not depicted) and analyzes the wireless signals to determine the identity of the user.

[0088] exist Figure 4A In the example, user 101A is not identifiable. Therefore, user 101A is required to use one or more default hot words to invoke automated assistant 120. Figure 4A , user 101A says “Hey assistant, play 'We Wish You a Merry Christmas. ” In response to “Hey assistant,” automated assistant 120 is invoked. The rest of the utterance is processed using the pipeline described previously so that automated assistant 120 initiates playback of the song on client device 406.

[0089] On the contrary, Figure 4B In the example, user 101B has been identified based on the signal generated by sensor 470. The action of identifying user 101B may constitute a context signal (e.g., Figure 2 CS1 in the example above) that causes the automated assistant to transition to a context-specific state in which additional or alternative hotwords are activated and can be used to transition the automated assistant 120 from a default listening state to an enhanced listening state. Figure 4B As shown in FIG. 1 , user 101B can say something like "Play 'We Wish You a Merry Christmas" without first invoking automated assistant 120. Nevertheless, automated assistant 120 still initiates playback of the song. Other hot words that may be active in this context include, but are not limited to, "fast forward", "skip ahead", and "fast forward".<number seconds> "skip forward <number of seconds>", "stop", "pause", "rewind <time increment>(rewind <time increment>)", "turn up / down the volume", etc.

[0090] Additionally or alternatively, it may not be necessary to detect the specific identity of user 101B. In some embodiments, recognizing some visual attributes of the user may be sufficient to activate certain dynamic hot words. For example, in Figure 4B In FIG. 1 , user 101B is a doctor wearing clothing typically worn by doctors. The clothing (or a badge with a tag, RFID, etc.) can be detected and used to determine that user 101B is a member of a group (e.g., medical staff) that should activate certain hot words.

[0091] Figure 5 Describes situations where proximity and / or identity can be used. Figure 5 , user 501 (shaded in white) holding client device 506 is in an area crowded with multiple other people (shaded in gray). It is assumed that the other people also have client devices (not depicted) that can be used to engage with automated assistant 120. If other people nearby speak the default hotword to invoke their automated assistants, they may inadvertently invoke automated assistant 120 on client device 506. Similarly, if user 501 speaks the default hotword to invoke automated assistant 120 on client device 506, they may inadvertently invoke automated assistants on other devices carried by other people.

[0092] Thus, in some implementations, automated assistant 120 may detect the identity of user 501 and / or that user 501 is within a predetermined proximity and activate a customized set of dynamic hotwords required to invoke automated assistant 120. For example, Figure 5 In the example, if user 501 is within a first proximity 515A of client device 506 (and in some cases is identified), a first customized set of hot words unique to user 501 (e.g., manually selected by user 501) can be activated. In this way, user 501 can speak these customized hot words without worrying about accidentally invoking the automated assistant on someone else's device. In some embodiments, different proximity ranges can cause different dynamic hot words to be activated. For example, in Figure 5 In some implementations, if user 501 is detected outside proximity range 515A but within another proximity range 515B, additional / alternative hot words may be activated. The same is true for a third proximity range 515C or any number of proximity ranges. In some implementations, if user 501 is outside proximity range 515C, only a default hot word may be activated, or no hot word may be activated.

[0093] In the alternative, in some embodiments, upon identifying the identity of user 501, automated assistant 120 may implement speaker recognition to respond only to the speech of user 501, while ignoring the speech of other users.

[0094] Although the embodiments described herein have focused on enabling the automated assistant 120 to take various actions (e.g., searching for information, controlling media playback, stopping a timer, etc.) in response to context-specific hotwords, this is not meant to be limiting. The techniques described herein can be extended to other use cases. For example, the techniques described herein can be applicable when a user wishes to fill out a form field on, for example, a search web page. In some embodiments, when there is a search bar or other similar text input element in a web page, one or more additional context-specific hotwords can be activated. For example, when a user navigates an assistant-enabled device to a web page with a search bar, the hotword "search for" can be activated, e.g., allowing the user to simply say "search for<desired topic> (search for <desired topic>)”, and then the user’s utterance after “search for” can be automatically transcribed into the search bar without the user having to first invoke the automated assistant 120.

[0095] In various embodiments, in addition to or in lieu of one or more context-specific hotwords, the transition of the computing device to a particular context may activate one or more context-specific gestures. For example, suppose a user is detected by a particular proximity of the assistant device. In some embodiments, one or more context-specific gestures may be activated. Detection of those gestures, for example by invocation module 113, may trigger a transition of automated assistant 120 to a general listening state and / or cause automated assistant 120 to perform some context-specific response action.

[0096] Figure 6 600 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as one or more components of automated assistant 120. Furthermore, although the operations of method 600 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0097] At box 602, the system can operate the automated assistant 120 at least partially on a computing device (e.g., client device 106, 306, 406, 506). For example, and as described above, in many cases, the automated assistant 120 can be implemented partially on the client device 106 and partially on the cloud (e.g., cloud-based automated assistant component 119). At box 604, the system can monitor audio data captured by a microphone (e.g., 109) for one or more default hot words. For example, the audio data (or other data indicative thereof, such as an embedding) can be used as an input to a call model 114 across one or more currently active activities to generate an output. The output can indicate that one or more default hot words were detected (box 606). At box 608, the system can remove the automated assistant 120 from a limited hot word listening state (e.g., Figure 2 The default inactive state in the Figure 2 general listening status in ).

[0098] In some implementations, in parallel with (or in series with) the operations of blocks 604 through 608, the system may monitor the status of the client device at block 610. For example, the system may monitor one or more contextual signals, such as detected user presence, detected user identity, detected user membership in a group, detected visual attributes, etc.

[0099] If, at block 612, the system detects a contextual signal, then at block 614, the system may transition the computing device to a given state. For example, the system may detect a contextual signal, such as a user being sufficiently close to the client device, the identity of the user, etc. Following the transition at block 614, at block 616, the system may monitor the audio data captured by the microphone for one or more context-specific hot words in addition to or in lieu of the one or more default hot words monitored at block 604.

[0100] As previously described, in some contexts, some hot words may transition the automated assistant 120 to a general listening state, while other hot words may cause the automated assistant 120 to perform a context-specific response action (e.g., stop a timer, pause media playback, etc.). Thus, at block 618, if the system detects a first one or more contextual hot words (e.g., a hot word intended to cause the automated assistant 120 to perform a context-specific task), then at block 620, the system may perform or cause the automated assistant 120 to perform one or more context-specific response actions. On the other hand, if the first one or more contextual hot words are not detected at block 618, but a second one or more contextual hot words are detected at block 622 (e.g., a hot word generally intended to simply invoke the automated assistant 120), then the method 600 may return to block 606, where the automated assistant 120 is in a general listening state.

[0101] In some implementations, if no context-specific action is requested, one or more timeouts may be employed to ensure that the automated assistant 120 returns to a stable or default value. For example, if the first context-specific hotword or the second context-specific hotword is not detected at box 618 and box 622, respectively, then at box 624, it may be determined whether a timeout (e.g., ten seconds, thirty seconds, one minute, etc.) has expired. If the answer at box 624 is yes, the method 600 may transition back to box 604, where the automated assistant 120 is transitioned to a default inactive state. However, if the answer at box 624 is no, then in some implementations, the method 600 may return to box 616, at which point the system may monitor for context-specific hotwords.

[0102] Figure 7 An example method 700 similar to method 600 for practicing selected aspects of the present disclosure according to various embodiments is depicted. For convenience, the operations of the flowchart are described with reference to a system that executes the flowchart. The system may include various components of various computer systems. In addition, although the operations of method 700 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0103] At block 702, the system can execute the automated assistant in a default listening state. While in the default listening state, at block 704, the system can monitor audio data captured by one or more microphones for one or more hot words from one or more default hot word sets. Detection of one or more hot words from the default set can trigger the automated assistant to transition from the default listening state to a speech recognition state (e.g., Figure 6 608).

[0104] At block 706, the system may detect one or more sensor signals generated by one or more hardware sensors (such as microphones, cameras, PIR sensors, wireless components, etc.) integrated with one or more computing devices. At block 708, the system may analyze the one or more sensor signals to determine an attribute of the user. This attribute of the user may be, for example, a presence attribute about the user itself, or the physical proximity of the user to one or more assistant devices. Non-limiting examples of attributes that may be determined at block 708 include, but are not limited to, the user being within a predetermined proximity of an assistant device, the identity of the user, the user's membership in a group, the user being detected by a proximity sensor (where distance may not be considered), visual attributes of the user (such as a uniform, badge, trinkets, etc.), and the like.

[0105] At box 710, the system can transition the automated assistant from the default listening state to the enhanced listening state based on the analysis. While in the enhanced listening state, at box 712, the system can monitor audio data captured by one or more microphones for one or more hot words in an enhanced set of one or more hot words. Detection of one or more hot words in the enhanced set can trigger the automated assistant to perform a response action without detecting one or more hot words in the default set. One or more hot words in the enhanced set may not be in the default set.

[0106] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only after receiving explicit authorization from the relevant user.

[0107] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information related to that user, providing permission or authorization as to whether information is collected and which portions of the information are collected. For example, one or more such control options may be provided to a user over a communications network. In addition, certain data may be processed in one or more ways before it is stored or used so as to remove personally identifiable information. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0108] Figure 8 810, which can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the user-controlled resource engine 134, and / or other components can include one or more components of the example computing device 810.

[0109] The computing device 810 typically includes at least one processor 814 that communicates with a plurality of peripheral devices via a bus subsystem 812. These peripheral devices may include: a storage subsystem 824, which includes, for example, a memory subsystem 825 and a file storage subsystem 826; a user interface output device 820; a user interface input device 822; and a network interface subsystem 816. The input and output devices allow a user to interact with the computing device 810. The network interface subsystem 816 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0110] User interface input devices 822 may include: a keyboard; a pointing device (such as a mouse, trackball, touchpad, or graphics tablet); a scanner; a touch screen incorporated into a display; an audio input device (such as a voice recognition system, a microphone); and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 810 or into a communication network.

[0111] User interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include: a cathode ray tube (CRT); a flat panel device (such as a liquid crystal display (LCD)); a projection device; or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computing device 810 to a user or to another machine or computing device.

[0112] The storage subsystem 824 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include an executable Figure 6 to Figure 7 Selected aspects of the method and implementation Figure 1 The logic of the various components is depicted in .

[0113] These software modules are typically executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include multiple memories, including a main random access memory (RAM) 830 for storing instructions and data during program execution and a read-only memory (ROM) 832 in which fixed instructions are stored. The file storage subsystem 826 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines that can be accessed by the processor 814.

[0114] The bus subsystem 812 provides a mechanism for the various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0115] The computing device 810 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purpose of illustrating some embodiments, Figure 8 The description of computing device 810 depicted in FIG. 8 is intended only as a specific example. Many other configurations of computing device 810 may have more Figure 8 The computing devices shown may have greater or fewer components.

[0116] Although several embodiments have been described and shown herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each such variation and / or modification is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application for which this teaching is used. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are given only by way of example, and within the scope of the attached claims and their equivalents, the embodiments may be implemented in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, the scope of the present disclosure includes any combination of two or more such features, systems, articles, materials, kits, and / or methods.< / time> < / topping> < / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. A method implemented using one or more processors, comprising: executing an automated assistant in a first listening state, wherein the automated assistant executes at least in part on one or more computing devices operated by a user; When in the first listening state: monitoring audio data captured by one or more microphones for one or more hot words in a default set of one or more hot words, wherein detection of one or more of the hot words in the default set triggers a transition of the automated assistant from the first listening state to a speech recognition state in which the audio data or a portion of the additional audio data is processed using speech-to-text (STT) processing; detecting one or more sensor signals generated by one or more hardware sensors integrated with one or more of the computing devices; analyzing the one or more sensor signals to determine attributes of the user; Based on the analysis, transitioning the automated assistant from the first listening state to a second listening state; and When in the second listening state: ceasing to monitor the audio data captured by one or more of the microphones for one or more of the hot words in the default set, and The audio data captured by one or more of the microphones is monitored for one or more hot words in a customized set of one or more hot words, replacing one or more of the hot words in the default set, wherein detection of one or more of the hot words in the customized set triggers the automated assistant to perform a response action.

2. The method according to claim 1, wherein: The one or more hardware sensors include a proximity sensor, and the attributes of the user include: detecting the user via the proximity sensor, or The user is detected by the proximity sensor to be within a predetermined distance of one or more of the computing devices.

3. The method according to claim 1, wherein: The one or more hardware sensors include a camera, and the attributes of the user include: detecting the user via the camera, or The user is detected by the camera to be within a predetermined distance of one or more of the computing devices.

4. The method according to claim 3, wherein: The analysis includes facial recognition processing.

5. The method according to claim 4, wherein: The attributes of the user include the identity of the user or the membership of the user in a group.

6. The method according to claim 1, wherein: The responsive action includes transitioning the automated assistant from the default listening state to the speech recognition state.

7. The method according to claim 1, wherein: The one or more hardware sensors include one or more of the microphones.

8. The method according to claim 7, wherein: The one or more attributes include auditorily detecting the user based on the audio data captured by one or more of the microphones.

9. The method according to claim 7, wherein: The analysis includes speech recognition processing.

10. The method according to claim 9, wherein: The attributes of the user include the identity of the user or the membership of the user in a group.

11. A method implemented using one or more processors, comprising: executing an automated assistant in a first listening state, wherein the automated assistant executes at least in part on one or more computing devices operated by a user; When in the first listening state: monitoring audio data captured by one or more microphones for one or more hot words in a default set of one or more hot words, wherein detection of one or more of the hot words in the default set triggers a transition of the automated assistant from the first listening state to a speech recognition state in which the audio data or a portion of the additional audio data is processed using speech-to-text (STT) processing; detecting one or more sensor signals generated by one or more hardware sensors integrated with one or more of the computing devices; analyzing the one or more sensor signals to determine attributes of the user; Based on the analysis, transitioning the automated assistant from the first listening state to a second listening state; and When in the second listening state: raising the confidence threshold required to invoke the automated assistant using one or more of the hot words in the default set, and The audio data captured by one or more of the microphones is monitored for one or more hot words in a customized set of one or more hot words, wherein detection of one or more of the hot words in the customized set triggers the automated assistant to perform a response action.

12. The method according to claim 11, further comprising, in the second listening state: A confidence threshold required to activate invoking the automated assistant using one or more of the hot words in the customized set.

13. The method according to claim 11, further comprising, in the second listening state: A confidence threshold required to invoke the automated assistant using one or more of the hot words in the customized set is lowered.

14. The method according to claim 11, wherein: The one or more hardware sensors include a proximity sensor, and the attributes of the user include: detecting the user via the proximity sensor, or The user is detected by the proximity sensor to be within a predetermined distance of one or more of the computing devices.

15. The method according to claim 11, wherein: The one or more hardware sensors include a camera, and the attributes of the user include: detecting the user via the camera, or The user is detected by the camera to be within a predetermined distance of one or more of the computing devices.

16. The method according to claim 15, wherein: The analysis includes facial recognition processing.

17. The method according to claim 16, wherein: The attributes of the user include the identity of the user or the membership of the user in a group.

18. The method according to claim 11, wherein: The responsive action includes transitioning the automated assistant from the default listening state to the speech recognition state.

19. A method implemented using one or more processors, comprising: executing an automated assistant in a default listening state, wherein the automated assistant executes at least in part on one or more computing devices operated by a user; monitoring audio data captured by one or more microphones for one or more hot words in a default set of one or more hot words while in the default listening state, wherein detection of one or more of the hot words in the default set triggers a transition of the automated assistant from the default listening state to a speech recognition state; detecting one or more sensor signals generated by one or more hardware sensors integrated with one or more of the computing devices; analyzing the one or more sensor signals to determine attributes of the user; Based on the analysis, transitioning the automated assistant from the default listening state to an enhanced listening state; and While in the enhanced listening state, the audio data captured by one or more of the microphones is monitored for one or more hot words in an enhanced set of one or more hot words, wherein detection of one or more of the hot words in the enhanced set triggers the automated assistant to perform a response action without performing speech-to-text (STT) processing, wherein the response action is performed without detecting one or more of the hot words in the default set, and wherein one or more of the hot words in the enhanced set are not in the default set.

Images