Selecting device to respond to device-independent user request

Abstract

Implementations relate to selecting a particular device from an ecosystem of devices to provide a response to a device-independent request of a user when a scene is occurring. A user specifies a scene, and a contextual feature is identified from one or more devices in the ecosystem to generate a scene feature indicative of the occurrence of the scene. The scene features are stored along with relevance of a device specified by a user to handle a response while the scene is occurring. When a subsequent device-independent request is received, a current contextual feature is identified and compared to a scene feature. Based on the comparison, a designated assistant device is selected to respond to a device-independent request.

Description

Background Art

[0001] A human may engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide explicit input (e.g., commands, queries, and / or requests) to the automated assistant, which may cause the automated assistant to generate and provide responsive output, control one or more Internet of Things (IoT) devices, and / or perform one or more other functionalities (e.g., assistant actions). This explicit input provided by the user may be, for example: spoken natural language input (i.e., spoken utterances), which in some cases may be converted to text (or other semantic representations) and then further processed; and / or typed natural language input.

[0002] In some cases, an automated assistant may include an automated assistant client that is executed locally by an assistant device and interacted with directly by a user, and a cloud-based counterpart that leverages the nearly unlimited resources of the cloud to help the automated assistant client respond to user input. For example, the automated assistant client may provide audio data of the user's spoken utterances (or a textual conversion thereof) to the cloud-based counterpart, and optionally provide data indicating the user's identity (e.g., credentials). The cloud-based counterpart may perform various processing on the explicit input to return the results to the automated assistant client, which may then provide corresponding output to the user. In other cases, the automated assistant may be executed exclusively locally by the assistant device and interacted with directly by the user to reduce latency.

[0003] Many users may engage automated assistants in performing daily tasks via assistant actions. For example, a user may routinely provide one or more explicit user inputs that cause an automated assistant to check the weather, check traffic along a route to work, start a vehicle, and / or other explicit user inputs that cause an automated assistant to perform other assistant actions while the user is eating breakfast. As another example, a user may routinely provide one or more explicit user inputs that cause an automated assistant to play a specific playlist, track a workout, and / or other explicit user inputs that cause an automated assistant to perform other assistant actions to prepare the user for a run. However, in some instances, multiple devices may be located near the user and may be configured to respond to the user's request. Therefore, determining which device best provides audio and / or visual output in response to a user providing a device-agnostic request may improve the user experience when interacting with multiple devices configured in an ecosystem of connected devices. Summary of the invention

[0004] Some implementations disclosed herein involve selecting a specific device from an ecosystem of devices to provide a user with a response to an automated assistant request based on the occurrence of a user-specified scenario. A user may indicate via input to a user interface that a specific device is utilized to provide a response from an automated assistant executed on multiple devices that are eligible to provide a response when a scenario is occurring. A user may specify that a scenario is occurring / occurred at a specific time, and one or more contextual features generated from sensor data from one or more sensors of a user device may be identified as being usable to generate a scene feature indicating that the corresponding scenario occurred. The scene feature may be stored together with an indication of a response request to the device. When a user submits a request to an automated assistant, the current context feature may be identified and compared to the scene feature. If the similarity between the scene feature and the current context feature meets a threshold (e.g., within a threshold Euclidean distance in an embedding space), the response of the automated assistant may be provided by the corresponding user-specified device.

[0005] As an example, a user may submit an indication of “When I am cooking, use my kitchen speaker” as a spoken utterance and / or via a text interface. In response, a “cooking” scene may be generated and associated with the “kitchen speaker”. The user may then indicate a time when the scene is occurring (i.e., the user is cooking), and contextual features may be identified from one or more devices that are part of an ecosystem of devices (e.g., “kitchen speaker”, “watch”, and “smartphone” that are part of an ecosystem of connected devices). The contextual features may include, for example, location data, temperature data, accelerometer data, and / or data from other sensors of devices that are part of an ecosystem of devices. The contextual features may be stored as contextual features along with the indication of the “kitchen speaker”. Subsequently, while the user is cooking, the user may utter the utterance “OK Assistant, set a time for 10 minutes”, and the current contextual features may be identified from one or more of the devices in the ecosystem. The current context characteristics can be compared to the scene characteristics, and if the similarity between the current context characteristics and the scene characteristics meets a threshold, the "kitchen speaker" can be used to render an audio response of "Setting a timer for 10 minutes" and / or provide an audio alert after ten minutes have passed.

[0006] In some implementations, a user may have an ecosystem of devices, each of which communicates with each other and with one or more cloud-based devices. Each device in the ecosystem of the device may execute an automated assistant client, wherein the cloud-based device performs one or more actions based on a request received by one or more of the automated assistant clients. For example, a user may have a smart phone, a smart watch, and a smart speaker, all of which are within the user's range and all of which have microphones that can receive audio data and determine based on the audio data whether the user has spoken a hot word and / or a call phrase to activate one or more of the devices. The request may be processed by one or more of the automated assistant clients and / or provided to a cloud-based automated assistant component for processing. Processing may include, for example, speech-to-text processing (STT), automatic speech recognition (ASR), natural language understanding processing (NLU), and / or other processing. One or more of the automated assistant components may include a fulfillment engine that may generate one or more actions that may be performed by one or more of the automated assistant clients and / or one or more other components based on the request.

[0007] As an example, a user may say “OK Assistant, set a timer,” and an automated assistant client executing on a smart phone may be invoked by recognizing the invocation phrase “OKAssistant” in the utterance. The automated assistant client may perform some processing of the utterance, such as STT, and provide a textual representation of the utterance as a request to a cloud-based automated assistant component for further processing. The cloud-based automated assistant component may perform additional processing, such as NLU, determine that the utterance includes a request to set a timer, and generate a response that may be provided to an automated assistant client executing, for example, on a smart speaker as part of the ecosystem, causing the smart speaker to set an alarm and / or perform one or more other actions, such as, for example, responding with “Setting a timer.” Thus, in some implementations, the device that received the request may not be the same device that performs the requested action.

[0008] In some implementations, a user may utilize a client device to indicate to an automated assistant client executing on the device that a specific device handles a response when a specified scenario occurs. For example, a user may say “OK Assistant, I want my smartwatch to respond when I’m exercising.” The scenario (i.e., “exercising”) may be stored on one or more of the devices in the ecosystem along with an indication of which device the user has specified to handle the request for subsequent calibration. Subsequently, the user may be prompted whether the scenario is occurring, such as prompting “Are you exercising right now? (Are you exercising now?)” or “What are you doing right now? (What are you doing now?)” with options for the user to select. In addition, for example, when the user is subsequently exercising, the user may say “I’m exercising right now (I’m exercising now)” and / or otherwise provide an indication of the current scenario that is occurring. In some implementations, this may occur when the user first specifies which device to handle the request. For example, when an as-yet-unspecified scenario occurs, the smartphone may have responded to the user, and the user may respond with “From now on, I want my watch to respond when I'm doing this.”

[0009] Once the user has confirmed that the scenario is occurring (or has occurred during a previous time period, such as by indicating "I worked today between 9 and 5"), one or more context features generated from sensor data of sensors of one or more of the devices in the ecosystem may be identified to generate a context feature indicating that the scenario occurred. For example, temperature, location, accelerometer data, and / or other context features during the time period during which the scenario occurred may be identified from sensors of a smart phone, a smart watch, and a smart speaker. In some implementations, the automated assistants of the devices in the ecosystem may identify context features from the sensors of their devices and generate encodings of context features specific to the device. Thus, for example, an automated assistant client of a smart watch may identify accelerometer data and temperature data from the sensors of the smart watch and generate encodings from the data. Additionally, for example, an automated assistant client of a smart phone may identify accelerometer data and location data from the sensors of the smart phone and generate separate encodings. For example, each encoding may be a vector in an embedding space having a lower dimension than the sensor data, such that, for example, computational resources may be saved by sending the encodings rather than the entirety of the sensor data. Further, encoding can be performed on the device that collected the contextual features so that once encoded, the sensor data is only available to the device and is not sent to, for example, a cloud-based device. Because the sensor data cannot be determined from the encoding, security is increased by not sending sensor data associated with the user to devices that do not belong to the user, thereby reducing the possibility of the user's sensor data being leaked to a third party.

[0010] In some implementations, once the contextual features of each device are identified, the contextual features (or encodings of sensor data as described above) can be provided to devices in the ecosystem for storage. For example, each of the user devices can be designated as a hub for storage of features and / or a cloud-based automated assistant component can receive features from devices in the ecosystem. In some implementations, the receiving automated assistant component can store these features and the association of the scene with the device to handle the response when the scene occurs. In some implementations, the receiving automated assistant component can combine, average, and / or otherwise aggregate the features before storing the contextual features to reduce storage requirements and improve security, as described above.

[0011] After receiving user input indicating a device to handle a response when the user is in the scene and further generating scene features identified from sensors when the scene occurs, the user can submit a request that does not indicate which device should respond to the request. For example, the user can first indicate "Use the kitchen speaker when I'm cooking", and when the user indicates that the current scene is "cooking" (e.g., the location data indicates that the user is in the kitchen and the stove is on), the scene feature can be generated and stored with the instruction to use the "kitchen speaker" when the user is "cooking". Subsequently, when the user indicates "OK Assistant, set a timer for 10 minutes", the current scene features can be identified from one or more devices in the ecosystem and processed in the same manner as previously described with respect to the scene features (e.g., the features are encoded and sent to the automated assistant component for aggregation / average), and the resulting current scene features can be compared with the stored scene features. The request can be processed by one or more automated assistant components in the ecosystem, and based on the comparison, the current scene (i.e., "cooking") can be identified. A correlation between the scene and a specified device (i.e., "kitchen speaker") can be further identified. Response data can then be provided to the "kitchen speaker" to cause one or more actions to occur. For example, the automated assistant of the "kitchenspeaker" can set a timer locally on the device, and / or the speaker of the "kitchen speaker" can provide "setting a timer for 10 minutes" as a response. Further, when the timer has expired, an alert can be provided to the user via the "kitchen speaker."

[0012] The above description is provided as an overview of only some of the implementations disclosed herein. Those implementations and other implementations are described in more detail herein.

[0013] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0015] Figure 2 is an illustration of an example connected device ecosystem.

[0016] Figure 3 is a flow chart illustrating an example method for associating a particular assistant device with a context based on user interface input.

[0017] Figure 4A is a flow chart illustrating an example method for generating scene features.

[0018] Figure 4B is a diagram of an embedding space indicating embedded context features and generated scene features.

[0019] Figure 5 is a flow chart illustrating a method for selecting a specific assistant device to respond to a device-independent request based on current context characteristics.

[0020] Fig. 6A , Figure 6B , Figure 6C and Fig.6D Each shows an embedding space with the scene features and current context features shown.

[0021] Fig. 7A and Figure 7B is a flow chart illustrating an example method according to various implementations disclosed herein.

[0022] Fig. 8A and Figure 8B is a flow chart illustrating an example method according to various implementations disclosed herein.

[0023] Fig. 9A and Fig. 9B is a flow chart illustrating an example method according to various implementations disclosed herein.

[0024] Fig.10 An example architecture of a computing device is shown. DETAILED DESCRIPTION

[0025] Now turn to Figure 1 , showing an example environment in which the technology disclosed herein can be implemented. The example environment includes multiple assistant input devices 106 1-N (also referred to herein as “assistant input device 106 ”), one or more cloud-based automated assistant components 119 , and / or one or more assistant non-input devices 185 1-N (Also referred to herein as "assistant non-input device 185"). Figure 1 The assistant input device 106 and the assistant non-input device 185 may also be collectively referred to as "assistant devices" in this article.

[0026] One or more (eg, all) of the assistant input devices 106 may execute a corresponding automated assistant client 118 1-N However, in some implementations, one or more of the assistant input devices 106 may optionally lack a corresponding automated assistant client 118. 1-N , and still includes engines and hardware components (e.g., microphones, speakers, speech recognition engines, natural language processing engines, speech synthesis engines, etc.) for receiving and processing user input to the automated assistant. 1-N An example of an automated assistant client 118 may be an application that is separate from (e.g., installed “on top of”) the operating system of the corresponding assistant input device 106—or may alternatively be implemented directly by the operating system of the corresponding assistant input device 106. As further described below, the automated assistant client 118 1-N Each instance of may optionally interact with one or more cloud-based automated assistant components 119 in response to a corresponding user interface component 107 of any of the corresponding assistant input devices 106. 1-N Various requests provided. Further, and also as described below, other engines of the assistant input device 106 may optionally interact with one or more of the cloud-based automated assistant components 119.

[0027] One or more cloud-based automated assistant components 119 may be implemented on one or more computing systems (e.g., servers collectively referred to as “cloud” or “remote” computing systems) that are communicatively coupled to respective assistant input devices 106 via one or more local area networks (“LANs”, including Wi-Fi LANs, Bluetooth networks, near field communication networks, mesh networks, etc.), wide area networks (“WANs”, including the Internet, etc.), and / or other networks. The communicative coupling of the cloud-based automated assistant components 119 to the assistant input devices 106 is generally accomplished by Figure 1 In some implementations, the assistant input devices 106 may be communicatively coupled to each other via one or more networks (e.g., LAN and / or WAN), which is generally referred to as Figure 1 1102 instructions.

[0028] One or more cloud-based automated assistant components 119 may also be communicatively coupled to the assistant non-input device 185 via one or more networks (e.g., LAN, WAN, and / or other networks). The communicative coupling of the cloud-based automated assistant component 119 to the assistant non-input system 180 is generally determined by Figure 11103 indicated. In some implementations, one or more corresponding assistant non-input systems (not depicted for clarity) can be communicatively coupled to one or more (e.g., one or more groups) of assistant non-input devices 185 via one or more networks (e.g., LAN, WAN, and / or other networks). For example, a first assistant non-input system can be communicatively coupled to and receive data from a first group of one or more assistant non-input devices 185, a second assistant non-input system can be communicatively coupled to and receive data from a second group of one or more assistant non-input devices 185, and so on. In addition, in some implementations, one or more (e.g., one or more groups or all) of the assistant non-input devices 185 can be communicatively coupled to each other via one or more networks (e.g., LAN, WAN, and / or other networks), which is generally indicated by Figure 1 1104 instructions. Figure 1 Network 110 1-4 This may also be collectively referred to herein as “network 110 ”.

[0029] An instance of automated assistant client 118, through its interaction with one or more of cloud-based automated assistant components 119, may form content that appears to be a logical instance of an automated assistant with which the user can have a human-computer conversation from the user's perspective. For example, a first automated assistant may be covered by automated assistant client 1181 of assistant input device 1061 and one or more cloud-based automated assistant components 119. A second automated assistant may be covered by assistant input device 1061. N Automated Assistant Client 118 N and one or more cloud-based automated assistant components 119. The first automated assistant and the second automated assistant may also be referred to herein simply as "automated assistants." Thus, it should be understood that a user interacting with an automated assistant client 118 executing on one or more of the assistant input devices 106 may actually be interacting with his or her own logical instance of the automated assistant (or a logical instance of the automated assistant shared among a family or other group of users and / or shared among multiple automated assistant clients 118). Although Figure 1 Only a plurality of assistant input devices 106 are shown in the figure, but it should be understood that the cloud-based automated assistant component 119 can additionally serve many additional assistant input device groups. In addition, although the various engines of the cloud-based automated assistant component 119 are described herein as being implemented separately from the automated assistant client 118 (e.g., at a server), it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, one or more (e.g., all) of the engines described with respect to the cloud-based automated assistant component 119 can be implemented locally by one or more of the assistant input devices 106.

[0030] Assistant input device 106 may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), an interactive stand-alone speaker (e.g., with or without a display), a smart appliance such as a smart TV or a smart washer / dryer, a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device), and / or any IoT device capable of receiving user input for an automated assistant. Additional and / or alternative assistant input devices may be provided. Assistant non-input device 185 may include many of the same devices as assistant input device 106, but is not capable of receiving user input for an automated assistant (e.g., does not include a user interface input component). Although assistant non-input device 185 does not receive user input for an automated assistant, assistant non-input device 185 may still be controlled by an automated assistant.

[0031] In some implementations, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other in various ways to facilitate the execution of the techniques described herein. For example, in some implementations, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other due to being connected via one or more networks (e.g., via Figure 1 The assistant input devices 106 and the assistant non-input devices 185 are communicatively coupled to each other through a network 110 of the present invention. For example, it may be the case that multiple assistant input devices 106 and assistant non-input devices 185 are deployed across a particular area or environment, such as a home, a building, and the like. Additionally or alternatively, in some implementations, multiple assistant input devices 106 and assistant non-input devices 185 may be associated with each other because they are members of a coordinated ecosystem that are selectively accessible by at least one or more users (e.g., individuals, family members, employees of an organization, other predefined groups, etc.). In some of those implementations, the ecosystem of multiple assistant input devices 106 and assistant non-input devices 185 may be manually and / or automatically associated with each other in a device topology representation of the ecosystem.

[0032] Assistant non-input devices 185 and corresponding non-input systems may include one or more first-party (1P) devices and systems and / or one or more third-party (3P) devices and systems. A 1P device or system refers to a system controlled by the same party as the party controlling the automated assistant described herein. In contrast, a 3P device or system refers to a system controlled by a different party than the party controlling the automated assistant described herein.

[0033] The assistant non-input device 185 can selectively send data (e.g., state, state change, and / or other data) to the automated assistant via the network 110 and optionally via the corresponding assistant non-input system. For example, assume that the assistant non-input device 1851 is a smart doorbell IoT device. In response to an individual pressing a button on the doorbell IoT device, the doorbell IoT device can send the corresponding data directly to the automated assistant and / or the assistant non-input system managed by the manufacturer of the doorbell, which can be a 1P system or a 3P system. The automated assistant (or assistant non-input system) can determine the change in the state of the doorbell IoT device based on such data. For example, the automated assistant (or assistant non-input system) can determine the change of the doorbell from an inactive state (e.g., no button was pressed recently) to an active state (a button was pressed recently). It is worth noting that although the user input is received at the assistant non-input device 1851 (e.g., pressing a button on the doorbell), the user input is not directed to the automated assistant (hence the term "assistant non-input device"). As another example, assume that the assistant non-input device 1851 is a smart thermostat IoT device with a microphone, but the smart thermostat does not include an automated assistant client 118. The individual can interact with the smart thermostat (e.g., using touch input or verbal input) to change the temperature, set a specific value as a set point for controlling the HVAC system via the smart thermostat, etc. However, the individual cannot communicate directly with the automated assistant via the smart thermostat unless the smart thermostat includes an automated assistant client 118.

[0034] In various implementations, one or more of the assistant input devices 106 may include one or more corresponding sensors 105 1-N (also referred to herein simply as "sensors 105"), which are configured to provide signals indicating one or more environmental conditions present in the device environment with approval from the corresponding user. In some of those implementations, the automated assistant can recognize that one or more of the assistant input devices 106 satisfy a spoken utterance from the user associated with the ecosystem. The spoken utterance can be satisfied by rendering responsive content at one or more of the assistant input devices 106 (e.g., audibly and / or visually), by causing the assistant input device 106 and / or one or more of the assistant non-input devices 185 to be controlled based on the spoken utterance, and / or by causing the assistant input device 106 and / or one or more of the assistant non-input devices 185 to perform any other action to satisfy the spoken utterance.

[0035] The corresponding sensors 105 can have various forms. Some assistant input devices 106 can be equipped with one or more digital cameras that are configured to capture and provide signals indicative of movement detected in their field of view. Additionally or alternatively, some assistant input devices 106 can be equipped with other types of light-based sensors 105, such as a passive infrared ("PIR") sensor that measures infrared ("IR") light radiated from objects within its field of view. Additionally or alternatively, some assistant input devices 106 can be equipped with sensors 105 that detect acoustic (or pressure) waves, such as one or more microphones. In addition, in addition to the assistant input device 106, one or more of the assistant non-input devices 185 can additionally or alternatively include the corresponding sensors 105 described herein, and signals from such sensors can additionally be utilized by the automated assistant to determine whether and / or how to satisfy the spoken utterance in accordance with the implementations described herein.

[0036] Additionally or alternatively, in some implementations, the sensor 105 may be configured to detect other phenomena associated with an environment that includes at least a portion of the ecosystem. For example, in some implementations, a given assistant device in the assistant devices 106, 185 may be equipped with a sensor 105 that detects various types of wireless signals (e.g., waves, such as radio, ultrasonic, electromagnetic waves, etc.) emitted, for example, by other assistant devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by a particular user and / or other assistant devices in the ecosystem. For example, some of the assistant devices 106, 185 may be configured to emit waves that are imperceptible to humans, such as ultrasonic or infrared waves, which can be detected by one or more of the assistant input devices 106 (e.g., via an ultrasonic / infrared receiver, such as a microphone with ultrasonic capabilities). In addition, for example, in some embodiments, a given assistant device among assistant devices 106, 185 can be equipped with sensors 105 to detect movement of the device (e.g., an accelerometer), temperature near the device, and / or other environmental conditions that can be detected near the device (e.g., a heart monitor that can detect the user's current heart rate).

[0037] Additionally or alternatively, various assistant devices may emit other types of waves that are imperceptible to humans, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which may be detected by other assistant devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by a particular user and used to determine the particular location of the operating user. In some implementations, GPS and / or Wi-Fi triangulation may be used to detect the location of a person, for example, based on GPS and / or Wi-Fi signals to / from the assistant device. In other implementations, other wireless signal characteristics such as flight time, signal strength, etc. may be used individually or collectively by various assistant devices to determine the location of a particular person based on signals emitted by other assistant devices carried / operated by a particular user.

[0038] Additionally or alternatively, in some implementations, one or more of the assistant input devices 106 may perform speech recognition to identify the user based on the user's speech. For example, some instances of the automated assistant may be configured to match the speech to the user's profile, for example, to provide / restrict access to various resources. Various techniques for user identification and / or authentication of automated assistants have been utilized. For example, when identifying a user, some automated assistants utilize text-dependent techniques (TD) that are constrained to a call phrase for the assistant (e.g., "OK Assistant" and / or "Hey Assistant"). Using such techniques, a registration process is performed in which the user is explicitly prompted to provide one or more instances of spoken utterances of the call phrase to which the TD features are constrained. Speaker features (e.g., speaker embeddings) of the user may then be generated by processing instances of audio data, wherein each instance in the instance captures a corresponding one of the spoken utterances in the spoken utterances. For example, speaker features may be generated by processing each instance in the instance of the audio data using a TD machine learning model to generate a corresponding speaker embedding for each utterance in the utterance. Speaker features can then be generated as a function of the speaker embeddings and stored (e.g., on a device) for use in TD techniques. For example, the speaker features can be cumulative speaker embeddings that are a function (e.g., an average) of the speaker embeddings. It has also been proposed to utilize text-independent (TI) techniques as a supplement or alternative to TD techniques. TI features are not constrained to a subset of phrases as in TD. Like TD, TI can also utilize speaker features of the user, and those features can be generated based on user utterances obtained through an enrollment process and / or other verbal interactions, although more instances of user utterances may be required to generate useful TI speaker features.

[0039] After the speaker features are generated, the speaker features can be used to identify the user who spoke the spoken utterance. For example, when the user speaks another spoken utterance, the audio data capturing the spoken utterance can be processed to generate utterance features, those utterance features are compared to the speaker features, and based on the comparison, a profile associated with the speaker features can be identified. As a specific example, the audio data can be processed using a speaker identification model to generate an utterance embedding, and the utterance embedding can be compared to a previously generated speaker embedding for the user to identify the user's profile. For example, if a distance metric between the generated utterance embedding and the user's speaker embedding satisfies a threshold, then the user can be identified as the user who spoke the spoken utterance.

[0040] Each of the assistant input devices 106 further includes a corresponding user interface component 107 1-N Each of the user interface components may include one or more user interface input devices (e.g., microphone, touch screen, keyboard, and / or other input devices) and / or one or more user interface output devices (e.g., display, speaker, projector, and / or other output devices). As an example, the user interface component 1071 of the assistant input device 1061 may only include a speaker and a microphone, while the assistant input device 1061 may only include a speaker and a microphone. N User interface component 107 N A speaker, a touch screen, and a microphone may be included. Additionally or alternatively, in some implementations, assistant non-input device 185 may include one or more user interface input devices and / or one or more user interface output devices of user interface component 107, but the user input device (if any) of assistant non-input device 185 may not allow the user to interact directly with the automated assistant.

[0041] Each of the assistant input devices 106 and / or any other computing device that operates one or more of the cloud-based automated assistant components 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by one or more of the assistant input devices 106 and / or by the automated assistant may be distributed across multiple computer systems. The automated assistant may be implemented, for example, as a computer system that communicates with the computer over a network (e.g., Figure 1 A computer program running on one or more computers located in one or more locations coupled to each other via a network 110.

[0042] As described above, in various implementations, each of the assistant input devices 106 may operate a corresponding automated assistant client 118. In various embodiments, each automated assistant client 118 may include a corresponding speech capture / text-to-speech (TTS) / speech-to-text (STT) module 114. 1-N (Also referred to herein simply as “speech capture / TTS / STT module 114”.) In other implementations, one or more aspects of a corresponding speech capture / TTS / STT module 114 may be implemented separately from a corresponding automated assistant client 118 (e.g., by one or more of cloud-based automated assistant components 119).

[0043] Each respective voice capture / TTS / STT module 114 may be configured to perform one or more functions, including, for example: (voice capture, e.g., via respective microphones (which in some cases may include one or more of sensors 105)) capturing the user's voice; using a speech recognition model stored in ML model database 192 to convert the captured audio into text and / or other representations or embeddings (STT); and / or using a speech synthesis model stored in ML model database 192 to convert text into speech (TTS). Instances of these models may be stored locally at each of the respective assistant input devices 106 and / or may be accessed by the assistant input device (e.g., via Figure 1 In some implementations, because one or more of the assistant input devices 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the corresponding voice capture / TTS / STT module 114 local to each of the assistant input devices 106 can be configured to use a speech recognition model to convert a limited number of different spoken phrases into text (or other forms, such as low-dimensional embeddings). Other voice inputs can be sent to one or more of the cloud-based automated assistant components 119, which can include a cloud-based TTS module 116 and / or a cloud-based STT module 117.

[0044] The cloud-based STT module 117 can be configured to utilize the nearly unlimited resources of the cloud to convert audio data captured by the speech capture / TTS / STT module 114 into text using the speech recognition model stored in the ML model database 192 (which can then be provided to the natural language processing (NLP) module 122). The cloud-based TTS module 116 can be configured to utilize the nearly unlimited resources of the cloud to convert text data (e.g., text formulated by an automated assistant) into computer-generated speech output using the speech synthesis model stored in the ML model database 192. In some implementations, the cloud-based TTS module 116 can provide the computer-generated speech output to one or more of the assistant devices 106, 185 for direct output, for example, using the corresponding speakers of the corresponding assistant devices. In other implementations, text data generated by the automated assistant using a cloud-based TTS module 116 (e.g., a client device notification included in a command) can be provided to a speech capture / TTS / STT module 114 of a corresponding assistant device, which can then use a speech synthesis model to convert the text data locally into computer-generated speech and cause the computer-generated speech to be rendered via a local speaker of the corresponding assistant device.

[0045] The NLP module 122 processes natural language input generated by a user via the assistant input device 106, and can generate annotated output for use by the automated assistant, the assistant input device 106, and / or one or more other components of the assistant non-input device 185. For example, the NLP module 122 can process natural language free-form input generated by a user via one or more corresponding user interface input devices of the assistant input device 106. The annotated output generated based on processing the natural language free-form input can include one or more annotations to the natural language input, and optionally one or more (e.g., all) of the terms of the natural language input.

[0046] In some implementations, the NLP module 122 is configured to recognize various types of grammatical information in natural language input and annotate. For example, the NLP module 122 may include a part of speech tagger that is configured to annotate terms by the grammatical role of the terms. In some implementations, the NLP module 122 may additionally and / or alternatively include an entity tagger (not depicted) that is configured to annotate entity references in one or more segments (such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc.). In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting nodes and representing relationships between entities.

[0047] The entity tagger of the NLP module 122 may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to a class of entities such as persons) and / or annotate references to entities at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve the specific entity.

[0048] In some implementations, the NLP module 122 may additionally and / or alternatively include a coreference resolver (not depicted) that is configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, the coreference resolver may be used to resolve the term "it" in the natural language input "lock it" as "front door lock" based on a mention of "front door lock" in a client device notification rendered immediately before receiving the natural language input "lock it".

[0049] In some implementations, one or more components of the NLP module 122 can rely on annotations from one or more other components of the NLP module 122. For example, in some implementations, the named entity tagger can rely on annotations from a coreference resolver and / or a dependency syntax parser to annotate all mentions of a particular entity. Also, for example, in some implementations, the coreference resolver can rely on annotations from a dependency syntax parser to cluster references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the NLP module 122 can determine one or more annotations using relevant data other than the particular natural language input—such as an assistant input device notification rendered immediately prior to receiving the natural language input on which the assistant input device notification is based.

[0050] refer to Figure 2 , for example purposes, an illustration of an ecosystem of connected devices is provided. In some implementations, the ecosystem is Figure 2 Additional components, fewer components, and / or different components may be included than those shown in FIG. Figures 3 to 5 and Fig. 6A For purposes of the example further described in relation to FIG. 6F , an ecosystem will be used as an example, the example including a smart phone 205, a smart watch 210, and a smart speaker 215, each of which includes one or more components of an assistant input device 106. For example, the smart phone 205 may be executing an automated assistant client 118, which further includes Figure 1 105 and / or user interface components 107). Figure 2 The ecosystem may include devices that are connected to each other and further located near the user. For example, the user may place the smart phone 105 on the kitchen counter, wear the smart watch 210, and the smart speaker may be permanently connected in the kitchen. Therefore, for explanation purposes only, the speakers of the devices 205, 210 and 215 can be heard by the user (and therefore can respond to the request) and / or the microphones of the devices 205, 210 and 215 can capture audio data of the user issuing the invocation phrase and / or request.

[0051] Further, for exemplary purposes only, the examples described herein will assume that the cloud-based assistant component 119 will perform one or more actions. However, it should be understood that one or more of the assistant input devices 106 may include Figure 119 as components of a cloud-based assistant, and may be executed by a device in an ecosystem that includes one or more of the components. For example, the smart speaker 215 may include one or more components of the cloud-based assistant component 119, and may receive and process requests in some configurations in the same manner as described herein with respect to the cloud-based automated assistant component 119. Further, one or more other components such as a scene database 194 and / or a scene feature database 193 may be accessible to one or more of the assistant input devices 106. Therefore, although some requests and / or other information are described as being provided to a cloud-based device for further processing, storage, and / or retrieval, it should be understood that the scene database 194 and / or the scene feature database 193 may be directly accessed by the assistant input device 106 and / or may be a component of the assistant input device 106 (e.g., one or more of the assistant input devices 106 may have a scene database 194 and / or may directly store entries in a scene database 194 shared by the assistant input devices 106).

[0052] In some implementations, the action processing engine 180 may receive a request from a user indicating that the user is interested in a particular device for handling a response when a scenario occurs. Figure 3 , a flowchart is shown, through which a user initiates a request indicating that the user is interested in a response provided by a specific device when a scene occurs. Request 305 can be received by one or more of the assistant input devices 106, and can be provided to, for example, one or more cloud-based automated assistant components 119 for further processing. For example, request 305 can be received by smart phone 205 and provided to an action processing engine 180 executed on another device. In response, and using one or more techniques described herein, the action processing engine 180 can generate response data 310, which indicates that an entry is stored in a scene database 194 where scenes can be stored and a corresponding device is used to respond when the scene occurs. In this example, response data 310 includes an indication that the scene is "cooking", and further includes an indication that a "kitchen speaker" (e.g., smart speaker 215) is used to respond when the identified scene is "cooking". This correlation may be stored in the scenario database 194 for later utilization in determining the current scenario and which device to utilize for a response received when the current scenario occurs, as further described herein.

[0053] In some implementations, a user can indicate that a scene is currently occurring and / or that a scene occurred during a previous time period. For example, a user can indicate that a scene is occurring by being prompted via an interface of a device that is part of the device's ecosystem (asking the user "Are you cooking right now" and / or providing the user with an option to select a current scene that is occurring) and / or by otherwise indicating a previously specified scene and a time period in which the scene occurred ("I'm cooking right now," "I was working today between 8 and 5").

[0054] refer to Figure 4A , receiving a scenario indication 405 indicating a current scenario (i.e., "Cooking") that may be provided to the action processing engine 180 by one of the devices in the ecosystem of connected devices (e.g., the user indicates via, for example, the smart phone 205 that the current scenario that is occurring is "Cooking"). In response, the action processing engine 180 may provide a request 410 to one or more of the devices 106 (e.g., a smart phone, a smart watch, and a smart speaker) to provide a contextual feature indicating the value of one or more sensors of the corresponding device. In response, the smart phone may provide a phone feature 415A, the watch may provide a watch feature 415B, and the smart speaker may provide a speaker feature 415C. Each device may provide a feature that reflects the type of sensor with which the corresponding device is equipped. For example, a smart phone may provide a phone feature 415A that includes location and accelerometer data. Further, a smart watch may provide a watch feature 415B that includes heart rate monitor data. Further, the smart speaker may provide a speaker feature 415C that includes audio data information (e.g., the amplitude of the captured audio data) and / or other information (e.g., feature data indicating the presence of a user near the speaker and / or indicating the presence of other devices near the speaker).

[0055] In some implementations, one or more of the devices may perform encoding of the sensor data before providing the features to the feature aggregation engine 130 for further processing. Figure 1, each of automated assistant clients 118 may include an encoder 120 that takes as input data from one or more sensors of a corresponding device and provides as output an encoded version of the sensor data. The encoded version of the sensor data may include, for example, a vector embedded in an embedding space that may reduce the dimensionality of the sensor data (e.g., reducing multiple sensor readings from multiple sensors to a single vector), as well as improve security by not providing the sensor data directly (e.g., encoding such that the resulting vector represents multiple sensor data values ​​without providing the sensor data directly). Thus, in Figure 4A , phone feature 415A, watch feature 415B, and speaker feature 415C can be sensor data captured by sensors of each respective device, or can be vectors (or other representations) indicative of the sensor data such that, for example, the sensor data is not available to components that are not directly in the user's environment.

[0056] In some implementations, an initial scene (or set of scenes) may be determined based on multiple users having specified scenes and associated contextual features. For example, before a user explicitly indicates a scene, one or more default scenes may be identified using default contextual features that may be utilized to suggest possible scenes to the user and / or otherwise identify when a scene is occurring but the user has not yet specified the scene. Thus, one or more candidate scenes may be identified, the user may be prompted whether a particular scene is currently occurring, and if the user responds affirmatively that a candidate scene is occurring, contextual features may be identified. Thus, before the user gives an explicit request to set a scene, the user may be proactively prompted to do so.

[0057] The feature aggregation engine 130 may receive sensor data from other devices and / or representations of sensor data from other devices as input and generate as output a scene feature 420 that indicates all sensor data captured while the scene is occurring. For example, the feature aggregation engine 130 may receive a phone feature 405A in the form of a vector representing sensor data from sensors of a smart phone 205, a watch feature 410A in the form of a vector representing sensor data from sensors of a smart watch 210, and a speaker feature 415C in the form of a vector representing sensor data from sensors of a smart speaker 215. In response, the feature aggregation engine 130 may generate a single embedding representing the various features, which may be embedded into an embedding space that may be utilized to compare subsequently received device features with the scene feature 420. The scene feature 420 may be stored in the scene feature database 193 along with an indication of the corresponding scene for subsequent comparison.

[0058] As an example, refer to Figure 4B, context features from three devices are shown in the embedding space of features. As shown, each of context features 415A, 415B, and 415C occupies a position in the embedding space. Feature aggregation engine 130 can determine scene features 420 representing context features in the embedding space based on the device context features. Further, feature aggregation engine 130 can determine region 425, which indicates that the corresponding scene is occurring when a current context feature embedded within region 425 is subsequently received, as further described herein.

[0059] In some implementations, one or more of the devices 106 in the ecosystem can receive a device-independent assistant request. Figure 5 , a flowchart is provided that illustrates the use of one or more current context features to determine which device to utilize from an ecosystem of devices for a response. According to one or more techniques described herein, a device-independent request 505 may be received from one or more of the devices in the ecosystem, such as a smart phone 205. The request indicates an action that a user is interested in performing, but does not indicate which device should handle the request (e.g., which device should set a timer, which device should respond to the request). The action processing engine 180 may determine response data 510 to initiate a response to the request. As shown, the device-independent request 505 "OK Assistant, set a timer for 10 minutes" is processed, and the action processing engine 180 may determine that no responding device is specified. In order to determine which device should respond to the user, the response data 510 may be provided to the devices in the ecosystem to request the current context features. In response to being provided the response data 510, the client device 106 (e.g., the smart phone 205, the smart watch 210, and / or the smart speaker 215) may each provide a current context feature indicating sensor data captured by the sensors of the corresponding device, as previously described with respect to FIG. 4A to FIG. 4B Thus, in some implementations, the phone feature 515A, the watch feature 515B, and / or the speaker feature 515C may each be an encoding of sensor data from the respective device, which may then be aggregated by the feature aggregation engine 130 into a single embedding in the embedding space, thereby producing an aggregated current context feature 520.

[0060] The device determination engine 140 can determine which scene (if any) is currently occurring based on the current scene features and previously generated scene features. For example, the scene features 420 can include embeddings of a "cooking" scene, a "working" scene, and an "exercising" scene. As an illustration, referring to FIG. 6, three scene features are shown. For exemplary purposes only, scene feature 605A is a scene feature for an "exercising" scene, scene feature 610A is a scene feature for a "working" scene, and scene feature 615A is a "cooking" signal. Each of scene features 605A, 610A, and / or 615A can be as shown in FIG. FIG. 4A to FIG. 4B Generated as described and shown in .

[0061] The device determination engine 140 can determine whether the current context feature 620 is within a threshold distance to one of the scene features and / or which scene feature the current context feature 620 is closest to. As shown, each of the scene features is associated with an area (as shown, a circle, but the threshold area can be any geometric shape). For example, area 605B surrounds scene feature 605A, and any current context feature embedded in area 605B can be identified as matching scene feature 605A. Similarly, scene feature 610A is surrounded by area 610B, and scene feature 615A is surrounded by area 615B. Further, as shown, the current context feature 620 is embedded in area 615B, which in this example indicates a "cooking" scene. Therefore, the device determination engine 140 can identify devices associated with the "cooking" scene in the scene database 194. Back to About Figure 3 In the previous example, the "cooking" scene is associated with the "kitchen speaker" device. In response, fulfillment data 525 is generated indicating that the requested timer is set using the "kitchen speaker".

[0062] In some implementations, when the current context feature is identified (and / or aggregated into a single embedding by the feature aggregation engine 130), the current context feature may not be within a threshold distance of the scene feature. Figure 6B, provides an illustration of an embedding space, the illustration including three scene features 625A, 630A, and 635A, each scene feature having a corresponding threshold region 625B, 630B, and 635B. Further, the illustration includes a current context value 640. The current context value 640 is not within the region of any of the scene features, so that the user has not indicated to utilize any device for a response in the current scene. In some implementations, one or more other techniques may be utilized to determine which device to utilize for handling a response to the user. For example, one or more techniques may be utilized to determine the device closest to the user, and when the current scene feature is not within a threshold, the determined nearest device may provide a response to the user. In some implementations, the user may be prompted to provide an indication of the current scene and / or which device the user is interested in interacting with in the current scene. For example, an audio prompt of "What are you doing right now?" may be provided to the user, and the user may respond with a known scene (e.g., "I'm cooking") or a new scene (e.g., "I'm driving"). If the current scene is a known scene, one or more of the scene features and / or regions may be updated to better refine what current context features indicate one or more of the known scene features. In addition, for example, if the user indicates a new scene, the current context features may be utilized to generate new scene features, and the user may be further prompted to provide an indication of which device should handle the interaction when the scene is occurring.

[0063] In some implementations, the current context feature may be within a threshold region of multiple context features. Figure 6C , provides an illustration of an embedding space that includes current context features within a threshold of two different scene features. As shown, scene feature 645A is associated with region 645B, and scene feature 650A is associated with region 650B. As previously described, those current context features that map to locations within a threshold region close to a scene feature can be considered to be indicative of the scene. However, as shown, current context feature 660 falls within multiple scene regions. Therefore, the device determination engine 140 cannot determine whether to select a device associated with scene feature 645A or a device associated with scene feature 650A. In response, the user may be prompted to indicate which scene (if any) is currently occurring. The device determination engine 140 may then select a corresponding device to respond to the user. Further, one or more of the regions may be updated and / or otherwise changed to reflect that the current context feature is associated with one of the scenes, rather than with other scenes. For example, referring to Fig.6D, in response to an indication from the user that current context feature 660 indicates a scene associated with 645A, new regions 665B and 670B are determined for each of the scene features. Thus, for subsequent features received, the regions do not overlap and are more refined to indicate which current context features indicate each of the scenes.

[0064] Fig. 7A and Figure 7B A flow chart illustrating an example method of selecting a device in an ecosystem of devices for responding to a user when a determined scenario is occurring is depicted. For convenience, the operations of the method are referred to as a system performing these operations, such as Figure 1 The system shown in the method - described. This system of the method includes one or more processors and / or other components of the client device. In addition, although the operations of the method are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, or added.

[0065] At step 705, an input is received from a user indicating a scenario and a specific device from the ecosystem of devices that the user has instructed to respond when the scenario occurs. Figure 1 The assistant input device 106 receives a device that shares one or more characteristics. For example, a user can indicate during the setup of the device that for the "cooking" scene, the "kitchen speaker" device should be used to respond to the request. In addition, for example, the user can indicate "When I am cooking, use the kitchenspeaker" via audio and / or video input via the user interface component 107. In some implementations, the user can indicate which device to select for response while the scene is occurring. For example, a user interface input indicating the user's designation of a device can be provided by the user while the scene is occurring. In some implementations, the user can first indicate the scene and later indicate that the scene is occurring.

[0066] At step 710, a correlation indicating an association between a scenario and a specific assistant device indicated by the user is stored. The correlation can be stored in a database that shares one or more characteristics with the scenario database 194. For example, in some implementations, the scenario can be stored on an assistant input device 106 in the ecosystem. In some implementations, the scenario can be stored in a database that is a component of a cloud-based automated assistant 119.

[0067] At step 715, an input is received indicating that a scenario has occurred during a time period. For example, an input from a user may be received via one of the devices 106 indicating that a scenario is currently occurring (e.g., indicating "cooking" via an interface provided to the user). In addition, for example, the user may indicate a scenario that occurred during a previous time period (e.g., "I worked today from 9 to 5").

[0068] At step 720, one or more contextual features are identified based on sensor data from sensors of one or more devices of the ecosystem. Figure 1 The encoder 120 in the embodiment of the present invention is generated by a component that shares one or more characteristics. For example, sensor data from sensor 105 can be encoded by encoder 120 into a code that can be provided as a context feature to one or more other components such as cloud-based automated assistant component 119. The context feature can be encoded so that it is impossible to determine the sensor data used to generate the code based on the code. Therefore, by encoding the sensor data, the sensor data is retained at the device level and is not provided to other components, thereby improving security.

[0069] At step 725, scene features are generated from the context features. Scene features can be generated by Figure 1 The feature aggregation engine 130 generates components that share one or more features. For example, Figure 4A As shown, context features may be received from multiple devices, and feature aggregation engine 130 may combine device context features to generate scene features. Scene features may be stored in a database that shares one or more characteristics with scene feature database 193, as described herein.

[0070] refer to Figure 7B At step 730, a device-independent request is received from a user. A device-independent request may include a request for an automated assistant to perform an action, but does not indicate a device that should respond and / or perform the action. In an instance where each device in an ecosystem of connected devices executes an automated assistant client, any of the devices may be configured to respond. For example, referring to Figure 2 , showing an ecosystem of devices, each of which can execute an automated assistant client and can further be configured to respond to requests.

[0071] At step 735, one or more current context characteristics are identified from sensor data of one or more sensors of devices in the ecosystem. The sensors may share one or more characteristics with sensor 105. In some implementations, the current context characteristics may be identified by each device, encoded using a component that shares one or more characteristics with encoder 120 and provides an encoding based on the sensor data to the component, as previously described with respect to step 715.

[0072] At step 740, the current context feature is compared with the scene feature to determine whether the scene is occurring. Comparing the scene feature with the current context feature may include, for example, determining a distance (e.g., Euclidean distance) between the current context feature and the scene feature and / or determining that the current context feature is embedded in a region including the scene feature.

[0073] At step 745, a specific assistant device is selected to respond to the device-independent request based on the comparison of the current context features with the scene features. For example, the scene can be identified based on the correlation of the scene features with the scenes in the scene feature database 193. Further, the correlation between the scene and the specified device can be identified from the scene database 194. Then, when the determined scene is occurring, the relevant device can be used to respond to the user request.

[0074] FIG. 8A to FIG. 8B A flow chart illustrating another example method of selecting a device in an ecosystem of devices for responding to a user when a determined scenario is occurring is depicted. For convenience, the operations of the method are referred to as a system performing these operations, such as Figure 1 The system shown in the method - described. This system of the method includes one or more processors and / or other components of the client device. In addition, although the operations of the method are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, or added.

[0075] At step 805, input is received specifying a scenario currently occurring and a particular assistant device from the device's ecosystem. As previously described herein, and in particular with respect to step 705 of FIG. 7 , input may be received from one or more devices that are part of an ecosystem of connected devices. In some implementations, the input may be received simultaneously with the one or more contextual features identified at step 810. For example, a user may indicate that a current scenario is occurring, and one or more of the devices in the ecosystem may generate contextual features and provide the contextual features to the cloud-based automated assistant component 119. At step 815, as previously described, the feature aggregation engine 130 may generate scenario features and store the scenario features along with the indication of a particular assistant device (at step 820). Steps 830 to 845 may be associated with the contextual features identified at step 810. Figure 7BSteps 725 to 745 share one or more features.

[0076] FIG. 9A to FIG. 9B A flow chart illustrating yet another example method of selecting a device in an ecosystem of devices for responding to a user when a determined scenario is occurring is depicted. For convenience, the operations of the method are referred to as a system performing these operations, such as Figure 1 The system shown in the method - described. This system of the method includes one or more processors and / or other components of the client device. In addition, although the operations of the method are shown in a particular order, this is not intended to be limiting. One or more operations can be reordered, omitted, or added.

[0077] At optional step 905, an initial device-independent request is received. As previously described herein, a device-independent request may include a request that does not specify which device should handle the request and / or provide a response. At step 910, one of the devices in the device's ecosystem is selected. For example, using one or more techniques (e.g., audio data analysis, presence sensor data), the device closest to the user may be selected to provide a response. At step 915, the selected device renders a response to the request.

[0078] At step 920, user input is received that instructs different devices to handle responses when the current scenario is occurring. For example, the user may initially request "OK Assistant, set a timer for 10 minutes" and may select a smartphone in the device's ecosystem to respond. The user may then provide user input that instructs "Use my kitchen speaker when I'm cooking."

[0079] At step 925, one or more context features are identified. Step 925 may share one or more characteristics with steps 725 and 815. For example, context features may be received from one or more devices in the ecosystem and combined by feature aggregation engine 130. At step 928, the context features are stored together with their association with the designated assistant device, such as in context feature database 193. Steps 930 to 945 may be similar to steps 725 and 815. Figure 7B Steps 730 to 745 share one or more features.

[0080] Although the implementations and examples described herein relate to a single user, in some implementations, multiple users may utilize the same ecosystem of devices and / or each user may have an ecosystem of devices that share one or more devices. In those instances, each user may have one or more scenarios associated with a user account and / or profile, so that when a user speaks a request to generate a new scenario, as described herein, the scenario may be stored with an indication of the user profile. Therefore, subsequently, when the user speaks a request that is not device-related, the TD-SID may be first executed using the utterance, and the user's profile may be identified. Therefore, as described herein, the scenario of the particular user may be used to determine which device to select to respond to the request. In other words, given the same or similar current context features, two users may select different scenarios and / or different devices to respond to environmental conditions that otherwise appear to be the same.

[0081] Further, although the implementations and examples described herein relate to selecting a particular device based on current context characteristics that indicate a scenario that is currently occurring, other implementations may provide a user with instructions for specific profiles and / or automated assistants to respond to requests that are unrelated to profiles and automated assistants. For example, a user may have a work profile and a home profile, each of which may access different applications and / or other information. A user may indicate “I am currently working. Use my work profile,” and when the current context characteristics indicate that a “working” scenario is occurring, the automated assistant may select the work profile when responding to the user. In addition, for example, a user may have a first automated assistant and a second automated assistant executing on a device, and indicate “When I am exercising, I want to talk to Assistant 1.” Scene features may be identified and stored along with an indication of “Assistant 1” for the “exercising” scene, and when the user subsequently indicates “What is on TV right now” while identifying the “exercising” scene from the current scene features, “Assistant 1” may be selected to respond to the request.

[0082] Fig.1010 is a block diagram of an example computing device 1010 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 1010 generally includes at least one processor 1014 that communicates with a number of peripheral devices via a bus subsystem 1012. These peripheral devices may include a storage subsystem 1024 (including, for example, a memory subsystem 1025 and a file storage subsystem 1026), a user interface output device 1020, a user interface input device 1022, and a network interface subsystem 1016. The input and output devices allow for interaction with a user of the computing device 1010. The network interface subsystem 1016 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0083] The user interface input device 1022 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 1010 or onto a communication network.

[0084] User interface output device 1020 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and methods for outputting information from computing device 1010 to a user or to all possible types of another machine or computing device.

[0085] The storage subsystem 1024 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1024 may include a computer program for performing selected aspects of the methods of FIGS. 4 to 6 and / or for implementing Figure 1 and Figure 3 The logic of the various components depicted in .

[0086] These software modules are typically executed by the processor 1014 alone or in combination with other processors. The memory 1025 used in the storage subsystem 1024 may include multiple memories, including a main random access memory (RAM) 1030 for storing instructions and data during program execution and a read-only memory (ROM) 1032 in which fixed instructions are stored. The file storage subsystem 1026 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 1026 in the storage subsystem 1024 or in other machines accessible by the processor 1014.

[0087] The bus subsystem 1012 provides a mechanism for the various components and subsystems of the computing device 1010 to communicate with each other as intended. Although the bus subsystem 1012 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0088] The computing device 1010 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Fig.10 The description of the computing device 1010 depicted in FIG. 1 is intended only as a specific example for illustrating some implementations. Many other configurations of the computing device 1010 are possible, and these configurations are similar to those of FIG. Fig.10 Computing devices are depicted as having more or fewer components than those shown.

[0089] In some implementations, a method is disclosed and includes the following steps: receiving a user interface input that specifies a scenario and a specific assistant device in an ecosystem of linked assistant devices that is to be utilized to interact with a user during an automated assistant interaction when a subsequent device-independent request is received while the scenario is occurring; in response to receiving the user interface input, storing an entry defining the relevance of the scenario to the specific assistant device; and receiving an instance of a further user interface input indicating the scenario occurring during a given time period. In response to receiving the instance of the further user interface input, the method further includes: identifying one or more context features present during the given time period, the one or more context features generated based on one or more sensors of one or more of the linked assistant devices; and generating a context feature for the scenario based on the context feature. After receiving the user interface input and the further user interface input, the method further includes: receiving a device-independent assistant request via one or more assistant devices in the ecosystem of the linked assistant device; identifying one or more current context features for the device-independent assistant request from one or more of the sensors; determining that the scene is occurring based on comparing the current context features with the scene features; and selecting the specific assistant device to respond to the assistant request, wherein selecting the specific device is in response to determining that the scene is occurring and in response to an entry defining the relevance of the scene to the specific assistant device.

[0090] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0091] In some implementations, the one or more context features include an encoding of given sensor data from a sensor of a given one of the link assistant devices. In some of those implementations, the encoding of the sensor data is generated by the given link assistant device. In other of those implementations, generating the context feature includes combining the encoding with one or more other encodings from other ones of the link assistant devices in the ecosystem.

[0092] In some implementations, at least one of the user interface input and the further user interface input is received from a given linked assistant device in the ecosystem that is different from the specific assistant device used to respond to the assistant request.

[0093] In some implementations, determining that the scene is occurring based on comparing the current context feature to the scene feature includes determining that the current context feature is within a threshold distance of the scene feature.

[0094] In some implementations, the method further includes generating a region in the embedding space based on the scene features, wherein the context features embedded in the region indicate the scene.

[0095] In some of those implementations, determining that the scene is occurring based on comparing the current context characteristic to the scene characteristic includes determining that the current context characteristic is within the area.

[0096] In some implementations, the given time period occurs concurrently with an instance of receiving the further user interface input. In other implementations, a method implemented by one or more processors of a client device is disclosed, and the method includes the following steps: receiving a user interface input that specifies a specific assistant device in an ecosystem of linked assistant devices, the specific assistant device being utilized to interact with the user during an automated assistant interaction when a currently occurring scenario is subsequently occurring upon receiving a device-independent request. In response to receiving the user interface input, the method further includes: identifying one or more first context features generated based on sensor data from one or more sensors of one or more of the linked assistant devices; generating one or more context features for the scenario based on the first context features; and storing an entry defining the relevance of the one or more context features to the specific assistant device. After receiving the user interface input, the method further includes: receiving a device-independent assistant request via one or more assistant devices in the ecosystem of the linked assistant device; identifying one or more current context features for the device-independent assistant request from one or more of the linked assistant devices; determining that the scene is occurring based on comparing the current context features with the scene features; and selecting the specific assistant device to respond to the assistant request, wherein selecting the specific device is in response to determining that the scene is occurring and in response to an entry defining a correlation between the one or more scene features and the specific assistant device.

[0097] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0098] In some implementations, the method further includes: based on the user interface input, identifying a user profile associated with the user; and storing an indication of the user profile with the entry. In some of those implementations, the user interface input is audio data captured by one or more microphones of one or more of the devices in the ecosystem. In some instances, identifying the user profile includes performing speaker recognition using the audio data. In other instances, the method further includes receiving user feedback indicating whether the scenario is occurring when the particular assistant device is selected. In yet other instances, the method further includes updating the one or more scenario features in response to the user feedback and based on the one or more current context features.

[0099] In some implementations, another method implemented by one or more processors of a client device is provided, and the method includes the following steps: while a first scenario is occurring, identifying one or more scenario features generated based on sensor data from one or more sensors of one or more assistant devices in an ecosystem of linked assistant devices; storing the one or more scenario features together with a correlation with a specific assistant device in the ecosystem of linked assistant devices, the specific assistant device being specified by a user to be utilized to interact with the user during an automated assistant interaction when the scenario is occurring; receiving a device-independent assistant request via one or more assistant devices in the ecosystem of the linked assistant device; identifying one or more current context features for the device-independent assistant request; determining that the scenario is occurring based on comparing the current context features with the scenario features; and selecting the specific assistant device to respond to the assistant request, wherein selecting the specific assistant device is in response to determining that the scenario is occurring and in response to the correlation of the one or more scenario features with the automated assistant device.

[0100] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0101] In some implementations, the method includes, prior to identifying the one or more scene features, receiving a user interface input specifying the particular assistant device along with an indication that the scene is occurring.

[0102] In some implementations, prior to receiving the user interface input, the method includes: receiving an initial device-independent assistant request via one or more assistant devices in the ecosystem of the linked assistant device; and selecting a different device among the devices in the ecosystem that is different from the specific assistant device to respond to the initial device-independent assistant request; and causing the response to the initial device-independent assistant request to be rendered by a different device among the devices in the ecosystem. In some of those implementations, selecting a different device among the devices in the ecosystem includes: determining a proximity of each of the devices in the ecosystem to the user; and selecting a different device among the devices based on the proximity of the devices. In some of those implementations, the user interface input specifying the specific assistant device further indicates a user preference for the specific assistant device relative to the different device when the scenario is occurring.

[0103] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0104] Where certain implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user to collect, store, and / or use the user's personal information.

[0105] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user whose personal information is to be collected to allow control over the collection of information associated with that user, providing permission or authorization as to whether to collect information and as to which portions of the information are to be collected. For example, one or more such control options may be provided to a user via a communications network. In addition, specific data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0106] Although several implementations have been described and shown herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be used, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors of a client device, comprising: Receiving user interface input specifying: Scenes, and linking a specific assistant device in the ecosystem of assistant devices, the specific assistant device to be utilized to interact with the user during the automated assistant interaction when a subsequent device-independent request is received while the scenario is occurring; In response to receiving the user interface input, storing an entry defining a relevance of the scenario to the particular assistant device; receiving an instance of a further user interface input indicative of said scenario occurring during a given time period; In response to receiving said instance of further user interface input: identifying one or more contextual characteristics present during the given time period, the one or more contextual characteristics generated based on one or more sensors of one or more of the link assistant devices; generating a scene feature for the scene based on the scene feature; After receiving the user interface input and the further user interface input: receiving a device-independent assistant request via one or more assistant devices in the ecosystem of linked assistant devices; identifying, from one or more of the sensors, one or more current context characteristics for the device-independent assistant request; Determining that the scene is occurring based on comparing the current context characteristics with the scene characteristics; and The specific assistant device is selected to respond to the assistant request, wherein selecting the specific device is responsive to determining that the scenario is occurring and to the entry defining the relevance of the scenario to the specific assistant device.

2. The method of claim 1, wherein the one or more contextual characteristics include an encoding of given sensor data from the sensor of a given one of the link assistant devices.

3. The method of claim 2, wherein the encoding of the sensor data is generated by the given link assistant device.

4. The method of claim 2 or 3, wherein generating the scenario signature comprises combining the code with one or more other codes from other link assistant devices in the link assistant device in the ecosystem.

5. A method as claimed in any preceding claim, wherein at least one of the user interface input and the further user interface input is received from a given linked assistant device in the ecosystem, the given linked assistant device being different from the specific assistant device used to respond to the assistant request.

6. The method of any preceding claim, wherein determining that the scene is occurring based on comparing the current context characteristics with the scene characteristics comprises: It is determined that the current context feature is within a threshold distance of the scene feature.

7. The method of any preceding claim, further comprising: A region in an embedding space is generated based on the scene features, wherein the context features embedded in the region indicate the scene.

8. The method of claim 7, wherein determining that the scene is occurring based on comparing the current context feature with the scene feature comprises: It is determined that the current context feature is within the area.

9. A method as claimed in any preceding claim, wherein the given time period occurs concurrently with the instance of receiving further user interface input.

10. A method implemented by one or more processors of a client device, comprising: receiving a user interface input specifying a particular assistant device in an ecosystem of linked assistant devices to be utilized to interact with a user during an automated assistant interaction when a currently occurring scenario is subsequently occurring upon receiving a device-independent request; In response to receiving the user interface input: identifying one or more first contextual features generated based on sensor data from one or more sensors of one or more of the link assistant devices; generating one or more scene features for the scene based on the first context feature; storing an entry defining a correlation between the one or more scene features and the specific assistant device; After receiving the user interface input: receiving a device-independent assistant request via one or more assistant devices in the ecosystem of linked assistant devices; identifying, from one or more of the linked assistant devices, one or more current context characteristics for the device-independent assistant request; Determining that the scene is occurring based on comparing the current context characteristics with the scene characteristics; and The specific assistant device is selected to respond to the assistant request, wherein selecting the specific device is responsive to determining that the scenario is occurring and responsive to the entry defining the correlation between the one or more scenario features and the specific assistant device.

11. The method of claim 10, further comprising: identifying a user profile associated with the user based on the user interface input; as well as An indication of the user profile is stored with the entry.

12. The method of claim 11, wherein the user interface input is audio data captured by one or more microphones of one or more of the devices in the ecosystem.

13. The method of claim 12, wherein identifying the user profile comprises performing speaker identification using the audio data.

14. The method of any one of claims 10 to 13, further comprising: User feedback is received, the user feedback indicating whether the scenario is occurring when the particular assistant device is selected.

15. The method of claim 14, further comprising: In response to the user feedback, the one or more scene characteristics are updated based on the one or more current context characteristics.

16. A method implemented by one or more processors of a client device, comprising: While the first scenario is occurring, identifying one or more scenario features generated based on sensor data from one or more sensors of one or more assistant devices in an ecosystem of linked assistant devices; storing the one or more scenario characteristics together with an association with a specific assistant device in an ecosystem of linked assistant devices, the specific assistant device specified by a user to be utilized to interact with the user during an automated assistant interaction when the scenario is occurring; receiving a device-independent assistant request via one or more assistant devices in the ecosystem of linked assistant devices; identifying one or more current context characteristics for the device-independent assistant request; Determining that the scene is occurring based on comparing the current context characteristics with the scene characteristics; and The specific assistant device is selected to respond to the assistant request, wherein the selection of the specific assistant device is responsive to determining that the scenario is occurring and responsive to the correlation of the one or more scenario features with the automated assistant device.

17. The method of claim 16, further comprising: Prior to identifying the one or more scene characteristics, receiving a user interface input specifying the particular assistant device and an indication that the scene is occurring.

18. The method of claim 17, further comprising: Before receiving the user interface input: receiving an initial device-independent assistant request via one or more assistant devices in the ecosystem of linked assistant devices; as well as selecting a different device among the devices in the ecosystem that is different from the particular assistant device to respond to the initial device-independent assistant request; as well as A response to the initial device-independent assistant request is caused to be rendered by the different one of the devices in the ecosystem.

19. The method of claim 18, wherein selecting a different one of the devices in the ecosystem comprises: determining a proximity of each of the devices in the ecosystem to the user; as well as The different one of the devices is selected based on the proximity of the devices.

20. The method of claim 19, wherein the user interface input specifying the particular assistant device further indicates a user preference for the particular assistant device relative to the different device when the scenario is occurring.

21. A system comprising: one or more hardware processors; as well as A memory storing instructions which, when executed by the one or more hardware processors, cause the one or more hardware processors to perform the method of any one of claims 1 to 20.

22. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform the operations of the method according to any one of claims 1 to 20.

Images