Adjusting assistant responsiveness according to characteristics of multi-assistant environment

Abstract

The invention relates to adjusting assistant responsiveness according to characteristics of a multi-assistant environment. Implementations set forth herein relate to employing a dynamic procedure to manage responsiveness of multiple automated assistant devices, and in particular responsiveness of automated assistants to a given spoken utterance that has been confirmed by two or more of the automated assistants. The dynamic procedure may be contextually dependent and may be adapted over time so that the automatic assistant may accommodate assistant interaction preferences that may vary by the user. For example, based on a context in which a user provides a spoken utterance, a spoken utterance, such as "stop", may be intended to affect different assistant actions. The context may refer to the location of the user relative to other rooms in the home, the time of day, the user providing spoken utterances, the placement of assistant devices in the home, and / or the status of each device in the home.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880095112.6, filed on August 23, 2018. Background Art

[0003] Humans may participate in human-computer conversations using interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0004] Many users may use multiple client devices to engage with the automated assistant. For example, some users may have a coordinated "ecosystem" of client devices, such as one or more smartphones, one or more tablet computers, one or more vehicle computing systems, one or wearable computing devices, one or more smart TVs, and / or one or more stand-alone interactive speakers, among other client devices. A user may use any of these client devices to engage in a human-computer conversation with the automated assistant (assuming the automated assistant client is installed). In some cases, these client devices may be scattered around the user's primary residence, secondary residence, workplace, and / or other structures. For example, mobile client devices such as smartphones, tablet computers, smart watches, and the like may be on the user's person and / or wherever the user last placed them. Other client devices, such as traditional desktop computers, smart TVs, and stand-alone interactive speakers may be more fixed, but may still be located in various places (e.g., rooms) in the user's home or workplace.

[0005] When a user has multiple automated assistant devices in their home, each corresponding automated assistant device may have different operating conditions due to performing different actions. At such times, a user may request to modify a particular action in progress at an automated assistant device, but inadvertently cause a different automated assistant device to modify a different action. This may be in part because some assistant devices may rely solely on whether the corresponding assistant device heard the user speak a command in order to modify a particular action. As a result, the adaptability of automated assistant devices to certain multi-assistant environments can be limited when the user is not speaking directly to the assistant device with which they intended to interact. For example, a user may accidentally stop an ongoing action being performed at an assistant device with which they did not intend to communicate, requiring the user to repeat the previous spoken utterance in order to recall the stopped action.

[0006] Undesirably, in response to accidentally stopping an action, memory allocated for the stopped action may be reused by the assistant device, and when the user subsequently recalls the stopped action, the assistant device may have to fetch any relevant data again. The consequence of this seemingly redundant behavior may be a waste of network resources because, for example, some spoken utterances may be processed by a natural language model that is accessible only via a network connection. Furthermore, any data related to the unintentionally canceled action will have to be re-downloaded to facilitate completing the action, and any energy wasted due to canceling an energy-intensive action (e.g., controlling a display backlight, a heating element, and / or a powered appliance) may not be recoverable. Summary of the invention

[0007] The present disclosure generally relates to methods, apparatus, and computer-readable media (transient and non-transient) for managing the responsiveness of multiple automated assistant devices using dynamic protocols. Specifically, the responsiveness of an automated assistant to a given spoken utterance that has been confirmed by two or more of the assistant devices can be adapted according to dynamic protocols. Dynamic protocols can be rules for allowing a particular automated assistant device to perform a particular action based on a given context that can change over time. Additionally or alternatively, each automated assistant device can operate according to multiple different dynamic protocols that manage the ability of the automated assistant to delegate specific actions for execution to a particular automated assistant device. Such dynamic protocols can identify certain contexts in which a particular automated assistant device will perform a particular action, such as when multiple automated assistant devices simultaneously confirm an incoming request for a particular action to be performed.

[0008] As an example, a user may be listening to music in their living room through a first assistant device, however, a second assistant device can be set to provide an alarm in the kitchen to alert the user of their cooking progress. When the alarm sounds, the user can provide verbal utterances to the second assistant device to modify the action corresponding to the sounding of the alarm. For example, the user can provide the verbal utterance "Assistant, stop!" from their living room, but intend for the verbal utterance to affect the second assistant device. Because the user is in their living room, the first assistant device can receive the verbal utterance at a higher volume compared to the second assistant device in the kitchen. However, in order to stop the alarm action in progress at the second assistant device without affecting the music playing action in progress at the first assistant device, the first assistant device can access one or more dynamic procedures before affecting any ongoing actions.

[0009] The dynamic protocol can be associated with the user's current context, which can be characterized by context data indicating the user's location in the living room, the location of each assistant device in the living room and the kitchen, respectively, and / or each ongoing action performed at each assistant device, respectively. For example, the dynamic protocol can be based on training data indicating that the user has historically provided the spoken word "stop" within a threshold time period (e.g., 5 seconds) after and / or within the second assistant device emitting an alarm sound. Such a dynamic protocol can provide a basis on which the automated assistant can determine whether to stop the music in response to the spoken word "Assistant, stop" within the current context. For example, in response to the spoken word "Assistant, stop", the automated assistant can access the dynamic protocol, determine that the spoken word was provided within a threshold time period of the alarm sounding, and modify the alarm action of the second assistant device. In other words, regardless of whether the music playback and the alarm sounding are concurrently performed at different assistant devices, the automated assistant will only affect the alarm sounding based at least on the dynamic protocol and the current context.

[0010] In some implementations, the training data can be used as a basis for generating and / or modifying dynamic procedures over time. The training data can be based on various different spoken utterances provided by the user to initialize and / or modify one or more ongoing actions being performed by one or more applications and / or devices. For example, a dynamic procedure can be generated based on the automated assistant confirming that the user has provided one or more corrective spoken utterances in similar contexts. The user can provide corrective spoken utterances in response to the automated assistant incorrectly modifying a particular action relative to the user's intent. As an example, the user may have previously provided the spoken utterance "Assistant, stop" after the second assistant device identified above began to sound an alarm, and in response the automated assistant may have incorrectly stopped the ongoing music playback action at the first assistant device. To address the incorrect modification of the music playback action, the user can provide the corrective spoken utterance, "No, stop the alarm". In response, the automated assistant can process the corrective spoken utterance, start the music playback action again, and stop the alarm action. Furthermore, the automated assistant enables a dynamic protocol to be generated that characterizes the current context and the user's preference to stop the alarm only when the user says "stop" while both the alarm and music are audible, and optionally when the user is in a living room when saying "stop." In this way, the user will not have to continue to provide corrective verbal utterances relative to the context, thereby conserving computing resources that would otherwise be spent on language processing and signaling between assistant devices to comply with corrective verbal utterances.

[0011] In some implementations, a dynamic procedure can have one or more context dependencies. A context dependency can refer to a specific context characteristic that affects whether a specific action will be initialized and / or modified given a specific context. As an example, a dynamic procedure can identify occupancy-related data that must be satisfied in order to perform a specific action in a specific room of a user's home. For example, when a person (e.g., a sleeping baby) is detected in a bedroom at home, the user may prefer not to disturb the person with sound after a specific time (e.g., 6 p.m.). Therefore, the dynamic procedure can identify one or more rooms, one or more affected actions (e.g., playing music, sounding an alarm, making noise in other ways, etc.), one or more affected assistant devices, one or more affected accounts, one or more output modalities associated with the action, one or more output modalities available at the assistant device, and / or any other features that can be associated with the automated assistant. Specifically, according to the above example, the corresponding dynamic procedure can identify the bedroom of the home, the assistant device located in the bedroom, and the specific time at which certain actions are restricted thereafter. Therefore, when the automated assistant receives a request for an action to be performed, the automated assistant can access the dynamic procedure to ensure that the automated assistant acts within the limits established by the dynamic procedure.

[0012] In some implementations, a dynamic procedure for controlling one or more assistant devices can be based on whether a first assistant device identified in a given context is associated with two or more users. In addition, when a first assistant device is associated with two or more users, a second assistant device identified in a given context can be controlled by the two or more users regardless of whether the second assistant device is not immediately and / or directly associated with the two or more users. As an example, two users may be watching a movie on a television in a living room of a first user's home, and each of the two users can be identified as a contact in each other's corresponding portable computing device. Specifically, a second of the two users can have a history of using its portable computing device to call a first of the two users.

[0013] While watching a movie, the first user may leave the living room to get a snack, and the second user, after observing the first user walking away, can provide a verbal utterance to the automated assistant interface of the television to pause the movie. For example, the second user can provide the command "Assistant, pause" to the automated assistant interface (e.g., microphone) of the television. In response, the automated assistant operating according to the account of the first user can access and / or generate a dynamic procedure for allowing the second user to control the playback of the movie while the second user and the first user are at the home of the first user. Specifically, the dynamic procedure may require that the second user can control the playback of the movie when the first user has the second user as a contact in his portable computing device and the first user and the second user are both at the home of the first user. In some implementations, the second user can be identified by the automated assistant by determining that the second user's portable computing device is connected to the home WiFi of the user's home and / or the portable computing device also hears the verbal utterance provided by the second user. Additionally or alternatively, the automated assistant can determine that the second user's portable computing device is within a threshold distance of the second user when the second user is providing the verbal utterance, at least to verify that the second user is providing the verbal utterance.

[0014] In some implementations, in response to allowing the second user to pause a movie being played at the television, the automated assistant can modify the dynamic protocol associated with pausing the movie at the television to allow the second user to pause the movie. In this manner, the automated assistant can adapt the dynamic protocol to relax specific protocols, thereby preserving computing resources. For example, once the dynamic protocol has been updated and the second user issues another request to the television, the automated assistant will not need to confirm any correspondence between the portable computing devices of the first user and the second user. Instead, the automated assistant may only need to verify the presence of the second user or its corresponding device at home, or to verify that the first user and the second user are both at home and / or in a common room (e.g., both are in a living room).

[0015] In some implementations, the context dependency of the dynamic program can be based on a layout of the locations of one or more stored automated assistant devices. The stored layout can characterize room adjacency, number of rooms, room size, location of the device within each respective room, most occupied portion of each particular room (e.g., heat map per user and / or per room), and / or any other characteristic associated with the location. Additionally or alternatively, the context dependency of the dynamic program can be based on the specific assistant device that receives the specific spoken utterance, the available modalities of the assistant device within a specific location, the actions that can be performed by each assistant device, and / or any other contingency that can be associated with the actions of the assistant device. For example, a user can have an audio-only assistant device in his kitchen and an assistant-enabled TV in his living room. The automated assistant for the user can operate according to a dynamic program that allows the assistant-enabled TV to be controlled from the audio-only assistant device when the user provides a request corresponding to a modality-specific action. For example, when a user is in his kitchen and the user's child is in the living room, the user can ask the audio-only assistant device in his kitchen to play a movie for the children on the TV without explicitly specifying the device. The spoken utterance may be, for example, "Assistant, play a kid's movie," and in response to the audio-only assistant device receiving the spoken utterance, the audio-only assistant device can identify the relevant dynamic protocol. The identified dynamic protocol can allow the audio-only assistant device to control other assistant devices with video modality when the user provides a video-related request to the audio-only assistant device in their kitchen.

[0016] In some implementations, for requests from a user that do not specify a particular assistant device, a particular modality, the location of an action to be performed, or otherwise contain ambiguous language, the automated assistant can identify one or more dynamic procedures to resolve the ambiguity. In some cases, a request can be considered ambiguous when a request to identify an assistant device with the same description has another assistant device. For example, a user can have two or more assistant-enabled televisions in two or more different locations in their home. Thus, a spoken utterance such as "Assistant, play the movie Pi on mytelevision" can be considered ambiguous because it can be interpreted to refer to multiple devices. As a result, when the automated assistant receives a request to identify a single assistant device but ambiguously refers to multiple assistant devices, the automated assistant can access one or more dynamic procedures to resolve the ambiguity.

[0017] As an example, in response to receiving the aforementioned spoken utterance, the automated assistant can determine that the spoken utterance includes an ambiguity regarding a particular type of device (e.g., "television"), and access a dynamic procedure corresponding to the ambiguity regarding the particular type of device. The accessed dynamic procedure can indicate that when the user is in their kitchen and requests that a movie be played on their TV, the user prefers that the movie be played on their living room TV rather than their bedroom TV or the children's room TV. In some implementations, this dynamic procedure can be different depending on the user who is providing the spoken utterance. Thus, if one of the children is providing the spoken utterance in the kitchen, the automated assistant can cause the movie to start playing on the children's room TV rather than the living room TV.

[0018] In some implementations, selection of an assistant device to perform a requested action corresponding to a spoken utterance can be based on the degree to which the assistant device heard the spoken utterance relative to other assistant devices. For example, a user who provides a spoken utterance from their kitchen will likely have an assistant device in their kitchen perform a particular action at least as well as a different assistant device located in a different room of the house. Additionally or alternatively, the automated assistant can select candidate assistant devices from a collection of assistant devices in the user's home based on which assistant devices confirmed the spoken utterance. Thereafter, the automated assistant can compare the degree to which each candidate assistant device heard the spoken utterance and select the assistant device that heard the spoken utterance to the highest degree.

[0019] As an example, the automated assistant can select candidate assistant devices based on whether the candidate assistant devices heard the user speak the spoken utterance at all, and then identify the volume level at which each assistant device heard the spoken utterance. The automated assistant can determine a tolerance level based on the highest determined volume level, and further reduce the number of candidate assistant devices considered. Any remaining candidate assistant devices whose corresponding determined volume levels fall within the tolerance can be further analyzed to identify the most appropriate assistant device for performing the requested action. For example, a user can provide a spoken utterance requesting music to be played at a target assistant device, but the target assistant device may not be the assistant device that heard the spoken utterance the loudest. However, the target assistant device can fall into a group of candidate assistant devices that have confirmed that the spoken utterance was received within a specific threshold tolerance. The identifiers of the candidate assistant devices can be compared with one or more dynamic procedures related to the requested action (e.g., playing music) to determine the user's most preferred assistant device to perform the requested action using additional data (e.g., context data).

[0020] In some implementations, the trajectory of a user while providing particular spoken utterances can be identified for one or more dynamic protocols. In this way, the trajectory of a user moving while providing spoken utterances can be a limiting factor for identifying assistant devices that will perform the action requested by the user. As an example, a dynamic protocol can dictate how to handle a spoken utterance such as "Assistant, stop" while a user is walking through their house, music is playing at a first assistant device, and an alarm is being sounded by a second assistant device. Because alarms can be associated with things that need attention, such as an oven being on for a certain amount of time, a user may have a history of walking toward the source of the alarm sound when providing a "stop" request. The dynamic protocol can capture such learned user habits to prevent the music from being stopped by the user when the user actually intended to stop the alarm.

[0021] In some implementations, the dynamic protocol can operate according to one or more machine learning models that are configured to receive as input context data associated with the context of a particular spoken utterance. For example, context data can be provided to a machine learning model in order to derive outputs that would otherwise be dictated by the dynamic protocol. The context data can characterize the layout of the home, identifiers of one or more users in the home, identifiers of one or more assistant devices in the home, the location of each assistant device in the home, adjacency data corresponding to each room in the home, the location of each person in the home, the time of day, the condition of each device in the home, voice characteristics derived from audio based on the spoken utterance, the type of ongoing action being performed by each device (e.g., passive action, active action, etc.), and / or any other data that can describe the context in which the user provides a particular spoken utterance. In this way, each assistant device can appear to adapt to changes in the user's preferences for certain automated assistant devices to perform specific actions.

[0022] A process for selecting an assistant device that will ultimately perform an action indicated by a spoken utterance can be performed at one or more remote computing devices, such as one or more server devices operating at a network separate from the assistant device. In other words, the process for selecting a preferred assistant device to perform an action can include selecting a specific server device or a designated cluster of co-located server devices that will process input for identifying a preferred assistant device. For example, each assistant device can provide status information to a corresponding server device or co-located server device cluster associated with each assistant device. Since each assistant device is associated with one or more different accounts and / or one or more different users, each assistant device can be associated with a different server device. Therefore, in order to rationalize the delegation of actions to the automated assistant, a specific server device can be selected for other server devices to respond to specific spoken utterances received at multiple assistant devices. For example, a first assistant device can provide status information to a first server device, and a second assistant device can provide other status information to a second server device. Based at least on the first automated assistant hearing the spoken utterance loudest, the first server device can be designated as the selected server device for processing the status information and selecting the target assistant device. Thereafter, the second server device and / or the second assistant device can determine that the first server device has been designated as the selected server device and forward other situation information and / or any other data related to the spoken utterance to the first server device. The first server device can then use the situation information, which can include context data associated with the spoken utterance, to designate one of the assistant devices to perform an action inferred from the spoken utterance.

[0023] Implementations of the present disclosure provide a method for improving the control of a group of devices in a specific environment so as to reduce the number of incorrect device triggering instances. In addition to ensuring that the results of user commands correspond to the user's intentions, this also reduces the waste of technical resources that would otherwise be associated with the execution of incorrect device triggers. This waste of resources may include, but is not limited to, communication overhead associated with instructing devices to perform specific functions that are not required to be performed, power consumed by activating specific functions in devices that are not intended to be controlled, and memory and / or power consumed by suspending devices in unnecessary specific states. Instances of correct device triggering can occur more quickly, thereby ensuring that the automated assistant operates more responsively.

[0024] Validity is further ensured by taking into account contextual information for multiple devices, enabling device control on a dynamic basis appropriate to the specific environment.

[0025] By training a database of dynamic procedures, it is possible to ensure that the system is able to learn the user's behavior, thereby improving the efficiency with which device control can be achieved, further reducing instances of incorrect device triggering, and ensuring that future conflicts in device control will be resolved as effectively as possible.

[0026] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.

[0027] In some implementations, a method implemented by one or more processors is described as including, such as receiving audio data capturing a spoken utterance of a user, wherein the spoken utterance embodies a request for a modification action to be performed via an automated assistant and is received by at least one of two or more computing devices, each of which is capable of performing a modification operation via an automated assistant. The method can further include, using the audio data capturing the spoken utterance, determining that the modification action is capable of modifying each of: a first ongoing action being performed at a first device of the two or more computing devices, and a second ongoing action being performed at a second device of the two or more computing devices. The method can also include, based on determining that the modification action is capable of modifying both the first ongoing action and the second ongoing action, identifying a stored procedure corresponding to the modification action, wherein the stored procedure characterizes a context dependency of execution of the modification action. The method can further include, based on identifying the stored procedure, accessing context data associated with the stored procedure, wherein the context data characterizes a first type of the first ongoing action and a second type of the second ongoing action. The method can also include determining, based on the context data and the stored procedure, a target computing device at which the modification action in the two or more computing devices is to be performed via the automated assistant. The method can further include causing the modification action to be performed at the target computing device via the automated assistant based on determining the target computing device at which the action is to be controlled.

[0028] In some implementations, determining a stored procedure includes identifying a stored procedure from a plurality of different procedures accessible by the automated assistant, and the plurality of different procedures are determined based on one or more previous instances in which a user requested the automated assistant to perform a modification action. In some implementations, the stored procedure is determined based on one or more corrective verbal utterances previously received by the automated assistant from the user, each of the one or more corrective verbal utterances being provided by the user after a corresponding previous instance in which the modification action was incorrectly performed via the automated assistant as indicated by the corrective verbal utterance. In some implementations, the first ongoing action includes providing an audible sound via a speaker of the first device.

[0029] In some implementations, the context data includes occupancy data characterizing occupancy of one or more rooms of a building to which the user provided the spoken utterance, and the stored protocol indicates a preference of the user to restrict at least one action to a specific room in the one or more rooms of the building. In some implementations, the context data further indicates a location of the user within a specific room in the one or more rooms of the building, and the stored protocol indicates another preference of the user for a target computing device to control the at least one action via the automated assistant. In some implementations, causing the modified action to be performed at the target computing device includes restricting execution of a specific ongoing action at the target computing device of the two or more computing devices. In some implementations, causing the at least one action to be performed at the target computing device includes modifying an ongoing rendering of audio data or visual data, and the context data indicates a time when the ongoing rendering of the audio data or visual data is initialized. In some implementations, the context data identifies an operating condition of each of the two or more computing devices, and each of the two or more computing devices is configured to provide corresponding operating condition data to a common co-located server device cluster for processing the operating condition.

[0030] In other implementations, a method implemented by one or more processors is described as including determining, such as at a server device in communication with a plurality of client devices, status information corresponding to an activity state of a plurality of client devices disposed around a location of a user, wherein each of the plurality of client devices is accessible to an automated assistant. The method can also include receiving data based on a spoken utterance provided by a user to at least one of the plurality of client devices that is operating according to the determined status information, wherein the spoken utterance embodies a request for an action to be performed by one or more of the plurality of client devices. The method can further include accessing a stored set of dynamic procedures associated with the plurality of client devices in response to receiving the data based on the spoken utterance, wherein at least one stored procedure in the stored set of dynamic procedures characterizes a contextual dependency of performance of an action by the one or more client devices on the status information. The method can also include identifying a target client device for performing an action based on at least one stored procedure in the stored set of dynamic procedures and the status information, wherein the contextual dependency characterized by the at least one stored procedure includes at least one volume-agnostic condition for specifying a target client device for performing an action. The method can further include causing the target client device to perform an action via the automated assistant.

[0031] In some implementations, the location includes multiple rooms characterized by a stored topology accessible to the server device, and the method further includes: accessing current context data that characterizes the context dependency as a user being located in a first room including at least one client device and a target client device being located in a second room of the multiple rooms. In some implementations, the method can include determining, based on receiving the data, whether the spoken utterance is at least one type of utterance selected from an ambiguous type of utterance and a specific type of utterance, wherein the following specific operations are performed when the spoken utterance is determined to be an ambiguous type of utterance: accessing a stored set of dynamic procedures associated with the multiple client devices, identifying a target client device to perform an action, and causing the target computing device to perform the action.

[0032] In some implementations, the method can include, when the spoken utterance is determined to be an utterance of a specific type: identifying a designated client device specifically identified by the user via the spoken utterance, and causing the designated client to perform an action while bypassing access to a stored set of dynamic procedures associated with the plurality of client devices. In some implementations, characterizing the received spoken utterance as an utterance of at least one type selected from an utterance of an ambiguous type and an utterance of a specific type is based on a prior interaction between the user and the automated assistant.

[0033] In still other implementations, a method implemented by one or more processors is described as including, such as receiving status information from multiple client devices associated with a user, the status information indicating that one or more client devices in the multiple client devices are performing an ongoing action. The method can also include, based on receiving the status information, selecting a designated co-located server device cluster for processing the status information provided by the multiple client devices, wherein the designated co-located server device cluster is selected from a plurality of different co-located server device clusters. The method can further include: determining that a spoken utterance from the user is received at an automatic assistant interface of a client device in the multiple client devices, wherein the automatic assistant interface is configured to facilitate interaction between the user and an automatic assistant that can be accessed by multiple client devices, and the spoken utterance embodies a request for the automatic assistant to perform an action. The method can also include, in response to determining that the spoken utterance is received from the user at the automatic assistant interface of the client device, causing audio data generated at the client device to be provided to a designated co-located server device cluster, wherein the audio data captures at least a portion of the spoken utterance. The method can further include providing the audio data to the designated co-located server device cluster based on the client device, so that the designated co-located server device cluster identifies a designated client device in the multiple client devices for performing an action requested by the user. The method can also include causing the designated client device to perform the action requested by the user

[0034] In some implementations, the method can include: accessing a stored device topology in response to receiving status information, the stored device topology indicating an arrangement of various client devices within a location, wherein the plurality of client devices is a subset of the various client devices, and determining that the plurality of client devices are in a common room of a location where a user provides a spoken utterance to an automated assistant interface. In some implementations, an action corresponds to a specific output modality, and the method further includes: determining that the plurality of client devices in the common room of the location include a specific output modality corresponding to the action identified by the request. In some implementations, two or more of the plurality of client devices are associated with different user accounts, and the two or more client devices are configured to communicate with a plurality of different co-located server device clusters based on the two or more client devices being associated with different user accounts. In some implementations, the specific status information provided by the respective client devices indicates whether the respective client devices of the plurality of client devices are performing an ongoing action, and when the respective client devices are performing an ongoing action, the status information indicates a modality in which the respective client devices are employed to perform the ongoing action. In some implementations, the designated client device is selected based on stored training data indicating correspondence between actions identified by the request and context information provided by a plurality of client devices.

[0035] In some implementations, a device is configured to include an audio interface configured to generate a signal in response to receiving a spoken utterance, the spoken utterance being received to facilitate communication between a user and an automated assistant, wherein the spoken utterance is received at both the audio interface and another audio interface of another client device, and the other client device is a client device that is one of a plurality of client devices that can be accessed by the automated assistant. The device can also include a network interface configured to receive communications from another device that also receives the spoken utterance, wherein the communications provided by the other device include first context data representing a context associated with the other device. The device can further include a memory configured to store second context data representing another context associated with the spoken utterance being received at the audio interface. The device may also include one or more processors configured to perform operations including: determining a ranking of multiple client devices, wherein the ranking includes at least a first ranking based on first context data and a second ranking based on second context data, and wherein each ranking in the ranking at least partially affects selection of a particular device among the multiple client devices for responding to the spoken utterance via the automated assistant, and when a candidate device among the multiple client devices has a corresponding ranking that is distinguishable from other rankings in the ranking: causing the automated assistant to respond to the spoken utterance via the candidate device based at least on the corresponding ranking of the ranking.

[0036] In some implementations, the network interface is further configured to receive a communication from another device before the spoken utterance is received at the audio interface. In some implementations, the determination of the ranking by the one or more processors is performed before the spoken utterance is received at the audio interface or while the spoken utterance is being received at the audio interface. In some implementations, the one or more processors are further configured to perform operations including: generating second context data before the spoken utterance is received at the audio interface, and causing the network interface to provide a separate communication to another device, wherein the other device is configured to determine whether the other device is assigned a task to respond to the spoken utterance via an automated assistant. In some implementations, the one or more processors are further configured to perform operations including: sending a separate communication to a separate device that is communicating with the network interface but has not yet confirmed the spoken utterance; and determining whether the separate device will be assigned a task to respond to the spoken utterance via an automated assistant.

[0037] In some implementations, the one or more processors are further configured to perform operations including: generating third context data characterizing an operating condition of the individual device. In some implementations, the one or more processors are further configured to perform operations including: when two or more of the plurality of client devices have respective rankings that result in a tie ranking between the two or more client devices: determining supplemental context data characterizing the two or more client devices, and based on the supplemental context data, causing the tie ranking to be resolved in a manner that favors the respective client device whose ranking is distinguishable from the rankings of other client devices in the plurality of client devices.

[0038] In some implementations, a method implemented by one or more processors is described as including, for example, determining that a spoken utterance has been received at a first client device, wherein the spoken utterance is provided by a user to facilitate communication with an automated assistant accessible via an assistant interface of the first client device, and the spoken utterance is received by a plurality of client devices including the first client device. The method can also include, in response to the second client device receiving the spoken utterance, receiving a communication from a second client device receiving the spoken utterance, wherein the communication provided by the second client device includes first context data characterizing a context associated with the second client device. The method can further include determining a ranking of a plurality of client devices, wherein the ranking includes at least a first ranking based on the first context data and a second ranking based on the second context data, and wherein each ranking in the ranking at least partially affects the selection of a particular client device in the plurality of client devices to respond to the spoken utterance. The method can further include, when a candidate client device in the plurality of client devices has a corresponding ranking that is distinguished from other rankings in the ranking: causing the automated assistant to respond to the spoken utterance via the candidate device based at least on the corresponding ranking of the ranking.

[0039] In some implementations, the communication is received from the second client device before the spoken utterance is received by the first client device. In some implementations, determining the ranking is performed before the spoken utterance is received at the first client device or while the spoken utterance is being received at the first client device. In some implementations, the method can include: generating second context data before the spoken utterance is received at the audio interface; and causing the network interface to provide a separate communication to another device, wherein the other device is configured to determine whether the other device will be tasked with responding to the spoken utterance via the automated assistant. In some implementations, the method can include: sending a separate communication to a separate device that is in communication with the first client device but has not yet confirmed the spoken utterance, and determining whether the separate device will be tasked with responding to the spoken utterance via the automated assistant. In some implementations, the method can include generating third context data characterizing the operating condition of the separate device. In some implementations, the method can include, when two or more client devices among a plurality of client devices have respective rankings that result in a tie ranking between the two or more client devices: determining supplemental context data characterizing the two or more client devices, and, based on the supplemental context data, resolving the tie ranking in a manner that facilitates ranking the respective client devices as distinguishable from rankings of other client devices among the plurality of client devices.

[0040] In other implementations, a method implemented by one or more processors is described as including operations such as: storing a plurality of dynamic procedures in a database, each dynamic procedure being associated with a user command, and indicating a correspondence between each of a plurality of controlled devices for performing an action corresponding to the user command and context information of a user and / or at least one of a plurality of client devices including the corresponding controlled device. The method can also include training the stored plurality of procedures by: receiving a first input representing a user command from one of a plurality of client devices, identifying one of the stored plurality of dynamic procedures corresponding to the received user command, requesting and obtaining context information of a user and / or at least one of the plurality of client devices including the controlled device, identifying one of the plurality of client devices as a controlled device based on the context information associated with the stored dynamic procedures, initiating execution of the identified dynamic procedure by the identified client device, receiving a second input representing an update of the user command from one of the plurality of client devices, correcting the update indicating that the updated client device is controlled, and updating the stored dynamic procedures in the database to indicate the correspondence between the updated client device and the obtained context information. The method can also include initiating, by the updated client device, execution of the identified dynamic procedure.

[0041] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0042] It should be appreciated that all combinations of the above-described concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1A Illustrated is a perspective view of a user providing spoken utterance within a location that includes multiple different assistant devices.

[0044] Figure 1B A perspective diagram illustrating how each device responds to spoken utterances provided by the user.

[0045] Figure 2 Illustrated is a system for selecting a target device from a plurality of candidate devices for responding to a spoken utterance in order to avoid inadvertently causing an unintended device to respond to the spoken utterance.

[0046] Figure 3A and Figure 3B Diagram illustrating a system for selecting a server that will determine a target client device that will respond to a particular spoken utterance.

[0047] Figure 4A and Figure 4B The illustration shows an implementation that allows for the arbitration of the selection of a client device for responding to a spoken utterance from a user to be performed locally.

[0048] Figure 5A and Figure 5B Illustrated is a method for modifying ongoing actions being performed by client devices in response to spoken utterances received at multiple client devices.

[0049] Figure 6Illustrated is a method for selecting a co-located server device cluster for processing status information from a plurality of client devices and selecting a target client device for performing an action based on a result of processing the status information.

[0050] Figure 7 is a block diagram of an example computer system.

[0051] Specific implementation method

[0052] Figure 1A A stereogram 100 illustrates a user 116 providing a spoken utterance 140 within a location 138 that includes multiple different assistant devices. Each assistant device can be individually operated to exhibit a variety of different states depending on the action being performed by each assistant device. To ensure that a particular spoken utterance affects only the intended target device, each of the assistant devices can operate according to one or more dynamic protocols. Dynamic protocols are rules or instructions that can determine how a particular device will react to a particular spoken utterance or request given a particular context associated with the particular spoken utterance. Dynamic protocols can be modified over time based on changes in user preferences, changes in the number of devices within a location, and / or any other characteristics that can affect the environment in which a device can operate.

[0053] For example, Figure 1A As provided in , location 138 can include vehicle 114, which can have assistant device 104 that can help the user control vehicle 114. Location 138 can also include assistant device 118, which can be a standalone assistant device that can receive touch and / or audio input and provide audio and / or visual output. Additionally, location 138 can include television 128, which can allow user 116 to access television automated assistant 130 via assistant interface 132 of television 128.

[0054] Each of the television 128, the assistant device 118, and the assistant device 104 can include an automated assistant. For example, the assistant device 104 can include a vehicle automated assistant 106 that can be accessed by the user 116 via the assistant interface 108. In addition, the assistant device 118 can include a client automated assistant 120 that can be accessed via an assistant interface 122. The assistant interface 122 can include one or more of a microphone, a speaker, a display panel, a touch display panel, a camera, a tactile interface such as a keyboard, and / or any other interface that a user can use to control a computer device. In some implementations, the user 116 can control the automated assistant and / or the device using the assistant interface of another device. For example, each of the vehicle 114, the assistant device 118, and the television 128 can be connected at a public local area network such as a Wi-Fi network, thereby allowing data to be transmitted between two or more devices and / or applications via the network.

[0055] In some implementations discussed herein, each of the devices at location 138 is capable of operating according to one or more dynamic regulations in a dynamic regulation set. Dynamic regulations can be provided to ensure that the device on which the user intends to perform an action receives instructions for performing the action, rather than accidentally executing instructions at different devices. As a non-limiting example, user 116 is able to watch a movie with his friends via television 128, which can be located in the living room of location 138. While they are watching the movie, assistant device 118 may be performing an active action of executing a timer. The timer may, for example, be intended to monitor the status of the food that user 116 is baking in his oven. The actions being performed by both television 128 and assistant device 118 can be characterized by action data accessible and / or stored at each device or a server device communicating with each device. When the ongoing action is initialized, executed, and / or completed at each perspective device, the action data corresponding to each corresponding device can be updated to indicate the operating status of each device and / or one or more actions of each device.

[0056] Following the foregoing example, when the timer has completed, assistant device 118 can provide output 142, such as sounding an alarm. In response to the timer completing, action data 124 can be updated to indicate an operating condition of assistant device 118. For example, the operating condition can indicate that assistant device 118 is "sounding an alarm" and the operating condition of the timer is "timer completed." Action data 124 can identify one or more operating conditions 126 (e.g., condition (N), where N is any positive integer).

[0057] Because user 116 was watching a movie when the alarm sounded, user 116 may wish to stop the alarm without stopping any actions occurring at television 128 (e.g., playing video and / or audio as indicated by status 136) and any actions occurring at vehicle 114 (e.g., charging the vehicle and remaining parked as indicated by status information 112). For example, to facilitate modifying the operating status of assistant device 118, user 116 can provide spoken utterance 140 "Assistant, stop". Figure 1B As indicated in , each of the assistant devices is capable of confirming spoken utterances 140. Specifically, Figure 1BA perspective diagram 144 illustrates how each device can respond to a spoken utterance 140 provided by a user 116. In response to confirming the spoken utterance, each automated assistant at each device can have its corresponding status information updated to indicate that the spoken utterance was received. For example, assistant device 104 can update its status information 112 to indicate that the user input was received. Similarly, television 128 and assistant device 118 can update their corresponding status information to indicate that the spoken input has been received. However, to ensure that the devices are not inadvertently affected by the spoken utterance, each device can operate according to one or more dynamic procedures and / or sets of dynamic procedures.

[0058] In some implementations, dynamic regulations can be generated based on training data that reflects user habits, and the dynamic regulations are generated with permission from the user 116. For example, one or more dynamic regulations can be associated with contextual data that characterizes previous instances of user requests and / or device operating conditions that were apparent when previous user requests were provided. Therefore, contextual data collected by one or more assistant devices can be used in conjunction with one or more dynamic regulations to determine a target device or multiple target devices with which the user intends to interact. For example, when the assistant device 118 is sounding an alarm, the television 128 is playing a video, and the vehicle 114 is charging, the user may have previously provided the verbal utterance "Assistant, stop". If an action other than sounding an alarm is inadvertently stopped in response to the verbal utterance "Assistant, stop", the user may have provided a corrective verbal utterance such as "No, stop the alarm". The automated assistant that receives the corrective verbal utterance can cause a dynamic regulation to be generated and / or updated to reflect the user's correction and / or preference. Specifically, the dynamic protocol can characterize an action that the user 116 intends to perform (eg, a stop action), an operating condition of a device in the location 138 , and / or a device that the user 116 intends to interact with (eg, the assistant device 118 ).

[0059] In some implementations, dynamic regulations can be based on contextual data that characterizes the occupancy of location 138 and / or the current operating conditions of one or more devices within location 138. For example, user 116 and his friends may be watching a movie at TV 128. During the movie, user 116 can get up to check the status of something cooking in his kitchen. In order to ensure that user 116 does not miss any movies, the friend can try to provide a spoken word "Assistant, stop" to pause the movie. Because the spoken word is ambiguous about the action to be modified and / or the device to be affected, the automatic assistant that has received the spoken word can access the dynamic regulations to determine how to respond to the spoken word. Specifically, the automatic assistant can determine that the spoken word "Assistant, stop" corresponds to a plurality of different dynamic regulations. Additionally, the automatic assistant can determine that the voice profile of the spoken word corresponds to a "guest" voice profile, which is associated with one or more dynamic regulations in a plurality of different dynamic regulations. One or more dynamic regulations can be generated based on training data that characterizes previous instances of friends / guests providing "Assistant, stop" spoken words. During the previous instance, the spoken utterance did not affect any device, so in response, the user 116 provided the corrective spoken utterance "Assistant, pause the movie." Based on the previous context in which the friend requested the automated assistant to "stop" something, the device did not respond, and the user 116 caused the movie to "pause," the automated assistant can cause a dynamic protocol to be updated or generated to reflect this preference of the user 116. Specifically, the preference can be characterized as giving permission to a friend with a particular voice profile to pause a movie playing at the television 128 when the user 116 and the friend are in the location 138.

[0060] In some implementations, when multiple devices confirm spoken words from user 116, it is possible to select a target device for responding to the spoken words based on a loudness tolerance generated in response to the spoken words. Specifically, a unique loudness tolerance corresponding to the received spoken words can be generated. Each local automated assistant at location 138 can arbitrate the formulation of a loudness tolerance so as to reduce multiple candidate target devices to those candidate target devices that hear the spoken words at a volume within the formulated loudness tolerance. For example, in response to assistant device 104, assistant device 118, and television 128 confirming the spoken words "Assistant, stop", each device can determine the volume at which they heard the spoken words and pass the determined volume to each of the other devices. Each device can then use each determined volume from each device to formulate a loudness threshold. The loudness threshold can be generated by each device according to the functions accessible by each device.

[0061] Each specific device can compare the determined volume of the specific device with the established loudness threshold to determine whether the specific device is a candidate target device. If the specific device is not a candidate target device, the specific device can communicate to other devices that it is no longer a candidate target device. If only one device is left as a candidate target device once all devices have compared their determined volumes with the loudness threshold, the single remaining device can be designated as the target computing device by the automated assistant. The target computing device can then be tasked with responding to the spoken utterance. However, if more than one candidate target device remains, one or more dynamic procedures can be employed in accordance with any implementation discussed herein to break a "tie" between candidate target devices.

[0062] Figure 2 The diagram shows a system 200 for selecting a target device from a plurality of candidate devices for responding to a spoken utterance in order to avoid inadvertently causing an unintended device to respond to the spoken utterance. The client-side automated assistant 214 can operate as part of an assistant application provided at one or more computing devices (e.g., a stand-alone speaker device, a display device, a personal computer, a portable computing device, a vehicle) such as a client device 202 and / or a remote computing device 208 such as a server device. A user can interact with the client-side automated assistant 214 via an assistant interface 204, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the client-side automated assistant 214 by providing spoken, textual, or graphical input to the assistant interface to cause the client-side automated assistant 214 to perform a function (e.g., provide data, control peripherals, access an agent, initialize an action, stop an action, modify an action, etc.). The client device 202 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures so as to allow a user to control an application 206 of the client device 202 via the touch interface. In some implementations, the client device 202 may lack a display device, thereby providing an audible user interface output without providing a graphical user interface output. In addition, the client device 202 can provide a user interface for receiving oral natural language input from a user, such as a microphone.

[0063] The client device 202 can communicate with the remote computing device 208 via a network 234 such as the Internet. The client device 202 can offload computing tasks to the remote computing device 208 in order to save computing resources at each client device 202. For example, the remote computing device 208 can host a server automated assistant 210, and the client device 202 can transmit input received at one or more assistant interfaces to the remote computing device 208. However, in some implementations, the automated assistant can be hosted at multiple different client devices. In various implementations, all aspects or less than all aspects of the automated assistant can be implemented on multiple different client devices and / or server devices. In some of those implementations, various aspects of the automated assistant are implemented via a local assistant application of the client device 202 and docked with a remote computing device 208 that implements other aspects of the automated assistant via a server automated assistant 210. The remote computing device 208 can optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of the automated assistant are implemented via a client automated assistant 214 of a client device 202, the client automated assistant 214 can be an application that is separate from the operating system of the client automated assistant 214 (e.g., installed "on top of" the operating system) - or can alternatively be implemented directly by the client device 202 (e.g., considered an application of the operating system but integral to the operating system).

[0064] In some implementations, the remote computing device 208 can include an input processing engine 212 that can process audio data received at the assistant interface to identify requests embodied in the audio data. The process for converting audio data to text can be performed by a speech processing module 222 that can use a speech recognition algorithm that can use a neural network, a word2vec algorithm, and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data can be parsed by a data parsing module 216 and made available to the automatic assistant as text data that can be used to generate and / or identify command phrases from users and / or third-party applications. The result data from the data parsing module 216 can be used by a parameter engine 218 to determine the action that the user is requesting to be performed and / or one or more parameters that may still be needed to complete the action. In response to determining that one or more corresponding slots that require one or more parameters to complete a function, the output generation engine 220 of the server automatic assistant 210 can generate a natural language output. The natural language output can be transmitted back to the client device 202 and provided to the user so as to require the user to provide additional parameters. For example, a user can provide a spoken utterance of "Assistant, play a movie," which can correspond to an action of "play media," which may require the user to identify a movie. Parameter engine 218 can determine that the movie was not identified, cause output generation engine 220 to generate a natural language output (e.g., "What movie would you like to play?"), and cause client device 202 to provide the natural language output.

[0065] In some implementations, to avoid inadvertently affecting the wrong client device in response to the assistant interface receiving a spoken utterance, each automated assistant can operate according to a set of one or more dynamic procedures 232. Each dynamic procedure 232 can be associated with one or more contextual features that can be used to identify a user's preference for a particular device when the user provides a spoken utterance within a given context. In some implementations, before the user provides a spoken utterance, each client device 202 can provide status information 240 to the remote computing device 208. The status information 240 can indicate one or more actions being performed by each client device 202, such as, but not limited to, playing media, performing updates, operating in sleep mode, operating in wake mode, charging, downloading data, and / or any other action that can be performed by a computing device. The status information 240 can be generated by an automated assistant that can access application data 242 that identifies one or more actions being performed by an application 206 and / or an automated assistant. In some implementations, the state information 240 can identify actions previously performed at the client device 202 , actions expected to be performed by the client device 202 , and / or actions that have been performed that are currently being performed and / or are expected to be performed by a remote device in communication with the client device 202 .

[0066] The remote computing device 208 can pre-process any multi-device state information 224 (i.e., state information corresponding to multiple client devices) before one or more client devices receive the spoken utterance. In this manner, the remote computing device 208 can delegate any actions corresponding to the spoken utterance with minimal or no delay. In some implementations, based at least on the processing of the multi-device state information 224, a particular client device 202 can be delegated to perform a particular action even before the spoken utterance has been fully received by the receiving client device.

[0067] In some implementations, when a spoken utterance is received at a client device 202, at least a portion of the spoken utterance can wake up the client device 202 (e.g., "Assistant, ...") and the remainder can be converted to audio data and transmitted to a remote computing device 208 for further processing. The input processing engine 212 can use the audio data to determine whether the spoken utterance is an ambiguous query or a specific query. An ambiguous query can be a query that does not specify a specific device and / or does not specify a specific action to be initialized, modified, and / or interrupted. For example, the spoken utterance "Assistant, stop" can be considered an ambiguous spoken utterance because it does not specify a specific device or a specific action to be modified, such as "play music". A specific query is a spoken utterance that does identify a device in which a request embodied by the spoken utterance should be fulfilled. When the spoken utterance is classified as a specific request, the device identified by the spoken utterance can be assigned the task of fulfilling the specific request. However, when the spoken utterance is classified as an ambiguous request, the dynamic procedure 232 and / or the multi-device status information 224 can be used to determine the target client device that the user intends to call.

[0068] For example, one of the dynamic regulations 232 can indicate an operation to be performed when a particular user provides an ambiguous spoken utterance of "Assistant, stop". Specifically, the dynamic regulations can indicate that when the multi-device status information 224 indicates that the media is playing from the TV and the alarm is sounding at a separate assistant device, the ambiguous spoken utterance should stop the alarm. Additionally or alternatively, another dynamic regulation in the dynamic regulations 232 can indicate what to do when a child's voice signature is detected when the ambiguous spoken utterance of "Assistant, play a movie" is provided. Specifically, another dynamic regulation can indicate that when the multi-device status information 224 indicates that when the movie is already playing at the living room TV and the child is requesting to play the movie, the ambiguous spoken utterance should cause the movie to start playing in the bedroom (e.g., the child's bedroom). In this way, an ambiguous request from a child does not interrupt the movie that is already playing in the living room. This can preserve computing resources at the living room TV because the living room TV does not inadvertently lose any cached data dedicated to the current movie being played, or consume network bandwidth that inadvertently downloads the movie requested by the child.

[0069] Dynamic procedures 232 can be generated at the remote computing device 208 and / or each client device 202 that communicates with the remote computing device 208. The dynamic procedures 232 can be considered "dynamic" because they can be modified and / or newly generated according to the preferences and / or habits of each user who engages with the automated assistant. For example, in response to the automated assistant causing an unintended action to be performed in response to an initial spoken utterance, the user's preference can be determined when the user provides a corrective spoken utterance. The initial spoken utterance can be, for example, "Assistant, turn down," which can be provided by the user while the user is participating in a telephone conversation on their cell phone while listening to music on their assistant device. The unintended action can be that the automated assistant causes the audio from the phone call to be lowered in volume.

[0070] When the user provides a corrective spoken utterance such as “No, turn down the music,” the automated assistant can return the audio from the phone call to its original settings and turn the volume of the music down. Additionally, in response to receiving the spoken utterance, the automated assistant can cause a new or modified dynamic procedure to be generated and accessible to the automated assistant. The dynamic procedure can indicate that when the user is participating in a phone call on their cell phone while music is playing from the assistant device and the user provides the request “turn down,” the music should be turned down instead of the phone call. This dynamic procedure can depend on current status information 240 at the cell phone and at the assistant device, which would indicate that a phone call is occurring simultaneously and music is playing. Thus, if the user provides the spoken utterance “Assistant, turn down” when those particular ongoing actions are not occurring simultaneously (thereby indicating a different context), the dynamic procedure 232 may affect the automated assistant to do something different.

[0071] In some implementations, the dynamic protocol 232 can be embodied as one or more tables that can be accessed by the automated assistant. The table can include columns that identify requests and / or actions and rows that identify operating conditions of specific devices. The values ​​corresponding to the respective rows and respective columns can indicate a score that can be used to identify the device that the request should affect. For example, a request to "stop" can be identified in one row, the first column can identify that an alarm is being issued at the assistant device, and the second column can indicate that a movie is playing at the television. Depending on the particular user to which the table corresponds, the table can provide a higher score or relatively more valuable metric in the first column than in the second column, thereby causing the assistant device to be affected rather than the television when the user provides a "stop" request in the corresponding context.

[0072] In some implementations, one or more dynamic procedures can be based on one or more machine learning models. The machine learning model can be operable to receive as input context data associated with a particular spoken utterance and / or one or more client devices, and provide as output an indication of a device and / or action that should be implemented in response to the spoken utterance. The machine learning model can be trained using training data representing previous instances in which a user provided a corrective spoken utterance. Additionally or alternatively, the machine learning model can be trained using training data representing previous instances in which an action was performed in a given context without a subsequently received corrective spoken utterance.

[0073] Figure 3A and Figure 3B A system diagram illustrating a method for selecting a server that will determine a target client device that will respond to a particular spoken utterance. Specifically, Figure 3A A system 300 is illustrated for adjudicating a decision for selecting a particular local device to respond to a particular spoken utterance provided by one or more users and acknowledged by one or more local devices. Prior to acknowledging the spoken utterance, and / or in response to receiving the spoken utterance, each of a first local device 336, a second local device 338, and an Nth local device 340 can provide status information to a corresponding server device. Status 320 can be provided to a first server device 302, status 322 can be provided to a second server device 304, and status 324 can be provided to an Nth server device 306, where "N" is any positive integer. Each server device can store status information 314, status information 316, and status information 318, respectively, and use the stored status information to designate a particular local device for responding to a particular spoken utterance.

[0074] In some implementations, each of the local devices can be located around a location (e.g., a user's home) and one or more of the local devices can be associated with one or more accounts that are different from the other local devices. As a result, the local devices within the location can communicate with different server devices. The server device can arbitrate the selection of a particular local device to respond to a particular spoken utterance within a given locality and / or based on conditions provided to the server device. In this way, the delay between providing a spoken utterance and receiving a response from a particular local device can be mitigated.

[0075] Figure 3BThe system diagram 326 that illustrates each server device sharing status information related to a specific local device that provides status information. Specifically, the first server device 302 can share status information about the first local device 336 with the second server device 304 and the Nth server device 306. The second server device 304 can share status information with the first server device 302 and the Nth server device 306, and the Nth server device 306 can share status information with the first server device 302 and the second server device 304. The server devices can know that they communicate with local devices in the same location and / or are connected to a public network, so that each server device can provide status information to other server devices. In some implementations, the client device can send status information about another client device connected to the client device. In this way, the client device can transmit status information of other devices that may not be equipped to send status information to the server device through the network but can be connected to the client device and / or share data with the client device in other ways. Each server device can then predict specific spoken words and / or requests of specific users that may be received while all local devices are operating according to their status information. For example, each server device can predict that user 334 may provide a spoken utterance such as “Assistant, pause” while a movie is playing at first local device 336 and a phone call is currently in progress at second local device 338. Given the aforementioned context, each server device can also determine which device user 334 intended to affect when user 334 provided the spoken utterance “Assistant, pause”.

[0076] Each server device can include a corresponding device selection engine 308, 310 or 312 for determining which device the user intends to affect based on the received status information 240. In some implementations, the device selection engine can determine the target local device based on the following: the user's location relative to one or more local devices, the user's voice characteristics, occupancy data associated with the location, the operating status of one or more local devices within the location, the proximity of one or more trusted devices (e.g., watches, mobile phones, etc.) to a specific local device, the operating status of one or more applications operating at the local device, and / or any other information that can be associated with the local device. If one or more of the server devices determines that the first local device 336 should respond to a spoken utterance, the first server device 302 that receives the status 320 from the first local device 336 can be assigned to handle any spoken utterance received during the current scene. In some implementations, the first server device 302 can be assigned to handle only ambiguous requests, and any specific request can be processed in whichever server device corresponds to the local device specified by the user 334.

[0077] For example, when a user provides the spoken utterance "Assistant, pause" while a movie is playing at the first local device 336 and a phone call is ongoing at the second local device 338, the spoken utterance can be confirmed by multiple local devices. However, because the server device has already arbitrated the decision of which local device and / or which server device will handle the ambiguous request, the spoken utterance will ultimately be processed by the first server device 302, and the first local device 336 will provide a response to the spoken utterance (e.g., "Ok, I will pause the movie.").

[0078] In some implementations, a tie may occur when a particular server device is trying to narrow down the selection of local devices for responding to a spoken utterance. Ties can be resolved based on device names established by a user for each of the local devices and / or one or more remaining devices to be selected from. For example, if there is a tie between two local devices for responding to a particular spoken utterance, and one of the two local devices has a user-created name that includes the word "favorite", then the one local device can be selected over the other local devices for responding to the particular spoken utterance. Additionally or alternatively, ties can be resolved based on user habits that indicate that the user has historically preferred one of the local devices involved in a tie for executing a request corresponding to the tie. Therefore, this identified preference can favor the preferred device to "tilt" the balance of the tie. Additionally or alternatively, ties between multiple local devices can be resolved by comparing the features of multiple local devices to identify the local device that is best suited for or equipped to respond to the spoken utterance that caused the tie. For example, if the user has requested to play music (e.g., "Assistant, play music."), and the server device has narrowed down the selection of local devices to play music to two independent speaker devices, the server device can select the local device that is best suited for playing the music. Specifically, the server device can select the local device that has the most speakers, is located in a part of the user's home with better acoustics, and / or has more network and / or processing bandwidth available for streaming the music.

[0079] Additionally or alternatively, the tie between two local devices can be based on the loudness of the spoken words as determined by each of the two local devices. Therefore, if one of the two local devices hears the user at a louder volume than the other local device in the two local devices, the local device that hears the user loudest can be selected. It should be noted that any information for resolving the tie can be provided to the server device to resolve the tie. For example, in response to multiple local devices confirming the spoken words, each of the local devices can determine that the local device heard the loudness measurement result of the user (e.g., in decibels) and provide the loudness measurement result to the server device for the purpose of allowing the server device to resolve the potential tie. Similarly, in the implementation of the local execution of the ruling or selection of the local device for responding to the spoken words, the information for resolving the tie can be shared by each local device with other local devices or at least those local devices that are the subject of the tie. In some implementations, depending on where the tie is being resolved, a deliberate delay can be programmed into the process for resolving the tie to ensure that all tie-breaking information has been received at one or more server devices or one or more local devices.

[0080] Figure 4A and Figure 4B The diagram illustrates implementations that allow for locally performing decisions on the selection of a client device to respond to a spoken utterance from a user. In some implementations, the decision on the selection of a target client device can be based on the location of the user relative to other client devices within a particular location 438. Additionally or alternatively, the decision on the selection of a target client device can be based on the user providing the spoken utterance. The decision can be made at least in part before the spoken utterance is received by one or more client devices in order to mitigate the amount of latency that might otherwise be present between when the user completes providing the spoken utterance and when a particular client device responds to the spoken utterance.

[0081] For example, before user 416 provides a spoken utterance, each client device within location 438 can share its corresponding state information 440 (and optionally state information corresponding to one or more devices connected to the corresponding client device) with other client devices in location 438. State information 440 can characterize the operating state of one or more devices and / or one or more applications. For example, state information 440 from vehicle 414 can characterize operating state 412, which can include "charging vehicle" and being "parked." Additionally, state information 440 from assistant device 418 can characterize operating state 426, which can include being held "idle" by user 416 and tracking "alert settings." Additionally, assistant device 442 can provide state information 440 characterizing operating state 426 of being held "idle" and tracking "alarm settings" for another user, and television 428 can provide state information 440 characterizing operating state 436 of being held "idle" and providing "screen saver."

[0082] Each client device (i.e., vehicle 414, assistant device 418, assistant device 442, television 428) can determine a score or metric based on any received situation information 440 to determine the selection of a target device when the spoken utterance is received by one or more of the client devices. Additionally or alternatively, contextual data can be generated individually by each client device to characterize the relative context in which the spoken utterance is provided. For example, in response to at least one client device receiving the spoken utterance, the client device can transmit the user's provided spoken utterance to other client devices via a local area network (e.g., a Wi-Fi network). Each of the other client devices can determine the volume (e.g., in decibels) at which each client device heard the spoken utterance. Each individual volume can be used in conjunction with the situation information to determine the corresponding score of each client device, or the ranking of each client device relative to the ranking of multiple client devices. Each score or ranking determined by each client device can be transmitted to all other client devices on the network so that each client device can determine whether their score corresponds to the highest score. The client device corresponding to the highest score can then be designated as the "target" client device and respond to the spoken utterance accordingly. In some implementations, the target client device can transmit data to other non-target client devices indicating that it is a designated target client device over the other non-target client devices. In response to each client device determining that they are not the designated target client device, each client device can exit the wake-up mode, suspend the activity of its microphone, and / or otherwise reduce the amount of sampling or processing that has been increased in response to receiving the spoken utterance.

[0083] In some implementations, if the client device receives data corresponding to a specific spoken utterance before the client device receives the spoken utterance, the client device can designate itself as not being targeted, and optionally delete the received data. Additionally or alternatively, if the data corresponding to a specific spoken utterance is received within or outside the client device within a threshold time period (e.g., 100ms, 1s, and / or any other suitable time period) of the spoken utterance reaching the client device, the client device can designate itself as a candidate target client device or designate itself as not being targeted. In this way, each device can exit from being considered as a candidate target device in order to retain computing resources and network bandwidth, which would otherwise be consumed to adjudicate the selection of a target client device among a larger pool of client devices. If a specific device is still a candidate target device after receiving the data, the specific device can determine its own score, compare the score with other scores received from other client devices, optionally determine whether a tie has occurred and identify information for resolving the tie, and then exit or designate itself as a target computing device. In some implementations, the transmitted "data" can include scores, contextual data associated with the sending client device, the sending client device's draw relative to other client devices, and / or any other information that can be used in adjudicating the selection of a client device.

[0084] In some implementations, if a particular client device (e.g., assistant device 418) does not receive data about the spoken utterance from other client devices (i.e., vehicle 414, assistant device 418, assistant device 442, television 428) when awakened in response to receiving the spoken utterance, the particular client device can designate itself as a target client device for responding to the spoken utterance. Optionally, the particular client device can also send data representing tie-breaking information that can be used by another client device to the other client devices if one of the other client devices also designates itself as a target client device to create a tie. In some implementations, the client device can send data corresponding to contextual data based on characteristics and / or operating parameters of a remote client device in communication with the client device to the other client devices. When the client device determines that the score or ranking of the remote client device is distinguishable from other scores or other rankings of other client devices, the client device can designate the remote client device as a target client device. In other words, the client device can determine the scores or rankings of itself and any other devices in communication with the client device, even if the other devices are not in communication with any other client devices that are candidates for responding to the spoken utterance. It should be noted that the remote client device may be a device connected to the client device, such as a Bluetooth enabled device, a display panel, a standalone speaker device, a peripheral device, and / or any other device capable of connecting to a computer (eg, client device).

[0085] In some implementations, the status information 440 provided from each client device can include information characterizing the distance of the user from the corresponding client device. The distance can be determined using one or more sensors accessible to each client device with permission from the user 416. Additionally or alternatively, the distance can be determined using voice characteristics of the spoken utterance compared to a stored topology indicating the proximity of the room and / or the location of the device within the location 438. In some implementations, the distance can be based on the determined location of a trusted device, which is typically carried by the user 416 and can therefore be "commissioned" to provide the location of the user 416 with permission from the user 416. For example, the television automated assistant 430 can determine that the smartwatch worn by the user 416 is 10 feet away and embodies the distance measurement in the status information 440 provided to the other client devices. In some implementations, each device can determine their respective scores based at least in part on a comparison of their respective distances from the user relative to other distances of other client devices relative to the user 416. In this manner, a respective client device located outside or inside a threshold distance from one or more of the client devices and / or the user 416 can designate itself as not targeted or targeted, respectively.

[0086] In some implementations, the decision on the selection of a target client device for responding to a particular spoken utterance can be independent of the request being made and / or the action the user intends the client device to perform. Additionally or alternatively, the decision on the selection of a target client device can be volume independent and thus can be based on information that does not include the volume at which the client device heard the user. Additionally or alternatively, the decision on the selection of a target client device can be based on at least one volume independent characteristic of the volume at which at least one client device heard the user and the context in which the spoken utterance was provided. In some implementations, in order to ensure that the client device responds only to the user and not another client device, the input processing engine of the client device can be equipped with a voice recognition feature that allows the client device to not react when other clients are responding to the spoken utterance.

[0087] In some implementations, when each client device has shared its respective status information 440 (as illustrated in diagram 400), the user 416 can then provide a spoken utterance 450 (as illustrated in diagram 402). The arbitration of the selection of the target client device that will respond to the spoken utterance can be performed before, during, or after receiving the spoken utterance. For example, the arbitration can result in the assistant device 442 having the lowest score due to the time of day (e.g., past 10PM) and the operating status 426 of the assistant device 442 indicating that the assistant device 442 is idle and has an alarm set. Therefore, the assistant device 442 can assume that because someone may be sleeping in the room where the assistant device 442 is located, the assistant device 442 should not have the highest score to respond to any utterance. For example, if the user 416 provides the spoken utterance 450 of "Assistant, play music", then as a result of the status information shared between the local client devices, the assistant device 442 will be ranked last according to any local arbitration.

[0088] In some implementations, one or more machine learning models can be employed by one or more of the client devices to determine the score of each corresponding client device, and / or otherwise determine the target client device that will respond to the spoken utterance. The machine learning model used by the client device to determine the score can use previous instances of the situation information 440 provided by one or more other client devices as training data. The situation information 440 provided by the other client devices can be used as an input to the machine learning model and / or a function generated by the machine learning model after a certain amount of training to identify the target client device and / or the score of the corresponding client device. Other contextual data can also be provided as input to the machine learning model, such as, but not limited to, voice characteristics of one or more users, occupancy data associated with the location 438, room adjacency data corresponding to the location 438, the current time, the current date, previous actions performed by one or more client devices, one or more actions expected to be performed at one or more client devices, a stored calendar accessible by the client device, the location of other remote devices relative to the client device, and / or any other information that may be suitable for providing as input to the machine learning model.

[0089] Figure 5A and Figure 5B Methods 500 and 512 are illustrated for modifying ongoing actions being performed by client devices in response to spoken utterances received at multiple client devices. Methods 500 and 512 can be performed by one or more computing devices, server devices, applications, and / or any other means or modules capable of interacting with an automated assistant. Method 500 can be performed via each of the plurality of client devices. Figure 5A and Figure 5B The circled sequential elements "A" and "B" in the figure continue to method 512. Method 500 can include an operation 502 of receiving audio data that captures a spoken utterance from a user and embodies a request for an automated assistant to perform an action. For example, the spoken utterance can be "Assistant, stop," and the spoken utterance can be heard by multiple client devices within the user's home. The user may intend the spoken utterance to affect only a single action, so a process must be used to identify the specific client device and / or action that the user intends to affect. The process can be performed at one or more server devices and / or one or more client devices.

[0090] The method 500 can further include an operation 504 of determining that a modification requested by a user via a spoken utterance can modify an ongoing action among multiple ongoing actions being performed by multiple client devices. Specifically, the audio data can be converted into text and then the text can be parsed to identify a portion of the text corresponding to the request. For example, the invocation phrase portion of the spoken utterance can be "Assistant" and the request portion of the spoken utterance can be "stop". The invocation phrase portion can cause multiple client devices to modify the operating state of their corresponding microphones to be in an awake state in order to capture audio data corresponding to the remainder of the spoken utterance. When the request portion of the spoken utterance has been identified, a determination can be made as to whether the request corresponds to a request to modify the ongoing action.

[0091] Method 500 can further include an operation 506 of determining whether the request specifically specifies a device to perform the request and / or the action. In other words, operation 506 can include determining whether the request is specific or ambiguous with respect to the action to be modified or the device to perform the modification on the action. If the request is specific with respect to the action to be modified and / or the device to perform the modification on the action, method 500 can proceed from method 500 at operation 506 to method 512 at operation 510 via continuation element "A". If the request is ambiguous with respect to the action to be modified and / or the device to perform the action, method 500 can proceed to operation 508.

[0092] Operation 508 can include identifying a stored procedure corresponding to a modification of an ongoing action. The stored procedure can be identified based at least on text parsed from audio data to identify a request. One or more procedures can be associated with a request, and each procedure can be generated based on learned preferences of one or more users. For example, the identified procedure can identify the aforementioned request "stop" and indicate one or more actions that should be affected by the request depending on available contextual data. For example, the contextual data can indicate operating conditions of one or more devices within a location where the user provided the request "stop", and the procedure can indicate an automated assistant that can cause a particular operation to be modified based on those specific operating conditions.

[0093] Method 500 may proceed from operation 508 to operation 514, as indicated by the continuation element "B". Operation 514 can include accessing contextual data associated with the stored protocol. The contextual data can identify and / or characterize: the type of device in the home that provided the spoken utterance, the characteristics and / or functional capabilities of the device in the home, the device that is temporarily present in the home or otherwise tends to follow the user (e.g., a watch, a cell phone, etc.), the operating state of one or more devices in the home or other devices connected to the device in the home, the media being accessed by one or more devices, the action being performed by one or more devices (e.g., a phone call, a timer countdown), the future operating conditions of one or more devices, the expected or predicted operating state of one or more devices, how long one or more devices have been in each operating state in their respective operating conditions, how long one or more devices are expected or predicted to be in each operating state in their respective operating states, and / or any other property or feature that can be associated with the device or application. For example, the contextual data can indicate that the user's home includes an assistant device that is sounding an alarm, a television that is playing a movie, and a cell phone that is receiving a phone call.

[0094] The method 512 can further include an operation 516 of determining a target computing device for performing an action via the automated assistant based on the contextual data and the stored procedures. For example, the stored procedures can indicate that when a user provides an ambiguous verbal utterance including a "stop" request and any device is sounding an alarm, the alarm action should be stopped at the active device. In this way, regardless of whether a non-alarm sounding device hears the ambiguous request from the user louder than any other device in the home, the ambiguous request will only affect the alarm sounding device. In this way, this corresponding stored procedure can include at least one volume-independent condition for modifying an ongoing action.

[0095] Method 512 can also include an operation 518 of causing a target computing device to perform an action (or modify an ongoing action) via the automated assistant (or not via the automated assistant). For example, in response to comparing the context data with the identified stored procedures, the automated assistant can determine that an alert being issued by one or more devices is a target action to be modified based on a request from a user. The automated assistant can then identify one or more devices as targets based on which devices are issuing alerts. If there is only a single alert issued at a single device, the automated assistant can designate the single device as the target device and cause the alert to be stopped based on a request from the user.

[0096] Figure 6The method 600 for selecting a co-located server device cluster for processing status information from a plurality of client devices and selecting a target client device for performing an action based on the results of processing the status information is illustrated. The method 600 can be performed by one or more computing devices, server devices, applications, and / or any other means or modules capable of interacting with an automated assistant.

[0097] Method 600 can include an operation 602 of receiving status information from a plurality of client devices associated with a user. The status information can indicate that at least one client device is performing an ongoing action. The plurality of client devices can be placed around a location such as a user's home, and can include devices connected via a local area network and / or one or more other connections, such as via near field communication. The status information can be provided periodically by the plurality of client devices so that the status information can be more easily processed by any receiving server device, as opposed to waiting for a user to make a request before sending the status information.

[0098] The method 600 can further include an operation 604 of selecting a designated co-located server device cluster for processing the status information. The designated co-located server device cluster can be selected from a plurality of different co-located server device clusters located in a plurality of different geographic regions, respectively. In some implementations, the designated co-located server device cluster can be selected based on the status information indicating that one or more user accounts associated with one or more user devices are predicted to be subject to requests from users. The selection of the co-located server device cluster can be based on processing the status information using a machine learning model and / or otherwise using the status information as an input and providing a selection of the co-located server device cluster as an output.

[0099] Method 600 can also include an operation 606 of determining that a spoken utterance from a user is received at an automated assistant interface of a client device among a plurality of client devices. The user may be requesting to initialize, modify, stop a particular action, and / or requesting that specific information be provided to the user. However, the request can be ambiguous as to the device from which the user intends to provide a response. For example, the user may have provided a request for information from the automated assistant, and a spoken utterance including the request can be received at a plurality of client devices. Therefore, the automated assistant should determine an appropriate device from which to respond to the user.

[0100] Method 600 can further include an operation 608 of providing audio data to a designated co-located server device cluster in response to determining that a spoken word is received. The audio data can be provided by at least one of the multiple client devices that confirms and / or hears the spoken word from the user. In some implementations, multiple client devices can perform local arbitration to select a client device that will be designated to provide audio data to a designated co-located server device cluster. Arbitration can include, for example, making each client device in the multiple client devices share loudness metrics with each other. The loudness metric can characterize the loudness of the spoken word heard by a specific client device. Determine that the device that they heard the spoken word loudest compared to other client devices in the multiple client devices can then transmit the received audio data to the designated co-located server device cluster. In some implementations, it can be specified to determine that it is closest to the user relative to other client devices to transmit audio data to the co-located server device cluster. In other implementations, it can be determined based on the context data and / or one or more dynamic procedures discussed herein that the client device designated to provide audio data to the co-located server device cluster. In some implementations, the client device designated to provide audio data to the co-located server device cluster can be determined based on which client device first acknowledged receipt of the spoken utterance. The determination can be made based on a timestamp transmitted between the client devices indicating the time at which each device heard the spoken utterance, so each of the client devices can individually determine whether they heard the spoken utterance first.

[0101] Method 600 can also include an operation 610 of enabling a designated co-located server device cluster to identify a designated client device among a plurality of client devices for performing an action. The designated client device for performing an action can be identified by the designated co-located server device cluster. In some implementations, the designated co-located server device cluster can select a designated client device for performing an action based on processing context data, the context data including, but not limited to, status information and dynamic procedures accessible to the designated co-located server device cluster. In this way, instead of multiple co-located server device clusters processing the same information, thereby wasting computing and network resources, a single co-located server device cluster can be designated for processing such information. In addition, this can reduce latency that might otherwise occur when a user provides a spoken utterance that is then received by multiple assistant-enabled devices that may be associated with different primary accounts and therefore communicate with different servers.

[0102] The method 600 can further include an operation 612 of causing the designated client device to perform an action requested by the user. Based on the output of the designated co-located server device cluster directed to the designated client device and / or other client devices co-located with the designated client device, the designated client device can be caused to perform the action. In some implementations, if the designated co-located server device cluster identifies a tie between two or more candidate client devices to perform an action, the designated co-located server device cluster can access and / or receive information related to resolving the tie as discussed herein.

[0103] Figure 7 is a block diagram of an example computer system 710. The computer system 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 including, for example, a memory 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow a user to interact with the computer system 710. The network interface subsystem 616 provides an interface to an external network and couples to corresponding interface devices in other computer systems.

[0104] The user interface input device 722 may include a keyboard, a pointing device such as a mouse, trackball, touch pad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 710 or onto a communication network.

[0105] The user interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 710 to a user or another machine or computer system.

[0106] The storage subsystem 724 stores program and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 724 may include logic for performing selected aspects of the methods 500, 512, 600, and / or implementing one or more of the following: an automated assistant, an assistant device 104, an assistant device 118, a television 128, a vehicle automated assistant 106, a client automated assistant 120, a television automated assistant 130, a client device 202, a remote computing device 208, a server device, a local device, a client device, and / or any other apparatus or module discussed herein.

[0107] These software modules are typically executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 can include many memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 in which fixed instructions are stored. The file storage subsystem 726 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored in the storage subsystem 724 by the file storage subsystem 726, or in other machines accessible by the processor 714.

[0108] The bus subsystem 712 provides a mechanism for the various components and subsystems of the computer system 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0109] Computer system 710 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 7 The description of the computer system 710 depicted in FIG. 7 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computer system 710 may have more Figure 7 The computer system depicted in FIG.

[0110] In situations where the systems described herein collect personal information about users (or often referred to herein as "participants") or can make use of personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, users may control how information is collected and / or used about them.

[0111] Although several implementations have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which these teachings are used. Those skilled in the art will recognize or be able to use more than routine experiments to ascertain many equivalents of the specific implementations described herein. Therefore, it should be understood that the foregoing implementations are given only as examples, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementations disclosed herein are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure when such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

Claims

1. A method implemented by one or more processors, the method comprising: determining, at a server device in communication with a plurality of client devices, status information corresponding to an activity state of a plurality of client devices disposed about a location of a user, wherein each of the plurality of client devices is accessible to the automated assistant; receiving data based on a spoken utterance provided by the user to at least one of the plurality of client devices that is operating in accordance with the determined condition information, wherein the spoken utterance embodies a request for an action to be performed by one or more of the plurality of client devices; in response to receiving the data based on the spoken utterance, accessing a stored set of dynamic procedures associated with the plurality of client devices, wherein at least one stored procedure in the stored set of dynamic procedures characterizes a contextual dependency of performance of the action by the one or more client devices on the state information; identifying a target client device for performing the action based on at least one stored procedure in the stored set of dynamic procedures and the condition information, wherein the context dependency characterized by the at least one stored procedure includes at least one volume-independent condition for specifying the target client device for performing the action; and The target client device is caused to perform the action via the automated assistant.

2. The method according to claim 1, wherein: The location includes a plurality of rooms characterized by a stored topology accessible by the server device, and the method further includes: Current context data is accessed that characterizes the context dependency as the user being located in a first room including the at least one client device and the target client device being located in a second room of the plurality of rooms.

3. The method according to claim 1, further comprising: Based on receiving the data, determine whether the spoken utterance is at least one type of utterance selected from an ambiguous type of utterance and a specific type of utterance, wherein the following specific operations are performed when the spoken utterance is determined to be an utterance of the ambiguous type: accessing the stored set of dynamic procedures associated with the multiple client devices, identifying a target client device for performing the action, and causing the target computing device to perform the action.

4. The method according to claim 1, further comprising: When the spoken utterance is determined to be of a specific type: identifying a designated client device specifically identified by the user via the spoken utterance; as well as The designated client device is caused to perform the action while bypassing access to the stored set of dynamic procedures associated with the plurality of client devices.

5. The method according to claim 4, wherein: Characterizing the received spoken utterance as at least one type of utterance selected from the ambiguous type of utterance and the particular type of utterance is based on previous interactions between the user and the interactive software application.

6. A method implemented by one or more processors, the method comprising: receiving status information from a plurality of client devices associated with a user, the status information indicating that one or more of the plurality of client devices are performing an ongoing action; selecting, based on receiving the status information, a designated co-located server device cluster for processing the status information provided by the plurality of client devices, wherein the designated co-located server device cluster is selected from a plurality of different co-located server device clusters; determining that a spoken utterance from the user is received at an automated assistant interface of a client device among the plurality of client devices, wherein the automated assistant interface is configured to facilitate interaction between the user and an automated assistant accessible by the plurality of client devices, and the spoken utterance embodies a request for the automated assistant to perform an action; responsive to determining that the spoken utterance was received from the user at the automated assistant interface of the client device, causing audio data generated at the client device to be provided to the designated cluster of co-located server devices, wherein the audio data captures at least a portion of the spoken utterance; Based on the client device providing the audio data to the designated cluster of co-located server devices, the designated cluster of co-located server devices identifies a designated client device among the plurality of client devices for performing the action requested by the user; and The designated client device is caused to perform the action requested by the user.

7. The method according to claim 6, further comprising: accessing a stored device topology in response to receiving the status information, the stored device topology indicating an arrangement of various client devices within a location, wherein the plurality of client devices is a subset of the various client devices, and It is determined that the plurality of client devices are in a common room at the location where the user provided the spoken utterance to the automated assistant interface.

8. The method according to claim 7, wherein: The action corresponds to a specific output modality, and the method further comprises: Determining that the plurality of client devices in the common room at the location include the particular output modality corresponding to the action identified by the request.

9. The method according to claim 6, wherein: Two or more of the plurality of client devices are associated with different user accounts, and the two or more client devices are configured to communicate with the plurality of different co-located server device clusters based on the two or more client devices being associated with the different user accounts.

10. The method according to claim 6, wherein: The specific status information provided by the corresponding client device indicates whether the corresponding client device among the multiple client devices is performing an ongoing action, and when the corresponding client device is performing the ongoing action, the status information indicates the modality adopted by the corresponding client device to perform the ongoing action.

11. The method according to claim 10, wherein: The designated client device is selected based on stored training data indicating a correspondence between the action identified by the request and the context information provided by the plurality of client devices.

12. A device comprising: an audio interface configured to generate a signal in response to receiving a spoken utterance, the spoken utterance being received to facilitate communication between a user and the automated assistant, wherein the spoken utterance is received at both the audio interface and another audio interface of another client device, and the other client device is a client device that is one of a plurality of client devices accessible by the automated assistant; a network interface configured to receive a communication from the other device that also received the spoken utterance, wherein the communication provided by the other device includes first context data characterizing a context associated with the other device; a memory configured to store second context data representative of another context associated with the spoken utterance being received at the audio interface; and One or more processors configured to perform operations including: determining a ranking of the plurality of client devices, wherein the ranking comprises at least a first ranking based on the first contextual data and a second ranking based on the second contextual data, and wherein each ranking in the ranking at least partially influences selection of a particular device of the plurality of client devices to respond to the spoken utterance via the automated assistant, and When a candidate device of the plurality of client devices has a corresponding ranking that is distinguishable from other rankings of the ranking: The automated assistant is caused to respond to the spoken utterance via the candidate device based at least on the corresponding ranking of the ranks.

13. The device according to claim 12, wherein: The network interface is further configured to receive the communication from the other device before the spoken utterance is received at the audio interface.

14. The device according to claim 13, wherein: Determining the ranking by the one or more processors is performed before the spoken utterance is received at the audio interface or while the spoken utterance is being received at the audio interface.

15. The apparatus according to claim 12, wherein: The one or more processors are further configured to perform operations including: generating the second context data before the spoken utterance is received at the audio interface, and The network interface is caused to provide a separate communication to the other device, wherein the other device is configured to determine whether the other device is tasked with responding to the spoken utterance via the automated assistant.

16. The apparatus according to claim 12, wherein: The one or more processors are further configured to perform operations including: sending a separate communication to a separate device that is in communication with the network interface but has not acknowledged the spoken utterance; as well as A determination is made as to whether the individual device is to be tasked with responding to the spoken utterance via the automated assistant.

17. The apparatus according to claim 12, wherein: The one or more processors are further configured to perform operations including: Third context data characterizing an operating condition of the separate device is generated.

18. The apparatus according to claim 12, wherein: The one or more processors are further configured to perform operations including: When two or more client devices of the plurality of client devices have respective rankings that result in a tie ranking between the two or more client devices: determining supplemental contextual data characterizing the two or more client devices, and The tied ranking is caused to be resolved in a manner that favors a respective client device having a ranking that is distinguishable from rankings of other client devices in the plurality of client devices based on the supplemental contextual data.

19. A method implemented by one or more processors, the method comprising: determining that a spoken utterance has been received at a first client device, wherein the spoken utterance is provided by a user to facilitate communication with an automated assistant accessible via an assistant interface of the first client device and is received by a plurality of client devices including the first client device; receiving a communication from a second client device that received the spoken utterance in response to the second client device receiving the spoken utterance, wherein the communication provided by the second client device includes first context data characterizing a context associated with the second client device; determining further context data characterizing another context associated with the first client device; determining a ranking of the plurality of client devices, wherein the ranking comprises at least a first ranking based on the first contextual data and a second ranking based on the second contextual data, and wherein each ranking in the ranking at least partially influences selection of a particular client device from the plurality of client devices to respond to the spoken utterance, and When a candidate client device among the plurality of client devices has a corresponding ranking that is distinguishable from other rankings of the ranking: The automated assistant is caused to respond to the spoken utterance via the candidate device based at least on the corresponding ranking of the ranks.

20. The method according to claim 19, wherein: The communication is received from the second client device before the spoken utterance is received by the first client device.

21. The method according to claim 20, wherein: Determining the ranking is performed before the spoken utterance is received at the first client device or while the spoken utterance is being received at the first client device.

22. The method of claim 19, further comprising: generating the second context data before the spoken utterance is received at the audio interface; and The network interface is caused to provide a separate communication to the other device, wherein the other device is configured to determine whether the other device is to be tasked with responding to the spoken utterance via the automated assistant.

23. The method of claim 19, further comprising: sending a separate communication to a separate device that is in communication with the first client device but has not acknowledged the spoken utterance, and A determination is made as to whether the individual device is to be tasked with responding to the spoken utterance via the automated assistant.

24. The method of claim 19, further comprising: Third context data characterizing an operating condition of the separate device is generated.

25. The method of claim 19, further comprising: When two or more client devices of the plurality of client devices have respective rankings that result in a tie ranking between the two or more client devices: determining supplemental contextual data characterizing the two or more client devices, and The tied ranking is resolved in a manner that favors a respective client device having a ranking that is distinguishable from a ranking of other client devices in the plurality of client devices based on the supplemental contextual data.

Images