It is recommended to include auto attendant actions in auto attendant routines

Abstract

An automated assistant action is recommended for inclusion in an existing automated assistant routine of a user, wherein the existing automated assistant routine includes a plurality of pre-existing automated assistant actions. If the user confirms the recommendation via affirmative user interface input, the automated assistant action may be automatically added to the existing automated assistant routine. Thereafter, when the automated assistant routine is initialized, the pre-existing automated assistant actions of the routine are executed, and the automated assistant actions that are automatically added to the routine in response to the affirmative user interface input are received in response to the recommendation.

Description

Background Art

[0001] A human may engage in a human-machine conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands, queries, and / or requests using spoken natural language input (i.e., utterances) that may in some cases be converted to text and then processed and / or by providing textual (e.g., typed) natural language input.

[0002] The automated assistant may perform routines of multiple actions in response to, for example, receiving a specific command (e.g., a shortcut command). For example, in response to receiving the spoken utterance of "Good Night", the automated assistant may cause a series of actions to be performed, such as causing networked lights to be turned off, tomorrow's weather forecast to be rendered, and tomorrow's agenda for the user to be rendered. Automated assistant routines may be specific to the ecosystem of the user and / or client device, and controls may be provided to the user to manually add certain actions to certain routines. For example, a first user may have: a "Good Morning" automated assistant routine with a first Good Morning automated assistant action set defined by the first user, a "good night" automated assistant routine with a first Good Night automated assistant action set defined by the first user, and additional automated assistant routines with additional automated assistant actions defined by the first user. A separate second user may have: a "good morning" automated assistant routine with a different Good Morning automated assistant action set defined by the second user, a "Good Night" routine with a different Good Night automated assistant action set defined by the second user, and the like. Summary of the invention

[0003] As described herein, various automated assistant routines may be initiated in response to detecting a shortcut command in a user's spoken or typed user interface input, initiated in response to a user's interaction with a virtual or hardware element at a client device, initiated in response to detecting a user gesture, and / or initiated in response to (multiple) other abbreviated user interface inputs. The abbreviated user interface inputs used to initiate the automated assistant routines are abbreviated because these abbreviated user interface inputs require less user input and / or less processing of user input than the user input required to cause the action of the automated assistant routine to be performed without the abbreviated user input. For example, a shortcut command for an automated routine that results in a series of actions being performed may be abbreviated because the length of the shortcut command is shorter than the length of the command that needs to be spoken / typed to cause the automated assistant to perform a set of actions without the compressed command. Additionally or alternatively, various automated assistant routines may be automatically initiated upon the occurrence of one or more conditions, and optionally without requiring explicit user interface input. Such automatic initiation of automated assistant routines may also require less user input and / or less processing of user input than the user input required to cause the action of the automated assistant routine to be performed.

[0004] Despite the existence of automated assistant routines, there are still various automated assistant actions that are not incorporated into any automated assistant routine of the user but are performed by the user instead through prolonged interaction with the automated assistant and / or through prolonged interaction with (multiple) other computer applications. Each occurrence of such prolonged interaction may consume a large amount of computer and / or network resources. For example, in order to turn on the networked lights in the kitchen via an automated assistant, the user may be required to speak "turn on the kitchen lights" or similar spoken words to the automated assistant interface of the client device. Audio data corresponding to the spoken words can be transmitted from the client device to the remote system. The remote system can process the audio data (e.g., speech to text processing, natural language processing of text, etc.) to determine the appropriate command for the spoken words, can transmit the command to the agent to turn on the kitchen lights, can receive confirmation from the agent that the kitchen lights have been turned on, and can transmit data to the client device so that the client device renders a notification that the lights have been turned on.

[0005] In view of these and other considerations, embodiments disclosed herein relate to recommending an auto assistant action for inclusion in an existing auto assistant routine of a user, wherein the existing auto assistant routine includes one or more of the pre-existing auto assistant actions. If the user confirms the recommendation via affirmative user interface input, the auto assistant action can be automatically added to the existing auto assistant routine. Thereafter, when the auto assistant routine is initialized, the pre-existing auto assistant actions of the routine will be executed, as well as the auto assistant actions that were automatically added to the routine in response to the affirmative user interface input in response to the recommendation being received.

[0006] This situation eliminates the need for the user to perform other resource-intensive persistent interactions instead to get the action to be executed. Instead, the action is executed as one of the multiple actions of the automated assistant routine to which it is added. This situation can, for example, produce improved human-computer interaction by reducing the number of inputs that need to be provided by the user (e.g., provided by the user to the automated assistant interface) to cause the execution of the action. In addition to improving human-computer interaction, this situation can directly produce various computer and / or network efficiencies relative to, for example, requiring the user to provide another user interface input instead to get the action to be executed. In addition, as described above and elsewhere herein, the initialization of the automated assistant routine can be responsive to abbreviated user interface inputs that can be effectively processed and / or can be automatic after the occurrence of (multiple) conditions that do not require the processing of the user interface input. Further still, recommending automated assistant actions for inclusion in existing routines and automatically adding actions to routines in response to single-touch and / or single-voice affirmative inputs (e.g., "yes, add") provides an effective supplement to existing routines. For example, automatically adding actions in response to simplified user interface inputs can be more computationally efficient than requiring the user to instead engage in a lengthy interaction to manually add actions (such as requiring the user to manually select a routine to be modified, specify an action to be added to the routine, confirm that the action should be added to the routine, etc.).

[0007] Embodiments disclosed herein can determine an automated assistant action for potential recommendation for inclusion in one or more automated assistant routines of a user. In addition, the automated assistant action can be compared to each of a plurality of automated assistant routines stored in association with the user, and based on the comparison (e.g., if one or more criteria are met), a subset of automated assistant routines (e.g., one of the automated assistant routines) can be selected. A graphical and / or audio user interface output can be rendered via the user's client device, wherein the user interface output prompts the user to add the action to the (multiple) selected routines of the subset. If an affirmative user interface input is received in response to the user interface output, the automated assistant action can be automatically (e.g., without requiring another user interface input) added to one of the (multiple) selected routines indicated by the affirmative user interface input. Thereafter, when one of the selected routines in the selected routine is initiated, the automatic execution of the pre-existing automated assistant action of the routine and the automatic execution of the added one of the (multiple) selected actions can be initialized.

[0008] Various techniques can be used to determine an auto-assistant action for potential recommendation for inclusion in one or more auto-assistant routines of a user. In some embodiments, determining an auto-assistant action is based on one or more instances of an auto-assistant action being initiated by a user via a user interface input provided by the user. For example, an "adjust the smartthermostat to 72°" auto-assistant action can be determined based on one or more previous instances of the user initiating an auto-assistant action by speech and / or other user interface input provided to the auto-assistant. In some embodiments, determining an auto-assistant action is based on one or more instances of different but related auto-assistant actions being initiated by a user via a user interface input provided by the user. For example, a "grilling tips" auto-assistant action that renders different tips related to grilling each time it is executed can be determined based on one or more past instances of a user providing user interface input related to grilling (e.g., "how long do I cook chicken on the grill") related to grilling. In some embodiments, determining an auto-assistant action is based on determining that the auto-assistant action is related to one or more non-auto-assistant interactions of the user. For example, an "adjust the smart thermostat to 72°" auto assistant action can be determined based on detecting one or more occurrences of a user manually adjusting the smart thermostat (e.g., through direct interaction with the smart thermostat) or adjusting the smart thermostat via a non-auto assistant application that is involved in controlling the smart thermostat. For example, the detection can be based on a state change indication that is transmitted to the auto assistant and indicates that the smart thermostat is adjusted to 72°. The state change indication can be transmitted to the auto assistant by an agent controlling the smart thermostat and can be pushed or provided in response to a status request from the auto assistant. In some embodiments, determining an auto assistant action is based on determining that the action can improve human-auto assistant interaction without having to determine that the user has performed the auto assistant action and / or related actions.

[0009] Selecting a subset of a user's (multiple) auto assistant routines to which to recommend adding an auto assistant action based on comparing the action to the (multiple) routines may increase the likelihood that the correct (multiple) routines, if any, are recommended for the action, thereby reducing the risk of providing recommendations that will be ignored or cancelled. In addition, rendering a subset of auto assistant routines instead of the entirety of the user's routines may save resources utilized in rendering. In some embodiments, a subset of auto assistant routines is selected based on comparing a temporal attribute of past occurrences of the routine to a temporal attribute of (multiple) past occurrences of the auto assistant action (or of an action related to the auto assistant action). For example, if past occurrences of an auto assistant action for a user occurred only on weekday mornings, the user's routines that are frequently initiated on weekday mornings may be more likely to be selected for the subset than the user's routines that are frequently initiated on weekday evenings or the user's routines that are frequently initiated on weekend mornings.

[0010] In some implementations, a subset of automated assistant routines is selected based on additionally or alternatively comparing device topology attributes of device(s) utilized in the routine to device topology attributes of device(s) utilized in the automated assistant action. For example, if an automated assistant action involves control of a television having an assigned device topology attribute of "location = livingroom", routines that include actions to control other device(s) having an assigned device topology attribute of "location = livingroom" (e.g., living room lamp, living room speaker lamp) may be more likely to be selected for the subset than routines that lack any actions to control devices having the device topology attribute of "location = living room". As another example, if an automated assistant action requires displaying content via a device having an assigned device topology attribute of "capable of display", routines that include actions to control device(s) located in an area having devices with the device topology attribute of "capable of display" may be more likely to be selected for the subset than routines that lack any actions to control devices in an area having devices with the device topology attribute of "capable of display".

[0011] In some implementations, a subset of auto assistant routines is selected based on additionally or alternatively excluding from the subset any routines that include a conflict with an auto assistant action. For example, for an "adjust the smart thermostat to 72°" auto assistant action, a given routine can be excluded based on it having a conflicting "adjust the smart thermostat to 75°" action. Additionally, for example, for an auto assistant action that interfaces with an agent to perform a service that is only available in the evening, a given routine can be excluded based on it automatically occurring in the morning rather than in the evening.

[0012] In some embodiments, multiple competing actions may be considered for addition to an automated assistant routine, and a subset of competing actions (e.g., one of the competing actions in) may be selected for outputting an actual recommendation via a user interface for addition to the routine. In some of those embodiments, each of the multiple competing actions may be compared to the routine, and the selected action may be one of the multiple competing actions that most closely conforms to the routine, lacks any conflict with the routine, and / or satisfies other criteria. In some additional or alternative embodiments, the multiple competing actions may include a first action performed by a first agent and a second action performed by a second agent. For example, a first action may cause a first agent of streaming music to render jazz music, while a second action may cause jazz music to be rendered by a second agent of streaming music. In some of those embodiments, one of the first agent and the second agent may be selected based on one or more measurements associated with the first agent and the second agent. Such measurements may include, for example, frequency of usage of the first agent and / or the second agent by users; frequency of usage of the first agent and / or the second agent by a user group; ratings assigned to the first agent and / or the second agent (by users and / or user groups); and / or data provided by or on behalf of the first agent and / or the second agent, which indicates the ability and / or desirability of the respective agent(s) to perform an action.

[0013] In some implementations, a user interface output prompting a user to add an automated assistant action to at least one selected automated assistant routine may be provided at the completion of execution of a selected routine or at the completion of execution of an automated assistant action. For example, after executing the "Good Morning" routine, the automated assistant may provide a graphical and / or audible prompt "by the way, it looks like Action X would be good to add to this routine, want me to add it", and if an affirmative input is received in response, "Action X" may be added to the "Good Morning" routine. Further, for example, after executing "Action X" in response to a user's user interface input (e.g., a spoken input of "Assistant, perform Action X"), the automated assistant may provide a graphical and / or audible prompt "by the way, it looks like this would begood to add to your Good Morning routine, want me to add it". If affirmative input is received in response, "Action X" may be added to the "Good Morning" routine. In these and other ways, the presented user interface output facilitates human-automatic assistant interaction and, if affirmative input is received in response, reduces the amount of user input required for future execution of both pre-existing actions of the routine and actions added to the routine. For example, "Action X" may control a connected device, and the presented user interface input may facilitate human-automatic assistant interaction by enabling a user to provide affirmative input in response and thereby causing control of the connected device to occur automatically in response to another initiation of the corresponding routine.

[0014] In some embodiments, when an auto assistant action is added to a routine, the execution location of the auto assistant action among the pre-existing actions of the routine can be determined. In some versions of those embodiments, the location can be determined based on the duration required for the auto assistant action of the user interface output (if any) and the (multiple) durations required for the pre-existing actions of the routine of the user interface output (if any). For example, if the auto assistant action is controlling a smart device and no user interface output is required (or a very brief "device controlled" output), the auto assistant action can be positioned so that the auto assistant action is executed before executing one or more actions that require rendering of a more persistent user interface output. In some additional or alternative versions, the execution location can be final and can optionally occur with a time delay. The time delay can be, for example, based on the time delay (multiple) of the past execution of the auto assistant action (in response to the user's (multiple) past inputs) relative to the past execution of the routine. For example, assume that the auto assistant action is starting a brew cycle for a smart coffee machine and is added to the "Good Morning" routine. Assume further that, after completion of the Good Morning Routine, in response to the spoken utterance "Assistant, brew my coffee," the automated assistant action was previously executed three times (before being added to the "Good Morning" Routine). If the three spoken utterances occur two minutes, three minutes, and four minutes after completion of the Good Morning Routine, the automated assistant action can be added to the Good Morning Routine with a time delay, such as a time delay of three minutes (the average of the three delays).

[0015] As described above, some automated assistant routines may be initialized in response to detecting a shortcut phrase or command in a user's spoken or typed natural language input. The shortcut command provides an abbreviated command for causing the automated assistant to optionally perform a set of actions in a specific order. Providing a shortcut command to cause the execution of a set of actions instead of a longer command for the set of actions can enable less user input to be provided (and transmitted and / or processed), thereby saving computing and network resources. Additionally, thereafter, automatically adding the automated assistant action as an additional action of the routine to be executed in response to the existing shortcut command enables the additional action of the routine and the previously existing action to be executed in response to the shortcut command. Therefore, thereafter, the user can speak or type the shortcut command instead of speaking both the command for the additional action and the shortcut command. As an example of an abbreviated command for an automated assistant routine, when the user wakes up in the morning, the user can trigger the "Good Morning" routine by providing a spoken word to a kitchen assistant device (i.e., a client computing device located in the kitchen). The spoken utterance may be, for example, "Good Morning," which may be processed by the assistant device and / or a remote assistant device (in communication with the assistant device) for initializing the "Good Morning" routine. For example, the assistant device and / or the remote device may process audio data corresponding to the spoken utterance to convert the spoken utterance to text, and may also determine that the text "Good Morning" is assigned by the user to the set of automated assistant actions that are performed in response to the spoken utterance of "Good Morning." Adding a new action to the "Good Morning" routine according to embodiments described herein also assigns the new action to the text "GoodMorning" by the user.

[0016] Although various automated assistant routines can be initialized in response to spoken or typed shortcut commands, in some embodiments, automated assistant routines can be initialized in addition or alternatively in response to a user pressing a virtual or hardware element at a client device or peripheral device, performing a gesture detected via a sensor(s) of a client device, providing other tactile input(s) at a client device, and / or providing any other type of computer-readable user interface input. For example, a graphical user interface (GUI) can be presented at a client device with selectable icons that provide the user with suggestions for initializing automated assistant routines. When a user selects a selectable icon (e.g., a GUI button that says "Good Morning"), the automated assistant can initialize the corresponding automated assistant routine in response. Additionally or alternatively, for example, an automated assistant routine can be automatically initiated in response to the automated assistant detecting the presence of a user (e.g., using voice authentication and / or facial recognition to detect a particular user), the location of a user (e.g., using voice authentication and / or facial recognition to detect a particular user in a location such as at home, in a car, in a living room, and / or additional location(s)), disarming an alarm (e.g., a wakeup alarm set on an associated phone or other device), opening an application, and / or other user actions that can be recognized by the automated assistant (e.g., based on signals from one or more client devices). For example, a "Good Morning" routine can be initialized when a user is detected using facial recognition in the kitchen at "the morning" (i.e., at a particular time, during a time range, and / or additional time(s) associated with the routine).

[0017] An example "Good Morning" routine may include actions such as causing a user's schedule to be rendered, causing a particular appliance to be turned on, and causing a podcast to be rendered. Again, by enabling the automated assistant to respond to shortcut commands, the user does not necessarily need to provide a string of commands in order for the automated assistant to perform the corresponding actions (e.g., the user does not need to recite the spoken words: "Assistant, read me my schedule, turn on my appliance, and play my podcast"). Alternatively, the automated assistant may respond to a shortcut command, which the automated assistant may process to identify an action corresponding to the shortcut command. In some embodiments, routines may be personalized so that a particular shortcut command or other input can cause the execution of the routine to cause the automated assistant to perform a particular set of actions for one user, while the same input causes the automated assistant to perform a different set of actions for a different user. For example, a particular user may specifically configure the automated assistant to perform a first set of actions in response to a shortcut command, and a spouse of the particular user may configure the automated assistant to perform a second set of actions in response to the same shortcut command. The automated assistant may distinguish between users who provide shortcut commands using one or more sensor inputs and / or one or more determined characteristics (such as voice signatures, facial recognition, image feeds, motion characteristics, and / or other data). In addition, as described herein, the automated assistant may distinguish between users when determining automated assistant actions to recommend to a given user, determining automated assistant routines specific to the user(s), and determining when to provide user interface output that prompts the automated assistant action to be added to the user's routine (e.g., user-specific user interface output may be provided only when it is determined that the user is interacting with the automated assistant).

[0018] When a user provides a quick command (such as "Good Morning") to an assistant device (such as a kitchen assistant device), content corresponding to one or more actions of the routine may be initially rendered by the kitchen assistant device because the spoken utterance involves the kitchen assistant device. For example, the content may initially be rendered exclusively at the kitchen assistant device (i.e., and not simultaneously rendered at any other client device) despite recognition of the quick command by other devices. For example, multiple devices may recognize receipt of the quick command at their respective automated assistant interfaces, however, the device that receives the quick command loudest and / or least distorted may be designated as the device at which the automated assistant routine will be initialized.

[0019] If the user performs the action in the office via the office assistant device, the automated assistant may still suggest adding the action to the "Good Morning" routine. For example, if the user moves to their office after completing their "Good Morning" routine in the kitchen, and then asks the office assistant device for a spoken utterance about a news headline, the automated assistant may suggest adding an action of rendering the news headline to the "Good Morning" routine performed in the kitchen via the kitchen assistant device. In some embodiments, the office assistant device may suggest adding an action performed in the office (i.e., rendering the news headline) to the "Good Morning" routine even if the action is performed at a different device from the "Good Morning" routine. In some embodiments, when the user is in the office, the office device may be used to suggest adding an action to render the news headline to the "Good Morning" routine after the user asks the office assistant for the news headline. In other embodiments, the kitchen assistant device may suggest that the user add a new action of rendering the news headline to the "Good Morning" routine after the "Good Morning" routine is completed while the user is still near the kitchen device. Additionally or alternatively, in some embodiments, when the user is detected near a kitchen appliance before executing the "Good Morning" routine but after the user requests the routine via a shortcut command for the kitchen appliance, the automated assistant can suggest that the user add actions that are historically performed in the office.

[0020] In some embodiments, restrictions may be imposed on adding actions to the automated assistant routine. For example, a user who lives with a spouse may have a "Good Morning" routine, while his spouse may have his own "Good Morning" routine. The user may not want to add actions to his spouse's Good Morning routine (and vice versa). In some embodiments, the automated assistant may determine whether the user or his spouse is performing an action by sensor input and / or one or more determined characteristics (such as voice signatures, facial recognition, image feeds, motion characteristics, and / or other data). In some embodiments, the default settings on the automated assistant routines in a multi-user household may only allow the user of the routine to modify his personal routine. In some embodiments, the user may give permission to the automated assistant to allow other users (such as his spouse) to add actions to their personal routines. Additionally, in some embodiments, if appropriate consent is obtained, the automated assistant may check actions for other members of the family with similar routines before suggesting routines to add actions. For example, before suggesting adding an action to turn on a device (such as a smart coffee machine) to the user's Good Morning routine, the automated assistant may check the spouse's Good Morning routine and determine whether the action has been performed in the family for the morning. In other words, when determining a routine to which to suggest adding actions, the automated assistant can examine the user's routine and compare it to the routines associated with other members of the household to, for example, prevent duplicate actions (such as two users both turning on the coffee maker in the morning) as part of their "Good Morning" routine.

[0021] In some embodiments, when executing a routine, the automated assistant is docked with one or more local and / or remote agents. For example, for a routine including three actions, the automated assistant may be docked with a first agent when performing a first action, docked with a second agent when performing a second action, and docked with a third agent when performing a third action. As used herein, "agent" refers to one or more computing devices and / or software utilized by the automated assistant. In some cases, the agent may be separated from the automated assistant and / or may communicate with the automated assistant through one or more communication channels. In some of these cases, the automated assistant may transmit data (e.g., an agent command) from a first network node to a second network node that implements all or all aspects of the functionality of the agent. In some cases, the agent may be a third party (3P) agent because it is managed by a party that is separated from the party that manages the automated assistant. In some other cases, the agent may be a first party (1P) agent because it is managed by the same party that manages the automated assistant.

[0022] The agent is configured to receive call requests and / or other agent commands from the automated assistant (e.g., over a network and / or via an API). In response to receiving the agent command, the agent generates response content based on the agent command, and transmits the response content for providing a user interface output based on the response content. For example, the agent can transmit the response content to the automated assistant for the automated assistant to provide an output based on the response content. As another example, the agent itself can provide output. For example, a user can interact with the automated assistant via a client device (e.g., the automated assistant can be implemented on the client device and / or perform network communication with the client device), and the agent can be an application installed on the client device or an application that is executable away from the client device but "streamable" on the client device. When the application is called, it can be executed by the client device and / or brought to the forefront by the client device (e.g., its content can take over the display of the client device).

[0023] The above is provided as an overview of various embodiments disclosed herein. Additional details regarding those various embodiments as well as additional embodiments are provided herein.

[0024] In some embodiments, a method implemented by one or more processors is provided and includes determining an action initiated by an automated assistant. The action is initiated by the automated assistant in response to one or more instances of user interface input provided by a user via one or more automated assistant interfaces interacting with the automated assistant. The method also includes identifying a plurality of automated assistant routines stored in association with the user. Each automated assistant routine in the automated assistant routine defines a plurality of corresponding actions that will be automatically performed via the automated assistant in response to initialization of the automated assistant routine, and the action is an action other than the action of the automated assistant routine. The method also includes comparing the action with a plurality of automated assistant routines stored in association with the user. The method also includes selecting a subset of automated assistant routines stored in association with the user based on the comparison. The method also includes initiating the action in response to user interface input provided by the user and based on selecting a subset of automated assistant routines: causing the user interface output to be rendered via the user's client device. The user interface output prompts the user to add the action to one or more of the automated assistant routines of the subset. The method also includes receiving an affirmative user interface input in response to the user interface output, wherein the affirmative user interface input indicates a given routine in the automated assistant routine of the subset. The method also includes, in response to receiving the affirmative user interface input, automatically adding the action as an additional action in a plurality of corresponding actions of the given routine to be automatically performed in response to initialization of the given routine.

[0025] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0026] In some implementations, the one or more instances of user interface input include one or more spoken utterances that, when converted to text, are first text of a first length. In those implementations, initialization of the given routine occurs in response to a given spoken utterance that, when converted to text, is second text of a second length, and the second length is shorter than the first length.

[0027] In some implementations, causing the user interface output to be rendered via the user's client device is also based on and occurs at the completion of executing the given routine.

[0028] In some embodiments, one or more instances of user interface input that initiates an action include a spoken utterance. In some versions of those embodiments, the method also includes identifying a profile of the user based on one or more voice characteristics of the spoken utterance. In those versions, identifying a plurality of automated assistant routines stored in association with the user includes identifying a plurality of automated assistant routines based on the plurality of automated assistant routines being stored in association with the user's profile. Further, in those versions, executing a given routine may occur in response to an additional spoken utterance from the user, and causing the user interface output to be rendered via the user's client device may also be based on determining that the additional spoken utterance has one or more voice characteristics corresponding to the user's profile.

[0029] In some implementations, causing the user interface output to be rendered via the user's client device occurs at the conclusion of an action initiated by the automated assistant in response to one or more instances of the user interface input.

[0030] In some implementations, the action includes providing a command to change at least one state of a connected device managed by the automated assistant.

[0031] In some implementations, the action includes providing a command to an agent controlled by a third party that is different from the party controlling the automated assistant.

[0032] In some embodiments, selecting a subset of automated assistant routines includes comparing an action to each of a plurality of corresponding actions in each of the automated assistant routines; determining, based on the comparison and for each of the automated assistant routines, whether the action includes one or more inconsistencies with any of the plurality of corresponding actions in the automated assistant routine; and selecting the subset of automated assistant routines based on determining that the plurality of corresponding actions of the automated assistant routines of the subset lack one or more inconsistencies with the action.

[0033] In some embodiments, the one or more inconsistencies include a time inconsistency and / or a device incompatibility inconsistency. A time inconsistency may include an inability to complete an action during a time frame for each of the plurality of corresponding actions. A device incompatibility inconsistency may include an action requiring a device with certain capabilities and each of the plurality of corresponding actions is not associated with any device with the certain capabilities.

[0034] In some embodiments, automatically adding an action includes determining a location in a given routine for performing the action, wherein the location is relative to a plurality of corresponding actions of the given routine. In some versions of those embodiments, the location for performing the action is before the plurality of corresponding actions, after the plurality of corresponding actions, or between two corresponding actions in the plurality of corresponding actions. In some of those versions, the location is after the plurality of corresponding actions and includes a time delay. The time delay can be based on one or more past time delays initiated by the user for the action in the past after one or more past initiations of the given routine by the user.

[0035] In some implementations, one or more instances of user interface input initiating an action are received via an additional client device that is linked to but separate from the client device and via which user interface output is rendered.

[0036] In some implementations, the user interface output includes a graphical output, and the method further includes selecting an additional client device for providing the user interface output based on determining that the client device lacks display capabilities and that the additional client device includes display capabilities.

[0037] In some embodiments, one or more instances of user interface input that initiates the action include a spoken utterance, and the method further includes identifying a profile of the user based on one or more voice characteristics of the spoken utterance. In some of those embodiments, identifying a plurality of automated assistant routines stored in association with the user is based on a plurality of automated assistant routines being stored in association with a profile of the user, the profile being identified based on the voice characteristics of the spoken utterance.

[0038] In some embodiments, a method implemented by one or more processors is provided and includes determining an automatic assistant action and identifying a plurality of automatic assistant routines stored in association with a user. Each of the automatic assistant routines defines a plurality of corresponding actions that will be automatically performed via the automatic assistant in response to initialization of the automatic assistant routine. The determined automatic assistant action may be an action in addition to a plurality of corresponding actions of the automatic assistant routine. The method also includes comparing the automatic assistant action with a plurality of automatic assistant routines stored in association with the user. The method also includes selecting a subset of the automatic assistant routines stored in association with the user based on the comparison. The method also includes determining that the user is interacting with the client device based on sensor data from one or more sensors of the client device. The method also includes rendering a user interface output via the client device based on determining that the user is interacting with the client device and based on selecting a subset of automatic assistant routines, wherein the user interface output prompts the user to add the automatic assistant action to one or more of the automatic assistant routines of the subset. The method also includes receiving an affirmative user interface input in response to the user interface output, wherein the affirmative user interface input indicates a given routine in the automatic assistant routines of the subset. The method also includes, in response to receiving the affirmative user interface input: adding the auto-assistant action as an additional action in a plurality of corresponding actions of the given routine to be automatically performed in response to initialization of the given routine.

[0039] In some implementations, adding is performed in response to receiving an affirmative user interface input and without requiring any other user interface input.

[0040] In some embodiments, the sensor data includes voice data based on one or more microphones of the client device, and determining that the user is interacting with the client device includes: determining one or more voice characteristics based on the voice data; and identifying the user based on the voice characteristics matching a profile of the user.

[0041] In some embodiments, the auto assistant action includes controlling a particular connected device. In some versions of those embodiments, comparing the auto assistant action to a plurality of auto assistant routines stored in association with the user includes determining, based on a device topology of the user's connected devices, that a particular connected device and additional connected devices controlled by at least one of a plurality of corresponding actions of a given routine are all assigned to a same physical location in the device topology. In some of those versions, selecting the given routine for inclusion in the subset based on the comparison includes selecting the given routine for inclusion in the subset based on determining that the particular connected device and additional connected devices are all assigned to a same physical location. The same physical location can be, for example, a user-assigned semantic identifier of a room within a house of the user.

[0042] In some embodiments, the automated assistant action includes controlling a specific connected device, and determining the automated assistant action includes: determining that the specific connected device was controlled in a specific manner in response to a non-automated assistant interaction based on a reported state of the specific connected device; and determining that the automated assistant action caused the specific connected device to be controlled in a specific manner.

[0043] In some embodiments, comparing an auto-assistant action to a plurality of auto-assistant routines stored in association with a user includes: determining at least one action time characteristic of the auto-assistant action based on one or more past performances of the auto-assistant action or a related action by the user; determining at least one routine time characteristic of the auto-assistant routine based on one or more past performances of the auto-assistant routine; and comparing the at least one action time characteristic to the at least one routine time characteristic.

[0044] In some implementations, the automated assistant action includes an interaction with a third-party agent controlled by a particular third party, and determining the automated assistant action includes determining past performance by the user of separate interactions with separate third parties.

[0045] In some implementations, determining the auto-assisted action is based on determining a relationship between the individual interactions and the auto-assisted action.

[0046] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the foregoing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a block diagram illustrating an example environment in which various embodiments may be implemented.

[0048] Figure 2 is a diagram illustrating an example interaction between a user and an automated assistant.

[0049] Figure 3 is a diagram illustrating another example interaction between a user and an automated assistant.

[0050] Figure 4 is a flow chart illustrating an example process according to implementations disclosed herein.

[0051] Figure 5 is a block diagram illustrating an example architecture of a computing device. DETAILED DESCRIPTION

[0052] Embodiments disclosed herein relate to methods, apparatus, and computer-readable media (transitory and non-transitory) for determining whether to recommend an automated assistant action for incorporation into an automated assistant routine of a user. In some of those embodiments, when it is determined that an action is recommended for incorporation into a routine, a user interface output is rendered for presentation to the user. The user interface output may prompt the user as to whether he / she wishes to add the action to the routine. Further, in some versions of those embodiments, the action is added to the routine in response to an affirmative user interface input being received in response to the user interface output. For example, the action may be automatically added to the routine in response to an affirmative user interface input being received.

[0053] An example routine may be a morning routine, in which the automated assistant sequentially performs a plurality of different actions in the morning in order to prepare the user for their day. For example, the morning routine may involve the automated assistant causing the user's schedule for a particular day (e.g., the day) to be audibly rendered via a client device, the automated assistant causing the device (e.g., smart lighting) to be turned on, and then causing the podcast to be audibly rendered via the client device when the user is ready. After completion of the morning routine, the user may cause the execution of an action that may be included in the morning routine. For example, after the morning routine is completed, the user may provide a spoken word to the assistant interface of the second client device, wherein the spoken word causes the automated assistant to turn on an appliance (e.g., a smart coffee machine) associated with the automated assistant. Based at least in part on the temporal proximity of the execution of the morning routine and the user providing a spoken word that causes the automated assistant to turn on the appliance, a "turn on the appliance" action may be recommended to be included in the morning routine. If the user accepts the recommendation, the "turn on the appliance" action may be automatically added to the routine. Thereafter, instead of the user having to give the assistant a separate command to turn on the appliance, the automated assistant may cause the appliance to be turned on in response to the initiation of the morning routine.

[0054] Turning now to the accompanying drawings, Figure 1 An example environment 100 is illustrated in which various embodiments may be implemented. Example environment 100 includes one or more client computing devices 102. Each client device may execute a respective instance of an automated assistant client 112. One or more cloud-based automated assistant components 116, such as a natural language processor 122 and / or a routine module 124, may be implemented on one or more computing systems (collectively, "cloud" computing systems) communicatively coupled to client devices 102 via one or more local area networks and / or wide area networks (e.g., the Internet) as generally indicated at 114.

[0055] In various embodiments, an instance of automated assistant client 108, through its interaction with one or more cloud-based automated assistant components 116, may form what appears from the user's perspective to be a logical instance of automated assistant 112, with which the user may engage in a conversation. Figure 1 108 and 110. One instance of such an automated assistant 112 is depicted in dashed lines in FIG. 109. Thus, it should be understood that each user engaging with an automated assistant client 112 executing on a client device 102 may actually engage with their own logical instance of automated assistant 112. For the sake of brevity and simplicity, the term "automated assistant" as used herein to refer to "serving" a particular user may often refer to the combination of a client device 108 operated by the user and one or more cloud-based automated assistant components 116 (which may be shared among multiple automated assistant clients 108). It should also be understood that in some embodiments, automated assistant 112 may respond to requests from any user, regardless of whether that particular instance of automated assistant 112 is actually "serving" that user.

[0056] Client device 102 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a touch-sensitive computing device (e.g., a computing device that can receive input via touch from a user), a mobile phone computing device, a computing device in a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0057] In various embodiments, client device 102 may include one or more sensors (not depicted) that may take various forms. The sensors may sense different types of input to automated assistant 112, such as verbal, textual, graphical, physical (e.g., touch on a display device including a touch-sensitive projector and / or a touch-sensitive screen of a computing device), and / or visual (e.g., gesture) based input. Some client devices 102 may be equipped with one or more digital cameras configured to capture and provide signals indicative of movement detected in a field of view. Additionally or alternatively, some client devices may be equipped with sensors that detect sound waves (or pressure waves), such as one or more microphones.

[0058] The client device 102 and / or the cloud-based automatic assistant component 116 can communicate with one or more devices 104. The device 104 may include any of the following various devices: an Internet of Things such as a smart appliance, a smart thermostat, a smart coffee machine, a smart light, a smart office, etc. The device 104 is linked to the client device 102 (and / or a specific user of the client device 102) and to each other. For example, the device 104 can be linked to a profile assigned to the client device 102 (and optionally other client devices) and / or can be linked to a profile assigned to the user of the client device 102. The client device 102, other client devices, and the device 104 can together define a coordinated ecosystem of devices. In various embodiments, the devices are linked to each other via a device topology representation, which can be created by a user and / or automatically, and can define various auxiliary client devices, various smart devices, identifiers of each device, and / or attributes of each device. For example, an identifier for a device may specify a room (e.g., living room, kitchen) (and / or other area) of a structure in which the device is located and / or may specify a nickname and / or alias for the device (e.g., couch light, front door lock, bedroom speaker, kitchen assistant, etc.). In this manner, an identifier for a device may be a name, alias, and / or location of a corresponding device that a user may associate with the corresponding device. As described herein, such identifiers may be utilized in various embodiments disclosed herein. For example, such identifiers may be utilized in determining whether an automated assistant action complies with an automated assistant routine.

[0059] Device 104 may be controlled directly by automated assistant 112, and / or device 104 may be controlled by one or more third-party agents 106 hosted by remote device(s) (e.g., another cloud-based component). In addition, one or more third-party agents 106 may also perform function(s) in addition to controlling device 104 and / or controlling other hardware devices. For example, automated assistant 112 may interact with third-party agents 106 to cause services to be performed, transactions to be initiated, etc. For example, a user voice command "order a large pepperoni pizza from Agent X" may cause automated assistant client 108 (or cloud-based automated assistant component(s) 116) to send an agent command to a third-party agent "Agent X". The agent command may include, for example, an intent value indicating the "order" intent determined from the voice command and optional slot values, such as "type=pizza", "toppings=pepperoni", and "size=large". In response, the third-party agent may cause an order for a large pepperoni pizza to be initiated and provide content to automated assistant 112 indicating that the order has been successfully initiated. The content (or a transformation thereof) may then be rendered to the user via output device(s) (e.g., speaker(s) and / or display(s)) of client device 102. As described herein, some embodiments may recommend adding an "order a large pepperoni pizza from Agent X" action and / or similar actions to an existing routine of a user. When added to an existing routine of a user, the "order a large pepperoni pizza from Agent X" automated assistant action may be initiated in response to initiation of the existing routine without the user having to separately provide the spoken utterance of "order a large pepperoni pizza from Agent X." Thus, transmission and / or processing of audio data to capture such spoken utterance may be avoided, thereby conserving computer and / or network resources.

[0060] In many embodiments, the automated assistant 112 participates in a conversational session with one or more users via the user interface input and output devices of one or more client devices 102. In some embodiments, the automated assistant 112 may participate in a conversational session with a user in response to user interface input provided by the user via one or more user interface input devices of one of the client devices 102. In some of those embodiments, the user interface input is explicitly directed to the automated assistant 112. For example, a user may speak a predetermined invocation phrase, such as "OK, Assistant" or "Hey, Assistant", to cause the automated assistant 112 to begin actively listening. In many embodiments, a user may speak a predetermined shortcut phrase to start running a routine, such as "OK Assistant, Good Morning" to start running a good morning routine.

[0061] In some embodiments, even when the user interface input is not explicitly directed to the automated assistant 112, the automated assistant 112 may participate in a conversation session in response to the user interface input. For example, the automated assistant 112 may check the content of the user interface input in response to certain terms present in the user interface input and / or based on other prompts and participate in a conversation session. In many embodiments, the automated assistant 112 may utilize speech recognition to convert the speech from the user into text, and respond to the text, for example, by providing visual information, by search results, by providing general information and / or taking one or more responsive actions (e.g., playing media, starting a game, ordering food, etc.). In some embodiments, the automated assistant 112 may additionally or alternatively respond to the speech without converting the speech into text. For example, the automated assistant 112 may convert the speech input into an embedding, into an entity representation (indicating an entity / entities present in the speech input) and / or other "non-text" representations, and operate on such non-text representations. Therefore, the embodiments described herein as operating based on text converted from speech input may additionally and / or alternatively directly operate on the speech input and / or other non-text representations of the speech input.

[0062] Each of the client computing devices 102 and the computing devices operating the cloud-based automated assistant component 116 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. Operations performed by one or more computing devices 102 and / or by the automated assistant 112 may be distributed across multiple computer systems. The automated assistant 112 may be implemented, for example, as a computer program running on one or more computers in one or more locations coupled to each other over a network.

[0063] As described above, in various embodiments, client computing device 102 can operate automated assistant client 108. In various embodiments, each automated assistant client 108 can include a corresponding speech capture / text-to-speech (“TTS”) speech-to-text (“STT”) module 110. In other embodiments, one or more aspects of speech capture / TTS / STT module 110 can be implemented separately from automated assistant client 108.

[0064] Each speech capture / TTS / STT module 110 can be configured to perform one or more functions: for example, capturing the user's voice via a microphone (which in some cases may include a sensor in the client device 102); converting the captured audio to text (and / or other representations or embeddings); and / or converting text to speech. For example, in some embodiments, because the client device 102 may be relatively constrained in computing resources (e.g., processor cycles, memory, battery, etc.), the local speech capture / TTS / STT module 110 for each client device 102 can be configured to convert a limited number of different spoken phrases - particularly phrases that invoke the automated assistant 112 - to text (or other forms, such as lower-dimensional embeddings). Other speech inputs can be sent to a cloud-based automated assistant component 116, which can include a cloud-based TTS module 118 and / or a cloud-based STT module 120.

[0065] The cloud-based STT module 120 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the voice capture / TTS / STT module 110 into text (which can then be provided to the natural language processor 122). The cloud-based TTS module 118 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 112) into computer-generated speech output. In some embodiments, the TTS module 118 can provide the computer-generated speech output to the client device 102 for direct output, for example, using one or more speakers. In other embodiments, the text data (e.g., natural language responses) generated by the automated assistant 112 can be provided to the voice capture / TTS / STT module 110, which can then convert the text data into computer-generated speech that is output locally.

[0066] The automated assistant 112 (e.g., cloud-based assistant component 116) may include a natural language processor 122, the aforementioned TTS module 118, the aforementioned STT module 120, and other components, some of which are described in greater detail below. In some embodiments, one or more engines and / or modules of the automated assistant 112 may be omitted, combined, and / or implemented in components separate from the automated assistant 112. In some embodiments, to protect privacy, at least a portion of one or more of the components of the automated assistant 112, such as the natural language processor 122, the voice capture / TTS / STT module 110, the routines module 124, etc., may be implemented at least in part on the client device 102 (e.g., excluded from the cloud).

[0067] In some implementations, the automated assistant 112 generates responsive content in response to various inputs generated by a user of the client device 102 during a human-machine conversation session with the automated assistant 112. The automated assistant 112 may provide the responsive content to be presented to the user as part of the conversation session (e.g., when separate from the user's client device, over one or more networks). For example, the automated assistant 112 may generate responsive content in response to free-form natural language input provided via the client device 102. As used herein, free-form input is formulated by the user and is not limited to a set of options presented to the user for selection.

[0068] Natural language processor 122 of automated assistant 112 processes natural language input generated by a user via client device 102 and may generate annotated output for use by one or more components of automated assistant 112. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 102. The generated annotated output includes one or more annotations of the natural language input and, optionally, one or more (e.g., all) terms of the natural language input.

[0069] In some implementations, the natural language processor 122 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the natural language processor 122 may include a part-of-speech tagger that is configured to annotate terms with their grammatical roles. Further, for example, in some implementations, the natural language processor 122 may additionally and / or alternatively include a dependency analyzer (not shown) configured to determine syntactic relationships between terms in the natural language input.

[0070] In some embodiments, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more paragraphs, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. The entity tagger of the natural language processor 122 may annotate references to entities at a higher level of granularity (e.g., enabling identification of all references to an entity category such as a person) and / or a lower level of granularity (e.g., enabling identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve a specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve a specific entity.

[0071] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more context clues. For example, a coreference resolver may be utilized to resolve the term "there" in the natural language input "I liked Hypothetical Café last time we ate there." to "Hypothetical Café."

[0072] In many embodiments, one or more components of the natural language processor 122 can rely on annotations from one or more other components of the natural language processor 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency analyzer when annotating all mentions of a particular entity. Similarly, for example, in some embodiments, a coreference resolver can rely on annotations from a dependency analyzer when clustering references to the same entity. In many embodiments, when processing a particular natural language input, one or more components of the natural language processor 122 can use relevant previous input and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0073] Routines module 124 of automated assistant 112 can determine candidate actions for adding to the user's automated assistant routines, can determine a subset of the user's available routines to which the candidate action is recommended for addition, can provide a recommendation for adding the action to the subset of routine(s), and can add the action to one of the routines based on affirmative input indicating one of the routines received in response to the recommendation. Routines module 124 can be implemented by cloud-based automated assistant component 116 as depicted, and / or by automated assistant client 108 of client device 102.

[0074] In some embodiments, routines module 124 determines automated assistant actions for potential recommendation for addition to a user's routine based on actions initiated by a user through interaction with one or more instances of automated assistant 112. In some embodiments, routines module 124 additionally or alternatively determines automated assistant actions for potential recommendation for addition to a user's routine based on related but different actions initiated by a user through interaction with one or more instances of automated assistant 112. In some embodiments, routines module 124 additionally or alternatively determines automated assistant actions for potential recommendation for addition to a user's routine based on data associated with the user indicating that the user may have performed the action or related actions. The data associated with the user may include data based on the user's interactions with other non-automated assistant applications, data based on the state of smart devices in response to direct user control of those smart devices, data indicating locations and / or transactions, and the like.

[0075] In some embodiments, routine module 124 may access a list of stored routines for a user and determine a subset of routines to which the determined auto assistant action is recommended for addition. In some of those embodiments, routine module 124 selects the subset of auto assistant routines based on comparing a time attribute of past occurrences of the routine to a time attribute of (multiple) past occurrences of the auto assistant action (or an action related to the auto assistant action). For example, if the auto assistant action is a "decrease smartthermostat set point 3°" action determined based on past occurrences of a user manually adjusting a smart thermostat, the times of those past occurrences may be determined based on the times of status updates from the smart thermostat reflecting those decreases. Furthermore, routines for users that are frequently initiated near those times (e.g., within an hour) may be more likely to be selected for the subset than routines for users that are rarely or never initiated near those times.

[0076] In some of those embodiments, routines module 124 additionally or alternatively selects a subset of automated assistant routines based on comparing device topology attributes of the device(s) utilized in the routines to device topology attributes of the device(s) utilized in the automated assistant actions. For example, if an automated assistant action involves control of a component of a user's vehicle having an assigned device topology attribute of "location=vehicle," then routines that include actions to control other device(s) having an assigned device topology attribute of "location=vehicle" may be more likely to be selected for the subset than routines that lack any actions to control a device having the device topology attribute of "location=vehicle."

[0077] In some of those embodiments, routine module 124 additionally or alternatively selects to exclude from the subset a subset of auto assistant routines that includes any routines that conflict with an auto assistant action. For example, for a "dim the livingroom lights to 50%" auto assistant action, a given routine can be excluded based on its having a conflicting "turn off the living room lights" action. Further, for example, for an auto assistant action that interfaces with an agent to perform a service that results in an item being delivered on demand to a user's home, a given routine can be excluded based on the given routine being assigned to a "work" location for the user (e.g., explicitly or based on being associated only with linked devices having a "work" device topology attribute).

[0078] In some embodiments, the routine module considers multiple competing actions for addition to the automated assistant routine, and selecting a subset of the competing actions (e.g., one of the competing actions) can be used to output an actual recommendation via a user interface for addition to the routine. In some of those embodiments, each of the multiple competing actions can be compared to the routine, and the selected action(s) can be one or more of the multiple competing actions that most closely conform to the routine, lack any conflicts with the routine, and / or meet other criteria.

[0079] Regardless of the technique used to determine the action(s) and / or the routine(s) to which the action(s) may be added, the routine module 124 may cause a user interface output to be added that prompts the user as to whether to add the action(s) to the routine(s). In some embodiments, upon recommending the addition of an action to a routine, the routine module 124 causes the corresponding user interface output to be rendered in response to the user's interaction with the automated assistant to perform the action (e.g., after the action has been completed in response to the user interaction). In some embodiments, upon recommending the addition of an action to a routine, the routine module 124 causes the corresponding user interface output to be rendered in response to the initiation of the routine (e.g., before performing any actions of the routine or after completing all actions of the routine).

[0080] If an affirmative user interface input is received in response to a recommendation provided by routine module 124, routine module 124 may cause a corresponding action to be added to the corresponding routine. For example, routine module 124 may automatically add an action to a routine without requiring any other user interface input from the user. In some embodiments, when adding an auto-assistant action to a routine, routine module 124 may determine an execution location of the auto-assistant action among pre-existing actions of the routine. In some versions of those embodiments, the location may be determined based on a duration required for the auto-assistant action of the user interface output (if any) and a duration(s) required for the pre-existing actions of the routine of the user interface output (if any). In some additional or alternative versions, the location of execution may be final and may optionally occur with a time delay.

[0081] exist Figure 2 200 illustrates an example of a user interacting with a client device to add an action to an automated assistant routine. Image 200 includes a scene of a room, which includes a user 202 and a client device 204. User 202 interacts with client device 204 via a spoken utterance indicated in dialog box 206. The spoken utterance of dialog box 206 requests the automated assistant to perform an action to increase the set point of a smart thermostat linked to client device 204 by five degrees. The automated assistant associated with client device 204 can process the spoken utterance to determine an appropriate command for increasing the set point of the smart thermostat by five degrees. In some embodiments, the automated assistant associated with client device 204 can interface directly with the smart thermostat to change the temperature. For example, the automated assistant can provide the command directly to the smart thermostat. Additionally or alternatively, the automated assistant can interface with a third-party agent associated with the smart thermostat to change the temperature. For example, the automated assistant can provide the command to the third-party agent, which in turn generates a corresponding command and transmits the corresponding command to the smart thermostat to cause the temperature to change.

[0082] In some embodiments, as shown in dialog box 208, client device 204 can render an audible user interface input confirming that the temperature has been increased, and can also recommend adding increasing the temperature by five degrees to the user's "Good Morning" routine. The recommendation can include a prompt asking the user if they want to add the action to the recommended routine. In some embodiments, if the user typically increases the temperature in the morning after completing their "Good Morning" routine, the recommendation can be made. In some embodiments, the recommendation can be made even if the user performs the "Good Morning" routine using a first client device and the user changes the temperature using a second client device.

[0083] Before making this recommendation, the automated assistant associated with client device 204 may select the "Good Morning" routine from among the user's multiple other routines as the routine to which the "temperature increase" automated assistant action should be recommended for addition. Various techniques may be used to select the "Good Morning" routine to the exclusion of the user's other routines, such as the various techniques described in detail herein.

[0084] The user can decide whether they want to add the action to the recommended routine and respond to the prompt via another spoken utterance indicated by dialog box 210. If the user responds affirmatively to the prompt, as indicated by the spoken utterance of dialog box 210, the automated assistant will automatically add the action to the user's "Good Morning" routine. In addition, the automated assistant causes another audible user interface output to be provided via client device 204, as indicated by dialog box 212, wherein the other audible output confirms that raising the temperature by five degrees has been added to the "Good Morning" routine. However, in some embodiments, this confirmation may be optional. Additionally or alternatively, client device 204 may confirm that the action has been added to the routine the next time the user runs the routine. In various embodiments where the user has given permission, such as to their spouse, to add the action to the user's routine, this situation may provide the user with additional confirmation that they want to add the action to a particular routine.

[0085] In some implementations, user 202 may provide input to client device 204 to cause the client device to interact with one or more third-party agents to cause an action to be performed that is not an action to control a smart device. For example, instead of requesting a change in temperature, a user may use client device 204 to place an online order from a third-party agent associated with a "Local Coffee Shop" and then pick up their ordered coffee on their way to work. The user may interact with an automated assistant associated with client device 204 when making such an order, or with a separate application of client device 204. In many implementations, an automated assistant associated with client 204 (such as the one described above in Figure 1, its routine module 124 described in , can determine a coffee ordering action and determine whether to recommend adding the action to the user's "Good Morning" routine and / or (multiple) other automated assistant routines. For example, if the user uses client device 204 to order coffee from the "Local Coffee Shop" every morning immediately after the "Good Morning" routine is completed, the automated assistant action can determine to recommend adding the action to the "Good Morning" routine based at least in part on the temporal proximity of the action to the "Good Morning" routine. In some embodiments, the user can use client device 204 to order coffee from the "Local Coffee Shop" by, for example, delaying an average of 25 minutes after client device 204 completes rendering the "Good Morning" routine (i.e., the user places the order as they leave home, which occurs an average of 25 minutes after the "GoodMorning" routine is completed), so that when the user arrives at the "Local Coffee Shop" there is a hot cup of coffee waiting for them. In some of those embodiments, the order for coffee from the "Local Coffee Shop" may still be added to the "Good Morning" routine so that it is automatically executed in response to execution of the "Good Morning" routine and along with other pre-existing actions of the "Good Morning" routine. However, the automated assistant may optionally include a delay (e.g., 25 minutes after the last action in the pre-existing "Good Morning" routine is completed) before the "Good Morning" routine places an order for coffee from the "Local Coffee Shop" for the user via a third-party agent. Additionally or alternatively, the automated assistant may add it to the "Good Morning" routine in a manner such that a prompt (e.g., "ready to order your coffee") is provided after a delay after completion of the last action in the pre-existing "GoodMorning" routine, and the order is only initiated if affirmative input is received in response to the prompt.

[0086] Additionally or alternatively, in some implementations, the automated assistant may examine actions in the "Good Morning" routine if the user has only placed a coffee order from the "Local Coffee Shop" once using client device 204. For example, if the routine includes turning on a coffee machine, it is likely that the user has only run out of coffee that morning, and the automated assistant should not recommend adding a coffee order to the "Good Morning" routine. However, in some implementations, if the user frequently visits the "Local Coffee Shop" in the mornings even if they have not reserved coffee in advance using a client device, the automated assistant may decide to recommend adding the placing a coffee order action to the "Good Morning" routine.

[0087] Furthermore, the automated assistant may suggest automated assistant actions to add to a routine even if the user has not previously interacted with a third party via which the action will be performed. For example, the user may not be aware of a new coffee shop, "New Coffee Shop," that opened last month and is conveniently located on the route between the user's home and work. An action to have coffee ordered at the "New Coffee Shop" may be recommended for addition to the user's "Good Morning" routine that currently lacks any coffee ordering actions. Additionally or alternatively, in some implementations, an action to have coffee ordered at the "New Coffee Shop" may be recommended for addition to an existing routine and replace an existing action of the routine that has coffee ordered at another coffee shop.

[0088] In some embodiments, more than one action may be presented to the user for potential addition to a routine, wherein each of the actions is associated with a different third-party agent. For example, a user who places a daily morning coffee order from a "Local Coffee Shop" via a client device 204 may be presented with a recommendation to add any of three separate actions to an existing "Good Morning" routine. Each of the three actions may cause a coffee order to be initiated, but the first action may cause it to be ordered in the case of a "Local Coffee Shop", the second action may cause it to be ordered in the case of a "New Coffee Shop", and the third action may cause it to be ordered in the case of a "Classic Coffee Shop". In some embodiments, corresponding additional information may be provided to each of the actions, such as: the amount of time that visiting the corresponding coffee shop will increase and / or decrease the user's commute, the corresponding price of the coffee order, the corresponding reward program, and / or other information to help the user make an informed decision about which coffee shop to add to their "Good Morning" routine. In some embodiments, one or more of the recommended actions may be recommended based on the counterpart providing monetary consideration for inclusion. In some implementations, the amount of monetary consideration required may be related to how closely the action matches the routine to which the action was suggested for addition, how frequently the user performs the routine, and / or other criteria.

[0089] Despite Figure 2 204, additional or alternative user input may be provided by the user 202 and / or by the client device 204 via content rendered in additional or alternative modalities. For example, the user 202 may additionally or alternatively provide typed input, touch input, gesture input, etc. Also, for example, the output may additionally or alternatively be graphically rendered by the client device 204. As a specific example, the client device 204 may graphically render prompts for adding actions to the routine, as well as selectable interface elements that may be selected through touch interaction by the user 202 to cause the action to be automatically added to the routine.

[0090] exist Figure 3300 illustrates an additional example of a user interacting with a client device to add an action to an automated assistant routine. Image 300 includes a scene of a room including a user 302 and a client device 304. User 302 can interact with client device 304 to start a routine by providing a spoken utterance including a predefined shortcut phrase. For example, in dialog box 306, user 302 starts the "Good Morning" routine by providing a spoken utterance detected by client device 304 and including the phrase "Good Morning". An automated assistant associated with client device 304 can process audio data capturing the spoken utterance, determine that the spoken utterance includes a shortcut phrase for the "Good Morning" routine, and in response, cause execution of each action in the "Good Morning" routine. In some embodiments, the automated assistant can ask user 302 via client device 304 after the routine is completed whether he or she would like to add an action to the routine that has just been completed. For example, dialog box 308 indicates that client device 304 has executed the user's "Good Morning" routine, and then asks the user whether he or she would like to add lighting control to his or her "Good Morning" routine. Additionally or alternatively, in some implementations, the client device 304 may ask the user if they would like to add lighting controls to their "Good Morning" routine before running the routine and including the new action as part of the running routine.

[0091] In many embodiments, the automated assistant associated with the client device 304 may determine the recommendation to add a new action to the just completed routine (in this case, the "Good Morning" routine) based on the user's historical actions. For example, the recommendation may be determined based on the user always using the client device 304 to turn on their smart lights after the "Good Morning" routine is completed. Additionally or alternatively, in some embodiments, the user may have (multiple) smart lights in their house, but is not aware of the ability to control the smart lights with the automated assistant, and therefore has not previously been used to control the smart lights with the automated assistant. In such an example, the automated assistant may make the recommendation based on determining that the user has not yet performed the lighting control action in an attempt to add functionality to the user's routine that the user may otherwise not be aware of. Additionally or alternatively, in some embodiments, the automated assistant may make the recommendation based on determining that control of the smart lights is an action that other users often have in related routines. For example, if more than 90% of users with devices that require lighting control (as determined via the user's device topology) also have lighting control actions in their "Good Morning" routine, the assistant may make a recommendation to add lighting control to the "Good Morning" routine.

[0092] The user can decide whether to add the suggested action to the routine. For example, dialog box 310 indicates that the user has decided to add the suggested action (i.e., lighting control) to his "Good Morning" routine. However, the user does not need to add the recommended action to the suggested routine. In some embodiments, the client device 304 associated with the automated assistant can confirm that the recommended action is added to the routine. For example, dialog box 312 indicates that lighting control has been added to the user's "Good Morning" routine. However, in various embodiments, the client device 304 can confirm to the user that the action has been added to a particular routine the next time the user runs the routine. In some embodiments, this situation can provide additional backup to confirm that the user still wants the new action in the routine and has not changed his mind at the next routine. In addition, in some embodiments, the user can give permission to others to modify their routine. In some such embodiments, confirming that the new action has been added before the user runs the routine can allow the user to confirm the action added to the user's routine by a different user (such as the user's spouse).

[0093] Figure 4 400 illustrates a process for determining whether to add an action to an automated assistant routine according to various embodiments. Process 400 can be performed by one or more client devices and / or any other device capable of interacting with an automated assistant. The process includes determining (402) an automated assistant action. In some embodiments, an action can be determined based on having been performed one or more times in response to a user's interaction with the automated assistant. In some embodiments, an action can include controlling a physical device (e.g., controlling a smart thermostat, turning on a smart coffee machine, controlling a smart speaker, etc.) and / or interacting with a remote agent (e.g., a calendar agent that can provide a daily agent to a user, a weather agent that can provide a user with the weather of the day, etc.). In some embodiments, an action can be determined based on how many times the user has historically performed it. For example, an action for controlling certain smart lights can be determined based on the user controlling certain smart lights every night before going to bed through interaction with the automated assistant, through manual interaction, and / or through interaction with an application specifically for controlling smart lights. An action can be determined based on being performed at least a threshold number of times. In some embodiments, an automated assistant action can be determined even if the user has never caused the execution of the action. For example, an action related to controlling a smart lock may be determined in response to determining that the smart lock has just been installed by a user and / or added to the topology of linked devices that can be controlled by the automated assistant. For example, a user may have just installed a new smart lock, but the smart lock is not associated with any routines.

[0094] The automated assistant may identify (404) routines for the user. A user may have many routines, some of which may overlap. For example, a user may have a first "Good Morning" routine during the work week and a second "GoodMorning" routine on the weekends. Additionally or alternatively, a user may have several routines to choose from when they get home from get off work. For example, a user may have a "Veg Out" routine and a "Read a Book" routine. In some embodiments, a profile of a user may be identified, and the set of routines assigned to the user's profile may be the identified set of routines associated with the user. In some of those embodiments, the set of routines for the user's profile is identified for the action determined at (402) based on determining that the action at (402) is performed for the user's profile. The above with respect to Figure 1 Any of the various sensors described can be used to identify a user profile associated with an input to an automated assistant (e.g., voice matching based on audio data) and an automated assistant routine associated with the user profile. In various embodiments, a set of routines can be shared between members of the same family. For example, a user and his spouse can enjoy cooking together and have a shared "Grilling Night" routine, in which both users interact with one or more automated assistant clients and one or more client devices. For example, the barbecue night routine may include the following actions: turning on the lights in the kitchen, turning on the outside grill to start heating it, and rendering speakers in both the kitchen and the outdoor grilling area to play the same music. Both users can interact with the equipment in the kitchen and the grill at various points in time. In some embodiments, since the routine is a shared routine, the "Grilling Night" routine can be included in the user profiles for both users, and should be considered for both users when potentially adding new actions to the new routine. Allowing one user to add actions to a shared routine can also share computing resources, because the two users will not have to manually go through the process of adding new actions to the shared routine.

[0095] An automated assistant routine group may be selected (406) from the group of actions identified at (404), potentially adding the action identified at (402) to the automated assistant routine group. For example, if the user is performing an action when returning home from get off work, the "Veg Out" routine and the "Read a Book" routine are potential automated assistant routines to add to the action. However, the automated assistant may compare the action to those routine(s) and / or other routines to select an automated assistant routine group that is a subset of all available automated assistant routines. For example, the "Veg Out" routine may include turning on the living room TV and turning off the living room lights. Conversely, the "Read a Book" routine may include setting custom lighting for book reading in the den and playing soft jazz in the den. If the user action is to request information about the current TV show playing that evening, in many embodiments, the automated assistant will select a routine that includes TV for inclusion in the group to which the action is potentially added. Additionally, for example, using the device topology of linked devices for the user, the automated assistant can determine that there are no display-capable devices in the study, which is the room associated with all actions for the "Book Reading" routine. Thus, since the action involves pushing information to a display-capable device, and the "Book Reading" routine is currently only associated with rooms that have no display-capable devices, the automated assistant can exclude the "Book Reading" routine from the group of potential added actions. In some implementations, the automated assistant can analyze the action and compare the action to each action in each routine for the user and / or location where the routine is typically executed to generate the group of routines for potential added actions.

[0096] In some embodiments, potential actions may be compared to a number of routines before selecting a routine (or group of routines) to potentially add an action to. For example, a user may have four routines: "Party Time," "Good Morning," "Good Night," and "Drive Home." The "Party Time" routine may include the following actions: rendering party music to speakers in the living room, adjusting the networked lights in the living room to flash in a random color pattern, and ordering a pizza for delivery from a "Local Pizza Shop" via a third-party agent. The "Good Morning" routine may include the following actions: rendering the user's schedule for the day, turning on the networked lights in the bedroom, rendering the user's weather for the day, and rendering news headlines for the user. The user's "Good Night" routine may include turning off all networked lights in the house, rendering tomorrow's schedule for the user, rendering tomorrow's weather for the user, and rendering two hours of white noise for the user. Additionally, the user's "Drive Home" routine may include the following actions: rendering an evening schedule for the user, notifying the user's spouse on another client device that the user is driving home, rendering information about a user-determined group of stock prices, and rendering a podcast for the remainder of the dive.

[0097] As an example, the automated assistant may examine four routines of the user and select which routine to suggest adding an action to order coffee from the "Local Coffee Shop". In various embodiments, the automated assistant may analyze various information for each routine, including: the actions included in the routine compared to the new action, the historical relationship between the new action and the routine (if any), the temporal relationship between the new action and the routine (if any), whether other users typically have similar actions in routines, etc. With respect to the "Party Time" routine, the automated assistant may compare the new action of ordering coffee to the routine. Other users may typically order coffee before a party is held (i.e., so they are awake during the party). "Party Time" has inherent time constraints and needs to be run when the "Local Pizza Shop" is open. Users may historically order coffee at approximately the same time they run the "Party Time" routine (i.e., the user orders afternoon coffee while working from home, but the user also has a party at approximately the same time in the afternoon), so historical data of when the user runs the "Party Time" routine may indicate that this should be a suggestion for the user. However, the action of ordering a pizza for delivery from the "Local Pizza Shop" conflicts with leaving home to pick up a cup of coffee (i.e., if the user has already left home to pick up the coffee order, the user may not be home when the pizza is delivered). Therefore, the automated assistant generally does not recommend adding the action of ordering a cup of coffee from the "Local Coffee Shop" to the "Party Time" routine. Additionally or alternatively, in some implementations, the automated assistant can review the user's historical actions after running the "Party Time" routine.

[0098] Similarly, the automated assistant may analyze the "Good Morning" routine in comparison to a new coffee ordering action. Other users often order coffee in similar routines. During the time that the user runs their "Good Morning" routine, the "Local CoffeeShop" is likely to be open. Additionally, there are no pre-existing actions in the "Good Morning" routine that conflict with ordering coffee (e.g., not turning on a coffee machine included in the "Good Morning" routine). Furthermore, any action by the user ordering coffee in the morning at approximately the same time that the user runs the "GoodMorning" routine may be an indicator to suggest adding a new action to order coffee from the "Local CoffeeShop" to the "Good Morning" routine.

[0099] Conversely, the "Good Night" routine may be executed when the "Local Coffee Shop" is closed, which may indicate to the assistant that adding an action is not recommended. However, in some implementations, if the user's preferred coffee shop is closed, the automated assistant may recommend another coffee shop to order coffee from. Most users do not include ordering coffee in a similar evening routine, which may indicate that the automated assistant does not recommend the action. Additionally, a user's actions in the "Good Night" routine (such as turning off the networked lighting for the entire house, followed by an action to play white noise for two hours) conflict with leaving the house to get a cup of coffee. The action of turning off the lights for the entire house itself does not conflict with going to get a cup of coffee, but this action paired with other actions in the "Good Night" routine conflicts with leaving the house to get coffee.

[0100] Furthermore, the "Drive Home" routine may be compared to a new coffee ordering action. The user is already driving, so there is no contradiction between the "Drive Home" routine and any action in going to get a cup of coffee. However, the automated assistant typically checks many factors before suggesting a routine to add a new action. The automated assistant may compare to similar routines of other users and see that most users do not order coffee when driving home from get off work in the evening. This situation may indicate that the automated assistant does not recommend adding the action to the "Drive Home" routine. Additionally, the user's historical actions of ordering coffee may be analyzed, and the automated assistant may see that the user no longer orders coffee after 3 p.m., and that the user typically starts the "Drive Home" routine between 5 p.m. and 6 p.m. The time at which the user performs the "Drive Home" routine contradicts historical data about when the user orders coffee from the "Local Coffee Shop." This situation may also indicate that the automated assistant does not recommend adding the new action of ordering coffee to the "Drive Home" routine.

[0101] Therefore, after analyzing all routines for the user, the auto assistant in this example should select the "GoodMorning" routine to suggest adding a new action to order a cup of coffee from the "Local Coffee Shop".

[0102] The automated assistant can render a user interface output (408) that prompts whether the action should be added to any of the routines in the potential automated assistant routine set. In some embodiments, the user can add the same action to several routines. In some embodiments, the user may want to add the action to a single routine. In addition, in some embodiments, the user can select the action to be added to the routine after performing a process similar to that described above. Figure 2In addition or alternatively, a routine may be run similar to that described above. Figure 3 After a specific routine, the user is asked whether the action should be added to the routine.

[0103] If an affirmative user interface input is received (410) in response to the output at (408), the automated assistant can add the action to the corresponding automated assistant routine. For example, if the output at (408) prompts whether the action should be added to a particular routine, the spoken utterance of "yes" can cause the action to be added to the particular routine. Further, for example, if the output at (408) prompts whether the action should be added to a first particular routine or a second particular routine, the spoken utterance of "the first one" can cause the action to be added to the first particular routine instead of the second particular routine; the spoken utterance of "the second one" can cause the action to be added to the second particular routine instead of the first particular routine; and the spoken utterance of "both" can cause the action to be added to both the first and second particular routines. In many implementations, the automated assistant can prompt the user for the name of a particular routine, and the user can add the action to the particular routine by speaking the name of the routine. For example, if the output (408) prompts the user to suggest adding routines to a Good Morning Routine and a Drive to Work Routine, then the spoken utterance of the name of the routine (e.g., "Drive to Work") may cause an action to be added to the "Drive to Work" routine. In some implementations, the automated assistant may automatically add the action to the routine and may message the user once the action has been added to the routine. In some implementations, similar to the above, Figure 2 As discussed in , the automated assistant can decide where to add new actions within a sequence of actions in a pre-existing routine.

[0104] Figure 5 5 is a block diagram of an example computing device 510. The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory 525 and a file storage subsystem 526, a user interface input device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and couples to corresponding interface devices in other computer systems.

[0105] The user interface input device 522 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or over a communication network.

[0106] User interface output device 520 may include a display subsystem, a printer, a fax machine or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device or some other mechanism for creating a visible image. The display subsystem may also, for example, provide a non-visual display via an audio output device. Typically, the use of item "output device" is intended to include all possible types of devices and for outputting information from computer system 510 to a user or to another machine or computer system.

[0107] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include an executable Figure 1 selected aspects and / or processes of, any operations discussed herein, and / or logic implementing one or more of cloud-based automated components 116, automated assistant 112, client device 102, and / or any other devices or applications discussed herein.

[0108] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include many memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 storing fixed instructions. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functionality of certain embodiments can be stored in the storage subsystem 426 by the file storage subsystem 524, or stored in other machines accessible by the processor 514.

[0109] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0110] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more Figure 5 The computer system depicted in FIG.

[0111] In cases where the systems described herein collect personal information about users (or "participants," as they are often referred to herein) or may utilize personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social activities or events, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content more relevant to the user is received from content servers. Similarly, certain data may be processed in one or more ways prior to storage or use so as to remove personally identifiable information. For example, the identity of a user may be processed so that no personally identifiable information can be determined for the user, or the user's geographic location may be summarized where geographic location information is available (e.g., to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may have control over how information about the user is collected and / or used.

Claims

1. A method implemented by one or more processors, the method comprising: determining an action initiated by the automated assistant, the action being initiated by the automated assistant in response to one or more instances of user interface input provided by a user via one or more automated assistant interfaces with which the automated assistant interacts; identifying a plurality of automated assistant routines stored in association with the user, each of the automated assistant routines defining a plurality of corresponding actions to be automatically performed via the automated assistant in response to initialization of the automated assistant routine, wherein the actions are in addition to the actions of the automated assistant routine; comparing the action to the plurality of automated assistant routines stored in association with the user; selecting, based on the comparison, a subset of the automated assistant routines stored in association with the user based on at least one of: device topology attributes of the automated assistant routines and the actions, temporal attributes of the automated assistant routines and the actions, and one or more inconsistencies of the automated assistant routines and the actions; Based on the action being initiated in response to user interface input provided by the user and based on selecting the subset of the automated assistant routine: causing a user interface output to be rendered via a client device of the user, wherein the user interface output prompts the user to add the action to one or more of the automated assistant routines of the subset; receiving an affirmative user interface input in response to the user interface output, wherein the affirmative user interface input indicates a given routine in the subset of the automated assistant routines; as well as In response to receiving the affirmative user interface input: The action is automatically added as an additional action among the plurality of corresponding actions of the given routine to be automatically performed in response to initialization of the given routine.

2. A method according to claim 1, wherein the one or more instances of user interface input include one or more spoken utterances that, when converted to text, include first text of a first length; and wherein initialization of the given routine occurs in response to a given spoken utterance that, when converted to text, includes second text of a second length, wherein the second length is shorter than the first length. 3 . The method of claim 1 , wherein causing the user interface output to be rendered via the client device of the user is further based on and occurs at the completion of executing the given routine.

4. The method of claim 3, wherein the one or more instances of user interface input initiating the action comprise spoken words, and the method further comprises: identifying a profile of the user based on one or more voice characteristics of the spoken utterance; wherein identifying the plurality of automated assistant routines stored in association with the user comprises identifying the plurality of automated assistant routines based on the plurality of automated assistant routines being stored in association with the profile of the user; wherein executing the given routine occurs in response to an additional spoken utterance from the user; as well as Wherein causing the user interface output to be rendered via the client device of the user is further based on determining that the additional spoken utterance has one or more voice characteristics corresponding to the profile of the user.

5. The method of claim 1, wherein causing the user interface output to be rendered via the client device of the user occurs at the completion of the action initiated by the automated assistant in response to the one or more instances of user interface input.

6. The method of claim 1, wherein the action comprises providing a command to change at least one state of a connected device managed by the automated assistant.

7. The method of claim 1, wherein the action comprises providing commands to an agent controlled by a third party, wherein the third party is different from the party controlling the automated assistant.

8. The method of claim 1 , wherein selecting the subset of the automated assistant routines comprises: comparing the action to each of the plurality of corresponding actions in each of the automated assistant routines; as well as determining, based on the comparing and for each of the automated assistant routines, whether the action includes the one or more inconsistencies with any of the plurality of corresponding actions in the automated assistant routine; as well as The subset of the automated assistant routines is selected based on a determination that the plurality of corresponding actions of the subset of the automated assistant routines lack the one or more inconsistencies with the action.

9. The method of claim 8, wherein the one or more inconsistencies include a time inconsistency.

10. The method of claim 9, wherein the time inconsistency comprises a failure to complete the action during a time frame of each corresponding action in the plurality of corresponding actions. The method of claim 8 , wherein the one or more conflicts include a device incompatibility conflict.

12. The method of claim 11, wherein the device incompatibility contradiction comprises that the action requires a device with certain capabilities and each corresponding action in the plurality of corresponding actions is not associated with any device with the certain capabilities.

13. The method of claim 1, wherein automatically adding the action comprises determining a location in the given routine for performing the action, the location relative to the plurality of corresponding actions of the given routine. The method according to claim 13 , wherein the position for performing the action is before the plurality of corresponding actions, after the plurality of corresponding actions, or between two corresponding actions in the plurality of corresponding actions.

15. The method of claim 13, wherein the position is subsequent to the plurality of corresponding actions and includes a time delay, wherein the time delay is based on one or more past time delays of past initiations of the action by the user subsequent to one or more past initiations of the given routine by the user.

16. The method of any one of claims 1-15, wherein the one or more instances of user interface input that initiates the action are received via an additional client device that is linked to but separate from the client device, and the user interface output is rendered via the client device.

17. The method of claim 16, wherein the user interface output comprises a graphical output, the method further comprising: The additional client device is selected for providing the user interface output based on a determination that the client device lacks display capabilities and the additional client device includes display capabilities.

18. The method of claim 1, wherein the one or more instances of user interface input initiating the action comprise spoken words, the method further comprising: identifying a profile of the user based on one or more voice characteristics of the spoken utterance; Wherein identifying the plurality of automated assistant routines stored in association with the user comprises identifying the plurality of automated assistant routines based on the plurality of automated assistant routines being stored in association with the profile of the user.

19. A method implemented by one or more processors, the method comprising: Determine the auto-attendant action; identifying a plurality of automated assistant routines stored in association with a user, each of the automated assistant routines defining a plurality of corresponding actions to be automatically performed via the automated assistant in response to initialization of the automated assistant routine, wherein the automated assistant actions are in addition to the plurality of corresponding actions of the automated assistant routines; comparing the automated assistant action to the plurality of automated assistant routines stored in association with the user; selecting, based on the comparison, a subset of the automated assistant routines stored in association with the user based on at least one of: device topology attributes of the automated assistant routines and the automated assistant actions, temporal attributes of the automated assistant routines and the automated assistant actions, and one or more inconsistencies of the automated assistant routines and the automated assistant actions; determining that the user is interacting with the client device based on sensor data from one or more sensors of the client device; Based on determining that the user is interacting with the client device and based on selecting the subset of the automated assistant routines: causing a user interface output to be rendered via the client device, wherein the user interface output prompts the user to add the automated assistant action to one or more of the automated assistant routines of the subset; receiving an affirmative user interface input in response to the user interface output, wherein the affirmative user interface input indicates a given routine in the subset of the automated assistant routines; as well as In response to receiving the affirmative user interface input: The auto-assistant action is added as an additional action among the plurality of corresponding actions of the given routine to be automatically executed in response to initialization of the given routine.

20. The method of claim 19, wherein the sensor data comprises voice data based on one or more microphones of the client device, and wherein determining that the user is interacting with the client device comprises: determining one or more voice characteristics based on the voice data; as well as The user is identified based on the voice characteristics matching a user profile of the user.

21. The method according to claim 19, Wherein the automated assistant action includes controlling a specific connected device, wherein comparing the automated assistant action to the plurality of automated assistant routines stored in association with the user comprises; determining, based on the device topology of the user's connected devices, that the particular connected device and an additional connected device controlled by at least one of the plurality of corresponding actions of the given routine are assigned to the same physical location in the device topology; and wherein selecting the given routine for inclusion in the subset based on the comparing comprises: The given routine is selected for inclusion in the subset based on a determination that the particular connected device and the additional connected device are both assigned to the same physical location.

22. The method of claim 21, wherein the same physical location is a user-assigned semantic identifier of a room within the user's house.

23. The method of claim 19, wherein the automated assistant action comprises controlling a particular connected device, and wherein determining the automated assistant action comprises: determining, based on the reported status of the particular connected device, that the particular connected device is to be controlled in a particular manner in response to a non-automated assistant interaction; as well as The automated assistant action is determined to cause the particular connected device to be controlled in the particular manner.

24. The method of claim 19, wherein comparing the automated assistant action to the plurality of automated assistant routines stored in association with the user comprises; determining at least one action temporal characteristic of the auto-assistant action based on one or more past performances of the auto-assistant action or a related action by the user; determining at least one routine time characteristic of the automated assistant routine based on one or more past executions of the automated assistant routine; as well as The at least one action time characteristic is compared to the at least one routine time characteristic.

25. The method of claim 19, wherein the automated assistant action comprises interaction with a third party agent controlled by a particular third party, and wherein determining the automated assistant action comprises: determining past performance by the user of a single interaction with a single third party; The automated assistant action is determined based on a relationship between the separate interaction and the automated assistant action.

Images