Systems, methods, and apparatuses for pausing and resuming a virtual assistant

Abstract

Systems, methods, and apparatuses for pausing and resuming virtual assistants are disclosed. Methods, apparatuses, systems, and computer-readable media for allowing an automated assistant routine to be interrupted during execution of a routine are provided. The routine can correspond to a set of actions to be performed under the direction of an automated assistant. When the routine is initialized and the user subsequently issues a command to interrupt the routine, the automated assistant can modify a state identifier for the routine. The state identifier can be stored at a database and other applications and / or devices that are running to complete the routine are notified that the user has requested to interrupt the routine. The database is accessible by one or more devices and / or applications, such as third-party applications, to facilitate providing a medium through which devices and / or applications can check the status of various automated assistant routines.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201880088975.0, filed on January 12, 2018. Technical Field

[0003] This application relates to systems, methods, and apparatus for suspending and resuming virtual assistants. Background Technology

[0004] Humans can engage in human-computer dialogue using interactive software applications referred to herein as “automatic assistants” (also known as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “conversational agents,” etc.). For example, humans (who may be referred to as “users” when interacting with an automatic assistant) can provide commands and / or requests using spoken natural language input (i.e., speech) and / or by providing text-based (e.g., typed) natural language input, which in some cases can be converted to text and then processed. Some automatic assistants can perform multiple actions in response to commands, thus enabling the automatic assistant to undertake routines in response to receiving a specific spoken command. Unfortunately, once the automatic assistant has begun directing the execution of actions involved in a routine, the user may be unable to stop the routine. This can be problematic when the user attempts to have the automatic assistant perform another action or routine while performing the current one. As a result, there will inevitably be a delay before the automatic assistant can begin another action or routine. Furthermore, any processing resources and power expended in executing the initial routine will be wasted. Summary of the Invention

[0005] This disclosure generally relates to methods, apparatus, and computer-readable media (transitory and non-transitory) for allowing users to interrupt routines being executed by an automated assistant to conserve computational resources that would otherwise be wasted executing routines that the user is no longer interested in. The automated assistant may be pre-configured or configurable by one or more users to execute routines comprising a set of actions to be performed under the guidance of the automated assistant. For example, the automated assistant may execute a set of actions associated with a user's typical morning routine in response to a verbal command such as, "Assistant, please perform my morning routine." The set of actions corresponding to a "morning routine" may include: turning on an Internet of Things (IoT) device to provide an audio news broadcast, turning on lights in the home, and displaying the day's schedule on a tablet device. To stop the routine, the architecture through which the routine is executed can be modified to handle commands for interrupting the routine.

[0006] A system for allowing automated assistants to execute routines may include a routine management engine and a state management engine, both of which communicate with a database that may include state information for each routine. The routine management engine can identify the routines executed by the automated assistant so that the database can be updated to reflect the state of each routine. The system may also include a state management engine. The state management engine can handle requests to modify routines executed by the automated assistant and can update the database to indicate whether the routine is the subject of a user request. Any application associated with a particular routine can be delegated the task of checking the database to determine whether the routine has been modified.

[0007] For example, a user can provide an automation assistant with a command to execute a "morning routine." The automation assistant can then cause the routine management engine to write to and / or modify one or more entries in the database to indicate that the routine has been initialized. Because the automation assistant can operate as one or more applications or scripts executing on one or more devices, each application or script can be delegated the task of checking the database to determine if the routine has been modified. In some implementations, when the routine includes a third-party application or device, the third-party application or device can also be delegated the task of checking the database for changes to the routine. For example, after the user has initially provided the command "Assistant, start my morning routine," the user can provide a command to stop the routine, such as "Assistant, stop." Because the user has not yet provided any description of what should be stopped, the automation assistant can cause a search to be performed to identify candidate routines to be stopped.

[0008] To identify candidate routines, the automated assistant can allow the state management engine to access a database to identify routines executed by the automated assistant. The automated assistant can use context data to determine candidate routines to be modified. For example, the automated assistant can determine whether the user who provided the stop command corresponds to the user who initialized the routine identified by the state management engine. In this way, the automated assistant can ensure that the routine being modified is the one previously initialized by the user who provided the stop command.

[0009] In some implementations, context data may include data generated by the automation assistant, applications accessible to the automation assistant, one or more devices accessible to the automation assistant, and / or any other application or device that may collect context data. Context data may include data identifying the user, the time associated with the command, the location where the user issued the command, the device to which the user issued the command, the user's schedule, other users associated with the user who issued the command, and / or any other information that may be used to identify the context in which the user interacted with the automation assistant. For example, a routine identified in a database and previously initialized by the user may be selected as a candidate routine based at least on the correlation between the name of the user who issued the stop command and the name of the user who initialized the routine. Alternatively or additionally, a routine identified in a database and previously initialized at a device that also received the stop command may be selected as a candidate routine.

[0010] The state management engine can generate a state management request based on the identified candidate routines. This request can be provided to the database to indicate that a user has requested the candidate routine to be stopped. In some implementations, in response to the database receiving a state management request, the state identifier associated with the candidate routine can be modified to indicate that the candidate routine has been stopped. In this way, when the automation assistant and / or any other application that has been delegated a task to check the database accesses the database, the candidate routine will be associated with a state identifier indicating that the routine has been stopped. In response, the automation assistant and / or other applications operating in association with the candidate routine will terminate the execution of the actions associated with the candidate routine.

[0011] In some implementations, a routine may be executed by one or more devices and / or one or more applications. For example, a routine may be executed under the guidance of an automated assistant, but the actions associated with the routine may be performed by various different applications and / or different devices. A routine involving the action of booking a ride online to a destination may involve the automated assistant accessing or invoking a third-party application that can instruct a driver to drive to the user's location and take the user to the destination. The third-party application may be delegated the task of accessing a database to periodically check the status of the routine. If the user chooses to stop the routine by issuing a stop command to the automated assistant, the third-party application can confirm the stop command by accessing the database. In response to determining that the user has issued a stop command for the routine, the third-party application may cancel the operation being performed by the third-party application, reverse any changes made by the third-party application, and / or otherwise perform any tasks necessary to stop the completion of the part of the routine.

[0012] As the automation assistant guides the execution of a routine in various ways, the user can modify the routine. For example, the user can issue a stop command to halt the routine. The user can also issue a pause command to pause the routine. The user can then resume a paused routine by issuing a resume command to the automation assistant. In some implementations, the user can issue a rewind command to reverse one or more actions performed under the guidance of the automation assistant during the routine. For example, when a routine involves modifying the settings of an IoT device in the user's home and the user issues a rewind command during the routine, the IoT device's settings can be restored to their previous settings before the routine is initialized. This way, the user does not need to wait until the routine completes before reversing the settings. This can save computing resources and / or power at the affected device, especially when the modified settings correspond to a process that will increase power at the affected device. For example, if the settings include modifying the home temperature, any appliances involved in changing the home temperature can restore their settings much faster than if the routine were left to complete, thus saving energy otherwise applied to complete the routine. Typically, allowing users to interrupt the routine can protect network and computing resources, at least because the actions performed to facilitate the routine can be stopped at any time.

[0013] In some embodiments, a method executed by one or more processors is provided, the method comprising: receiving at an automation assistant interface a routine command corresponding to a request for an automation assistant application to initiate the execution of a routine comprising a set of actions to be performed under the guidance of the automation assistant application; in response to receiving the routine command, providing a first routine write request to a database, wherein the first routine write request is configured to cause a routine status identifier at the database to indicate that a routine is in progress; at the automation assistant interface, receiving an interrupt request from a user; in response to receiving the interrupt request from the user, providing a second routine write request to the database, the second routine write request being configured to modify the routine status identifier to indicate that a routine is at least temporarily interrupted; and when the routine status identifier is modified at the database in response to the database receiving the second routine write request: causing one or more actions in the routine's set of actions to be at least temporarily interrupted.

[0014] In other embodiments, a method executed by one or more processors is provided, the method comprising: receiving a routine write request from an auto-assistant application in response to receiving an input command from a user, the routine write request identifying a routine corresponding to a set of actions initiated by the auto-assistant application in response to receiving the input command; generating a routine state for the routine in response to receiving the routine write request, wherein the routine state is associated with an identifier of the routine and configured to be modifiable during the routine; receiving a state read request from the auto-assistant application, wherein the state read request is received during routine execution and in response to the auto-assistant application receiving a routine interruption request from the user; identifying an identifier of a routine being executed by the auto-assistant application in response to receiving the state read request; providing the auto-assistant application with the identifier of the routine being executed by the auto-assistant application; receiving a state write request from the auto-assistant application, the state write request being configured to at least temporarily interrupt the executing routine; modifying the routine state in response to receiving the state write request; and providing the modified routine state in response to a next routine state request, wherein providing the modified routine state causes the routine to be interrupted at least temporarily.

[0015] In other embodiments, a method executed by one or more processors is provided, the method comprising: receiving a request to perform an action associated with a set of actions in a routine employed by an auto-assistant application, wherein the request is received at a computing device in response to receiving an input command at an auto-assistant interface; initiating an action in response to receiving the request to perform the action; providing a write request to a database accessible to the auto-assistant application, wherein the write request is configured to provide the database with an indication that an action is in progress; providing a read request to the database, the read request being configured to determine a status identifier of a routine, wherein the status identifier is configured to be modifiable during the routine; determining, based on the read request, that the status identifier of the routine has been modified in response to the auto-assistant application receiving an interrupt request; and when the action is in progress: in response to determining that the status identifier of the routine has been modified, at least temporarily suspending the action.

[0016] By interrupting a routine, at least temporarily, these and other methods prevent the consumption of at least those computational and network resources used to facilitate the routine. This improves power efficiency by correspondingly reducing resource usage. For example, in cases where a routine includes one or more outputs, such as audio or video output, interrupting or terminating the routine prevents unnecessary power consumption required for such output. Similarly, in cases where a routine includes network communication (transmission of queries or instructions to remote devices), interrupting or terminating the routine prevents unnecessary power consumption required for such communication. In each case, the processing resources required for the computation, determination, and control of the routine output, as well as the associated processor power consumption, are protected. If the routine is allowed to complete, interrupting or terminating the routine can further protect those computational and network resources used to preserve the actions taken by the routine. This may include resources for processing at least one additional user input instructing the reversal of the routine, and may include resources for processing multiple user inputs for individually reversing multiple actions of the routine execution.

[0017] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features. In some embodiments, providing a second routine write request to the database may include selecting a routine status identifier from other identifiers stored at the database, the other identifiers corresponding to different routines that can be executed by the automation assistant application. In some embodiments, modifying the routine status identifier may include generating a value for the routine status identifier indicating that at least one action in the action set has been stopped. In some embodiments, the action set includes third-party actions associated with a third-party application. In some embodiments, when a third-party action has been completed before an interruption request is received: the routine status identifier may be modified to indicate that the third-party action has been completed. In some embodiments, a method may include providing a notification to the user indicating that the routine has been stopped and the third-party action has been completed. In some embodiments, the automation assistant interface may be part of a computing device, and the routine status identifier may be stored at a database provided at a separate computing device.

[0018] In some implementations, a method may include generating context data in response to receiving a routine command; and identifying a routine state identifier based on the context data in response to receiving an interrupt request. In some implementations, the identifier identifying the routine may include performing a query to select an identifier from other identifiers stored in a database, wherein the other identifiers may correspond to different routines that can be executed by an automation assistant application. In some implementations, modifying the routine state may include generating a value indicating that at least one action in a set of actions has been stopped. In some implementations, the set of actions may include third-party actions associated with a third-party application. In some implementations, a method may include modifying the routine state to indicate that the third-party action has been completed when a state write request has been received.

[0019] In some implementations, a method may include causing an auto-assistant application to provide a notification to a user indicating that a routine has stopped and a third-party action has been completed. In some implementations, the routine interruption request may be received at the auto-assistant interface of a computing device, and the routine state may be stored in a database provided on a separate computing device. In some implementations, the identifier identifying the routine may be based on context data available to the auto-assistant application. In some implementations, the action may be a third-party action associated with a third-party application. In some implementations, a method may include modifying a status identifier to indicate that the third-party action has been completed when the third-party action has been completed before the auto-assistant application receives the interruption request. In some implementations, a method may include causing the auto-assistant application to provide a notification to the user indicating that a third-party action has been completed. In some implementations, providing read requests to the database may include providing multiple read requests to the database during the execution of the action. In some implementations, a method may include causing a separate action to be executed to undo the action when the action has been completed and the routine's status identifier indicates that the routine has been reversed. In some implementations, a method may include determining that the routine's status identifier has been modified to indicate that the routine will continue; and causing the action to be completed.

[0020] Additionally, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein said instructions are configured to cause any of the methods described above to be performed. Some embodiments further include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above.

[0021] It will be understood that all combinations of the foregoing and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0022] Figure 1 The diagram illustrates a system that allows users to initialize and interrupt routines executed under the guidance of an automation assistant.

[0023] Figure 2 Provides a state diagram illustrating how a user can interrupt a routine executed by the automated assistant, thereby halting various actions of the routine.

[0024] Figure 3 The diagram illustrates a method for managing the database accessible to the automation assistant so that modifications can be made to routines executed by the automation assistant during routine execution.

[0025] Figure 4 The diagram illustrates a method for controlling the Auto Assistant to at least temporarily suspend the routines it is executing.

[0026] Figure 5 The diagram illustrates a method for interrupting actions performed by a device and / or application under the guidance of an automated assistant.

[0027] Figure 6 This is a block diagram of an example computer system. Detailed Implementation

[0028] Figure 1The illustration depicts a system 100 for allowing users to initialize and interrupt routines executed under the guidance of an automated assistant 104. System 100 may include a remote device 112, such as server device 102, which may host the automated assistant 104. In some embodiments, the automated assistant 104 may be hosted on server device 102 and / or one or more computing devices capable of hosting an assistant application. For example, in some embodiments, the automated assistant 104 may be hosted on client device 116 and / or server device 102. Processes and / or functions performed by the automated assistant 104 may be performed by server device 102 or distributed across separate devices. For example, a user may provide input, such as voice commands or text input, on an assistant interface 118 on client device 116. Input from the user may be transmitted from client device 116 to server device 102 via network 114, such as a private network (e.g., a peer-to-peer network) or a public network (e.g., the Internet). When the input is a voice command or other audible input from the user, the input can be received at a speech-to-text engine 108, which may be part of or separate from the automation assistant 104. The speech-to-text engine 108 can convert the input into text, which can be parsed by the automation assistant 104 to determine how adequately the automation assistant 104 can respond to the input from the user.

[0029] The automated assistant 104 may employ one or more machine learning algorithms to learn or otherwise adapt to user requests, preferences, and / or how users otherwise interact with the automated assistant 104. Interactions between one or more users and the automated assistant 104 may be stored as routine data 110, user data 106, and / or context data 122, all of which may be stored at server device 102 and / or client device 116. Routine data 110 may, for example, provide corresponding data identifying one or more inputs received from one or more users and one or more routines to be executed by the automated assistant 104. A routine may correspond to a set of actions to be performed by the automated assistant 104, a separate application, a device separate from server device 102, and / or any other application and / or device capable of interacting with the automated assistant 104.

[0030] The routine can be executed under the guidance of the automation assistant 104 and can be controlled by one or more users during routine execution. The state of the routine can be stored at a remote device 136, which may include a database 138. The database 138 may include a routine identifier 140, a state identifier 142, user data 144, and / or context data 146. Initially, when a user requests to initialize the routine, the assistant interface 118 may receive a command. Optionally, the client device 116 may generate and / or store user data 106 and / or context data 122, which may be associated with the command used to initialize the routine. The automation assistant 104 can confirm the command received at the assistant interface 118 and generate routine data 110 for the routine. In some embodiments, the automation assistant 104 may receive a command from the user and execute a query to identify the routine corresponding to the command. Subsequently, when a routine has been identified, the automation assistant can enable the routine management engine 124 to access the database 138 and modify the status identifier 142 associated with the routine to indicate that the routine is active.

[0031] When a user initializes a routine at assistant interface 118, the auto-assistant can identify a set of actions associated with that routine and authorize each action in that set accordingly. For example, at least one action in the action set can be performed by application 130 hosted on a separate computing device 128. Therefore, auto-assistant 104 can communicate with application 130 to cause application 130 to begin performing the action. Auto-assistant 104 can optionally delegate tasks to application 130 to access database 138 once or multiple times (e.g., periodically) during action execution to determine whether the status identifier 142 associated with the routine has been modified. In some implementations, when the status identifier 142 stored in database 138 has been modified, application 130 can store and update the routine state 134. If the routine state 134 changes during action execution, any action data 132 associated with the action being performed by application 130 can be modified based on the change in routine state 134. For example, if application 130 has been assigned a task to perform an action such as changing the settings of appliances in a user's home, action data 132 can correspond to the settings changed by the application. If the user issues a command to stop the routine associated with the action, action data 132 can be modified to restore the appliances to the settings they had before the action was initiated or before the routine was initialized.

[0032] When a user attempts to stop a routine after the Auto Assistant 104 has initialized it, the command to stop the routine may not explicitly identify the routine. Therefore, to identify the routine the user is referencing, the Auto Assistant 104 can use user data 106 and / or context data 122 to identify candidate routines that will be stopped in response to a command from the user. For example, when the user initializes a routine, user data 106 and / or context data 122 can be collected at client device 116 and provided to server device 102 and / or remote device 136. As a result, database 138 can identify multiple routines and any user data 144 and / or context data 146 that can be associated with each of the multiple routines. When the user provides an interrupt command at assistant interface 118, the Auto Assistant 104 can receive the interrupt command and allow the routine management engine 124 to identify candidate routines.

[0033] Candidate routines can be identified based on an interrupt command, user data 144 associated with each routine, context data 146 associated with each routine, and / or any other data that may be associated with the routine. For example, context data 146 may identify a timestamp associated with the routine, and thus a candidate routine could be the most recently initialized routine by the Auto Assistant 104. Alternatively, context data 146 may identify the device and / or location where the routine was initialized and compare that device and / or location with the most recently initialized device and / or location where the user provided the interrupt command. Thus, a candidate routine to be interrupted or otherwise modified by an interrupt command could be a routine initialized at the same device and / or the same location where the user provided the interrupt command. For example, if a user initializes a routine at a cellular device and subsequently provides an interrupt command at the same cellular device, context data 146 would provide a mapping between cellular devices and routines. This mapping can then be identified by a routine management engine 124, which can then select the routine as a candidate routine to be modified by the interrupt command.

[0034] Once a candidate routine has been identified, the state management engine 126 and / or the automated assistant 104 can provide a write request to the database 138 to modify the state identifier 142 associated with the candidate routine. The state management engine 126 can, for example, modify the state identifier 142 associated with the candidate routine to indicate that the routine has been stopped. As a result, any application and / or device that has been assigned a task to perform actions to facilitate the routine can confirm that the routine has been stopped under user guidance. In some implementations, the user can provide an interrupt command to the assistant interface 118 to modify the routine's state in various ways. For example, the user can provide an interrupt command to rewind the routine, pause the routine, stop the routine, delay the routine until certain conditions are met, and / or modify any other aspect of the routine.

[0035] In some implementations, a user can provide an interrupt command to delay a routine until certain conditions are met. The user can specify the conditions in the interrupt command. For example, when a user leaves their office, they can provide an initial command to initialize a routine such as “Assistant, please start my home routine.” In response, the automated assistant 104 can authorize action on the “home routine.” Subsequently, the user can make another commitment to prevent them from returning home at their expected time. To stop a user-initiated routine, at least until the user arrives home, the user can provide an interrupt command such as “Assistant, please delay this routine until I arrive home.” In response, the routine management engine 124 can identify the routine the user is referencing and modify the status identifier 142 associated with that routine to indicate that the routine has been delayed until the user arrives home (e.g., “Status_Identifier = home_routine.delay-until_location:home”).

[0036] The context data 146 associated with the routine can identify the user's location as their office. When the user arrives home, the automation assistant 104 can update the context data 146 and modify the stored user location to indicate that the user is at home. The state management engine 126 can confirm that the user has arrived home and modify the state identifier 142 to indicate that the routine is active. In response, any applications and / or devices that have been authorized to facilitate the routine can continue to execute their authorized actions in response to the state identifier 142 changing from delayed to active. In this way, the user does not have to repeat the initial command (e.g., "Assistant, start the home routine") to the automation assistant 104 when returning home to start the routine. This saves computational resources because the audio of the voice command does not have to be reprocessed, and the action does not have to be reauthorized to each application and / or device involved in executing the routine.

[0037] Figure 2A state diagram 200 is provided, illustrating how user 204 can interrupt a routine executed by automated assistant 208, thereby interrupting the individual actions of the routine. Initially, user 204 may provide input command 216 to automated assistant 208, which may be hosted on one or more computing devices 212. Input command 216 may correspond to natural language input on an automated assistant interface that can be connected to or communicate with computing device 212. Furthermore, input command 216 may correspond to a request for automated assistant 208 to execute a routine comprising a set of actions. This routine may be pre-configured at automated assistant 208, for example, by user 204 or the manufacturer of automated assistant 208. In response to receiving input command 216, automated assistant 208 may provide write request 222 to a separate computing device 214. Computing device 214 may host database 210, which may be accessed by automated assistant 208, and / or by any other application and / or device delegated with actions to facilitate the completion of the routine. For example, a third-party application 218 hosted on computing device 220 can access database 210 and perform actions assigned to the third-party application 218 by the automated assistant 208. The term "third party" can refer to an entity different from the entity that originally manufactured and / or configured the automated assistant 208.

[0038] In response to receiving input command 216, the automation assistant 208 may initialize routine actions 224. These actions may correspond to routines identified by input command 216 and / or by the automation assistant 208. When the automation assistant 208 initializes routine actions 224, it may also provide a write request 222 to the database 210. The write request 222 may provide an entry at the database 210 identifying a routine that has already been initialized at the automation assistant 208. In some embodiments, the write request 222 may modify the state of the routine provided at the database 210 to indicate that the routine is currently in progress. When a routine includes actions to be performed by a third-party application 218, the automation assistant 208 may provide a start action command 226 to the third-party application 218. In some embodiments, by providing the start action command 226, the automation assistant 208 may delegate the task of checking the state of the routine to the third-party application 218 while it is performing an action assigned to it. For example, when third-party application 218 is performing an action, third-party application 218 can provide read request 228 to database 210. In response to read request 228, third-party application 218 can identify the state of the routine. At the moment in state diagram 200 when read request 228 is provided to database 210, the state of the routine can be indicated as "active".

[0039] While a third party 218 is performing an action and the automation assistant 208 is directing the execution of a routine, the user 204 may provide an interrupt request 230 to the automation assistant 208. The interrupt request 230 may be a command to stop the program, pause the routine, rewind the routine, delay the routine until a specific time or until a condition is met, and / or any other command that may interrupt the routine. In response to receiving the interrupt request 230, the automation assistant 208 may provide a write request 232 to the database 210. The write request 232 may change the status of the routine stored in the database 210 from "active" to "stopped" or any other status identifier corresponding to the interrupt request 230. Furthermore, in response to receiving the interrupt request 230, the automation assistant 208 may interrupt the routine action 234 so that the automation assistant 208 and / or any other application or device that has been assigned a task to perform the action cease further execution of the action.

[0040] Because the third-party application 218 has been assigned the task of checking for changes in the routine's state in the database 210, the third-party application 218 can provide a read request 236 to the database 210 during the execution of an action. As a result, the third-party application 218 will be notified that the routine's state has changed from an active state to a different state, such as "stopped". In response to determining that the routine's state has changed, the third-party application 218 can interrupt its action 238 until the automation assistant 208 receives further instructions and / or the routine's state has changed at the database 210. Optionally, the automation assistant 208 can provide a notification 240 to the user indicating that the routine's state has changed, and thus, the action performed to facilitate the routine has been interrupted according to the interruption request 230. For example, if the user 204 provides a request to stop the routine, the notification 240 can confirm that the routine's state has changed to "stopped".

[0041] Figure 3The illustration illustrates a method 300 for managing a database accessible to an automated assistant to allow modification of routines performed by the automated assistant during routine execution. Method 300 can be performed by one or more applications and / or devices capable of interacting with and / or hosting the database. Method 300 may include an operation 302 receiving a routine write request in response to the automated assistant receiving an input command from a user. The routine write request may identify a routine comprising a set of actions to be performed under the guidance of the automated assistant. The input command may be natural language input provided to the automated assistant interface of a computing device. The automated assistant interface may be, for example, a microphone, camera, touchscreen display, peripheral device, and / or any other interface that can be used to communicate with the device application. In some embodiments, the input command may be spoken natural language input corresponding to a routine used to prepare a user for flight. For example, a user may provide an input command such as, “Assistant, prepare for my flight.” The input command may correspond to a routine comprising a set of actions, which may include actions such as setting a thermostat in the user’s home, downloading a podcast for the user to listen to during the flight, and booking a ride to the airport online. The thermostat can be controlled by an automation assistant, podcast downloads can be performed through third-party apps, and online ride-hailing bookings can also be performed through third-party apps. Therefore, in order to execute routines, the automation assistant will interact with other apps and devices. Additionally, the automation assistant can delegate tasks to each app and device involved in the routine to check for updates to the routine's status in the database.

[0042] Method 300 may further include operation 304 of generating a routine state in response to receiving a routine write request. The routine state may be associated with an identifier for the routine and may be configured to be modifiable during the routine. For example, the routine identifier may be "flight routine," at least based on the name of a routine initialized by the user via an input command to the automation assistant interface. In some implementations, the routine identifier may be stored in a database in association with the routine state, which may identify the state of the routine and / or one or more actions in a set of actions corresponding to the routine. For example, when the routine is initially initialized by the user, the routine state may be "active" or "in progress."

[0043] Method 300 further includes operation 306 of receiving a status read request from an auto-assistant and / or a status management engine. The status read request may be received during routine execution and may be received in response to the auto-assistant receiving a routine interruption request from the user. The status read request may be received at a database to enable the auto-assistant and / or any other application or device associated with the routine to identify a routine being performed or executed under the guidance of the auto-assistant. The routine interruption request may correspond to spoken natural language input received at the auto-assistant interface. A routine interruption request may be, for example, “Assistant, stop preparing my flight.” To identify the routine the user is referencing, the auto-assistant may provide the status read request to the database. In some embodiments, the status read request may be generated by a status management engine accessible to the auto-assistant and / or any other application or device associated with the routine. For example, the auto-assistant may convert the routine interruption request from speech to text and provide the corresponding command to the status management engine. In response, the status management engine may provide a request to the database to identify the ongoing routine.

[0044] Method 300 may further include operation 306 in response to receiving a status routine request to identify an identifier of a routine being executed under the guidance of the automation assistant. For example, the status management engine may generate a status routine request to identify routines associated with the term "flight". If the flight routine is the only routine associated with the term "flight", the status management engine will be able to quickly identify the flight routine in the database. However, if multiple routines are associated with the term "flight", the status management engine can filter the routine using context data. In this way, the status management engine can sort pending routines based on their similarity to the context data and identify the routine that most closely matches the routine the user intends to modify or discontinue.

[0045] Method 300 may further include operation 308 of providing an identifier of a routine executed by the automated assistant to the automated assistant. The identifier may include one or more identifiers that allow the automated assistant to determine the name of the routine, the status of the routine, and / or any other information that may be associated with the routine executed by the automated assistant. Providing the identifier may include allowing access to the identifier at a database.

[0046] Method 300 may further include operation 310 of receiving a state write request from the auto-assistant and / or state management engine, configured to at least temporarily interrupt a routine executed under the guidance of the auto-assistant. For example, the state write request may include instructions for modifying the state of the routine. If the user has provided a routine interruption request to stop the routine, the state write request may be configured to modify the state of the routine to indicate that the routine has stopped or has been instructed to stop.

[0047] Method 300 may further include operation 312 of modifying the state of a routine in response to receiving a state write request. The state of the routine may be modified from "active" to "stopped". In this way, any application or device that has been delegated to access the database to determine the state of a routine will be able to confirm that the routine has stopped or has been instructed to stop. This saves computing resources that would otherwise be wasted if the user could not stop the routine while it was running.

[0048] Method 300 may further include operation 314 of providing a modified routine status in response to a next routine status request. The routine status request may be provided by an automated assistant, application, device, and / or any other device or application that may be associated with the routine. For example, the routine status request may be provided by a third-party application that has been delegated to a user to book a ride online from their home to the airport. When the third-party application receives or identifies the modified routine status, it may stop the activity it is performing to facilitate the routine. For example, if the third-party application is attempting to contact a driver, it may stop providing queries to a remote device associated with the driver. Alternatively, if the routine status request is provided by an IoT device that can modify the temperature of the user's home, the IoT device may restore the previous temperature in response to confirmation of the modified routine status.

[0049] Figure 4 The illustration illustrates a method 400 for controlling an automated assistant to at least temporarily interrupt a routine being executed by the automated assistant. Method 400 can be performed by one or more applications or computing devices capable of interacting with or providing the automated assistant. Method 400 may include an operation 402 at the automated assistant interface receiving a routine command corresponding to a request to initiate the execution of a routine by the automated assistant application. This routine may be executed under the guidance of the automated assistant and includes a set of actions to be performed by the automated assistant, one or more applications, and / or one or more computing devices. The automated assistant interface may be a device and / or application connected to a computing device being operated by a user. For example, the automated assistant interface may include a microphone, touchscreen display, peripheral device, camera, and / or any other means through which the user can control the application or device. In some embodiments, the routine command received at the automated assistant interface may be “Assistant, prepare for the party.” The aforementioned routine command may correspond to a set of actions, including modifying home lighting settings, playing a party playlist, and setting reminders for movies to be shown. The set of actions may include causing one or more applications and / or devices to perform actions within that set.

[0050] Method 400 may further include operation 404 of providing a first routine write request to the database in response to receiving a routine command. The first routine write request may be provided by an automation assistant and / or a routine management engine associated with the automation assistant. The first routine write request may be provided to the database to provide a record or identifier for a user-requested routine executed under the guidance of the automation assistant. For example, when a user requests the automation assistant to execute a “party routine,” the first routine write request may be configured to create an entry identifying the “party routine” at the database. In some implementations, a routine status or condition may also be created at the database in response to the database receiving the first write request. The routine status may be, for example, “in progress,” “active,” and / or any other identifier that may be used to describe the routine status.

[0051] Method 400 may further include operation 406 of receiving an interruption request from the user at the auto assistant interface. The interruption request may correspond to spoken natural language input provided to the auto assistant interface. For example, the interruption request may correspond to a spoken command such as "Assistant, return" or "Assistant, rewind." The aforementioned command may cause the auto assistant to reverse any action in the set of actions already performed to advance a routine rather than stop it. In this way, the user can not only stop the continuation of a routine but also restore specific applications and devices to their operating settings before initializing the routine.

[0052] Method 400 may further include operation 408 of providing a second routine write request to the database in response to receiving an interruption request from the user. The second write request may be configured to modify a routine status identifier to indicate that the routine has been interrupted at least temporarily. The second write request may correspond to a verbal command from the user and cause the database to include an entry identifying the status of the routine. For example, the second write request may change the status identifier corresponding to the routine from "active" to "rewind" or "revert," or any other identifier indicating that the routine has been reversed. In this way, any application and / or device that has been assigned the task of checking the status of the routine during the routine can confirm the change in the routine's status. The change in status may cause the application and / or device responsible for performing the routine's actions to reverse any completed actions.

[0053] Method 400 may further include operation 408, which causes one or more actions of the routine to be interrupted at least temporarily when the routine status identifier has been modified. The routine status identifier may be modified in response to the database receiving a second routine write request. For example, one or more actions may include changing home lighting settings (e.g., dimming the lights), so that in response to the routine status identifier being modified, the lighting settings may be restored to their original settings (e.g., fully on), just as they were before they received the request to execute the routine. Alternatively, one or more actions may include setting a reminder to watch a movie, so that in response to the routine status identifier being modified, the reminder may not be created, or if created, the reminder may be deleted.

[0054] Figure 5 The illustration illustrates a method 500 for interrupting an action performed at a device or application under the guidance of an automation assistant. Method 500 can be performed by one or more applications and / or devices capable of performing one or more actions of a routine under the guidance of an automation assistant. Method 500 may include an operation 502 of receiving a request to perform an action associated with a set of actions in a routine adopted by the automation assistant. This request may be provided by the automation assistant, which has received a command from the user to perform the routine. For example, the automation assistant may receive a verbal command, such as “Assistant, please prepare dinner,” at the automation assistant interface of a computing device. In response, the automation assistant may convert the verbal command into text, identify the routine corresponding to the text, and communicate with any device and / or application configured to perform the action to facilitate the identified routine. For example, the device receiving the request to perform the action may be associated with the action of ordering takeout online to the user's home.

[0055] Method 500 may further include operation 504 of initiating an action in response to receiving a request to perform an action. Initiating an action may include processing data received from the automation assistant and / or the user, communicating with a separate application and / or device, and / or performing any other operations to facilitate a routine. For example, when the action corresponds to ordering takeout online, initiating the action may include accessing the user's account and providing the user with a variety of food options to choose from. Data corresponding to the food options may then be provided to the automation assistant so that the automation assistant can present the food options to the user, allowing the user to select from the food options.

[0056] Method 500 may optionally include an operation 506 of providing a write request to a database accessible to the automation assistant. The write request may provide the database with an indication that an action is in progress. In this way, the automation assistant and / or any other applications and / or devices associated with the routine can access the database and determine that the requested action is in progress. For example, in response to receiving a request to initialize a routine, the automation assistant may provide a request to each application and / or device involved in the routine to check the database to determine whether the state or condition of the routine has changed. In this way, if the state or condition of the routine has changed, each application and / or device involved can modify one or more actions that have been assigned to them to respond quickly to changes in the state or condition of the routine.

[0057] Method 500 may further include operation 508 of providing a read request to the database. The read request may be configured to determine a status identifier for a routine. In some implementations, the read request may be provided by an application and / or device that has been delegated to perform the action of the routine. Alternatively, the read request may be provided at periodic intervals and / or in response to one or more steps in completing the action. If the user has provided an interrupt command to the automation assistant interface, the status identifier of the routine can be modified, and therefore the read request will result in providing a status identifier such as "stopped" or "paused".

[0058] Method 500 may further include operation 510, which determines that the status identifier of the routine is modified based on a read request in response to the automation assistant receiving an interrupt request. For example, the automation assistant may receive an interrupt request, such as "Assistant, please stop." In response to the interrupt request, the automation assistant may cause the state management engine to provide a write request to the database to modify the state of the routine. Thereafter, any application and / or device that has been assigned actions to facilitate the routine can determine that the state of the routine has been modified and can adjust the execution of its actions accordingly. For example, when the state of the routine has been modified to indicate that the routine has stopped, each application and / or device may stop executing the corresponding action if the action has not yet been completed or is in progress.

[0059] Method 500 may further include operation 512, which at least temporarily interrupts the action in response to determining that the status identifier of the routine has been modified. Operation 512 can be performed while the action is in progress. For example, when the action corresponds to ordering takeout online and the action has not yet been completed, the application and / or device responsible for the action can stop the execution of the action. Specifically, if the user's account has been accessed to determine the user's food preferences, some actions have already been performed. However, because the state of the routine has been modified, no further steps will be taken to facilitate the action and the routine, and therefore any determined food preferences can be ignored. By allowing the user to interrupt the routine, network and computing resources can be protected, at least because actions performed to facilitate the routine can be stopped at any time. If the automated assistant and / or any other application or device involved in the routine cannot stop the execution of certain actions to facilitate the routine, these actions will waste resources, especially if the user is not interested in the outcome of using the actions.

[0060] Figure 6 This is a block diagram of an example computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, memory 625 and a file storage subsystem 626; an interface output device 620; a user interface input device 622; and a network interface subsystem 616. The input and output devices allow users to interact with the computer system 610. The network interface subsystem 616 provides an interface to an external network and couples to corresponding interface devices in other computer systems.

[0061] User interface input device 622 may include a keyboard; pointing devices such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touchscreen integrated into a display; an audio input device such as a voice recognition system; a microphone; and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and the manner in which information is input into the computer system 610 or a communication network.

[0062] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visual images. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and the manner in which information is output from computer system 610 to a user or another machine or computer system.

[0063] Storage subsystem 624 stores the programming and data construction that provide some or all of the functionality of the modules described herein. For example, storage subsystem 624 may include selected aspects of methods 300, 400, and 500 and / or logic that implements client device 116, remote device 112, server device 102, computing device 128, remote device 136, computing device 206, computing device 212, computing device 220, computing device 214, and / or one or more of any other devices or applications discussed herein.

[0064] These software modules are typically executed by processor 614 alone or in combination with other processors. The memory 625 used in storage subsystem 624 may include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 for storing fixed instructions. File storage subsystem 626 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 626 within storage subsystem 624 or on other machines accessible to processor 614.

[0065] Bus subsystem 612 provides a mechanism for enabling various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0066] The computer system 610 can be of various types, including workstations, servers, computing clusters, blade servers, server groups, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 6 The description of the computer system 610 depicted herein is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computer system 610 may have... Figure 6 The computer system depicted in the text has more or fewer components.

[0067] Where the system described herein collects or may use personal information about a user (or, as often referred to herein, a “participant”), the user may be given the opportunity to control whether a program or feature collects user information (e.g., information about the user’s social networks, social behavior or activities, occupation, user preferences, or the user’s current geographic location), or to control whether and / or how content is received from a content server that may be more relevant to the user. Additionally, certain data may be processed in one or more ways before storage or use, such that personally identifiable information is removed. For example, a user’s identity may be treated in a way that makes it impossible to determine any personally identifiable information for that user, or geographic location information (such as city, zip code, or state level) may be generalized to make it impossible to determine the user’s specific geographic location. Therefore, the user has control over how information about the user and / or information that has been used is collected.

[0068] While several embodiments have been described and illustrated herein, various other means and / or structures may be utilized for performing functions and / or obtaining results and / or one or more advantages described herein, and each of these variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on the specific application or use of the teachings. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using no more than conventional experimentation. Therefore, it is to be understood that the foregoing embodiments are presented as examples only, and embodiments may be practiced in ways different from those specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of this disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure if these features, systems, articles, materials, kits, and / or methods do not contradict each other.

Claims

1. A method implemented by one or more processors, the method comprising: In response to initiating the execution of a routine that includes a set of actions to be performed: An automated assistant provides a routine write request to the database, wherein providing the routine write request causes a routine status identifier at the database to indicate that the routine is in progress, and The automated assistant provides a start action command to a third-party application, which causes the third-party application to: Initiating the execution of a third-party action from the action set of the routine, and During the execution of the third-party action, a third-party read request is provided to the database to read the routine status identifier and to identify the state of the routine based on the routine status identifier. Specifically, when the status indicates that the routine is in progress, the third-party application continues to execute the third-party action, and when the status indicates that the routine is temporarily interrupted, the third-party application at least temporarily interrupts the execution of the third-party action. The automatic assistant receives an interrupt request from the user via its interface and during the execution of the third-party action in the routine. In response to receiving the interrupt request: The automated assistant provides an additional routine write request to the database, wherein providing the additional routine write request causes the routine status identifier to be modified to indicate that the routine is at least temporarily interrupted.

2. The method according to claim 1, wherein, The interrupt request is a request to stop the routine, and further includes: In response to providing the additional routine write request: The automated assistant interface provides a notification indicating that the routine has been stopped.

3. The method according to claim 1, wherein, The interrupt request is a request to suspend the routine, and further includes: In response to providing the additional routine write request: The automated assistant interface provides a notification indicating that the routine has been paused.

4. The method of claim 3, further comprising: The automatic assistant interface receives a recovery request from the user after receiving the interruption request; In response to receiving the recovery request: The automated assistant provides a further routine write request to the database, wherein providing the further routine write request causes the routine status identifier to be modified to indicate that the routine is in progress.

5. The method of claim 1, further comprising: It is determined that the interrupt request corresponds to the routine; The additional routine write request is provided in response to determining that the interrupt request corresponds to the routine.

6. The method according to claim 5, wherein, Determining that the interrupt request corresponds to the routine includes: It is determined that the user who provided the interrupt request is the same user who caused the execution of the routine.

7. The method according to claim 5, wherein, Determining that the interrupt request corresponds to the routine includes: The Auto Assistant interface through which the interrupt request is received is the Auto Assistant interface of the client device through which the execution of the routine is initiated.

8. The method according to claim 1, wherein, The automated assistant interface is integrated with the client device, and the database is provided on a server device located remotely from the client device.

9. A method implemented by one or more processors, the method comprising: In response to initiating the execution of a routine that includes a set of actions to be performed, a routine write request is received from the automation assistant that identifies the routine and indicates that the routine is in progress. In response to receiving the routine write request, an entry is generated in the database, the entry including a routine identifier that identifies the routine and a status identifier that indicates the routine is in progress; During the execution of a third-party action on the routine by a third-party application, a third-party read request indicating the routine identifier is received from the third-party application; In response to the third-party read request, and based on the status identifier of the entry indicating that the routine is in progress, the third-party application is provided with an indication that the routine is in progress; Following the indication that the routine is in progress: In response to the Auto Assistant receiving an interruption request, an Additional Routine Write Request is received from the Auto Assistant, the Additional Routine Write Request identifying the routine and indicating that the routine has been interrupted; In response to receiving the additional routine write request, the status identifier of the entry in the database is modified to indicate that the routine has been interrupted; After modifying the status identifier of the entry: During the execution of the third-party action of the routine by the third-party application, an additional third-party read request indicating the routine identifier is received from the third-party application; In response to the additional third-party read request and based on the status identifier indicating that the routine has been interrupted, the third-party application is provided with an indication that the routine has been interrupted, wherein the indication that the routine has been interrupted causes the third-party application to at least temporarily suspend the execution of the third-party action.

10. The method according to claim 9, wherein, The interruption request is a request to suspend the routine, and wherein modifying the status identifier of the entry in the database to indicate that the routine is interrupted in response to receiving the additional routine write request includes modifying the status identifier to indicate that the routine is suspended.

11. The method of claim 10, further comprising: After modifying the status identifier of the entry, a further routine write request is received from the automation assistant; as well as In response to receiving the further routine write request, the status identifier of the entry in the database is further modified to indicate that the routine is in progress.

12. The method of claim 11, further comprising: Following further modification of the status identifier of the entry: When the execution of the third-party action of the routine by the third-party application is suspended, an additional third-party read request indicating the routine identifier is received from the third-party application. as well as In response to the additional third-party read request, and based on the status identifier indicating that the routine is in progress, the third-party application is provided with an indication that the routine is in progress.

13. A system for controlling a virtual assistant, comprising: Memory that stores instructions; One or more processors, said one or more processors being operable to execute the instructions to: In response to initiating the execution of a routine that includes a set of actions to be performed: A routine write request is provided to the database, wherein providing the routine write request causes a routine status identifier at the database to indicate that the routine is in progress, and Provide a start action command to a third-party application, the start action command causing the third-party application to: Initiating the execution of a third-party action from the action set of the routine, and During the execution of the third-party action, a third-party read request is provided to the database to read the routine status identifier and to identify the state of the routine based on the routine status identifier. Specifically, when the status indicates that the routine is in progress, the third-party application continues to execute the third-party action, and when the status indicates that the routine is temporarily interrupted, the third-party application at least temporarily interrupts the execution of the third-party action. Receive interruption requests from the user via the automated assistant interface and during the execution of the third-party action in the routine; In response to receiving the interrupt request: Provide an additional routine write request to the database, wherein providing the additional routine write request causes the routine status identifier to be modified to indicate that the routine is at least temporarily interrupted.

14. The system according to claim 13, wherein, The interrupt request is a request to suspend the routine, and wherein one or more of the processors are operable to execute the instructions further: In response to providing the additional routine write request: The automated assistant interface provides a notification indicating that the routine has been paused.

15. The system according to claim 14, wherein, One or more of the processors are operable to execute the instructions further: The automatic assistant interface receives a recovery request from the user after receiving the interruption request; In response to receiving the recovery request: A further routine write request is provided to the database, wherein providing the further routine write request causes the routine status identifier to be modified to indicate that the routine is in progress.

16. The system according to claim 13, wherein, One or more of the processors are operable to execute the instructions further: It is determined that the interrupt request corresponds to the routine; The additional routine write request is provided in response to determining that the interrupt request corresponds to the routine.

Images