Assistant adaptation of graphical user interface to guide interaction with user to fulfill user requests

Abstract

Implementations set forth herein relate to an automated assistant that can actively identify and complete tasks that may be associated with a user having scheduled activities assisted by the automated assistant. Tasks may be identified and completed before the user has scheduled the time at which the automated assistant assists the user, thereby eliminating certain manual tasks that the user may otherwise perform at the scheduled time. When an activity involves communication with a separate entity, such as another person and / or organization, an automated assistant may initialize communication with the entity prior to a scheduled time when a user requests assistance. The customized GUI may be rendered at a device supporting the assistant to provide the user with an in-progress state of completing various tasks associated with the scheduled activity.

Description

Background Art

[0001] A human may engage in a human-computer conversation with an interactive software application referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] Automated assistants are often used to handle quick tasks that users can otherwise handle via their personal computing devices (e.g., mobile phones, laptops, etc.). For example, a user may request their automated assistant to set a reminder for the next day, and in response, the automated assistant may generate a calendar entry that may be stored by a calendar application on their personal computing device. Although this interaction may save time for the user, the user may still manually perform other tasks that may be associated with the reminder, and those tasks may also involve interactions between the user and their device. For example, a user may call their automated assistant to set a reminder about "scheduling a plumber", which may involve the user searching for plumbing services in various applications and calling each search result to schedule an available plumber. However, when the automated assistant is initialized to render a reminder, the automated assistant may only repeat an explicit request from the user (e.g., "Schedule a plumber.") without any supplemental data and / or assistance (e.g., assistance in scheduling a plumber) on the subject of the reminder. As a result, some interactions with the automated assistant may be duplicative and waste computing resources, particularly where the user immediately invokes the automated assistant (e.g., “Assistant, call my plumber.”) and / or another application for assistance after confirming an alert from the automated assistant. Summary of the invention

[0003] The implementation described herein relates to an automated assistant that can provide an adaptable graphical user interface (GUI) in response to a user request related to certain tasks. The adaptable GUI can be dynamically updated based on data provided to the automated assistant from one or more other applications and / or devices to facilitate the elimination of repetitive tasks that may be associated with certain reminders and / or other requests. For example, a user can provide the automated assistant with a request to set a reminder for the user to perform an activity that may involve multiple different tasks. The request can be embodied in a spoken utterance, such as "Assistant, remind me to bake bread tomorrow." This request can refer to an activity of baking bread, which typically can involve multiple tasks, such as: finding a bread recipe, ensuring that all ingredients are available, arranging a time to prepare the recipe, and finally making the bread. Therefore, in response to the automated assistant receiving the spoken utterance, multiple different reminders and / or options can be presented to the user to complete the task within the time period requested by the user (e.g., from the moment the user provides the spoken utterance to "tomorrow").

[0004] For example, in response to receiving a spoken utterance, the automated assistant may identify one or more tasks that may be associated with a request embodied in the spoken utterance. Tasks may be identified using one or more heuristic processes and / or one or more trained machine learning models. In some implementations, the automated assistant may process data from one or more different data sources (e.g., the Internet, one or more other applications, instructional videos, etc.) to determine multiple tasks that may be involved in fulfilling the request. The data may be processed to identify certain data sources and / or data instances that may be particularly reliable for identifying steps to complete a particular task (e.g., as determined by crowdsourcing, data content, data organization, etc.). Alternatively or additionally, one or more trained machine learning models (e.g., trained using supervised learning and / or another training process) may be used to process data from different data sources to generate an action list for a particular task.

[0005] When generating an action list for a specific task (e.g., reminding the user to make bread), the automated assistant can render a customized GUI to initialize the execution of certain tasks. For example, the user may have an independent display device in the kitchen of their home, and the automated assistant can render a customized GUI at the display interface of the independent display device the next day after receiving the spoken utterance. The customized GUI may include a plurality of different selectable GUI elements that may correspond to each identified task of the request embodied in the spoken utterance. In some implementations, one or more of the selectable GUI elements may be rendered based on data generated in advance in response to the spoken utterance.

[0006] For example, the customized GUI may include a selectable link to a particular bread recipe that has been selected by the automated assistant based on contextual data indicating that the user might prefer the particular bread recipe. Alternatively or additionally, the customized GUI may include a selectable link to a particular shopping application that has been selected by the automated assistant based on the automated assistant determining, via the shopping application, the availability of certain ingredients for the particular recipe. In some implementations, the customized GUI may be dynamically adjusted based on contextual data and / or other data processed after the verbal utterance is provided by the user and / or after the customized GUI is rendered.

[0007] For example, the automated assistant may render a customized GUI during the next morning (e.g., the day after the user provides the verbal utterance), and the customized GUI may include a selectable link to a shopping application. However, after the customized GUI is rendered for the user, the user may order groceries from a grocery application that is different from the shopping application, and grocery data characterizing the purchased groceries may be provided to the automated assistant with prior permission from the user. When the grocery data indicates that the user purchased one or more ingredients for making bread that the automated assistant determines that the user did not previously have, the automated assistant may cause the customized GUI interface to be dynamically updated. In this way, the user will not be reminded of certain tasks that may have been completed, even if the user did not explicitly indicate their completion to the automated assistant. This can save time and resources that might otherwise be consumed to render comprehensive reminders about tasks that may no longer be relevant over time.

[0008] In some implementations, selectable links pointing to shopping applications can be removed and / or modified from a custom GUI interface. For example, if the automated assistant determines that the user now has all the raw materials, the automated assistant can remove the selectable elements for the shopping application from the custom GUI interface. However, if the automated assistant determines that the user still needs to buy one or more of the raw materials, the automated assistant can modify the selectable elements. For example, the selectable elements at the custom GUI can initially include content such as "Buy cinnamon and yeast with shopping application (buy cinnamon and yeast with shopping application)", and can also be associated with deep links, which add "cinnamon (cinnamon)" and "yeast (yeast)" to the digital shopping cart of the shopping application when selected. However, then based on grocery data, the selectable elements can be modified by the automated assistant to include content such as "Buy yeast with the shopping application (buy yeast with shopping application)" (without cinnamon), and associated with different deep links, which add "yeast" to the digital shopping cart of the shopping application when selected, instead of both "yeast" and "cinnamon".

[0009] In some implementations, the customized GUI rendered in response to a user request can be dynamically updated based on interactions between the automated assistant and one or more other entities (e.g., other people, organizations, devices, applications, etc.). For example, a user can provide a request for assistance with home repairs by providing a verbal utterance to the automated assistant, such as "Assistant, help me repair my furnace tomorrow." In response, the automated assistant can identify one or more tasks that the user typically attempts to complete to facilitate fulfillment of the request. For example, the automated assistant can identify one or more tasks, such as: calling an HVAC company, and finding helpful instructional videos on the Internet. The next day, the automated assistant can cause the customized GUI interface to be rendered at a display interface of an assistant-enabled device, such as a standalone display device in the living room of the user's home.

[0010] For example, before the normal business hours of an HVAC company, the customized GUI interface may render a reminder to "Call HVACcompanies" and a selectable element linking to an instructional video on HVAC repairs identified by the automated assistant. When the current time reaches the normal business hours of the HVAC company, the automated assistant (with prior permission from the user) may interact with a variety of different HVAC companies via one or more assistant protocols. For example, the automated assistant may access the chat module of the first HVAC company to schedule a time to meet with the user, and call the assistant phone of the second HVAC company to schedule another time to meet with the user. The automated assistant may schedule a specific HVAC company depending on which company is available that day, the user's expected preferences, crowdsourced data about each company, and / or any other information that may be helpful in determining whether to schedule a specific service.

[0011] Depending on the results of the interaction between the automated assistant and the HVAC company, the automated assistant may cause the customized GUI to be updated. For example, when the interaction between the first HVAC company and the automated assistant results in the first HVAC company being scheduled to arrive at 3:00 PM that day, the customized GUI interface may be updated to indicate the scheduled arrival (e.g., "First HVAC is arriving at 3:00PM."). Alternatively or additionally, the automated assistant may update the customized GUI interface to include additional content that may be determined based on the interaction between the automated assistant and the first HVAC company. For example, during the interaction between the first HVAC company and the automated assistant, the first HVAC company may indicate that the user should "shut off their HVAC system" before the first HVAC company arrives. Based on this indication, the automated assistant may update the customized GUI interface to remind the user to shut down the HVAC system before the appointment time of 3:00 PM.

[0012] Alternatively or additionally, the automated assistant may interact with the IoT application with prior permission from the user to automatically shut down the HVAC system at the user's home before the 3:00 PM appointment (e.g., the automated assistant may shut down the HVAC system at 2:55 PM). In such an instance, the automated assistant may modify the custom GUI interface to indicate that the automated assistant will utilize the IoT application to shut down the HVAC system before 3:00 PM. Alternatively or additionally, the automated assistant may cause the custom GUI interface to be updated to include a selectable element that causes the automated assistant to interact with the IoT application to shut down the HVAC system when selected. Alternatively or additionally, the selectable element may include natural language content, such as "Shut off HVAC system with IoT application," thereby indicating that the automated assistant will shut down the HVAC system in response to the user selecting the selectable element. In this way, the automated assistant may proactively assist with tasks related to reminders and / or other requests that the user may submit to the automated assistant. This may save time and resources by inferring certain tasks and attempting to complete such tasks when the user is unable to complete such tasks due to their limited schedule and / or availability.

[0013] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in more detail below.

[0014] Other implementations may include a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0015] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A , Figure 1B , Figure 1C as well as Figure 1D , Figure 1E1 , Figure 1E2 , Figure 1F1 and Figure 1F2 The automated assistant is shown dynamically rendering a corresponding view of a custom GUI that indicates the status of tasks that the automated assistant may be performing to assist in performing an activity.

[0017] Figure 2 A system is shown that provides an automated assistant for proactively identifying and completing tasks associated with scheduled activities and rendering a customized GUI that indicates the status of any ongoing tasks.

[0018] Figure 3 A method for operating an automated assistant to provide assistance in performing a specific activity at a specific time by allowing the automated assistant to complete certain tasks without user intervention before the specific time is shown.

[0019] Figure 4 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0020] Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 1E1 , Figure 1E2 , Figure 1F1 and Figure 1F2 The automated assistant dynamically renders views 100, 120, 140, 160, 180, 184, 190, and 194 of a custom GUI, respectively, indicating the status of a task that the automated assistant may be performing to assist in performing an activity. The user 102 may request assistance for the activity during a subsequent time instance, and the automated assistant may assist in performing certain tasks of the activity between the time when the user 102 provides the request and the subsequent time instance specified by the user 102. For example, the user 102 may provide a spoken utterance 112 to the automated assistant, which may be accessible via a computing device 104 located in a living room 110 of the user's 102 home. The spoken utterance 112 may be, for example, "Assistant, help me get a wedding cake tomorrow." The automated assistant may process audio data corresponding to the spoken utterance 112 to determine that the user 102 is requesting a reminder to perform a specific activity tomorrow (e.g., "get a wedding cake") and / or is requesting assistance to perform a specific activity tomorrow.

[0021] In some implementations, when the automated assistant determines that the user 102 is requesting assistance with an activity, the automated assistant may determine whether the activity is associated with other tasks that, if completed, would help fulfill the request of the user 102. In some implementations, one or more heuristic processes and / or one or more trained machine learning models may be used to process data from one or more different sources to identify one or more tasks associated with the activity. For example, content accessed by the user 102 and / or one or more other users may be processed with prior permission from the user to identify specific content instances that may identify tasks that may be associated with the requested activity. For example, video and / or audio rendered via the display device 108 and / or the portable device 106 may indicate that a particular activity (e.g., going to pick up a wedding cake) may involve certain tasks (e.g., selecting a design, purchasing wedding decorations, contacting a local bakery, etc.). Alternatively or additionally, certain tasks may be identified by processing data from one or more different web pages and / or application interfaces to identify, rank, and / or sort tasks that may be identified by other users as being associated with the identified activity.

[0022] After user 102 provides spoken utterance 112, automated assistant may cause customized GUI 132 to be rendered at display interface 122 of computing device 104, such as Figure 1B Alternatively or additionally, the automated assistant may cause selectable element 182 to be rendered at display interface 122, such as Figure 1E1 In some implementations, the customized GUI 132 can be rendered simultaneously with one or more other application interfaces (e.g., a media player 134 with a status indicator 128) being rendered at the display interface 122. In this way, the user 102 can be informed of the task that the automated assistant may be assisting with and / or the task that the user 102 may desire to begin to facilitate the execution of the activity. For example, a few minutes after the user 102 provides the spoken utterance 112, the automated assistant can cause the customized GUI 132 to be rendered with one or more selectable elements. For example, and as Figure 1B As shown, the customized GUI 132 may include a first selectable element 124 "See cake pictures", a second selectable element 126 "Buy wedding cake toppers nearby", a third selectable element 128 "Call local cake shops" and a fourth selectable element 130 "Get a wedding cake tomorrow". Alternatively, and as Figure 1E1As shown in view 180 of FIG. 1 , customizing GUI 132 may include causing other selectable elements to be rendered (e.g., Figure 1E2 Selectable element 182 of view 184).

[0023] Each of the selectable elements can correspond to a task that the automated assistant has determined to be associated with the activity for which the user 102 requested assistance. In some implementations, the user 102 can select a particular selectable element to cause the automated assistant to initiate performance of one or more operations to fulfill the corresponding task. For example, the user 102 can provide an input gesture (e.g., a tap gesture with their hand 142) to the computing device 104 at the first selectable element 124 to cause the automated assistant to employ a search engine to find “cake pictures” and render the search results at the display interface 122. However, in some implementations, after the user 102 requests assistance (e.g., provides the spoken utterance 112), the automated assistant can complete the one or more tasks associated with the activity without explicit user input and / or direct input to the automated assistant.

[0024] For example, prior to a time instance at which user 102 requests assistance (e.g., “tomorrow”), the automated assistant may initiate the performance of one or more operations to facilitate the completion of one or more tasks associated with an activity (e.g., “Getting a wedding cake.”). For example, the automated assistant may proactively identify one or more images of cakes that user 102 may like, thereby facilitating the completion of the task associated with first selectable element 124. Alternatively or additionally, the automated assistant may identify a local store that sells “wedding cake toppers” and add the item to a digital shopping cart of a shopping application that user 102 prefers to use, thereby facilitating the completion of the task associated with second selectable element 126. Alternatively or additionally, the automated assistant may initiate communications with one or more local entities (e.g., a local cake shop) to facilitate the completion of the task associated with third selectable element 128. When the automated assistant initiates the performance of certain tasks, the status of the initialized tasks may be indicated at each corresponding customized GUI 132.

[0025] For example, after the user 102 provides the spoken utterance 112 but before the time instance (e.g., "tomorrow") at which the user 102 requests assistance, the automated assistant may perform one or more operations 136, such as "identifying local cake shops" and "communicating with certain cake shops." In some implementations, the content of the communication may be based on one or more tasks that may be associated with the activity identified by the user 102. For example, the automated assistant may initiate communication with the local cake shops via audio communication, text communication, and / or another modality (e.g., via an API of an application associated with the entity). The content of the communication may include a query from the automated assistant to the entity regarding whether certain items (e.g., "wedding cake toppers") are available and / or the entity's hours of operation during the time instance (e.g., "tomorrow") at which the user 102 requests assistance. When the automated assistant identifies additional data associated with the activity and / or completes tasks associated with the activity, the automated assistant may cause the customized GUI 132 to be updated based on the additional data and / or based on any results of completing the tasks. In some implementations, when the selectable element 182 is rendered, an update to the task can be rendered in response to the user 102 selecting the selectable element 182. For example, when the user 102 selects Figure 1F1 The selectable element 192 sometimes causes the rendering Figure 1F2 When user 102 selects selectable element 182, the selected element 182 may be rendered differently compared to the updated customized GUI 174 of FIG. Figure 1E2 This difference can be reflected in the customized GUI 132. Figure 1E1 and / or Figure 1A The corresponding time and Figure 1F1 and / or Figure 1B The changes in state of certain tasks and / or operations performed by the automated assistant between corresponding times.

[0026] In some implementations, the customization GUI 132 may include a selectable element (e.g., fourth selectable element 130) that, when selected by the user, may render a state of any action taken by the automated assistant to facilitate completion of the task. Figure 1CAs shown in view 140 of , the user can tap the fourth selectable element 130 using their hand 142 to cause the status GUI element 142 to be rendered. The status GUI element 142 can be rendered with content that characterizes one or more tasks that the automated assistant is currently completing and / or has completed. For example, the automated assistant may be performing a task associated with the third selectable element 128, such as calling a local bakery (e.g., "Calling Cardinal Bakery"). In some implementations, the user 102 can view a transcription of the progress of the communication between the automated assistant and the entity (e.g., the local bakery) and / or can control any actions that the automated assistant may currently be taking. In this way, if the user 102 has decided to contact a particular entity without giving this communication to the automated assistant, the user 102 can cancel any ongoing actions that the automated assistant may be taking to complete this task.

[0027] In some implementations, the automated assistant may render a different and / or updated customized GUI 174 at the computing device 104 and / or at a different display interface 176 during or before the time instance at which the user 102 requests assistance. The display interface 1162 may be integrated into a different computing device 176 associated with the user 102, and the updated customized GUI 174 may be rendered simultaneously with other application interfaces (e.g., a media player 134 with a status indicator 128 and a news application 172). In some implementations, the automated assistant may render an updated customized GUI 174 based on operations performed to facilitate completion of any identified tasks associated with the activity. Alternatively or additionally, the updated customized GUI 174 may be based on other tasks that may not have been rendered after the user 102 provided the spoken utterance 112. For example, a first selectable element 164 may be rendered for a task of "See popular wedding cake flavors," which may be rendered based on an entity (e.g., a local cake shop) requesting to select a cake flavor. For example, an incoming communication from “Cardinal Bakery” to the automated assistant may include natural language content that embodies a request for user 102 to select a cake flavor when user 102 orders from “Cardinal Bakery.” Alternatively or additionally, first selectable element 164 may be rendered in response to spoken utterance 112 along with a task identified separately for the activity. Alternatively or additionally, third selectable element 168 may be rendered for the task of “View Recipe Video,” which may correspond to a recipe that may have been identified by the automated assistant when performing a task associated with another first selectable element 164 (e.g., “See cake pictures”). In some implementations, the updated customized GUI 174 may not be rendered until user 102 selects selectable element 192, such as Figure 1F1 and Figure 1F2 shown.

[0028] In some implementations, selectable element 192 and selectable element 182 can be rendered along with an indication of the request and / or task associated with selectable element 182 and selectable element 192. For example, each selectable element can include natural language content and / or graphical content (e.g., wedding cake) to indicate the request and / or task associated with each selectable element. In this way, user 102 can understand the purpose of each selectable element without requiring each selectable element to occupy more screen area. In some implementations, the content of each selectable element can be rendered to indicate whether a state change has occurred to one or more tasks associated with the selectable element. For example, when an assistant call to another entity (e.g., a cake shop) is already in the state corresponding to Figure 1F1 The time corresponding to Figure 1E1 , selectable element 192 may appear different from selectable element 182. For example, the color, text, shading, graphics, and / or other features of selectable element 192 may appear different from selectable element 182 based on a change in one or more states of a task and / or operation and / or based on information obtained during the performance of the task and / or operation. Figure 1F1 Selectable element 192 at and view the Figure 1F2 The updated customized GUI 174 at the address to check status and / or information.

[0029] In some implementations, selecting a selectable element 192 may "spawn" a new task compared to when the user 102 selected the selectable element 182 at a previous time instance. For example, information obtained during a call between an automated assistant and a particular entity (e.g., Cardinal Bakery) may indicate that the particular entity is suitable for completing a task (e.g., because the particular entity is open tomorrow, while other entities are closed tomorrow). Based on this determination, depending on how the user context changes over time and / or depending on data obtained during certain assistant operations, an updated customized GUI 174 may be dynamically rendered with new tasks, information, and / or selectable elements for the user 102 to interact with. In this way, the user 102 may be guided by the automated assistant to more efficiently fulfill their requests by eliminating certain optional tasks that may not make sense in certain contexts (e.g., when certain stores are closed) and / or certain optional tasks that may be redundant at certain times (e.g., when certain stores have already been contacted by the automated assistant).

[0030] In some implementations, the updated customized GUI 174 may include selectable elements corresponding to specific entities that the automated assistant has determined will be most helpful to the user 102 based on the context of the spoken utterance 112. For example, while there may be multiple entities operating near the user 102 (e.g., a cake shop), only a certain number of entities may be operating during the time instance when the user 102 requests assistance. Based on the aforementioned example, the automated assistant may determine that "Cardinal Bakery" corresponding to the second selectable element 166 is open during the request time instance (e.g., "tomorrow"). When the user 102 selects the second selectable element 166, the automated assistant may initialize communication between the user 102 and the corresponding entity (e.g., "Cardinal Bakery") and / or between the automated assistant and the corresponding entity. Alternatively or additionally, when the user 102 selects the fourth selectable element 170, the automated assistant may initialize execution of a custom routine involving performing one or more operations corresponding to any remaining tasks of the activity. For example, in response to selecting fourth selectable element 170, the automated assistant may present popular wedding cake flavors to user 102 at display interface 162 (e.g., per first selectable element 164) and then initiate a call with "Cardinal Bakery" (e.g., per second selectable element 166).

[0031] By allowing the automated assistant to identify tasks for a particular activity of interest to the user, the automated assistant can proactively begin and / or complete certain tasks, thereby minimizing the amount of time and / or resources that the user may spend on the particular activity. Additionally, when the particular activity involves interacting with another application and / or device with which the automated assistant is able to interact, the automated assistant can reduce the amount of time and effort that may be consumed during such interaction by replacing the user in such interaction. For example, by allowing the automated assistant to make decisions about certain tasks based on contextual data (e.g., the user's schedule, the organization's hours of operation, the user's location, the time of day, etc.), the automated assistant can eliminate steps that the user may have to take manually when preparing for and / or participating in the activity.

[0032] Figure 2A system 200 is shown that provides automation 204 for proactively identifying and completing tasks associated with scheduled activities and rendering a customized GUI indicating the status of any ongoing tasks. The automated assistant 204 can operate as part of an assistant application provided at one or more computing devices (such as computing device 202 and / or server device). The user can interact with the automated assistant 204 via an assistant interface 220, which can be a microphone, camera, touch screen display, user interface, and / or any other device capable of providing an interface between the user and the application. For example, the user can initialize the automated assistant 204 by providing speech, text, and / or graphical input to the assistant interface 220 to cause the automated assistant 204 to initiate one or more actions (e.g., provide data, control peripheral devices, access agents, generate input and / or output, etc.). Alternatively, the automated assistant 204 can be initialized based on processing context data 236 using one or more trained machine learning models. The context data 236 can characterize one or more features of the environment in which the automated assistant 204 is accessible, and / or one or more features of the user who is predicted to intend to interact with the automated assistant 204. The computing device 202 may include a display device, which may be a display panel, including a touch interface for receiving touch input and / or gestures to allow a user to control an application 234 of the computing device 202 via the touch interface. In some implementations, the computing device 202 may lack a display device, thereby providing an audible user interface output instead of a graphical user interface output. In addition, the computing device 202 may provide a user interface for receiving oral natural language input from a user, such as a microphone. In some implementations, the computing device 202 may include a touch interface and may not contain a camera, but may optionally include one or more other sensors.

[0033] Computing device 202 and / or other third-party client devices may communicate with the server device via a network such as the Internet. Additionally, computing device 202 and any other computing devices may communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 202 may offload computing tasks to a server device in order to conserve computing resources at computing device 202. For example, a server device may host automated assistant 204, and / or computing device 202 may transmit input received at one or more assistant interfaces 220 to a server device. However, in some implementations, automated assistant 204 may be hosted at computing device 202, and various processes that may be associated with automated assistant operations may be performed at computing device 202.

[0034] In various implementations, all or less than all aspects of automated assistant 204 may be implemented on computing device 202. In some of those implementations, aspects of automated assistant 204 are implemented via computing device 202 and may interface with a server device, which may implement other aspects of automated assistant 204. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 204 are implemented via computing device 202, automated assistant 204 may be an application separate from (e.g., installed “on top of”) an operating system of computing device 202, or may alternatively be implemented directly by (e.g., treated as an application of, but integrated with) the operating system of computing device 202.

[0035] In some implementations, automated assistant 204 can include an input processing engine 206 that can employ a plurality of different modules to process input and / or output of computing device 202 and / or server devices. For example, input processing engine 206 can include speech processing engine 208 that can process audio data received at assistant interface 220 to identify text embodied in the audio data. The audio data can be transmitted from, for example, computing device 202 to a server device in order to preserve computing resources at computing device 202. Additionally or alternatively, the audio data can be processed exclusively at computing device 202.

[0036] The process for converting audio data to text may include a speech recognition algorithm that can use a neural network and / or a statistical model to identify grouped audio data corresponding to a word or phrase. The text converted from the audio data can be parsed by the data parsing engine 210 and used as text data for the automated assistant 204, which can be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 210 can be provided to the parameter engine 212 to determine whether the user has provided input corresponding to a specific intention, action, and / or routine that can be performed by the automated assistant 204 and / or an application or agent that can be accessed via the automated assistant 204. For example, the assistant data 238 can be stored at a server device and / or a computing device 202, and can include data defining one or more actions that can be performed by the automated assistant 204, and parameters necessary for performing the action. The parameter engine 212 can generate one or more parameters for the intention, action, and / or slot value, and provide one or more parameters to the output generation engine 214. Output generation engine 214 may use the one or more parameters to communicate with assistant interface 220 for providing output to a user, and / or to communicate with one or more applications 234 for providing output to one or more applications 234 .

[0037] In some implementations, the automated assistant 204 may be an application that may be installed “on top of” the operating system of the computing device 202, and / or may itself form part (or the entirety) of the operating system of the computing device 202. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally at the computing device 202. The on-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. Additionally, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes the recognized text generated using on-device speech recognition and optionally context data to generate NLU data.

[0038] The NLU data may include an intent corresponding to the spoken utterance and optionally parameters for the intent (e.g., slot values). On-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine actions to be taken to resolve the intent (and optionally parameters for the intent) of the spoken utterance. This may include determining local and / or remote responses (e.g., replies) to the spoken utterance, interactions with locally installed applications performed based on the spoken utterance, commands transmitted to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The on-device fulfillment may then initiate local and / or remote execution / execution of the determined actions to resolve the spoken utterance.

[0039] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be at least selectively utilized. For example, the recognized text may be at least selectively transmitted to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, the recognized text may be optionally transmitted for remote execution in parallel with on-device execution or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the reduced latency they provide when resolving spoken utterances (because resolving spoken utterances does not require client-server round trips). In addition, on-device functionality may be the only functionality available when there is no network connection or when network connection is limited.

[0040] In some implementations, computing device 202 may include one or more applications 234, which may be provided by a third-party entity different from the entity that provided computing device 202 and / or automated assistant 204. Automated assistant 204 and / or an application state engine of computing device 202 may access application data 230 to determine one or more actions that can be performed by one or more applications 234, as well as a state of each of one or more applications 234 and / or a state of a corresponding device associated with computing device 202. Automated assistant 204 and / or a device state engine of computing device 202 may access device data 232 to determine one or more actions that can be performed by computing device 202 and / or one or more devices associated with computing device 202. Additionally, application data 230 and / or any other data (e.g., device data 232) can be accessed by automated assistant 204 to generate context data 236, which can characterize the context in which a particular application 234 and / or device is executing, and / or the context in which a particular user is accessing computing device 202, accessing application 234, and / or any other device or module.

[0041] While one or more applications 234 are executing at computing device 202, device data 232 may characterize the current operational state of each application 234 being executed at computing device 202. In addition, application data 230 may characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces being rendered under the direction of one or more applications 234. Alternatively or additionally, application data 230 may characterize action paradigms based on the current operational state of the respective applications, which action paradigms may be updated by the respective applications and / or by automated assistant 204. Alternatively or additionally, one or more action paradigms of one or more applications 234 may remain static but may be accessed by the application state engine in order to determine appropriate actions to be initiated via automated assistant 204.

[0042] The computing device 202 may also include an assistant invocation engine 222, which may use one or more trained machine learning models to process application data 230, device data 232, context data 236, and / or any other data accessible to the computing device 202. The assistant invocation engine 222 may process this data to determine whether to wait for the user to explicitly say an invocation phrase to invoke the automated assistant 204, or to treat the data as indicating the user's intention to invoke the automated assistant, rather than requiring the user to explicitly say an invocation phrase. For example, one or more trained machine learning models may be trained using instances of training data based on scenarios in which the user is in an environment where multiple devices and / or applications are exhibiting various operating states. Instances of training data may be generated to capture training data that characterizes contexts in which the user invokes the automated assistant and other contexts in which the user does not invoke the automated assistant. When one or more trained machine learning models are trained based on these instances of training data, the assistant invocation engine 222 may enable the automated assistant 204 to detect or limit detection of verbal invocation phrases from the user based on features of the context and / or environment.

[0043] In some implementations, system 200 can include a task identification engine 216 that can determine whether a request from a user corresponds to one or more tasks that can be performed by automated assistant 204 alone. For example, a user can provide automated assistant 204 with a request seeking assistance from automated assistant 204 for an activity during a subsequent time instance. Data embodying the request can be processed by task identification engine 216 to determine one or more tasks for automated assistant 204 to perform during this time to facilitate the preparation of the activity. In some implementations, tasks can be identified using one or more heuristic processes and / or one or more trained machine learning models using data associated with the user and / or one or more other users.

[0044] When a corresponding task for an activity is identified, the task identification engine 216 can determine how to assist in performing the corresponding task. In some implementations, the application data 230 can be processed to determine whether one or more applications 234 can assist the user and / or the automated assistant 204 in performing one or more of the identified tasks. For example, when a user provides a request (such as "Assistant, help me start a garden tomorrow"), the task identification engine 216 can determine that the task of "Identifying nearby Nurseries" may be useful to complete before the time the user requests help (e.g., "tomorrow"). The task identification engine 216 can call a navigation application and / or any other suitable application to initialize the execution of one or more operations to facilitate the completion of this task. For example, the automated assistant 204 can initialize a search operation via the navigation application to identify "Nurseries" within a specific distance from the user's current location. Therefore, the application 234 can be selected and / or initialized to a specific state based on the user's current context and / or the context associated with the activity.

[0045] In some implementations, the system 200 may include a context decision engine 218 that may process context data to facilitate determining a specific application to be invoked for a specific task, a specific application state of the invoked application, and / or data to be utilized to facilitate the completion of the task. For example, the context data 236 may indicate the time of day and / or location of the user after the user provides a request for assistance for a specific activity (with prior permission from the user). The automated assistant 204 may determine whether and / or how to initialize the execution of a specific task based on processing the context data 236. For example, when the user is at a specific location after providing their request, the context processing engine 218 may initialize a navigation application and search for entities near the specific location. However, when the user is at another location after providing their request, the context processing engine 218 may initialize navigation and search for other entities near the other location. Alternatively or additionally, the context processing engine 218 may determine that a specific application and / or application state is suitable for initialization at a specific time based on historical usage data and / or historical interaction data associated with the user and the specific application. However, the context processing engine 218 can determine that different applications and / or different application states are suitable for initialization at different times based on historical usage data and / or historical interaction data. In some implementations, the historical usage data can indicate that a user previously accessed a particular application and / or application state in a particular context, and that the user previously accessed different applications and / or different application states in different contexts. Thus, the historical usage data can be used to determine that a particular application and / or application state is useful for facilitating completion of a particular task and / or activity.

[0046] In some implementations, system 200 may include a state generation engine 226 for determining the state of one or more tasks that the automated assistant may perform to facilitate assisting a user in performing a particular activity. State generation engine 226 may process application data 230, device data 232, and / or context data 236 to determine the state of a particular task. For example, when the automated assistant initiates communication with a separate entity, state generation engine 226 may determine state information about an interaction between the automated assistant and the separate entity. In some implementations, state data generated by state generation engine 226 may characterize the content of the interaction, such as, but not limited to, additional information requested by the entity, additional information provided by the entity, information provided by the automated assistant, and / or any other information that may be associated with an interaction with the same entity.

[0047] In some implementations, the system 200 may include a custom GUI engine 234 that can generate interface data using data generated by the state generation engine 226, the task identification engine 216, and / or the context decision engine 218. The interface data may characterize a custom GUI that may be rendered based on a request for the automated assistant to assist the user in performing an activity during a subsequent time instance. Using the available data, the automated assistant may dynamically update the custom GUI to indicate the state of each corresponding task being performed by the automated assistant to facilitate assisting the user in performing an activity. In some implementations, the custom GUI engine 224 may generate data representing one or more selectable state elements that, when selected, may provide the user with an indication of the ongoing state of one or more tasks. For example, when a user selects a selectable state element corresponding to a search for nearby entities, the display interface that renders the selectable state element may indicate that the automated assistant is still searching and / or may indicate certain search results identified by the automated assistant.

[0048] In an implementation, when there is a change to the context associated with the user and / or task, the customized GUI can be updated accordingly. For example, an application and / or application state can be selected based on the user's current context. However, when the current context changes, another application and / or another application state can be selected. For example, if the user selects a selectable state element (e.g., rendered at a home stand-alone display device) when the user is in a first position, the customized GUI can provide an animation showing the search results identified for the first position. However, if the user selects a selectable state element (e.g., rendered at a vehicle display interface) when the user is in a second position different from the first position, the customized GUI can provide an animation showing that the automated assistant is still searching and / or showing the search results identified for the second position.

[0049] Figure 3A method 300 for operating an automated assistant to provide assistance for performing a specific activity at a specific time by allowing the automated assistant to complete certain tasks before the specific time without user intervention is shown. The method 300 can be performed by one or more computing devices, applications, and / or any other device or module that may be associated with the automated assistant. The method 300 may include an operation 302 for determining whether a user has provided input to the automated assistant. The input may be, for example, a spoken word and / or other assistant input, which may include a request for the automated assistant to perform one or more specific operations. For example, the assistant input may be a spoken word, such as "Assistant, help me find a carmechanic tomorrow at 2:00PM. (Assistant, help me find a car mechanic tomorrow at 2:00PM.)". The audio data representing the spoken word may be processed to determine one or more operations that the user may request the automated assistant to perform. When it is determined that the user input has been received, the method 300 may proceed from operation 302 to operation 304, otherwise the automated assistant may continue to determine whether the user has provided input.

[0050] Operation 304 may include determining whether the input from the user corresponds to a request for assistance with an activity to be performed at a subsequent time instance. For example, based on the aforementioned example, the spoken utterance may refer to a request for the automated assistant to assist the user in finding a "car mechanic" during a subsequent time instance (e.g., "tomorrow at 2:00PM"). In some implementations, when the user is requesting assistance with an activity, the automated assistant may determine whether the activity includes one or more tasks that can assist the user in performing the activity if completed by the automated assistant. For example, the automated assistant may use one or more heuristic processes and / or one or more trained machine learning models to process data from one or more sources to identify other tasks that may be associated with the activity. In some implementations, embeddings may be generated based on the activity and / or related data, and the embeddings may be mapped to a latent space in which the task embeddings are mapped. The multiple tasks may then be identified based on their distances from the activity embeddings to the task embeddings in the latent space. For example, tasks identified as being associated with "Finding a car mechanic" may include: (1) using a search engine to identify a nearby car mechanic, (2) calling a nearby car mechanic to see if they are available, and (3) taking the broken-down car to the car mechanic.

[0051] When it is determined that the user input corresponds to a request for assistance with an activity to be performed during a subsequent time instance, the method 300 may proceed from operation 304 to operation 306. Otherwise, the method 300 may proceed from operation 304 to operation 314 to perform one or more operations to facilitate fulfilling the request from the user. Operation 306 may include causing a customized GUI for the activity to be rendered at a display interface of a computing device (e.g., a computing device to which the user input is directed, or a separate computing device). The customized GUI may include one or more selectable elements, and each selectable element may correspond to a corresponding task that may be performed to facilitate assisting the user in performing the activity.

[0052] For example, and in accordance with the foregoing example, a customized GUI may be rendered with a selectable element that, when selected, causes the automated assistant to initialize execution of an Internet search for nearby auto mechanics. In some implementations, the selectable element may be rendered with natural language content, such as “Search for Nearby Car Mechanics.” As another example, the customized GUI may be rendered with another selectable element that, when selected, causes the automated assistant to initialize a phone call between the user and a particular auto mechanic and / or to initialize a phone call between the automated assistant and a particular auto mechanic. In some implementations, other selectable elements may be rendered with natural language content, such as “Call Louisville Mechanic Shop.”

[0053] From operation 306, method 300 may proceed to operation 308, which may include determining whether the automated assistant can assist in performing specific tasks of the activity without direct, explicit, and / or indirect user intervention before the time instance. For example, the automated assistant may determine, based on the identified tasks of the activity, whether one or more of the tasks correspond to one or more actions that can be performed by the automated assistant and / or another application associated with the automated assistant. For example, the task of performing an Internet search for "nearby car mechanics" may be completed by the automated assistant alone, while calling a nearby car mechanic may be performed using the automated assistant and a separate phone application. When the automated assistant determines that one or more tasks of the activity can be performed by the automated assistant before the time instance, method 300 may proceed to operation 310. Otherwise, method 300 may proceed from operation 308 to operation 314.

[0054] Operation 310 may include initializing the execution of one or more operations to facilitate the completion of a specific task before a time instance. In some implementations, the specific task may include initializing communication between the automated assistant and a separate entity that is different from the user and the automated assistant. For example, the separate entity may be a separate application, a separate device, an organization, and / or another person associated with the specific task. In some implementations, the communication may include communicating messages via audio data and / or text data to facilitate the completion of the specific task. Alternatively or additionally, operation 310 may involve the automated assistant interacting with one or more separate applications to facilitate the completion of the specific task. For example, the automated assistant may communicate with a contracting party application to facilitate arranging for an auto mechanic to assist the user. For example, the automated assistant may also communicate with a calendar application to facilitate determining whether a specific time when the auto mechanic is available corresponds to a portion of the user's calendar that includes an entry.

[0055] Alternatively or additionally, the automated assistant may initiate the performance of one or more operations by identifying a nearby auto mechanic using a search engine application. The automated assistant may then identify one or more search results corresponding to a suitable nearby auto mechanic and initiate communication with the identified auto mechanic. In some implementations, the automated assistant may access the user's schedule data to determine when to schedule a meeting with the auto mechanic. When the automated assistant has initiated the performance of one or more operations to facilitate completion of a particular task before a time instance, method 300 may proceed from operation 310 to operation 312.

[0056] Operation 312 may include rendering an updated custom GUI based on operations performed and / or initialized by the automated assistant. For example, the initial custom GUI rendered at operation 306 may include a first set of selectable elements with corresponding natural language content and / or other information. In response to the automated assistant initializing certain operations, the initial custom GUI may be updated to include a second set of selectable elements, which may indicate the status of certain tasks and / or operations that may have been performed and / or initialized by the automated assistant. For example, the first set of selectable elements may include a selectable element that causes a search engine operation to be performed when selected, while the second set of selectable elements may not have the selectable element. Alternatively or additionally, the second set of selectable elements may remove the selectable element, but the selectable element may be replaced with another selectable element for calling a specific entity (e.g., an auto mechanic organization) associated with the search results from the search engine.

[0057] From operation 312, method 300 may proceed to operation 316 to determine whether contextual data indicates a change to the requested activity and / or specific task before the time instance at which the user requested assistance for the activity. For example, contextual data may indicate a time of day, a location associated with the user (with prior permission from the user), a change in content that the user is interested in or may be interested in, a change in an activity in which the user participates (e.g., the user was previously at work, but is now out having dinner with friends), and / or any other feature of a context that may be associated with the user. According to the foregoing example, the time of day may change to be outside certain business hours indicated by certain entities identified via search engine results. Alternatively or additionally, the user's location may be updated to be less close to certain entities and closer to other entities identified in the search engine results. When contextual data indicates a change to the requested activity and / or specific task before the time instance, method 300 may proceed to operation 310 and / or operation 312 from operation 316. Otherwise, method 300 may proceed to operation 314 from operation 316 to perform one or more operations to facilitate fulfilling a request from the user at or before the time instance.

[0058] When method 300 returns to operation 310, the automated assistant may initialize one or more operations and / or execution of one or more other operations to facilitate completion of a particular task prior to the time instance. For example, when the contextual data indicates that the current time exceeds the operating hours of certain entities, the automated assistant may initialize communications with different entities that may be open for business at the current time. Alternatively or additionally, when the contextual data indicates that the user's vehicle is at a different location (e.g., as determined using data provided by an application corresponding to the vehicle), the automated assistant may initialize communications with a particular entity that may be closer to the location than the previous location. Depending on any communication data generated during the automated assistant's interaction with the different entities, the automated assistant may cause the customized GUI to be updated based on the communication data in accordance with operation 312.

[0059] Figure 4 4 is a block diagram 400 of an example computer system 410. The computer system 410 generally includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include a storage subsystem 424 (including, for example, a memory 425 and a file storage subsystem 426), a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with the computer system 410. The network interface subsystem 416 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0060] The user interface input device 422 may include a keyboard, a pointing device (such as a mouse, a trackball, a touch pad or a graphics tablet, a scanner, a touch screen incorporated into a display), an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 410 or over a communication network.

[0061] User interface output device 420 can include display subsystem, printer, fax machine or non-visual display, such as audio output device.Display subsystem can include cathode ray tube (CRT), flat panel device such as liquid crystal display (LCD), projection device or some other mechanisms for creating visible image.Display subsystem can also provide non-visual display such as via audio output device.In general, the use of term "output device" is intended to include all possible types of devices and modes for outputting information from computer system 410 to user or another machine or computer system.

[0062] Storage subsystem 424 stores programming and data constructs that provide some or all of the functionality of the modules described herein. For example, storage subsystem 424 may include logic for performing selected aspects of method 300 and / or implementing one or more of system 400, computing device 104, automated assistant, and / or any other application, device, apparatus, and / or module discussed herein.

[0063] These software modules are typically executed by processor 414 alone or in conjunction with other processors. The memory 425 used in storage subsystem 424 may include multiple memories, including a main random access memory (RAM) 430 for storing instructions and data during program execution and a read-only memory (ROM) 432 in which fixed instructions are stored. File storage subsystem 426 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in storage subsystem 424 by file storage subsystem 426, or in other machines accessible to processor 414.

[0064] The bus subsystem 412 provides a mechanism for the various components and subsystems of the computer system 410 to communicate with each other as intended. Although the bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0065] Computer system 410 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 4 The description of the computer system 410 depicted in FIG. 4 is intended only as a specific example for illustrating some implementations. Many other configurations of the computer system 410 are possible, with similar configurations. Figure 4 The computer systems may have more or fewer components than those depicted.

[0066] In cases where the systems described herein collect personal information about users (or "participants" as often referred to herein) or may make use of personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social behavior or activities, occupation, a user's preferences, or a user's current geographic location), or to control whether and / or how content is received from a content server that may be more relevant to the user. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that personally identifiable information about the user cannot be determined, or, in the case of obtaining geographic location information, the user's geographic location may be generalized (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may control how information about users is collected and / or used.

[0067] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize or confirm many equivalents of the implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

[0068] A method implemented by one or more processors is described as including operations, such as receiving, by an automated assistant, user input seeking assistance from the automated assistant for a user to perform an activity, wherein the automated assistant is accessible via a computing device associated with the user. The method may also include determining, based on the user input, that the activity for which the user requests assistance corresponds to one or more tasks, wherein the one or more tasks are also determined based on data accessible to the automated assistant. The method may also include causing, by the automated assistant, a display interface of the computing device to render a customized graphical user interface (GUI) including one or more selectable elements, wherein each of the one or more selectable elements corresponds to a corresponding task in one or more tasks. The method may also include, after the customized GUI is rendered at the display interface of the computing device: performing, by the automated assistant, one or more operations to facilitate completion of a specific task in the one or more tasks without explicit input from the user, and causing the customized GUI to be modified to indicate a state of completion of a specific task in the one or more tasks based on the automated assistant performing the one or more operations.

[0069] In some implementations, the state of completing a particular task is based on an interaction between the automated assistant and a separate entity different from the user. In some implementations, causing the customized GUI to be modified to indicate a state of completing the particular task includes: causing the customized GUI to render one or more additional selectable elements corresponding to one or more additional tasks associated with the activity. In some implementations, causing the customized GUI to be modified to indicate a state of completing a particular task includes: causing a particular selectable element of the one or more selectable elements to be removed from the customized GUI, wherein the particular selectable element corresponds to a particular application with which the automated assistant interacts to facilitate completing the particular task. In some implementations, wherein causing the display interface of the computing device to render a customized GUI including one or more selectable elements includes: causing a particular selectable element of the one or more selectable elements to be rendered at the customized GUI, wherein the particular selectable element is associated with a first application; and wherein causing the customized GUI to be modified to indicate a state of completing a particular task of the one or more tasks includes: causing an additional selectable element to be rendered at the customized GUI, wherein the additional selectable element is associated with a second application different from the first application. In some implementations, performing one or more operations to facilitate completion of a particular task of the one or more tasks includes processing context data indicating whether facilitating completion of the particular task can be performed by the automated assistant alone for a given context.

[0070] In other implementations, a method implemented by one or more processors is described as including operations such as receiving, by an automated assistant, user input seeking the automated assistant to assist a user in performing an activity during a subsequent time instance, wherein the automated assistant is accessible via a computing device associated with the user. The method may also include communicating, by the automated assistant, a message with an entity separate from the user before the subsequent time instance to facilitate completing the activity, wherein the message includes a message sent from the automated assistant to the entity and an additional message received by the automated assistant from the entity in response to the message. The method may also include generating state data indicating a state of completing the activity based on the additional message received by the automated assistant. The method may also include causing, by the automated assistant, a display interface of the computing device to render a custom graphical user interface (GUI) based on the state data before or during the subsequent time instance, wherein the custom GUI includes one or more selectable elements, and a specific selectable element of the one or more selectable elements is generated based on the state data and indicates a state of completing the activity.

[0071] In some implementations, communicating a message with an entity separate from the user includes causing an audio call to be initiated between the automated assistant and the entity, wherein the message corresponds to audio data communicated between the entity and the automated assistant during the audio call. In some implementations, communicating a message with an entity separate from the user includes causing a text conversation to be initiated between the automated assistant and the entity, wherein the message corresponds to text data communicated between the entity and the automated assistant during the text conversation. In some implementations, the method may further include processing, by the automated assistant, context data indicating that the entity is available for communication with the automated assistant, wherein the context data is provided by an application associated with the entity and distinct from the automated assistant.

[0072] In yet other implementations, a method implemented by one or more processors is described as including operations such as receiving, by an automated assistant, user input seeking the automated assistant to assist a user in performing an activity during a subsequent time instance, wherein the automated assistant is accessible via a computing device associated with the user. The method may also include processing, by the automated assistant, context data representing a context of the user after providing the user input to the automated assistant, wherein the context data is processed to facilitate determining a specific arrangement of selectable elements to be rendered at a display interface or a separate display interface of the computing device to assist the user in performing the activity. The method may also include, when the context data indicates that the user is associated with a first context after providing the user input: causing a customized GUI to be rendered at a display interface or a separate display interface together with a first set of selectable elements, wherein each selectable element in the first set of selectable elements corresponds to a respective task in each task associated with the activity. The method may also include, when the context data indicates that the user is associated with a second context after providing the user input: causing a different customized GUI to be rendered at the display interface or a separate display interface along with a second set of selectable elements, wherein each selectable element in the second set of selectable elements corresponds to a different respective specific task among the tasks associated with the activity.

[0073] In some implementations, processing context data representing a context of a user after providing user input to an automated assistant includes: determining, based on the context data, a current time of day after the user provided the user input, wherein a particular arrangement of selectable elements to be rendered at a display interface or a separate display interface is selected based on the current time of day. In some implementations, the user is associated with a first context when the current time of day corresponds to a calendar entry stored by a calendar application associated with the user, and the user is associated with a second context when the current context does not correspond to a calendar entry stored by a calendar application associated with the user. In some implementations, processing context data representing a context of a user after providing user input to an automated assistant includes: determining, based on the context data, a current location of the user after the user provided the user input, wherein the particular arrangement of selectable elements to be rendered at the display interface or a separate display interface is selected based on the current location of the user.

[0074] In some implementations, when the current location corresponds to the specific location, the user is associated with the first context, and when the current location does not correspond to the specific location, the user is associated with the second context. In some implementations, the selectable elements in the first set of selectable elements include content provided by the first application, and the additional selectable elements in the second set of selectable elements include other content provided by a second application different from the first application. In some implementations, the method may also include, when the context data indicates that the user is associated with the first context after providing the user input: determining to render the first set of selectable elements based on historical usage data indicating that the first application is associated with the first context.

[0075] In some implementations, selectable elements in the first set of selectable elements are stored in association with links that, when selected, initialize the application to a first state, and additional selectable elements in the second set of selectable elements are stored in association with separate links that, when selected, initialize the application to a second state different from the first state. In some implementations, the method may also include, when the context data indicates that the user is associated with the first context after providing the user input: determining to render the first set of selectable elements based on historical usage data indicating that the user previously accessed the first state of the application in the first context. In some implementations, the first context corresponds to a specific time and / or a specific location, and the second context corresponds to a separate time and / or a separate location.

Claims

1. A method implemented by one or more processors, the method comprising: receiving, by an automated assistant, user input seeking assistance from the automated assistant in performing an activity for the user, wherein the automated assistant is accessible via a computing device associated with the user; determining, based on the user input, that the activity for which the user requests assistance corresponds to one or more tasks, wherein the one or more tasks are further determined based on data accessible to the automated assistant; causing the display interface of the computing device to render, by the automated assistant, a customized graphical user interface (GUI) including one or more selectable elements, wherein each of the one or more selectable elements corresponds to a respective task of the one or more tasks; After the customized GUI is rendered at the display interface of the computing device: performing, by the automated assistant without explicit input from the user, one or more operations to facilitate completion of a particular task of the one or more tasks, and The customized GUI is modified to indicate a status of completing the particular task of the one or more tasks based on the automated assistant performing the one or more operations.

2. The method of claim 1, wherein the status of completion of the particular task indicated in the modified customized GUI is based on an interaction between the automated assistant and a separate entity distinct from the user.

3. The method of claim 2, wherein causing the customized GUI to be modified to indicate the status of completing the specific task comprises: The customized GUI is caused to render one or more additional selectable elements corresponding to one or more additional tasks associated with the activity.

4. The method of any preceding claim, wherein causing the customized GUI to be modified to indicate the status of completion of the particular task comprises: causing a particular selectable element of the one or more selectable elements to be removed from the customized GUI, The specific selectable element corresponds to a specific application with which the automated assistant interacts to facilitate completion of the specific task.

5. A method as claimed in any preceding claim, Wherein causing the display interface of the computing device to render the customized GUI including the one or more selectable elements comprises: causing a specific selectable element of the one or more selectable elements to be rendered at the custom GUI, wherein the particular selectable element is associated with a first application; and Wherein causing the customized GUI to be modified to indicate the status of completing the specific task comprises: causing additional selectable elements to be rendered at the custom GUI, The additional selectable element is associated with a second application that is different from the first application.

6. The method of claim 5, wherein performing the one or more operations to facilitate completion of the particular task in the one or more tasks comprises: Context data is processed that indicates whether, for a given context, facilitating completion of the particular task can be performed by the automated assistant alone.

7. A method implemented by one or more processors, the method comprising: receiving, by the automated assistant, user input seeking the automated assistant to assist the user in performing an activity during a subsequent time instance, wherein the automated assistant is accessible via a computing device associated with the user; communicating, by the automated assistant, a message to an entity separate from the user prior to the subsequent time instance to facilitate completion of the activity, wherein the message comprises a message sent from the automated assistant to the entity and an additional message received by the automated assistant from the entity in response to the message; generating status data indicating a status of completing the activity based on the additional message received by the automated assistant; as well as causing, by the automated assistant, a display interface of the computing device to render a customized graphical user interface (GUI) based on the state data before or during the subsequent time instance, The customized GUI includes one or more selectable elements, and a specific selectable element of the one or more selectable elements is generated based on the state data and indicates the state of completing the activity.

8. The method of claim 7, wherein communicating the message with the entity separate from the user comprises: causing an audio call to be initiated between the automated assistant and the entity, The message corresponds to audio data communicated between the entity and the automated assistant during the audio call.

9. The method of claim 7, wherein communicating the message with the entity separate from the user comprises: causing a text conversation between the automated assistant and the entity to be initiated, The message corresponds to text data communicated between the entity and the automated assistant during the text conversation.

10. The method according to any one of claims 7 to 9, further comprising: processing, by the automated assistant, context data indicating that the entity is available for communication with the automated assistant, The context data is provided by an application associated with the entity and distinct from the automated assistant.

11. A method implemented by one or more processors, the method comprising: receiving, by the automated assistant, user input seeking the automated assistant to assist the user in performing an activity during a subsequent time instance, wherein the automated assistant is accessible via a computing device associated with the user; After providing the user input to the automated assistant, processing, by the automated assistant, context data characterizing the context of the user, wherein the contextual data is processed to facilitate determining a particular arrangement of selectable elements to be rendered at a display interface or a separate display interface of the computing device to assist the user in performing the activity; In response to the context data indicating that the user is associated with a first context after providing the user input: causing a customized GUI to be rendered at the display interface or the separate display interface together with a first set of selectable elements, wherein each selectable element in the first set of selectable elements corresponds to a respective task in each task associated with the activity; and In response to the context data indicating that the user is associated with a second context after providing the user input: causing a different customized GUI to be rendered at the display interface or the separate display interface together with a second set of selectable elements, Each selectable element in the second set of selectable elements corresponds to a different respective specific task in the respective tasks associated with the activity.

12. The method of claim 11, wherein processing the context data characterizing the context of the user after providing the user input to the automated assistant comprises: determining, based on the contextual data, a current time of day after the user provided the user input, Wherein the particular arrangement of selectable elements to be rendered at the display interface or the separate display interface is selected based on the current time of day.

13. The method according to claim 12, wherein the user is associated with the first context when the current time of day corresponds to a calendar entry stored by a calendar application associated with the user, and Wherein when the current context does not correspond to the calendar entry stored by the calendar application associated with the user, the user is associated with the second context.

14. The method of claim 11 or 12, wherein processing the context data characterizing the context of the user after providing the user input to the automated assistant comprises: determining, based on the contextual data, a current location of the user after the user provided the user input, Wherein the particular arrangement of selectable elements to be rendered at the display interface or the separate display interface is selected based on the current location of the user.

15. The method of claim 14, wherein when the current location corresponds to a specific location, the user is associated with the first context, and When the current location does not correspond to the specific location, the user is associated with the second context.

16. The method of any one of claims 11 to 15, wherein the selectable elements in the first set of selectable elements include content provided by a first application, and the additional selectable elements in the second set of selectable elements include other content provided by a second application different from the first application.

17. The method of claim 16, further comprising: When the context data indicates that the user is associated with the first context after providing the user input: A determination is made to render the first set of selectable elements based on historical usage data indicating that the first application is associated with the first context.

18. A method as claimed in claim 16 or claim 17, wherein selectable elements in the first set of selectable elements are stored in association with links that, when selected, initialize the application to a first state, and Wherein additional selectable elements in the second set of selectable elements are stored in association with separate links that, when selected, initialize the application to a second state different from the first state.

19. The method of claim 18, further comprising: When the context data indicates that the user is associated with the first context after providing the user input: Determining to render the first set of selectable elements is based on historical usage data indicating that the user previously accessed the first state of the application in the first context.

20. The method of claim 19, wherein the first context corresponds to a specific time and / or a specific location, and the second context corresponds to a separate time and / or a separate location.

21. A system comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform the operations according to any one of claims 1 to 20.

22. A computer-readable storage medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 20.

Images