Proactively Incorporating Unsolicited Content into Human-Computer Dialogue

Abstract

The automated assistant is configured to proactively incorporate unsolicited content of potential interest to the user into a human-computer dialogue session. The method includes determining that the automated assistant has responded to all natural language inputs received from the user in an existing human-to-computer dialog session between the user and the automated assistant. Information of potential interest to the user or actions of potential interest to the user are identified based on characteristics of the user. Unsolicited content indicating the information or actions of potential interest to the user is generated by the automated assistant and incorporated into the existing human-to-computer dialog session. The incorporation is performed in response to determining that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session.

Description

[Technical field]

[0001] The present invention relates to the proactive incorporation of unsolicited content into human-to-computer dialogue. [Background technology]

[0002] Humans can engage in human-computer dialogues using interactive software applications referred to herein as “automated assistants” (which may also be referred to as “chatbots”, “conversational personal assistants”, “intelligent personal assistants”, “conversational agents”, etc.). For example, a human (who may be referred to as a “user” when interacting with an automated assistant) can provide commands and / or requests using spoken natural language input (i.e., speech), which may in some cases be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input. Automated assistants are generally reactive, as opposed to proactive. For example, at the beginning of a human-computer dialogue session between a user and an automated assistant (e.g., when no currently adaptable conversational context exists), the automated assistant may at best provide a general greeting, such as “hey”, “good morning”, etc. An automated assistant does not proactively obtain and provide specific information of potential interest to the user. Thus, the user must provide an initial natural language input (e.g., spoken or typed) before the automated assistant will respond with substantive information and / or initiate one or more tasks on behalf of the user. Summary of the Invention [Means for solving the problem]

[0003] Techniques are described herein for configuring an automated assistant to proactively incorporate Unsolicited Content of potential interest to a user into an already existing or newly initiated human-to-computer dialog session. In some implementations, an automated assistant configured with selected aspects of the present disclosure, and / or one or more other components working in conjunction with the automated assistant, can perform such incorporation when it determines that the automated assistant has effectively fulfilled its obligations to the user in an existing human-to-computer dialog session (e.g., the automated assistant is waiting for further instructions). This may be as simple as a user uttering "Good morning" and the automated assistant providing a generic response "Good morning." In such a scenario, the user may still be (at least briefly) involved in the human-to-computer dialog session (e.g., a chatbot screen showing an ongoing transcript of the human-to-computer dialog may still be open, the user may still be within hearing range of an audio input / output device through which the human-to-computer dialog is implemented, etc.). Thus, any unsolicited content embedded within a human-to-computer dialogue session has a high likelihood of being consumed (e.g., heard, seen, recognized, understood, etc.) by the user.

[0004] Incorporating unsolicited content of potential interest to a user into a human-computer dialogue session may have several technical advantages. It may free the user from affirmatively requesting such content, which may conserve computing resources that would otherwise be used to process the user's natural language input, and / or may be useful for users with reduced capacity to provide input (e.g., driving, physical limitations, etc.). In addition, the user may receive potentially useful content that may not have otherwise been requested by the user. As yet another example, the incorporation of unsolicited content may provide the user with information that the user may have otherwise sought by submitting an additional request. Avoiding such additional requests may conserve computing resources (e.g., network bandwidth, processing cycles, battery power) required to parse and / or interpret these requests.

[0005] In some implementations, the automated assistant can initiate a human-to-computer dialog (to incorporate unsolicited content) and / or incorporate unsolicited content into an existing human-to-computer dialog in response to various events. In some implementations, an event can include a determination that a user is within hearing range of the automated assistant. For example, a standalone conversational speaker running the automated assistant can detect that a user is nearby, for example, using various types of sensors (e.g., IP webcams, or motion sensors / cameras built into appliances such as smart thermostats, smoke detectors, carbon monoxide detectors, etc.) or by detecting the co-presence of another computing device carried by the user. In response, the automated assistant can audibly provide unsolicited content to the user, such as "Don't forget your umbrella today, it's forecasting rain," "Don't forget it's your sister's birthday today," "Did you hear about the injury to the power forward of <sports team> last night?", "<stock> has gone up 8% in the last few hours," etc.

[0006] In some implementations, an automated assistant running on a first computing device of a coordinated ecosystem of computing devices associated with a user can receive one or more signals from another computing device of the ecosystem. Such signals can include a computing interaction by the user (e.g., the user is performing a search, researching a topic, reading a particular article), the state of an application running on another computing device (e.g., consuming media, playing a game, etc.), and the like. For example, assume that a user is listening to a particular musical artist on a standalone interactive speaker (which may or may not be running an instance of an automated assistant). The automated assistant on the user's smartphone can audibly detect the music and / or receive one or more signals from the standalone interactive speaker and, in response, incorporate unsolicited content into a newly initiated or existing human-to-computer dialogue, such as additional information about the artist (or song), recommendations about other similar artists / songs, notifications of upcoming tour dates or other artist-related events, and the like.

[0007] The unsolicited content embedded within the human-to-computer dialogue may include information of potential interest to the user (e.g., weather, scores, traffic information, answers to questions, reminders, etc.) and / or actions of potential interest to the user (e.g., play music, create a reminder, add or create an item to a shopping list, etc.). The information and / or actions of potential interest are selected based on various signals. In some implementations, the signals may include past human-to-computer dialogues between the user and the automated assistant. Assume that during a first human-to-computer session, the user researches flights to a particular destination but does not purchase a ticket. Assume further that a subsequent human-to-computer dialogue between the automated assistant and the user is triggered and the automated assistant determines that it has responded to all natural language inputs from the user. In such a scenario, the user has not yet provided any additional natural language inputs to the automated assistant. Thus, the automated assistant can take the initiative to incorporate unsolicited content containing information about the user's previous flight search, such as "Have you already purchased a ticket to your destination?" or "I don't know if you're still looking for flights, but I found some cheap flights on <website>."

[0008] Other signals that may be used to select information and / or actions to be incorporated as unsolicited content within the human-to-computer dialogue include, but are not limited to, user location (e.g., the automated assistant may be prompted to proactively suggest certain menu items, specials, etc.), calendar entries (e.g., "Remember your anniversary is next Monday"), appointments (e.g., an upcoming flight may prompt the automated assistant to proactively remind the user to perform online check-in and / or start packing), reminders, search history, browsing history, topics of interest (e.g., interest in a particular sports team may prompt the automated assistant to proactively ask the user, "Did you see the score last night?"). In some cases, the automated assistant may proactively remind the user about the upcoming event, such as by sending a notification to the user, or by sending a notification to the user. In other cases, the automated assistant may proactively remind the user about the upcoming event, such as ...

[0009] In some implementations, a method is provided that includes the steps of: one or more processors determining, in an existing human-to-computer dialog session between the user and the automated assistant, that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session; one or more of the processors identifying information of potential interest to the user or one or more actions of potential interest to the user based on one or more characteristics of the user; one or more of the processors generating unsolicited content indicating the information of potential interest to the user or the one or more actions of potential interest to the user; and the automated assistant incorporating the unsolicited content into the existing human-to-computer dialog session. In various implementations, at least the incorporating step is performed in response to determining that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session.

[0010] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0011] In various implementations, the unsolicited content may include unsolicited natural language content. In various implementations, the identifying step may be based at least in part on one or more signals obtained from one or more computing devices operated by the user. In various implementations, the one or more computing devices operated by the user may include a given computing device currently operated by the user.

[0012] In various implementations, the one or more signals may be received from another of the one or more computing devices operated by the user that is different from the given computing device currently operated by the user. In various implementations, the one or more signals may include an indication of a state of an application running on the other computing device. In various implementations, the indication of the state of the application may include an indication that the application is providing media playback. In various implementations, the indication of the state of the application may include an indication that the application has received a search query from the user or provided search results to the user.

[0013] In various implementations, the method may further include one or more of the processors determining a desire criterion indicative of a desire of the user to receive the unsolicited content, the desire criterion being determined based on the one or more signals, and at least the incorporating step being performed in response to a determination that the desire criterion satisfies one or more thresholds. In various implementations, the unsolicited content may include one or more user interface elements, each selectable by a user to cause the automated assistant to provide information of potential interest to the user or to trigger one or more actions of potential interest to the user.

[0014] In another aspect, a method may include determining that a user is within hearing range of one or more audio output devices based on one or more signals, identifying information of potential interest to the user or one or more actions of potential interest to the user based at least in part on one or more characteristics of the user, generating unsolicited content indicative of the information of potential interest or the one or more actions of potential interest, and incorporating the unsolicited content into an audible human-to-computer dialogue session between the automated assistant and the user. In various implementations, the incorporating may be performed by the automated assistant in response to determining that the user is within hearing range of the one or more audio output devices.

[0015] In addition, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause any of the aforementioned methods to be performed. Some implementations also include one or more non-transitory computer-readable storage media having stored thereon computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0016] It should be appreciated that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Diagram 2]1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Diagram 3] 1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Figure 4] 1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Diagram 5] 1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Figure 6] 1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Figure 7] 1A-1D illustrate example dialogues between various users and automated assistants according to various implementations. [Figure 8] 1 is a flow diagram illustrating an example method according to an implementation disclosed herein. [Figure 9] 1 is a flow diagram illustrating an example method according to an implementation disclosed herein. [Figure 10] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] 1, an example environment in which the techniques disclosed herein may be implemented is shown. The example environment includes a plurality of client computing devices 106. 1~N and an automated assistant 120. The automated assistant 120 is connected to the client computing device 106 in FIG. 1~N Although shown as separate from the client computing device 106, in some implementations, all or aspects of the automated assistant 120 may be implemented as part of the client computing device 106. 1~NFor example, the client device 1061 may implement an instance of one or more aspects of the automated assistant 120, and the client device 106 N It is also possible for each of the client computing devices 106 to implement separate instances of one or more of those aspects of the automated assistant 120. 1-N In an implementation implemented by one or more computing devices remote from the client computing device 106 1-N and those aspects of the automated assistant 120 may communicate over one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0019] Client Device 106 1~Nmay include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, a vehicle navigation system), a standalone interactive speaker, and / or a wearable device of the user including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality computing device or an augmented reality computing device). Additional and / or alternative client computing devices may be provided. In some implementations, a given user may utilize multiple client computing devices that collectively form a coordinated "ecosystem" of computing devices to communicate with the automated assistant 120. In some such implementations, the automated assistant 120 may be considered to "serve" that particular user, e.g., to give the automated assistant 120 enhanced access to resources (e.g., content, documents, etc.) whose access is controlled by the "served" user. However, for the sake of brevity, some examples described herein will focus on a user operating a single client computing device 106.

[0020] Client Computing Device 106 1~N Each of the message exchange clients 107 1~N A variety of different applications can be run on the messaging client 107, such as a corresponding one of the following: 1~N may appear in a variety of forms, which may be implemented by the client computing device 106 1~N and / or multiple forms may vary depending on the client computing device 106 1~NIn some implementations, the messaging client 107 may be operated on a single client computing device. 1~N One or more of the messaging clients 107 may appear in the form of a short messaging service ("SMS") and / or multimedia messaging service ("MMS") client, an online chat client (e.g., an instant messenger, Internet Relay Chat, or "IRC"), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with the automated assistant 120, and the like. In some implementations, the messaging client 107 1~N One or more of may be implemented via a web page or other resource rendered by a web browser (not shown) or other application of the client computing device 106.

[0021] In addition to the messaging client 107, the client computing device 106 1~N Each of these applications can be used to access various other applications (e.g., “MISC.APP” 109 in Figure 1). 1~N ) can also be run on the device. These other apps may include, but are not limited to, gaming applications, media playing applications (e.g., music players, video players, etc.), productivity applications (e.g., word processors, spreadsheet applications, etc.), web browsers, map applications, reminder applications, cloud storage applications, photography applications, etc. As described in more detail below, these other applications 109 1~N The various states of can be used, in some embodiments, as signals to prompt the automated assistant 120 to incorporate unsolicited content into the human-to-computer dialogue.

[0022] As described in more detail herein, the automated assistant 120 may be configured to communicate with one or more client devices 106. 1~N In some implementations, the automated assistant 120 may be connected to a client device 106 and may engage in human-to-computer dialogue sessions with one or more users via the client device 106's user interface input devices and user interface output devices. 1~N In some of these implementations, the user interface input is explicitly targeted to the automated assistant 120. For example, the messaging client 107 may be configured to communicate with the automated assistant 120 in response to user interface input provided by the user via one or more user interface input devices of one of the messaging clients 107. 1~N One of the messaging services may be a personal assistant messaging service dedicated to conversations with the automated assistant 120, and user interface input provided via the personal assistant messaging service may be automatically provided to the automated assistant 120. Also, for example, the user interface input may be provided to the messaging client 107 based on a particular user interface input that explicitly indicates that the automated assistant 120 should be invoked. 1~N, may be explicitly targeted to the automated assistant 120 in one or more of the user interface inputs. For example, a particular user interface input may be one or more typed characters (e.g., @AutomatedAssistant), a user interaction with hardware and / or virtual buttons (e.g., tap, long tap), a verbal command (e.g., “hey, automated assistant”), and / or other particular user interface input. In some implementations, the automated assistant 120 can engage in a dialogue session in response to a user interface input even when the user interface input is not explicitly targeted to the automated assistant 120. For example, the automated assistant 120 can inspect the content of the user interface input and engage in a dialogue session in response to the presence of certain terms in the user interface input and / or based on other cues. In many implementations, the automated assistant 120 can engage in interactive voice response (“IVR”) such that a user can issue commands, searches, and the like, and the automated assistant can utilize natural language processing and / or one or more grammars to convert the utterances to text and respond to the text accordingly.

[0023] Client Computing Device 106 1~N Each of the client computing devices 106 and the automated assistant 120 may include one or more memories for storing data and software applications, one or more processors for accessing the data and executing the applications, and other components that facilitate communication over the network. 1~N Operations performed by one or more of and / or by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented, for example, as a computer program running on one or more computers in one or more locations coupled together through a network.

[0024] The automated assistant 120 may include a natural language processor 122 and a responsive content engine 130. In some implementations, one or more of the engines and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. The automated assistant 120 may be configured to communicate with an associated client device 106. 1~N via which the responsive content engine 130 may engage in human-to-computer dialogue sessions with one or more users to provide responsive content that is generated and / or maintained by the responsive content engine 130.

[0025] In some implementations, the responsive content engine 130 may communicate with the client device 106 during a human-computer dialogue session with the automated assistant 120. 1~N The responsive content engine 130 generates the responsive content in response to various inputs generated by a user of one of the client devices 106. The responsive content engine 130 provides the responsive content (e.g., over one or more networks, when separate from the user's client device) for presentation to the user as part of a dialog session. For example, the responsive content engine 130 may be 1~N Responsive content can be generated in response to free-form, natural language input provided via one of the following: As used herein, free-form input is input that is constructed by a user and is not constrained to a group of options presented for selection by the user.

[0026] As used herein, a "dialog session" may include a logically independent exchange of one or more messages between a user and the automated assistant 120. The automated assistant 120 may distinguish between multiple dialog sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled appointment, etc.) between sessions, detection of one or more intervening interactions between the user and the client device other than the dialog between the user and the automated assistant (e.g., the user switches applications for a while, the user walks away from a standalone voice-activated product and then returns later), locking / sleeping the client device between sessions, changes in the client device used to interact with one or more instances of the automated assistant 120, etc.

[0027] In some implementations, when the automated assistant 120 provides a prompt requesting user feedback, the automated assistant 120 may preemptively activate one or more components of the client device (through which the prompt is provided) configured to process user interface input to be received in response to the prompt. For example, if the user interface input is to be provided through the microphone of the client device 1061, the automated assistant 120 may preemptively "open" the microphone (so that an interface element or a "hot word" does not have to be hit to open the microphone), preemptively activate a local voice to the text processor of the client device 1061, preemptively establish a communication session between the client device 1061 and a remote voice to the text processor, and / or provide one or more commands to cause a graphical user interface (e.g., an interface including one or more selectable elements that may be selected to provide feedback) to be rendered on the client device 1061. This allows the user interface input to be provided and / or processed more quickly than if these components were not preemptively activated.

[0028] The natural language processor 122 of the automated assistant 120 is 1~N and generate annotated output for use by one or more other components of the automated assistant 120, such as the responsive content engine 130. For example, the natural language processor 122 can process natural language free-form input generated by a user via one or more user interface input devices of the client device 1061. The generated, annotated output includes one or more annotations of the natural language input and, in some cases, one or more (e.g., all) of the terms of the natural language input.

[0029] In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information within the natural language input. For example, the natural language processor 122 may include a portion of a speech tagger configured to annotate terms with their grammatical role. For example, the portion of the speech tagger may tag each term with its part of speech, such as "noun," "verb," ​​"adjective," "pronoun," etc. Also, for example, in some implementations, the natural language processor 122 may additionally and / or alternatively include a dependency parser configured to determine syntactic relationships between terms within the natural language input. For example, the dependency parser may determine which terms modify other terms, subjects and verbs of a sentence, etc. (e.g., a parse tree) and annotate such dependencies.

[0030] In some implementations, the natural language processor 122 may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (e.g., including literary properties), organizations, locations (real and imaginary), etc. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to entity classes, such as people), and / or at a low level of granularity (e.g., to enable identification of all references to a particular entity, such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities, and / or in some cases may communicate with a knowledge graph or other entity database to resolve particular entities.

[0031] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be utilized to resolve the term "there" to "Hypothetical Cafe" in the natural language input "I liked the Hypothetical Cafe where I ate last time."

[0032] In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122. For example, in some implementations, a named entity tagger may rely on annotations from a coreference resolver and / or a dependency parser in annotating all references to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from a dependency parser in clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 122 may determine one or more annotations using related prior inputs and / or other related data external to the particular natural language input.

[0033] As described above, the responsive content engine 130 is 1~N The responsive content engine 130 may utilize one or more resources in generating suggestions and / or other content for providing during a dialog session with a user of one of the responsive content engines 130. In various implementations, the responsive content engine 130 may include an action module 132, an entity module 134, and a content module 136.

[0034] The action module 132 of the responsive content engine 130 is 1~NThe natural language input received from and / or annotations of the natural language input provided by the natural language processor 122 are utilized to determine at least one action responsive to the natural language input. In some implementations, the action module 132 can determine an action based on one or more terms included in the natural language input. For example, the action module 132 can determine an action based on a mapping, in one or more computer-readable media, of an action to one or more terms included in the natural language input. For example, the action "add <item> to my shopping list" can be mapped to one or more terms, such as "I need <item> from the market...", "I need to get <item>", "We're out of <item>", etc.

[0035] The entity module 134 determines the candidate entities based on input provided by one or more users via user interface input devices during a dialog session between the users and the automated assistant 120. The entity module 134 utilizes one or more resources in determining the candidate entities and / or in refining those candidate entities. For example, the entity module 134 can utilize the natural language input itself and / or annotations provided by the natural language processor 122.

[0036] The proactive content module 136 may be configured to proactively incorporate unsolicited content of interest to the user into an existing or newly initiated human-computer dialogue session. For example, in some implementations, the proactive content module 136 may determine, based on data received from other modules, such as, for example, the natural language processor 122, the action module 132, and / or the entity module 134, that in an existing human-computer dialogue session between the user and the automated assistant 120, the automated assistant 120 has responded to all natural language inputs received from the user during the human-computer dialogue session. Assume that the user operates the client device 106 to request a search for specific information, and the automated assistant 120 performs (or causes a search to be performed) the search and returns response information as part of the human-computer dialogue. At this point, unless the user has also requested other information, the automated assistant 120 has sufficiently responded to the user's request. In some implementations, the proactive content module 136 may wait for some pre-determined time interval (e.g., 2 seconds, 5 seconds, etc.) for the automated assistant 120 to receive additional user input. If none is received during that time interval, the proactive content module 136 may determine that it has responded to all natural language input received from the user during the human-computer dialogue session.

[0037] The proactive content module 136 may further be configured to identify information of potential interest to the user or one or more actions of potential interest to the user (collectively referred to herein as "content" of potential interest to the user) based on one or more characteristics of the user. In some implementations, this identification of content of potential interest to the user may be performed by the proactive content module 136 at various time intervals (e.g., periodically, continuously, periodically, etc.). Thus, in some such implementations, the proactive content module 136 may be "ready" to continuously (or at least periodically) provide unsolicited content of potential interest to the user. Additionally or alternatively, in some implementations, this identification of content of potential interest may be performed by the proactive content module 136 in response to various events. One such event may be a determination that the automated assistant 120 has responded to all natural language inputs received from the user during the human-computer dialogue and no additional user inputs have been received by the expiration of the aforementioned time interval. Other events that may trigger the proactive content module 136 to identify content of potential interest to the user may include, for example, the user performing a search using the client device 106, the user running a particular application on the client device 106, the user traveling to a new location (e.g., as detected by a location coordinate sensor of the client device or by the user "checking in" to a location on social media), the user being detected to be within hearing range of a speaker under the control of an automated assistant, etc.

[0038] User characteristics that may be used, for example, by the proactive content module 136 to determine content of potential interest to the user may come in various forms and may be determined from various sources. For example, topics of interest to the user may be determined from sources such as the user's search history, browsing history, user setting preferences, location, media play history, travel history, past human-computer dialogue sessions between the user and the automated assistant 120, etc. Thus, in some implementations, the proactive content module 136 may have access to various signals or other data from one or more client devices 106 operated by the user, for example, directly from the client device 106 and / or indirectly through one or more computing systems operating as a so-called "cloud". Topics of interest to the user may include, for example, a particular hobby (e.g., golf, skiing, games, painting, etc.), literature, movies, music genres, a particular entity (e.g., an artist, an athlete, a sports team, a company), etc. Other characteristics of the user may include, for example, age, location (e.g., determined from a location coordinate sensor of the client device 106, such as a Global Positioning System ("GPS") sensor or other triangulation-based location coordinate sensor), user setting preferences, whether the user is currently in a moving vehicle (e.g., determined from an accelerometer of the client device 106), the user's scheduled events (e.g., determined from one or more calendar entries), and the like.

[0039] In various implementations, the proactive content module 136 may be configured to generate unsolicited content that indicates information of potential interest to the user and / or one or more actions of potential interest to the user and incorporate the unsolicited content into the human-to-computer dialogue. This unsolicited content may appear in various forms that may be incorporated into an existing human-to-computer dialogue session. For example, in some implementations where a user is interacting with an automated assistant 120 using a text-based messaging client 107, the unsolicited content generated by the proactive content module 136 may take the form of text, images, videos, or any combination thereof that may be incorporated into a transcript of the human-to-computer dialogue rendered by the messaging client 107. In some implementations, the unsolicited content may include or take the form of so-called "deep links" that are selectable by the user to expose a different application interface to the user. For example, a deep link, when selected by the user, may cause the client device 106 to launch (or operate) a particular application 109 in a particular state. In other implementations where the user is interacting with the automated assistant 120 using a voice interface (e.g., when the automated assistant 120 operates on a standalone interactive speaker or on an in-vehicle system), the unsolicited content may take the form of natural language output that is audibly provided to the user.

[0040] In some implementations, the incorporation of unsolicited content may be performed in response to a determination, for example, by proactive content module 136, that automated assistant 120 has responded to all natural language input received from the user during the human-computer dialog session. In some implementations, one or more of the other operations described above with respect to proactive content module 136 may also be performed in response to such an event. Or, as noted above, the operations may be performed by proactive content module 136 on a periodic or continuous basis such that proactive content module 136 (and thus automated assistant 120) remains "ready" to rapidly incorporate unsolicited content of potential interest to the user into an existing human-computer dialog session.

[0041] In some implementations, the automated assistant 120 may provide unsolicited output even before the initiation of a human-computer dialogue session by the user. For example, in some implementations, the proactive content module 136 is configured to determine that the user is within hearing range of one or more audio output devices (e.g., a standalone interactive speaker or a passive speaker operably coupled to the client device 106 that operates all or a portion of the automated assistant 120) based on one or more signals. These signals may include, for example, co-presence of one or more client devices carried by the user with the audio output device, detection of the physical presence of the user (e.g., passive infrared, sound detection (e.g., detection of the user's voice), etc.), and the like.

[0042] When the proactive content module 136 determines that the user is within hearing range of one or more audio output devices, the proactive content module 136 can identify information of potential interest to the user (as described above) or one or more actions of potential interest to the user based at least in part on one or more characteristics of the user, generate unsolicited content indicating the information of potential interest or one or more actions of potential interest, and / or incorporate the unsolicited content into an audible human-to-computer dialogue session between the automated assistant 120 and the user. As described above, one or more of these additional operations may be performed in response to determining that the user is within hearing range of an audio output device. Additionally or alternatively, one or more of these operations may be performed periodically or continuously such that the proactive content module 136 is always (or at least normally) "ready" to incorporate unsolicited content into the human-to-computer dialogue.

[0043] Figure 2 illustrates an example of a human-computer dialog session between a user 101 and an instance of an automated assistant (120 in Figure 1, not shown in Figure 2). Figure 2 illustrates an example of a dialog session that may occur between a user 101 of a computing device 210 (shown as a standalone interactive speaker, but this is not meant to be limiting) and the automated assistant 120 via a microphone and speaker according to implementations described herein. One or more aspects of the automated assistant 120 may be implemented on the computing device 210 and / or on one or more computing devices in network communication with the computing device 210.

[0044] In FIG. 2, user 101 initiates a human-to-computer dialog session between user 101 and automated assistant 120 by providing natural language input 280 of "Good morning. What's on the schedule today?". In response to natural language input 280, automated assistant 120 provides responsive natural language output 282 of "I have a dentist appointment at 9:30 AM, followed by a meeting at Hypothetical Cafe at 11 AM." Assuming there are only two events on the user's schedule for that day, automated assistant 120 has responded adequately to the user's natural language input (e.g., via action module 132). However, rather than waiting for additional user input, automated assistant 120 can proactively incorporate additional content of potential interest to the user into the human-to-computer dialog of FIG. 2 (e.g., via proactive content module 136). For example, the automated assistant 120 may search (or request another component to search) one or more travel routes between the dentist location and the appointment location and determine, for example, that the most direct route is undergoing major construction. Because the two appointments are relatively close to one another, the automated assistant 120 proactively incorporates the following unsolicited content (shown in italics) into the human-to-computer dialogue: "There is major construction going on on the direct route between your dentist and Hypothetical Cafe. How about taking <alternate route> instead?"

[0045] 3 illustrates another exemplary dialogue between user 101 and automated assistant 120 running on computing device 210 during different sessions. At 380, user 101 speaks the phrase, "What's the temperature outside?" After determining the outside temperature from one or more sources (e.g., weather-related web services), automated assistant 120 can respond, at 382, ​​with, "75 degrees Fahrenheit." Again, automated assistant 120 can determine (e.g., via proactive content module 136) that it has sufficiently responded to the user's natural language input. In response, based on user 101's interest in a particular team and a determination that that team won a game the previous night, automated assistant 120 can proactively incorporate the following unsolicited content into the human-to-computer dialogue: "Did you see that <team> won by 20 points last night?"

[0046] 4 illustrates another exemplary dialogue between a user 101 and an automated assistant 120 running on a computing device 210 during different sessions. In this example, the user 101 does not provide natural language input. Instead, the automated assistant 120 or another component running on the computing device 210 determines, for example, based on one or more signals provided by the client device 406 (in this example, a smartphone), that the user 101 is co-present with the computing device 210 and thus within hearing range of an audible output provided by the computing device 210. In response, at 482, the automated assistant 120 proactively incorporates unsolicited content (same as the unsolicited content of FIG. 3) into a new human-to-computer dialogue initiated by the automated assistant based on the co-presence of the user 101 with the computing device 210. The one or more signals provided by the client device 406 to the computing device 210 may include, for example, a wireless signal (e.g., Wi-Fi, BlueTooth), a shared network (e.g., the client device 406 has joined the same Wi-Fi network as the computing device 210), etc.

[0047] In some implementations, the automated assistant 120 can proactively incorporate other content of potential interest to the user into the human-to-computer dialogue as soon as it determines that the user 101 is co-present with the computing device 210. In some implementations, this other content can be determined, for example, based on the state of an application running on the client device 406. Assume that the user 101 is playing a game on the client device 406. The automated assistant 120 on the computing device 210 can determine that the client device 406 is in a particular gameplay state and can provide various unsolicited content of potential interest to the user, such as tips, tricks, recommendations of similar games, etc., as part of the human-to-computer dialogue. In some implementations where the computing device 210 is a standalone interactive speaker, the computing device 210 can even output background music (e.g., duplicating or adding background music) and / or sound effects related to the game being played on the client device 406, at least as long as the user 101 remains co-present with the computing device 210.

[0048] FIG. 5 illustrates an exemplary human-computer dialogue between a user 101 and an instance of an automated assistant 120 running on a client device 406. In this example, the user 101 again does not provide natural language input. Instead, the computing device 210 (again in the form of a standalone interactive speaker) is playing music. This music is detected at one or more audio sensors (e.g., microphones) of the client device 406. One or more components of the client device 406, such as a software application configured to analyze the audibly detected music, can identify one or more attributes of the detected music, such as artist / song. Another component, such as the entity module 134 of FIG. 1, can use these attributes to search one or more online sources for information about the entity. The automated assistant 120 running on the client device 406 can then provide (at 582) unsolicited content that informs the user 101 of various information about the entity, e.g., audibly via one or more speakers of the client device 406 of FIG. 5. For example, at 582 in Figure 5, the automated assistant 120 states, "You're listening to <artist>. Did you know <artist> has a tour date in <your city> on <date>?" Similar techniques can be applied by an instance of the automated assistant 120 running on a client device (e.g., a smartphone, tablet, laptop, standalone interactive speaker) when the instance detects (by sound or visual detection) that audiovisual content (e.g., a movie, a television program, a sporting event, etc.) is being presented on the user's television.

[0049] In FIG. 5, computing device 210 audibly outputs music that is "heard" by client device 406. Assume, however, that user 101 was listening to music using client device 406 instead of computing device 210. Assume further that user 101 was listening to music using earplugs such that the music was only audible to user 101 and not necessarily to other computing devices, such as computing device 210. In various implementations, particularly where client device 406 and computing device 210 are part of the same ecosystem of computing devices associated with user 101, computing device 210 can determine that a music playback application of client device 406 is in a state in which the application is currently playing music. For example, client device 406 can provide an indication of the state of the music playback application (and / or an indication of the state of other applications) to nearby devices (such as computing device 210) using wireless communication technologies, such as, for example, Wi-Fi, Bluetooth, etc. Additionally or alternatively, in the case of an ecosystem of computing devices operated by user 101, a global index of currently running applications and their respective states may be maintained (e.g., by an automated assistant serving user 101) and available among computing devices in the ecosystem. In either case, once the automated assistant 120 associated with computing device 210 learns the state of the music playback application on client device 406, the automated assistant 120 may proactively incorporate content similar to that shown at 582 in FIG. 5, for example, into the human-to-computer dialog between user 101 and automated assistant 120 via computing device 210 (which may be triggered by automated assistant 120).

[0050] 2-5 illustrate a human-computer dialogue in which the user 101 is engaged by the automated assistant 120 using audio input and output. However, this is not meant to be limiting. As mentioned above, in various implementations, the user can engage the automated assistant using other means, such as a messaging client 107. FIG. 6 illustrates an example in which a client device 606 in the form of a smartphone or tablet (not meant to be limiting) includes a touch screen 640. Visually rendered on the touch screen 640 is a transcript 642 of the human-computer dialogue between the user of the client device 606 ("you" in FIG. 6) and an instance of the automated assistant 120 running on the client device 606. An input field 644 is also provided in which the user can provide natural language content, as well as other types of input, such as images, sounds, etc.

[0051] In FIG. 6, a user initiates a human-to-computer dialog session with the question, "What time does <store> open?" The automated assistant 120 ("AA" in FIG. 6), for example, by action module 132 or another component, performs one or more searches on information about store hours and responds, "<store> opens at 10:00 AM." At this point, the automated assistant 120 has responded only to natural language input provided by the user in the current human-to-computer dialog session. However, for this example, assume that the user recently operated client device 606, or another client device in an ecosystem of client devices that also includes client device 606, to search for airline tickets to New York. The user may have performed this search by engaging in one or more human-to-computer dialog sessions with the automated assistant 120, by a web browser, or by some combination thereof.

[0052] Based on this past search activity, in some implementations, the automated assistant 120 (e.g., by the proactive content module 136) can search one or more online sources for information related to this search, and therefore of potential interest to the user, either periodically / continuously, or in response to a determination that the automated assistant 120 has responded to all received natural language inputs in a current human-to-computer dialog session. The automated assistant 120 can then proactively incorporate the following unsolicited content into the human-to-computer dialog session illustrated in FIG. 6: "Have you bought a flight to New York? We found great deals on direct flights and hotels." The automated assistant 120 can then (e.g., by the proactive content module 136) incorporate additional unsolicited content into the human-to-computer dialog in the form of a user interface element (e.g., a deep link) 646 that is selectable by the user to open a travel application installed on the client device 606. When user interface element 646 is selected, the travel application may open to a predefined state, e.g., already pre-populated with inputs for booking a flight to New York, so that the user can edit the pre-populated information and / or issue relatively few additional input commands to purchase an airline ticket. Other unsolicited content, such as one or more hyperlinks 648 to web pages for, e.g., booking an airline ticket or for general information about the destination, may be presented as selectable options as well.

[0053] The user interface element 646 in FIG. 6 is a visual element that may be selected by tapping on it or, in some cases, touching it, although this is not meant to be limiting. A human-to-computer dialogue similar to that shown in FIG. 6 may occur audibly between the user and an audio output device (e.g., a standalone interactive speaker as shown in the previous figure). In some such implementations, the user interface element may instead take the form of an audible prompt, such as a question or an option that may be "selected" if answered affirmatively by the user. For example, instead of presenting the visual user interface element 646, the automated assistant 120 may audibly output something like "Tell me if you want to open <travel application> so that you can book a flight to New York." In some implementations, the travel application itself may include its own automated assistant specifically tailored to relate to the human-to-computer dialogue with the user for booking travel arrangements. In some such implementations, the user may be "directed" to a travel application-specific automated assistant. In other implementations, the automated assistant 120 can utilize various information and state related to the travel application to construct a natural language output that solicits information from the user needed to book a flight using the travel application. The automated assistant 120 can then interact with the travel application on behalf of the user (e.g., in response to spoken natural language input provided by the user).

[0054] FIG. 7 again shows a client device 606 with a touch screen 640 and a user input field 644, as well as a transcript 742 of the human-to-computer dialogue session. In this example, a user ("you") initiates a human-to-computer dialogue by typing and / or speaking (which may be recognized and converted to text) the natural language input "Good morning." The automated assistant 120 ("AA") responds with "Good morning." At this point, there is no outstanding natural language request from the user that needs to be fulfilled. In response, using techniques described herein, the automated assistant 120 determines that the user has made a reservation at <restaurant> tonight, e.g., from a calendar associated with the user, from communications exchanged by the user with others (e.g., emails, text messages), from a restaurant application installed on the client device 606, etc. In response, the automated assistant 120 proactively incorporates the following statement into the human-to-computer dialogue: "Don't forget that you made a reservation at <restaurant> tonight." The automated assistant 120 then proactively incorporates a user interface element 746 selectable by the user to open the restaurant application ("My Reservations") where the user can view and / or edit / cancel the reservation. In addition, the automated assistant 120 proactively incorporates other content of potential interest to the user, such as hyperlinks 748 to websites related to the restaurant and / or attractions near the restaurant. In some implementations, when the user arrives at the restaurant, different unsolicited content, such as photos previously taken at the restaurant (by the user and / or by others), ratings, recommendations, specials, etc., can be incorporated by the automated assistant 120 into the same human-computer dialogue session or into a new human-computer dialogue session.

[0055] The examples of proactively incorporated unsolicited content described above are not meant to be limiting. Using the techniques described herein, other unsolicited content of potential interest to the user can be proactively incorporated into the human-to-computer dialogue. For example, in some implementations where a user has an upcoming flight (or train departure, or other travel arrangements) scheduled, the automated assistant 120 can proactively incorporate unsolicited content into a human-to-computer dialogue session with the user. This unsolicited content can include, for example, a reminder that the user's flight is approaching, one or more user interface elements selectable (by touch, voice, gesture, etc.) to open an application that allows the user to view or edit the scheduled flight, information about travel times to the airport (or selectable user interface elements that link to travel times), etc. Or, if the automated assistant 120 determines (e.g., based on the user's schedule, location coordinate sensors, etc.) that the user's flight has arrived at its destination, the automated assistant 120 can proactively incorporate various information and / or user interface elements that may be of interest to the user into a new or already existing human-to-computer dialogue session, such as information / user interface elements regarding calling a car service (or initiating a ride-sharing application), getting directions to a hotel or other attraction, nearby restaurants, etc.

[0056] As another example, the automated assistant 120 can determine that a change has been made to one or more computing devices operated by the user (which may in some cases be part of a coordinated ecosystem of computing devices associated with the user). For example, the automated assistant 120 can determine that one or more applications installed on one or more client devices associated with the user (including the automated assistant 120 itself) have been updated since the last human-to-computer dialogue session with the user. Because the user may potentially be interested in being informed about such updates, the automated assistant can incorporate unsolicited content into the human-to-computer dialogue, such as "Welcome back. While you were away, I learned how to call a cab. Let me know whenever you need a cab."

[0057] As yet another example, in some implementations, the automated assistant 120 can determine various information that may be of potential interest to the user at a particular time (e.g., depending on one or more topics of general interest to the user, the user's browsing history, etc.) and can proactively incorporate unsolicited content related to these information into a human-computer dialogue session with the user. For example, assume that a particular user is interested in history and electronics. In various implementations, when the automated assistant 120 determines that it has responded to all natural language inputs received from the user during an existing human-computer dialogue session, the automated assistant 120 can proactively incorporate information of potential interest to the user, for example, related to the current day. For example, on Nicola Tesla's birthday, a user who is interested in history and electronics can be presented with a user interface element selectable by the user to open an app or webpage related to Tesla. As another example, assume that today is the user's wedding anniversary. The automated assistant can proactively incorporate graphical elements or other information into an existing human-to-computer dialogue session that will likely be of interest to the user regarding the anniversary, such as links to flower websites, restaurants, etc.

[0058] As yet another example, in some implementations, the user's location (e.g., as determined by a position coordinate sensor of a computing device carried by the user) can prompt the automated assistant 120 to proactively incorporate unsolicited content into a human-to-computer dialog session with the user. For example, assume that the user is at or near a grocery store. The automated assistant 120 can determine, for example, based on one or more grocery store lists (e.g., stored locally on the client device or cloud-based) associated with the user, that the grocery store has items that the user is to pick up. The automated assistant 120 can then proactively incorporate unsolicited content into a human-to-computer dialog with the user, where the unsolicited content includes the needed items, information about those items, discounts available for those items, etc.

[0059] As yet another example, in some implementations, information or actions that a user frequently requests can be proactively incorporated into a human-to-computer dialogue session as unsolicited content. For example, assume that a user is conversing with the automated assistant 120 on various topics and it is approaching the time when the user would normally have dinner. In some implementations, the automated assistant 120 can incorporate unsolicited eating-related content into an existing human-to-computer dialogue session, such as user interface elements selectable by the user to order pizza, open a recipe (e.g., from local memory or from a frequently visited recipe webpage), etc. In other implementations, unsolicited content that can be incorporated into an existing human-to-computer dialogue session can include, but is not limited to, trending news articles, trending searches, updated search results for a search query previously issued by the user, and the like.

[0060] Of course, a user may not always desire unsolicited content. For example, a user may be driving in heavy traffic, may be in an emergency, may be operating a computing device in a manner that suggests that the user does not want to receive unsolicited content (e.g., in a video call), etc. Thus, in some implementations, the automated assistant 120 may be configured to determine a user's desire criteria for receiving unsolicited content (e.g., based on signals such as location signals, conversation context, state of one or more applications, accelerometer signals, etc.) and may provide the unsolicited content only if the criteria meet one or more thresholds.

[0061] Similarly, in some implementations, the automated assistant 120 can provide unsolicited content (as part of a new or existing human-computer dialogue session) during a particular period of time. For example, if a user is detected within hearing range of a client device running the automated assistant 120 between 7:00 and 8:00 a.m., the automated assistant 120 can automatically output unsolicited greetings such as "Good morning," "It's raining, don't forget your umbrella," "There's traffic on the 405," "Today's headlines are...," "Today's schedule is...," etc.

[0062] As another example, in some implementations, the automated assistant 120 can take into account the activity of multiple users at a particular time and / or location to determine that a particular user is likely to receive unsolicited content. In various implementations, the automated assistant 120 can analyze search queries from multiple users to identify spikes, trends, and / or other patterns in searches that may be related to a particular location, a particular time of day, etc. For example, assume that many users visiting a historical site perform similar web searches on their mobile devices, such as "how many stories does it have," "when was it built," "how old is it," etc. After detecting clear patterns or trends in these searches, the automated assistant 120 can proactively provide unsolicited content to new users when they arrive at the historical site.

[0063] 8 is a flow diagram illustrating an example method 800 according to an implementation disclosed herein. For convenience, the operations of the flow diagram are described with reference to a system that performs those operations. The system may include various components of various computer systems, such as one or more components of the automated assistant 120. Moreover, although the operations of the method 800 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0064] In block 802, the system can determine that in an existing human-computer dialog session between the user and the automated assistant, the automated assistant has responded to all natural language inputs received from the user during the human-computer dialog session. In some implementations, this may include waiting some pre-determined time interval to elapse after responding to all natural language inputs, although this is not required.

[0065] In some implementations, the system may proceed with execution of one or more of operations 806-810 only in response to a further determination in block 804 that the user is likely to desire unsolicited content (i.e., as may be represented by the "desire criteria" discussed above). This further determination may be made based on various sources, such as the conversational context of the human-computer dialogue session, the user's context determined from signals unrelated to the human-computer dialogue session (e.g., location signals, accelerometer signals, etc.), or a combination thereof. For example, if the user is determined to be currently driving based on the user's accelerometer and / or location coordinate sensor signals (e.g., after the user asks for traffic updates or directions), the system may determine that the user is unlikely to desire to be distracted by unsolicited content. As another example, the context of the human-computer dialogue session may suggest that the user would not want to be distracted by unsolicited content. For example, if a user asks an automated assistant about the location of the nearest emergency room or about treatment for an injury, the determined desirability criteria may be relatively low (e.g., unable to meet a threshold) and the automated assistant may refrain from providing unsolicited content as a follow-up to the requested information. As yet another example, if a user asks an automated assistant to trigger some action (e.g., starting a video call, starting a phone call, playing a movie, etc.) that is likely to take some time to complete and will require the user's attention, the user will likely not want to be distracted by additional unsolicited content.

[0066] In block 806, the system can identify information of potential interest to the user or one or more actions of potential interest to the user based on one or more characteristics of the user. As described above, the operations of block 806 may be performed in response to the determinations of blocks 802-804, or may be performed on an ongoing basis such that the automated assistant is "ready" to provide unsolicited content at a given time. In various implementations, the automated assistant can identify information or actions of potential interest to the user based on various sources, including, but not limited to, the user's search history, browsing history, human-computer dialogue history (including the same session, and / or previous sessions on the same or different client device), the user's location (e.g., determined from the user's schedule, social network state (e.g., check-in), location coordinate sensor, etc.), schedule / calendar, general topics of interest to the user (which may be set manually by the user and / or learned based on the user's activity), etc.

[0067] At block 808, the system may generate unsolicited content indicating information of potential interest to the user or one or more actions of potential interest. This unsolicited content may include, for example, a natural language output (e.g., an audible output or visual form) that provides information of potential interest to the user in a natural language format, user interface elements (graphical or audible) selectable by the user to obtain additional information and / or trigger one or more tasks (e.g., setting a reminder, creating a calendar entry, making an appointment, opening an application in a pre-determined state, etc.), etc.

[0068] At block 810, the system can incorporate the unsolicited content generated at block 808 into an existing human-to-computer dialog session. For example, the unsolicited content can be presented as natural language output from an automated assistant, user interface elements such as cards, hyperlinks, audible prompts, and the like. Incorporating unsolicited content into an existing human-to-computer dialog is different from, for example, simply showing the information to the user (e.g., a card on a lock screen or pull-down menu). The user is already engaged in a human-to-computer dialog session with the automated assistant, and thus the unsolicited content may be more likely to be seen / heard and acted upon by the user than if the content were simply presented to the user on a lock screen (which the user may often ignore and / or may be inundated with too many notifications).

[0069] 9 is a flow diagram illustrating an example method 900 according to an implementation disclosed herein. For convenience, the operations of the flow diagram are described with reference to a system that performs these operations. The system may include various components of various computer systems, such as one or more components of the automated assistant 120. Moreover, although the operations of the method 900 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0070] In block 902, the system can determine that the user is within hearing range of one or more audio output devices (e.g., one or more speakers operably coupled to a computing device running an instance of an automated assistant, a standalone interactive speaker running an instance of an automated assistant, etc.) based on one or more signals. These signals can take various forms. In some implementations, the one or more signals can be triggered by a user-operated computing device separate from the system and received at one or more communication interfaces operably coupled to the one or more processors. For example, one computing device can push a notification to another computing device that the user is engaged in a particular activity, such as driving, operating a particular application (e.g., to play back music or movies). In some implementations, the one or more signals can include detection of coexistence of the system and the computing device. In some implementations, the one or more signals can include an indication of the state of an application running on a computing device separate from the system, such as a user preparing a document, performing various searches, playing back media, viewing photos, engaged in a phone / video call, etc. In some implementations, the human-to-computer dialogue may be initiated in response to determining that the user is within hearing range of one or more audio output devices, although this is not required. Blocks 904-908 of FIG. 9 may be similar to blocks 804-808 of FIG. 8. Although not shown in FIG. 9, in various implementations, the automated assistant may determine whether the user is likely to desire unsolicited content before providing the unsolicited content, as described above with respect to block 804.

[0071] 10 is a block diagram of an example computing device 1010 that may be optionally utilized to perform one or more of the techniques described herein. In some implementations, one or more of the client computing device, the automated assistant 120, and / or other components may include one or more components of the example computing device 1010.

[0072] The computing device 1010 typically includes at least one processor 1014 that communicates with a number of peripheral devices via a bus subsystem 1012. These peripheral devices may include, for example, a storage subsystem 1024 including a memory subsystem 1025 and a file storage subsystem 1026, user interface output devices 1020, user interface input devices 1022, and a network interface subsystem 1016. The input / output devices allow a user to interact with the computing device 1010. The network interface subsystem 1016 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0073] The user interface input devices 1022 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods of inputting information into the computing device 1010 or onto a communications network.

[0074] The user interface output devices 1020 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computing device 1010 to a user or to another machine or computing device.

[0075] The storage subsystem 1024 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1024 may include logic for performing selected aspects of the methods of Figures 8 and 9, as well as for implementing the various components illustrated in Figure 1.

[0076] These software modules are generally executed by the processor 1014 alone or in combination with other processors. The memory 1025 used within the storage subsystem 1024 may include several memories including a main random access memory (RAM) 1030 for storing instructions and data during program execution, and a read only memory (ROM) 1032 in which fixed instructions are stored. The file storage subsystem 1026 may provide persistent storage for program files and data files, and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 1026 within the storage subsystem 1024 or within other machines accessible by the processor 1014.

[0077] The bus subsystem 1012 provides a mechanism for allowing the various components and subsystems of the computing device 1010 to communicate with each other as intended. Although the bus subsystem 1012 is illustrated generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0078] The computing device 1010 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 1010 shown in Figure 10 is intended only as a specific example to illustrate some implementations. Many other configurations of the computing device 1010 are possible having more or fewer components than the computing device shown in Figure 10.

[0079] In situations where some implementations discussed herein may collect or use personal information about the user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and user activity and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only upon receiving explicit permission to do so from the associated user.

[0080] For example, a user is provided with control over whether a program or feature collects user information about a particular user or other users associated with that program or feature. Each user for whom personal information is collected is presented with one or more options to provide permission or authorization as to whether information is collected and what portions of the information will be collected to enable control over information collection related to that user. For example, one or more such control options may be provided to the user via a communication network. In addition, some data may be treated in one or more ways before the data is stored or used such that personally identifiable information is removed. In one example, the user's identity may be treated such that personally identifiable information cannot be determined. As another example, the user's geographic location may be generalized to a broader area such that the user's specific location cannot be determined. In the context of the present disclosure, any relationships captured by the system, such as parent-child relationships, may be maintained in a secure manner such that they are not accessible outside of the automated assistant, for example, using the relationships to parse and / or interpret natural language input.

[0081] In addition to the advantages described above, it should be appreciated that the techniques described herein can make the automated assistant appear more “natural” or “human” to the user, which may encourage increased interaction with the automated assistant.

[0082] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results, and / or one or more of the advantages described herein, may be utilized, and each such variation and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain, using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, the implementations described above are presented by way of example only, and it will be understood that within the scope of the appended claims and their equivalents, implementations other than those specifically described and claimed may be practiced. Implementations of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0083] 101 users 106 1~N Client Computing Device, Client Device 107 1~N Conversation Client 109 1~N Application, MISC.APP 120 Automated Assistants 122 Natural Language Processors 130 Responsive Content Engine 132 Action Module 134 Entity Module 136 Content Module, Proactive Content Module 210 Computing Devices 280 Natural Language Input 282 Responsive Natural Language Output 406 Client Device 606 Client Device 640 Touchscreen 642 Transcript 644 Input Fields 646 User interface elements (e.g. deep links) 746 User Interface Elements 748 Hyperlinks 800 ways 900 methods 1010 Computing devices 1012 Bus Subsystem 1014 Processor 1016 Network Interface Subsystem 1020 User interface output device 1022 User Interface Input Device 1024 Storage Subsystem 1025 Memory Subsystem 1026 File Storage Subsystem 1030 Main Random Access Memory (RAM) 1032 Read-Only Memory (ROM)

Claims

1. 1. A method implemented using one or more processors, comprising: In an existing human-to-computer dialog session between a user and an automated assistant occurring at a given computing device operated by the user, determining that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session; Detecting that the user has arrived at a location based on the user's schedule or one or more sensor signals generated by one or more computing devices operated by the user; generating unsolicited content comprising user interface elements selectable by the user to facilitate procurement of a service associated with the location, the unsolicited content comprising user interface elements for initiating an application for facilitating procurement of a service associated with the location, excluding directions, the application for facilitating procurement of the service comprising an application for a ride-hailing service; incorporating the unsolicited content by the automated assistant into the existing human-to-computer dialogue session, at least the incorporating being performed in response to determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialogue session; A method comprising:

2. The method of claim 1 , wherein the unsolicited content comprises unsolicited natural language content.

3. determining a desire criterion indicative of a desire of the user to receive the unsolicited content; the desirability criteria are determined based on one or more signals; The method of claim 1 , wherein at least the incorporating is performed in response to determining that the desired criteria meet one or more thresholds.

4. The method of claim 3 , wherein the one or more signals include an indication of a state of an application executing on another computing device.

5. The method of claim 4 , wherein the indication of the state of the application includes an indication that the application is providing media playback.

6. The method of claim 3 , wherein the one or more signals include an action that the user has requested the automated assistant to trigger during the human-to-computer dialogue session.

7. The method of claim 3 , wherein the one or more signals include a mention of an emergency in the human-to-computer dialogue session.

8. The method of claim 3 , wherein the one or more signals indicate that the user is driving.

9. The method of claim 3 , wherein the one or more signals include an accelerometer signal.

10. The method of claim 3 , wherein the one or more signals include one or more topics detected in the human-to-computer dialogue session.

11. The method of claim 10 , wherein the one or more topics include transportation or destinations.

12. 1. A system comprising: one or more processors; a memory operatively coupled to the one or more processors; a memory storing instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations comprising: In an existing human-to-computer dialog session between a user and an automated assistant occurring at a given computing device operated by the user, determining that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session; detecting that the user has arrived at a location based on the user's schedule or one or more sensor signals generated by one or more computing devices operated by the user; generating unsolicited content comprising user interface elements selectable by the user to facilitate procurement of a service associated with the location, the unsolicited content comprising user interface elements for initiating an application for facilitating procurement of a service associated with the location, excluding directions, the application for facilitating procurement of the service comprising an application for a ride-hailing service; and an operation of incorporating the unsolicited content by the automated assistant into the existing human-to-computer dialogue session, at least the incorporating being performed in response to determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialogue session. Including, the system.

13. The system of claim 12 , wherein the unsolicited content includes unsolicited natural language content.

14. the plurality of operations further comprising an operation of determining a desire criterion indicative of a desire of the user to receive unsolicited content; the desirability criteria are determined based on one or more signals; The system of claim 12 , wherein at least the incorporating is performed in response to determining that the desired criteria meet one or more thresholds.

15. The system of claim 14 , wherein the one or more signals include an indication of a state of an application executing on another computing device.

16. The system of claim 15 , wherein the indication of the state of the application includes an indication that the application is providing media playback.

17. 17. The system of claim 16, wherein the one or more signals include an action that the user has requested the automated assistant to trigger during the human-computer dialogue session.

18. The system of claim 14 , wherein the one or more signals include a mention of an emergency in the human-to-computer dialogue session.

19. 20. The system of claim 18, wherein the one or more signals indicate that the user is driving.

20. When executed by one or more processors, the one or more processors are In an existing human-to-computer dialog session between a user and an automated assistant occurring at a given computing device operated by the user, determining that the automated assistant has responded to all natural language inputs received from the user during the human-to-computer dialog session; Detecting that the user has arrived at a location based on the user's schedule or one or more sensor signals generated by one or more computing devices operated by the user; generating unsolicited content comprising user interface elements selectable by the user to facilitate procurement of a service associated with the location, the unsolicited content comprising user interface elements for initiating an application for facilitating procurement of a service associated with the location, excluding directions, the application for facilitating procurement of the service comprising an application for a ride-hailing service; incorporating the unsolicited content by the automated assistant into the existing human-to-computer dialogue session, at least the incorporating being performed in response to determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialogue session. A computer program that performs the following:

Images