Providing prompts in automated conversation sessions based on previous conversation content

Abstract

The present disclosure relates to providing prompts in automated conversation sessions based on previous conversation content. Methods, apparatus, and computer-readable media are provided for soliciting feedback from a user regarding one or more parameters of suggestions or other content provided by an automated assistant. The user's feedback can be used to influence future suggestions and / or other content that the automated assistant subsequently provides to the user and / or other users in future conversation sessions. In some embodiments, in a conversation session between a user and an automated assistant, the automated assistant provides content to the user—and the automated assistant provides a prompt that solicits user feedback related to the provided content in a future conversation session between the user and the automated assistant. In some of these embodiments, the prompt is provided after input from the user and / or output from the automated assistant in a future conversation session that is unrelated to the content provided in the previous conversation session.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710882194.6, filed on September 26, 2017. Technical Field

[0003] This application provides prompts in automated conversation sessions based on previous conversation content. Background Art

[0004] An automated assistant (also known as a "personal assistant module," "mobile assistant," or "chatbot") can be interacted with by a user through various computing devices, such as smartphones, tablet computers, wearable devices, automotive systems, standalone personal assistant devices, etc. The automated assistant receives input from the user (e.g., typed and / or spoken natural language input) and responds with responsive content (e.g., visual and / or auditory natural language output). Summary of the Invention

[0005] In response to input from a user during a conversational session, the automated assistant can provide suggestions and / or other content that is responsive to the input and includes one or more content parameters selected by the automated assistant from a plurality of candidate content parameters based on one or more factors. For example, in response to the input “where can I get a good burger?”, the automated assistant can identify multiple candidate restaurants that serve burgers, but can select only a subset of those candidate restaurants (e.g., one of them) for inclusion in content responsive to the input.

[0006] This specification relates to methods, apparatus, and computer-readable media related to soliciting feedback from a user regarding one or more content parameters of suggestions or other content provided by an automated assistant. The user's feedback can be used to influence future suggestions and / or other content that the automated assistant subsequently provides to the user and / or other users in future conversational sessions.

[0007] In some implementations, during a conversational session between a user and an automated assistant, the automated assistant provides content to the user—and the automated assistant provides a prompt to solicit user feedback related to the provided content in a future conversational session between the user and the automated assistant. In some of these implementations, the prompt is provided immediately following input from the user and / or output from the automated assistant in the future conversational session that is unrelated to the content provided in the previous conversational session.

[0008] As an example, a future conversation session may be initiated by a user, and the automated assistant may be prompted to perform some "routine action," such as providing a summary of the user's calendar entries, providing a news report to the user, playing music for the user, etc. The automated assistant may perform some or all of the routine actions and then provide the prompt. This may add some variation to the routine actions and / or enhance the user experience by injecting conversational prompts into everyday operations. Furthermore, as described above, user feedback (directly or indirectly) provided in response to the prompt may be used to influence future suggestions or other content subsequently provided by the automated assistant for presentation to the user—thereby increasing the likelihood that subsequent content provided to the user by the automated assistant will be relevant to the user. Such improvements to the relevance of content provided by the automated assistant may enable desired content to be provided to the user more quickly, which may reduce various computing resources that may be consumed in a more lengthy conversation that may otherwise be required to identify the desired content.

[0009] In some embodiments, a method performed by one or more processors is provided, comprising: receiving natural language input based on user interface input provided by the user via a user interface input device; and providing, as a reply to the natural language input by the automated assistant, content responsive to the natural language input. The content is provided for presentation to the user via a user interface output device, and includes at least one content parameter selected by the automated assistant from a plurality of candidate content parameters. The method further comprises providing a prompt, soliciting feedback on the selected content parameter, receiving additional input in response to the prompt, and using the additional input to influence a value stored in association with the content parameter as part of an additional conversational session between the user and the automated assistant, separated in time from the conversational session. The value stored in association with the content parameter influences future provision of further content containing the content parameter. The prompt is provided for presentation to the user via the user interface output device or an additional user interface output device. The prompt is generated based on the content parameters to solicit feedback on the content parameters, the content parameters previously selected by the automated assistant and previously provided for presentation to the user as part of the conversational session. Additional input in response to the prompt is based on additional user interface input provided by the user via a user interface input device or an additional user interface input device.

[0010] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0011] In some embodiments, the user initiates the additional conversation session with additional natural language input that is unrelated to the content of the previous conversation session. In some of these embodiments, the method further includes, as part of the additional conversation session: providing an additional conversation session output that is responsive to the additional natural language input and is also unrelated to the content of the previous conversation session. Providing the prompt occurs immediately after providing the additional conversation session output. In some versions of these embodiments, the method further includes determining that one or more criteria are satisfied by at least one of: additional natural language input and additional conversation session output. Providing the prompt can be further based on determining that the criteria are satisfied. For example, the criteria can include: the additional conversation session output has a certain semantic type and / or at least one n-gram in the n-gram set appears in the user prompt.

[0012] In some embodiments, the content is a suggestion for future action by the user, and the method further comprises: determining that the user acts on the suggestion after providing the content including the content parameters. Providing the prompt is further based on determining that the user acts on the suggestion.

[0013] In some embodiments, the user interface input device for generating additional input in response to the prompt or the additional user interface input device includes a microphone, and the method further includes: based on providing the prompt, pre-activating at least one component configured to process the user interface input provided via the microphone.

[0014] In some implementations, the user initiates the additional conversation session, and providing the prompt is contingent upon the user having initiated the additional conversation session.

[0015] In some implementations, the method further includes, as part of the additional conversation session: providing an additional conversation session output before providing the prompt. Providing the prompt occurs immediately after providing the additional conversation session output.

[0016] In some embodiments, the method further comprises: identifying additional content parameters provided to the user as part of an additional previous conversational session between the user and the automated assistant; and determining, based on one or more criteria, to provide a prompt based on the content parameters rather than an alternative prompt based on the additional content parameters. In some of these embodiments, the one or more criteria comprise a corresponding temporal proximity of the provision of the content parameters and the provision of the additional content parameters. In some of these embodiments, the one or more criteria comprise semantic types assigned to the content parameters and the additional content parameters.

[0017] In some implementations, the content includes a recommendation for a specific physical location and for a specific item that can be consumed at the specific location, and wherein the content parameter identifies the specific item.

[0018] In some implementations, the content includes a suggestion for a particular physical location, and wherein the content parameter identifies a category to which the particular physical location belongs.

[0019] In some embodiments, both the input and the additional input are generated via the user interface input device, and wherein both the content and the feedback prompt are provided for presentation via the user interface output device. In some of these embodiments, the user interface input device comprises a microphone of a single device, and the user interface output device comprises a speaker of the single device.

[0020] In some implementations, the input is generated via a user interface input device of a first computing device, the content is provided for presentation via a user interface output device of the first computing device, and the prompt is provided for presentation via an additional user interface output device of an additional computing device.

[0021] In some embodiments, a method performed by one or more processors is provided, comprising: identifying, from a computer-readable medium, stored content parameters for content previously provided to a user as part of a previous conversational session between a user and an automated assistant implemented by the one or more processors. The method further comprises, as part of an additional conversational session between the user and the automated assistant, temporally separate from the previous conversational session, providing a prompt soliciting feedback on the content parameters. The prompt is provided for presentation to the user via a user interface output device of the user's computing device, and the prompt is generated based on the content parameters previously provided for presentation to the user as part of the previous conversational session to solicit feedback on the content parameters. The method further comprises, as part of the additional conversational session, receiving additional input in response to the prompt, the additional input based on additional user interface input provided by the user via a user interface input device of the computing device. The additional input is based on the additional user interface input provided by the user via the user interface input device of the computing device, and the stored value influences future provision of further content including the content parameters.

[0022] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described above.

[0023] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram of an example environment in which embodiments disclosed herein may be implemented.

[0025] Figure 2A Illustrated is an example of a conversational session in which an automated assistant provides content to a user having content parameters selected from a plurality of candidate content parameters.

[0026] Figure 2B1 、 Figure 2B2 、 Figure 2B3 and Figure 2B4 The diagram shows Figure 2A The conversation session separated by the conversation session provides users with Figure 2A Different examples of feedback prompts for the selected content parameter.

[0027] Figure 3A Another example of a conversation session is illustrated in which an automated assistant provides content to a user having content parameters selected from a plurality of candidate content parameters.

[0028] Figure 3B The diagram shows Figure 3A The conversation session separated by the conversation session provides users with Figure 3A Example of a feedback prompt for the selected content parameter.

[0029] Figure 4 is a flowchart illustrating an example method according to implementations disclosed herein.

[0030] Figure 5 An example architecture for a computing device is illustrated. DETAILED DESCRIPTION

[0031] Now turn Figure 1 , illustrates an example environment in which the technology disclosed herein may be implemented. The example environment includes multiple client computing devices 106 1-Nand automated assistant 120. Although Figure 1 The automated assistant 120 is shown as being associated with the client computing device 106. 1-N Separately, in some embodiments, all or multiple aspects of automated assistant 120 may be managed by one or more client computing devices 106 1-N For example, client device 1061 may implement an instance or aspects of automated assistant 120, and client device 106 N Separate instances of these one or more aspects of automated assistant 120 may also be implemented. 1-N In some embodiments implemented as one or more computing devices, the client computing device 106 1-N Aspects of automated assistant 120 may communicate via one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (eg, the Internet).

[0032] For example, client device 106 1-N The client computing devices may include one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), and / or a wearable device of the user that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual reality or augmented reality computing device). Additional and / or alternative client computing devices may be provided. In some embodiments, a given user may utilize multiple client computing devices from a coordinated "ecosystem" of computing devices to communicate with automated assistant 120. However, for the sake of brevity, some examples described in this specification will focus on a user operating a single client computing device 106.

[0033] Client computing device 106 1-N Each of them can operate various different applications, such as message exchange client 107 1-N The corresponding one in the message exchange client 107 1-N can have various forms, and these forms can be across client computing devices 106 1-N Different and / or multiple formats may be used on a single client computing device 106 1-N In some embodiments, the message exchange client 107 1-NOne or more of the messaging clients 107 may be in the form of a Short Messaging Service ("SMS") and / or Multimedia Messaging Service ("MMS") client, an online chat client (e.g., an instant messenger, Internet Relay Chat, or "IRC"), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with automated assistant 120, and the like. In some embodiments, message exchange client 107 1-N One or more of may be implemented via a web page or other resource rendered by a web browser (not shown) or other application of the client computing device 106 .

[0034] As described in greater detail herein, automated assistant 120 communicates with one or more client devices 106 via user interface input. 1-N In some embodiments, the automated assistant 120 may be operable in response to a user's input via the client device 106. 1-N One or more user interface input devices provide user interface input to the dialog session with the user. In some of these embodiments, the user interface input is explicitly directed to the automated assistant 120. For example, the message exchange client 107 1-N One of these may be a personal assistant messaging service dedicated to conducting conversations with automated assistant 120, and user interface input provided via the personal assistant messaging service may be automatically provided to automated assistant 120. Furthermore, for example, based on a specific user interface input indicating that automated assistant 120 is to be invoked, the user interface input may be explicitly directed to automated assistant 120 in one or more messaging clients 1071. For example, the specific user interface input may be one or more typed characters (e.g., @AutomatedAssistant), user interaction with a hardware button and / or virtual button (e.g., a tap, a long tap), a spoken command (e.g., "Hey Automated Assistant"), and / or other specific user interface input. In some embodiments, automated assistant 120 may engage in a conversational session in response to a user interface input even when the user interface input is not explicitly directed to automated assistant 120. For example, automated assistant 120 may examine the content of the user interface input and engage in a conversational session in response to certain terms present in the user interface input and / or based on other cues.

[0035] Client computing device 106 1-NEach of the client computing device 106 and the automated assistant 120 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components to facilitate communication over the network. 1-N One or more of and / or the operations performed by automated assistant 120 can be distributed across multiple computer systems. For example, automated assistant 120 can be implemented as a computer program running on one or more computers in one or more locations coupled to each other via a network.

[0036] Automated assistant 120 may include a natural language processor 122, a response content engine 130, and a feedback engine 140. In some implementations, one or more of the engines and / or modules of automated assistant 120 may be omitted, combined, and / or implemented in a component separate from automated assistant 120. Automated assistant 120 communicates with the client device 106 via an associated client device 106. 1-N A dialog session is conducted with one or more users to provide response content generated by the response content engine 130 and / or to provide feedback prompts generated by the feedback engine 140 .

[0037] In some embodiments, response content engine 130 responds to a request from client device 106 during a conversation session with automated assistant 120. 1-N The response content engine 130 generates response content based on each input generated by a user of one of the client devices. The response content engine 130 provides the response content (e.g., when provided over one or more networks when separate from the user's client device) for presentation to the user as part of the conversation session. For example, the response content engine 130 may generate response content based on each input generated by a user of one of the client devices 106. 1-N Response content is generated by a free-form natural language input provided in one of the . As used herein, a free-form input is an input formulated by a user and not constrained by a group of options presented for selection by the user.

[0038] In response to some input, the responsive content engine 130 can generate content having one or more content parameters selected from a plurality of candidate content parameters. For example, the input provided can be "give me directions to a good coffee shop," and the responsive content engine 130 can determine directions to a particular coffee shop based on one or more factors by first selecting "good coffee shop" from a plurality of available coffee shops. Furthermore, the responsive content engine 130 can determine directions based on one or more factors by selecting them from a plurality of candidate directions (e.g., shortest vs. fastest; including highways vs. excluding highways). Additional descriptions of embodiments of the responsive content engine 130 are provided below.

[0039] In some embodiments, feedback engine 140 stores various content parameters selected by response content engine 130 and provided to the user in a conversation session—and generates feedback that solicits prompts from the user regarding one or more of the selected content parameters. Feedback engine 140 provides feedback prompts for presentation to the user as part of a conversation session. In many embodiments, feedback engine 140 provides feedback prompts for presentation to the user as part of a conversation session that is separate from the conversation session in which the content parameter that was the focus of the feedback prompt was provided to the user. Feedback engine 140 can further utilize feedback provided by the user in response to prompts to influence future suggestions and / or other content subsequently provided by automated assistant 120 to the user and / or other users in future conversation sessions.

[0040] As described above, in some embodiments, content is provided to a user via automated assistant 120 during a conversational session between the user and the automated assistant—and automated assistant 120 provides a feedback prompt in a future conversational session between the user and the automated assistant that solicits user feedback related to the provided content. In some of these embodiments, the prompt is provided in the future conversational session immediately following input from the user and / or output from automated assistant 120, where the input and / or output is unrelated to the content provided in the previous conversational session.

[0041] As used herein, a "conversation session" may include a logically self-contained exchange of one or more messages between a user and automated assistant 120. Automated assistant 120 may distinguish between multiple conversation sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intermediary interactions between the user and a client device other than the conversation between the user and the automated assistant (e.g., the user switches apps for a period of time, leaves, and then returns to a standalone voice-activated product), locking / hibernation of a client device between sessions, changes in the client device used to interface with one or more instances of automated assistant 120, and the like.

[0042] In some embodiments, when automated assistant 120 provides a prompt to solicit user feedback, automated assistant 120 can preemptively activate one or more components of the client device (via which the prompt is provided) that are configured to process user interface input to be received in response to the prompt. For example, in the case where user interface input is provided via a microphone of client device 1061, automated assistant 120 can provide one or more commands to cause: the microphone to be preemptively "turned on" (thereby avoiding the need to click an interface element or say a "hot word" to turn the microphone on), the local speech-to-text processor of client device 1061 to be preemptively activated, a communication session between client device 1061 and a remote speech-to-text processor to be preemptively established, and / or a graphical user interface to be rendered on client device 1061 (e.g., an interface including one or more selectable elements that can be selected to provide feedback). This can enable user interface input to be provided and / or processed more quickly than would be possible without preemptive activation of the components.

[0043] In some embodiments, automated assistant 120 can provide a prompt to solicit user feedback in a future conversation session based on user input and / or automated assistant 120's response output in the future conversation session that meets one or more criteria. For example, a prompt can be provided only when the user initiates a conversation session with any user input and / or a certain user input, such as any natural language input and / or a certain natural language input. Furthermore, for example, a prompt can be provided only when the user input and / or response output have one or more certain semantic types and / or do not belong to one or more other certain semantic types. As another example, a prompt can be provided only when the user input and / or response output include one or more specific n-grams and / or do not include one or more other specific n-grams. In some embodiments, criteria can be selected to increase the likelihood of providing a prompt when the conversation of the conversation session is conversational and / or light and / or to decrease the likelihood of providing a prompt when the conversation of the conversation session is "task-oriented" (e.g., to prevent distracting the user from the task).

[0044] In some embodiments, criteria may additionally or alternatively be selected to maintain the privacy of the user to whom the prompt is provided. For example, if speech-based natural language input is provided, the criteria may be that the speech-based input conforms to the user's speech profile and / or that the speech-based input lacks a certain amount of background noise (e.g., lacks background noise that indicates that other users may be present). Further, for example, the criteria may be that the user input and / or response output includes content that is private to the user (e.g., a response output that provides a summary of the user's calendar items for the day), which may indicate that the user is in a setting that the user has deemed private.

[0045] In some embodiments, content parameters from a plurality of different suggestions provided to a user may be available to the feedback engine 140 for use in generating a prompt to be provided to the user in a given future conversation session. In some of these embodiments, the feedback engine 140 may select a subset (e.g., one of) of those plurality of suggestions to provide in a given prompt based on one or more criteria. For example, whether to utilize the content parameters of a given suggestion when generating and / or providing a prompt may be based on the category of the given suggestion (e.g., a dining suggestion may be more likely to be selected than a music suggestion), the subcategory of the given suggestion (e.g., a French food suggestion may be more likely to be selected than a Mexican food suggestion), the time at which the given suggestion was provided (e.g., a more recent suggestion is more likely to be selected than a less recent suggestion), and / or the category of the given suggestion and the time at which the given suggestion was provided (e.g., a music suggestion from three days ago may not be provided, while a dining suggestion from three days ago may be provided).

[0046] In some embodiments, based on determining that the user actually acted on a suggestion, automated assistant 120 can provide a prompt to solicit feedback on content parameters of the provided suggestion that the user may act on in the future. This can mitigate the risk of the user being prompted to provide feedback on content parameters with which the user has never interacted, which could confuse the user and / or unnecessarily consume computing resources when providing the prompt. In some embodiments, determining that the user acted on the suggestion can be based on a conversation between the user and automated assistant 120 in a conversation session prior to the conversation session in which the prompt was provided (e.g., a conversation in which the user actually acted on the suggestion). In some embodiments, determining that the user acted on the suggestion can be based on one or more additional or alternative signals, such as signals generated by automated assistant 120 that indicate the user acted on the suggestion. For example, suppose automated assistant 120 provides a user with a suggestion for a coffee shop to visit. Before providing the user with a prompt to solicit feedback on the coffee shop in a future conversation session, it can first be determined that the user actually visited the coffee shop based on location data from the user's mobile client device, transaction data associated with the user, and / or other signals.

[0047] The natural language processor 122 of the automated assistant 120 processes the natural language input by the user via the client device 106 1-N The generated natural language input and generates annotated output for use by one or more other components of automated assistant 120, such as response content engine 130 and / or feedback engine 140. For example, natural language processor 122 can process natural language free-form input generated by a user via one or more user interface input devices of client device 1061. The generated annotated output includes one or more annotations of the natural language input and optionally one or more (e.g., all) terms of the natural language input.

[0048] In some embodiments, the natural language processor 122 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the natural language processor 122 may include a part-of-speech tagger configured to annotate terms with their grammatical role. For example, the part-of-speech tagger may label each term with its part of speech, such as "noun," "verb," ​​"adjective," "pronoun," and so on. Furthermore, for example, in some embodiments, the natural language processor 122 may additionally and / or alternatively include a dependency parser configured to determine syntactic relationships between terms in the natural language input. For example, the dependency parser may determine which terms modify other terms, subjects, and verbs of a sentence, and so on (e.g., a parse tree) - and may perform annotations of such dependencies.

[0049] In some embodiments, the natural language processor 122 may additionally and / or alternatively include an entity tagger configured to annotate entity references, such as references to people, organizations, locations, etc., in one or more segments. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity class, such as a person) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity, such as a specific person). The entity tagger may rely on the content of the natural language input to resolve a specific entity and / or may optionally communicate with a knowledge graph or other entity database (e.g., content database 152) to resolve a specific entity.

[0050] In some embodiments, the natural language processor 122 can additionally and / or alternatively include a coreference resolver that is configured to group or "cluster" entities based on one or more contextual clues. For example, a coreference resolver can be utilized to resolve the term "it" in the natural language input "I like the stir fry dish at Asia Village. Order it please." to "Asia Village."

[0051] In some embodiments, one or more components of the natural language processor 122 can rely on annotations from one or more other components of the natural language processor 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. Additionally, for example, in some embodiments, a coreference resolver can rely on annotations from a dependency resolver when clustering references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the natural language processor 122 can use relevant previous input and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0052] As described above, the responsive content engine 130 is useful in generating suggestions and / or other content to be used in communication with the client device 106. 1-N The responsive content engine 130 may include an action module 132 , an entity module 134 , a content generation module 136 , and an attribute module 138 .

[0053] The action module 132 of the response content engine 130 utilizes the 1-NAt least one action associated with the natural language input is determined based on the received natural language input and / or the annotation of the natural language input provided by the natural language processor 122. In some implementations, the action module 132 may determine the action based on one or more terms included in the natural language input. For example, the action module 132 may determine the action based on actions mapped to one or more terms included in the natural language input in one or more computer-readable media. For example, the action of "making a restaurant reservation" may be mapped to one or more terms such as "book it," "reserve," "reservation," "get me a table," and the like. Furthermore, for example, the action of "providing a daily briefing" may be mapped to one or more terms such as "tell me about my day," "what's going on today," "good morning," and the like. As another example, the action of "providing small talk" may be mapped to one or more terms such as "hello," "what's up," and the like.

[0054] In some implementations, action module 132 may determine an action based, at least in part, on one or more candidate entities determined by entity module 134 based on the natural language input of the conversation session. For example, assume the natural language input of "book it," and "book it" is mapped to multiple different actions, such as "making a restaurant reservation," "making a hotel reservation," "making an appointment," and so on. In such a scenario, action module 132 may determine which action is the correct action based on the candidate entities determined by entity module 134. For example, if entity module 134 only identifies multiple restaurants as candidate entities, action module 132 may determine the action of "making a restaurant reservation" as the correct action.

[0055] Entity module 134 determines candidate entities based on input provided by one or more users via user interface input devices during a conversational session between the user and automated assistant 120. Entity module 134 utilizes one or more resources in determining candidate entities and / or refining these candidate entities. For example, entity module 134 may utilize the natural language input itself, annotations provided by natural language processor 122, attributes provided by attribute module 138, and / or the contents of content database 152.

[0056] The content database 152 may be provided on one or more non-transitory computer-readable media and may define a plurality of entities, attributes of each of the entities, and optionally, relationships between the entities. For example, the content database 152 may include an identifier for a particular restaurant and one or more attributes of the restaurant, such as its location, type of cuisine, dishes available, hours of operation, nicknames, whether reservations are accepted, a rating for the restaurant, an indication of prices, and the like. The content database 152 may additionally or alternatively include a plurality of protocols, each of which may be applied to one or more entities and / or one or more actions. For example, each protocol may define one or more required and / or desired content parameters to perform an associated action and / or utilize an associated entity or entities to perform an associated action.

[0057] The content generation module 136 conducts a conversation with one or more users via an associated client device to generate suggestions for performing actions and / or generate other content. The content generation module 136 optionally utilizes one or more resources when generating content. For example, the content generation module 136 may utilize: the user's current and / or past natural language input during the conversation session, annotations of the input provided by the natural language processor 122, attributes provided by the attribute module 138, one or more entities determined by the entity module 134, and / or one or more actions determined by the action module 132.

[0058] The content generation module 136 can generate and provide content including one or more content parameters selected from a plurality of candidate content parameters in the conversation session. The content generation module 136 can also provide the selected content parameters of the provided content to the feedback engine 140 for generating feedback prompts as described herein.

[0059] As an example, input provided in a conversation session may be "give me directions to a good coffee shop," and content generation module 136 may determine directions to a specific coffee shop based on one or more factors by first selecting "good coffee shop" from a plurality of available coffee shops (e.g., based on candidate actions from action module 132 and / or candidate entities from entity module 134). Furthermore, content generation module 136 may determine directions based on one or more factors by selecting them from a plurality of candidate directions (e.g., shortest vs. fastest routes; including highways vs. excluding highways). Content generation module 136 may optionally communicate with one or more external components when determining the specific coffee shop and / or directions. Content generation module 136 may provide selected content parameters to feedback engine 140. For example, content generation module 136 may provide content parameters indicating the specific coffee shop selected, and may provide content parameters indicating one or more parameters of the selected directions (e.g., indicating that they are "fastest route" directions).

[0060] As another example, the input provided may be "tell me about my day," and the content generation module 136 may select one or more calendar entries of the user, the user's local weather, one or more news stories customized to the user, and / or other content to be provided in response to the input. The content generation module 136 may select content based on various factors such as the type of content (e.g., "calendar entry," "news story"), the individual and / or overall size of the content (e.g., length of time), etc. The content generation module 136 may provide selected content parameters to the feedback engine 140. For example, the content generation module 136 may provide content parameters indicating that "calendar entry" and "news story" are provided to the user and / or may provide content parameters indicating that the length of the audible presentation of the entire content in the conversation session is "2 minutes."

[0061] Attribute module 138 determines one or more attributes applicable to a user engaging in a conversational session with automated assistant 120 and provides these attributes to one or more other components of response content engine 130 for use in generating content to be provided to the user during the conversational session. For example, other components of response content engine 130 may utilize these attributes when determining a specific entity for an action, when determining one or more criteria for an action, and / or when generating output for a conversation with one or more users. Attribute module 138 is in communication with attribute database 156, which may store attributes that are personal to a user and / or applicable to a group of users that includes the user. As described herein, the values ​​of various attributes of attribute database 156 may be influenced based on input provided by one or more users in response to a feedback prompt. For example, an attribute of attribute database 156 may include a value associated with a particular restaurant that indicates the desirability of the particular restaurant for a particular user. This value may be based on input provided by the particular user in response to a feedback prompt for the particular restaurant provided to the particular user during the conversational session. The attribute database 156 may additionally or alternatively include other attributes that are personal to the user, but whose values ​​are not necessarily affected by the response to the feedback prompt. For example, such attributes may include the user's current location (e.g., based on GPS or other location data), the user's time constraints (e.g., based on the user's electronic calendar), and / or attributes of the user based on the user's actions across multiple Internet services.

[0062] As described above, the feedback engine 140 stores various content parameters selected by the response content engine 130 and provided to the user in the conversation session—and generates prompts that solicit feedback from the user regarding one or more of the content parameters. The feedback engine 140 may include a provided content parameter module 142, a prompt generation module 144, and a feedback module 146.

[0063] The provided content parameter module 142 stores content parameters provided for content presented to the user in a conversation session with the user in association with the user in the provided content parameter database 154. For example, in response to a provided input in a conversation session of "what's a good coffee shop," the content generation module 136 may select "good coffee shop" from a plurality of available coffee shops, provide a response output indicating the selected "good coffee shop," and provide content parameters indicating the selected "good coffee shop" to the feedback engine 140. The provided content parameter module 142 may store the provided content parameters in association with the user in the provided content parameter database 154.

[0064] Prompt generation module 144 generates a feedback prompt that solicits feedback from the user regarding one or more content parameters stored in association with the user in provided content parameter database 154. In some embodiments, prompt generation module 144 further determines when and / or how the feedback prompt should be provided to the user. For example, prompt generation module 144 may determine to provide a feedback prompt for presentation to the user as part of a conversation session based on: a conversation session separate from the conversation session that provided the user with the content parameter that is the focus of the feedback prompt; user input and / or response output of the conversation session meeting one or more criteria; based on verifying that the user acted in accordance with a suggestion based on the content parameter; and / or based on other criteria. Prompt generation module 144 may operate in cooperation with response content engine 130 to insert the generated feedback prompt into certain conversation sessions managed by response content engine 130. In some embodiments, prompt generation module 144 and / or other components of feedback engine 140 may be included in response content engine 130.

[0065] In some implementations, content parameters from a plurality of different suggestions provided to a user may be available to prompt generation module 144 for use in generating a prompt to provide to the user in a given conversation session. In some of these implementations, prompt generation module 144 may select a subset (e.g., one of) those plurality of suggestions to provide in a given prompt based on one or more criteria.

[0066] Feedback module 146 utilizes user-provided feedback in response to prompts that influence future recommendations and / or other content subsequently provided to the user and / or other users by automated assistant 120 in future conversational sessions. In some embodiments, feedback module 146 utilizes one or more instances of feedback to adjust values ​​associated with one or more attributes in attribute database 156. The adjusted values ​​can be personal to the user and / or applicable to a group of users (e.g., all users). In some embodiments, feedback module 146 can utilize annotations provided by natural language processor 122 when determining the impact one or more instances of feedback will have on the associated attributes. For example, natural language processor 122 can include a sentiment classifier that can provide annotations indicating the sentiment of the provided feedback, and feedback module 146 can utilize the indicated sentiment to adjust the value. For example, for the feedback prompt "How did you like Coffee Shop A?" and the user-provided "it was great," feedback module 146 can receive the annotation "it was great" associated with highly positive feedback. Based on such annotations, the feedback module 146 may increase the value of the attribute associated with "CoffeeShop A."

[0067] Now refer to Figure 2A 、 Figure 2B1 、 Figure 2B2 、 Figure 2B3 ,and Figure 2B4 , which describes examples of various embodiments disclosed herein. Figure 2A Illustrated is an example of a conversational session in which an automated assistant provides content to a user having content parameters selected from a plurality of candidate content parameters. Figures 2B1-2B4 Each illustrates different examples of providing feedback prompts to a user in a separate conversation session, where the feedback prompts are based on selected content parameters.

[0068] Figure 2A According to embodiments described herein, a computing device 210 including one or more microphones and one or more speakers is illustrated, and an example of a conversational session that can occur between a user 101 of the computing device 210 and an automated assistant 120 via the microphone and speakers is illustrated. One or more aspects of the automated assistant 120 can be implemented on the computing device 210 and / or on one or more computing devices in network communication with the computing device 210.

[0069] exist Figure 2AIn the example embodiment, a user provides natural language input 280A of "Can you order dinner at 6 PM" to initiate a conversation session between the user and automated assistant 120. In response to natural language input 280A, automated assistant 120 provides natural language output 282A of "Sure, what kind of food." The user then provides natural language input 280B indicating that the user likes Mexican food. Automated assistant 120 then provides natural language output 282B asking the user whether the user would like the automated assistant to select a particular restaurant, to which the user responds affirmatively using natural language input 280C.

[0070] Automated assistant 120 then provides natural language output 282C as a suggestion. The suggestion in natural language output 282C includes a specific restaurant (Café Lupe) selected from a plurality of candidate "Mexican cuisine" restaurants, and also includes a specific dish ("burrito") selected from a plurality of candidate dishes available at the specific restaurant.

[0071] The user then provides natural language input 280D, which directs automated assistant 120 to order a burrito and some fries from Café Lupe. Automated assistant 120 then provides natural language output 282D to confirm that the user request of input 280D has been fulfilled by automated assistant 120 (optionally via one or more additional external components).

[0072] exist Figure 2A In a conversation session, automated assistant 120 selects the location "Café Lupe" and the dish type "Tacos" from a plurality of candidate options and presents these selections to the user as part of the conversation session. Furthermore, automated assistant 120 may determine that the user acted on the suggestion because the user had previously done so in the same conversation session (by ordering tacos from Café Lupe). Based on the recommended location and dish type, and optionally based on determining that the user acted on the suggested location and dish type, automated assistant 120 may store content parameters indicating the suggested location and dish type. For example, automated assistant 120 may store an indication of those selected and provided content parameters in association with the user in provided content parameter database 154.

[0073] Figure 2B1 Provided based on the response Figure 2A An example of providing feedback prompts to users using the "CaféLupe" content parameters stored in the conversation session. Figure 2B1, the user provides natural language input 280A1 of “Tell me about my day” to initiate a conversation session between the user and the automated assistant 120 . Figure 2B1 Conversational conversation with Figure 2A For example, Figure 2B1 The conversation session can be based on Figure 2A The conversation sessions are determined to be separate conversation sessions based on the elapse of at least a threshold amount of time and / or based on other criteria.

[0074] The automated assistant 120 responds to the natural language input 280A1 with a responsive natural language output 282A1 that includes a summary of the user's calendar along with local forecasts and traffic reports. Figure 2A The content of the conversation session is irrelevant, but the automated assistant 120 then provides a feedback prompt 282B1 that solicits feedback on the content parameters of "Café Lupe". As described herein, the automated assistant 120 can provide feedback based on, for example, Figure 3B Conversational conversation with Figure 3A Feedback prompt 282B1 is provided based on individual criteria for separating the conversation session, where input 380A1 and / or output 382A1 satisfies one or more criteria, such as being considered "routine" and / or other criteria.

[0075] The user responds to feedback prompt 282B1 with positive natural language input 280B1, and the automated assistant responds with affirmative automated language output 282C1. Automated assistant 120 can utilize positive natural language input 280B1 to positively influence the value associated with the "Café Lupe" content parameter. For example, automated assistant 120 can adjust the value to increase the likelihood that "Café Lupe" and / or restaurants similar to Café Lupe will be offered in future conversation sessions with the user and / or in future conversation sessions with other users.

[0076] Figure 2B2 Provided based on the response Figure 2A An example of providing feedback prompts to users based on the "burrito" content parameter stored in the conversation session. Figure 2B2 A conversation session can be replaced by Figure 2B1 Conversational sessions or in addition Figure 2B1 In the conversation session outside the conversation session. Figure 2B2 , the user provides natural language input 280A2 of “Good morning” to initiate a conversation session between the user and automated assistant 120. Figure 2B2 Conversational conversation with Figure 2A For example, Figure 2B2The conversation session can be based on Figure 2A The conversation sessions are determined to be separate conversation sessions based on the elapse of at least a threshold amount of time and / or based on other criteria.

[0077] Automated assistant 120 responds to natural language input 280A2 with response natural language output 282A2 including the response content "Good morning John". Figure 2A Regardless of the content of the conversation session, natural language output 282A2 also includes a feedback prompt: "What did you think about the burrito from Café Lupe last night?" As described herein, automated assistant 120 can provide the feedback prompt included in output 282A2 based on various criteria.

[0078] The user responds to output 282A2 with positive natural language input 280B2. The automated assistant responds with natural language output 282B2 that includes the confirmatory language "great" and a suggestion for another popular dish at Café Lupe. Automated assistant 120 can use positive natural language input 280B2 to positively influence the value associated with the "taco" content parameter. For example, automated assistant 120 can adjust the value to increase the likelihood of recommending "taco" as a dish at Café Lupe and / or at other restaurants in future conversation sessions with the user and / or with other users.

[0079] Figure 2B3 Provided based on the response Figure 2A Another example of providing feedback prompts to the user based on the "burrito" content parameter stored in the conversation session. Figure 2B3 A conversation session can be replaced by Figure 2B1 and / or Figure 2B2 Conversational sessions or in addition Figure 2B1 and / or Figure 2B2 In the conversation session outside the conversation session. Figure 2B3 , the user provides natural language input 280A3 of “Play me some music” to initiate a conversation session between the user and the automated assistant 120. Figure 2B2 Conversational conversation with Figure 2A Separation of conversation sessions.

[0080] Automated assistant 120 responds to natural language input 280A3 with response output 282A3 that includes a response song (indicated by musical notes). Figure 2A Regardless of the content of the conversational session, output 282A3 also includes a feedback prompt: "By the way, what did you think about the burrito from Café Lupe last night?" This feedback prompt can be provided after playing all or part of the response song. As described herein, automated assistant 120 can provide the feedback prompt included in output 282A3 based on various criteria.

[0081] The user responds to output 282A3 with positive natural language input 280B3. The automated assistant responds with output 282B3 including the confirmatory language "good to hear," and then continues playing the response song or additional songs. Automated assistant 120 can utilize positive natural language input 280B3 to positively influence the value associated with the "burrito" content parameter.

[0082] Figure 2B4 Provided based on the response Figure 2A Another example of providing feedback prompts to users based on the "Café Lupe" content parameters stored in the conversation session. Figure 2B4 A conversation session can be replaced by Figure 2B1 、 Figure 2B2 and / or Figure 2B3 Conversational sessions or in addition Figure 2B1 、 Figure 2B2 and / or Figure 2B3 A conversation session that occurs outside of a conversation session.

[0083] Figure 2B4 The diagram shows Figure 2A101 and a display screen 340 of client device 310. Client device 310 may include and / or communicate with automated assistant 120 and / or another instance thereof (which may access user 101's entry in content parameter database 154). Display screen 340 includes a reply interface element 388, which the user may select to generate user interface input via a virtual keyboard, and a voice reply interface element 389, which the user may select to generate user interface input via a microphone. In some embodiments, the user may generate user interface input via the microphone without selecting voice reply interface element 389. For example, during a conversational session, active monitoring of audible user interface input via the microphone may occur to obviate the need for the user to select voice reply interface element 389. In some of these and / or other embodiments, voice reply interface element 389 may be omitted. Furthermore, in some embodiments, reply interface element 388 may additionally and / or alternatively be omitted (e.g., the user may only provide audible user interface input). Display screen 340 also includes system interface elements 381 , 382 , 383 , which can interact with a user to cause client device 310 to perform one or more actions.

[0084] exist Figure 2B4 In FIG. 2 , the user provides natural language input 280A4 of “What’s my day look like tomorrow” to initiate a dialog session between the user and the automated assistant 120 . Figure 2B4 Conversational conversation with Figure 2A In some embodiments, the automated assistant 120 can be based on Figure 2B4 The conversation session occurs via a separate client device to determine that it is a separate conversation session.

[0085] Automated assistant 120 responds to natural language input 280A4 with response output 282A4 that includes a summary of the user's calendar. Figure 2AThe content of the conversation session is irrelevant, but the automated assistant then provides 282B4 a feedback prompt, “By the way, did you like Café Lupe? (By the way, did you like Café Lupe)?” The user responds to the feedback prompt 282B4 with positive natural language input 280B4. In some embodiments, the natural language input 280B4 can be a free-form input. In some other embodiments, the automated assistant 120 can present multiple options in the conversation for the user to select. For example, the automated assistant 120 can provide an interface including multiple options such as “Yes,” “No,” and “It was OK” in combination with the feedback prompt 282B4—and the user can select an option to provide a corresponding response input.

[0086] Figures 2B1-2B4 An example is provided in which a user provides positive feedback and thereby increases a corresponding content parameter. However, it should be understood that a user may alternatively provide negative feedback that decreases a corresponding content parameter.

[0087] Now refer to Figure 3A and Figure 3B , which describes additional examples of various embodiments disclosed herein. Figure 3A An example of a conversation session in which an automated assistant provides content to a user having content parameters selected from a plurality of candidate content parameters is illustrated. Figure 3B Illustrated is an example of providing feedback prompts to a user in a separate conversation session, where the feedback prompts are based on selected content parameters.

[0088] Figure 3A The diagram shows Figure 2B4 The same client device 310 is shown. Figure 3A In FIG. 4 , a dialog session exists between a user (“You”), an additional user (“Tom”), and an automated assistant 120 (“Automated Assistant”). The user provides natural language input 380A1 directed to the additional user, “Coffee in the morning?” The additional user provides a response natural language input 381A1 of “Sure.”

[0089] The user then invokes automated assistant 120 into the conversational session by including “@AutomatedAssistant” in input 380A2 and requests automated assistant 120 to “Pick a good coffee shop.” In response, automated assistant 120 provides output 382A1, which includes a recommendation for “Hypothetical Roasters,” which is reputable and close to the user and the additional user. Output 382A1 may be provided for presentation to the user (via client device 310) and the additional user (via their respective client devices). Automated assistant 120 selects the location “Hypothetical Roasters” from among multiple candidate options. Furthermore, automated assistant 120 may determine that the user acted on the recommendation based on further signals associated with the user (e.g., the user issued a navigation request to “Hypothetical Roasters,” location data indicating the user visited “Hypothetical Roasters,” and / or other signals). Based on recommending the location, and optionally based on determining that the user acted on the suggested location, automated assistant 120 may store content parameters in association with the user indicating the suggested location. For example, automated assistant 120 may store, in association with the user, an indication of those selected and provided content parameters in provided content parameter database 154. Automated assistant 120 may additionally or alternatively store, in association with additional users, content parameters indicating suggested locations (optionally after determining that the additional users acted upon the suggested locations).

[0090] Figure 3B The diagram shows Figure 3A The conversation session separated by the conversation session provides the user 101 with a Figure 3A Example of a feedback prompt for the selected content parameter. Figure 3B In FIG. 3 , the user 101 provides a natural language input 380B1 of “How's the commute home?” to the separate computing device 210 . Figure 3B Conversational conversation with Figure 3A For example, Figure 3B The conversation sessions can be determined to be separate conversation sessions based on that they occur via separate computing devices 210.

[0091] Automated assistant 120 responds to natural language input 380B1 with a responsive natural language output 382B1 that includes a summary of current traffic conditions. Figure 3AThe content of the conversation session is irrelevant, but the automated assistant 120 then provides a feedback prompt 382B2 that solicits feedback on the content parameters of "HypotheticalRoasters". As described herein, the automated assistant 120 can provide feedback based on, for example, Figure 3B Conversational conversation with Figure 3A The dialogue session is separated into individual criteria to provide feedback prompt 382B2, where input 380B1 and / or output 382B1 satisfies one or more criteria, such as being considered "routine" and / or other criteria.

[0092] The user responds to output 382B1 with negative natural language input 380B2. The automated assistant provides a further feedback prompt 382B3, which solicits feedback about whether the user particularly dislikes anything about "Hypothetical Roasters." The user responds to output 382B3 with further natural language input 380B3 specifying that "Hypothetical Roasters" is too crowded. Automated assistant 120 can utilize negative natural language input 380B2 to negatively influence the value associated with the "Hypothetical Roasters" content parameter. Automated assistant 120 can additionally or alternatively utilize natural language input 380B3 to influence the value associated with a content parameter indicating a level of crowding. For example, automated assistant 120 can adjust the value to reduce the likelihood of recommending a restaurant associated with "heavy crowds" one or more times in future conversation sessions with the user and / or in future conversation sessions with other users.

[0093] Figure 3B The example of a user 101 of a client device 310 is shown. However, based on the provided content parameter database 154 and the Figure 3A , a separate instance of automated assistant 120 can additionally and / or alternatively provide feedback prompts to an additional user ("Tom") via one of the additional user's client devices.

[0094] Although the examples provided in the figures focus on location suggestions, the techniques described herein can be implemented for other types of suggestions and / or other content. As one example, in response to a user input of “play me some bluegrass,” the automated assistant 120 can select Bluegrass Album A to play in response. The automated assistant 120 can then provide a prompt in a future conversation session, such as “How did you like Bluegrass Album A?” As another example, in response to a user input of “current news,” the automated assistant 120 can select current news from Source A to provide in response. The automated assistant 120 can then provide a prompt in a future conversation, such as “Did you like Source A for the news or would you prefer another source?” As another example, in response to a user input of “current news,” the automated assistant 120 can select five news stories to provide in response. The automated assistant 120 can then provide a prompt in a future conversation session, such as, "Did you like the number of news stories provided earlier, or would you prefer more or fewer?" As another example, in response to the user input "navigate to a coffee shop," the automated assistant 120 can select the nearest coffee shop and provide directions to the coffee shop. The automated assistant 120 can then provide a prompt in the conversation session the next day, such as, "I gave you directions to the closest coffee shop yesterday. In the future would you prefer a more highly rated coffee shop that is a bit farther away?"

[0095] Figure 44 is a flowchart illustrating an example method 400 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of automated assistant 120. Furthermore, when the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0096] At block 452 , the system receives natural language input based on user interface input provided by a user during a conversation session.

[0097] At block 454 , the system generates content that is responsive to the natural language input and includes at least one content parameter selected from a plurality of candidate content parameters.

[0098] At block 456 , the system provides content for presentation to the user as part of the conversation session and stores the selected content parameters for the content.

[0099] At block 458, the system identifies additional conversation sessions that include the user. In some implementations, block 458 may include block 460, where the system generates and provides additional content for presentation to the user as part of the additional conversation sessions.

[0100] At block 462 , the system provides a prompt soliciting feedback on the stored selected content parameters for presentation to the user as part of an additional conversation session.

[0101] At block 464 , the system receives additional input in response to the prompt and based on user interface input provided by the user during the additional conversation session.

[0102] At block 466 , the system uses the additional input to affect the value stored in association with the selected content parameter.

[0103] Figure 5 is a block diagram of an example computing device 510 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, automated assistant 120, and / or other components can include one or more components of the example computing device 510.

[0104] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, which includes, for example, a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0105] The user interface input devices 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or a communication network.

[0106] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as provided via an audio output device. The term "output device" is generally used to include all possible types of devices and methods for outputting information from the computing device 510 to a user or another machine or computing device.

[0107] The storage subsystem 524 stores the programming and configuration that provides the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include a Figure 4 The logic of the selected aspects of the method.

[0108] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions. The file storage subsystem 526 may provide persistent storage for program and data files and may include a hard drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other computers accessible by the processor 514.

[0109] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0110] The computing device 510 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The depiction of the computing device 510 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of the computing device 510 are possible with more Figure 5 The computing devices shown may have greater or fewer components.

[0111] In situations where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and the user's activities and demographic information), the user is provided with one or more opportunities to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how the information about the user is collected, stored, and used. That is, the systems and methods discussed herein may only collect, store, and / or use a user's personal information upon explicit authorization from the relevant user. For example, the user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user whose personal information is to be collected is presented with one or more options allowing control over the collection of information related to that user, providing permission or authorization regarding whether that information is collected and which portions of that information are to be collected. For example, one or more such control options may be provided to the user via a communication network. Furthermore, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. As an example, the user's identity may be processed so that personally identifiable information cannot be determined. As another example, the user's geographic location may be generalized to a larger area such that the user's specific location cannot be determined.

[0112] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing the functions and / or achieving the results and / or one or more advantages described herein may be utilized, and each of such variations and / or modifications is considered within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications in which the teachings are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are presented by way of example only, and that within the scope of the appended claims and their equivalents, embodiments may be practiced otherwise than as specifically described and claimed. Embodiments of the present disclosure are directed to each individual feature, system, article, material, tool, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, tools, and / or methods is included within the scope of the present disclosure, provided such features, systems, articles, materials, tools, and / or methods are not mutually inconsistent.

Claims

1. A method implemented by one or more processors, the method comprising: receiving spoken input from the user during a previous dialog session between the user and the automated assistant; processing the spoken input to select a suggestion from a plurality of candidate suggestions responsive to the spoken input; causing the certain suggestion to be presented to the user audibly and / or graphically during the previous conversation session; After presenting the suggestion to the user, verifying that the user actually acts according to the suggestion; determining that a current conversation session satisfies one or more criteria, the current conversation session being temporally separate from the previous conversation session and unrelated to the suggestion of the previous conversation session, and the current conversation session being via a client device and between the user and the automated assistant; In response to determining that the current conversation session satisfies the one or more criteria, and in response to determining that the user actually acted on the suggestion: causing a prompt soliciting feedback regarding the suggestion to be presented to the user at an output component of the client device; receiving a user interface input in response to the prompt; using the user interface input to change a value stored in association with the suggestion; as well as After changing the value, the changed value is used when selecting further suggestions to be provided to the user in a further dialog session between the user and the automated assistant.

2. The method according to claim 1, wherein The user interface input is further a spoken utterance and also includes: Based on providing the prompt, at least one microphone of the client device is turned on in advance.

3. The method according to claim 1, wherein The user interface input is further a spoken utterance and also includes: Based on providing the prompt, a local speech-to-text processor of the client device is activated.

4. The method according to claim 1, wherein The current conversation session includes speech-based input from the user, and wherein determining that the current conversation session satisfies the one or more criteria includes: It is determined that the speech-based input conforms to the user's speech profile.

5. The method according to claim 1, wherein The current conversation session includes speech-based input from the user, and wherein determining that the current conversation session satisfies the one or more criteria includes: A determination is made that the speech-based input lacks certain background noise.

6. The method according to claim 5, wherein: The certain background noise indicates that other users may be present.

7. The method according to claim 1, wherein Determining that the current dialog session satisfies the one or more criteria includes: It is determined that content provided during the current conversation session includes private content, the private content being private to the user.

8. The method according to claim 1, wherein The previous conversation session was via an additional client device, wherein the current conversation session includes speech-based input from the user, and wherein determining that the current conversation session satisfies the one or more criteria comprises: It is determined that the speech-based input conforms to the user's speech profile.

9. The method according to claim 1, wherein: After presenting the suggestion to the user, verifying that the user actually acted on the suggestion is based on one or more signals that were not generated using the automated assistant.

10. The method according to claim 1, wherein After presenting the suggestion to the user, verifying that the user actually acts on the suggestion is based on location data associated with the suggestion or a navigation request associated with the suggestion.

11. A client device comprising: at least one microphone; at least one speaker; Network interface; One or more processors configured to: As part of an additional conversation session between a user and an automated assistant, the additional conversation session being temporally separate from a previous conversation session between the user and the automated assistant and initiated by the user using speech-based natural language input: Based on the speech-based natural language input matching the user's speech profile: providing a prompt via the speaker, the prompt soliciting feedback on a suggestion previously provided to the user for presentation as part of the previous conversation session, wherein the prompt solicits feedback regarding the suggestion based on the suggestion being previously provided to the user for presentation as part of the previous conversation session and based on a determination that the user acted on the suggestion; and Based on providing the prompt, a communication session is pre-established between the client device and a remote speech-to-text processor to process user interface input provided via the microphone.

12. The client device according to claim 11, wherein The one or more processors are further configured to: receiving additional input via the microphone in response to the prompt; and Data based on the additional input is provided via the web interface, the data being provided with an influence value that is stored in association with the suggestion and influences future provision of further content including the suggestion.

13. The client device according to claim 11, wherein The one or more processors are configured to provide the prompt further based on the speech-based natural language input lacking certain background noise indicating a possible presence of other users.

14. The client device according to claim 11, wherein The one or more processors are further configured to render a graphical user interface based on providing the prompt, the graphical user interface including one or more selectable elements that are selectable to provide feedback.

15. A system comprising a memory and one or more processors, wherein: the memory stores instructions; and The one or more processors are capable of executing the instructions so that the one or more processors perform the following operations: receiving spoken input from the user during a previous dialog session between the user and the automated assistant; processing the spoken input to select a suggestion from a plurality of candidate suggestions responsive to the spoken input; causing the certain suggestion to be presented to the user audibly and / or graphically during the previous conversation session; After presenting the suggestion to the user, verifying that the user actually acts according to the suggestion; determining that a current conversation session satisfies one or more criteria, the current conversation session being temporally separate from the previous conversation session and unrelated to the suggestion of the previous conversation session, and the current conversation session being via a client device and between the user and the automated assistant; In response to determining that the current conversation session satisfies the one or more criteria, and in response to determining that the user actually acted on the suggestion: causing a prompt soliciting feedback regarding the suggestion to be presented to the user at an output component of the client device; receiving a user interface input in response to the prompt; using the user interface input to change a value stored in association with the suggestion; as well as After changing said value: selecting a further suggestion in a further dialog session between the user and the automated assistant, wherein upon selecting the further suggestion, one or more of the processors selecting the further suggestion based on the changed value; as well as The further suggestion is caused to be presented to the user at the client device or an additional client device of the user.

16. The system according to claim 15, wherein: The user interface input is a further spoken utterance, and wherein, upon executing the instructions, one or more of the processors: Based on providing the prompt, at least one microphone of the client device is turned on in advance.

17. The system according to claim 15, wherein: The current conversation session includes speech-based input from the user, and wherein, upon determining that the current conversation session satisfies the one or more criteria, one or more of the processors perform the following operations: It is determined that the speech-based input conforms to the user's speech profile.

18. The system according to claim 15, wherein: Upon determining that the current dialog session satisfies the one or more criteria, one or more of the processors may perform the following operations: It is determined that content provided during the current conversation session includes private content, the private content being private to the user.

Images