Use user input to adapt search results for presentation to the user

Abstract

The present application relates to using user input to adapt search results provided for presentation to a user. Methods, apparatus, and computer-readable media relate to interactions between a user and an automated assistant during a conversation between the user and the automated assistant. Some embodiments relate to adapting a graphical and / or audible presentation of search results provided by the automated assistant. The adaptation is responsive to attributes in the search results referenced in input provided by the user during the conversation. Some of those embodiments enable a user to provide text input to navigate search results within a conversation and within resource and / or interface constraints associated with the conversation, and / or enable adaptation of when and / or whether the automated assistant provides search results with certain attributes based on text input provided by the user to the automated assistant. Example embodiments use an improved interaction protocol for an automated assistant to reduce computing resources used during an interaction between the automated assistant and its user.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201710763373.8, filed on August 30, 2017. Technical Field

[0003] The present application relates to using user input to adapt search results provided for presentation to a user. Background Art

[0004] An automated assistant (also referred to as a "personal assistant module," "mobile assistant," or "chatbot") interacts with a user via various computing devices, such as smartphones, tablet computers, automotive systems, standalone personal assistant devices, etc. The automated assistant receives text input (e.g., typed and / or spoken) from the user and responds with responsive output (e.g., visual and / or audible).

[0005] Some user interactions with the automated assistant may only request a single response from the automated assistant. For example, the text inputs "what time is it in London", "what is the Capital of Delaware", and "how many ounces are in a cup" may request a single response from the automated assistant.

[0006] In contrast, other user interactions with the automated assistant would be more common and request that a response group be provided by the automated assistant. For example, "news headlines", "nearby restaurants", and "search results for mustang" would request that the automated assistant issue a search of one or more corpora and return a search result group responsive to the search.

[0007] However, resource and / or interface constraints associated with existing automated assistants present one or more disadvantages in providing search result groups for presentation to a user via an automated assistant. For example, some automated assistants may be implemented via a "chat" type graphical user interface. Displaying a large number of search results simultaneously in such an interface may clutter the interface, make it more difficult to follow the conversation, and / or may use a considerable amount of computing resources. For example, computing resources may be used due to the simultaneous presentation of a large number of results and / or due to the "scrolling" and / or other actions required to view a large number of results. In addition, making the search results viewable in an interface and / or application independent of the automated assistant interface and / or application may consume more user and / or computing resources. For example, switching to other interfaces and / or applications may distract the user from an ongoing conversation with the automated assistant and / or require the computing device to execute a separate application and / or present a separate interface.

[0008] As another example, some automated assistants may include (or be limited to) providing audible user interface output and accepting spoken user interface input. When search result groups are provided by such automated assistants, directional navigation of the result groups for a user via spoken user interface input is impossible and / or limited. In addition, many automated assistants lack the ability to adapt when and / or whether search results with certain attributes are provided to the user by the automated assistant based on text input provided to the automated assistant by the user. Additional and / or alternative disadvantages of these and / or other technologies may occur. Summary of the invention

[0009] The present specification is directed to methods, apparatus, and computer-readable media relating to interactions between a user and an automated assistant during a conversation between at least the user and the automated assistant. Some embodiments are directed to adapting a graphical and / or audible presentation of search results provided by the automated assistant for presentation to the user. The search results may be adapted in response to attributes of one or more search results referenced in spoken and / or typed text input provided by the user during the conversation. Some of those embodiments may enable the user to provide text input to navigate the search results within the conversation and within resource and / or interface constraints associated with the conversation. Some of these embodiments may additionally or alternatively implement adaptation of when and / or whether search results having certain attributes are provided to the user by the automated assistant based on text input provided by the user to the automated assistant.

[0010] Some of these and other embodiments of the specification can achieve various technical advantages. Some example embodiments use an improved interaction protocol for an automated assistant to reduce the computing resources used during the interaction between the automated assistant and its user. For example, some embodiments that adapt when and / or whether to provide search results to a user make it possible to provide fewer search results to the user in some cases, but still meet the user's information needs. This can reduce the use of various computing resources, such as resources of a computing device required to visually and / or audibly present search results to the user. At the same time, for example, some embodiments that navigate search results within a conversation can allow the user to freely navigate forward and / or backward search results (optionally, non-sequentially) without having to present all search results at the same time and / or present all of one or more search results. This can make it possible to provide search results to be presented to the user and navigated by the user, while also making it possible to meet required and / or necessary resource and / or interface constraints, and / or during the presentation of search results, provide less than all of one or more search results to be provided.

[0011] As one example, during a conversation between a user and an automated assistant, the user may cause text input to be provided to the automated assistant. The text input may initiate a conversation or may be a continuation of a previously initiated conversation. The text input may be a natural language free-form input, such as text input based on a user interface input generated by a user via one or more user interface input devices (e.g., based on typed input provided via a physical or virtual keyboard or based on spoken input provided via a microphone). As described herein, a free-form input is an input formulated by a user and is not limited to a group of options presented for selection by the user (e.g., not limited to a group of options presented in a drop-down menu).

[0012] In response to some text input provided to the automated assistant as part of a conversation, the automated assistant may obtain a plurality of responsive search results. For example, the automated assistant may cause a search of one or more databases to be issued (or may search the databases itself) based on the text input, and may obtain a plurality of search results in response to the search. For example, in response to the text input of "news headlines", a database of news story documents may be searched, and the plurality of search results obtained may be based on respective documents in a plurality of recent news story documents identified in response to the search. The obtained search results may be selected and / or ranked based on various signals, such as popularity of the search results, a match between the search parameters and the search results, attributes of the user, and the like.

[0013] Meanwhile, in some embodiments, the obtained search results may be additionally and alternatively selected and / or ranked based on past text input provided by the user during the conversation with the automated assistant. For example, if: during the conversation with the automated assistant, search results from source 1 were previously presented to the user, search results from "Source 1" (e.g., a specific resource name of a publisher and / or author's URL or other indicator) are excluded from the search results, and a response text input "no more from this source" is provided to the user. As another example, if: during the conversation with the automated assistant, search results from source 2 were previously presented to the user, search results from "Source 2" are advanced in the ranking of the search results, and a response text input "I like this source" is provided to the user.

[0014] Regardless of the technique for obtaining and / or ranking search results, the automated assistant may sequentially provide groups of search results for presentation (visually and / or audibly) to the user as part of a conversation. The order in which the search results are provided for presentation may be based on the ranking. For example, a first group including only the highest ranked search results may be provided for presentation, then a second group including only the second highest ranked search results may be provided for presentation, and so on. In some embodiments, the second group may be automatically provided after the first group (e.g., immediately thereafter or after a time delay). In some embodiments, a user interface input is required before providing the second group. For example, during or after providing the first group, the second group may only be provided after the user provides a specific spoken user interface input, such as "next", "another", "continue", etc.

[0015] In some embodiments, during the provision of search results, a user may provide further text input that "breaks" the order in which the sequential provision is made and causes one or more "out of order" search results to be provided. For example, a user may say or type "previous" to return to the immediately previous search result; "second result" to return to the second presented search result, "back to the one from Source A" to return to the search result from "Source A"; "back to the one about Topic A" to return to the search result about "Topic A"; "more from Source A" to go forward to another search result from "Source A" (even if it is not the next one in the sequential order); "more about Topic A" to go forward to another search result about "Topic A" (even if it is not the next one in the sequential order); and / or "more like this" to go forward to an additional search result that is similar to at least a portion of the most recently presented search result (even if the additional search result is not the next one in the sequential order). These and other implementations are described in additional detail herein.

[0016] In some embodiments, a method implemented by one or more processors is provided, comprising: receiving an input and obtaining a plurality of search results in response to the input. The input is based on a user interface input generated by a user via a user interface input device, and the user interface input is generated by the user as part of a conversation between the user and an automated assistant implemented at least in part by one or more processors. The method further comprises sequentially providing search result groups for presentation to the user via a user interface output device. Each search result group comprises at least one search result, and sequentially providing search result groups comprises providing each group according to an order of the groups. The method further comprises, during the provision, receiving further input. The further input is based on further user interface input generated by the user via the user interface input device or another user interface input device. The method further comprises, based on one or more terms of the further input and based on at least one attribute of a previously presented search result, determining that the further input corresponds to the previously presented search result. The previously presented search result is a search result previously provided during the sequential provision. The method further comprises, in response to determining that the further input corresponds to the previously presented search result, providing output related to the previously presented search result. The output is provided in the reverse order of the groups.

[0017] These and other implementations of the technology disclosed herein can optionally include one or more of the following features.

[0018] In some embodiments, sequentially providing search result groups for presentation to a user includes providing an initial group of the groups in response to an input; and providing a second group following the initial group. The second group follows the initial group in a sequential order, providing the second group in response to receiving an additional user interface input, and the additional user interface input precedes the further user interface input. In some of those embodiments, the additional user interface input is a verbal user interface input.

[0019] In some embodiments, the user interface input device is a microphone, the user interface output device is a speaker, and the further user interface input is generated by the user via the microphone. In some of those embodiments, at least a portion of the further user interface input is received during audible presentation of a search result via the speaker.

[0020] In some embodiments, the further user interface input is a verbal user interface input, and the method further comprises, during the providing, actively monitoring for the further input. In some of those embodiments, receiving the further input occurs during the active monitoring.

[0021] In some embodiments, the user interface output device is a display, the output includes the previously presented search results, and providing the output includes: causing the output to be presented in a graphical user interface presented on the display separately from any previous presentation of continued output from the previously presented search results.

[0022] In some implementations, the output is an additional search result associated with a base content item that is related to the previously presented search result but is different from the previously presented search result.

[0023] In some embodiments, the user interface output device is a display, and sequentially providing the groups for presentation to the user via the display includes causing each group to replace a corresponding immediately preceding one of the groups in a graphical user interface presented on the display. In some of those embodiments, providing output related to the search results includes causing the output to replace a most recently provided one of the groups provided during the sequential presentation in the graphical user interface.

[0024] In some embodiments, based on the one or more terms of the further input and based on an attribute of the previously presented search result, determining that the further input corresponds to the previously presented search result includes: identifying an attribute of the previously presented search result; and determining that the one or more terms match the attribute. In some of those embodiments, the attribute is one of: a name of a source of the previously presented search result, a name of an entity included in the previously presented search result, and a reference to an order of presentation of the previously presented search results in a sequential presentation of the groups.

[0025] In some embodiments, the method further includes determining, based on one or more terms of the further input, that the further input also corresponds to an additional previously presented search result that was previously provided in the search results during the sequential provision. In some of those embodiments, determining that the further input corresponds to the previously presented search result further includes selecting the previously presented search result in place of the additional previously presented search result based on one or more additional criteria. In some of those embodiments, determining that the further input corresponds to the previously presented search result includes: generating a prompt based on additional attributes of the additional previously presented search result, providing the prompt for presentation to a user via a user interface output device; receiving additional user interface input in response to the prompt, and selecting the previously presented search result in place of the additional previously presented search result based on the additional user interface input.

[0026] In some embodiments, the input and the further text input each comprise a text input, such as a text input generated based on a spoken user interface input.

[0027] In some embodiments, a method implemented by one or more processors is provided, including: as part of a conversation between a user and an automated assistant at least partially implemented by the one or more processors: providing search results for presentation to the user, and in response to providing the search results, receiving text input referencing an attribute of the search result and the user's sentiment toward the attribute; based on the attribute of the search result and the user's sentiment toward the attribute, determining, in response to further conversation between the user and the automated assistant, a parameter that affects whether or when one or more attribute search results having the attribute are provided by the automated assistant; and as part of a further conversation between the user and the automated assistant: using the parameter by the automated assistant to affect whether or when at least one of the attribute search results is provided for presentation to the user.

[0028] These and other embodiments of the technology disclosed herein may optionally include one or more of the following features. In some embodiments, the parameter: affects when the attribute search result is provided as part of a further conversation by affecting the ranking of the attribute search result, or affects whether the attribute search result is provided as part of a further conversation by preventing the attribute search result from being provided. In some versions of those embodiments, the parameter affects the ranking of the attribute search result to a degree based on the user's sentiment toward the attribute of the attribute search result. In some other versions of those embodiments, the parameter prevents the attribute search result from being provided based on the user's sentiment toward the attribute as expressed in the text input by one or more predetermined terms.

[0029] In some embodiments, the attribute of the search result is the source of the search result, the text input includes terms referencing the source and additional terms referencing the sentiment, and the parameter affects the ranking of the attribute search results from the source, or prevents the attribute search results from being provided for presentation to the user.

[0030] In some embodiments, the reference to the attribute in the text input is a reference to the search result rather than an explicit reference to the attribute. In some of those embodiments, the method further includes identifying the attribute based on the attribute assigned to the search result in one or more computer-readable media. In some versions of those embodiments, the attribute is one of: a source of the search result, an entity referenced in the search result, and a document type of the search result.

[0031] In some embodiments, the search results are provided for presentation to the user in response to a first text input by the user, the further conversation is a continuation of the conversation and is a continuation of the sequential presentation of the search results in response to the first text input, and at least one of the attribute search results is in response to the first text input.

[0032] In some embodiments, the search results are provided for presentation to the user in response to a first text input by the user, the further dialog includes providing additional search results to the user in response to a new search issued for additional text input by the user, and at least one of the attribute search results is in response to the additional text input.

[0033] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are executable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the above methods to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the above methods.

[0034] It should be recognized that all combinations of the above concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0036] Figure 2 The diagram illustrates an example of adapting search results provided by a user to an automated assistant during a conversation, wherein the search results are adapted in response to attributes referenced in one or more of the search results in spoken and / or typed text input provided by the user during the conversation.

[0037] Figure 3 An example client computing device having a display screen showing an example of a conversation occurring between a user of the client computing device and an automated assistant according to embodiments disclosed herein

[0038] Figure 4A , 4B 4C and 4D illustrate a portion of a display screen having another example of a conversation occurring between a user of a client computing device and an automated assistant according to an embodiment disclosed herein. Figure 3 An example client computing device.

[0039] Figure 4E FIG. 1 illustrates a display screen having another example of a conversation occurring between a user of a client computing device and an automated assistant according to embodiments disclosed herein. Figure 3 An example client computing device.

[0040] Figure 5 Another example client computing device is illustrated, along with an example of an audible conversation occurring between a user of the client computing device and an automated assistant, in accordance with implementations disclosed herein.

[0041] Figure 6 and 7FIG. 1 illustrates a display screen having another example of a conversation occurring between a user of a client computing device and an automated assistant according to embodiments disclosed herein. Figure 3 An example client computing device.

[0042] Figure 8 is a flow chart illustrating an example method according to an embodiment disclosed herein.

[0043] Fig. 9 is a flowchart illustrating another example method according to an embodiment disclosed herein.

[0044] Fig.10 The diagram illustrates an example architecture for a computing device. DETAILED DESCRIPTION

[0045] exist Figure 1 , an example environment in which the technology disclosed herein can be implemented is illustrated. The example environment includes one or more user interface input devices 102, one or more user interface output devices 104, and an automated assistant 120. The user interface input device 102 may include, for example, a physical keyboard, a touch screen (e.g., implementing a virtual keyboard or other text input mechanism), and / or a microphone. The user interface output device 104 may include, for example, a display screen and / or a speaker. The user interface input and output devices 102, 104 may be integrated on one or more computing devices of the user. For example, the user's mobile phone may include user interface input and output devices 102, 104, or an independent personal assistant hardware device may include user interface input and output devices 102, 104, or a first computing device may include a user interface input device 102 and a separate computing device may include a user interface output device 104, etc.

[0046] Although the automated assistant 120 Figure 1 Although illustrated as being independent of user interface input and output devices 102, 104, in some embodiments, all or aspects of automated assistant 120 may be implemented on a computing device that also includes user interface input device 102 and / or user interface output device 104. For example, all or aspects of output generation engine 130 and / or output adaptation engine 124 of automated assistant 120 may be implemented on a computing device. In some embodiments, all or aspects of automated assistant 120 may be implemented on a computing device that is separate and remote from the computing device that includes user interface input device 102 and / or user interface output device 104 (e.g., all or aspects may be implemented "in the cloud"). In some of these embodiments, those aspects of automated assistant 120 may communicate with the computing device via one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0047] Some non-limiting examples of client computing devices that may include user interface input device 102 and user interface output device 104 include one or more of the following: a desktop computing device, a laptop computing device, a standalone hardware device at least partially dedicated to the automated assistant, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), or a wearable device of a user including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided. In some embodiments, a given user may communicate with all or aspects of automated assistant 120 using multiple client computing devices that together form a collaborative "ecosystem" of computing devices. However, for the sake of brevity, some of the examples described in this disclosure will focus on users operating a single client computing device.

[0048] The client computing devices and automated assistant 120 may each include (when they are separate devices) one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. Operations performed by one or more client computing devices and / or by automated assistant 120 may be distributed across multiple computing devices. Automated assistant 120 may be implemented as a computer program running on one or more computers in one or more locations coupled to each other over a network, for example.

[0049] like Figure 1 As shown, the user provides input to the automated assistant 120 via the user interface input device 102. The automated assistant 120 provides a response output for presentation to the user via the user interface output device 104 (optionally after further processing by one or more components). For simplicity, the input is Figure 1 104. The automated assistant 120 is shown as being provided directly to the automated assistant 120 via the user interface input device 102, and the output is illustrated as being provided directly by the automated assistant 120 to the user interface output device 104. However, it should be noted that in various embodiments, one or more intermediate hardware components are functionally inserted between the automated assistant 120 and the user interface input and / or output devices 102, 104, and may optionally process the input and / or output. For example, one or more components may process the output provided by the automated assistant 120 and, based on the processing of the output, generate one or more signals for presenting search results and / or other content via the user interface output device 104. For example, in the case where the user interface output device 104 is on a client computing device that is separate from all or aspects of the automated assistant 120, the hardware processor and / or other components may process the output and, based on the output, generate signals for driving the user interface output device 104.

[0050] In some embodiments, the input received by the automated assistant 120 is already in text format. For example, the user interface input device 102 may include a keyboard that generates text input in response to user interface input to the keyboard, and the text input is provided to the automated assistant 120. Also, for example, the user interface input device 102 may include a microphone, a speech-to-text processor separate from the automated assistant 120 may convert the speech input received at the microphone into text input, and provide the text input to the automated assistant 120. In some other embodiments, the input initially received by the automated assistant 120 is not in text format, but is converted into text format by the automated assistant 120 and provided to one or more other components of the automated assistant 120 in text format. For example, the user interface input device 102 may include a microphone, speech input may be received at the microphone and provided to the automated assistant 120, and the speech-to-text processor of the automated assistant 120 may convert the speech input into text input.

[0051] The client computing device optionally operates one or more applications that implement the conversation with the automated assistant 120. These applications may come in various forms, such as a short message service ("SMS") and / or multimedia message service ("MMS") client, an online chat client (e.g., an instant messenger, Internet Relay Chat or "IRC"), etc., a messaging application associated with a social network, an automated assistant messaging service dedicated to conversations with the automated assistant 120, etc. In some implementations, one or more applications may be implemented via a web page or other resource presented by a web browser or other application of the client computing device.

[0052] In various embodiments, in response to certain textual input provided to the automated assistant 120 during a conversation with the automated assistant 120, the automated assistant 120 may obtain a plurality of responsive search results using search parameters based at least in part on the textual input. The obtained search results may be selected and / or ranked based on various signals, such as popularity of the search results, a match between the search parameters and the search results, attributes of the user, and the like. In some embodiments, the obtained search results are additionally or alternatively selected and / or ranked based on past textual input provided by the user during a conversation with the automated assistant. For example, if: during a conversation with the automated assistant 120, search results from source 1 were previously presented to the user, search results from "source 1" are excluded from the search results to be presented to the user, and the user is provided in response to the textual input "no more from this source".

[0053] The automated assistant 120 may sequentially provide the search result groups for presentation (visually and / or audibly) to the user as part of the conversation as output for providing to the user via the user interface output device 104. The automated assistant 120 may sequentially provide the search result groups in an order based on the ranking of the search results. In some embodiments, each of the provided groups may include only a single search result. In some other embodiments, one or more of the provided groups may include multiple search results (e.g., each group may include two search results).

[0054] In some implementations, during the provision of search results by automated assistant 120, the user may provide further text input that “breaks” the order of the sequence provided by automated assistant 120 and causes one or more “out of order” search results to be provided by automated assistant 120. For example, the user may say or type: “previous” to return to the immediately previous search result; “second result” to return to the two previous presented search results; “back to the one from Source A” to return to the search results from “Source A”; “back to the one about Topic A” to return to the search results about “Topic A”; “more from Source A” to move forward to another search result from “Source A” (even if it is not the next search result in the sequential order), and / or “more about Topic A” to move forward to another search result about “topic A” (even if it is not the next search result in the sequential order).

[0055] In various implementations, automated assistant 120 may include a message processing engine 122, an output adaptation engine 124, a search engine 126, a ranking engine 128, and an output generation engine 130. In some implementations, one or more of engines 122, 124, 126, 128, and / or 130 may be omitted, combined, and / or implemented in a component separate from automated assistant 120. For example, one or more of engines 122, 124, 126, 128, and / or 130, or any operable portion thereof, may be implemented in a component executed by a client computing device including user interface input and / or output devices 102 and 104 and separate from automated assistant 120. Also, search engine 126 and / or ranking engine 128 may be implemented in whole or in part by, for example, a system separate from automated assistant 120 (e.g., a separate search system in communication with automated assistant 120).

[0056] Message processing engine 122 processes text input submitted to automated assistant 120 and generates annotation output for use by one or more other components of automated assistant 120. For example, message processing engine 122 may process natural language free-form text input generated based on user interface input generated by a user via user interface input device 102. The generated annotation output includes one or more annotations of the text input and, optionally, one or more (e.g., all) terms of the text input.

[0057] In some embodiments, the message processing engine 122 is configured to recognize and annotate various types of grammatical information in the text input. For example, the message processing engine 122 may include a portion of a part-of-speech tagger that is configured to annotate terms by their grammatical roles. For example, the part-of-speech tagger may tag each term by its part of speech, such as "noun", "verb", "adjective", "pronoun", etc. At the same time, for example, in some embodiments, the message processing engine 122 may additionally and / or alternatively include a dependency analyzer that is configured to determine syntactic relationships between terms in the text input. For example, a dependency syntactic analyzer can determine which terms modify other terms, subjects, verbs, etc. of a sentence (e.g., a parse tree) - and make annotations of such dependencies.

[0058] In some embodiments, the message processing engine 122 may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to persons, organizations, locations, etc. The entity tagger may annotate references to entities at a high level of granularity (e.g., enabling identification of all references to an entity class such as a person) and / or at a lower level of granularity (e.g., enabling identification of all references to a specific entity such as a specific person). The entity tagger relies on the content of the natural language input to parse the specific entity and / or optionally communicates with a knowledge graph or other entity database to parse the specific entity.

[0059] In some implementations, the message processing engine 122 may additionally and / or alternatively include a coreference resolver configured to group or "cluster" references to the same entity based on one or more contextual clues. For example, a coreference resolver may be utilized to resolve the term "it" in the natural language input "I hate Source 1. No more results from it." into "Source 1".

[0060] In some embodiments, the message processing engine 122 can additionally and / or alternatively determine the emotion of one or more segments of the text input, and optionally, the emotion magnitude. The emotion of the segment can be determined based on the terms of the segment, other terms of the text input, and / or data other than the text input itself (e.g., voice characteristics included in the voice input on which the text input is based, previous text input, and / or other data). In some embodiments, the message processing engine 122 includes a trained emotion classifier that is trained to predict the classification / direction of the emotion of the segment and, optionally, the emotion magnitude based on the segments of the text input and / or other data. For example, the emotion classifier can predict whether the segment is positive or negative based on the terms of the segment and, optionally, based on other data, and, optionally, the magnitude of positive / negative. In some embodiments, the message processing engine 122 additionally and / or alternatively uses a mapping between terms and emotions (and, optionally, emotion magnitude) to determine the emotion of the segment. For example, a mapping may define that: segments including "never" have a strong magnitude of negative sentiment; "I amnot a fan of" has a smaller magnitude of negative sentiment, "always" has a strong magnitude of positive sentiment, "I like" has a smaller magnitude of positive sentiment, etc.

[0061] In some embodiments, one or more components in the message processing engine 122 rely on annotations from one or more components of the message processing engine 122. For example, in some embodiments, a designated entity annotator depends on annotations from a coreference resolver and / or a dependency analyzer in annotating all records of a particular entity. Also, for example, in some embodiments, the coreference resolver depends on annotations from a dependency analyzer in clustering references to the same entity. In some embodiments, when processing a particular text input, one or more components of the message processing engine 122 may use relevant previous inputs and / or other relevant data other than the particular text input to determine one or more annotations. For example, the first text input of a user in a conversation with the automated assistant 120 may be "Is that result from source 1?" and the user's subsequent text input may be "never provide results from them." In processing “never provide results from them”, using the previous input of “Is that result from source 1”, the coreference resolver can resolve “them” into “Source 1”.

[0062] The search engine 126 searches one or more search databases 154 in response to at least some text input submitted by the user as part of a conversation between the automated assistant 120 and the user. The search engine 126 searches the search databases 154 to identify content in response to the text input. In some embodiments, the search databases 154 include a database that indexes publicly available content and / or a database that indexes content dedicated to the user. The search engine 126 can use the databases 154 to identify responsive content and generate search results based on the identified responsive content. In some embodiments, one or more search databases 154 can be remote from the automated assistant 120 and / or any individual client computing device, and / or one or more search databases 154 can be local to the automated assistant 120 and / or any individual client computing device. In this specification, the term "database" is used to refer to any collection of structured or unstructured information stored in one or more computer-readable media.

[0063] The search engine 126 utilizes various techniques, such as conventional and / or other information retrieval techniques, when searching the databases 154 in response to the text input. In some embodiments, the search engine 126 may search one or more databases 154 based on search parameters that strictly match the text input. For example, for the text input "mustangs", the only search parameter is the term "mustangs". In some embodiments, the search engine 126 may search one or more databases 154 based on one or more search parameters that are based on but not necessarily strictly match the text input. For example, for the text input "local news", the search engine 126 may search one or more databases 154 based on search parameters that limit the databases 154 to "news" databases and / or content to "news" content, and based on search parameters that limit content to content local to the user. As another example, for the text input "restaurantsnearby", the search engine 126 can search one or more databases 154 based on search parameters that limit databases 154 to "points of interests" databases and content to "restaurant" content, and based on search parameters that limit content to content within a threshold distance of the user's current location. As yet another example, for the text input "my photos", the search engine 126 can search one or more databases 154 based on search parameters that limit databases 154 to the user's personal databases and / or content to "image" content.

[0064] The ranking engine 128 calculates a score for the content identified by the search engine 126 using one or more ranking signals, such as popularity of the content, a match between the search parameters and the content, attributes of the user (e.g., location of the user, native language of the user, etc.). The ranking engine 128 then uses the score to rank the responsive content.

[0065] Search engine 126 uses the identified responsive content ranked by ranking engine 128 to generate search results. Search results include search results corresponding to the content of the search issued in response to text input. For example, each search result can include a title or other summary in response to the content item, an overview of the content item, a link to the responsive content item, other information about the responsive content item, and / or even the entire content item. As an example, an overview of a news report content item may include a specific "segment" or news report chapter. At the same time, for example, for search results associated with an image, the search results may include a display of a reduced size of the image, a title associated with the image, and / or a link to the image. At the same time, for example, for search results associated with a video, the search results may include an image from the video, a segment of the video, a title of the video, and / or a link to the video.

[0066] As described herein, in some embodiments, during a conversation with the automated assistant 120, content is identified and / or ranked based on past text input provided by the user. For example, the search engine 126 may search the search database 154 based on parameters determined by past text input, and / or the ranking engine 128 may rank content based on parameters determined by past text input. For example, when ranking content identified by the search engine 126, if the user was previously presented with search results related to Topic A, the ranking engine 128 may filter all content items related to "Topic A", and provide the user with a response text input of "never provide results about this topic". As another example, when ranking content, if the user was previously presented with search results from Source 2, the ranking engine 128 may promote content items from "Source 2", and provide the user with a response text input of "I like this source".

[0067] The ranked search results generated by search engine 126 and ranking engine 128 are provided by output generation engine 130 for presentation to a user via user interface output device 104. For example, output generation engine 130 may provide for audible and / or visual presentation of the search results via user interface output device 104, and may provide the search results as part of a conversation between a user and automated assistant 120.

[0068] In some embodiments, the output generation engine 130 sequentially provides search result groups for presentation to the user as part of a conversation, and the order in which the search results are provided can be based on the ranking of the search results. For example, the output generation engine 130 can provide a first group including only the highest ranked search results, then provide a second group including only the second highest ranked search results, then provide a third group including the third highest ranked search results, etc. In some embodiments, the output generation engine 130 automatically provides a second search result group after the first group (e.g., immediately before or after a time delay), automatically provides a third group after the second group, etc. In some embodiments, before providing the second group, the output generation engine 130 waits for further user interface input, then, before providing the third group, waits for further user interface input, etc. For example, in some of these embodiments, if during or after providing the first group, the user provides a spoken user interface input, such as "next", "another", etc., the output generation engine 130 only provides the second group. One user interface input can be used to cause the output generation engine 130 to sequentially advance through the search result groups.

[0069] The output adaptation engine 124 adapts the search results provided by the output generation engine 130 in response to some further text input by the user. In some embodiments, the output adaptation engine 124 disrupts the order of the search results sequentially provided by the output generation engine 130 in response to some further text input. For example, the output adaptation engine 124 can determine, based on the order, that at least one term of the further text input includes an attribute of one or more search results that is not included in the next group to be provided. Based at least in part on this determination, the output adaptation engine 124 can provide one or more search results with the attribute and provide the search results in place of the next group to be provided based on the order.

[0070] To illustrate some examples, assume that search result A is provided, and then the output generation engine 130 that provided search result B is providing search result C and specifies that search result D is to be provided after search result C. In response to a text input "back to the first result" provided before search result D is provided, the output adaptation engine 124 causes the output generation engine 130 to provide search result A again in place of search result D. This may be based on determining that the term "back to" refers to a previously presented search result and based on determining that the term "first result" matches an attribute of the previously presented search result A (an attribute of the search result first provided by the output generation engine 130). In response to a text input "back to the search result about Topic A" provided before search result D is provided, the output adaptation engine 124 may cause the output generation engine 130 to provide search result B again in place of search result D. This is based on determining that the term "back to" refers to a previously presented search result and based on determining that the previously presented search result B is related to "Topic A" (and that search results A and C are not related to Topic A and / or are less relevant to Topic A). Determining that a search result is related to a topic, related to an entity, from a particular source, and / or has other attributes can be based on those attributes assigned to the search result and / or underlying content in the search database 154 and / or other databases.

[0071] In response to a text input “more results like this one” provided before providing search result D, the output adaptation engine 124 may cause the output generation engine 130 to provide a search result X that has not yet been presented in place of search result D. This may be based on determining that “more” has a positive sentiment (e.g., based on the annotation of the message processing engine 122), that “like this one” references (at least in part) the attributes of a previously presented search result C, and that one or more attributes of search result C are more similar to attributes of search result X than to attributes of search result D. In response to a text input “I don't like this source” provided before providing search result D, the output adaptation engine 124 may cause the output generation engine 130 to provide a search result Y that has not yet been presented in place of search result D. This can be based on determining that "don't like" has a negative sentiment (e.g., based on the annotation of the message processing engine 122), "this source" refers to (at least in part) the source attributes of the previously presented search result C, and that search result D has the same source attributes as search result C, while the next search result Y in the sequence does not have the same source attributes as search result C. In various example implementations provided, the output adaptation engine 124 relies on one or more annotations provided by the message processing engine 122 and uses various techniques for semantic understanding of the text fragment, such as rule-based techniques, template-based techniques, machine learning models (e.g., deep neural networks), and / or other techniques.

[0072] In some implementations, the output adaptation engine 124 additionally and / or alternatively uses the sentiment expressed about the attribute to modify the identification and / or ranking of search results for downstream searches based on downstream text input in the conversation between the user and the automated assistant 120. For example, the above-mentioned text input of "I don't like this source" can be used to reduce the ranking of downstream search results having the same source attribute as the search result C.

[0073] Now back to Figure 2 , provides additional descriptions of the various components of automated assistant 120. Figure 2In the example, the message processing engine 122 receives the text input 201 and generates an annotation input 203 including annotations of the text input and / or the terms of the text input 201 itself. The search engine 126 uses the annotation input 203 to determine the search parameters, issues a search to one or more search databases 154 based on the search parameters, and identifies the content in response to the search. The ranking engine 128 uses one or more signals to rank the responsive content. For example, the ranking engine 128 ranks the responsive content based on the popularity of the responsive content (e.g., as indicated by the entry of the content in the search database 154) and / or based on one or more parameters of the parameter database 156. As described herein, one or more parameters of the parameter database 156 can be determined based on past text inputs in a conversation with the user. A given parameter can define an attribute and how to affect search results with the attribute. For example, a parameter can define an attribute and define how to promote or reduce search results with the attribute in the ranking of search results, or should prevent search results with the attribute from being included in the search results provided for presentation to the user.

[0074] The ranking engine 128 provides the search results and the order 205 of the search results to the output generation engine 130. The search results are based on the content identified by the search engine 126 and the order of the search results may be based on the ranking determined by the ranking engine 128.

[0075] The output generation engine 130 starts to sequentially provide search results based on the order. Figure 2 As shown, the output generation engine 130 provides search result group 1 207 and, optionally, provides further search result groups as indicated by ellipses.

[0076] At some point during the sequential provision of search results by the output generation engine 130, the message processing engine 122 receives further text input 211. The message processing engine 122 may determine that the further text input 211 is relevant to the adaptation of the presentation of the search results and, therefore, provide further annotation input 213 to the output adaptation engine 124. The further annotation input 213 may include annotations of the further text input 211 and / or one or more terms of the further text input 211.

[0077] In some implementations, based on the further annotation input 213, the output adaptation engine 124 determines an adaptation 215 so that the search results are provided sequentially and communicates the adaptation 215 to the output generation engine 130. The output generation engine 130 can adapt the current sequential provision of the search results based on the adaptation 215. For example, instead of providing the next search result in the current sequential provision, the output generation engine 130 can instead provide out-of-order search results based on the adaptation 215.

[0078] In some embodiments, based on the further annotation input 213, the output adaptation engine 124 additionally or alternatively determines parameters 217 for storage in the parameter database 156. These stored parameters 217 can be used to modify additional downstream searches and / or rankings by the engines 126 and / or 128 in response to new searches based on new text input. It should be noted that in some embodiments, the output adaptation engine 124 can provide both adaptations 215 and parameters 217 simultaneously. For example, for the further text input 211 "I never want results from Source A": the adaptation 215 can be provided and the search results from Source A and specified by the output generation engine 130 to be provided are no longer provided, and the parameters 217 can be stored and additional search results from Source A can be prevented from being provided in response to new searches based on the new text input.

[0079] Reference now Figure 3-7 , various examples of implementations of automated assistant 120 are described. Figure 3 , 4A -4E, 6, and 7 each illustrate a computing device 110 having a display screen 140 showing an example of a conversation occurring between a user of the computing device 110 and the automated assistant 120, according to embodiments disclosed herein. One or more aspects of the automated assistant 120 may be implemented on the computing device 110 and / or one or more computing devices in network communication with the computing device 110.

[0080] Figure 3 , 4A -4E, 6 and 7 display screens 140 further include a text reply interface element 188 that the user can select to generate a user interface input via a virtual keyboard and a voice reply interface element 189 that the user can select to generate a user interface input via a microphone. In some embodiments, the user can generate a user interface input via a microphone without selecting the voice reply interface element 189. For example, during a conversation, active monitoring of audible user interface inputs via a microphone can occur to eliminate the need for the user to select the voice reply interface element 189. In some of those embodiments and / or other embodiments, the voice reply interface element 189 can be omitted. In addition, in some embodiments, the text reply interface element 188 can be omitted in addition and / or alternatively (e.g., the user only provides audible user interface input). Figure 3 , 4A -4E, 6 and 7 display screens 140 also include system interface elements 181, 182, 183 that can interact with a user to cause computing device 110 to perform one or more actions.

[0081] Figure 5The diagram illustrates a computing device 110 including one or more microphones and one or more speakers and illustrates an example of a conversation occurring via the microphone and speaker between a user 101 of the computing device 510 and an automated assistant 120. One or more aspects of automated assistant 120 may be implemented on computing device 510 and / or on one or more computing devices in network communication with computing device 510.

[0082] exist Figure 3 , the user provides initial text input 380A as part of a conversation between the user and automated assistant 120. In response to text input 380A, automated assistant 120 obtains search results responsive to text input 380A and provides search results 382A for presentation on display screen 140 as a transcript of the conversation. Automated assistant 120 first provides search result 382A for presentation based on it being first in the order of presentation of the search results (e.g., based on the ranking of the search results). Then, the user provides text input 380B "next" to cause automated assistant 120 to provide search result 382B for presentation. According to the order, search result 382B is the next search result.

[0083] The user then provides text input 380C "more like this". The automated assistant 120 determines that "like this" refers to the most recently presented search result 382B. The automated assistant 120 further determines one or more attributes of the most recently presented search result 382B, such as the following attributes: the search result is associated with a first entity corresponding to the fictional chef Jon Doe, and the search result is associated with a second entity corresponding to the fictional restaurant Hypothetical Café. The automated assistant 120 further determines that like the search result 382B, the search result 382C also has an attribute associated with the first entity corresponding to the fictional chef Jon Doe. Based on this determination, the automated assistant 120, in response to the text input 380C, provides the search result 382C for presentation. In some embodiments, the search result 382C is not the next search result in the sequence according to the original order. However, based on the text input 380C, the automated assistant 120 can provide the search result 382C instead of the next search result designated to be provided according to the original order. In other words, text input 380C may cause automated assistant 120 to adapt the original order of presentation of search results so that one or more search results that are similar to search result 382B are promoted in the order.

[0084] The user then provides text input 380D "No, about Hypothetical Café". Automated assistant 120 determines, based on text inputs 380C and 380D, that "No, about Hypothetical Café" refers to a search result that the user wants more information about a fictional restaurant, Hypothetical Café, rather than about a fictional chef, Jon Doe. Automated assistant 120 further determines that search result 382D has attributes associated with an entity corresponding to a fictional restaurant, Hypothetical Café, and based on this determination, provides search result 382D for subsequent presentation. In some embodiments, search result 382D may not be the next search result according to the order as modified by text input 380C. However, based on text input 380D, automated assistant 120 may provide search result 382 instead of the next search result designated to be provided according to the modified order. In other words, text input 380D may cause automated assistant 120 to further adapt the modified order of presentation of search results so as to elevate one or more search results related to "Hypothetical Café" in the order.

[0085] It should be noted that in some embodiments, in response to text input 380C, automated assistant 120 identifies multiple search results that share one or more attributes with search result 382B, and based on one or more factors, selects search result 382C in place of those other search results. For example, automated assistant 120 may determine that the entity corresponding to the fictitious chef Jon Doe has a greater weight (e.g., is more relevant) to search result 382B than the entity corresponding to the fictitious restaurant Hypothetical Café. Based on search result 382C being more relevant to the entity corresponding to the fictitious chef Jon Doe, search result 382C may be initially selected in place of search result 382D and / or other search results.

[0086] In some embodiments, in response to text input 380C requesting disambiguation of the input "like this", the automated assistant 120 may optionally generate a prompt. For example, the automated assistant 120 may identify one or more attributes of the search result 382B and formulate a prompt that requires the user to specify which attributes the user "like this" refers to. For example, the automated assistant 120 may determine that the search result 382B is associated with the source attribute of "Source 2" and is associated with the entity attributes of "Jon Doe (Chef)" and "Hypothetical Café". Based on this determination, the prompt may be "do you want more from Source 2, more about the chef, or more about Hypothetical Café?". Further text input provided in response to the prompt may be used by the automated assistant to select appropriate further search results to provide. For example, further text input of "aboutHypothetical Café" may result in the initial provision of search result 382D, replacing search result 382C.

[0087] exist Figure 4A , the user provides initial text input 480A as part of a conversation between the user and automated assistant 120. In response to text input 480A, automated assistant 120 obtains search results responsive to text input 480A and provides search results 482A for presentation on a display screen as part of the conversation. Automated assistant 120 first provides search result 482A based on it being first in the presentation order of the search results (e.g., based on the ranking of the search results).

[0088] After the search results 482A are displayed to the user, the user then provides a text input 488B "next", such as Figure 4B In response, automated assistant 120 provides an output that replaces search result 482A with search result 482B, as shown in FIG. Figure 4B 4. According to this order, search result 482B is the next search result. Text input 488B of FIG. 4 is provided by typing (e.g., by selecting interface area 184) or by voice (e.g., by selecting microphone interface element 185 - or just by speaking without necessarily selecting element 185 (i.e., the automated assistant can monitor for voice input during provision)).

[0089] Note that text input 480A is the same as text input 380A, search result 482A is the same as search result 382A, text input 488B is the same as text input 380B, and search result 482B is the same as search result 382B. Figure 3 , the text input 380B is added to the copy of the dialog presented in the graphical interface and persists in that copy. Figure 4B The text input 488B is part of the dialog, but is not added to the copy of the dialog presented by the graphical interface. Figure 3 In , after providing search result 382B, search result 382A persists in the copy. Figure 4B In the example, search result 482B replaces search result 482A, thereby removing it from the copy. In other words, search result 482B replaces search result 482A, thereby preventing search result 482A and search result 482B from being displayed simultaneously. Figures 4A-4E The described techniques are advantageous, for example, for limited size display screens and / or interfaces where it is not distracting and / or computationally burdensome to maintain a full transcript of the conversation.

[0090] After the search results 482B are displayed to the user, the user then provides a text input 488C "next", such as Figure 4C In response, automated assistant 120 provides an output that replaces search result 482B with search result 482C, as shown in FIG. Figure 4C According to this order, search result 482C is the next search result.

[0091] After search result 482C is displayed to the user, the user then provides text input 488D "back to the one from source 1". Automated assistant 120 can determine that "back to" refers to a previously presented search result and that "source 1" is an attribute of previously presented search result 482A. In response, automated assistant 120 causes search result 482C to be replaced by expanded search result 482D. Expanded search result 482D includes the same content as search result 482A, but also includes some additional text from the base content item. Figure 4D, the user can optionally further navigate the search results (e.g., "Ok, now to search result 5") or can select (e.g., audibly select or "touch select") a link associated with the expanded search result 482D to view all base content items associated with the expanded search result 482D. Viewing all base content items can optionally occur in a separate interface and / or a separate application.

[0092] Figure 4E The diagram shows Figures 4A-4D A similar example, but in which the various conversation items persist in the conversation. In particular, Figure 4E In the text input 480B, 480C and 480D and Figures 4B-4D 488B, 488C and 488D. However, in Figure 4E , text inputs 480B, 480C, and 480D are still in the transcript of the conversation, while their counterparts are not Figures 4B-4D At the same time, Figure 4E , search results 482A, 482B, and 482C are still in the transcript of the conversation, but they are not Figures 4B-4D middle.

[0093] exist Figure 5 , the user provides initial text input 580A as part of an interaction between the user and automated assistant 120. In response to text input 580A, automated assistant 120 obtains search results responsive to text input 580A and provides search results 582A for audible presentation via a speaker of computing device 510. Automated assistant 120 provides search result 582A first, based on it being first in the presentation order of the search results.

[0094] During the audible provision of search results 582A, the user provides text input 580B while the provision of search results 582A is stopped. Text input 580B "I never want news from Source 1" can be used to prevent further search results from Source 1 from being provided to the user by the automated assistant 120. For example, the automated assistant 120 can suppress search results with attributes of Source 1 based on the sentiment indicated by "never". In some embodiments, further search results from Source 1 are suppressed only for those search results that are responsive to text input 580A. In some other embodiments, further search results from Source 1 are suppressed for search results that are responsive to further text input. For example, search results from Source 1 can continue to be suppressed by the automated assistant 120 until the user explicitly indicates, via further text input and / or other interface input, that the user wishes to receive search results from Source 1 again. In some embodiments, a smaller magnitude of negative sentiment results in less duration and / or degree of suppression of search results from Source 1. For example, "I'm not a big fan of Source 1" may result in a downgrade of search results from Source 1 based on a sentiment associated with "not a big fan of" being lower in magnitude than the sentiment associated with "never," but not excluded. In some implementations, the direction (e.g., positive, negative) and / or magnitude of sentiment for terms associated with attributes of search results is determined based on a sentiment classifier, one or more rules, and / or based on other techniques.

[0095] In response to text input 580B, providing search result 582A ceases and providing search result 582B. In some implementations, search result 582B is next in the order after search result 582B. In some other implementations, one or more intermediate search results are provided in the order, between search result 582A and search result 582B, based on the search result associated with the source attribute of "Source 1," but the search results are skipped by automated assistant 120.

[0096] During the audible provision of search results 582B, the user provides text input 580C to stop providing search results 582B. In response to text input 580C "next", the automated assistant may provide the next search result 582C in the sequence. During the audible provision of search results 582C, the user provides text input 580D to stop providing search results 582C. In response to text input 580D "next", the automated assistant may provide the next search result 582D in the sequence.

[0097] During the audible presentation of search results 582D, the user provides text input 580E to stop providing search results 582D. Automated assistant 120 can determine that text input 580E "back" refers to the previously presented search results and that "Local Business" is an attribute of previously presented search results 582B. In response, automated assistant 120 again presents search results 582B, and since the user did not stop the audible presentation by further text input, a majority of search results 582B are audibly presented.

[0098] Figure 6 and 7 The diagram illustrates an example of how, via further conversation with automated assistant 120, a user may explore additional details associated with a given search result and then navigate forward and / or backward to other search results.

[0099] exist Figure 6 , the user provides initial text input 680A as part of a conversation between the user and automated assistant 120. In response to text input 680A, automated assistant 120 obtains search results responsive to text input 680A and provides search results 682A for presentation as part of a transcript of the conversation. Automated assistant 120 first provides search result 682A based on it being first in the order of presentation of the search results.

[0100] The user then provides text input 680B "What time does it open?" to cause the automated assistant 120 to present additional details 682B related to the search result 682A and in response to the text input 680B. The user then provides text input 680C "next." The automated assistant 120 may determine that "next" refers to the next search result in the order and, in response to the text input 680C, provide the next search result 682C. In some implementations, based on the additional details 682B being a single item, rather than a list of items, the automated assistant 120 may determine that "next" refers to the next search result, rather than to further details related to the search result 682A. In other words, having only a "time that it opens" and "next" cannot refer to additional times of operation.

[0101] exist Figure 7 In the text input 780A and Figure 6 The text input 680A is the same as that of the search result 782A. Figure 6 The search results for 682A are the same. However, in Figure 7 In the text input 780B, it is different from Figure 6The text input 780B "Reviews?" causes the automated assistant 120 to present additional details 782B related to the search result 782A and in response to the text input 780B. The additional detail 782B is one of a plurality of available reviews for "Restaurant A."

[0102] The user then provides text input 780C "next review". Based on the presence of "review", automated assistant 120 can determine that "next" refers to the next review, rather than the next search result in the sequence. Therefore, in response to text input 780C, automated assistant 120 provides additional details 782C of the next review as "Restaurant A".

[0103] The user then provides text input 780D "next restaurant". Based on the presence of "restaurant", automated assistant 120 can determine that "next" refers to the next restaurant search result. Therefore, in response to text input 780D, automated assistant 120 provides additional search result 782D as another restaurant search result in response to text input 780A.

[0104] Figure 8 800 is a flowchart illustrating an example method 800 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as automated assistant 120. Furthermore, although the operations of method 800 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0105] At block 850, the system receives text input. The text input may be generated as part of a conversation between the user and the system, and may be based on user interface input generated by a user interface input device, such as a microphone or a virtual keyboard.

[0106] At block 852, the system obtains search results in response to the text input and obtains a presentation order for the search results. The presentation order can be based on the ranking of the search results. In some embodiments, the system itself performs the search to obtain the search results and / or ranks the search results. In some embodiments, the system provides the search parameters to one or more separate systems and, in response, obtains the search results and the ranking of the search results.

[0107] At block 854, the system provides an initial set of search results. For example, depending on the presentation order, the system may only provide the first search result. The initial set of search results is provided to be presented to the user audibly and / or visually as part of the conversation.

[0108] At block 856, the system receives further text input. The system then determines what type of input the further text input is at one or more of blocks 856A, 856B, 856C, and 856D.

[0109] If it is determined at block 856A that the further text input is a continuing input, the system proceeds to block 858. At block 858, the system provides the next search result group according to the presentation order. As an example of block 856A, the system may determine that the further text input is a continuing input if only one or more terms in a predetermined "continuing" term set are included, such as "next", "continue", "go on", etc. As an example of block 858, the system may only provide the next search result according to the presentation order.

[0110] If, on the other hand, at block 856B, it is determined that the further text input is an adapting input, the system proceeds to block 860. At block 860, the system modifies the presentation order based on the further text input. The system then proceeds to block 858 and provides the next search result group based on the presentation order (as modified at block 860). The system additionally or alternatively proceeds to block 862, where the system determines and stores parameters based on the further text input. As described herein, in downstream iterations of block 852, the parameters can modify the recognition and / or search result ranking.

[0111] As an example of block 856B, the system determines that the further text input is an adapted input if it matches one or more adaptation templates (e.g., "back to result [#]", "forward to result [#]") and / or based on other techniques along with other terms, including one or more of a predetermined set of "adaptation" terms (e.g., "back to", "forward to", "more like", "less like", "less about", "never", "remove"). As an example of block 860, the system can modify the presentation order to promote, demote, or remove one or more search results having attributes explicitly or implicitly referenced in the further text input.

[0112] On the contrary, if it is determined at 856C that the further text input is a result detail input, the system proceeds to block 864. At block 864, the system provides further search result details based on the input. As an example of block 856C, if further content related to the search result provided at block 854 is requested, the system can determine the further text input as a result detail input. For example, if the "restaurant" search result is provided at block 854, a further text input of "reviews for thisrestaurant" can be determined as a result detail input.

[0113] On the contrary, if it is determined in block 856D that the further text input is other input, the system proceeds to block 866 and performs other actions. For example, the text input can be an input intended to initiate a new search, and the other actions of block 866 can return to block 852 and obtain search results in response to the further text input. At the same time, for example, the text input can be an input requesting a single response, such as "what time is it" and the further actions of block 866 can be to provide an output indicating the current time, and then return to block 856.

[0114] During a conversation with a user, multiple iterations of blocks 856, 856A / B / C / D, 858, 860, 862, 864, and / or 866 may occur, allowing the user to navigate and explore multiple search results according to the techniques disclosed herein. Although blocks 856A, 856B, 856C, and 856D are shown in a particular order, it should be understood that the order may be changed, one or more blocks may be performed in parallel, and / or one or more blocks may be performed only selectively.

[0115] Fig. 9 900 is a flowchart illustrating an example method 900 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as automated assistant 120. Furthermore, although the operations of method 900 are shown in a particular order, this is not a limitation. One or more operations may be reordered, omitted, or added.

[0116] At block 950, the system receives text input. The text input may be generated as part of a conversation between the user and the system, and may be based on user interface input generated by a user interface input device, such as a microphone or a virtual keyboard.

[0117] At block 952, the system obtains search results responsive to the text input. In some implementations, at block 952, the system also obtains a presentation order for the search results.

[0118] At block 954, the system provides one or more search result groups. For example, the system provides the first search result, then the second search result, then the third search result, and so on.

[0119] At block 956 , the system receives text input during the provision at block 954 .

[0120] At block 958, the system determines, based on the text input, one or more attributes of the one or more search results provided at block 954. For example, the text input may be "never show results from this source". Based on "this source", the system may determine the source attribute assigned to the most recently provided search result. As another example, the text input may be "more like this". Based on "likethis", the system may determine the attribute assigned to the most recently provided search result, such as a source attribute, a subject attribute, a document type attribute (e.g., an image, video, or web page), etc. As yet another example, the text input may be "no more from Source 1" and the system may determine the source attribute of "Source 1", which is also the source attribute of the one or more search results provided at block 954.

[0121] At block 960, the system determines parameters based on the determined attribute and the emotion expressed for the attribute in the further text input of block 956. A given parameter can define an attribute and how it affects search results with the attribute. The system can determine how to affect search results with the attribute based on the emotion expressed for the attribute in the further text input. In some embodiments, the system can access mappings of certain emotion terms with certain effects to determine the effect. For example, "never" can be mapped to suppress search results with the attribute. "don't like" can be mapped to be downgraded to the first-level search results with the attribute. "really don't like" can be mapped to be downgraded to a larger second-level search result with the attribute. "more like" can be mapped to be promoted to the first-level search results with the attribute, etc. In some embodiments, the system can additionally or alternatively utilize an emotion classifier to determine the direction and / or magnitude of the emotion, and can determine how to affect search results with the attribute based on the direction and / or magnitude.

[0122] At block 962, the system uses the parameter to influence whether or when to provide one or more attribute search results having the attribute for presentation to the user. In some embodiments, the system uses the parameter to influence one or more search results obtained at block 952, and ultimately influence further provision of search results in response to the text input received at block 950. In some embodiments, the system additionally or alternatively uses the parameter to influence one or more search results for downstream searches based on downstream text input in a conversation between the user and the system.

[0123] Various examples described herein are described with reference to visually presenting search results via a graphical user interface and / or audibly presenting search results via a speaker. However, in some embodiments, search results may be provided for presentation to a user using additional and / or alternative techniques. For example, in some embodiments, search results may be provided for tactile presentation to a user. As another example, in some embodiments, search results may include a plurality of lighting commands and / or lighting scenes to be implemented by a lighting system, and each of the search results may be presented as a lighting output from the lighting system. For example, providing lighting scene search results for presentation to a user may include providing a command or other output to a lighting system controller and / or an independent component of the lighting system (e.g., an independent light bulb or other lighting unit) to generate a lighting scene corresponding to the lighting scene search result to be generated by the lighting system. As a specific example, a first lighting scene search result may be presented to a user as a lighting output from the lighting system, and in response to a "next" user interface input, a next lighting scene in the order of the search results may be presented; in response to another "next" user interface input, a next lighting scene in the order may be presented, and in response to a further user interface input identifying a previously presented search result, another search result in the reverse order of the search results may be presented. For example, further text input "back to the first lighting scene" may cause the initially presented lighting scene to be presented again, further text input "back to the one with a lot of red" may cause the previously provided lighting scene with the attribute "red" to be presented again, further text input "more like this" may cause an additional scene to be presented, wherein the scene is in the reverse order and is identified based on sharing attributes with the currently presented scene. As another specific example, a given lighting scene search result may be presented to the user as a lighting output from a lighting system, and when the given lighting scene search result is presented, the user may provide further user interface input "never any lighting scenes like this". The further user interface input may be used to determine whether or when to present, in response to a further dialog, a further lighting scene search result having one or more attributes in common with the given lighting scene search result.

[0124] Fig.101010, which may optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, automated assistant 120, and / or other components may include one or more components of the example computing device 1010.

[0125] The computing device 1010 typically includes at least one processor 1014, which communicates with a number of peripheral devices via a bus subsystem 1012. These peripheral devices may include a storage subsystem 1024, for example including a memory subsystem 1025 and a file storage subsystem 1026, a user interface output device 1020, a user interface input device 1022, and a network interface subsystem 1016. The input and output devices allow a user to interact with the computing device 1010. The network interface subsystem 1016 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0126] The user interface input device 1022 may include a keyboard, a pointer device (such as a mouse, a trackball, a touchpad, or a drawing tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 1010 or over a communication network.

[0127] User interface output device 1020 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include devices and methods for outputting information from computing device 1010 to all possible types of users or another machine or computing device.

[0128] The storage subsystem 1024 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1024 may include an executable Figure 8 methods and / or Fig. 9 The logic of the selected aspect of the method.

[0129] These software modules are typically executed by the processor 1014 independently or in combination with other processors. The memory 1025 used in the storage subsystem 1024 can include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 1026 can provide permanent storage for program and data files, and can include a hard drive, a floppy disk drive together with associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functions of certain embodiments can be stored by the file storage subsystem 1026 in the storage subsystem 1024, or stored in other machines accessible by the processor 1014.

[0130] The bus subsystem 1012 provides a mechanism for the various components and subsystems of the computing device 1010 to communicate with each other as intended. Although the bus subsystem 1012 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0131] The computing device 1010 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Fig.10 The description of computing device 1010 depicted in the drawings is intended only as a specific example for illustrating some embodiments. Many other configurations of computing device 1010 may have more features than those described in the drawings. Fig.10 The computer system may have more or fewer components than those depicted.

[0132] In situations where the systems described herein collect personal information about a user or may use personal information, the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, before certain data is stored or used, it may be processed in one or more ways so as to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information is not determined for the user, or the user's geographic location may be generalized to the location where geographic location information is obtained (such as a city, zip code, or state level) so that the user's specific geographic location is not determined. Thus, the user may control how information is collected about the user and / or how the information is used.

Claims

1. A method for adapting search results, comprising: As part of a conversation between a user and an automated assistant implemented at least in part by one or more processors: Receiving spoken input from the user; obtaining search results responsive to the spoken input; providing the search results for presentation to the user; In response to providing the search results, receiving further spoken input from the user; Based on processing the further spoken input, it is determined that the further spoken input references both: the attributes of the search results, and the sentiment expressed by the user regarding the attribute; In response to determining that the further spoken input references the attribute of the search result and the sentiment expressed by the user with respect to the attribute: A determination is made based on the attribute and the sentiment expressed by the user with respect to the attribute whether or when to provide as part of the conversation additional search results that are also responsive to the spoken input and also have the attribute.

2. The method according to claim 1, wherein: Determining whether or when to provide the additional search results as part of the conversation based on the attribute and the sentiment expressed by the user with respect to the attribute includes: The additional search results are prevented from being provided based on the sentiment expressed by the user with respect to the attribute being a negative sentiment.

3. The method according to claim 1, wherein: Determining whether or when to provide the additional search results as part of the conversation based on the attribute and the sentiment expressed by the user with respect to the attribute includes: The additional search result is provided as a next search result after providing the search result based on that the emotion expressed by the user with respect to the attribute is a positive emotion.

4. The method according to claim 1, wherein: At the time the further spoken input from the user is received, the additional search results have not yet been provided in the conversation.

5. The method according to claim 4, in, The search result and the additional search result are included in a plurality of search results responsive to the spoken input; wherein providing the search results comprises providing the search results according to an initial order of the plurality of search results in response to the spoken input; and Wherein, determining whether or when to provide the additional search results as part of the conversation based on the attribute and the emotion expressed by the user with respect to the attribute includes: A determination is made to provide the additional search results in the conversation in a reverse order of the initial order of the search results.

6. The method according to claim 1, further comprising: determining a magnitude of emotion expressed by the user with respect to the attribute based on processing the further spoken input; Wherein determining whether or when to provide the additional search results that are also responsive to the spoken input and also have the attribute as part of the conversation is further based on the magnitude of the emotion.

7. The method according to claim 6, wherein: Further determining, based on the magnitude of the emotion, whether or when to provide, as part of the conversation, the additional search results that are also responsive to the spoken input and also have the attribute, comprises: A determination is made based on the magnitude of the sentiment whether to demote the additional search results in a presentation order for the conversation or alternatively prevent the additional search results from being presented in the conversation. 8 . The method of claim 1 , further comprising determining a sentiment expressed by the user with respect to the attribute based on one or more terms included in a textual transformation of the spoken input.

9. The method of claim 1, further comprising determining a sentiment expressed by the user with respect to the attribute based on speech characteristics of the further spoken input.

10. The method according to claim 1, wherein: The attribute of the search result is a source of the search result, and wherein the further spoken input includes a term referencing the source and an additional term referencing the sentiment.

11. The method according to claim 1, wherein: The reference to the property in the further spoken input is a reference to the search result without an explicit reference to the property, the method further comprising: The attributes are identified based on the attributes assigned to the search results in one or more computer-readable media.

12. The method according to claim 1, wherein: The attribute of the search result is the entity referenced in the search result.

13. The method according to claim 1, wherein: The further spoken input is received during presentation of the search results, wherein the search results are presented in an order according to an initial sequential order of the search results, and wherein no other of the search results are presented during presentation of the search results.

14. A method for adapting search results, comprising: As part of a conversation between a user and an automated assistant implemented at least in part by one or more processors: Receiving spoken input from the user; obtaining search results responsive to the spoken input; providing the search results for presentation to the user; In response to providing the search results, receiving further spoken input from the user; Based on processing the further spoken input, it is determined that the further spoken input references both: the attributes of the search results, and the sentiment expressed by the user regarding the attribute; As part of an additional conversation between the user and the automated assistant and subsequent to the conversation: receiving additional dialogue spoken input from the user; obtaining additional search results that are responsive to the additional dialog spoken input and also have the attributes; In response to determining that the further spoken input of the conversation references the attribute of the search result and the sentiment expressed by the user with respect to the attribute, and in response to the additional search result also having the attribute: A determination is made based on the attribute and the sentiment expressed by the user with respect to the attribute whether or when to provide the additional search results as part of the additional conversation in response to the additional conversation spoken input.

15. The method according to claim 14, wherein: Determining whether or when to provide the additional search results as part of the additional conversation in response to the additional conversation spoken input based on the attribute and the emotion expressed by the user with respect to the attribute includes: The additional search results are prevented from being provided based on the sentiment expressed by the user with respect to the attribute being a negative sentiment.

16. The method according to claim 14, further comprising: determining a magnitude of emotion expressed by the user with respect to the attribute based on processing the further spoken input; Wherein determining whether or when to provide the additional search results as part of the additional conversation in response to the additional conversation spoken input is further based on the magnitude of the emotion.

17. The method of claim 14, further comprising determining a sentiment expressed by the user with respect to the attribute based on one or more terms included in a textual transformation of the further spoken input.

18. The method of claim 14, further comprising determining a sentiment expressed by the user with respect to the attribute based on speech characteristics of the further spoken input.

19. The method according to claim 14, wherein: The attribute of the search result is an entity referenced in the search result, and wherein the further spoken input includes a term referencing the entity and an additional term referencing the sentiment.

20. The method according to claim 14, wherein: The reference to the property in the further spoken input is a reference to the search result without an explicit reference to the property, the method further comprising: The attributes are identified based on the attributes assigned to the search results in one or more computer-readable media.

21. A method for adapting search results, comprising: receiving an audible input based on a first user interface input generated by a user via a user interface input device, wherein the first user interface input is generated by the user as part of a conversation between the user and an automated assistant implemented at least in part by one or more processors; obtaining a plurality of lighting scenes responsive to the audible input; causing a lighting scene of the lighting scenes to be presented as lighting output from a lighting system; After causing the one lighting scene to be rendered as the lighting output, causing additional lighting scenes in the lighting scene to be rendered as additional lighting output from the lighting system; receiving a further audible input during the presentation of the additional lighting output, the further audible input being based on an additional user interface input generated by the user via the user interface input device or another user interface input device; determining that the additional user interface input corresponds to the lighting scene based on one or more terms of the additional user interface input and based on at least one attribute of the lighting scene, Wherein, based on the one or more terms of the additional user interface input and based on the at least one attribute of the lighting scene, determining that the additional user interface input corresponds to the lighting scene comprises: determining that the one or more terms of the terms of the additional user interface input match the at least one attribute of the lighting scene; In response to determining that the additional user interface input corresponds to the lighting scene, the lighting scene is again caused to be presented by the lighting system.

22. The method according to claim 21, wherein: The user interface input device is a microphone, and wherein the additional user interface input is generated by the user via the microphone.

23. The method according to claim 21, further comprising: Actively monitoring for the additional user interface input occurs during the time the additional lighting output is presented, wherein receiving the additional user interface input occurs during the active monitoring.

24. The method according to claim 21, wherein: The at least one property of the lighting scene comprises a color of the lighting scene.

25. A method for adapting search results, comprising: receiving an audible input based on a first user interface input generated by a user via a user interface input device, wherein the first user interface input is generated by the user as part of a conversation between the user and an automated assistant implemented at least in part by one or more processors; obtaining a plurality of lighting scenes responsive to the audible input; causing a lighting scene of the lighting scenes to be presented as lighting output from a lighting system; receiving a further audible input while the lighting output is rendered, the further audible input being based on an additional user interface input generated by the user via the user interface input device or another user interface input device; determining, based on one or more terms of the additional user interface input and based on at least one property of the lighting scene, that the additional user interface input refers to a property of the lighting scene and to a desire for an additional lighting scene having the property, In response to determining that the additional user interface input references a property of the lighting scene and references the desire for an additional lighting scene having the property: selecting an additional lighting scene from the plurality of lighting scenes also having the properties; and The additional lighting scene is caused to be presented as additional lighting output from the lighting system.

26. The method according to claim 25, further comprising: Actively monitoring the additional user interface input while the lighting output is rendered, wherein receiving the additional user interface input occurs during the active monitoring.

27. The method according to claim 25, wherein: The at least one property of the lighting scene comprises a color of the lighting scene.

28. A method for adapting search results, comprising: As part of a conversation between a user and an automated assistant implemented at least in part by one or more processors: enabling lighting scenes to be realized by lighting systems, and In response to the lighting scene being implemented, receiving a text input referencing an attribute of the lighting scene and referring to a sentiment expressed by the user with respect to the attribute, the text input being based on a user interface input generated by the user via a user interface input device; determining, based on the attribute of the search result and the sentiment expressed by the user with respect to the attribute, a parameter that affects whether and when one or more additional lighting scenes having the attribute are provided by the automated assistant in response to further conversation between the user and the automated assistant; and As part of the further dialogue between the user and the automated assistant: The parameter is used by the automated assistant to influence whether and when at least one of the one or more additional lighting scenes is provided for presentation to the user, wherein the parameter: influencing when the one or more additional lighting scenes are provided as part of the further dialogue by influencing the ranking of the one or more additional lighting scenes, or Whether the one or more additional lighting scenes are provided as part of the further dialog is influenced by preventing the one or more additional lighting scenes from being provided.

29. The method according to claim 28, wherein: The property of the lighting scene is a color of the lighting scene, and wherein the text input comprises a term referencing the color and an additional term referencing the emotion.

30. The method of claim 28, wherein: The reference to the property in the text input is a reference to the lighting scene without an explicit reference to the property, the method further comprising: The attributes are identified based on the attributes assigned to the lighting scene in one or more computer-readable media.

Images