Suggestions for interacting with automated assistants in multi-user messaging threads

Abstract

The present disclosure relates to providing suggestions for interacting with an automated assistant in a multi-user message interaction topic. At least one context-related suggestion is provided to one or more users of an ongoing message exchange topic between users. The suggestion is provided for presentation to the user via a user interface output device of the user's computing device. A suggestion indication may be submitted to the automated assistant to cause the automated assistant to include content responsive to the query in the message exchange topic. In some embodiments, the suggestion is a selectable suggestion, and the content responsive to the query is included in the message exchange topic in response to a user interface input for the selectable suggestion. In some embodiments, the suggestion is determined based on one or more messages that have been communicated between users of the message exchange topic.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201611236617.9, filed on December 28, 2016. Technical Field

[0003] The present disclosure relates to providing suggestions for interacting with an automated assistant in a multi-user message interaction topic. Background Art

[0004] Multiple users may communicate textually with each other via a message exchange topic, which may be implemented using various technologies, such as text message exchange, online chat (e.g., instant messenger, Internet relay chat), social network messaging topic, etc. Client message exchange applications, such as text messaging applications, online chat applications, social network applications, etc., may display at least a portion of a transcript of an ongoing message exchange topic (e.g., a live or near live transcript of text and / or images that is updated in real time as the users converse) as part of a graphical user interface so that the users can follow the conversation.

[0005] Typically, the ongoing message exchange topic and therefore the displayed log book is limited to text or other data, such as pictures that are intentionally and explicitly inserted into the message exchange topic by the user. Therefore, a user seeking external information related to the session must switch to a different application, such as a web browser, to obtain the desired information. Switching applications may have one or more disadvantages in various situations, such as when using a mobile computing device that typically displays only a single application at a time. For example, switching applications may be distracting and cumbersome, may interfere with the flow of the session and / or may burden one or more computing resources (e.g., memory, processor and / or battery resources of the mobile computing device). Other and / or alternative disadvantages of client message exchange applications may be presented. Summary of the invention

[0006] Some embodiments of the present specification are directed to providing at least one context-relevant suggestion to one or more users of an ongoing message exchange topic between users. The suggestion is provided for presentation to the user via a user interface output device of the user's computing device. The suggestion indicates a query that can be submitted to an automated assistant (e.g., submitted to the automated assistant via a message exchange topic) to include content in the message exchange topic that the automated assistant responds to the query. Including the context-relevant suggestion in the message exchange topic can reduce the number of user inputs that the user must make to the device in order to access content responsive to the query. This is particularly advantageous for users with low dexterity or users who are otherwise able to make user inputs to the device. For at least these users, including selectable suggestions in the message exchange topic can increase the reliability and use of the device, for example, by saving errors made by the user due to low dexterity.

[0007] In some embodiments, the suggestion is a selectable suggestion (e.g., a selectable graphical element provided for display) and content responsive to the query is included in the message exchange topic in response to a user interface input (e.g., a "tap," "click," "verbal command") to the selectable suggestion. In some embodiments, the suggestion is determined based on one or more messages that have been communicated between users of the message exchange topic. For example, the query indicated by the suggestion can be determined based on one or more terms, images, and / or other content of the messages that have been communicated.

[0008] As a specific example of some embodiments described herein, assume that "Tom", "Jerry" and "Spike" join a message exchange topic. In addition, assume that "Tom" uses a message exchange client on one of his computing devices to express the message "I heard that Band A is playing a concert in Louisville, are you interested?" and includes the message in the topic. In addition, assume that Jerry uses a text messaging application on his computing device to express a reply message "Any ideas for dates?" and includes the message in the topic. A query for "dates for Band A's Louisville concert" can be determined based on the message and based on the reply message based on one or more techniques described herein. In addition, a suggestion of a selectable graphical element as an indication of providing a query (e.g., including all terms of the query) can be provided to be displayed to Tom, Jerry and / or Spike via their respective text messaging applications. In response to a user interface input from one of the users for a selectable graphical element displayed to the users, content responsive to the query can be automatically included in the message exchange topic and displayed to one or more (e.g., all) of the users. For example, the content can include text indicating the date, time, and / or location of Band A's concert in Louisville. For example, the content is determined based on one or more search results returned from a search issued for the query.

[0009] In some implementations, providing context-relevant suggestions during an ongoing message exchange thread can facilitate the user to submit queries indicated by those suggestions and their submission causes the automated assistant to include context-relevant content in the message exchange thread. This can reduce and / or eliminate the need for the user to switch from an application presenting the message exchange thread to another application in order to satisfy certain information needs of the user. This is particularly useful for users with low dexterity or users who may otherwise have difficulty performing input to the device.

[0010] This can further reduce the use of certain computing resources that might otherwise be consumed in switching to another application to meet information needs. For example, switching to another application may require the use of processor, memory and / or battery resources via the launch and / or appearance of the application. In addition, switching to another application to obtain content from an automated assistant instead of obtaining content in a message exchange topic may increase the amount of time it takes for the user to obtain information-which may lead to a corresponding increase in the consumption of computing resources when obtaining content. In some embodiments, the techniques described herein are used to teach the user various types of inputs that can be provided by the user to the automated assistant in future ongoing message exchanges, such as when no suggestions are provided and / or the suggestions provided do not meet the user's information needs. In at least these cases, the optional suggestions included in the message exchange topic can have an effect outside the device displaying the suggestions-at least by causing data to be downloaded to the device from one or more external computers. The process can be made more reliable by optional suggestions, which ensures that the risk of erroneous data being downloaded to the device due to one or more erroneous user inputs (such as user selection of erroneous manual search results) is significantly reduced.

[0011] In some embodiments, a method performed by one or more processors is provided, the method comprising determining at least one candidate query based on one or more already communicated messages of an ongoing message exchange topic. The ongoing message exchange topic is at least between the first user and the second user via a first message exchange client of the first user and a second message exchange client of the second user. The method further comprises providing a selectable graphical element for display to at least the first user via the first message exchange client, the selectable graphical element providing an indication of the candidate query. The method further comprises, in response to a user interface input of the first user for the selectable graphical element, including content responsive to the candidate query in the ongoing message exchange topic. Including the content in the ongoing message exchange topic causes the content to be displayed to the first user via the first message exchange client and / or to the second user via the second message exchange client.

[0012] In some implementations, including the content in the ongoing message exchange topic includes requesting at least one of the message exchange clients to insert the content into a log of the ongoing message exchange topic displayed in a graphical user interface of at least one of the message exchange clients.

[0013] In some implementations, the content includes one or more of text and images.

[0014] In some implementations, determining the candidate query includes selecting the candidate query from the query corpus based on consistency between one or more query terms of the candidate query and one or more message terms of already communicated messages of the ongoing message exchange topic.

[0015] In some embodiments, determining the candidate query includes determining at least one entity referenced in a message that has been communicated in the ongoing message exchange topic, and determining the candidate query based on the at least one entity. In some of those embodiments, determining the candidate query based on the at least one entity includes generating one or more query terms for the candidate query based on an alias of the at least one entity.

[0016] In some implementations, determining candidate queries and providing selectable graphical elements occurs independently of the user's expression of any additional natural language input for inclusion in the message exchange thread.

[0017] In some embodiments, the method further includes: determining a score for the candidate query based on one or more terms in at least one of the messages that have been communicated in the ongoing message exchange topic. Providing a selectable graphical element can be based on the score satisfying a threshold. In some of those embodiments, determining the score based on the one or more terms includes determining the score based on whether a specified entity is included in the term and / or determining the score based on whether the multiple terms conform to at least one predefined search query pattern.

[0018] In some implementations, the method further includes determining a score for the candidate query based on: a popularity metric of the candidate query based on a number of submissions of the candidate query; and / or a quality metric of content responsive to the candidate query. Providing a selectable graphical element may be based on the score satisfying a threshold.

[0019] In some embodiments, the method further includes submitting the candidate query to a search engine; receiving one or more search results in response to submitting the candidate query to the search engine; and determining content for inclusion in the ongoing message exchange topic based on at least one of the search results. In some of those embodiments, submitting the candidate query to the search engine is in response to a user interface input to a selectable graphical element.

[0020] In some implementations, including the content in the ongoing message exchange topic includes including the content in the ongoing message exchange topic along with a graphical indication of the content being generated by the automated assistant.

[0021] In some embodiments, the method further includes providing an additional selectable graphical element for display to the first user via the first message exchange client, the additional selectable graphical element identifying one or more terms and, when selected, causing the one or more terms to be included in the message exchange topic along with an indication that the one or more terms originated from the first user, and causing the one or more terms to be included in the message exchange topic without including any content responsive to the one or more terms.

[0022] In some implementations, providing the graphical element further comprises providing the graphical element for display to the second user via the second message exchange client.

[0023] In some embodiments, the content includes a selectable first content item focused on a first entity and a selectable second content item focused on a second entity. In some of those embodiments, a user interface input to the first content item causes additional first entity content focused on the first entity to be included in the message exchange topic, and / or a user interface input to the second content item causes additional second entity content focused on the second entity to be included in the message exchange topic.

[0024] In some implementations, the content includes a first image or text focused on a first entity, and further includes at least one selectable deep link that, when selected, causes one or more additional computing actions specific to the first entity to be performed.

[0025] In some embodiments, the method further includes determining additional related queries based on the candidate queries and / or content, and upon a user interface input of the first user, providing an additional selectable graphical element for display to at least the first user via a first message exchange client, the additional selectable graphical element providing an indication of the additional related query. The additional selectable graphical element is provided for display without simultaneously displaying the graphical elements. The method may further include, in response to an additional user interface input of the first user for the additional selectable graphical element, including additional content responsive to the additional related query in an ongoing message exchange topic. Including the additional content in the ongoing message exchange topic causes the additional content to be displayed to the first user via the first message exchange client and / or to the second user via the second message exchange client.

[0026] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause performance of any of the above methods. Some embodiments include a non-transitory computer-readable storage medium storing computer instructions executable by the one or more processors to perform any of the above methods.

[0027] It should be appreciated that all combinations of the above concepts and additional concepts described in greater detail herein are understood to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are understood to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a diagram of an exemplary environment in which embodiments disclosed herein may be implemented.

[0029] Figure 2A An example of providing at least one context-relevant suggestion to one or more users of an ongoing message exchange topic between users is shown.

[0030] Figure 2B An example of including content responsive to a query in a message exchange topic in response to a user interface input by one of the users of the message exchange topic, such as a user interface input for a suggestion provided to the user and indicative of the query, is shown.

[0031] Figure 3A , 3B , 3C, and 3D illustrate examples of how a client computing device may present a graphical user interface for a message exchange client according to various embodiments.

[0032] Figure 4 A flow chart illustrating an exemplary method of providing at least one context-relevant suggestion to one or more users of an ongoing message exchange thread between users and including content responsive to a query indicated by the suggestion in the message exchange thread.

[0033] Figure 5 An exemplary architecture of a computing device is shown. DETAILED DESCRIPTION

[0034] Embodiments of the present specification are directed to providing at least one context-related suggestion to one or more users of an ongoing message exchange topic between users. The suggestion is provided for presentation to the user via a user interface output device of the user's computing device. The suggestion indicates a query that can be submitted to an automated assistant (e.g., submitted within a message exchange topic) so that the automated assistant includes content responsive to the query in the message exchange topic. In some embodiments, the suggestion is a selectable suggestion (e.g., a selectable graphical element is provided for display) and the content responsive to the query is included in the message exchange topic in response to a user interface input (e.g., "tap", "click", "verbal command") for the selectable suggestion. In some embodiments, the suggestion is determined based on one or more messages that have been communicated between users of the message exchange topic. For example, the query indicated by the suggestion can be determined based on one or more terms, images, and / or other content of messages that have been communicated.

[0035] Now go to Figure 1 , showing an exemplary environment in which the techniques described herein may be implemented. The exemplary environment includes a plurality of client computing devices 106 1-N and automated assistant 120. Although Figure 1 Automated assistant 120 is shown as being associated with client computing device 106 1-N separate, but in some embodiments, can be provided by the client computing device 106 1-N 120. For example, client computing device 1061 may implement an instance of one or more aspects of automated assistant 120, and client computing device 106 N A separate instance of one or more aspects of automated assistant 120 may also be implemented. 1-N In embodiments where one or more (eg, all) aspects of automated assistant 120 are implemented by one or more computing devices, client computing device 106 1-N Those aspects of automated assistant 120 may communicate via one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (eg, the Internet).

[0036] Client computing device 106 1-NIt may include, for example, one or more of the following: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0037] Client computing device 106 1-N Each of the client computing device 106 and the automated assistant 120 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components to facilitate communication over the network. 1-N One or more of the operations performed by the automated assistant 120 and / or the automated assistant 120 may be distributed across multiple computer systems. The automated assistant 120 may be implemented as a computer program running on one or more computers in one or more locations, for example, coupled to each other via a network.

[0038] In some implementations, a given user may communicate with automated assistant 120 using multiple client computing devices of the user that collectively form a coordinated "ecosystem" of computing devices. For example, a user may join an ongoing message thread via multiple computing devices of the user. For example, a user may provide a first message for inclusion in a message thread via user interface input at a first computing device (e.g., a desktop computing device), and may later provide a second message for inclusion in the message thread via user interface input at a second computing device (e.g., a mobile computing device). However, for simplicity, many of the examples described in this disclosure will focus on operating client computing device 106. 1-N A single given user.

[0039] Client computing device 106 1-N Each of which can operate multiple different applications, such as message exchange client 107 1-N A corresponding one of the message exchange client 107 1-N may appear in various forms and the forms may be on the client computing device 106 1-N Changes between and / or multiple forms can be made on the client computing device 106 1-N In some embodiments, the message exchange client 107 1-NOne or more of the message exchange client 107 may be in the form of a short messaging service ("SMS") / multimedia messaging service ("MMS") client, an online chat client (e.g., instant messenger, Internet relay chat or "IRC"), etc., a messaging application associated with a social network, etc. In some embodiments, the message exchange client 107 may be implemented via a web page or other resource presented by a web browser (not shown) or other application of the client computing device. 1-N One or more of .

[0040] The automated assistant 120 checks the multiple message exchange clients 107 via the multiple users. 1-N One or more messages in an ongoing message exchange topic between the multiple users that occurred. In some embodiments, the automated assistant 120 serves as a central exchange to facilitate the message exchange topic between the users. In other embodiments, the automated assistant 120 may not serve as a central exchange, but still continuously, periodically, or at other regular and / or irregular intervals to check for messages in the ongoing message exchange. For example, the automated assistant 120 may be implemented in whole or in part on one or more client computing devices of the users participating in the ongoing message exchange, and check the message exchange via the client computing devices. Also, for example, the automated assistant 120 may be implemented in whole or in part on one or more computing devices that are remote from the client computing devices of the users participating in the ongoing message exchange - but may receive messages of the ongoing message exchange from one or more of the client computing devices and / or from another computing device that facilitates the message exchange.

[0041] In some implementations, automated assistant 120 determines at least one context-relevant suggestion based on one or more of the examined messages to provide via message exchange client 107. 1-N Presentation to one or more of those users (eg, for display as a graphical element).

[0042] The suggestion indication determined by the automated assistant may be communicated to the client 107 via the message exchange client 107. 1-NIn some embodiments, the content in response to the query indicated by the suggestion can be submitted to the automated assistant 120, the automated assistant 120 can initiate a search based on the query, and the automated assistant 120 can determine the content based on one or more search results of the search. In some other embodiments, the content in response to the search can be previously determined and can be implemented as a message exchange topic by the automated assistant 120 in response to receiving the query, or as other data in response to the user interface input for the suggestion. In other embodiments, the user can type or speak the query indicated by the suggestion and submit the query to the automated assistant 120 without explicitly directing the user interface input to the suggestion. In some of those embodiments, the automated assistant 120 can also respond by including the content in the message exchange topic.

[0043] In some implementations, automated assistant 120 can provide suggestions for presentation to a user independent of one or more user interface inputs provided by the user. For example, in expressing messages for possible inclusion in a message exchange topic, the suggestions can be provided independent of any natural language user interface inputs provided (e.g., typed or spoken) by the user. For example, in some implementations, the suggestions can be determined based at least in part on a given message being the most recent message in the message exchange topic, and can be presented to the one or more users before the one or more users provide any user interface input for expressing additional messages that, if submitted to the message exchange, would become the most recent messages.

[0044] In some implementations, the suggestion may be provided automatically and independently of any user interface input. In some other implementations, the suggestion may be provided in response to certain user interface inputs, such as user selection of a user interface element, a spoken command provided by a user, etc. For example, the suggestion may be provided in response to a user selecting a user interface element provided by the messaging client 107. 1-N The suggestion may be provided by "tapping" or selecting a "reply area" in a graphical user interface presented in one of the embodiments. As another example, the suggestion may be provided in response to a user interaction with a virtual button (e.g., tapping, long pressing), a verbal command (e.g., "Hey personal assistant, give me some suggestions", etc.).

[0045] In some implementations, whether and / or how automated assistant 120 provides a suggestion can be based on a score for the query indicated by the suggestion. For example, if the score for the query meets a threshold, automated assistant 120 can automatically provide the suggestion for presentation, but if the score does not meet the threshold, user interface input may be required before providing the suggestion. Also, for example, automated assistant 120 can: automatically provide the suggestion for presentation if the score meets a first threshold; require certain user interface input before providing the suggestion for presentation if the score does not meet the first threshold but meets a second threshold; and in any case, no suggestion may be provided if the score does not meet the first threshold or the second threshold. Additional descriptions of implementations for determining a score for a query are provided herein.

[0046] In some implementations, automated assistant 120 may include a message processing engine 122, an entity determination engine 124, a suggestion engine 126, a content engine 128, and a message exchange engine 130. In some implementations, one or more of engines 122, 124, 126, 128, and / or 130 may be omitted. In some implementations, all or aspects of one or more of engines 122, 124, 126, 128, and / or 130 may be combined. In some implementations, one or more of engines 122, 124, 126, 128, and / or 130 may be implemented in a component separate from automated assistant 120. For example, the engine 126 may be implemented in a component that is executed by client computing device 106. 1-N One or more of engines 122, 124, 126, 128, and / or 130, or any operational portion thereof, are implemented in an executed component.

[0047] Message processing engine 122 processes one or more messages of a message exchange topic and generates annotated output for use by one or more other components of automated assistant 120. For example, message processing engine 122 may process free-form natural language input expressed by a user via one or more user interface input devices of client computing device 1061 and submitted to the message exchange topic by message exchange client 1071. The generated annotated output of the message includes one or more annotations of the content of the message, and optionally includes at least some of the content itself.

[0048] In some embodiments, the message processing engine 122 is configured to recognize and annotate various grammatical information in the natural language content of the message. For example, the message processing engine 122 may include a portion of speech tags configured to annotate terms by their grammatical solution. For example, a portion of the speech tags may label each term with its part of speech, such as "noun", "verb", "adjective", "pronoun", etc. Moreover, for example, in some embodiments, the message processing engine 122 may additionally and / or alternatively include a relevance analyzer that is configured to determine syntactic relationships between terms in the natural language input. For example, the relevance analyzer can determine which terms modify which terms, subjects, verbs, etc. of a sentence (e.g., a parse tree) - and annotations to these relevances can be performed.

[0049] In some embodiments, the message processing engine 122 may additionally and / or alternatively include an entity tagger configured to annotate entity references, such as references to people, organizations, locations, etc., in the content of the message (e.g., natural language content and / or images). The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to an entity category such as a person) and / or at a lower level of granularity (e.g., to enable identification of references to a specific entity such as a specific person). The entity tagger may rely on the content itself to resolve specific entities and / or may optionally communicate with one or more resources to resolve specific entities. For example, to resolve entities in natural language content, the entity tagger may communicate with a knowledge graph or other entity database to resolve specific entities. Also, for example, to resolve entities in image content, the entity tagger may communicate with an image processing system, such as a convolutional neural network configured to identify one or more entities (e.g., entity categories) present in an image.

[0050] In some embodiments, the message processing engine 122 may additionally and / or alternatively include a cross-reference resolver that is configured to group, or "cluster," references to the same entity based on one or more contextual clues. For example, the cross-reference resolver may be used to resolve the term "it" into "restaurant A" in the natural language input "I think restaurant A sounds good, where is it?"

[0051] In some embodiments, one or more components of the message processing engine 122 may rely on annotations from one or more other components of the message processing engine 122. For example, in some embodiments, a designated entity tagger may depend on annotations from a cross-reference resolver and / or a relevance analyzer in annotating all records of a particular entity. Also, for example, in some embodiments, a cross-reference resolver may rely on annotations from a relevance analyzer when grouping references to the same entity. In some embodiments, in processing a particular message, one or more components of the message processing engine 122 may use one or more previous messages of a message exchange topic to determine one or more annotations for a given message of the message exchange topic. For example, a first user in a message exchange topic may include a message "How does restaurant A sound?" in the topic, and a second user may include a response message "Where is it?" in the message exchange topic. In processing "Where is it," the cross-reference resolver may resolve "it" to "Restaurant A" based on the previous message "How does restaurant A sound?".

[0052] The entity determination engine 124 determines one or more entities referenced in one or more messages of a message exchange topic. In some implementations, the entity determination engine 124 may determine one or more entities that are "recently" referenced in a message exchange topic based on an analysis of a subset of the messages of the message exchange topic. For example, the engine 124 may determine recently referenced entities based on the latest N messages (e.g., the last 1, 2, or 4 messages) and / or based on messages that have been included in the message exchange topic within a time threshold (e.g., within the last 1 minute, the last 10 minutes).

[0053] In determining entities referenced in one or more messages, entity determination engine 124 may rely on the content of the messages and / or annotations provided by message processing engine 122 (e.g., entity tags provided by an entity tagger of engine 122). Entity determination engine 124 may also utilize one or more additional resources in determining entities, such as a knowledge graph or other entity database and / or user attributes of one or more users referencing the topic of the message exchange.

[0054] As an example, assume the latest message "What's the weather like then" and the immediately preceding message "Are you interested in the Louisville Open in May?" Based on these two messages, the entity determination engine 124 can determine the entity corresponding to the city of Louisville, Kentucky based on, for example, an entity in the knowledge base with an "alias" attribute of "Louisville" for the entity, and based on an entity with a strong relationship to an entity for an entity associated with "the Open". The entity determination engine 124 can also optionally determine the entity corresponding to May based on the occurrence of the term "May" in the message and / or its syntactic relationship in the message (e.g., based on annotations provided by the engine 122). The entity determination engine 124 can also optionally determine the entity corresponding to "weather" based on the occurrence of the term in the message. In some embodiments, the entity determination engine 124 can be omitted.

[0055] The suggestion engine 126 determines one or more suggestions related to one or more message contexts based on one or more messages of the message exchange topic. The suggestion engine 126 may provide one or more of the determined suggestions for presentation (e.g., for display as a graphical element) to one or more of the users participating in the message exchange topic (e.g., via the message exchange client 107). 1-N For example, the suggestion may be provided as a graphical element that can be viewed by one or more of the users who joined the topic. For example, the graphical element may be displayed in a dialog box with the message exchange client 107. 1-N The message exchange topic is displayed on one or more associated graphical user interfaces in the message exchange topic, and can be displayed together with (but optionally not included in) a transcript of the ongoing message exchange topic. As used herein, a "transcript" can refer to a live record of text, images, sounds, etc. exchanged and / or viewed by one or more users participating in the message exchange topic. In some embodiments, the transcript of the message exchange topic can be updated in real time or near real time while the users are engaged in the conversation.

[0056] In some implementations, the suggestion engine 126 may submit the suggestion to the message exchange engine 130, which may then provide the suggestion to the message exchange client 107. 1-N For example, the message exchange engine 130 may request the message exchange client 107 1-N At least one of the message exchange clients 107 1-N A graphical element is provided in at least one presented graphical user interface.

[0057] In some implementations, the suggestion engine 126 determines suggestions based on the annotated output determined by the message processing engine 122, based on one or more entities determined by the entity determination engine 124, and / or based on one or more queries of the query database 156. The query database 156 may include, for example, one or more queries previously submitted to an automated assistant and / or a search engine and / or one or more query "templates." A query template may include one or more terms and one or more placeholders that may be "filled in" with aliases for specific entities to generate a query. For example, a query template may take the form of "actors in [movie]," and "movie" may be filled in with aliases for specific movies to generate a query based on the template.

[0058] In some implementations, the suggestion engine 126 determines candidate search queries based on one or more messages in the message exchange topic, and generates suggestions based on the candidate search queries. For example, the suggestion can be a selectable graphical element that provides an indication of the candidate search query. The graphical element can include the entire search query or a subset of terms of the search query or a rewrite of the search query (e.g., when the search query is longer than expected to be displayed).

[0059] The suggestion engine 126 can utilize various techniques to determine candidate search queries. In some embodiments, the suggestion engine 126 can determine candidate search queries based on one or more entities determined by the entity determination engine 124. For example, the suggestion engine 126 can determine one or more search queries that are strongly associated with one or more entities in one or more databases (e.g., in the query database 156). Moreover, for example, the suggestion engine 126 can determine one or more search queries based on aliases of one or more entities. For example, as described above, the entity determination engine 124 can determine an entity corresponding to the city of Louisville, Kentucky based on the message "Are you interested in the Louisville Open in May?" after the message "What's the weather like then?" The suggestion engine 126 can determine the candidate query "Louisville, KY, May weather" based on the aliases of the entities determined by the engine 124 ("Louisville, KY"), based on the terms of the message ("weather", "May"), and based on the annotated output provided by the message processing engine 122 (e.g., determining that "then" refers to "May").

[0060] In some implementations, the suggestion engine 126 can additionally and / or alternatively determine the candidate search query based on selecting it from a corpus of previously submitted search queries, such as queries in the query database 156. For example, the candidate search query can be selected based on consistency between the candidate search query and terms included in the messages of the message exchange topic, consistency between the candidate query and aliases of entities determined by the entity determination engine 124, and / or based on a popularity metric of the search query (e.g., based on a frequency of submission of the candidate query or other metric indicating a number of submissions of the candidate query).

[0061] In some implementations, the suggestion engine 126 can additionally and / or alternatively determine candidate search queries based on identifying query templates associated with one or more terms and / or entities of the message (e.g., determined by the entity determination engine 124) in the query database 156. The suggestion engine 126 can further generate candidate queries by filling in the placeholders of the query template with one or more appropriate terms. For example, assume that the query template of "nearby [type of cuisine] restaurants" is associated with one or more entities such as the entity category "restaurant", the entity category "type of cuisine", and / or is associated with one or more terms such as "food", "dinner", "reservation", "restaurant", "seafood", "barbecue", etc. The suggestion engine 126 can identify the query template based on a message exchange including the message "Anyone wanna go to lunch? - I want to eat BBQ". The suggestion engine 126 can further fill in the "[type of cuisine]" placeholder with "barbecue" based on the fact that BBQ is a type of cuisine and is a term included in the message.

[0062] In some implementations, whether the suggestion engine 126 provides a suggestion may be based on the score of the query indicated by the suggestion. For example, in some implementations, the suggestion engine 126 may determine one or more groups of candidate queries using one or more techniques such as those described herein. In some implementations, the suggestion engine 126 may determine a score for each of the group of candidate queries and provide suggestions based only on those candidate queries (if any) that have scores that satisfy a threshold.

[0063] In some embodiments, the suggestion engine 126 may additionally and / or alternatively utilize the score of the candidate query to determine how to provide suggestions indicating the candidate query. For example, in some embodiments, the suggestion engine 126 may use the score of the candidate query to determine the display order, size, and / or other visual prominence of the suggestions provided as graphical elements indicating the candidate query. Moreover, for example, in some embodiments, if the score for the candidate query meets the threshold, the suggestion engine 126 may automatically provide the suggestion for presentation, but if the score does not meet the threshold, user interface input may be required before providing the suggestion. Moreover, for example, the suggestion engine 126 may: automatically provide the suggestion for presentation if the score meets the first threshold; require certain user interface input before providing the suggestion for presentation if the score does not meet the first threshold but meets the second threshold; and if the score does not meet the first threshold or the second threshold, no suggestion is provided in any case.

[0064] In some embodiments, the suggestion engine 126 determines an analysis of the candidate query based on: (1) one or more attributes of the candidate query itself; (2) one or more attributes of a search structure responsive to the candidate query; and / or (3) one or more attributes of one or more messages of an ongoing message exchange topic. The one or more attributes of the candidate query may include, for example, a popularity metric for the candidate query. The one or more attributes of one or more search results responsive to the candidate query may include, for example, a quality metric for the search results and / or whether the search results are of a certain type, such as a navigation search result, a "curated" search result, a "knowledge card" search result.

[0065] One or more attributes of one or more messages in an ongoing message exchange topic may include, for example, a measure of the "chatty" and / or the "searchability" of one or more messages, such as the latest message or the most recent N messages. In some embodiments, the measure of the "searchability" of a message may be determined based on a comparison of the terms and / or syntactic structure of the message with the terms and / or syntactic patterns of a previously submitted search query. For example, the syntactic structure of the message may be compared with a common syntactic structure in a search submitted to a search engine, and if there is a high degree of consistency, it may be indicated that the message is "searchy". Moreover, for example, the presence and / or absence of certain terms in the message may indicate that the message is "searchy", such as the presence of "what", "when", "?", etc. Moreover, for example, the presence of any specified entity, the presence of a certain amount of specified entities, and / or the presence of certain categories of specified entities in the message may indicate that the message is "searchy".

[0066] In some implementations, a measure of "chatty" of a message may be determined based on comparing the lexical terms and / or syntactic structure of the message to the lexical terms and / or syntactic patterns of previous message exchanges between two or more users. For example, the syntactic structure of the message may be compared to syntactic structures common in previous message exchanges between the users, and if there is a high degree of consistency, it may be indicated that the message is "chatty." Also, for example, the presence and / or absence of certain lexical terms in the message may indicate that the message is "chatty." Also, for example, the absence of any specified entities and / or certain classes of specified entities in the message may indicate that the message is "chatty."

[0067] In some implementations, the suggestion engine 126 may utilize a trained classifier and / or other machine learning system to determine measures of "chatty" and / or "searchable." For example, a classifier may be trained to receive as input one or more features of one or more messages, such as terms of the message, annotations of the message (e.g., provided by the engine 122), etc.—and provide as output an indication of whether and / or to what extent the message is "queryable" (i.e., indicating a desire for search results) and / or "chatty" (i.e., indicating a desire for further communication with one or more users).

[0068] Content engine 128 determines content responsive to a query to automated assistant 120, such as content responsive to a query to automated assistant 120 in response to a user selecting or otherwise interacting with suggestions based on the query. The content may include, for example, documents, images, textual content, and / or other information responsive to the query, and may be identified from, for example, public databases 154 and / or personal databases 152. In various implementations, the content responsive to the search query may further include further suggestions related to other determined content.

[0069] In some embodiments, the content engine 128 may submit the query to a separate search query (not shown) and obtain one or more search results in response to the submission. In some embodiments, the search engine may search one or more public content resources, such as a public database 154. In some embodiments, the search engine may additionally and / or alternatively search one or more personal content resources of the user (and / or other users of the message exchange topic) who submitted the query, such as a personal database 152. For example, the personal database 152 may include entries of the user personally, such as the user's calendar entries, the user's contact entries, the user's email entries, content derived from the user's email, etc. The search engine may access the personal database 152 (or a subset of entries of the user's personal database 152) based on, for example, user authorization information provided by the content engine 128 with the query submission. In some other embodiments, the content engine 128 itself may perform a search of the public database 154, the personal database 152, and / or other resources based on the query.

[0070] In many instances, the content in response to the query may be too unwieldy (e.g., too long, too detailed, etc.) to be conveniently included in the message exchange topic without interrupting the flow of the conversation. In these cases, the content engine 128 may select a subset of the entire response content (e.g., the entire response content may be a selection) to include in the topic. In some embodiments, the content engine 128 may select a subset (e.g., a selection of the response content) for inclusion in the message exchange topic. In other embodiments, the content engine 128 may provide the comprehensive response information to the message exchange client 107. 1-N One or more of, and it can perform the selection. For example, in some embodiments, the message exchange client 1071 can select information based on the screen size of the client computing device 1061. In other embodiments, both the content engine 128 and the message exchange topic can perform the selection. In some embodiments, the top N search results (or other content) can be selected for inclusion in the message exchange topic and / or certain types of search results (or other content) can be selected for inclusion in the message exchange topic. N can be selected based on, for example, screen size, user preferences, etc., or can be manually selected (for example, N can be set to 5, 10, 2, 1, etc.).

[0071] In various implementations, at least some of the content included in the message exchange topic in response to the query can be linked to one or more electronic documents responsive to the query, such that the user need only select content for a new resource (e.g., a web browser that displays information). In some implementations, at least some of the content included in the message exchange topic can be linked to an application on a client computing device 106 that has at least one of the operating message exchange clients 107 installed that is responsive to the query. For example, if the user selects to indicate a query for "dinner reservations at Franks" suggestions, one or more graphical elements associated with the location "Franks" can be included in the message exchange topic, and those graphical elements can be selected to open another application that is specifically operated to create a reservation at the location Franks (e.g., a deep link).

[0072] Message exchange engine 130 may facilitate inclusion of content (determined by content engine 128) in message exchange topics and / or may facilitate providing suggestions (determined by suggestion engine 126) for display via one or more message exchange topics.

[0073] The content contained in the message exchange topic can be viewed by one or more users who have joined the topic. For example, the content can be viewed on a graphical user interface associated with a message exchange client that illustrates a log of the ongoing message exchange topic. The message exchange engine 130 can include the content in multiple message exchange clients 107 in various ways. 1-N In some embodiments, the message exchange engine 130 may request the message exchange client 107 to join the message exchange topic. 1-N At least one of the message exchange clients inserts the content into a logbook of the ongoing message exchange topic displayed in the graphical user interface. In some cases, the content can be included in a logbook presented by more than one message exchange client, such as a logbook presented by all message exchange clients that have joined the message exchange topic. In some embodiments, the message exchange engine 130 can simply provide the response content to the message exchange client, and the message exchange topic can then select which content to include in the message exchange topic.

[0074] Now go to Figure 2A and 2B , provides additional descriptions of various components of automated assistant 120.

[0075] exist Figure 2A and 2B In the message exchange topic, the message exchange client 1071 and the message exchange client 107 N is ongoing. Figure 2AIn the example, the message exchange engine 130 obtains one or more messages of the topic message 201 and provides those messages to the message processing engine 122. In some embodiments, the message exchange engine 130 can be used as a convenient message exchange client 1071 and 107 N In other embodiments, the message exchange engine 130 may not be used as a central exchange, but the message 201 of the topic can still be obtained.

[0076] The message processing engine 122 processes the topic's message 201 and generates various annotations of the topic's message 201. The message processing engine 122 provides the annotated input (terms and / or other content of the topic's message 201 and the generated annotations) 202 to the entity determination engine 124 and the suggestion engine 126.

[0077] The entity determination engine 124 determines one or more entities 203 referenced in the topic's message 201 based on the annotated input 202 , an entity database, and / or other resources. The entity determination engine 124 provides the one or more entities 203 to the suggestion engine 126 .

[0078] The suggestion engine 126 determines one or more suggestions 204 and provides the suggestions 204 to the message exchange engine 130. The message exchange engine 130 provides the suggestions 204 for use via the message exchange clients 1071 and 107 N In some implementations, one or more of suggestions 204 can be selectable graphical elements that include terms of the candidate search query and / or otherwise indicate the candidate search query. In some implementations, one or more of suggestions 204 can be provided as audible suggestions.

[0079] As described herein, the suggestion engine 126 can utilize one or more of a variety of techniques to determine suggestions. For example, the suggestion engine 126 can determine candidate search queries based on the annotated input 202, the entity 203, and / or the query database 156, and can generate suggestions based on the candidate search queries. In some embodiments, the suggestion engine 126 determines a score for the candidate query associated with the suggestion 204, and determines whether to provide the suggestion 204 based on the score. In some embodiments, the suggestion engine 126 utilizes the score to determine how the suggestion 204 will be provided. For example, the suggestion engine can utilize the score to generate computer-readable instructions to provide the suggestion 204 indicating a specific display position of the suggestion 204, a specific display size of the suggestion 204, and / or other specific display attributes of the suggestion 204.

[0080] Figure 2BAn example is shown of including content responsive to a query in a message exchange topic in response to user interface input by a user of the message exchange topic, such as user interface input for a suggestion provided to the user and indicative of the query.

[0081] For example, in Figure 2B In the message exchange engine 130, the message exchange engine 130 may respond to the Figure 2A 205 by receiving a user interface input of one of the provided suggestions 204 (ie, the suggestion may indicate the query 205). Figure 2A As described above, suggestions may be provided to clients 1071 and 107 via message exchange. N The client 1071 and the client 1072 may be displayed in response to the message exchange client 1071 and the client 1073 providing the suggestion. N , or any one of the user interfaces input for the suggestion, to provide query 205 to message exchange engine 130 .

[0082] The message exchange engine 130 provides the query 205 to the content engine 128, and the content engine 128 obtains content 206 in response to the query 205. The content 206 can be obtained directly or indirectly by the content engine 128 from the personal database 152 and / or the public database 154. In some embodiments, the content engine 128 determines a set of search results or other content in response to the submission of the query 205, and selects the content for generating the content 206. The content engine 128 provides the content to the message exchange engine 130. The message exchange engine 130 includes the content 206 in the message exchange clients 1071 and 107 N For example, content 206 may be included in a message exchange topic for display as a message exchange topic between message exchange clients 1071 and 107 N The two are part of the topic of the message exchange.

[0083] Now go to Figures 3A-3D , provides additional description of various components and techniques described herein. Figures 3A-3D Each of the Figure 1 1, wherein display screen 140 illustrates a diagram of how client computing device 1061 may render a graphical user interface of client computing device 1061 during a message exchange topic. Figures 3A-3D , a user of client computing device 1061 is involved in an ongoing message exchange topic with two other users, "Tom" and "Alex".

[0084] Figures 3A-3DDisplay screen 140 further includes a text reply interface element 388, which a user can select to generate user input via a virtual keyboard, and a voice reply interface element 389, which a user can select to generate user input via a microphone. Display screen 140 also includes system interface elements 381, 382, ​​383, which can interact with the user to cause computing device 1061 to perform one or more actions.

[0085] exist Figure 3A , user has previously expressed natural language message 380A and submitted message 380A for inclusion in the message exchange topic. In addition, Tom has previously expressed natural language message 382A and submitted message 382 for inclusion in the message exchange topic after including message 380A. Alex has previously expressed natural language message 384A and submitted message 384A for inclusion in the message exchange topic after including message 382A.

[0086] exist Figure 3A Also shown in the figure is a selectable graphical element 361 indicating a candidate query "Acme cast" that can be submitted to the automated assistant 120. Graphical element 361 includes the text of the candidate query and also includes an "asterisk" to indicate to the user that selection of graphical element 361 will result in content responsive to the candidate query "Acme cast" being included in the message exchange topic. Also shown is a selectable graphical element 371 including the text "No". Selection of graphical element 371 will result in "No" being automatically populated in text reply interface element 388 (which the user can then submit for inclusion in the topic via one or more user interface inputs) or will result in "No" being automatically submitted for inclusion in the message exchange topic. In either case, "No" will be included in the message exchange topic as a reply from the user without causing content responsive to "No" to also be included. In other words, selection of graphical element 371 can enable the user to generate a "No" reply as if the user had typed "No" in text reply interface element 388 and / or spoken "No" after selecting voice reply interface element 385.

[0087] In some implementations, automated assistant 120 can determine the candidate search query "Acmet cast" indicated by graphical element 361 based on one or more of the communicated messages 380A, 382A, and 384A of the message exchange topic. For example, the candidate search query can be determined based on the occurrence of the term "Acme" (the name of a fictional movie) in message 382A from Tom and based on the occurrence of the term "cast" in message 384A from Alex.

[0088] It should be noted that in some embodiments, Tom's message exchange client may also display graphical element 371 and / or graphical element 361 for selection by Tom. Moreover, Alex's message exchange client may also display graphical element 371 for selection by Alex (graphic element 371 may not be displayed to Alex because "no" is not an appropriate response for Alex in the context of his message 384A).

[0089] Figure 3B Shows that the user has provided Figure 3A The graphical user interface after the user interface input of the graphical element 361 in the graphical user interface. For example, the user can "tap" the graphical element 361. Figure 3B , the user selection of graphical element 361, along with an asterisk indicating that message 380B is provided to obtain content responsive to "Acme cast" for inclusion in the message exchange topic, is included in the message exchange topic as a message 380B from the user that includes the text of the candidate query "Acme cast." For example, the message exchange client of computing device 1061 can include message 380B by submitting "Acme cast" and an indication that causes automated assistant 120 to process "Acme cast" as a query for automated assistant 120. The indication can be a human-recognizable indication and / or a human-non-recognizable indication. Note that in some implementations, Tom and / or Alex's message exchange client can also display message 380B (along with an indication of the user's name instead of "you").

[0090] exist Figure 3B, content 390A is also included in the message exchange topic as a message from the "automated assistant". Content 390A includes the text "I found:" to indicate that the content is in response to message 380B and includes three "business cards" in response to the query "Acme cast" and a selectable option "...". The business cards are three cast members of "Acme Movies" and the selectable option ("...") enables the user to see other business cards for other cast members. Each of the business cards includes an image of the cast member, the "known" name of the cast member, and the name of the cast member in "Acme Movies" (shown in italics). Note that in some embodiments, Tom and / or Alex's message exchange clients may also display content 390A. In some of those embodiments, a user interface input by any of the users for the selectable option ("...") for seeing other business cards for other cast members may cause the other business cards to be displayed in the graphical user interfaces of all message exchange clients, or only in the client of the user who provided the user interface input. In some embodiments, one or more business cards may be selected by the user to cause other content about the cast member of that business card to be provided. For example, the selection of one of the business cards may cause the other content to be included in a message exchange topic for the user and optionally for other users. Also, for example, the selection of one of the business cards may cause the other content to be provided to the user in another application (such as a separate web browser application) separate from the message exchange client.

[0091] Figure 3B Also shown in FIG. 1 are selectable graphical elements 362 and 363, and indicate candidate queries of "Acme trailers" and "Acme showtimes" that may be submitted to automated assistant 120. Each of graphical elements 362 and 363 includes text indicating its candidate query (but not the entirety of the text of the candidate query) and also includes an "asterisk" indicating to the user that selection will result in content responsive to its candidate query being included in the message exchange topic. As shown in graphical element 361 ( Figure 3A ), Tom and / or Alex's messaging client may also display graphical elements 362 and 363. In some implementations, automated assistant 120 may determine candidate queries for graphical elements 362 and 363 based on those candidate queries being related to candidate query "Acme movies" and / or being related to content 390A. For example, automated assistant 120 may determine those candidate queries based on a database entry indicating that those candidate queries are the two most frequently submitted queries by other users after and / or before submitting "Acme showtimes."

[0092] Figure 3CThe graphical user interface is shown after Alex provides a user interface input (e.g., a "tap") to an instance of graphical element 363 displayed in his graphical user interface. Figure 3C , Alex's selection of graphical element 363, along with an asterisk indicating that message 384B is provided to obtain content responsive to "Acme cast" for inclusion in the message exchange topic, is included in the message exchange topic as message 384B from Alex including the text of the candidate query "Acme showtimes."

[0093] exist Figure 3C , content 390B is also included in the message exchange topic as a message from the "automated assistant". Content 390B includes the text "I found:" to indicate that the content is responsive to message 384B and includes natural language output indicating the showtimes of "Acme Movies" at two theaters. Note that in Figure 3C , two movie theaters are located close to Tom, Alex, and the user—and the natural language output of content 390B includes an indication of the distance of Tom, Alex, and the user to each movie theater. For example, automated assistant 120 can determine content based on a search of location information and / or other user attributes of each of the users who initiated the message exchange topic, determining content 390B that is contextually relevant to the user's location and / or other attributes of the user that are not discernible from the messages of the message exchange topic.

[0094] Selectable graphic elements 362 and 365 are still Figure 3C , and indicates candidate queries of "Acme trailers" and "Acme stills" that may be submitted to automated assistant 120. Each of graphical elements 362 and 365 includes text indicating its candidate query, and each of them also includes a "star" indicating to the user that selection will result in content responsive to its candidate query being included in the message exchange topic. In some implementations, automated assistant 120 may determine candidate queries for graphical elements 362 and 365 based on those candidate queries related to candidate query "Acme showtimes" and / or related to content 390B.

[0095] exist Figure 3C , the natural language input "We go to the cinema 1@7" is shown in the text reply interface element 388 and can be expressed by the user. For example, after including content 390B in the message exchange topic, the user can provide natural language input using a virtual keyboard.

[0096] Figure 3D The graphical user interface is shown after the user has submitted the natural language input "Let's go to the cinema 1@7" into the message exchange topic. Figure 3D, the user submits natural language input to be included in the message exchange topic as message 380C from the user, message 380C does not include an asterisk indicating that the message is from the user, and is not intended to obtain response content from the automated assistant. Note that in many embodiments, Tom and / or Alex's message exchange client also displays message 380C (with an indication of the user's name instead of "you").

[0097] Still Figure 3D , a selectable graphical element 366 indicating a candidate query "directions to theater 1" that may be submitted to automated assistant 120 is shown. Graphical element 366 includes the text of the candidate query and also includes an "asterisk" for indicating to the user that selection of graphical element 366 will result in inclusion of content responsive to the candidate query "directions to theater 1" in the message exchange topic.

[0098] In some implementations, automated assistant 120 may determine the candidate search query "directions to theater 1" indicated by graphical element 366 based on one or more of the already communicated messages, such as message 380C, and / or based on content 390B.

[0099] Despite Figures 3A-3D 38, but in some embodiments, the user of the message exchange topic and / or other users can explicitly submit a query to the automated assistant 120 without selecting a provided suggestion. For example, the user can express natural language input in the text reply interface element 388, and include a specific term and / or symbol (e.g., @automatedassistant as a prefix) in conjunction with the input so that the input is submitted to the automated assistant 120 (e.g., shown as a message with "asterisks" replied with content responsive to the input). Also, for example, the user can verbally express the natural language input after selecting the voice reply interface element 389, and can speak a specific term in conjunction with the input so that the input is submitted to the automated assistant 120. As another example, a separate graphical interface element can be provided in the graphical user interface that, when selected, causes the natural language input expressed by the user to be submitted to the automated assistant 120.

[0100] Figure 4The diagram illustrates a flow chart of an exemplary method 400 for providing at least one context-related suggestion to one or more users in an ongoing message exchange topic between users and including content responsive to a query indicated by the suggestion in the message exchange topic. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. The system may include various components of various computer systems, such as automated assistant 120. In addition, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0101] At block 450, the system examines ongoing message exchange topics between multiple users operating multiple message exchange clients.

[0102] At block 452, the system determines one or more candidate queries based on one or more messages of the message exchange topic.

[0103] At block 454 , the system determines a score for each of the candidate queries.

[0104] At 456, the system determines for each of the candidate queries whether the score of the candidate query satisfies the threshold. If the system determines at block 456 that none of the candidate queries has a score that satisfies the threshold, the system proceeds to block 458 and waits for another message in the message exchange topic. In response to another message in the message exchange topic, the system again proceeds to block 450.

[0105] If, at block 456 , the system determines that at least one of the candidate queries has a score that satisfies the threshold, the system proceeds to block 460 .

[0106] At block 460 , the system provides one or more suggestions for one or more candidate queries whose scores satisfy the threshold for presentation to at least one of the users of the message exchange topic.

[0107] At block 462 , the system identifies that an interface input occurs for one of the provided suggestions.

[0108] At block 464, the system determines the content of the suggested candidate query for which the user interface input is responsive.

[0109] At block 466, the system includes the content in the message exchange topic and proceeds to block 458. At block 458, the system waits for another message in the message exchange, and in response to another message in the message exchange topic, the system proceeds to block 450 again.

[0110] Figure 5is a block diagram of an exemplary computing device 510 that may optionally be used to perform one or more aspects of the techniques described herein. 1-N , automated assistant 120 , and / or one or more of the other components may include one or more components of example computing device 510 .

[0111] The computing device 510 generally includes at least one processor 514, which communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, for example, including a memory subsystem 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface for an external network and is coupled to corresponding interface devices in other computing devices.

[0112] User interface input device 522 may include a keyboard, a pointing device such as a mouse, trackball, or graphic tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the term "input device" is used to include all possible types of devices and ways to input information into computing device 510 or over a communication network.

[0113] User interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display via, for example, an audio output device. Typically, the term "output device" is used to include all possible types of devices and methods for outputting information from computing device 510 to a user or another machine or computing device.

[0114] The storage subsystem 524 stores programs and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include an executable Figure 4 The logic of the selected aspects of the method.

[0115] These software modules are typically executed by processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem can include a variety of memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 524 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media card box. Modules that implement the functions of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 524, or in other machines accessible to the processor 514.

[0116] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computing device 510 to communicate with each other as desired. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0117] Computing device 510 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server group, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computing device 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computing device 510 are possible, which have more Figure 5 The computing devices depicted may have greater or fewer components.

[0118] In situations where the systems described herein collect personal information about a user or may use personal information, the user may have the opportunity or choice to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. At the same time, certain data may be processed in one or more ways prior to being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or the user's geographic location from which geographic location information is obtained may be generalized (such as to a city, ZIP code, or state level) so that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and how the information is used.

[0119] Although various embodiments have been described and illustrated herein, various other devices and / or structures for performing the functions described herein and / or obtaining the results and / or one or more advantages thereof may be utilized, and each such variation and / or modification is intended to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which the teaching is used. A person skilled in the art will recognize or be able to confirm many specific embodiments equivalent to those described herein using only conventional experiments. Therefore, it is to be understood that the above embodiments are given only as examples, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure when such features, systems, articles, materials, kits, and / or methods do not conflict with each other.

Claims

1. A method for providing context-relevant advice, comprising: determining a candidate query based on one or more communicated messages of the ongoing message exchange topic, The ongoing message exchange topic is between at least the first user and the second user via a first message exchange client of the first user and a second message exchange client of the second user; providing a selectable graphical element for display to at least the first user via the first message exchange client and to the second user via the second message exchange client, the selectable graphical element providing an indication of the candidate query; In response to a first user interface input by the first user to the selectable graphical element: determining, from a personal database of the first user, first content responsive to the candidate query; as well as including the first content in the ongoing message exchange topic, wherein including the first content in the ongoing message exchange topic enables the first content to be displayed to the first user via the first message exchange client, and to the second user via the second message exchange client; In response to a second user interface input by the second user to the selectable graphical element: determining, from the second user's personal database, second content responsive to the candidate query; and including the second content in the ongoing message exchange topic, Including the second content in the ongoing message exchange topic enables the second content to be displayed to the first user via the first message exchange client, and to be displayed to the second user via the second message exchange client.

2. The method of claim 1, wherein: Including the first content in the ongoing message exchange topic includes requesting at least one of the first message exchange client and the second message exchange client to insert the first content into a log of the ongoing message exchange topic displayed in a graphical user interface of at least one of the first message exchange client and the second message exchange client.

3. The method of claim 2, wherein: The first content includes one or more texts and images.

4. The method of claim 1, wherein: Determining the candidate query includes: The candidate query is selected from the query corpus based on consistency between one or more query terms of a query in the query corpus and one or more message terms of one or more communicated messages of the ongoing message exchange topic.

5. The method of claim 1, wherein: Determining the candidate query includes: determining at least one entity referenced in the one or more communicated messages of the ongoing message exchange topic; and The candidate query is determined based on the at least one entity.

6. The method of claim 5, wherein: Determining the candidate query based on the at least one entity comprises: One or more query terms for the candidate query are generated based on the alias of the at least one entity.

7. The method of claim 1, wherein: Determining the candidate query and providing the selectable graphical element occurs independently of expression by the first user of any additional natural language input for inclusion in the ongoing message exchange thread.

8. The method of claim 1, further comprising: determining a score for the candidate query based on one or more terms of at least one of the one or more communicated messages of the ongoing message exchange topic; Wherein providing the selectable graphical element is based on the score satisfying a threshold.

9. The method of claim 8, wherein: Determining a score based on the one or more terms includes at least one of: determining the score based on whether a specified entity is included in the one or more terms; as well as The score is determined based on whether the one or more terms conform to at least one predefined search query pattern.

10. The method of claim 1, further comprising: A score for the candidate query is determined based on at least one of: a popularity measure of the candidate query based on the number of submissions of the candidate query, and a quality metric of the first content responsive to the candidate query; Wherein providing the selectable graphical element is based on the score satisfying a threshold.

11. The method of claim 1, wherein: Including the first content in the ongoing message exchange topic includes: The first content is included in the ongoing message exchange topic along with a graphical indication that the first content was generated by an automated assistant.

12. The method of claim 1, further comprising: An additional selectable graphical element is provided for display to the first user via the first messaging exchange client, the additional selectable graphical element identifying one or more terms and causing the one or more terms to be included in the ongoing messaging exchange topic when the additional selectable graphical element is selected, along with an indication that the one or more terms originated from the first user, and causing the one or more terms to be included in the ongoing messaging exchange topic without the need to include any content responsive to the one or more terms.

13. The method of claim 1, wherein: The first content includes a selectable first content item focused on a first entity and a selectable second content item focused on a second entity; wherein user interface input to the first content item causes additional first entity content focused on the first entity to be included in the ongoing message exchange topic; and Wherein, the user interface input for the second content item causes additional second entity content focused on the second entity to be included in the ongoing message exchange topic.

14. The method of claim 1, wherein: The first content includes a first image or text focused on a first entity, and further includes at least one selectable deep link that, when selected, causes one or more additional computing actions specific to the first entity to be performed.

15. The method of claim 1, further comprising: determining additional relevant queries based at least on at least one of the candidate query and the first content; providing, subsequent to the user interface input by the first user, an additional selectable graphical element for display to at least the first user via the first message exchange client, the additional selectable graphical element providing an indication of the additional related query, wherein the additional selectable graphical element is provided for display without simultaneously displaying the selectable graphical element; In response to additional user interface input by the first user for the additional selectable graphical element: including additional content responsive to the additional related query in the ongoing message exchange topic, Wherein, including the additional content in the ongoing message exchange topic causes the additional content to be displayed to the first user via the first message exchange client and to the second user via the second message exchange client.

16. A system comprising a memory and one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to: determining a candidate query based on one or more communicated messages of the ongoing message exchange topic, The ongoing message exchange topic is between at least the first user and the second user via a first message exchange client of the first user and a second message exchange client of the second user; providing a selectable graphical element for display to at least the first user via the first message exchange client and to the second user via the second message exchange client, the selectable graphical element providing an indication of the candidate query; as well as In response to a first user interface input by the first user to the selectable graphical element: determining first content responsive to the candidate query; as well as including the first content in the ongoing message exchange topic, wherein including the first content in the ongoing message exchange topic enables the first content to be displayed to the first user via the first message exchange client, and to the second user via the second message exchange client; In response to a second user interface input by the second user to the selectable graphical element: determining, from the second user's personal database, second content responsive to the candidate query; and including the second content in the ongoing message exchange topic, Including the second content in the ongoing message exchange topic enables the second content to be displayed to the first user via the first message exchange client, and to be displayed to the second user via the second message exchange client.

Images