Contextual Estimation of Link Information Gain

Abstract

Described herein are techniques for determining an information gain score for one or more documents of interest to a user and presenting information from the documents based on the information gain score. The information gain score for a given document indicates additional information included in the document beyond information contained in documents previously viewed by the user. In some implementations, the information gain score for one or more documents may be determined by applying data from the documents across a machine learning model to generate the information gain score. Based on the information gain scores for a collection of documents, documents may be presented to a user in a manner that reflects the possible information gain that the user may be able to obtain if the user views the document.

Description

Background Art

[0001] Information from one or more documents of a shared topic may be provided to an individual (also referred to herein as a "user") via one or more computing devices. For example, a user may submit a search request and be provided with a collection of documents and / or links to documents in response to the submitted search request. Likewise, for example, documents may be provided to a user based on the user's identified interests, the user's previously viewed documents, and / or other criteria that may be used to identify and provide documents of interest. Information from the documents may be provided via, for example, an automated assistant and / or provided as a result to a search engine. In addition, information from the documents may be provided to the user in response to a search request and / or may be automatically served to the user based on a continued search after the user has ended a search session.

[0002] In some cases, a subset of information may be extracted from a document for presentation to a user. For example, when a user engages in a verbal human-computer dialogue via an automated assistant software process (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," "virtual assistant," etc.), the automated assistant may perform various types of processing to extract salient information from a document so that the automated assistant may present the information in an abbreviated form. As another example, in response to a user's search query, some search engines will provide summary information from one or more responsive and / or relevant documents in addition to or in lieu of links to responsive and / or relevant documents.

[0003] However, when identifying a collection of documents that share a topic, many of the documents may include similar information. For example, a user may submit a search related to resolving a computer problem (e.g., failure, malware, configuration, etc.) and may subsequently be provided with multiple documents that include a similar list of solutions, remediation steps, resources, etc. Thus, although two documents that share a topic may be relevant to the user's request or interest, the user may be less interested in viewing the second document after having viewed the same or similar information in the first document or collection of documents. Summary of the invention

[0004] Implementations described herein relate to determining information gain scores for one or more documents of potential interest to a user, and presenting information from one or more of the documents selected based on their respective information gain scores. The information gain score for a given document indicates additional information included in the given document over and above information contained in other documents presented to the user. Information from the document can be presented to the user in a variety of ways, such as opening the entire document (e.g., in a web browser or other applicable software application), audibly reading the entire contents of the document to the user, extracting and audibly / visually presenting to the user salient information extracted from the document, and the like.

[0005] In some embodiments, an information gain score for one or more documents may be determined by applying data indicating documents such as their entire contents, salient extracted information, semantic representations (e.g., embeddings, feature vectors, bag-of-words representations, histograms generated from words / phrases in the documents, etc.) across a machine learning model to generate an information gain score. For example, in some embodiments, data indicating one or more previously presented documents and data indicating one or more not yet presented (or "new") documents may be applied as input across a trained machine learning model to generate an output indicating information gain scores for one or more new documents. Based on the information gain scores, information contained in one or more new documents may be selectively provided to a user in a manner that reflects the possible information gain that the user might obtain if the user were presented with information from the selected documents.

[0006] In some embodiments, a first set of documents that have been presented to a user (e.g., viewed by the user, audibly rendered in whole or in part, etc.) is identified. For example, by an automated assistant in response to a user's natural language input, in a web browser, in another applicable software application, etc., the documents of the first set share a common topic and can be identified based on being previously presented to the user. The user can search for a topic, and one or more document responsive files can be returned (e.g., as a list of search results and / or queued by the automated assistant awaiting potential presentation). As an example, a user can ask the automated assistant "Give me information about fixing my computer," and the automated assistant can provide data indicating one or more responsive (and / or related) files via a visual interface or via text to speech. The first responsive document can include information related to troubleshooting a software application problem, and the user can view and / or listen to the contents of the first document. In addition, the set can include a second document that includes hardware repair information, and the user can additionally view the second document.

[0007] A second set of documents is identified that are also related to the topic of the first set of documents, but have not yet been viewed by the user. For example, documents that include information related to hardware and software repair may be identified. For each new document in the second set of documents, an information gain score is determined that indicates, for the new document, whether the new document includes information that is not included in documents in the first set of documents (i.e., documents that have been presented to the user). For example, if the user has previously viewed documents that include information about repairing common software applications, then an information gain score for a document that includes information related to general computer repair (perhaps including software troubleshooting information) may be less indicative of information gain.

[0008] Based on the information gain scores, one or more new documents may be selected to be presented to the user, and / or the new documents may be ranked based on their corresponding information scores. For example, ten documents may be identified in the second document collection, and information of the document having the most information gain scores indicative of information gain may be presented to the user by the automated assistant. In some embodiments, the new documents may be ranked, and when the user views the additional documents, the second document collection may be re-ranked based on the new information gain scores. For example, the information scores of the second document collection may be recalculated based on the one or more new documents presented to the user, and the first document collection may be moved from the second collection accordingly.

[0009] In some embodiments, data from each document of the second document collection, for example in combination with data from the first document collection, can be applied as input across a machine learning model, and an information gain score can be determined based on the output of the machine learning model. In some embodiments, rather than applying the content of the document itself across a machine learning model, an alternative representation of the document, such as a semantic feature vector or embedding, a "bag of words" representation, etc., can be generated from each document and applied as input across a machine learning model.

[0010] In some embodiments, one or more new documents in the second collection may be presented in a manner selected based on the information gain score. For example, the one or more new documents may be rendered as part of a search result interface that is presented to a user in response to a query that includes a topic of the document (such as a reference to one or more documents). In some embodiments, these search results may be ranked based at least in part on their corresponding information gain scores. The user may then select one of the references, and the information contained in the particular document may be presented to the user. Subsequently, the user may return to the search results, and the user may again be provided with a reference to the document, but the document may be updated based on the new information gain score of the referenced document. In some embodiments, the references may be re-ranked and / or one or more documents may be excluded (or significantly downgraded) from the search results based on the new information gain score determined based on the documents that the user has already viewed. As the user views additional documents, the updated information gain score will be updated to reflect the information that the user has already viewed and the information contained in the unviewed documents.

[0011] The above provides an overview of some embodiments disclosed herein. Further description of these and other embodiments is provided below.

[0012] In some embodiments, a method performed by one or more processors is provided, the method comprising: identifying a first set of one or more documents that share a topic, wherein information extracted from each document in the first set was previously presented to the user by an automated assistant executing at least in part on one or more computing devices operated by a user in response to free-form natural language input from the user; identifying a second set of multiple new documents that also share the topic, wherein the automated assistant has not yet presented information contained in each new document in the second set to the user; determining an information gain score for each new document in the second set, wherein the information gain score for a given new document of the second set indicates additional information that the user would obtain over information extracted from one or more documents in the first set if the information contained in the given new document was extracted and presented to the user; selecting a new document from the second set based on the information gain score; and presenting the information extracted from the selected new document to the user via one or more computing devices.

[0013] In various embodiments, the method may further include ranking the second set of new documents based on the information gain score, wherein the selecting is based on the ranking. In various embodiments, determining the information gain score for the given new document may include applying first data indicating the information extracted from the first set of documents and second data indicating information extracted from the given new document across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

[0014] In various implementations, the first data may include a semantic feature vector generated based on the information extracted from the first document set, and the second data may include a semantic feature vector generated based on the information extracted from the given new document.

[0015] On the other hand, a method implemented using one or more processors may include: identifying a first collection of one or more documents that share a topic, wherein a user accesses each document in the first collection on one or more computing devices operated by the user; identifying a second collection of multiple new documents that also share the topic, wherein the user has not previously accessed the multiple new documents in the second collection; determining an information gain score for each new document in the second collection, wherein the information gain score for a given new document in the second collection indicates that if the user accesses the given new document, the user will obtain additional information beyond the information obtained by the user when accessing the first collection of the one or more documents; selecting a new document from the second collection based on the information gain score; and presenting information indicating the one or more new documents in the second collection to the user via one or more computing devices, wherein the information indicating the one or more new documents in the second collection is presented in a manner selected based on the information gain score.

[0016] In various embodiments, the method may further include: presenting a ranked list of a second collection of new documents, wherein the list is ranked based on the information gain score. In various embodiments, identifying the first collection may include: presenting a reference to one or more documents in the first collection as part of a search results interface presented to the user in response to a query including the topic; receiving user input from the search results interface indicating a selection of a reference to a particular document of the first collection, wherein at least a portion of the particular document is provided to the user in response to the selection; receiving user input indicating a command to return to the search results interface; and in response to receiving the user input indicating the command to return to the search results interface, rendering a reference to one or more documents in the second collection as part of the search results interface.

[0017] In various embodiments, rendering the reference to one or more documents of the second collection may include excluding references to at least certain documents of the first collection from the search results interface. In various embodiments, determining the information gain score for the given new document may include applying first data indicating the first document collection and second data indicating the given new document across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

[0018] In another related aspect, a method implemented using one or more processors may include: identifying a first document including a first information element and a second information element, wherein the first information element is different from the second information element; processing one or more portions of the document including the first information element and the second information element using text-to-speech (“TTS”) processing to generate TTS output that conveys the first information element and the second information element; providing the TTS output at one or more computing devices operated by a user; identifying a second document including the second information element and a third information element, wherein the third information element is different from the first information element and the second information element; based on the providing, determining that the second information element has been conveyed to the user; in response to the determination, processing one or more portions of the second document including the third information element using additional TTS processing to generate additional TTS output that conveys the third information element and excludes the second information element; and providing the additional TTS output at one or more computing devices operated by the user.

[0019] In addition, some embodiments include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units) (TPUs) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and the instructions are configured to cause any of the foregoing methods to be performed. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the foregoing methods.

[0020] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0022] Figure 2 Illustration of an interface for an automated assistant that can provide information in a document to a user based on an information gain score.

[0023] Figure 3 Illustrate an interface for a search interface that can provide references to documents based on their information gain scores.

[0024] Figure 4 Illustrated is a collection of documents that are tagged based on whether a user has viewed the document.

[0025] Figure 5 A flow chart illustrating an example method for presenting information extracted from a new document to a user based on the information gain score of the new document.

[0026] Figure 6 An example architecture of a computing device is illustrated. DETAILED DESCRIPTION

[0027] Now turn to Figure 1 , illustrates an example environment in which the technology disclosed herein can be implemented. The example environment includes a client device 105 and a remote computer 110. Although the client device 105 and the remote computer 110 are Figure 1 110, but it is understood that one or more modules and / or any aspect may be implemented in whole or in part by one or more other devices. For example, in some embodiments, a first set of modules and / or aspects are implemented by one or more processors of a first remote system, and a second set of modules and / or aspects are implemented by one or more processors of one or more separate remote server devices in network communication with the remote computer 110. The remote server device may be, for example, a cluster of high-performance remote server devices that processes requests from one or more client devices and requests from additional devices.

[0028] The client device 105 may be a mobile phone computing device, a tablet computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), and / or a wearable device of a user, which includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided. In addition, one or more components of the client device 105 may be implemented on a separate device. For example, the microphone 108 and / or the automated assistant client 106 may be implemented on one or more alternative computing devices that communicate with the client device 105. The components of the client device 105 and the components of the remote computer 1110 may communicate via a communication network. The communication network may include, for example, a wide area network (WAN) (e.g., the Internet). In addition, the components of the client device 105 may communicate with one or more other components via a communication network. For example, the communication network may include a local area network (LAN) and / or Bluetooth, and may communicate with one or more other devices (e.g., an automated assistant device that communicates with a user's handheld computing device) via a LAN and / or Bluetooth.

[0029] The client device 105 includes a microphone 108 that allows a user to provide natural language input to one or more modules of the client device 105. The user's speech may be captured by the microphone 108, and one or more modules of the client device 105 may process the resulting audio data to generate text and / or other audio data, which may be further processed by one or more modules of the client device 105 and / or provided to the remote computer 110 via network communication for further processing. For example, the client device may include a speech-to-text ("STT") module (not depicted) that processes the captured audio into text for provision to the remote computer 110. Additionally or alternatively, the audio data received by the microphone 108 may be provided to the remote computer 110, which may include its own STT module (not depicted) for further processing of the audio data.

[0030] In some embodiments, the audio data captured by microphone 108 may include one or more topics of interest to the user. For example, a user may utter an audio phrase "Help me fix my computer" via microphone 108 to an automated assistant client 106 executing on client device 105. Automated assistant client 106 may be part of an executing automated assistant engine having a component on client device 105 and a component on remote computer 110 (e.g., remote automated assistant 115, also referred to as a "cloud-based automated assistant component"). Automated assistant client 106 may receive audio data captured by microphone 108 and / or from one or more other input devices that the user may utilize. For example, a user may provide natural language input via a keyboard, which may then be provided to automated assistant client 106 and / or search interface 107, as described herein.

[0031] The automated assistant client 106 and the remote automated assistant 115 can process the user's natural language input and provide a response in the form of a dialogue including one or more dialogue turns. A dialogue turn may include, for example, a natural language input provided by the user and a response to the natural language input by the automated assistant. Thus, a dialogue between the user and the automated assistant may be generated, which allows the user to interact with the automated assistant (the phrase "automated assistant" as used herein without reference numerals may refer to any combination of 105 and 115) in a conversational manner. As an example, a user may submit a natural language input "Help me fix my computer" to the automated assistant client 106. The automated assistant may process the natural language input and identify one or more documents (e.g., files related to computer repair) that are responsive to and / or related to the user's request. The automated assistant client 106 may provide one or more documents (or a portion of one or more documents) via a graphical interface of the client device 105. Additionally or alternatively, the automated assistant client 106 may include a text-to-speech ("TTS") module (not depicted) that converts a portion of the identified document into speech and renders the speech via a speaker (not depicted) of the client device 105. The user may then submit additional natural language input to automated assistant client 106 to further refine the topic search, ask for additional details about the document, and / or submit new topics.

[0032] As used herein, a "conversation session" may include a logically self-contained exchange of one or more messages (or turns) between a user and the automated assistant (and in some cases, other human participants). The automated assistant may distinguish between multiple conversation sessions with a user based on various signals (such as the passage of time between sessions, changes in user context between sessions (e.g., location, before / during / after a scheduled meeting, etc.), detection of one or more intervening interactions between the user and the client device other than the conversation between the user and the automated assistant (e.g., the user switches apps for a while, the user walks away and then returns to a standalone voice-activated product), locking / sleeping of the client device between sessions, changes in the client device used to interface with one or more instances of the automated assistant, etc.

[0033] For audio output and especially, the audio output using TTS module, this information output can be considered as linear or serial.That is, in " dialogue session ", dialogue turn can include natural language input provided by user and the response of automatic assistant to natural language input, and therefore user can wait for substantially all responses to be output before proceeding.Compared with reading, user can passively receive audio information, but, the time spent by output is longer, and the ability of scanning or scrolling / skipping information is reduced.Therefore, in the context of audio output, it is important that the output information is relevant, accurate and concise, so as to avoid unnecessary long output, redundant output or extra dialogue turn.The information density of output information is particularly important for improving the efficiency of dialogue session.Technology described herein solves these problems by reducing and / or eliminating the presentation of information included in the audio man-machine dialogue context that the user has provided.

[0034] The remote computer 110 further includes a search engine 120 that identifies one or more documents related to the search request. In some embodiments, the natural language input received by the remote automated assistant 115 can be converted to text via the STT module and provided to the search engine 120. In some embodiments, the client device 105 includes a search interface that allows a user to submit text directly to the search engine 120. For example, the search interface 107 can be implemented via a web browser, and the user can submit the natural language input via the search interface 107 for further processing by the search engine 120. The automated assistant can also submit natural language requests from the user to the search engine 120.

[0035] The search engine 120 identifies the topic of the user input and identifies documents that are responsive to and / or related to the input. As an example, a user may submit a natural language input of "Help me fix my computer," and the search engine 120 may identify references to one or more topics in the search phrase to identify responsive documents and / or related documents. The identified documents may be scored via a variety of methods to determine the relevance of each document based on the submitted natural language input, as well as the likelihood that the document is of interest to the user. For example, in response to a user submitting the search phrase "Help me fix mycomputer," the search engine 120 may identify documents that include information related to computer hardware repair, computer software repair, common problems with computer software, troubleshooting guides for specific software applications, and / or other documents.

[0036] Then, one or more of the identified documents, portions of the documents, and / or references to one or more documents may be provided to the user based on the identified documents and the scores determined for each document. For example, in response to a user submitting the spoken phrase "Help me fix my computer", the search engine 120 may identify multiple documents and may provide the highest scoring document to the automated assistant client 106. A portion of the highest scoring document may be converted to speech via the TTS module, and then the automated assistant client 106 may provide a portion of the document as audio in a conversation session between the user and the automated assistant client 106. For example, the automated assistant may provide the user with the first few sentences of a document including a troubleshooting guide for common software as a conversation turn in response to the user providing the natural language input "Help me fix my computer". Similarly, for example, additional information may be provided to the user from the document by submitting audio indicating interest in continuing the current document (e.g., "Tell me more"), or interest in receiving information about other documents in future conversation turns (e.g., "What other information have you found").

[0037] As an example, see Figure 2, provides an example interface of an automated assistant. The interface includes a conversation session between a user and the automated assistant as text. However, in some embodiments, the text of each turn of the conversation session may alternatively be phrases provided by the user and / or automated assistant client 106 during an audio conversation session. For example, turn 201 includes a user's natural language input, which may be provided to the automated assistant client 106 via microphone 108 as audio input or via a keyboard as text input. The user's natural language input may be provided to the automated assistant client 106 via a speaker of the client device 105 or via a similar Figure 2 200, providing the automated assistant's response 202 as audio. Response 202 includes information from a document that can be identified by search engine 120 from the natural language input of turn 201. Subsequently, the user indicates to automated assistant client 106 "What else have you found?" as a dialog turn 203, and then provides another dialog turn 204 in response to dialog turn 203, which includes information from a second document that can be identified by search engine 120.

[0038] In some implementations, the first provided document (i.e., the document 202 having the information provided in round 202) may have a generated score that is more indicative of the topicality of the initially submitted phrase 201 than the second provided document. For example, the search engine 120 may rank the resultant response documents and initially and subsequently provide information from the first document for each ranked document, such that the information provided to the user may be from a higher ranked document before a lower ranked document.

[0039] Additionally or alternatively, a list of documents and / or a list of references to documents identified by the search engine 120 may be provided to the user in response to a natural language input including a topic. Figure 3, illustrating an interface 300 for providing search results. The search results include a reference to a first document 301, a reference to a second document 302, a reference to a third document 303, and a reference to a fourth document 304. The interface may be provided to a user's client device 105 by a component that shares one or more characteristics with the search interface 107. Each of the references may be selectable (e.g., clickable) by the user, and when the user selects the reference, the search interface may provide the user with a related document, a portion of a document, and / or information extracted from the document. In addition, each optional link includes information from an associated document to allow the user to view a portion or summary of the document before selecting the reference. For each reference, the search engine 120 may determine a score for the associated document, and may provide the reference to the user via the interface 300 in a manner that reflects the score of the document. For example, the first reference document 301 may have a generated score that indicates more relevance and / or a related relationship to a topic of interest to the user than the second reference document 302, which may have a score that indicates more of the same criteria than the third reference document 303.

[0040] Return to reference Figure 1 , user document database 140 includes references to and / or documents that have been presented to the user in whole, in part, and / or in summary form. When information from a document is presented to the user, user document database 140 may be updated to include information related to the presented document, such as entities in the document, identifiers of the location of the document, semantic vectors representing the text of the document, and the like. The data stored in user document database 140 may be applied as input across a trained machine learning model, for example, in conjunction with data indicating one or more new documents, to determine an information gain score for the one or more new documents.

[0041] refer to Figure 4 , a document collection 400 may be provided to a user, for example, in response to a search query provided to a search engine and / or in response to natural language input provided to an automated assistant. The collection 400 includes four documents, which have not yet been presented to the user. These documents may be, for example, Figure 3 responsive documents and / or related documents provided in response to a gesture query in the search results interface 300 of the automated assistant, and / or documents related to and / or in response to natural language input provided by the user to the automated assistant, and which have not yet been used by the automated assistant to Figure 2 Provide information during the conversation turn.

[0042] Assume that, for example, during a conversation with an automated assistant or as a user from a Figure 3401a and 401b depict how these four documents can be affected. Collection 401a includes three documents of collection 400, and collection 401b includes one document of collection 400. Collection 401a represents documents that have not yet been viewed by the user, and collection 401b represents documents that have not yet been presented to the user (e.g., information from documents in collection 401b provided in a conversational turn of the automated assistant and / or the user in Figure 3 The information in the referenced document has been selected in the interface).

[0043] Now, assume that the user is next presented with information indicating document 2. Sets 402a and 402b describe how these four documents may be affected. Set 402a includes two documents that have not yet been presented to the user, and set 402b includes two documents that have previously been presented to the user (e.g., after a subsequent conversation turn and / or after viewing the documents, returning to Figure 3 In each case, as the user is presented with information from more documents, the user document database 140 may be updated to include information about each presented document. In some implementations, the references in the user document database 140 may be stored after the search session has terminated. In some implementations, each session (i.e., each time a user provides a topic) will have a separate set of viewed documents that are not persisted through multiple topic submissions.

[0044] In various embodiments, the information scoring engine 125 may be configured to determine an information score gain for each document identified by the search engine 120 that has not yet been presented to the user. Figure 4 , the user has not viewed all of the documents of collection 400, and therefore information gain scores may not be generated for any document (or an arbitrary score may be assigned to each document indicating an information gain score that is equal for all documents). Once information from documents in collection 401b has been presented to the user, a new information gain score may be determined for each document in collection 401a. For a given non-presented document, the generated information gain score indicates the information gain of the given non-presented document in view of the documents that have been presented to the user. Thus, information gain scores are calculated based on documents that have been presented to the user and / or information contained in those documents, taking into account the information contained in the new document.

[0045] As an example, a user may provide the topic "Help me fix my computer" and be provided with Figure 4. The user may select document 1 and view the document. Based on viewing the documents, the information gain scoring engine 125 may determine the information gain scores of document 2, document 3, and document 4. Assuming that the user has viewed document 1, each information gain score indicates the information gain of the document. For example, document 1 may include common software troubleshooting techniques for repairing computers, while document 2 may additionally include common software application problems that can be solved by users. However, document 3 may include common computing hardware troubleshooting techniques as well as software solutions. Finally, document 4 may only include hardware solutions for computer problems. Therefore, because document 1 and document 2 have overlapping content that may be the same, the information gain scoring engine 125 may determine an information gain score that indicates less information gain for document 2 than document 3 and document 4, which may include information that is not included in document 1. In addition, the score determined for document 4 may indicate more than document 2 and document 3 that the information gain is that the information in document 4 is not included in document 2 and document 3 at all (i.e., the gain of completely different information).

[0046] In various embodiments, information to be provided to the user may be extracted from documents selected based on information gain scores. In this manner, the information provided to the user may be simplified by prioritizing the output of new information. In some embodiments, unnecessary output of repeated or redundant information may be reduced by determining information gain scores. In embodiments including audio or TTS output as described above, corresponding reductions may be provided in total output time, number of user interruptions, or number of additional conversation turns.

[0047] In some embodiments, the information gain scoring engine 125 can utilize documents that have been presented to the user and other documents related to the same topic across the inputs of the machine learning model. For example, the input of the machine learning model may include a pair of documents <d 1 ,d 2 >, where d 1 is the document that has been consumed / rendered, and d 2 are documents that have not yet been consumed / presented. The output of the machine learning model can indicate that the document d 2 Exceeds Document D 1 Various types of machine learning models can be used, such as various types of neural networks (e.g., feedforward, convolution, etc.), support vector machines, Bayesian classifiers, etc.

[0048] In some implementations, a document that also includes pairs may be used. <d 1 ,d 2 > (or semantic representation pairs generated from paired documents such as embeddings) to train a machine learning model. Each pair of documents (or corresponding semantic representation pairs) can be used with the second document d2 associated information gain score (i.e., indicating if the 1 After the user is presented with information from d 2 How much information should the user get?) Training examples <d 1 ,d 2 > can be applied as input to a cross-training model to generate an output. The output can be compared to the labels assigned to the training examples to determine the error. The error can be used to train a machine learning model, for example, by using techniques such as gradient descent (e.g., randomized, batched) and / or backpropagation (in the case of a neural network).

[0049] In some such implementations, the labels assigned to the training examples may be manually generated. For example, one or more individuals may read the document and then provide a subjective information gain score that represents the difference in consumption of the document. 1 After consumption 2 How much additional (or novel) information is gained in the listing. The information gain score, whether represented in the training data labels or output by the trained machine learning model, can take various forms, such as a value between zero and one, a value along a numerical range, etc. This information gain score can be assigned as a label for a training example, which includes a pair of documents consumed by a user.

[0050] Additionally or alternatively, in some embodiments, training data may be developed while one or more individuals simply search and consume documents in their daily lives. As they view successive documents returned in response to a search query, they may be asked questions (e.g., via a web browser plug-in) for each new document they consume, such as "What this document / information is helpful in view of what you've already read?" or "Was this document redundant?" The individuals' answers to these questions may be used to label training examples generated from these documents. Training may then be performed in a manner similar to that described above.

[0051] In some embodiments, a semantic representation of each document presented to the user and each other document identified by the search engine 120 but not presented to the user may be provided as an input to a machine learning model and the output of the machine learning model may be used to determine an information gain score for each unviewed document. For example, a representation of the documents of collection 402b may be provided as an input across a machine learning model and indicated as "not presented", and a representation of two documents of collection 402a may be provided as an input and indicated as "previously presented" (e.g., based on identifying documents that have been viewed in the user document database 140). The output from the machine learning model may be utilized to determine an information gain score for document 3 and an information gain score for document 4, assuming that the user has viewed document 1 and document 2, each of which indicates an information gain for that document. Additionally, if the user then views document 3 (or is provided information from document 3 by the automated assistant client 106), a new information gain score for document 4 may be determined assuming that the information contained in the other three documents has been presented to the user.

[0052] In some embodiments, a semantic vector for each document may be determined and utilized as input across machine learning models. A semantic vector may be, for example, a digital representation of a document as a vector, and may be used to determine similarities between documents based on proximity in vector space. In some embodiments, the semantic vector may be generated by another trained machine learning model such as a word2vec model.

[0053] For example, an autoencoder (e.g., word2vec) can be trained to receive text as input, encode semantic representations using an encoder portion, and then reconstruct the original text input using a decoder portion. Once trained, the encoder portion can be used alone to generate a semantic representation (e.g., a semantic vector) of a document, which can then be used as input across the previously trained machine learning. Thus, for each document identified by the search engine 120, a semantic vector can be generated. When a user views a document, the semantic vector can be provided as input across the machine learning model, where the labels of "viewed" and "not viewed" also change when the user views the document.

[0054] As previously described, in some embodiments, the information gain annotation engine 130 can generate annotated training data based on annotations by human curators of documents. For example, a curator can be provided with a first document and a second document (described previously). 1 and d 2 ), and the curator can submit a second document d 2Assigning a value indicating the information gain of a second document after viewing a first document. Additionally, for example, during a search session, a user in a session of search interface 107 may be asked to rate and / or rank the information gain of a document (e.g., a pop-up asking the user to rate based on information gained from the viewed document based on other documents viewed during the session). The annotations may be used to generate training data indicating information gain determined by human curators, and the training data may be provided as input to a machine learning model to train the model to determine meaningful outputs (e.g., a score indicating information gain for a given set of documents).

[0055] In some embodiments, semantic representations of multiple documents that have already been presented to the user can be applied as inputs across the model along with the semantic representation of the new document. For example, a first set of n inputs can be retained for semantic features extracted from documents that have already been presented, and a second set of m inputs can be retained for semantic features extracted from new documents. Thus, the overall dimensionality of the inputs is set to n+m. In some cases, if all n inputs of the first set are not needed (e.g., not enough semantic features were extracted), these inputs can be left blank or padded with arbitrary values ​​(such as zeros) to minimize or eliminate any impact they may have on the final output of the machine learning model.

[0056] In some implementations, documents that have not yet been viewed by a user may be re-ranked based on the information gain scores of the documents. For example, information from a first document may be provided to a user by the automated assistant client 106 based on the first document having a score indicating that it is most relevant to a topic provided by the user. Other documents (i.e., new documents) may then be re-ranked based on the previously determined scores and the information gain scores generated based on providing information to the user from the first document. Re-ranking based on information gain scores may result in one or more documents being promoted and / or demoted in a ranked list. Thus, if the user subsequently requests information for a new document, the user may be provided with documents that have a greater information gain than documents that have already been viewed, rather than simply based on the initial ranked list of documents.

[0057] As an example, a user may submit a natural language input "Help me fix my computer". As described herein, documents may be identified by search engine 120 and ranked based on content into a ranked list, such as Figure 4 The information from document 1 can be provided as the most relevant document based on the ranking and can be provided as Figure 2The remaining documents may be scored by the information gain scoring engine 125 and re-ranked according to the determined scores. In the re-ranking, it may be determined that document 4 has a greater information gain (e.g., document 4 includes more information than documents 2 and 3 that is not included in document 1) based on the user being provided with document 1. Therefore, if the user subsequently requests a new document, as indicated in dialog turn 203, the information in dialog box 204 may be generated from document 4 rather than from document 2 (i.e., the higher ranked document before considering the information gain score).

[0058] In this way, a user can be directed to another document with a greater information gain, which can result in the user receiving the desired or required information item more quickly or with fewer input interactions. Thus, by preferentially ranking documents based on greater information gain, the query session between the user and the search engine can be made shorter or more efficient. In some embodiments, a shorter query session or fewer conversation turns can provide a corresponding reduction in resource demands on the system, for example with respect to the system's memory and / or power usage.

[0059] Similarly, Figure 4 The references to the documents provided in the list may be updated based on calculating the information gain scores of the documents associated with the references. The user may first be provided with a list of ranked references such that document 1 appears first, then document 2 appears, and so on. If the user selects a link to document 1, the remaining documents may be scored by the information gain scoring engine 125, re-ranked, and presented in a different order if the user navigates back to the list of references (e.g., document 4 may now appear higher in the list than document 2). Likewise, for example, based on the determined information gain scores, one or more documents may no longer appear in the list of ranked documents. For example, if the user subsequently navigates back to the list, it may be determined that the document includes the same information as another unviewed document and the same unviewed document may be removed from the list of references (or at least substantially downgraded). If the user views the same new document, the reference no longer included in the ranked list of documents may indicate, for example, a user-determined information gain of zero.

[0060] Figure 5 A flow chart illustrating an example method for presenting information from a document to a user based on an information gain score determined for an unviewed document. Figure 5 The steps may be performed by one or more processors (such as one or more processors of a client device). Other embodiments may include in addition to Figure 5 Additional steps beyond those shown in the figure may be performed in a different order and / or in parallel. Figure 5 steps, and / or can be omitted Figure 5 one or more steps.

[0061] At step 505, a first set of documents that share a common topic and have been viewed by a user is identified. Figure 1 The topic is identified by natural language input from an automated assistant client 106. In some implementations, the input may be spoken and received by a component that shares one or more characteristics with microphone 108. In some implementations, the input may be textual and may be provided by the user via a search interface. The first document set includes documents that may have been viewed (or listened to) by the user based on previous conversational turns of a conversational session with automated assistant client 106 and / or based on a previous selection of a document by the user via search interface 107.

[0062] At step 510, a second document set is identified that also shares the topic of the first document set but has not yet been viewed by the user. The second document set may be identified by a component that shares one or more characteristics with the search engine 120. For example, the topic identified from the natural language input may be used by the search engine 120 to identify documents that are responsive to and / or related to the search topic. Additionally, the component that shares one or more characteristics with the user document database 140 may include information about documents that have been viewed by the user. Thus, the second document set may be a subset of a larger document set, with documents removed that have been viewed by the user.

[0063] At step 515, an information gain score is determined for each document in the second document collection. For a document in the second collection, the information gain score indicates the information gain for the user, the information gain for the user being presented with information from the document, assuming that the user has been presented with information from documents in the first document collection. In some embodiments, the information gain score can be determined based on the output of a machine learning model, the output of which has been provided with information related to documents in the first document collection and the second document collection as input. For example, a semantic vector for each document in the first document collection and a semantic vector for a new document in the second document collection can be provided as input across the machine learning model, and the output of the machine learning model can be used to determine an information gain score for the new document. The score can be a quantitative score, for example, between 0.00 and 1.00, where 0.00 indicates that no information gain is expected if the user is assumed to have viewed the document, and a score of 1.00 indicates that the new document only includes information not included in previously viewed documents (e.g., total information gain).

[0064] At step 520, one or more new documents are selected from the second document set based on the information gain score. In some embodiments, the single new document is selected based on the information gain score of the document being more indicative of information gain than all other documents of the second set. In some embodiments, one or more scores already associated with documents in the second document set may be utilized along with the information gain score to select the document. For example, each document in the second document set may be scored by the search engine 120, and the score may be adjusted based on the information gain score determined for the document by the information gain scoring engine 125.

[0065] At step 525, based on the information gain score, one or more documents in the second set and / or information extracted from the one or more documents are presented to the user. In some embodiments, the information extracted from the selected new document can be provided to the user via a component that shares one or more characteristics with the automated assistant client 106. For example, the TTS module can convert a portion of the new document into speech and provide the portion of the text to the user via a speaker of the client device 105. In some embodiments, a list of references to the second document collection can be provided to the user via a component that shares one or more characteristics with the search interface 107. For example, a ranked list of documents can be provided to the user, and the ranked list of documents is ranked based on the information gain score and / or one or more other scores determined for the documents. In some embodiments, the information gain scoring engine 125 can score the reference list and / or document collection as the user views the additional documents. For example, the user can be provided with a top-ranked list of references, select one of the new documents to view, and navigate back to the reference list. When the user returns to the list based on the updated information gain score, the reference can be updated (e.g., re-ranked, reference removed, reference presented differently).

[0066] In this manner, the information presented to the user may be improved given the amount of information provided. The information density of the output provided by automated assistant client 106 may be increased by, for example, prioritizing the output of new information. In this way, automated assistant client 106 may avoid outputting repetitive or redundant information. In some implementations, the TTS module may be required to convert a smaller amount of text to speech output, thereby providing a corresponding reduction in the processing required by the TTS module and the total output time, the number of user interruptions, and the number of additional conversational turns.

[0067] For example, assume that a user engages an automated attendant using spoken natural language input. Further assume that the user seeks help with a computer error message and speaks the request: "Why am I receiving the error message,<hypothetical error message text> ? (Why did I receive the error message, <hypothetical error message text>?)". A first document responsive to the user request can be identified. Assume that the first document includes a first information element (e.g., topic, fact, etc.) associated with a first potential cause of the error message and a second information element associated with a second, different potential cause of the error message.

[0068] In various embodiments, the automated assistant may process one or more portions of a first document including a first information element and a second information element using text-to-speech ("TTS") processing to generate TTS output that conveys the first information element and the second information element. For example, the automated assistant may identify one or more sentences that mention and / or are related to a first potential cause of an error message. Based on these identified one or more sentences, the automated assistant may generate a natural language output (e.g., directly from the document as an excerpt or using a natural language generation technique such as Tensorflow to summarize the first information element) that conveys the first potential cause of the error message. The automated assistant may perform the same operation for a second potential cause of the error message. The TTS output may be provided at one or more computing devices of the user, for example, as an audible natural language output.

[0069] Now, assume that a second document is also identified in response to the user's original request, and that after the previously discussed TTS output is presented to the user, the user issues a subsequent request, such as, "what else could be the cause of this error message?" Assume further that the second document includes a second information element and a third information element that is different from the first and second information elements. For example, the third information element can be a third potential cause of a computer error message that is different from the first and second potential causes of the computer error message. In various embodiments, for example, it can be determined by the automated assistant that the second information element has been communicated to the user based on a previous provision of the TTS output.

[0070] Therefore, the automated assistant can use additional TTS processing to process one or more parts of the second document including the third information element to generate additional TTS output that conveys the third information element and excludes the second information element. For example, one or more sentences, phrases, or other short texts or fragments that mention the third potential cause of the computer error or are related to it in the third document can be identified. Then, TTS processing can be applied to generate, for example, extracts directly from the second document and / or natural language output that summarize these sentences, phrases, or other short texts or fragments. The additional TTS output can be provided at one or more computing devices operated by the user. Therefore, summary information about each of the first, second, and third potential causes of the computer error message can be presented to the user without providing redundant information. As described above, even in summary form, it takes a long time to audibly provide information from the document to the user, and eliminating any redundant information can substantially improve the user experience.

[0071] Figure 6 6 is a block diagram of an example computing device 610 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626; an interface output device 620; a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0072] User interface input devices 622 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into a display; an audio input device such as a voice recognition system; a microphone; and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways of inputting information into computing device 610 or a communication network.

[0073] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visual images. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 610 to a user or another machine or computing device.

[0074] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include executing selected aspects of the methods described herein and implementing Figure 1 The logic of the various components is depicted in .

[0075] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments can be stored by the file storage subsystem 626 in the storage subsystem 624, or stored in other machines accessible to the processor 614.

[0076] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0077] The computing device 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 610 may have more Figure 6 The computing devices depicted may have greater or fewer components.

[0078] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only after receiving explicit authorization to do so from the relevant user.

[0079] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to that user to provide permission or authorization as to whether information is to be collected and which portions of the information are to be collected. For example, one or more such control options may be provided to a user over a communications network. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be summarized as a larger area so that the user's specific location cannot be determined.

[0080] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which the teachings are applied. Those skilled in the art will recognize or be able to use no more than routine experiments to ascertain many equivalents of the specific embodiments described herein. It should therefore be understood that the above embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented using one or more processors, comprising: identifying a first set of one or more documents that share a topic, wherein information extracted from each document in the first set was previously presented to a user by an automated assistant executing at least in part on one or more computing devices operated by the user in response to free-form natural language input from the user; identifying a second set of a plurality of new documents that also share the topic, wherein the automated assistant has not yet presented information contained in each of the new documents in the second set for presentation to the user; determining an information gain score for each new document in the second set, wherein the information gain score for a given new document in the second set is based on an amount of new information in the given new document in the second set that would be different from information extracted from each document in the first set if the new information contained in the given new document were extracted and presented to the user; selecting a new document from the second collection based on the information gain score; and Information extracted from the selected new document is presented to the user via one or more of the computing devices.

2. The method of claim 1, further comprising ranking the second set of new documents based on the information gain scores, wherein: The selection is based on the ranking.

3. The method according to claim 1, wherein: Determining the information gain score for the given new document includes: First data indicating the information extracted from the first document set and second data indicating information extracted from the given new document are applied across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

4. The method according to claim 3, wherein: The first data includes a semantic feature vector generated based on the information extracted from the first document set, and the second data includes a semantic feature vector generated based on the information extracted from the given new document.

5. A method implemented using one or more processors, comprising: identifying a first set of one or more documents that share a topic, wherein each document in the first set is accessed by a user on one or more computing devices operated by the user; identifying a second set of a plurality of new documents that also share the topic, wherein the user has not previously accessed the plurality of new documents in the second set; determining an information gain score for each new document in the second collection, wherein the information gain score for a given new document in the second collection is based on an amount of new information in the given new document in the second collection that would be different from information extracted from each document in the first collection if the user had accessed the given new document; and The new information from one or more new documents in the second collection is presented to the user via one or more of the computing devices, wherein the new information from one or more new documents in the second collection is presented in a manner selected based on the information gain score.

6. The method of claim 5, further comprising presenting a ranked list of a second set of new documents, wherein: The list is ranked based on the information gain score.

7. The method according to claim 5, wherein: Identifying the first set includes: presenting a reference to one or more documents of the first collection as part of a search results interface presented to the user in response to a query including the topic; receiving user input from the search results interface indicating a selection of a reference to a particular document in the first collection, wherein at least a portion of the particular document is provided to the user in response to the selection; receiving a user input indicating a command to return to the search results interface; and In response to receiving the user input indicative of the command to return to the search results interface, rendering references to one or more documents in the second collection as part of the search results interface.

8. The method according to claim 7, wherein: Rendering the references to one or more documents in the second collection includes excluding at least references to particular documents in the first collection from the search results interface.

9. The method according to claim 5, wherein: Determining the information gain score for the given new document includes: First data indicating a first set of documents and second data indicating the given new document are applied across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

10. A system comprising one or more processors and a memory operably coupled to the one or more processors, wherein the memory stores instructions that, in response to the one or more processors executing the instructions, cause the one or more processors to perform the following operations: A first set of one or more documents that share a topic is identified, wherein: information extracted from each document in the first collection was previously presented to the user by an automated assistant executing at least in part on one or more computing devices operated by the user in response to free-form natural language input from the user; identifying a second set of a plurality of new documents that also share the topic, wherein the automated assistant has not yet presented information contained in each of the new documents in the second set for presentation to the user; determining an information gain score for each new document in the second set, wherein the information gain score for a given new document in the second set is based on an amount of new information in the given new document in the second set that would be different from information extracted from each document in the first set if the new information contained in the given new document were extracted and presented to the user; selecting a new document from the second collection based on the information gain score; and Information extracted from the selected new document is presented to the user via one or more of the computing devices.

11. The system of claim 10, further comprising ranking a second set of new documents based on the information gain scores, wherein: The selection is based on the ranking.

12. The system according to claim 10, wherein: Determining the information gain score of the given new document includes: First data indicating the information extracted from the first document set and second data indicating information extracted from the given new document are applied across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

13. The system according to claim 12, wherein: The first data includes a semantic feature vector generated based on the information extracted from the first document set, and the second data includes a semantic feature vector generated based on the information extracted from the given new document.

14. A system comprising one or more processors and a memory operably coupled to the one or more processors, wherein the memory stores instructions that, in response to the one or more processors executing the instructions, cause the one or more processors to perform the following operations: identifying a first set of one or more documents that share a topic, wherein each document in the first set is accessed by a user on one or more computing devices operated by the user; identifying a second set of a plurality of new documents that also share the topic, wherein the user has not previously accessed the plurality of new documents in the second set; determining an information gain score for each new document in the second collection, wherein the information gain score for a given new document in the second collection is based on an amount of new information in the given new document in the second collection that would be different from information extracted from each document in the first collection if the user had accessed the given new document; selecting a new document from the second collection based on the information gain score; and The new information from one or more new documents in the second collection is presented to the user via one or more of the computing devices, wherein the new information from one or more new documents in the second collection is presented in a manner selected based on the information gain score.

15. The system of claim 14, further comprising presenting a ranked list of a second set of new documents, wherein: The list is ranked based on the information gain score.

16. The system of claim 14, wherein: Identifying the first set includes: rendering a reference to one or more documents in the first collection as part of a search results interface presented to the user in response to a query including the topic; receiving user input from the search results interface indicating a selection of a reference to a particular document in the first collection, wherein at least a portion of the particular document is provided to the user in response to the selection; receiving a user input indicating a command to return to the search results interface; and In response to receiving the user input indicative of the command to return to the search results interface, rendering references to one or more documents in the second collection as part of the search results interface.

17. The system of claim 16, wherein: Rendering the references to one or more documents in the second collection includes excluding at least references to the particular documents in the first collection from the search results interface.

18. The system of claim 14, wherein: Determining the information gain score of the given new document includes: First data indicating a first set of documents and second data indicating the given new document are applied across a machine learning model to generate an output, wherein the information gain score is determined based on the output.

19. The system of claim 18, wherein: The machine learning model includes a neural network.

20. The system of claim 18, wherein: The first data indicative of the first document set includes one or more semantic representations generated from the first document set, and the second data indicative of the given new document includes a semantic representation generated from the given new document.

Images