Large language model output implication

Abstract

Implementations are described herein for identifying potential error information in generative model outputs by performing implication evaluation of the generative model outputs. In various implementations, data indicative of a query may be processed to generate a generative model output. Text fragments may be extracted from the generative model output, and a subset of the text fragments may be classified as suitable for text implication analysis. A text implication analysis may be performed on each text segment of the subset, including formulating a search query based on the text segment, retrieving a document responsive to the search query, and processing the text segment and the document using an implication machine learning model to generate a prediction of whether the at least one document confirms or counteracts the text segment. When a natural language (NL) responsive to the query is rendered at the client device, an annotation may be rendered to express the prediction.

Description

Background Technology

[0001] Generative models are a type of machine learning model that generates a sequence of output data based on a sequence of input data. Large Language Models (LLMs) are a type of generative model capable of performing a wide variety of Natural Language Processing (NLP) tasks, such as language generation, machine translation, and question answering, to name a few. LLMs are typically trained on massive amounts of diverse data, including, but not limited to, data from web pages, ebooks, software code, e-news articles, and machine translation data. Therefore, these LLMs leverage the underlying data on which they were trained to perform these diverse NLP tasks. For example, when performing a language generation task, these LLMs can process natural language (NL)-based input received from a client device and generate NL-based output that is in response to the NL-based input and is to be rendered at the client device.

[0002] A more general challenge with LLMs and generative models is that they occasionally generate misinformation, sometimes referred to as “illusions.” Misinformation undermines user trust and leads to reduced use and reliance on generative models. However, LLM-generated illusions are not always easily identifiable, and manually verifying them can be time-consuming. Summary of the Invention

[0003] This document describes implementations for identifying and presenting potential misinformation and / or illusions contained in documents and / or text content (including, but not limited to, generative model outputs) to users. More specifically, but not exclusively, this document describes techniques for performing implication evaluation of document and / or generative model outputs and providing user interface (UX) functionality that enables users to interact with various visual elements and / or annotations associated with that implication evaluation.

[0004] In some implementations, a method may be implemented by one or more processors and may include: receiving a query associated with a client device operated by a user; generating a generative model output based on data indicating the query processed using a generative model; extracting multiple text fragments from the generative model output; classifying a subset of the text fragments as suitable for textual implication analysis; performing textual implication analysis separately for each text fragment in the subset, wherein for each text fragment in the subset, the textual implication analysis includes: formulating a search query based on the text fragment, retrieving at least one document responding to the search query, and processing the text fragment and the at least one document using one or more implication machine learning models to generate one or more predictions that the at least one document confirms or refutes the text fragment; causing natural language (NL) in response to the query to be rendered at the client device; and causing one or more annotations to be rendered at the client device, wherein the one or more annotations express one or more predictions among the predictions for one or more text fragments in the subset.

[0005] In various implementations, one or more implication machine learning models may include: a confirmatory machine learning model trained to generate a first output indicating whether a document confirms a text fragment; and a rebuttal machine learning model trained to generate a second output indicating whether a document refutes a text fragment. In various implementations, one or more predictions in the predictions may be determined based on a comparison of the first and second outputs. In various implementations, one or more visual attributes selected based on this comparison may be used to render one or more annotations in the annotations.

[0006] In various implementations, a classifier machine learning model can be used to categorize each text segment in the subset as suitable for textual implication analysis. This classifier machine learning model is trained to classify text segments as either capable or unsuitable for textual implication analysis. Alternatively, a text segment can be categorized as suitable for textual implication analysis based on an implication score predicted for each text segment in the subset using a regression machine learning model trained to predict the text segment's textual implication analysis suitability score.

[0007] In various implementations, the subset may include multiple text fragments, and textual implication analysis is performed on these multiple text fragments in parallel. In various implementations, one or more annotations expressing one or more predictions in the prediction may include: a first annotation that visually highlights a text fragment in the text fragment that is confirmed by a document in the document, using a first color; and a second annotation that visually highlights another text fragment in the text fragment that is refuted by a document in the document, using a second color different from the first color.

[0008] In various implementations, a given annotation in the annotations expressing one or more predictions in the prediction can be operable to retrieve at least a portion of the text segment under that given annotation that confirms or refutes the document. In various implementations, the method may include causing a pop-up window to be rendered at a client device, wherein the pop-up window conveys that portion of the text segment under the given annotation that confirms or refutes the document. In various implementations, the method may include causing a new web browser tab to be rendered at a client device, wherein the new web browser tab conveys all or part of the text segment under the given annotation that confirms or refutes the document. In various implementations, the new web browser tab may be automatically scrolled to the location of the document containing that portion of the text segment under the given annotation that confirms or refutes the document.

[0009] In various implementations, the method may include causing one or more interactive feedback elements to be rendered at a client device, wherein the one or more interactive feedback elements are operable to accept or reject one or more predictions for one or more text fragments in a subset of text fragments. In various implementations, the NL in response to the query may be rendered at the client device without annotations before rendering one or more annotations. In various implementations, one or more predictions of whether the at least one document confirms or refutes a text fragment may be conditionally generated based on a response content quality metric determined for at least one document.

[0010] In another related aspect, a method may be implemented by one or more processors and may include: receiving a query associated with a client device operated by a user; generating a generative model output based on data indicative of the query processed using a generative model; extracting multiple text fragments from the generative model output; performing textual implication analysis separately for each text fragment, wherein for each text fragment, the textual implication analysis includes: formulating a search query based on the text fragment, retrieving at least one document responding to the search query, and processing the text fragment and the at least one document using a confirmatory machine learning model and a refutation machine learning model to generate a confirmatory metric and a refutation metric for the text fragment, respectively; causing natural language (NL) in response to the query to be rendered at the client device; and causing one or more annotations to be rendered at the client device, wherein the one or more annotations include one or more visual features selected based on a comparison of the confirmatory metric and the refutation metric. In various implementations, the one or more visual features may include colors for highlighting one or more text fragments in the text fragments and / or colors for rendering one or more text fragments in the text fragments.

[0011] Additionally, some implementations include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the methods described above.

[0012] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0013] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0014] Figure 2 Examples of how the various components described herein can collaborate to perform selected aspects of this disclosure are illustrated schematically.

[0015] Figure 3A and Figure 3B Examples of graphical user interfaces (GUIs) used in the various implementations described herein are illustrated schematically.

[0016] Figure 4 Examples are depicted showing how confirmatory and disproven metrics can be compared to select annotation attributes based on various implementations.

[0017] Figure 5 A flowchart illustrating an example method of practicing a selected aspect of this disclosure is depicted.

[0018] Figure 6 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0019] Now go to Figure 1 The diagram depicts a block diagram of an example environment 100 illustrating various aspects of this disclosure and in which implementations disclosed herein may be carried out. Example environment 100 includes a client device 110, a natural language (NL)-based response system 120, and a search system 140. Although shown separately, in some implementations, all or all aspects of the NL-based response system 120 and all or all aspects of the search system 140 may be implemented as part of an integrated system.

[0020] In some implementations, all or all aspects of the NL-based response system 120 may be implemented locally at the client device 110. In additional or alternative implementations, all or all aspects of the NL-based response system 120 may be obtained from, for example... Figure 1 The depicted client device 110 is implemented remotely (e.g., at a remote server). In those implementations, the client device 110 and the NL-based response system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi LANs, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet) .

[0021] Client device 110 may be one or more of the following, for example: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart home appliance (such as a smart TV), and / or user's wearable device including a computing device (e.g., user's watch with computing device, user's glasses with computing device, virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0022] Client device 110 may execute one or more applications, such as application 115, through which queries can be submitted and / or (e.g., audibly and / or visually) rendered NL-based summaries and / or other responses to queries. Application 115 may be an application separate from the operating system of client device 110 (e.g., an application installed "on top of" the operating system) – or it may alternatively be implemented directly by the operating system of client device 110. For example, application 115 may be a web browser installed on top of the operating system, or it may be an application integrated as part of the functionality of the operating system. Application 115 may interact with NL-based response system 120.

[0023] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of client device 110. Alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in the field of view of one or more visual components. Alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110. Some examples of queries described herein may be queries formulated based on user input provided by a user of client device 110 and detected via user input engine 111. For example, the query could be a typed query entered via a physical or virtual keyboard, a suggested query selected via a touchscreen or mouse, a spoken voice query detected by the microphone of the client device, or an image query based on an image captured by the visual component of the client device.

[0024] In various implementations, client device 110 may include rendering engine 112 configured to provide content (e.g., NL-based summaries, creative generative model outputs, chat outputs, etc.) for audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers enabling the provision of audible content to a user via client device 110. Alternatively, client device 110 may be equipped with a display or projector enabling the provision of visual content to a user via client device 110.

[0025] In various implementations, client device 110 may include a context engine 113 configured to determine the context (e.g., current or recent context) of client device 110 and / or its user. In a multi-turn session between the user and an automated assistant (or, alternatively, a “virtual assistant,” “chatbot,” etc.), the context of the client device and / or the user can be maintained as a “user state” across multiple turns.

[0026] In some implementations, the context engine 113 may utilize current or recent interactions of the client device 110, the location of the client device 110, profile data of the client device 110's user (e.g., an active user when multiple profiles are associated with the client device 110), and / or other data accessible to the context engine 113 to determine the context and / or update the user's state. For example, the context engine 113 may determine the current context based on one or more recent queries of a search session, profile data, and / or the current location of the client device 110. For example, the context engine 113 may determine the current context "looking for a healthy lunch restaurant in Louisville, Kentucky" based on recently published queries, profile data, and the location of the client device 110.

[0027] As another example, the context engine 113 can determine the current context based on which application is active in the foreground of the client device 110, the current or recent state of the active application, and / or the content currently or recently rendered by the active application. The context determined by the context engine 113 can be used, for example, to supplement or rewrite queries based on user input, generate implicit queries (e.g., queries independent of user input), and / or determine the submission of implicit queries and / or determine the rendering results for implicit queries (e.g., NL-based summaries). Furthermore, the user's context across multiple rounds of a search session can be used as the aforementioned user state to enrich the output rendered, for example, by a search chatbot companion device, at each round of a multi-round human-computer dialogue session.

[0028] In various implementations, client device 110 may include an implicit input engine 114 configured to: generate an implicit query independently of any user input intended to formulate the implicit query; submit the implicit query, optionally independently of any user input requesting the submission of the implicit query; and / or cause the rendering of the result of the implicit query, optionally independently of any user input requesting the rendering of the result. For example, implicit input engine 114 may use the current context from current context engine 113 to generate the implicit query, determine to submit the implicit query, and / or determine to cause the rendering of the result of the implicit query. For example, implicit input engine 114 may automatically generate and automatically submit the implicit query based on the current context. Further, implicit input engine 114 may automatically push the result of the implicit query so that the result is automatically rendered, or may automatically push a notification of the result, such as an optional notification, which causes the rendering of the result when selected. As another example, the implicit input engine 114 can generate implicit queries (e.g., implicit queries related to the user's interests) based on profile data, submit the queries at regular or irregular intervals, and cause the automatic provision of corresponding results for the submissions (or automatic notification of such results). For example, the implicit query could be "patent news" based on profile data indicating interest in patents, which is submitted periodically, and the corresponding NL-based summary results are automatically rendered. It should be noted that, given the existence of, for example, new / fresh search result documents over time, the provided NL-based summary results may change over time.

[0029] Furthermore, the client device 110, the NL-based response system 120, and / or the search system 104 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication via one or more networks in network 199. In some implementations, one or more software applications may be locally installed on the client device 110, while in other implementations, one or more software applications may be remotely hosted (e.g., by one or more servers) and may be accessible from the client device 110 via one or more networks in network 199.

[0030] although Figure 1The aspects described herein relate to a single client device with a single user, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form a coordinated ecosystem of devices that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users (e.g., user groups, families) in a shared setting.

[0031] The NL-based response system 120 is shown as including a generative model (GM) selection engine 124, a GM input engine 126, a GM response generation engine 128, a fragment extraction engine 130, an implication filtering engine 132, an implication evaluation engine 134, a query formulation engine 137, and a user interface (UX) engine 136. Some of these engines may be omitted and / or combined in various implementations.

[0032] The GM selection engine 124 can, for example, select zero or more generative models from a plurality of candidate generative models. For instance, in some iterations, the system will determine that none of the candidate generative models will be utilized; in some iterations, the system will determine that only one candidate generative model will be utilized; and in some iterations, the system will determine that multiple candidate generative models will be utilized. The GM selection engine 124 may optionally utilize one or more rules and / or one or more classifiers 125, which are trained to generate outputs identifying which generative models are best suited to generate a response to the current query given the current user state / context.

[0033] The GM input engine 126 can be configured to assemble generative model input hints based on data such as the current query, the current user state / context, past queries, and past generative outputs (which may be included in the current user state / context). In some implementations, generative model input hints may include sequences of lexical units, which may be words, phrases, or embeddings generated from data such as text, images, and audio.

[0034] The GM response generation engine 128 can be configured to apply one or more generative models stored in the GM database 129 to input prompts generated by the GM input engine 126 to generate generative model outputs. The generative model outputs can take various forms, such as sequences of lexical units that correspond to, represent, or directly convey words, phrases, embeddings, images, videos, audio, etc. The generative models stored in the GM database 129 can take many forms, including but not limited to models such as PaLM, BERT, LaMDA, Meena, and / or any other generative model, such as encoder-only, decoder-only, sequence-to-sequence based models, and optionally including attention mechanisms or other memories, diffusion models, etc. The generative model can have hundreds of millions or even trillions of parameters. In some implementations, the generative model can include multimodal models such as visual language models (VLM) and / or visual question answering (VQA) models, which can have any of the aforementioned architectures and can be used to process multiple data modalities, particularly, for example, images and text and / or images and audio, to generate one or more output modalities. Non-limiting examples of VLMs that can be applied as described herein include Gemini and / or Flamingo, to name just a few.

[0035] The fragment extraction engine 130 can be configured to extract meaningful fragments of text (e.g., statements, complete sentences, independent clauses) from the output of the generative model generated by the GM response generation engine 128. For example, in the generative model response “George Washington was the first President of the United States. He was born on February 22, 1732, on his family's plantation on Papes Creek in Westmoreland County, Virginia”, the following fragments can be extracted: (i) “George Washington was the first President of the United States”; (ii) “He was born on February 22, 1732”; and (iii) “He was born on his family's plantation on Papes Creek in Westmoreland County, Virginia”.

[0036] The implication filtering engine 132 can be configured to evaluate text fragments extracted from the output of the generative model by the fragment extraction engine 130 using various criteria and / or techniques to determine which text fragments are suitable for text implication analysis (or, in other words, "attribution required"). For example, phrases such as "Good afternoon" and "Thank you" are not suitable for text implication analysis because they do not state facts that can be verified or refuted / disproven. In contrast, the statement "George Washington was the first President of the United States" is suitable for text implication analysis because it states at least one verifiable fact: George Washington was the first President of the United States.

[0037] The implication filtering engine 132 can evaluate the suitability of text fragments extracted from the output of a generative model for text implication analysis in various ways. In some implementations, the implication filtering engine 132 can use a data source such as a knowledge graph to identify entities and / or facts associated with those entities in the text fragments. If entities and / or facts are identified based on the fragment, the fragment may be suitable for text implication analysis. If no entities or facts are identified, such as for statements like “Good morning” or “thank you”, the fragment may not be suitable for text implication analysis and may be discarded (or “filtered”) and not considered.

[0038] Alternatively or concurrently, in some implementations, the implication filtering engine 132 may use one or more classifiers 133 trained to generate outputs indicating whether an input text fragment is suitable for text implication analysis. For example, such classifiers may be trained on a corpus of training data comprising text fragments labeled as suitable or unsuitable for text implication analysis. The classifier 133 may take various forms, including machine learning models such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, gated recurrent unit (GRU) networks, (e.g., self-attention) Transformers, or various types of generative models previously mentioned, including “scaled-down” generative models with fewer parameters than, for example, those used by the GM response generation engine 128. For instance, an LLM, which may include billions of parameters and is used by the GM response engine 128, may be used as a “teacher” model to train a “student” model (e.g., another LLM) with far fewer parameters to perform implication filtering.

[0039] The implication evaluation engine 134 can be configured to evaluate text fragments based on documents retrieved using text fragments classified by the implication filtering engine 132 as suitable for text implication analysis. For example, the implication evaluation engine 134, or another component such as the query formulation engine 137, can formulate search queries for each text fragment remaining after the implication filtering engine 132 has filtered out inappropriate text fragments, seeking documents or other content that can be used to confirm or refute that text fragment. In some implementations, the query formulation engine 137 can utilize one or more machine learning models, such as one or more generative models 129 or individual models (e.g., LLM, recurrent neural networks, etc.), to formulate these search queries. The implication evaluation engine 134 can then use one or more implication models 135 to process text fragments and responsive documents / content.

[0040] For example, in some implementations, the implication evaluation engine 134 can assemble input hints for both confirming implication model 135A and refuting implication model 135B. Each input hint can include content extracted or otherwise derived from one or more responsive documents returned by the search system 140 based on one or more search queries formulated by the query formulation engine 137. Models 135A-B can generate two distinct outputs: one indicating whether the response content confirms (or "implications") the text fragment and another indicating whether the response content refutes the text fragment. In other implementations, the implication evaluation engine 134 can use a single implication model that generates outputs such as a confirmation score and / or a classification of "confirmed" or "refuted". In either case, based on these outputs, the implication evaluation engine 134 can generate various predictions as to whether at least one document confirms or refutes the text fragment.

[0041] UX engine 136 can be configured to cause data at client device 110 to indicate predictions made by implication evaluation engine 134. For example, in some implementations, UX engine 136 can cause annotations to be added to the generative model output presented at client device 110, for example, by providing data indicating annotations to rendering engine 112, such that rendering engine 112 can modify the GUI to annotate the text fragment based on the implication predictions of the text fragment. For example, one color can be used to visually emphasize (e.g., highlight) a confirmed sentence, another color can be used to visually emphasize a refuted sentence, and so on. In some implementations, these visual annotations can be operable to present additional information. For example, clicking on a paragraph marked as refuted can cause a pop-up window or similar visual element at client device to render a document or other content that tends to refute or question the veracity of the paragraph.

[0042] Search system 140 is shown as including an SRD engine 142 and a results engine 144. Some of these engines may be omitted or combined with each other in various implementations. For example, SRD engine 142 may utilize index 143 and / or other resources to identify search result documents in response to queries as described herein. For example, SRD engine 142 may use queries formulated by components of the NL-based response system 120, such as query formulation engine 137, to identify search result documents or other content that potentially confirm and / or refute text fragments. Results engine 144 may generate search results, which may optionally be presented along with the NL-based summary described herein, and / or may be used by GM response generation engine 128 to generate generative model responses.

[0043] Figure 2 schematically depicted Figure 1 Examples of how the various components depicted can collaborate to perform selected aspects of this disclosure are provided. Starting from the upper right of the dashed rectangle, a NL-based query 250 may be received, for example, at the user input engine 111 of the client device 110. The NL-based query 250 may be typed, transcribed from spoken utterance using ASR, or even an implicit query. In any case, the GM response generation engine 128 may process the data indicative of the NL-based query 250 using one or more generative models from the database 129 to generate a GM response 252.

[0044] Although for the sake of simplicity, it was not included Figure 2 As shown, but it should be understood that in various implementations, other upstream components of the GM response generation engine 128 (such as one or more engines among engines 124-126) can perform various operations to prepare input prompts, which are ultimately processed by the GM response generation engine 128 using a generative model such as LLM to generate a GM response 252. The GM response 252 may include, for example, one or more sentences, paragraphs, images, audio, video, etc., in response to an NL-based query 250. It should also be understood that in Figure 2 The technique described in this paper, indicated outside the dashed box at the top, can be used to perform implication analysis on content other than the generative model response 252. For example, and as... Figure 2 As shown in the upper left corner, any other type of document 251 (e.g., web pages, social media posts, videos, or their transcriptions) can be generated by Figure 2 The fragment extraction engine 130 and other components described downstream are used to perform implication analysis on this document 251.

[0045] GM response 252 (or document 251, as appropriate) can be processed by fragment extraction engine 130 to extract a number of content fragments 254 (e.g., sentences, clauses, expressions, phrases, claims, images or portions thereof, audio or portions thereof, video or portions thereof, etc.) that can be processed independently by downstream components. In various implementations, fragment extraction engine 130 can employ rule-based or machine learning-based methods to extract fragments 254. In the former case, fragment extraction engine 130 can extract individual sentences, clauses separated by punctuation, etc. Alternatively, fragment extraction engine 130 can use a machine learning model, such as a deep neural network, to extract fragments 254. For example, the machine learning model can be trained to process data indicating GM response 252 (or document 251) and output data indicating predicted boundaries between text fragments of GM response 252 (or document 251).

[0046] Regardless of how fragment 254 is extracted, the implication filtering engine 132 can be configured to classify a subset 254' of fragment 254 as suitable for textual implication analysis. For example, the implication filtering engine 132 can employ one or more trained classifiers 133 to classify individual fragments of content as suitable or unsuitable for textual implication analysis, or, in the case of regression, to predict a suitability score for textual implication analysis. As previously noted, the classifier 133 can take various forms, such as a Transformer-based generative model (e.g., LLM), RNN, LSTM network, GRU network, or other types of encoder-decoder models, encoder-only models, or decoder-only models. In other implementations, the implication filtering engine 132 can use a rule-based approach to filter out content fragments unsuitable for textual implication analysis. For example, the implication filtering engine 132 can leverage knowledge graphs or other databases to identify entities and / or facts in content fragments that can then indicate suitability for textual implication analysis.

[0047] A subset 254' of text segments deemed suitable for textual implication analysis can be provided to both the search system 140 and the implication evaluation engine 134. The search system 140 can use the subset of text segments to formulate a certain number of search queries. Figure 2 (Not depicted in the text). These search queries can be used by engines 142 and / or 144 of search system 140 to retrieve response content 256. Response content 256 may include documents (e.g., text-based, image-based, audio-based, etc.), snippets / parts of documents that most directly respond to the search query, or other content derived from the response document (e.g., using LLM, VLM, etc.). Response content 256 may be paired with text fragments of subset 254' where applicable.

[0048] In some implementations, the SRD engine 142 and / or the results engine 144 may be configured to evaluate the quality, credibility, and / or reliability of the search result documents (or more generally, the response content 256) and provide one or more response content quality metrics to other components such as the components of the NL-based response system 120 described herein. These response content quality metrics can be determined in various ways. In some implementations, the response content quality metrics may include query-related metrics that take into account both the search query and the response content. Alternatively, in some implementations, these response content quality metrics may include query-independent metrics that may include, for example, metrics determined by the search system 140 independently of any query.

[0049] In various implementations, these response content quality metrics can be used, for example, by the implication filtering engine 132 and / or the implication evaluation engine 134 as thresholds for determining whether to continue textual implication analysis. For instance, while a particular text fragment may be deemed suitable for textual implication analysis by the implication filtering engine 132, the implication evaluation engine 134 may determine that the response content 256 retrieved based on that particular text fragment lacks sufficient quality to be considered as confirming or refuting the text fragment. For example, the website from which the response content 256 is obtained may be considered lacking credibility. In some such cases, the implication evaluation engine 134 and / or the UX engine 136 may avoid visually annotating text fragments in the GM response 252 (or document 251). In other such cases, the implication evaluation engine 134 and / or the UX engine 136 may cause the client device 110 to render output indicating potentially weak confirmatory evidence or potentially weak refusal evidence.

[0050] The implication evaluation engine 134 can use one or more implication models 135 to process pairs of text fragments 254' and corresponding response content 256 (the entire document or fragments / excerpts extracted from it) to generate a prediction 258 that the response content 256 either confirms or refutes the corresponding text fragment. Figure 2 For example, the implication evaluation engine 134 uses a confirmation model 135A and a rebuttal model 135B to process the pairs of text fragments 254' and their corresponding response content 256. Therefore, the prediction 258 includes both a confirmation metric 260 and a rebuttal metric 262.

[0051] In some implementations, models 135A, 135A can be applied in parallel (e.g., as separate threads) to copies of the pair to speed up processing and / or reduce latency. Alternatively, in some implementations, the implied evaluation engine 134 can process multiple different pairs of text fragments 254' and corresponding response content 256 in parallel to further reduce latency.

[0052] In various implementations, the confirmatory metric 260 and the rebuttal metric 262 can be evaluated, for example, by an implication evaluation engine 134 and / or another component such as a UX engine 136, to determine how to cause the rendering of visual or audio output at the client device 110. Figure 2 For example, when the GM response 252 / document 251 is initially rendered by the rendering engine 112 of the client device 110, or after the GM response 252 / document has been rendered without annotations, for example, the UX engine 136 uses the confirmatory metric 260 and the disproving metric 262 to generate visual and / or audio annotations 264 that can be applied to the GM response 252 and / or document 251. Figure 4The document presents a non-limiting example of such rules and / or heuristics that can be used by UX Engine 136 or another component. In some implementations, UX Engine 136 can base its rules on a comparison of confirmatory metric 260 and refutation metric 262 (e.g., Figure 4 (As illustrated in the example) to select various attributes of the annotation. For example, visual attributes such as text color, highlight color, etc., can be selected based on this comparison. For example, if a text fragment is determined to be refuted by the corresponding response content, a color such as orange or red can be selected. If a text fragment is determined to be confirmed by the corresponding response content, a color such as gray or blue can be selected, or even no color can be selected. In various implementations, the UX engine 136 can render the confirmed and / or refuted content as a selectable element, which, when selected, displays the response content 256 that forms the basis for the confirmation and / or refutation.

[0053] Figure 3A An example client device 310 in the form of a tablet computer is depicted for interacting with an NL-based response system 120. The client device 310 includes a display 370 on which a query input field 372 is rendered. A user (not depicted) has entered the query “Who was the first to land on the moon?” into the query input field 372 (either by typing or by having spoken speech recognized). Figure 2 The GM response 252 shares various characteristics with the GM response 352, which is generated by the GM response generation engine 128 and rendered on the display 370. The display 370 also renders thumbs-up and thumbs-down indicators that can be manipulated by the user to provide positive or negative feedback regarding the GM response 352, as well as operable graphical elements 374 that the user can select to initiate chosen aspects of this disclosure. Specifically, the user can manipulate element 374 to initiate… Figure 2 The process described in the text is used to perform textual implication analysis and provide visual annotations of the constituent text fragments of the GM response 352.

[0054] exist Figure 3B Element 374 has already been selected. Therefore, execution... Figure 2The process has been documented, and text fragment 376 of GM response 352 has been annotated. Text fragment 376 reads: “Armstrong's first words on the Moonwere, 'That's one small step for man, one giant leap for mankind.'” Text fragment 376 has been visually annotated because… Figure 2 The processing revealed the existence of data sources that tend to refute this statement. Text fragments can be visually annotated in various ways based on various combinations, such as confirmatory metric 260 and / or rebuttal metric 262. In some implementations, a first color, such as green, can be used, for example, via highlighting, font color, etc. wait Visual annotation can be used to annotate validated text fragments. Secondary colors such as red or orange can be used to visually annotate refuted text fragments. In some implementations, continuous or discrete color ranges can be used to visually annotate text fragments based on their position on the continuous or discrete spectrum between validated and refuted text fragments. In addition to or instead of font color, other visual annotations can be used, such as font type (e.g., Calibri, Times New Roman, Helvetica). wait ), font size, bold, underline, italic, underline wait .

[0055] In various implementations, the user can select text fragment 376 to cause a pop-up window 378 or other similar auxiliary interface to be rendered at the client device 310 (e.g., opening and, in some cases, scrolling to a new browser tab that refutes the content). Window 378 can depict information about the content, such as search results documents or excerpts / fragments from those documents that tend to refute text fragment 376. Figure 3BFor example, window 378 displays an excerpt from www.hypothetical.com that tends to refute the following text snippet 376: “Armstrong said that wasn't what he planned to say. Instead, Armstrong planned to say, 'That's one small step for 'A' man, one giant leap for mankind.' It's just that people didn't hear it.” Although in Figure 3B The text shows a counter-example, but it should be understood that similar annotations and auxiliary interfaces can be rendered (e.g., in different colors) and / or made available for verified text fragments.

[0056] Figure 4 This schematically depicts an example of the logic that can be implemented, for example, by a UX engine 136, to determine how to visually annotate a text fragment based on a confirmatory metric 260 and a rebuttal metric 262. Starting at the top, for a given text fragment ( Figure 2 254' in the middle) and the corresponding response content ( Figure 2 The highest confirmed score (256) in the middle. e This can be generated, for example, by the implication evaluation engine 134 using the confirmation model 135A. Similarly, the maximum dissent score... c It can be generated, for example, by the implication evaluation engine 134 using the refutation model 135B.

[0057] Then the overall "incorrectness" score can be calculated. i In some implementations, the incorrectness score is the maximum rebuttal score. c With the highest confirmed score e (or in some cases, 1- e The weighted average of the data is calculated between the given data and the provided data. Using a weighted average can account for cases where no strong rebuttal or strong supporting evidence was found in the documents or other responses. In some implementations, the following formula can be used, where... α The weights are determined from empirical evidence:

[0058] if i If it is greater than the first threshold, for example, 0.75, then... c Compare it to a second threshold—say, 0.9. Ifc If the value exceeds the second threshold, the given text fragment can be considered to be contradicted by the response content. Therefore, the UX engine 136 can, for example, highlight the given text fragment with a color such as red or orange, or otherwise visually emphasize it. However, if c If the value is less than the second threshold, a definitive or similar inclusion output of "no relevant content found" can be generated. In such cases, the given text fragment can be left unannotated, or it can be visually emphasized using a neutral color such as gray.

[0059] Returning to equation (1), if i If it is less than the first threshold (e.g., 0.75), then it can be... e Compare it to a third threshold—say, 0.9 (which may be the same as or different from the second threshold). If e If the value exceeds the third threshold, the given text fragment can be considered verified. Therefore, the given text fragment can be annotated at the client device (110, 310) with another color such as green or blue. However, if e If the text is below the third threshold, then the given text segment can be emphasized without visual emphasis. Figure 4 The thresholds depicted are for illustrative purposes only and are not intended to impose any restrictions.

[0060] Now go to Figure 5 A flowchart illustrating an example method 500 for implementing selected aspects of this disclosure is depicted. For convenience, the operation of method 500 is described with reference to a system performing the operation. This system of method 500 includes one or more processors, memories, and / or other components of a computing device (e.g., Figure 1 (The client device 110, one or more servers and / or other computing devices). Furthermore, although the operations of method 500 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0061] At box 501, for example via user input engine 111 of client device 110, the system can receive queries associated with a user-operated client device. For example, user input engine 111 can capture spoken utterances and transcribe them into a text query, and then provide the text query to NL-based response system 120. Alternatively, the user can type in a text query.

[0062] At box 503, for example via GM response generation engine 128, the system can use a generative model from database 129 to generate generative model output (e.g., 252, 352) based on the data indicating the query. The data indicating the query may include, for example, lexical units or sequences of embedded units generated therefrom, where lexical units include words, phrases, and / or characters of the query.

[0063] At box 505, for example, via fragment extraction engine 130, the system can extract multiple fragments (textual or otherwise) from the output of a generative model. At box 507, for example, via implication filtering engine 132, the system can, for example, use one or more classifier models from database 133 to classify a subset of fragments (e.g., 254') as suitable for textual implication analysis.

[0064] At box 509, for example via implication evaluation engine 134, the system can perform text implication analysis individually for each text segment of the subset. In various implementations, for each text segment of the subset, text implication analysis may include performing the operations associated with boxes 509A-C. At box 509A, for example, the system can formulate a search query based on the text segments of the currently considered subset. For example, words and / or phrases from the segments of the subset can be used to formulate the search query.

[0065] At box 509B, the system may retrieve at least one document or a portion thereof (or more generally, content) in response to the search query. At box 509C, the system may use one or more implication machine learning models 135 to process the considered text fragment and the at least one document retrieved at box 509B to generate one or more predictions as to whether the at least one document confirms or refutes the text fragment. For example, as Figure 2 As shown, both the confirmation model 135A and the rebuttal model 135B can be used, for example, to process text fragments and response content in parallel to generate confirmation metric 260 and rebuttal metric 262, respectively.

[0066] At box 511, for example via UX engine 136, the system can cause the rendering of natural language (NL) in response to a query at the client device. In some implementations, the NL in response to a query can initially be rendered without annotations and can then be updated with annotations at a later time (e.g., after a predetermined time interval, after time has been given to complete the techniques described herein, or in response to the use of a user interface element such as 375).

[0067] At box 513, for example via UX engine 136, the system can cause one or more annotations to be rendered at client device 110. In various implementations, the one or more annotations may (visually or audibly) express, convey, or otherwise represent one or more predictions of predictions for one or more text segments in a subset of text segments.

[0068] While the examples described herein primarily concern performing implication analysis on the output of a generative model, this is not intended to be limiting. In various implementations, the techniques described herein can be used to perform implication analysis on any type of document. For example, web pages can be created by anyone without any supervision, and therefore, inaccurate information spreads on the internet. The techniques described herein can be used to perform implication analysis on web pages and / or any other type of document such as text documents, (e.g., videos with transcripts that can be evaluated), etc. In some such cases, starting at box 505, the operations of method 500 can be performed, where the considered document replaces the output of the generative model.

[0069] Turn now Figure 6 This diagram depicts a block diagram of an example computing device 610 that can be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automation assistant component, and / or other components may include one or more components of the example computing device 610.

[0070] Computing device 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow user interaction with computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0071] User interface input device 622 may include a keyboard, pointing devices (such as a mouse, trackball, touchpad or graphics tablet, scanner, touchscreen integrated into the display), audio input devices (such as a voice recognition system, microphone), and / or other types of input devices. Generally, the term "input device" is used to include all possible types of means and methods for inputting information into computing device 610 or into a communication network.

[0072] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of means and methods for outputting information from computing device 610 to a user or another machine or computing device.

[0073] Storage subsystem 624 stores the functional programming and data constructs of some or all of the modules described herein. For example, storage subsystem 624 may include selected aspects and implementations that perform the methods disclosed herein. Figure 1 or Figure 2 The logic of the various components described in the document.

[0074] These software modules are typically executed separately by processor 614 or in combination with other processors. The memory 625 used in storage subsystem 624 may include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 for storing fixed instructions therein. File storage subsystem 626 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored in file storage subsystem 626 within storage subsystem 624, or in other machines accessible by processor 614. Bus subsystem 612 provides mechanisms for enabling the various components and subsystems of computing device 610 to communicate with each other as intended. Although bus subsystem 612 is schematically shown as a single bus, alternative implementations of bus subsystem 612 may use multiple buses.

[0075] The computing device 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, therefore... Figure 6 The description of the computing device 610 depicted herein is intended only as a specific example to illustrate some implementation methods. Many other configurations of the computing device 610 are possible, which have the same characteristics as... Figure 6 The computing device depicted in the document has more or fewer components compared to other computing devices.

[0076] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be given the opportunity to control whether programs or features collect user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Furthermore, some data may be altered before it is stored or used, resulting in the removal of personally identifiable information. For example, a user's identity may be processed to the point that their personally identifiable information cannot be determined, or, if geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level), making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0077] While several implementations have been described and illustrated herein, a variety of other means and / or structures may be utilized to perform functions and / or obtain results and / or one or more of the advantages described herein, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications for which this teaching is used. Those skilled in the art will recognize, or can determine, many equivalents of the particular implementations described herein using routine experimentation alone. Therefore, it should be understood that the foregoing implementations are presented by way of example only, and it should be understood that implementations may be practiced in ways other than those specifically described and claimed within the scope of the appended claims and their equivalents. Implementations of this disclosure relate to each individual feature, system, article of manufacture, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles of manufacture, materials, kits, and / or methods is included within the scope of this disclosure provided that such features, systems, articles of manufacture, materials, kits, and / or methods are not inconsistent with each other.

Claims

1. A method implemented by one or more processors, comprising: Receive queries associated with client devices operated by the user; Generative model output is generated based on the data indicating the query processed using a generative model; Extract multiple text fragments from the output of the generative model; Subsets of the text fragments are classified as suitable for textual implication analysis; Perform textual implication analysis separately on each text segment of the subset, wherein for each text segment of the subset, the textual implication analysis includes: Formulate a search query based on this text fragment. Retrieve at least one document in response to the search query, and One or more implication machine learning models are used to process the text fragment and the at least one document to generate one or more predictions as to whether the at least one document confirms or refutes the text fragment; This causes the rendering of the natural language (NL) in response to the query at the client device; and This causes one or more annotations to be rendered at the client device, wherein the one or more annotations express one or more predictions of the predictions for one or more text segments of the subset.

2. The method of claim 1, wherein the one or more implied machine learning models comprise: A validation machine learning model, which is trained to generate a first output indicating whether a document validates a text fragment; as well as A rebuttal machine learning model, which is trained to generate a second output indicating whether a document refutes a text fragment.

3. The method of claim 2, wherein one or more predictions in the predictions are determined based on a comparison of the first output and the second output.

4. The method of claim 3, wherein one or more visual attributes selected based on the comparison are used to render one or more annotations in the annotations.

5. The method of any of the preceding claims, wherein a classifier machine learning model is used to classify each text segment of the subset as suitable for textual implication analysis, the classifier machine learning model being trained to classify the text segments as capable or incapable of textual implication analysis.

6. The method of any of the preceding claims, wherein the text segment is classified as suitable for textual implication analysis based on an implication score predicted for each text segment of the subset using a regression machine learning model, the regression machine learning model being trained to predict the textual implication analysis suitability score of the text segment.

7. The method of any of the preceding claims, wherein the subset comprises a plurality of text segments, and the text implication analysis is performed on the plurality of text segments in parallel.

8. The method as described in any of the preceding claims, wherein the one or more annotations expressing one or more predictions in the predictions include: A first annotation, which visually highlights a text fragment in the text fragment that is verified by a document in the document, in a first color; as well as A second annotation, which visually highlights another text fragment in the text fragment that is refuted by one of the documents, in a second color different from the first color.

9. The method of any of the preceding claims, wherein a given annotation in the annotation expressing one or more predictions of the prediction is operable to retrieve at least a portion of the text segment under the given annotation that confirms or refutes the document.

10. The method of claim 9, further comprising causing a pop-up window to be rendered on the client device, wherein the pop-up window conveys the confirmation or rebuttal of the portion of the text fragment under the given comment.

11. The method of claim 9 or 10, further comprising causing a new web browser tab to be rendered at the client device, wherein the new web browser tab conveys all or part of the text segment under the given annotation, confirming or refuting the document.

12. The method of claim 11, wherein the new web browser tab is automatically scrolled to the location of the document containing the portion of the text fragment under the given annotation that confirms or refutes it.

13. The method of any of the preceding claims, further comprising causing one or more interactive feedback elements to be rendered at the client device, wherein the one or more interactive feedback elements are operable to accept or reject one or more predictions for one or more text segments of the subset of text segments.

14. The method as claimed in any of the preceding claims, wherein the NL in response to the query is rendered at the client device without any annotations before rendering the one or more annotations.

15. The method of any of the preceding claims, wherein one or more predictions are conditionally generated based on a response content quality metric determined for the at least one document to confirm or refute the text fragment.

16. A system comprising one or more processors and a memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to: Receive queries associated with client devices operated by the user; Generative model output is generated based on the data indicating the query processed using a generative model; Extract multiple text fragments from the output of the generative model; Subsets of the text fragments are classified as suitable for textual implication analysis; Perform textual implication analysis separately on each text segment of the subset, wherein for each text segment of the subset, the textual implication analysis includes: Formulate a search query based on this text fragment. Retrieve at least one document in response to the search query, and One or more implication machine learning models are used to process the text fragment and the at least one document to generate one or more predictions as to whether the at least one document confirms or refutes the text fragment; This causes the rendering of the natural language (NL) in response to the query at the client device; and This causes one or more annotations to be rendered at the client device, wherein the one or more annotations express one or more predictions of the predictions for one or more text segments of the subset.

17. The system of claim 16, wherein the one or more implied machine learning models comprise: A validation machine learning model, which is trained to generate a first output indicating whether a document validates a text fragment; as well as A rebuttal machine learning model, which is trained to generate a second output indicating whether a document refutes a text fragment.

18. The system of claim 17, wherein one or more predictions in the predictions are determined based on a comparison of the first output and the second output.

19. At least one non-transitory computer-readable medium, said at least one non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause said one or more processors to: Receive queries associated with client devices operated by the user; Generative model output is generated based on the data indicating the query processed using a generative model; Extract multiple text fragments from the output of the generative model; Subsets of the text fragments are classified as suitable for textual implication analysis; Perform textual implication analysis separately on each text segment of the subset, wherein for each text segment of the subset, the textual implication analysis includes: Formulate a search query based on this text fragment. Retrieve at least one document in response to the search query, and One or more implication machine learning models are used to process the text fragment and the at least one document to generate one or more predictions as to whether the at least one document confirms or refutes the text fragment; This causes the rendering of the natural language (NL) in response to the query at the client device; and This causes one or more annotations to be rendered at the client device, wherein the one or more annotations express one or more predictions of the predictions for one or more text segments of the subset.

20. The non-transitory computer-readable medium of claim 19, wherein the one or more implied machine learning models comprise: A validation machine learning model, which is trained to generate a first output indicating whether a document validates a text fragment; as well as A rebuttal machine learning model, which is trained to generate a second output indicating whether a document refutes a text fragment.

21. A method implemented using one or more processors, comprising: Receive queries associated with client devices operated by the user; LLM output is generated based on the data indicating the query processed using a large language model (LLM). Extract multiple text fragments from the LLM output; Perform textual implication analysis separately for each text segment, wherein for each text segment, the textual implication analysis includes: Formulate a search query based on this text fragment. Retrieve at least one document in response to the search query, and The text fragment and the at least one document are processed using a confirmatory machine learning model and a rebuttal machine learning model to generate a confirmatory metric and a rebuttal metric for the text fragment, respectively. This causes the rendering of the natural language (NL) in response to the query at the client device; and This causes one or more annotations to be rendered at the client device, wherein the one or more annotations include one or more visual features selected based on a comparison of the confirming metric and the rebutting metric.

22. The method of claim 21, wherein the one or more visual features include colors for highlighting one or more text segments in the text segment.

23. The method of claim 21 or 22, wherein the one or more visual features include colors for rendering one or more text segments in the text segment.

Images