Preventing Hallucination in Language Models by Mapping Facts as Explanations

Abstract

A computer-implemented method for improving the accuracy and reliability of a language model system includes encoding a query using a language model, the language model being pre-trained to generate responses to text queries and further trained to follow retrieved facts for queries related to domain text. The retrieved facts are a subset of text chunks of the domain text. One or more relevant facts to the encoded query are retrieved based on the text chunks from the domain text. A response to the encoded query is generated using the retrieved relevant facts. Example applications include, but are not limited to, use cases in materials informatics, data security, cyber threat intelligence, data extraction, digital transformation, and medical / healthcare.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO PRIOR APPLICATIONS Priority is claimed to European Patent Application No. EP 23153495, filed January 26, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to artificial intelligence (AI) and machine learning, and in particular to methods, systems, and computer-readable media for improving accuracy and preventing fact hallucination in language models. [Background technology]

[0003] Language models such as ChatGPT have attracted incredible attention from both academia and business due to their remarkable ability to generate fluent, compelling answers to human queries. However, these answers are all too often factually incorrect, which can be deceptive and potentially harmful to users and downstream applications, making such language models unusable in many real-world application settings. For example, Mok, Aaron, "Google's management has reportedly issued a 'code red' amid the rising popularity of the ChatGPT AI," Business Insider, Tech, online (December 21, 2022), notes that "ChatGPT is unable to fact-check what it says and is unable to distinguish verified facts from misinformation." As an example, note the following interaction between a human asking a question and a language model (ChatGPT) answer: Q (Human user): Why didn't Geoffrey Hinton win the Turing Award? A (Language Model): Geoffrey Hinton could not receive the Turing Award because it is not awarded posthumously. Hinton passed away in 2020, and the Turing Award can only be given to living individuals.

[0004] In fact, the answer in the language model (ChatGPT) is factually incorrect because Geoffrey Hinton did not receive the Turing Award and passed away in 2020. Worse, the answer is stated with complete certainty. Therefore, until a significant academic milestone, the use of such a language model is dangerous in real-world applications. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Mok, Aaron, “Google’s management has reportedly issued a ‘code red’ amid the rising popularity of the ChatGPT AI,” Business Insider, Tech, online (December 21, 2022) [Non-patent document 2] Galassi, Andrea et al., “Attention in Natural Language Processing,” IEEE Transactions on Neural Networks and Learning Systems, Vol. 32, No. 10 (October 2021) [Non-patent document 3] Jang, Eric et al., “Categorical Reparameterization with Gumbel-Softmax”, arXiv:1611.01144, ICLR (2017) [Non-patent document 4] Sukhbaatar, S. et al., "End-to-End Memory Networks," arXiv:1503.08895 (November 2015) [Non-patent document 5] Nakano, R. et al., "WebGPT: Browser-assisted question-answering with human feedback," arXiv 2112:09332 (June 2022) Summary of the Invention [Means for solving the problem]

[0006] In one embodiment, the present invention provides a computer-implemented method for improving the accuracy and reliability of a language model system. The method includes encoding a query using a language model, the language model being pre-trained to generate responses to text queries and further trained to follow retrieved facts about queries related to domain text. The retrieved facts are a subset of text chunks of the domain text. One or more relevant facts for the encoded query are retrieved based on the text chunks from the domain text. A response to the encoded query is generated using the retrieved relevant facts.

[0007] Embodiments of the present invention may be used in a variety of applications, including but not limited to material informatics, data security, cyber threat intelligence, data extraction, digital transformation, and medical / healthcare use cases.

[0008] The embodiments of the present invention will be described in further detail below based on the exemplary figures. The present invention is not limited to the exemplary embodiments. All features described and / or illustrated herein can be used alone or in different combinations in the embodiments of the present invention. The features and advantages of various embodiments of the present invention will become apparent from reading the following detailed description with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0009] [Figure 1] 1 illustrates generally a method and system for improving the accuracy and reliability of a language model according to an embodiment of the present invention; [Figure 2] FIG. 1 illustrates a schematic diagram of a training method and system architecture for training a language model, a fact searcher, and a compressor, and for generating modified responses of the language model according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram of an example processing system that may be configured to perform any of the operations disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments of the present invention improve the accuracy and reliability of language models, and in particular their computational capabilities to enhance their ability to generate fluent responses, while at the same time increasing confidence in the language model's output by overcoming the technical problem of hallucinating facts. Hallucinating facts refers to a language model's response being factually incorrect, which is impossible for a human to recognize from the response alone. Thus, by avoiding such hallucination, embodiments of the present invention improve language models' computational capabilities to be more accurate and reliable, and increased confidence in language models also enables their use to improve in some practical applications. In particular, embodiments of the present invention avoid hallucinating facts by first mapping to facts before generating a response.

[0011] Another example of language model hallucinating fact hallucination is given by the following chat between a human user and a language model (ChatGPT): Q (Human user): Who is the CEO of Twitter? A(Language Model): The CEO of Twitter is Elon Musk. Q (Human user): Since when? A (Language Model): Sorry, my information is inaccurate. Elon Musk is not the CEO of Twitter. My knowledge cutoff is 2021, and the current CEO of Twitter as of 2021 is Jack Dorsey. Q (Human user): So, who is the CEO of Twitter? A (Language Model): At my knowledge cutoff for 2021, the CEO of Twitter is Jack Dorsey. Q (Human user): What about Elon Musk? A (Language Model): Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, CEO, and chief engineer / designer of SpaceX, an early investor, CEO, and product architect of Tesla, Inc., founder of The Boring Company, and co-founder of Neuralink. He was also a co-founder and early co-chairman of OpenAI. Elon Musk is not the CEO of Twitter.

[0012] The language model's (ChatGPT) response is problematic for two reasons. First, if the ChatGPT designer's claim that it was trained on 2021 cutoff data is true, then Elon Musk did not acquire Twitter before October 2022, making the initial response factually incorrect. Second, if that claim is untrue, then the language model listened and learned from data from 2021 onward. This effectively raises significant technical issues regarding data leaks and poses significant risks to data privacy.

[0013] Therefore, the current computational capabilities of existing language modeling systems, such as ChatGPT, are technically unable to avoid fact hallucination and the generation of incorrect information, making such systems impractical or unusable for many higher-risk applications, such as medical AI, drug or treatment development, smart cities, public services, law enforcement, and detecting risks to public safety. Existing language modeling systems are particularly dangerous when true facts are unknown and / or queries are potentially dangerous, limiting their use to situations where humans can easily verify responses and where factual accuracy is less of an issue. Furthermore, many language models are large, computationally expensive, and burdensome to use. Embodiments of the present invention improve language modeling systems by helping to avoid fact hallucination while making them more accurate and reliable by using compression to reduce computational load and save computing power and resources. This also allows for further improvements in some applications, particularly higher-risk applications where language modeling systems have not previously been applied in practice. Moreover, the benefits of language model systems, such as advancing and accelerating natural language processing applications, are further enhanced by the improvements in accuracy and reliability imparted by embodiments of the present invention.

[0014] According to a first aspect, the present invention provides a computer-implemented method for improving accuracy and reliability of a language model system. The method includes encoding a query using a language model, the language model being pre-trained to generate responses to text queries and further trained to follow retrieved facts about queries related to a domain text. The retrieved facts are a subset of text chunks of the domain text. One or more related facts to the encoded query are retrieved from the domain text based on the text chunks. A response to the encoded query is generated using the retrieved relevant facts.

[0015] According to a second aspect, the present invention provides a method according to the first aspect, wherein the relevant facts are retrieved by a fact searcher trained to extract one or more of the text chunks that are known to be valid and / or have a shorter distance to the encoded query and / or higher attention compared to others of the text chunks.

[0016] According to a third aspect, the present invention provides a method according to the first or second aspect, wherein the fact searcher is a neural network trained to extract one or more text chunks with higher attention using a Gumbel-Softmax layer.

[0017] According to a fourth aspect, the present invention provides a method according to any of the first to third aspects, wherein the fact searcher is trained using an objective to bring the retrieved facts closer to their query counterparts.

[0018] According to a fifth aspect, the present invention provides a method according to any of the first to fourth aspects, further comprising using a language model to generate an initial response to the encoded query, and using the initial response to retrieve relevant facts.

[0019] According to a sixth aspect, the present invention provides a method according to any of the first to fifth aspects, further comprising the step of verifying that the generated response does not contradict one of the retrieved relevant facts.

[0020] According to a seventh aspect, the present invention provides a method according to any of the first to sixth aspects, further comprising the step of displaying to the user which of the text chunks of the domain text have been used as relevant facts retrieved to generate the response.

[0021] According to an eighth aspect, the present invention provides a method according to any of the first to seventh aspects, further comprising the step of concatenating the generated response and the encoded query to generate additional text or a follow-up query.

[0022] According to a ninth aspect, the present invention provides a method according to any of the first to eighth aspects, wherein the language model is trained to follow the retrieved facts using constraints and / or external rewards with the objective of increasing the likelihood of the generated response.

[0023] According to a tenth aspect, the present invention provides a method according to any of the first to ninth aspects, wherein constraints are incorporated directly into the language model to increase mutual information between generated responses and retrieved relevant facts.

[0024] According to an eleventh aspect, the present invention provides a method according to any of the first to tenth aspects, wherein external rewards are used in reinforcement learning based training of the language model, in which penalties are based on the distance between generated responses and retrieved relevant facts.

[0025] According to a twelfth aspect, the present invention provides a method according to any of the first to eleventh aspects, wherein a language model is trained to follow the retrieved facts by encoding a representation of each of text chunks of the domain text, queries related to the domain text, and a response from the language model, compressing the representations, retrieving retrieved facts from the text chunks of the domain text for each of the queries using the compressed representations, and updating the language model using an objective to configure the language model to follow the retrieved facts.

[0026] According to a thirteenth aspect, the present invention provides a method for producing a medicament for the treatment of a disease having an inflammatory bowel disease, the method comprising:

[0027]

number

[0028] It has the form of however,

[0029]

number

[0030] is the parameterization of the language model, S is the set of text chunks, and s i is the relevant text chunk, and h i is the encoded representation of the relevant text chunk, and q i is the corresponding query, and s j is a random chunk from a set S of text chunks, and h j is the encoded representation of the random chunk, and q r is a random query q r wherein d is a distance measure in the embedding space, E is the expectation to be minimized, and λ1 and λ2 are hyperparameters to be fine-tuned.

[0031] According to a fourteenth aspect, the present invention provides a computer system for improving the accuracy and reliability of a language model, the computer system comprising one or more hardware processors configured, alone or in combination, to enable the execution of a method according to any of the first to thirteenth aspects.

[0032] According to a fifteenth aspect, the present invention provides a tangible, non-transitory computer-readable medium having instructions thereon which, when executed by one or more processors, enable performance of a method according to any of the first to thirteenth aspects.

[0033] FIG. 1 schematically illustrates a method and system 100 for improving the accuracy and reliability of a language model B according to an embodiment of the present invention, comprising a compression component D and a fact retriever E. The language model B is trained with data from a database of domain text C, but is iteratively fine-tuned using compression of representations of retrieved facts after receiving a user query. A user interacts with the language model B via a control unit A, which acquires user input. The user input can be used as a query to the language model B or can be used to generate a query. The language model B provides a response to the control unit A, which can be used to generate a decision signal in response to the user input. For example, the user can utilize the language model's response to make a follow-up decision. In a different application, the user can also be another technical system or downstream application that utilizes the response or decision signal, for example, to generate automated decisions or actions for various machine learning tasks. The database of domain text C contains all facts accessible to the language model B. The compression component D is a module that compresses the representations of the query and associated "facts" returned by the language model B. The fact finder E is used to update and fine-tune the language model B.

[0034] According to one embodiment of the present invention, a method for improving the accuracy and reliability of a language model comprises the following steps: compressing representations of queries to and responses returned by the language model; retrieving facts about a query; iteratively fine-tuning a language model using the retrieved facts; Includes:

[0035] 2 shows a schematic diagram of a system architecture 200 according to an embodiment of the present invention. The pre-trained language model (LM) 202 with generation can be an existing or off-the-shelf LM that is trained with any known technique and is to be improved and further fine-tuned according to an embodiment of the present invention, but the model is trained on a set of sentences s i A text chunk s ∈ S contains all the text that can be accessed as evidence for the facts contained in the production. i ∈S as input. LM achieves text generation by creating tokens with maximum probability, usually after a softmax operation. First, the domain text is divided into text chunks, which can be, for example, sentences or paragraphs, but in this example, sentence s i , also referred to herein as the context. Then, for each sentence s i is encoded with LM202 with the existing pre-trained generation, and the LM-encoded representation h i ∈H206. For example, encoding can be done using multiple transformer blocks and multi-layer perceptrons and LM attention heads. The sentence is input to these blocks, which translate the input into vectors in the embedding space. During generation, sentence s is used as a factual scaffold for generating responses. i A subset of sentences s is selected, preferably just sentences s rather than the entire set due to space and memory constraints. i A subset of s is used, and the subset is selected by a sampling technique. Thus, for each sentence s selected, i There is a hidden expression h i can be stored and used in later steps. In many practical applications, the number of chunks and the hidden representation h i The size of h is too large to store easily. For this reason, embodiments of the present invention employ a compression 209 step. This step compresses the hidden representation h i The compressed representation of the chunk c while reducing the size of c and preserving semantic meaning and / or distance. i∈C 211.

[0036] In addition, each of them has a corresponding response r from LM202 with pre-trained generation qi The set of queries q associated with 208 i ∈Q 201 are each encoded with LM202 with pre-trained generation, and the LM-encoded representation h q 205. In addition, the LM202 with pre-trained generation also obtains the corresponding response r qi 208 is encoded into the LM encoded representation h rq 207. Generate the LM encoded representation of the query h q 205 and the LM-encoded representation of the response h rq 207 similarly undergoes a compression 209 step to obtain a compressed representation of the query c q 210 and a compressed representation of the response rq 212, which are the compressed representations of the chunks c i ∈C 211 serve as input to a fact finder 213 according to one embodiment of the present invention, which finds a set of text chunks s i ∈S cq Output 214. Set of text chunks s i ∈S cq 214 is input to the pre-trained generative LM 202 so that the fact detector 213 updates the parameterization of the generative LM 202, specifically so that the representations of queries and their corresponding responses are close in the embedding space. For example, the fact searcher 213 solves the objective of option 2.b. below to update the parameterization of the generative LM 202.

[0037] Thus, once all the domain text has been encoded and compressed, the training process is iteratively applied across the training set. For each case, the following steps are performed: 1. For each query q i 201 (e.g., a question) is encoded by LM202 with pre-trained generation, and the standard response given by LM202 with pre-trained generation is response r qi208 and response r qi 208 may or may not be true. 2. Then, the LM encoded representation h q 205, and the hidden representation of the query, as well as the LM-encoded response representation h rq Both the initial response hidden representations as 207 undergo a compression step 209. Then, a subset of the text chunks s i ∈S cq 214 is extracted as a relevant fact by the fact detector 213. To this end, the fact detector 231 can take different forms, for example: a. The fact detector 213 can be configured by training a neural network to retrieve a set of facts by learning, for example, by learning an attention layer between the query / expression / join concatenation and each text chunk (learning the linkage between the query and the relevant text chunk), and then retrieving the chunk with the highest attention, for example, by sampling from Gumbel-Softmax. Applying the attention mechanism and Gumbel-Softmax can be done by known methods (see, for example, Galassi, Andrea et al., "Attention in Natural Language Processing," IEEE Transactions on Neural Networks and Learning Systems, Vol. 32, No. 10 (October 2021), and Jang, Eric et al., "Categorical Reparameterization with Gumbel-Softmax," arXiv:1611.01144, ICLR (2017), each of which is incorporated herein by reference). b. The fact detector 213 can be constructed by mapping representations of both the retrieved facts (the text chunks on which the response is based) and the query into an invariant space. In the training data, it is known which facts correspond to which queries. Training and objective minimization are iteratively performed to map retrieved facts and corresponding queries for each response.

[0038]

number

[0039] Assuming a parameterization of the language model referred to as i and query q, query q and other chunks s j Under the constraint that they should be closer to each other compared to

[0040]

number

[0041] The goal is to fine-tune s i is a valid fact for query q. At the same time, this proximity is i and its representation h i A random query q that cannot be answered by r Specifically, the goal is to minimize the following objectives:

[0042]

number

[0043] However, the first paragraph is a legitimate expression of fact h i and the corresponding query q i The second term tries to reduce the distance between the query q i and random fact representations j ∈S\{s i}. Similarly, the third term increases the distance between the random query q r and legitimate facts i Expression of h i This term increases the distance between , and simulates negative sampling in the space of queries. d is the distance measure in the embedding space. E is the mathematical notation for expectation. λ1 and λ2 are hyperparameters that can be fine-tuned. The objective function is to minimize the expectation E.

[0044]

number

[0045] We update the LM by attempting to find a parameterization of the LM, denoted as

[0046] Additionally or alternatively, the objective of option 2.b. is to

[0047]

number

[0048] The scalar can be implemented as a two-level program with a master objective and a slave objective, compression. These two objectives complement each other to achieve the min-max procedure defined above.

[0049] Additionally or alternatively, for the desired distance in option 2.b., chunks i is the query q i Another supervisory signal (e.g., from another AI system) may be employed that determines how likely the AI ​​system is to answer a given query. The supervisory signal may take a wide variety of forms, depending, for example, on the type of other AI system, and may indicate the other AI system's determination of which facts are relevant to which query.

[0050] Additionally or alternatively, for the desired distance in option 2.b., chunks iAnother supervisory signal (e.g., from another AI system) may be employed that determines the extent to which the AI ​​system is likely to contradict or disagree with a particular response from the LM. As noted above, the supervisory signal can take a wide variety of forms, depending, for example, on the type of other AI system, and indicates the other AI system's determination of which facts are relevant to which queries.

[0051] Additionally or alternatively, for the desired distance in option 2.b., chunks i is the query q i Another supervisory signal (e.g., from another AI system) may be employed that determines the likelihood of contradicting or disagreeing with a particular response from a given LM. As noted above, the supervisory signal may take a wide variety of forms, depending, for example, on the type of other AI system, and may indicate the other AI system's determination of which facts are relevant to which queries.

[0052] Additionally or alternatively, for the desired distance in option 2.b., query q i The distance between each text chunk s i All chunks s that are closer than a given distance value d are measured to i , or the top k nearest chunks are returned. The same procedure described above may be followed, but the first response r q Using the query q as a starting point. i and response r q are compressed by joint compression or by retransmitting the concatenation through a language model. q , h rq and then, as above, the objectives are concatenated as

[0053]

number

[0054] and a slave objective compression, these two objectives complement each other to achieve the minimax procedure defined above.

[0055] The above options may be combined into a single purpose and used alternately to train the model of the fact searcher 213. This is particularly advantageous when some monitoring signals are cheaper or faster to obtain than others. Overall, during training of the fact detector 213, it is possible to retrieve the currently closest fact to the query and / or use the correct fact directly if this information is relevant to the query-answer pair.

[0056] The query and extracted text chunks are then concatenated. These are then used to generate a refined response r' q 204, which can be fed together through LM202 with pre-trained generation to generate a refined response r' q 204 is used as input to LM with pre-trained generation 202 in the next iteration. In concatenation, it is possible to mark separate parts if desired (e.g., Query: q, Fact 1: s1, Fact 2: s2, ...). To ensure that LM with pre-trained generation 202 follows the facts, the language model is updated with an objective to encourage this. This can be, for example, an objective to increase the maximum likelihood of the response with additional constraints. This constraint is specifically applied to encourage the generation of factual responses based on both the query and the facts and can be directly incorporated, or an external reward estimate is obtained that indicates how well the newly generated response maintains factuality based on the given facts and can be utilized in a reinforcement learning-based setup together with the likelihood of the generated response.

[0057] For either additional constraints or external rewards, the following options may be used as examples: - Additional constraints can, for example, increase the mutual information between the generated and extracted responses. For example, a constraint could be to reduce the negative expected logarithm of the conditional distribution q(Y|X) estimated by the model. - Additional constraints keep the variation between the generated responses and the basic facts to a minimum, with larger differences in distance in the input space (e.g., measured by token overlap) resulting in larger penalties. - Natural language reasoning is used to check whether the generated response contradicts any of the facts, which can be implemented using machine learning techniques such as attention-based methods that detect contradictions between hypotheses and given premises.

[0058] Different objectives can be combined by linear combination or by iteratively applying back and forth between different objectives.

[0059] According to embodiments of the invention, different options for the compression 209 step are possible: a. The identity function, if the size of the representation does not need to be reduced. b. A neural network that is learned end-to-end during the training process, is initially randomly initialized, and has output dimensions smaller than the input dimensions, for example by adding linear layers, each with a number of outputs smaller than the number of inputs. c. Locality-sensitive hash function. d. Other dimensionality reduction techniques.

[0060] At inference or prediction time, a query q is input to a further trained language model according to an embodiment of the present invention, and is asked to generate a factual answer without integrating imaginary or unfactual statements. To this end, the query q and the context s i The distance between each of the facts ∈ S is measured. This can be the Euclidean distance measured in the embedding space of the encoded representation. The facts closest to the query are considered to be the most relevant ones.

[0061] Embodiments of the present invention provide for avoiding the generation of irrelevant descriptions when an Oracle is unable to answer a query despite available context, for example, the following exemplary procedure may be used to avoid this: 1. Heuristic rejection: Here, d(q,s * )>ασ, then refuse to answer the query, except that

[0062]

number

[0063] ,σ,is the standard deviation of the distances between all contexts, and ,α,is a domain-specific, parameter chosen by experts. 2. Out-of-distribution detection of queries: Here, the free energy function of a query is

[0064]

number

[0065] is defined as, where:

[0066]

number

[0067] is the dot product between two vectors, and the free energy function E(q;S,LM) is non-stochastic (not normalized). From these energies, a simple rule is derived to distinguish between in-distribution and out-of-distribution samples. This sample rule takes the form:

[0068]

number

[0069] where δ is a threshold learned and tuned from queries within the distribution that have already been seen.

[0070] Contexts i If at least one of ∈S is successfully retrieved, that at least one context (a fact or set of facts that support the correctness of the response) is used to generate a response (the output of the LM system), and both the response and the retrieved context are displayed to the user. In contrast to Sukhbaatar, S. et al., "End-to-End Memory Networks," arXiv:1503.08895 (November 2015), which is incorporated herein by reference and describes a probabilistic mixture of components, embodiments of the present invention ensure that a context is either retrieved or not retrieved.

[0071] Based on the retrieved facts, a new response is generated. The initial or new response may be further fine-tuned using the retrieved facts. This response may be returned only if, for example, a natural language inference model detects no contradictions between the generated response and the retrieved facts. In some cases, explainable AI (XAI) methods may be used to highlight which tokens (words or syllables) in the query and context were used to generate which response tokens.

[0072] Embodiments of the present invention also provide for follow-up queries. In this case, the system may decide to generate a longer response by re-running the process with the initial response as a prefix, for example, if the end of the sequence symbol has not been generated or an externally configured minimum response length has not been reached. Once a response is returned, the system may receive a follow-up query in which the previous query and the generated response can serve as a prefix to the new query.

[0073] Embodiments of the present invention can be practically applied to bring about further improvements in several technical fields that have not been possible until now due to the practical illusions of existing language model systems. For example, embodiments of the present invention can be applied to document generation in fields such as public safety, smart cities, cyber threat intelligence, or in medical AI fields such as AI-assisted drug discovery or materials design and development.

[0074] In the document generation use case for knowledge work, the goal is to generate the required document, which may be, for example, a summary or an answer, given an input document and a query. The data source or "facts" for this use case is the set of input documents of interest, and the training data are one or more queries about these documents (e.g., "Summarize X," "Did Hinton win the Turing Award?") and the corresponding answers. According to the application of a method according to one embodiment of the present invention, the input documents are used to create a set of concepts (retrieved facts / contexts / chunks). The query is then used to retrieve related concepts and corresponding sentences. Based on this, a final answer is generated. The output is thus generated text that answers the query based on the facts retrieved from the input documents.

[0075] In a medical AI use case, such as AI-assisted drug discovery or prescription, the goal is to automatically read published biomedical documents and derive pharmaceuticals. The data source or "facts" for this use case are medical publications or reports, and the training data are one or more queries and corresponding answers about these documents. Application of a method according to an embodiment of the present invention generates a report of facts that summarizes and contrasts insights from several publications to answer a query. The query may be a description of a patient's symptoms and several medical tests. The output may therefore be a personalized treatment recommendation or a report on which medication a particular patient should receive, along with underlying relevant facts, allowing a human to review the explanation and relevant facts before agreeing to administer the medication to the patient.

[0076] In a materials informatics use case, the goal is to automatically read materials informatics documents or reports and derive new materials. The data sources or "facts" include relevant publications or reports, and the training data is one or more queries and corresponding answers about these documents. For example, in a document describing steel, the query might be "How do I make steel more resistant to corrosion?" Application of a method according to an embodiment of the present invention generates a factual report that summarizes and contrasts insights from several publications to answer the query. Thus, the output can be a material design description along with underlying related facts, allowing a human to review the description and related facts before agreeing to manufacture the material.

[0077] In use cases in public safety, threat detection, law enforcement, government, or corporate management, the goal is to automatically read all relevant case files (e.g., relating to a single individual or a single topic of interest). The data source or "facts" for this use case are case files and associated text reports, and the training data are one or more queries and corresponding answers about these documents. For example, if a document describes the security threat "denial of service," an exemplary query might be "How does denial of service work?" or "What are the main damages caused by denial of service?" Application of a method according to an embodiment of the present invention generates a factual report that summarizes and contrasts insights from the underlying case files to answer the query. Thus, the output can be a summarized report of the case files that answers a given query and allows, for example, determining which category or benefit an individual should be assigned to.

[0078] In one embodiment, the present invention provides a method for generating factually correct responses from a language model, the method comprising the following steps. 1) Collecting domain text divided into chunks to serve as a fact base. 2) Collecting a set of queries and corresponding gold responses. For example, the gold responses may be manually created and checked by an expert to ensure that the training and other training from step 1) are correct. 3) Obtaining a pre-trained language model capable of generating text responses. 4) Defining a process for how the language model encoding representation is compressible. 5) The language model encodes and compresses all the domain text. 6) Training the language model, compressor (if applicable), and fact searcher by: a. Encode the query. b. Generate the first response. c. Use the encoded query and initial response to retrieve relevant facts based on text chunks from the domain text. d. Use the encoded query and the retrieved relevant facts to generate a new response. e. Update the language model, compression, and fact retriever using the objectives to encourage final responses and according to the extracted facts. 7) At inference time, given a query, generate the following response: a. Encode the query. b. Generate an initial response (optional). c. Implement a method to determine whether the system finds any relevant facts. d. Use the encoded query and possibly the initial response to find relevant facts based on text chunks from the domain text. e. Use the encoded query and relevant facts to generate a refined response. f. Validate, for example using natural language reasoning methods, that the refined responses generated are consistent with relevant facts retrieved from the domain text. g. Optionally, run a feature attribution explainable AI method that highlights which query and context tokens were used to generate a particular response token. 8) If additional text must be generated or the user submits a follow-up query, a refined response can be concatenated to the query (optional).

[0079] Embodiments of the present invention provide the following improvements over existing technology: 1. During inference or prediction time, for each query, a set of facts from the domain dataset (potentially private and only usable in certain circumstances) is retrieved and used to generate a factual response (see steps 1), 3), 4), 5), and 7) of the method above). 2. A supervisory signal is provided to identify appropriate facts, which are used to generate factual responses (see steps 6)a-c of the method above), while training a language model based on the objective of considering the facts ensures that the refined responses take the retrieved facts into account (see steps 6)d and 6)e of the method above). 3. Make it possible to guarantee that a response is generated only if at least one relevant fact is found (see step 7)c. of the method above). 4. Be able to guarantee that the generated response is returned only if it is consistent with the facts retrieved (see step 7)f. of the method above). 5. In contrast to existing language modeling systems such as ChatGPT, methods and systems according to embodiments of the present invention retrieve facts that are used to generate the final response. In this way, the chance of hallucinating the answer is explicitly reduced using a corresponding training procedure. In addition, both the response and the retrieved facts are shown to a human user, thus allowing the user to perform their own checks, which may significantly improve the safety and applicability of such language modeling systems. 6. In contrast to WebGPT (see Nakano, R. et al., “WebGPT: Browser-assisted question-answering with human feedback,” arXiv 2112:09332 (June 2022), which is incorporated herein by reference and describes retrieving websites from a search engine before generating a response), methods and systems according to embodiments of the present invention can operate on privately owned data that is not part of the Internet and does not require the use of highly performant search engines. Additionally, WebGPT does not have a training objective that encourages the generation of factual responses, but instead relies on reinforcement learning to learn preferences that are inconsistent with factuality (in contrast to Improvement 2. described above). WebGPT also does not have a mechanism for the system to refuse to answer a query based on its lack of factual knowledge (in contrast to Improvement 3. described above).

[0080] In contrast to the end-to-end memory network described by Sukhbaatar, S., methods and systems according to embodiments of the present invention overcome two technical limitations. Specifically, unlike embodiments of the present invention, end-to-end memory networks (1) do not address the concept of hallucinations or how to avoid them (see Improvements 2 and 3 described above), and (2) assume a softmax distribution for contexts, making it impossible to track how much influence each context has on the answer (in contrast to Improvement 1 described above). In one embodiment, the present invention explicitly searches for contexts, or does not search for them, thereby allowing a human to verify what information was used to generate the answer.

[0081] 3, a processing system 300 may include one or more processors 302, memory 304, one or more input / output devices 306, one or more sensors 308, one or more user interfaces 310, and one or more actuators 312. The processing system 300 may represent each computing system disclosed herein.

[0082] The processor 302 may include one or more separate processors, each having one or more cores. Each of the separate processors may have the same or different architecture. The processor 302 may include one or more central processing units (CPUs), one or more graphics processing units (GPUs), circuitry (e.g., application specific integrated circuits (ASICs)), digital signal processors (DSPs), etc. The processors 302 may be mounted on a common substrate or on multiple different substrates.

[0083] Processor 302 is configured to perform a function, method, or operation (e.g., configured to effectuate the performance of a function, method, or operation) when one of one or more of the separate processors is at least capable of performing the operations that embody the function, method, or operation. Processor 302 may perform the operations that embody a function, method, or operation, for example, by executing code stored in memory 304 (e.g., interpreting a script) and / or passing data through one or more ASICs. Processor 302, and therefore processing system 300, may be configured to automatically perform any of the functions, methods, and operations disclosed herein. Thus, processing system 300 may be configured to execute any (e.g., all) of the protocols, devices, mechanisms, systems, and methods described herein.

[0084] For example, when this disclosure states that a method or device performs task "X" (or that task "X" is performed), such statement should be understood to disclose that processing system 300 is configurable to perform task "X." Processing system 300 is configured to perform a function, method, or operation when processor 302 is configured at least to perform that function, method, or operation.

[0085] The memory 304 may include volatile memory, non-volatile memory, and any other medium capable of storing data. The volatile memory, non-volatile memory, and any other type of memory may each include multiple different memory devices in multiple separate locations, each with a different structure. The memory 304 may include remotely hosted (e.g., cloud) storage.

[0086] Examples of memory 304 include non-transitory computer-readable media such as RAM, ROM, flash memory, EEPROM, any type of optical storage disk such as a DVD, Blu-Ray® disk, magnetic storage, holographic storage, HDD, SSD, or any medium that can be used to store program code in the form of instructions or data structures. Any of the methods, functions, and operations described herein may be embodied entirely in the form of tangible and / or non-transitory machine-readable code (e.g., interpretable script) stored in memory 304.

[0087] The input / output devices 306 may include any components for passing data, such as ports, antennas (i.e., transceivers), printed conductive paths, etc. The input / output devices 306 may enable wired communication via USB, DisplayPort, HDMI, Ethernet, etc. The input / output devices 306 may enable electronic, optical, magnetic, and holographic communication with suitable memory 304. The input / output devices 306 may enable wireless communication via WiFi, Bluetooth, cellular (e.g., LTE, CDMA, GSM, WiMax, NFC), GPS, etc. The input / output devices 306 may include wired and / or wireless communication paths.

[0088] The sensors 308 can capture physical measurements of the environment and report them to the processor 302. The user interface 310 can include a display, physical buttons, a speaker, a microphone, a keyboard, etc. The actuators 312 can enable the processor 302 to control mechanical forces.

[0089] Processing system 300 may be distributed. For example, some components of processing system 300 may reside on a remotely hosted network service (e.g., a cloud computing environment), while other components of processing system 300 may reside on a local computing system. Processing system 300 may have a modular design in which some modules include multiple features / functionality as shown in FIG. 3 . For example, an I / O module may include volatile memory and one or more processors. As another example, individual processor modules may include read-only memory and / or a local cache.

[0090] While the subject matter of the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary, and not limiting. The present invention is defined by the claims, and any statements herein that characterize the invention should likewise be considered illustrative or exemplary, and not limiting. It will be understood by those skilled in the art that changes and modifications may be made within the scope of the following claims, which may include any combination of features from the different embodiments described above.

[0091] The terms used in the claims should be interpreted to have the broadest reasonable interpretation consistent with the above description. For example, the use of the articles "a" or "the" when describing an element should not be interpreted as excluding a plurality of elements. Similarly, the term "or" should be interpreted as inclusive, such that it does not exclude "A and B," unless it is clear from the context or the preceding description that the statement "A or B" refers to only one of A and B. Furthermore, the statement "at least one of A, B, and C" should be interpreted as one or more of the group of elements consisting of A, B, and C, and should not be interpreted as requiring at least one of each of the listed elements A, B, and C, regardless of whether A, B, and C are categorically related or not. Furthermore, references to "A, B, and / or C" or "at least one of A, B, or C" should be interpreted to include only any one entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B, and C. [Explanation of symbols]

[0092] 100 systems 200 System Architecture 201 queries 202 Language Model (LM) 203 Domain Text 204 Sophisticated Response 205 LM encoded representation 206 LM encoded representation 207 LM encoded representation 208 response 209 Compression 210 Compressed Representation of Queries 211 Compressed Representation of Chunks 212 Compressed Representation of Response 213 Fact Searcher 214 text chunks 300 Processing System 302 processor 304 memory 306 Input / Output Devices 308 Sensor 310 User Interface 312 Actuator

Claims

1. 1. A computer-implemented method for improving the accuracy and reliability of a language model system, comprising: encoding a query using a language model that has been pre-trained to generate responses to text queries and that has been further trained to follow retrieved facts about the query related to a domain text, the retrieved facts being a subset of text chunks of the domain text; retrieving one or more related facts to the encoded query based on the text chunks from the domain text; generating a response to the encoded query using the retrieved relevant facts; A computer-implemented method comprising:

2. 10. The computer-implemented method of claim 1, wherein the relevant facts are retrieved by a fact searcher trained to extract one or more of the text chunks that are known to be legitimate and / or have a shorter distance to the encoded query and / or higher attention compared to others of the text chunks.

3. 3. The computer-implemented method of claim 2, wherein the fact retriever is a neural network trained to extract the one or more text chunks with higher attention using a Gumbel-Softmax layer.

4. 4. The computer-implemented method of claim 2 or 3, wherein the fact searcher is trained with an objective to make the retrieved facts more similar to their query counterparts.

5. 5. The computer-implemented method of claim 1, further comprising using the language model to generate an initial response to the encoded query and using the initial response to retrieve the relevant facts.

6. 6. The computer-implemented method of claim 1, further comprising verifying that the generated response is consistent with one of the retrieved relevant facts.

7. 7. The computer-implemented method of claim 1, further comprising displaying to a user which of the text chunks of the domain text were used as the retrieved relevant facts to generate the response.

8. 8. The computer-implemented method of claim 1, further comprising concatenating the generated response and the encoded query to generate additional text or a follow-up query.

9. 9. The computer-implemented method of claim 1, wherein the language model is trained to follow the retrieved facts using constraints and / or external rewards with the goal of increasing the likelihood of the generated response.

10. 10. The computer-implemented method of claim 9, wherein the constraints are directly incorporated into the language model to increase mutual information between the generated response and the retrieved relevant facts.

11. 10. The computer-implemented method of claim 9, wherein the external reward is used in reinforcement learning-based training of the language model, where a penalty is based on the distance between the generated response and the retrieved relevant facts.

12. The language model is encoding a representation of each of the text chunks of the domain text, the query related to the domain text, and a response from the language model; compressing said representation; using the compressed representation to retrieve the retrieved facts from the text chunks of the domain text for each of the queries; updating the language model using an objective to configure the language model to conform to the retrieved facts; 12. The computer-implemented method of claim 1, wherein the computer is trained to follow the retrieved facts by

13. The purpose is to [Equation 1] It has the form of however, [Equation 2] is a parameterization of the language model, S is a set of text chunks, and s i is the relevant text chunk, and h i is the encoded representation of the relevant text chunk, and q i is the corresponding query, and s j is a random chunk from the set S of text chunks, and h j is the encoded representation of the random chunk, and q r is a random query q r where d is the distance measure in the embedding space, E is the expectation to be minimized, and λ 1 and λ 2 13. The computer-implemented method of claim 1, wherein is a hyperparameter to be fine-tuned.

14. 1. A computer system for improving the accuracy and reliability of a language model, comprising one or more hardware processors, said hardware processors performing, singly or in combination, the following steps: encoding a query using a language model that has been pre-trained to generate responses to text queries and that has been further trained to follow retrieved facts about the query related to a domain text, the retrieved facts being a subset of text chunks of the domain text; retrieving one or more related facts to the encoded query based on the text chunks from the domain text; generating a response to the encoded query using the retrieved relevant facts; 1. A computer system configured to enable the execution of

15. A tangible, non-transitory computer-readable medium having instructions thereon, said instructions, when executed by one or more processors, performing the following steps: encoding a query using a language model that has been pre-trained to generate responses to text queries and that has been further trained to follow retrieved facts about the query related to a domain text, the retrieved facts being a subset of text chunks of the domain text; retrieving one or more related facts to the encoded query based on the text chunks from the domain text; generating a response to the encoded query using the retrieved relevant facts; 2. A tangible, non-transitory computer-readable medium that enables the execution of

Images