Multi-agent filtering retrieval-augmented generation

Abstract

A method may be performed using a multi-stage framework with multiple large language model based agents. The method may comprise obtaining a query and retrieving a plurality of candidate documents from one or more data sources based on the query. A first language model agent can generate, for each candidate document of the plurality of candidate documents, an initial response to the query using the respective candidate document, thereby producing a set of document-query-response triplets. A second language model agent can evaluate, for each document-query-response triplet of the set of document-query-response triplets, a relevance of the initial response based on the respective candidate document. A subset of the candidate documents can be selected based on the relevance for each of the plurality of candidate documents. A third language model agent can generate a response to the query using the subset of candidate documents.

Description

PATENT Attorney Docket No.: 079900-1520010-9543W001 Client Reference No.: 9543WO01MULTI- AGENT FILTERING RETRIEVAL- AUGMENTED GENERATIONCROSS-REFERENCES TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 708,196, filed October 16, 2024, which is herein incorporated by reference in its entirety' for all purposes.BACKGROUND

[0002] Large language models (LLMs) are widely used for natural language processing tasks, but they often generate inaccurate or outdated information, which is a phenomenon referred to as hallucination. This is particularly problematic in applications that require up-to- date, precise, or domain-specific knowledge.

[0003] Retrieval-augmented generation (RAG) systems have emerged as a solution by integrating real-time document retrieval to ground LLM responses. RAG systems identify relevant documents to be provided to the LLM along with a query, together comprising an expanded prompt.

[0004] However, standard RAG systems frequently encounter a significant technical challenge: the retrieval of irrelevant or noisy documents, which can degrade the qualify and reliability' of the generated responses. The presence of these noisy documents can mislead the LLM. reduce recall of relevant information, and ultimately result in lower performance across question answering and related tasks.

[0005] Embodiments of the disclosure address this problem and other problems individually and collectively.SUMMARY

[0006] Provided herein are systems, methods, and medium that can overcome the challenges associated with hallucination of large language models.

[0007] The disclosed technology may overcome the hallucination problem by developing a multi-agent retrieval-augmented generation (RAG) system. This may include methods for1801 10344V 1filtering out irrelevant content. The disclosed technology may include intelligently selecting relevant documents related to a user query, thereby eliminating unnecessary noise and reducing hallucination.

[0008] One embodiment is related to a method comprising a computer-implemented method for generating a response to a query. The method can be performed by a computer system, for example. The method can include obtaining a query. After obtaining the query, a plurality of candidate documents can be retrieved from one or more data sources based on the query. A first language model agent can generate, for each candidate document of the plurality of candidate documents, an initial response to the query using the respective candidate document, thereby producing a set of documcnt-qucry-response triplets. A second language model agent can evaluate, for each document-query -response triplet of the set of document-query -response triplets, a relevance of the initial response based on the respective candidate document: A subset of the candidate documents can be selected based on the relevance for each of the plurality of candidate documents. A third language model agent can generate a response to the query using the subset of candidate documents.

[0009] Another embodiment is related to system comprising a processor and a non-transitory computer readable medium comprising code, executable by the processor for performing the aforementioned method.

[0010] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The novel features of the invention are set forth with particularity' in the claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which:

[0012] FIG. 1 illustrates an example framework 100 related to large language models according to embodiments of the present disclosure, according to aspects of the disclosed technology.

[0013] FIG. 2 shows a hybrid diagram illustrating a response generation process that includes filtering according to aspects of the disclosed technology.2801 10344V 1

[0014] FIG. 3 shows a hybrid diagram illustrating a response generation process that includes filtering and ordering according to aspects of the disclosed technology’.

[0015] FIG. 4 shows a graph that illustrates effects of the random ordering of candidate documents at different ratios of noisy candidate documents to related candidate documents, according to aspects of the disclosed technology.

[0016] FIG. 5 shows a flowchart illustrating a response generation process according to aspects of the disclosed technology.

[0017] FIG. 6 shows a flo chart that illustrates a method relating to additional aspects of the quantification of document scoring, filtering, and ordering, according to aspects of the disclosed technology.

[0018] FIG. 7 A illustrates aspects of establishing an adaptive threshold, which may separate relevant and noise documents for a query, according to aspects of the disclosed technology.

[0019] FIG. 7B illustrates aspects of determining an optimal adaptive threshold based on ratios of noisy candidate documents to related candidate documents, according to aspects of the disclosed technology.

[0020] FIG. 7C illustrates aspects of relevance indicator distribution to select an adaptive threshold based on a set of related candidate documents, according to aspects of the disclosed technology.

[0021] FIG. 7D illustrates aspects of relevance indicator distribution to select an adaptive threshold based on a set of noisy candidate documents, according to aspects of the disclosed technology.

[0022] FIG. 8 shows a graph that illustrates experimental results of efficacy on QA benchmarks for a TriviaQA dataset, according to aspects of the disclosed technology7.

[0023] FIG. 9 shows a graph that illustrates experimental results of efficacy on QA benchmarks for a PopQA dataset, according to aspects of the disclosed technology.

[0024] FIG. 10 shows a graph that illustrates experimental results of efficacy on QA benchmarks for an ARC challenge dataset, according to aspects of the disclosed technology.

[0025] FIG. 11 show s a flow-chart illustrating an example method 1100 related to generating responses to queries, according to aspects of the disclosed technology7.3801 10344V 1

[0026] FIG. 12 shows a block diagram of an example computer system usable with systems and methods, according to aspects of the disclosed technology.TERMS

[0027] A “uniform distribution’7may refer to a probability distribution (e.g., probability density function or probability mass function) where possible values associated with a random variable are equally possible. A fair dice is an example of a system corresponding to a uniform distribution (in that the probability of any two rolls are equal). The term “non-uniform distribution” may refer to a probability distribution where all possible values or intervals are not equally possible. A Gaussian distribution is an example of anon-uniform distribution.

[0028] “Classification” may refer to a process by which something (such as a data value, feature vector, etc.) is associated with a particular class of things. For example, an image can be classified as being an image of a dog. “Anomaly detection” can refer to a classification process by which something is classified as being normal or an anomaly. An “anomaly” may refer to something that is unusual, infrequently observed, or undesirable. For example, in the context of email communications, a spam email may be considered an anomaly, while a nonspam email may be considered normal. Classification and anomaly detection can be carried out using a machine learning model.

[0029] The term “artificial intelligence model” or “machine learning model” can include a model that may be used to predict outcomes to achieve a pre-defined goal. A machine learning model may be developed using a learning process, in which training data is classified based on known or inferred patterns.

[0030] "Machine learning" can include an artificial intelligence process in which software applications may be trained to make accurate predictions through learning. The predictions can be generated by applying input data to a predictive model formed from performing statistical analyses on aggregated data. A model can be trained using training data, such that the model may be used to make accurate predictions. The prediction can be, for example, a classification of an image (e.g., identifying images of cats on the Internet) or as another example, a recommendation (e.g., a movie that a user may like or a restaurant that a consumer might enjoy).

[0031] A “machine learning model” may include an application of artificial intelligence that provides systems with the ability' to automatically learn and improve from experience without4801 10344V 1explicitly being programmed. A machine learning model may include a set of software routines and parameters that can predict an output of a process (e.g., identification of an attacker of a computer network, authentication of a computer, a suitable recommendation based on a user search query, etc.) based on feature vectors or other input data. A structure of the software routines (e.g., number of subroutines and the relation between them) and / or the values of the parameters can be determined in a training process, which can use actual results of the process that is being modeled, e.g., the identification of different classes of input data. Examples of machine learning models include support vector machines (SVM), models that classify data by establishing a gap or boundary between inputs of different classifications, as well as neural networks, collections of artificial “neurons” that perform functions by activating in response to inputs. A machine learning model can be trained using “training data” (e.g., to identify patterns in the training data) and then apply this training when it is used for its intended purpose. A machine learning model may be defined by “model parameters,” which can comprise numerical values that define how the machine learning model performs its function. Training a machine learning model can comprise an iterative process used to determine a set of model parameters that achieve the best performance for the model. Example supervised learning models may include different approaches and algorithms including analytical learning, artificial neural network, backpropagation, boosting (meta-algorithm), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc ), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), random forests, ensembles of classifiers, ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn, a multicriteria classification algorithm.

[0032] The model may include linear regression, logistic regression, convolutional neural network (CNN), deep recurrent neural network (e.g., fully-connected recurrent neural network (RNN), Gated Recurrent Unit (GRU), long short-term memory, (LSTM)), transformed-based methods (e.g. XLNet, BERT, XLM, RoBERTa), hidden Markov model (HMM), linear discriminant analysis (LDA), k-means clustering, density-based spatial clustering of5801 10344V 1applications with noise (DBSCAN), random forest algorithm, adaptive boosting (AdaBoost), extreme Gradient Boosting (XGBoost), support vector machine (SVM), or any model described herein. A machine learning model can be trained using '‘training data” (e.g., to identify patterns in the training data) and then apply this training when it is used for its intended purpose. A machine learning model may be defined by “model parameters,” which can comprise numerical values that define how the machine learning model performs its function. Training a machine learning model can comprise an iterative process used to determine a set of model parameters that achieve the best performance for the model. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques (e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques).

[0033] An “agent” can include an independently operating program. An agent can be can be an language model agent. An language model agent can include a type of Al agent that uses a large language model as its core reasoning engine.

[0034] A “query” can include a question or a request for information. A query can include a text input by a user. A query can include a question to which a user is requesting a response. A query can include an input or instruction given to a large language model to generate a response.

[0035] A “response” can include something constituting a reply or a reaction. A response can be created due to a query’. A response can include an output from a large language model based on a query.

[0036] A '‘document” can include something that contains information. A document can be a computer file containing information. A document can include datasets, articles, descriptions, and / or other written information.

[0037] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can include a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, a server computer can include a database server coupled to a w eb server. The server computer may comprise one or more computational apparatuses and may use any of a variety7of computing structures, arrangements, and compilations for servicing the requests for one or more “client computers.”6801 10344V 1

[0038] A “memory” may include any suitable device or devices that may store electronic data. A suitable memory may comprise a non-transitory computer readable medium that stores instructions that can be executed by a processor to implement a desired method. Examples of memories include one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation.

[0039] A “processor” may include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU that comprises at least one high-speed data processor adequate to execute program components for executing user and / or system generated requests. The CPU may be a microprocessor such as AMD’s Athlon, Duron and / or Opteron; IBM and / or Motorola’s PowerPC; IBM’s and Sony’s Cell processor; Intel’s Celeron, Itanium, Pentium, Xenon, and / or Xscale; and / or the like processor(s).DETAILED DESCRIPTION

[0040] Provided herein are systems, methods, and medium that can overcome the challenges associated with outdated or incorrect information generated in outputs by large language models (LLMs).

[0041] The disclosed multi-agent retrieval-augmented generation (RAG) framework may use language model agents as predictors, document judges, and final predictors. In response to a query, several candidate documents may be retrieved for the query. A first language model agent, referred to as the predictor agent, may determine initial responses to the query based on each candidate document. For each candidate document, a document-query-response triplet data structure may be generated based on the outputs of the first agent. This triplet data structure may then be provided to the second agent, which acts as the judge agent, to evaluate the relevant information.

[0042] The second language model agent (e.g., a judge agent) can be prompted to provide a relevance indicator that indicates how relevant (e.g., an amount of relevance) each candidate document is to the query and initial response. The relevance indicator can be a binary value, such as "yes" or "no", or can be a numerical value, such as a probability or other derived value. In this manner, the judge agent may be used for filtering and ordering the candidate documents.

[0043] After the judge agent filters out noisy (e.g., irrelevant) candidate documents and, in some embodiments, orders the remaining candidate document list by their relevance indicators7801 10344V 1that indicate the relevance of the candidate document, the third language model agent (e.g., a final-predictor agent) may be used to generate a response to the query utilizing the remaining candidate documents (e.g., a subset of candidate documents). This multi-agent approach ensures that only the most relevant content is used in the final response generation process.

[0044] Embodiments include the use of multiple language model agents to collaboratively filter and score retrieved documents. Such a process allows for a training-free modification to a standard RAG-based system. Embodiments can be utilized without needing to retrain a large language model with billions of data points or to perform a fine-tuning process.

[0045] This training-free solution not only improves the recall of relevant information but also dynamically adapts to different queries, making it a versatile tool for a wide range of applications, as the various applications do not require fine-tuning to train an LLM with domain-specific data. The method enhances the reliability of the RAG system by minimizing the inclusion of noise and improving the overall recall of relevant documents. Collaborative filtering by multiple agents, combined with a dynamic and adaptive filtering mechanism, results in notable improvements in system performance and robustness, directly addressing critical issues in retrieval-augmented generation.

[0046] To further enhance adaptability and robustness, embodiments provide for an adaptive threshold that dynamically adjusts to different queries based on the distribution of relevance indicators. This adaptive threshold can adjust the relevance filtering threshold according to relevance indicator distributions, ensuring robust performance across diverse queries. This dynamic adjustment leads to a versatile and reliable system that maintains high performance across different query types, in contrast to systems that utilize static filtering thresholds.

[0047] Experimental results related to embodiments illustrate that the framework consistently outperformed existing RAG systems, demonstrating performance improvements ranging from 2 to 6 times across various question-answering (QA) benchmarks.I. RETRIEVAL AUGMENTED GENERATION

[0048] FIG. 1 shows an example framework 100 that illustrates a retrieval-augmented generation process. The framework 100 can include a pre-trained LLM that can generate a response to a query based on retrieved documents. The steps described in reference to FIG. 1 can be performed by a computer system.8801 10344V 1

[0049] Within FIG. 1, the process can begin with the submission of a query 102. The query' 102 may include a question or a request for information. The query’ 102 can originate from a user or another system component. For example, a user device (not shown) can provide the query 102 to the computer system.

[0050] As an illustrative example, a user can input, into the user device, a question of “Which team won the latest FIFA World Cup championship?” The user device can create a query' comprising the question. The user device can provide the query to the computer system.

[0051] After obtaining the query’ 102. the query 102 can be directed to a retriever module 104. The retriever module 104 can be a module in the computer system. The retriever module 104 can process the query' 102 and identify documents 108 from a database 106 that are related to the query 102.

[0052] In some embodiments, the retriever module 104 can query the database 106 to identify and extract one or more documents 108 that are related to the query 102. The selection criteria utilized by the retriever module 104 may include keyword matching, semantic analysis, metadata filtering, or other information retrieval techniques. The resulting documents 108 represent the subset of information identified in the database 106 as most relevant to the query’ 102.

[0053] For example, upon receiving the query 102, the retriever module 104 can perform a search operation within the database 106. The database 106 can be a structured repository' that may contain a plurality7of documents. A document can include datasets, articles, descriptions, and / or other written information.

[0054] In other embodiments, the retriever module 104 determine a query embedding for the query 102 and compare the query’ embedding to a plurality of document embeddings in a database to determine relevant documents. For example, when the retriever module 104 receives the query 102, the retriever module 104 can generate a query embedding for the query 102.

[0055] The query embedding can be vector representation that represents the query 102. The retriever module 104. or other suitable module, can generate the query embedding based on the query 102 using a large language model embedding process. For example, the query' 102 can be transformed into a vector using a pre-trained embedding model (e.g., Open Al’s Ada, Sentence-BERT, etc.).9801 10344V 1

[0056] After obtaining the query embedding, the retriever module 104 can compare the query' embedding to a plurality of document embeddings in a database that represent, and are associated with, documents. Each document embedding can be pre-generated using the same or similar pre-trained embedding model to create document embeddings from documents. In some embodiments, the document embeddings can be indexed in the database 106 using a prebuilt document index (e.g., stored in a vector database such as those provided by Pinecone, Weaviate, Qdrant, etc.). The retrieval of the documents can be performed using an approximate nearest neighbor (ANN) search that compares the query embedding to the document embeddings.

[0057] In some embodiments, the retriever module 104 can identify a top N number of documents in the database 106 related to the query 102. In other embodiments, the retriever module 104 can identify’ any number of documents in the database 106 that are within a particular matching threshold of the query 102. For example, the retriever module 104 can obtain the plurality7of candidate documents based on the comparison of the query embedding to the plurality of document embeddings, where each candidate document of the plurality of candidate documents is stored in association with a document embedding.

[0058] Following the retrieval of the documents 108, the process continues to the response generation phase. A pre-trained LLM 110 can process the documents 108 in conjunction with the query’ 102. The pre-trained LLM 110 can include a machine learning model that has been previously trained on diverse textual corpora, enabling it to comprehend, synthesize, and generate text.

[0059] The pre-trained LLM 110 may be based on transformer architectures, such as GPT, BERT, or similar models. The pre-trained LLM 1 10 can be trained on hundreds of billions of words from diverse textual sources including, for example, technical literature, legal documents, scientific papers, publicly available datasets, etc. This pre-training can provide the pre-trained LLM 110 with an understanding of language, context, and knowledge, enabling it to perform text generation.

[0060] Using the context provided by the related documents 108 and the query 102, the pretrained LLM 110 can generate a response 112. The computer system can generate a prompt (e.g., a composite input prompt) that concatenates or otherwise includes both the query 102 and the documents 108, ensuring that the pre-trained LLM 110 has access to all necessary contextual information. Such a process can also include other prompt engineering steps. For10801 10344V 1example, the computer system may also format inputs to fit within the model’s context window, manage token limits, and / or apply other heuristics for information prioritization. The pretrained LLM 110 can generate the response 112 based on the documents 108 and the query 102. The response 112 may take the form of an answer to the question of the query 102.

[0061] In some embodiments, after the pre-trained LLM 110 generates the response, the computer system can perform any suitable post-processing techniques, such as response truncation, formatting, and / or validation against any predefined criteria (e.g., a criteria imposed by a guardrail).

[0062] After generating the response 1 12, the computer system can provide the response 112 to the user device in response to the query 102.

[0063] The framework 100 depicted in FIG. 1 illustrates a system where the query7102 is processed through a retriever module 104, which interacts with the database 106 to obtain documents 108. These documents are subsequently utilized by the pre-trained LLM 110 to generate the response 112.

[0064] Framework 100 may include certain limitations. These limitations include the susceptibility7of the pre-trained LLM 110 in the framework 100 to generate outdated or incorrect information. This phenomenon may be referred to as ‘‘hallucination.’' This is especially problematic when LLMs are used in critical applications where accurate and up-to- date information is essential.II. MULTI-AGENT RETRIEVAL AUGMENTED GENERATION

[0065] Embodiments provide for a training-free, multi-agent filtering framework that leverages multiple LLM agents to collaboratively assess, score, and filter retrieved documents. This approach improves the recall of relevant documents and robustly7reduces the inclusion of noise, all without the need for retraining or fine-tuning the LLMs. The invention also features an adaptive, dynamic filtering mechanism that uses a score-based threshold (e.g., an adaptive threshold), which adjusts according to the distribution of document relevance indicators for each query. This ensures that the filtering process is both robust and versatile, consistently improving performance across diverse question-answering benchmarks.

[0066] Various aspects of a multi-agent retrieval augmented generation process will be described in reference to FIG. 2, FIG. 3, and FIG. 5. FIG. 2 shows a flowchart illustrating a11801 10344V 1response generation process that includes filtering. FIG. 3 shows a flowchart illustrating a response generation process that includes filtering and ordering. FIG. 5 shows a flowchart illustrating a response generation process according to aspects of the disclosed technology. FIG. 5 includes a multiple stage process to identify, obtain, and rank documents as well as provide the documents to an LLM for analysis, according to aspects of the disclosed technology. The methods illustrated in FIG. 2, FIG. 3, and FIG. 5 can be performed by a computer system.A. Agents

[0067] In addition to the document retrieval system described above, embodiments provide for agents (e.g., language model agents) that identify, evaluate, and process documents and queries. These agents can operate on retrieved documents to enhance the quality, relevance, and accuracy of information provided in responses to queries. An agent can be implemented as a language model agent, wherein the core reasoning capabilities are powered by a large language model. A language model agent can include an artificial intelligence entity that utilizes the natural language processing and understanding capabilities of an LLM to perform sophisticated analytical and evaluative tasks on the documents and queries.

[0068] The methods illustrated in FIG. 2, FIG. 3, and FIG. 5 include three distinct language model agents, each tasked with a specific role in the document refinement and response generation process. The language model agents depicted in FIG. 2, FIG. 3, and FIG. 5 include a predictor agent (e.g., as a first agent), a judge agent (e.g., a second agent), and a final-predictor agent (e.g., a third agent). The collaborative operation of these language model agents forms a multi-stage pipeline for processing, evaluating, and synthesizing information derived from the retrieved documents, thereby ensuring a robust and contextually accurate final output response.

[0069] In some embodiments, the first language model agent, the second language model agent, and the third language model agent can be instantiated using the same pre-trained language model.B. Filtering

[0070] Embodiments provide for systems and methods of filtering obtained candidate documents into a subset of candidate documents that can be utilized for generation of a response for a query.12801 10344V 1

[0071] FIG. 2 shows a hybrid diagram illustrating a response generation process that includes filtering according to aspects of the disclosed technology’. FIG. 2 includes a method 200 where a computer system can obtain candidate documents from a database, filter the candidate documents, and generate a response to a query7based on the filtered candidate documents.

[0072] The computer system can obtain a query 202. The query’ 202 can include a question. The computer system can utilize a retriever module 204 to query' a database 206 to obtain a plurality of candidate documents 208 that are related to the query7202. The computer system can obtain the plurality7of candidate documents 208 as described in reference to FIG. 1, for example.

[0073] After obtaining the plurality7of candidate documents 208 from the database 206 based on the query7202, the computer system can provide the plurality7of candidate documents 208 and the query 202 to a first language model agent 210 (e.g., a predictor agent).

[0074] The first language model agent 210 can include a machine learning model, such as a large language model, that can process inputs and determine outputs. The first language model agent 210 can individually process each candidate document of the plurality of candidate documents 208. For example, the first language model agent 210 can accept a first candidate document in conjunction with the query 202 as input. The first language model agent 210 can generate a first initial response to the query 202 based on the first candidate document. The first language model agent 210 can also accept a second candidate document in conjunction with the query7202 as input. The first language model agent 210 can generate a second initial response to the query 202 based on the second candidate document. The first language model agent 210 can also accept a third candidate document in conjunction with the query 202 as input. The first language model agent 210 can generate a third initial response to the query 202 based on the third candidate document.

[0075] The first language model agent 210 can process each combination of the query7and a candidate document from the plurality of candidate documents 208 in parallel or in series.

[0076] The first language model agent 210 can generate a document-query-initial response triplet data structure 212 from the query 202, each candidate document of the plurality of candidate documents 208, and each generate initial response. The document-query-initial response triplet data structure 212 (also referred to as a triplet) can include entries for each candidate document and corresponding initial response.13801 10344V 1

[0077] After generating the document-query-initial response triplet data structure 212, the computer system can provide the document-query-initial response triplet data structure 212 to a second language model agent 214.

[0078] The second language model agent 214 can filter the plurality of candidate documents into a subset of candidate documents. The second language model agent 214 can filter the plurality of candidate documents based on the data in the document-query-initial response triplet data structure 212. For example, the second language model agent 214 can filter the plurality of candidate documents based on how well a particular initial response relates to the corresponding candidate document and query 202.

[0079] The second language model agent 214 can iteratively determine a relevance indicator (e.g., a relevance score) for each entry in the document-query-initial response triplet data structure 212. For example, the second language model agent 214 can receive an input prompt that includes the query 202, a first candidate document, and a first initial response. The prompt can include a question or other text based input that prompts a large language model of the second language model agent 214 to output a relevance indicator that indicates whether or not the initial response is relevant to the query 202 and / or the corresponding candidate document. The second language model agent 214 can determine a binary relevance indicator (e.g., "Yes ’ or “No”) or a numerical relevance indicator (e.g., a value in a range of 0-1).

[0080] The second language model agent 214 can determine a plurality' of relevance indicators 216. The plurality of relevance indicators 216 can include a number of relevance indicators that is equal to the number of candidate documents in the plurality of candidate documents 208.

[0081] The computer system can filter the plurality of candidate documents 208 using the plurality of relevance indicators 216. If the relevance indicators are binary, then the computer system can filter the plurality of candidate documents 208 by removing candidate documents that are associated with a relevance indicator of “No” (or other suitable binary representation). The computer system can obtain a subset of candidate documents 218 by filtering candidate documents out of the plurality of candidate documents 208 using the relevance indicators.

[0082] As an illustrative example, the plurality of candidate documents 208 can include a first candidate document, a second candidate document, and a third candidate document. The second language model agent 214 can determine three relevance indicators, one for each candidate document. The first candidate document can be associated with a relevance indicator14801 10344V 1of “Yes,” the second candidate document can be associated with a relevance indicator of “No,” and the third candidate document can be associated w ith a relevance indicator of “Yes.” The computer system can obtain the subset of candidate documents 218 by removing the second candidate document from the plurality of candidate documents 208. In this example, the resulting subset of candidate documents 218 can include an unordered list that includes the first candidate document and the third candidate document.

[0083] In some embodiments, if the relevance indicators are numerical relevance indicators, then the computer system can filter the plurality of candidate documents 208 by removing candidate documents that are below a particular threshold value (e.g., 0.4, 0.5, 0.7, 0.9, etc.). The computer system can obtain a subset of candidate documents 218 by filtering candidate documents out of the plurality of candidate documents 208 using the relevance indicators. In some embodiments, the computer system can determine an adaptive threshold rather than utilizing a static threshold to filter candidate documents. The computer system can determine the adaptive threshold based on a mean of the distribution of relevance indicators, a standard deviation of the distribution of relevance indicators, or other suitable statistical values. Determination of the adaptive threshold is described in further detail herein.

[0084] After obtaining the subset of candidate documents 218, the computer system can provide the subset of candidate documents 218 as well as the query 202 to a third language model agent 220.

[0085] The third language model agent 220 can accept the subset of candidate documents 218 and the query 202 as input to a large language model and can generate a response 222. The response 222 can be a response to the query 202 based on information included in each candidate document in the subset of candidate documents 218.C. Filtering and Ordering

[0086] Embodiments provide for systems and methods of ordering candidate documents that have been obtained and filtered.

[0087] FIG. 3 shows a hybrid diagram illustrating a response generation process that includes filtering and ordering according to aspects of the disclosed technology7. Elements 302-312 are similar to elements 202-212 of FIG. 2, and will not be repeated here.15801 10344V 1

[0088] After obtaining a document-query -initial response triplet data structure 312 from a first language model agent 310, a second language model agent 314 can score each entry in the document-query-initial response triplet data structure 312. The second language model agent 314 can score how well each first language model agent 310 generated initial response relates to a corresponding candidate document of a plurality of candidate documents 308 and a query7302. To score the triplets, the second language model agent 314 can generate a relevance indicator for each entry in the document-query -initial response triplet data structure 312.

[0089] Prior to describing the ordering and filtering of candidate documents using relevance indicators, technical problems of how to create relevance indicators that allow for sorting of candidate documents will be discussed.

[0090] FIG. 4 shows a graph that illustrates effects of the random ordering of candidate documents at different ratios of noisy candidate documents to related candidate documents, according to aspects of the disclosed technology. Embodiments face and solve a technical problem of how to create a relevance indicator such that candidate documents can be ordered in a meaningful manner. Embodiments provide for such relevance indicators and ordering methods.

[0091] Research has observed that when processing long context inputs, LLMs tend to overlook information in the middle, placing greater emphasis on the beginning and end of the context (Nelson F Liu, et al., 2024, Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 11 : 157-173). This suggests that in RAG, the ordering of documents may influence prediction performance. To investigate the impact of document order in RAG. an experiment was conducted on a benchmark RGB dataset (Jiawei Chen, et al., 2024, Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754-17762), during which retrieved documents were randomly shuffled and evaluated. This process was repeated ten times for each noise ratio condition (e.g., 40% noisy documents, 60% noisy documents, and 80% noisy documents).

[0092] The results of the experiment, illustrated in FIG. 4, reveal that document order has a significant effect on performance. The maximum performances are substantially higher than the minimum ones, suggesting that certain document orders can provide stable and optimal results. To solve such technical problems, embodiments can order the filtered candidate documents.16801 10344V 1

[0093] To quantify the binary natural language outputs of "Yes" and "No" for a candidate document's relevance, embodiments can generate a numerical relevance indicator. The computer system can generate the relevance indicator based on numerical values output from the second language model agent 314.

[0094] For example, for a candidate document, the second language model agent 314 can output a first probability value that represents a probability of the candidate document being classified as “Yes” and can output a second probability value that represents a probability of the candidate document being classified as “No.”

[0095] The computer system can utilize the probability values to generate a relevance indicator. The computer system can generate the relevance indicator for the candidate document based on the first probability value and the second probability' value.

[0096] For example, the computer system can determine a difference between the log probabilities of the corresponding classifications. For example, the computer system can determine:Relevancelndicator = Log (F ir st Probability Value) — Log SecondProbabilityValue)

[0097] Since these are log probabilities, the difference can represents the logarithm of the ratio of the probabilities of the "Yes" classification to the "No" classification. By subtracting the log probabilities of the two classifications, the computer system can simplify the judgment process by consolidating the two factors into a single relevance indicator. This relevance indicator can sen e as the criterion for candidate document filtering.

[0098] Returning to FIG. 3, for each candidate document of the plurality of candidate documents 208, the computer system can generate a prompt comprising the query 302, the candidate document, and the corresponding initial response. The prompt can indicate the second language model agent 314 to generate a relevance indicator. The second language model agent 314 can determine a relevance indicator for each candidate document.

[0099] For each document-query-response triplet of the set of document-query-response triplets, the second language model agent can evaluate a respective relevance indicator of the initial response based on the respective candidate document. Evaluating the respective relevance indicator of the initial response can include generating a relevance indicator for the document-query-response triplet.17801 10344V 1

[0100] In some embodiments, the relevance indicator can be a value determined by the large language model of the second language model agent 314. For example, the second language model agent 314 can output a relevance indicator that is a value in a range from, for example, 0-1.

[0101] In other embodiments, the relevance indicator can be a probability value determined by the large language model that indicates a probability that the candidate document is to be classified as a particular classification, where the classifications can be binary relevance classifications (e.g., “Yes” or “No”).

[0102] In yet other embodiments, the relevance indicator can be a value that is derived from one or more outputs of the second language model agent 314. For example, the relevance indicator can be generated based on the probability that the candidate document is classified as “Yes” and the probability that the candidate document is classified as “No.”

[0103] The second language model agent 314 can determine a plurality of relevance indicators 316, where each relevance indicator corresponds to a document of the plurality of candidate documents 308. The computer system can filter the plurality of candidate documents 308 using the plurality of relevance indicators 316 and a threshold, as described herein.

[0104] After filtering the candidate documents of the plurality of candidate documents 308 to form a subset of candidate documents 318, the computer system can order the subset of candidate documents 318. The computer system can order the subset of candidate documents 318 in any suitable order. The computer system can order the candidate documents of the subset of candidate documents 318 respectively based on the relevance indicators. For example, the computer system can order the candidate documents of the subset of candidate documents 318 in descending order. As such, the computer system can order the subset of candidate documents 318 by relevance.

[0105] As an illustrative example, the plurality of candidate documents 308 can include a first candidate document, a second candidate document, and a third candidate document. The second language model agent 314 can determine three relevance indicators, one for each candidate document. The first candidate document can be associated with a relevance indicator of “0.2,” the second candidate document can be associated with a relevance indicator of “-0.4,” and the third candidate document can be associated with a relevance indicator of “0.6.” The computer system can obtain the subset of candidate documents 318 by removing the second candidate document from the plurality of candidate documents 308 based on the relevance18801 10344V 1indicator being less than a threshold value (e.g., -0.05, 0, 0.1, 0. 15, etc.), which may be a static threshold or an adaptive threshold as described herein. The computer system can also order the remaining candidate documents in the subset of candidate documents 318 based on the relevance indicator of each candidate document. In this example, the resulting subset of candidate documents 318 can include an ordered list that includes, in order, the third candidate document and the second candidate document.D. Query and Response

[0106] Embodiments provide for various stages of operation of an overall process of generating a response to a query. The method illustrated in FIG. 5 can include 1) document retrieval, 2) initial response generation, 3) relevance determination and filtering, and 4) response generation.1. Document retrieval

[0107] During document retrieval 502, the computer system can obtain a query 504. The computer system can utilize a retriever module 506 to obtain candidate documents 510 from a database 508. The computer system can obtain one or more candidate documents 510 from the database 508 as described in detail in reference to FIG. 1.

[0108] For example, the retriever module 506 can obtain a set of candidate documents from the database 508 (e.g., a public or a private database). The candidate documents in the set of candidate documents may include both relevant and noisy (e.g.. irrelevant) entries.2. Initial response generation

[0109] The first language model agent present in the sequence is the predictor agent 512. The predictor agent 512 can be configured to generate independent initial responses for the query 504 by utilizing each retrieved candidate document as a separate informational source. For example, for each candidate document that is retrieved in response to the query 504, the predictor agent 512 is capable of generating a distinct initial response that is based on the unique content of that candidate document. This process results in the creation of multiple initial responses, with each response uniquely associated with a corresponding candidate document of the candidate documents 510. Such a process ensures that the information contained within each candidate document is explicitly and individually considered during the initial response generation phase.19801 10344V 1

[0110] For each candidate document within the candidate documents 510, the predictor agent 512 can generate an initial response to the query 504 by referencing the respective candidate document. The predictor agent 512 or the computer system can generate a plurality of initial prompts. Each initial prompt can include the query 504 and a candidate document of the candidate documents 510. The predictor agent 512 can input each initial prompt into a machine learning model (such as a large language model) to generate an initial response based on the initial prompt. Through this iterative process, the predictor agent 512 produces a set of document-query-initial response triplets, with each triplet comprising a candidate document, the query 504, and the corresponding initial response generated based on that candidate document.[OHl] For example, for each candidate document, the predictor agent 512 can obtain two inputs: the query 504 (e.g., a question) and the candidate document (e.g., a passage, an article, a record, etc.) that is identified as being potentially relevant to the query 504. In some embodiments, the predictor agent 512 may preprocess these inputs to normalize, tokenize, or otherwise format the text to conform to the requirements of the underlying LLM architecture. Preprocessing steps may include removing extraneous characters, standardizing encodings, and truncating content to fit within the LLM’s input context window.

[0112] Once the query 504 and candidate document are prepared, the predictor agent 512 can create a composite initial prompt that presents both elements in a structured manner to the LLM. The initial prompt may concatenate the query 504 and the candidate document or may encode the query 504 and the candidate document using delimiters or special tokens, thereby explicitly instructing the LLM to consider the candidate document’s content when formulating an initial response to the query 504. The computer system can generate the initial prompt in any suitable manner, for example, using a prompt template.

[0113] After generating the initial prompt, the predictor agent 512 can provide the initial prompt to a large language model, which can be included within the predictor agent 512. The large language model can be pre-trained on text to processes the prompt using internal attention mechanisms and deep neural network layers. During inference, the large language model can utilize information from both the query' 504 and the candidate document to generate an initial response.

[0114] Upon completion of the inference phase, the predictor agent 512 can output a generated initial response that addresses the query 504 in the context of the candidate20801 10344V 1document. In some embodiments, the predictor agent 512 may perform additional postprocessing on this output, such as trimming extraneous text, normalizing formatting, etc.

[0115] The predictor agent 512 can then associate the initial response with the query 504 and the candidate document, forming part of a data structure (e.g., a triplet) that records the relationship between the query 504, the candidate document, and the initial response.

[0116] The result of the processing of the predictor agent 512 includes a structured record that can be structured in a data structure 514. The predictor agent 512 can output an entry for the data structure 514 for each candidate document. The data structure 514 can include a set of document-query-initial response triplets. The data structure 514 can include the same number of document-query-initial response triplets as there are candidate documents 510. The data structure 514 may, for each entry, include the query 504 that is associated with a particular candidate document of candidate documents 510, as well as the initial response corresponding to that candidate document. This arrangement of the data structure 514, where each entry includes the elements of a document, a query, and an initial response, can be referred to as a triplet. By systematically generating initial responses based on each candidate document paired with the query 504, the predictor agent 512 can ensures that a diverse set of candidate documents can be evaluated along with their associated initial responses in downstream tasks.

[0117] In some embodiments, the data structure 514 may be implemented as an ordered list of triplets. For example, the document-query-initial response triplets of the set of triplets can be ranked in order of relevance, with the order determined by the retriever module 506 based on the assessed relevance of each candidate document to the query’ 504.

[0118] As an illustrative example, the predictor agent 512 can generate a data structure 514 that includes a candidate document, a query, and an initial response triplet. The data structure 514 can be structured as illustrated in Table 1, below.Table 1: Example triplet data structure21801 10344 V l3. Relevance determination and filtering

[0119] After the predictor agent 512 generates the initial responses and organizes them within the data structure 514, the process advances to the evaluation of the initial responses and their relevance to the query 504.

[0120] In this stage, a judge agent 516 can be utilized to assess both the relevance and the quality of each document-query -initial response triplet produced by the predictor agent 512 and stored within the data structure 514. For each triplet, the judge agent 516 evaluates the degree to which the initial response effectively addresses the query 504 in view of the information contained within the associated candidate document. The judge agent 516 is further configured to determine and assign a relevance indicator to each document-query-initial response triplet. This relevance indicator serves to quantify a combination of factors, such as the relevance, coherence, and informational value of the triplet. By assigning such relevance indicators, the judge agent 516 enables the identification and ranking of the most relevant initial responses among the set of initial responses generated by the predictor agent 512, thus facilitating the prioritization and selection of high-quality answers in subsequent processing. The relevance indicator can be a binary flag (e.g., “Yes” or “No”) or a numerical value (e.g., a value in a range from 0 to 1, a value in a range from -10 to 10, etc.).

[0121] Details related to the judge agent 516 will be described in reference to FIG. 6. FIG. 6 shows a flowchart that illustrates a method relating to additional aspects of the quantification of document scoring, filtering, and ordering, according to aspects of the disclosed technology. During the method illustrated in FIG. 6, the judge agent 516, in conjunction with the computer system, can evaluate, for each document-query-initial response triplet, a relevance of the initial response based on the respective candidate document. The judge agent 516, in conjunction with the computer system, can then select candidate documents for a subset of the candidate documents 518 based on the relevance for each of the plurality of candidate documents. In some embodiments, the judge agent 516 can then order the candidate documents in the subset of the candidate documents.

[0122] At step 602, the judge agent 516 can perform a scoring process. The judge agent 516 can evaluate each entry’ within the data structure 514. The judge agent 516 may be prompted to classify whether the initial response is "supported" or "not supported" by the corresponding document with respect to the query’ 504.22801 10344 V l

[0123] To determine a relevance indicator for a particular triplet (e.g., including the query' 504, the candidate document, and the initial response), the computer system can generate a prompt based on the query 504, the candidate document, and the initial response. For example, the computer system can iteratively select a triplet from the data structure 514. The computer system can extracts the relevant textual content of each element of the triplet and prepares the textual content for prompt construction, which may include normalization, formatting, and truncation to ensure compliance with the input constraints of the large language model.

[0124] The computer system can create a composite input prompt designed to present the three components of the triplet in a clear and structured manner. The prompt can incorporate the query 504, the candidate document, and the initial response, each clearly delineated by appropriate labels (e.g., "‘Query:’’, “Document:”, “Response:”), as an example. Additionally, the prompt may include explicit instructions that direct the LLM to assess whether the information contained in the candidate document substantively supports the initial response in the context of the query 504. For example, the prompt may conclude with a question such as, “Does the document support the response to the query? Please answer ‘Yes’ or ’No' and explain.” Such prompt engineering can provide the LLM with contextual information such that the LLM can perform an evaluation.

[0125] Once the prompt has been constructed, the computer system can provide the prompt as input to the judge agent 516. The judge agent 516 can process the prompt by interfacing with the LLM, which can be included in the judge agent 516. The LLM can utilize a contextual understanding and inferential capabilities to analyze the relationships between the query 504, the candidate document, and the initial response. During inference, the LLM can generate an output that includes a direct answer to the support question (e.g., “Yes” or “No”, or other suitable Boolean or numerical value). In some embodiments, the LLM can provide an explanation for the assessment. In some embodiments, the LLM may also output probabilistic confidence scores or probabilities associated with the possible answers.

[0126] As an illustrative example, the judge agent 516 can generate an output that indicates that a particular initial response is not supported by the candidate document (e.g., an answer of “No”) as well as a probability of the answer being “No” (e.g., 85%) and a probability of the answer being “Yes” (e.g., 28%). The probabilities do not necessarily need to equal 100% as each probability is the probability that input is related to a given class (e.g., a class of the answer being “No” or a class of the answer being “Yes”).23801 10344V 1

[0127] The computer system can then interpret the LLM’s output to derive a relevance indicator for the evaluated triplet. This relevance indicator may be implemented as a binary value, such as ‘'Yes” or£'No,” which indicates whether the candidate document supports the initial response for the given query. Alternatively, the relevance indicator may be a numerical value derived from the underlying probabilities or log likelihoods produced by the LLM for each possible token output. In some embodiments, the system may employ hybrid approaches, combining both binary and probabilistic results to obtain a more granular and nuanced relevance indicator . The relevance indicator , can be recorded and associated with the original triplet.

[0128] In some embodiments, the classification can be implemented as a binary' output, which reflects whether the initial response is substantiated by the content of the candidate document for the query 504. This Boolean (e.g., yes or no) determination can be subsequently used as a criterion for filtering the set of candidate documents 510, such that documents not supporting the initial response may be excluded from further consideration.

[0129] In other embodiments, rather than relying solely on a binary yes or no classification, the judge agent 516 can quantify the relevance of each triplet by calculating the difference between the log probabilities of the "yes" and "no" token outputs produced by the language model. This approach yields a relevance indicator that provides a robust, quantitative measure of the degree of support that the candidate document offers for the initial response to the query 504. The resulting relevance indicator can provide for a more nuanced and graded assessment of document support, as compared to a simple binary judgment.

[0130] As such, the judge agent 516 may evaluate whether or not a candidate document provides information that is relevant to both the query and the initial response. As part of this evaluation, the judge agent 516 can be prompted to output a relevance indicator for each document-query-initial response triplet. The relevance indicators can later be utilized as a criterion for filtering and ranking documents in subsequent processing stages.

[0131] At step 604, after determining a relevance indicator for each document-query-initial response triplet, the judge agent 516 can perform a filtering process. Each candidate document associated with a document-query -initial response triplet can be evaluated based on its assigned relevance indicator. Candidate documents that have a relevance indicator that fall below a threshold (e.g., a static threshold or an adaptive threshold) are removed (e.g.. filtered) from further consideration.24801 10344V 1

[0132] In the case where the relevance indicators are binary, the relevance indicator may indicate a classification of "yes" or "no" that indicates whether a particular candidate document should be retained and included into the subset of candidate documents 518. In such embodiments, only those documents classified as “yes’’ (e.g., meaning that the judge agent 516 has determined that the candidate document supports the initial response and is relevant to the query 504) are retained for subsequent processing, while documents classified as “no” are filtered out. As such, the subset of candidate documents 518 can include candidate documents with a classification of “yes” as determined during step 602 by the judge agent 516.

[0133] In the case where the relevance indicators are non-binary (e.g., are a numerical value), the computer system can filter the candidate documents 510 into the subset of candidate documents 518 using a static threshold or an adaptive threshold. In these numerical embodiments, the system determines whether or not the relevance indicator for a given candidate document exceeds a specified threshold value, which can be static or adaptive. If the numerical relevance indicator is greater than the threshold, the candidate document can be retained for further use. If the numerical relevance indicator is less than the threshold, then the candidate document can be removed along with the associated initial response. This scoring and filtering mechanism ensures that only documents with adequate relevance are carried forward in the final response generation pipeline.

[0134] In some embodiments, a static threshold can be utilized. The computer system can compare the relevance indicators to the static threshold. For example, the computer system can compare the relevance indicators to a static threshold that is predetermined (e.g., 1.0. 4.2, 7.8, 9.9, etc.). The value of the static threshold can be predetermined based on typical relevance indicators.

[0135] In some embodiments, to refine the selection of candidate documents, the computer system can utilize an adaptive threshold to filter out candidate documents that are deemed noisy (e.g., insufficiently relevant). The computer system can dynamically determine the adaptive threshold based on the average relevance indicator across the set of evaluated triplets for the query 504. Candidate documents with relevance indicators falling below this threshold may be excluded from further consideration and are not included in the subset of candidate documents 518. The adaptive threshold is described in further detail in reference to FIGs. 7A-7D.

[0136] The threshold used for filtering candidate documents based on relevance indicators may be adaptive, rather than static, to account for variations in the quality and distribution of25801 10344V 1the retrieved documents for different queries. For example, a first query' may have more relevant documents than a second query based on the contents of the queries. In such embodiments, the system can dynamically determine an adaptive threshold for filtering, based on the statistical distribution of relevance indicators associated with the current set of retrieved candidate documents. The adaptive threshold can be calculated using statistical measures such as the mean and standard deviation of the relevance indicators. For example, the threshold may be set as the mean relevance indicator minus a configurable multiple (N) of the standard deviation, where N can be a hyperparameter that can be modified according to system requirements or desired filtering outcomes. This adaptive approach ensures that the filtering process remains robust and responsive to the characteristics of each individual query and its corresponding set of candidate documents. Such adaptive thresholds provide for reducing the likelihood of retaining noisy or weakly relevant information when proceeding to generate the final response.

[0137] At step 606, in some embodiments, after the filtering stage has been completed, the subset of candidate documents 518 can be ordered according to their respective relevance indicators. By arranging the subset of candidate documents 518 in descending order of relevance (or other suitable ordering), the computer system can leverage the inherent property of large language models to assign greater importance or attention to documents that are positioned earlier within the model’s context window during input processing. This ordered sequencing is particularly advantageous when the LLM is presented with a long context compnsing multiple documents. Such prioritization mitigates the risk of information loss commonly associated with the so-called "lost in the middle" phenomenon, wherein critical information embedded in the middle or latter portions of a lengthy input context may be overlooked or underweighted by the LLM. By ordering the filtered documents based on their relevance indicators, the system can maximize the likelihood that the most important documents are effectively utilized by a large language model during response generation.

[0138] Returning to FIG. 5, the computer system can obtain the subset of candidate documents 518 based on the relevance scoring process performed by the judge agent 516.4. Response generation

[0139] Following the identification and selection of a filtered and ranked subset of candidate documents 518, each having been evaluated for relevance and support by the preceding agents,26801 10344V 1the computer system can continue to a subsequent processing stage involving the finalpredictor agent 520.

[0140] The final-predictor agent 520 can be configured to utilize the subset of candidate documents 518, thereby ensuring that only the most relevant, highly scored, and substantively supportive documents are taken into consideration for final response generation. By restricting input documents to this subset of candidate documents 518, the computer system can maximize the likelihood that a response 522 is both accurate and contextually appropriate.

[0141] The computer system can generate a final prompt based on the subset of candidate documents 518 and the query 504. The final prompt can prompt the final-predictor agent 520 to generate the response 522 to the query' 504. To generate the final prompt, the computer system can create a composite prompt that can concatenate, or otherwise combine, the query' 504 with the subset of candidate documents 518, presenting the documents in order of descending relevance as determined by prior scoring. The final prompt may also include explicit instructions for the LLM to generate a comprehensive and contextually grounded answer based on the provided materials. For example, the prompt may be structured illustrated in Table 2, below.Table 2: Example final prompt

[0142] Such a prompt ensures that the LLM within the final-predictor agent 520 is provided with the essential information from the documents and the query 504, presented in a manner that optimally leverages the model’s attention mechanisms and maximizes the use of the most critical supporting evidence.27801 10344V 1

[0143] The computer system can provide the final prompt to the final-predictor agent 520. Upon receiving the final prompt, the final-predictor agent 520 can invoke an LLM. which can be included in the final -predictor agent 520, to generate a response 522 to the information from the subset of candidate documents in the context of the query 504. The LLM can process the prompt to generate the response 522. The response 522 can be created to reflect the highest possible degree of relevance, accuracy, and contextual fidelity’, due to the earlier retrieval, filtering, and ranking operations.

[0144] After generating the response, the computer system can provide the response to a user and / or device from which the response was received. The computer system can provide the response subsequent to obtaining the query.

[0145] The process implemented by the final-predictor agent 520 is training -free. This means that that none of the large language models involved need to undergo fine-tuning or additional model training beyond their initial pre-training when new domain specific questions are asked. The domain specific questions can be answered based on and using documents that relate to the relevant domain. This approach allows for rapid adaptation to new queries and document sets.III. ADAPTIVE THRESHOLD

[0146] After obtaining relevant indicators for each candidate document, a technical challenge is to determine an optimal adaptive threshold for filtering out noisy candidate documents. An optimal adaptive threshold can be a score that perfectly filters out all noisy candidate documents while retaining all relevant candidate documents.

[0147] FIG. 7A illustrates aspects of establishing an adaptive threshold, which may separate relevant and noise documents for a query7, according to aspects of the disclosed technology7. FIG. 7A includes an adaptive threshold example 700.

[0148] The adaptive threshold examples 700 includes a first example and a second example of adaptive thresholds based on a number of related documents and noise documents. A related document can include a candidate document that includes content that relates to a query. A noise document can include a candidate document that does not include content or includes little content that relates to the query.28801 10344V 1

[0149] The adaptive threshold examples 700 includes a first example, where a retriever module retrieves a higher number of noisy candidate documents than related candidate documents from a database for a query. In particular, the retriever module can retrieve three candidate documents including a first document 702, a second document 704, and a third document 706. The computer system can determine a relevance indicator for each candidate document using, for example, the judge agent. The computer system can determine a relevance indicator for each of the three candidate documents. The computer system, for example, can determine a relevance indicator of 10.6 for the first document 702, a relevance indicator of - 3.2 for the second document 704, and a relevance indicator of -9.4 for the third document 706.

[0150] The computer system can determine an adaptive threshold based on the three relevance indicators. The computer system can determine the adaptive threshold based on the average of the relevance indicators, the standard deviation of the relevance indicators, and a number of standard deviations (n) that are to be utilized. For example, the computer system can determine the adaptive threshold, in some embodiments, as follows:Threshold = Avg(RelevanceIndicators') — n * Std^Relevancelndicators)

[0151] For example, the computer system can determine an optimal adaptive threshold for this example of approximately 3.7.

[0152] The adaptive threshold examples 700 includes a second example, where a retriever module retrieves a higher number of related candidate documents than noisy candidate documents from a database for a query. In particular, the retriever module can retrieve three candidate documents including a fourth document 708, a fifth document 710, and a sixth document 712. The computer system can determine a relevance indicator for each of the three candidate documents. The computer system, for example, can determine a relevance indicator of 12.1 for the fourth document 708, a relevance indicator of 11.2 for the fifth document 710, and a relevance indicator of -2.4 for the sixth document 712.

[0153] The computer system can determine an adaptive threshold based on the three relevance indicators. For example, the computer system can determine an optimal adaptive threshold for this example of approximately 4.4. As an illustrative example, to determine an adaptive threshold value of 4.4 using the three scores of 12.1, 11.2, and -2.4 and an number of standard deviations (n) equal to 0.39, the computer system can determine the optimal adaptive threshold as being 4.4.29801 10344V 1

[0154] These examples of the adaptive threshold examples 700 illustrate that the optimal adaptive threshold varies with the document distribution for each query’. As such, each query may be associated with a different optimal adaptive threshold based on the candidate documents obtained and how well the initial responses generated based on the candidate documents relate to the original queries.

[0155] The examples illustrated in FIG. 7A illustrate that the optimal adaptive threshold varies with the document distribution for each query. FIG. 7B illustrates aspects of determining an optimal adaptive threshold based on ratios of noisy candidate documents to related candidate documents, according to aspects of the disclosed technology. FIG. 7B includes a graph 720.

[0156] During the analysis of the relevance indicator distributions (which can be non- uniform distributions) for both related candidate documents and noisy candidate documents on the RGB benchmark dataset, the relevance indicators of related candidate documents were skewed high with a small standard deviation, as shown in the graph 720. This indicates that the LLM (in this example, the experiment was performed with Mistral-7B) was more confident about the set of candidate documents that contained fewer noisy candidate documents. In contrast, the relevance indicators of the set of candidate documents that contained more noisy candidate documents were more uniformly distributed with a larger standard deviation, suggesting that the LLM was less confident and may misjudge the noisy candidate documents. Based on this biased LLM behavior, the average relevance indicator can be used as an adaptive threshold.

[0157] FIG. 7C illustrates aspects of relevance indicator distribution to select an adaptive threshold based on a set of related candidate documents. FIG. 7D illustrates aspects of score distribution to select an adaptive threshold based on a set of noisy candidate documents. FIG. 7C includes a graph illustrating a first distribution 730. FIG. 7D includes a graph illustrating a second distribution 732. These graphs visually represent the distribution of relevance indicators (e.g., relevance scores) assigned to candidate documents for two example distributions. The first distribution 730 shows a relevance indicator distribution for a set of related candidate documents. The second distribution 732 shows a relevance indicator distribution for a set of noisy candidate documents.

[0158] The graphs in FIGs. 7C-7D feature an average relevance indicator line as indicated by the line 734 for the first distribution 730 and line 736 for the second distribution 732. For both distributions, candidate documents that fall to the right of the average relevance indicator30801 10344V 1line (e.g., have a relevance indicator greater than average), are retained for further processing. Conversely, in some embodiments, candidate documents to the left of the average relevance indicator line (e.g., have a relevance indicator less than average) are filtered out. This approach enables the system to dynamically adapt document selection criteria based on the observed relevance indicator distributions for each query'.

[0159] When the average relevance indicator is high, as seen in the first distribution 730 for the example of the set of candidate documents containing many relevant candidate documents. Under such conditions, the computer system can effectively filter out most low-scoring outliers, which are likely to be noise (e.g., noisy candidate documents).

[0160] When the average relevance indicator is low, as seen in the second distribution 732 for the example of the set of candidate documents containing many noisy or less relevant candidate documents. In this scenario, the computer system can attempt to reduce the number of retained candidate documents while still maintaining a high recall rate for relevant content. This is achieved by using the average relevance indicator as a threshold, which typically results in approximately half of the candidate documents being filtered out.

[0161] Doing so can result in the candidate documents in a region 738 in the first distribution 730 being selected for inclusion into a subset of candidate documents, which can be utilized for further processing. This can also result in the candidate documents in a region 740 in the second distribution 732 being selected for inclusion into a subset of candidate documents.

[0162] To introduce additional flexibility, the computer system can further adjust the adaptive threshold rather than simply utilizing the average relevance indicator as the threshold. The computer system can adjust the adaptive threshold by adding N times the standard deviation of the relevance indicators within each set of candidate documents. This adjustment allows the adaptive threshold to be relaxed when necessary and can prevent N standard deviations of candidate documents from being filtered out. The candidate documents within N = 1 standard deviation are included in a region 742 in the first distribution 730 and in a region 744 in the second distribution 732. The parameter N can be a hyperparameter within the system, providing a tunable mechanism for threshold calibration.

[0163] By determining the adaptive threshold based on the average and the standard deviation, the computer system can identify the candidate documents in the region 738 and the region 742 for inclusion into the subset of candidate documents for a first query associated with a first obtained set of candidate documents. The computer system can also identify the31801 10344V 1candidate documents in the region 740 and the region 744 for inclusion into the subset of candidate documents for a second query- associated with a second obtained set of candidate documents.

[0164] Empirical analysis of the relevance indicator distributions for both related and noisy example sets of candidate documents, such as those observed on the RGB benchmark dataset, reveals distinct patterns in LLM behavior. For related candidate documents, the relevance indicators are skewed high and exhibit a small standard deviation, as shown in the first distribution 730, indicating that the LLM (e.g., Mistral-7B in the experiment) was more confident about these documents. For noisy candidate documents, the relevance indicators were more uniformly distributed with a larger standard deviation, as show n in the second distribution 732, suggesting that the LLM was less confident and may misjudge these cases. Based on this observed LLM behavior, the embodiments provide for determining the adaptive threshold based on the average relevance indicator, the standard deviation, and a hyperparameter N for each query-.IV. RESULTS SHOWING IMPROVED PERFORMANCE

[0165] Experiments were conducted to evaluate the performance of embodiments, aiming to demonstrate the effectiveness of using LLMs as a noisy document filter.A. Tasks and Datasets

[0166] The model according to embodiments was evaluated against various baselines across a range of downstream tasks, where the outputs were assessed for overall correctness. All evaluations were conducted in a zero-shot setting, where task instructions were provided without few-shot demonstrations (Sanh, et al., 2022, Multitask prompted training enables zeroshot task generalization, In International Conference on Learning Representations; and Wei, et al., 2021, Finetuned language models are zero-shot learners, arXiv preprint arXiv:2109.01652).

[0167] Embodiments were evaluated on open-domain question answering tasks. Embodiments were evaluated on two open-domain question and answer (QA) datasets, including TriviaQA-unfiltered (Joshi, et al., 2017, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, arXiv preprint arXiv: 1705.03551.) as illustrated in FIG. 8 and PopQA (Mallen, et al., 2022, When not to trust language models: Investigating effectiveness of parametric and non-parametric memories, arXiv preprint arXiv:2212. 10511.)32801 10344V 1as illustrated in FIG. 9, where LLMs were required to answer arbitrary questions about factual knowledge.

[0168] Since the test set of TriviaQA-unfiltered is not publicly available, the experiment utilized the validation and test sets provided by an existing work, comprising 11,313 test queries for evaluation. For PopQA, the experiment utilized the long-tail subset, consisting of 1,399 rare entity queries with monthly Wikipedia page views of less than 100. The performance was evaluated based on whether or not the gold answers are included in the model’s generations instead of strictly requiring exact matches.

[0169] Embodiments were also evaluated on closed-set tasks. Embodiments were evaluated on the ARC-Challenge dataset (Clark, et al., 2018, Think you have solved question answering? tty arc, the ai2 reasoning challenge, arXiv preprint arXiv: 1803.05457.) as illustrated in FIG. 10. The ARC-Challenge is a multiple-choice reasoning dataset collected from scientific exams. The experiment used accuracy as the evaluation metric and reported results on the test set.B. Baselines

[0170] For baselines without retrievals, the experiment evaluated strong publicly available pre-trained LLMs. particularly. Mistral-7B, as the baseline without leveraging external knowledge.

[0171] For baselines with retrievals, the experiment evaluated models augmented with retrieval at test time. The first category included standard RAG baselines, where an LLM (Mistral-7B) generated output given the query prepended with the top retrieved documents using the same retriever as according to embodiments. The second category includes two variants of embodiments: 1) embodiments where the judge agent outputs a natural language judgment of "Yes" or "No,” (which is referred to as a binary classification embodiment) and 2) embodiments with random ordering document list after filtering and scoring (which is referred to as a random sorting embodiment).C. Experimental Settings

[0172] As a training-free RAG framework, the three LLM agents according to embodiment can be instantiated by different pre-trained LLMs. As default settings, for the experiment, all three agents were instantiated by Mistral-7B without training.33801 10344V 1D. Quantitative Analysis

[0173] The performance of embodiments and baselines across three well-known QA benchmarks were evaluated, as illustrated in FIG 8, FIG. 9, and FIG. 10.

[0174] FIG. 8 shows a graph 800 that illustrates experimental results of efficacy on QA benchmarks for a TriviaQA dataset. The graph 800 shows a performance comparison between various implementations such as Mistral (with documents), a binary classification embodiment (e.g., where relevance indicators are binary), a random sorting embodiment (e.g., where documents are randomly sorted after filtering), and a descending value sorting embodiment (e.g., where documents are sorted in descending order after filtering).

[0175] FIG. 9 shows a graph 900 that illustrates experimental results of efficacy on QA benchmarks for a PopQA dataset. The graph 900 shows a performance comparison between the same implementations as the graph 800.

[0176] FIG. 10 shows a graph 1000 that illustrates experimental results of efficacy on QA benchmarks for an ARC challenge dataset. The graph 1000 shows a performance comparison between the same implementations as the graph 800 and the graph 900.

[0177] The experimental results demonstrate that embodiments outperform all training-free baselines by margins ranging from 2.3% to 6.1%. Notably, the questions in PopQA heavily rely on external knowledge to enable pre-trained LLMs to generate accurate answers. In this scenario, embodiments exhibits a significant technical advantage over the baselines, particularly because the retriever is not fine-tuned on the target question sets and may retrieve a large number of noisy candidate documents.V. EXAMPLE METHOD TO GENERATE RESPONSES

[0178] FIG. 11 shows a flowchart illustrating an example method 1100 related to generating responses to queries according to embodiments. The method illustrated in FIG. 11 can be performed by a computer system. The computer system can include one or more computers. The computer system can include a plurality of language model agents.

[0179] At step 1102, the computer system can obtain a query. The computer system can obtain the query from an external device (e.g., a user device, a client device, etc.) or from input (e.g., user input directly into the computer system).34801 10344V 1

[0180] The query can include a question or a request for information. The query can include query includes text input by a user. For example, a query can be a question such as “what is the city of Paris like?” As another example, the query can be “what are the allowed category codes for a resource provider in the Canada region according to the rules?”

[0181] At step 1104, after obtaining the query, the computer system can retrieve a plurality of candidate documents from one or more data sources (e.g., databases) based on the query.

[0182] The computer system can utilize the query to identify relevant candidate documents in one or more databases. For example, the computer system can generate a query embedding based on the query using a machine learning model that is trained to generate embeddings. The computer system can compare the query embedding to document embeddings stored in one or more databases. The computer system can compare the embeddings using a distance calculation that measures the distance between the vectors of the query embedding and the document embedding. The computer system can identify document embeddings that are similar to the query embedding (e.g., based on a distance metric or other suitable embedding comparison metric). The computer system can obtain candidate documents that are stored in association with the document embeddings.

[0183] In some embodiments, each document in a database can be associated with one or more document embeddings, where each document embedding represents a different excerpt of the document.

[0184] The computer system can obtain the plurality of candidate documents from the one or more data sources.

[0185] At step 1106, after obtaining the plurality of candidate documents, a first language model agent of the computer system can generate, for each candidate document of the plurality of candidate documents, an initial response to the query using the respective candidate document. For example, the computer system can generate an initial response for each candidate document.

[0186] The computer system can generate the initial response for each candidate document using the first language model agent. The first language model agent can accept the query and a candidate document of the plurality of candidate documents as input. For example, the input can include a prompt comprising the uery and the candidate document. The first language model agent can generate an initial response based on the prompt.35801 10344V 1

[0187] The computer system can repeat the initial response generation process for each candidate document of the plurality of candidate documents. The computer system can generate a set of document-query-initial response triplets.

[0188] At step 1108, a second language model agent of the computer system can evaluate, for each document-query-initial response triplet, a relevance of the initial response based on the respective candidate document.

[0189] To evaluate the relevance of the initial response, the computer system can generate a relevance indicator for the document-query -initial response triplet. In some embodiments, the relevance indicator can be a Boolean value that indicates whether or not the candidate document is relevant to the query based on the initial response. In other embodiments, the relevance indicator can be a numerical value that indicates a probability that the candidate document is relevant to the query’ based on the initial response.

[0190] For example, the second language model agent of the computer system can generate a Boolean value relevance indicator based on the candidate document, the query, and the initial response. The computer system can prompt the second language model agent to determine an output classification that indicates whether or not the candidate document is relevant to the query based on the initial response. The second language model agent can receive an input prompt that includes the request for the output classification as well as the query, the candidate document, and the initial response. The second language model agent can include a large language model capable of generating text outputs based on inputs. The second language model can output a classification of whether or not the candidate document is relevant to the query based on the initial response (e.g., an output of "yes” or "no").

[0191] As another example, the second language model agent of the computer system can generate a numerical value relevance indicator based on the candidate document, the query', and the initial response. The relevance indicator can be based on, for example, a difference between log probabilities that the candidate document supports or does not support the initial response for the query.

[0192] At step 1110, after evaluating the relevance of each candidate document, the computer system can select a subset of the candidate documents based on the relevance for each of the plurality’ of candidate documents. The computer system can filter the plurality’ of candidate documents into the subset of candidate documents. The computer can utilize the36801 10344V 1relevance from each candidate document to determine whether or not to select the candidate document for inclusion in the subset of candidate documents.

[0193] If the relevance indicators are Boolean values (e.g.. binary flags), then if the computer system can determine to include a candidate document into the subset of candidate documents if the relevance indicator indicates “yes.”

[0194] If the relevance indicators are numerical values, then the computer system can determine to include a candidate document into the subset of candidate documents based on a static threshold or an adaptive threshold.

[0195] If a static threshold is utilized, then the computer system can compare the relevance indicators to the static threshold. For example, the computer system can compare the relevance indicators to a static threshold that is predetermined (e.g., 1.0, 4.2, 7.8, 9.9, etc.). The value of the static threshold can be predetermined based on ty pical relevance indicators.

[0196] If an adaptive threshold is utilized, then the computer system can determine the adaptive threshold and then compare the relevance indicators to the adaptive threshold. The computer system can determine the adaptive threshold dynamically. For example, the computer system can determine the adaptive threshold based on the candidate documents, the query7, the initial responses, and / or data derived therefrom.

[0197] For example, the computer system can dynamically determine the adaptive threshold for each query based on a statistical analysis of the relevance indicators for the plurality of candidate documents. Each different query can utilize a different adaptive threshold value since different queries may relate to documents in the database by different absolute amounts.

[0198] The computer system can determine (e.g., calculate) the adaptive threshold as a function of a mean and a standard deviation of the relevance indicators for the plurality of candidate documents. As an illustrative example, the adaptive threshold can be determined based on an average of the relevance indicators minus a standard deviation of the relevance indicators multiplied by a number of standard deviations (n) that are to be utilized. For example:Threshold = Avg^Relevancelndicators) — n * Std(RelevanceIndicators')

[0199] In some embodiments, after filtering the candidate documents to form the subset of candidate documents, the computer system can order candidate documents of the subset of candidate documents. The computer system can order ordering the candidate documents in the37801 10344V 1subset of candidate documents according to their respective relevance indicators. Candiate documents with higher relevance indicators can be positioned earlier in subset of candidate documents, which can form the context to the third language model agent. In some embodiments, the ordering can be performed in descending order of the relevance indicators.

[0200] At step 1112, after forming the subset of candidate documents, the computer system can generate, using a third language model agent, a response to the query' using the subset of candidate documents.

[0201] For example, the computer system can generate a prompt comprising the query and the subset of candidate document. The computer system can input the prompt into the third language model agent. The third language model agent can generate and output a response (e.g., a final response) based on the prompt.

[0202] The computer system can provide the response to the originating device or user of the query. For example, the computer system can provide the response to a user device in response to the query. The user device can display the response to a user of the user device.VI. ADVANTAGES

[0203] Collectively, the methods and systems described herein deliver substantial advantages over traditional single-stage large language model document retrieval systems. Embodiments address the technical challenges of noisy document retrieval in RAG by introducing a training-free, multi-agent framework. Embodiments can utilize multiple LLM agents to collaboratively filter and score retrieved documents, enhancing the recall of relevant information while minimizing irrelevant content. Embodiments can utilize an adaptive threshold that dynamically adjusts based on the relevance indicator distribution of relevant and noisy candidate documents, improving robustness across different queries. Experimental results demonstrated that embodiments consistently outperformed RAG baselines across various QA benchmarks.

[0204] Embodiments provide for advantages in the relevance and contextual accuracy of generated responses. By incorporating a retriever module that selectively identifies documents pertinent to a query, the system ensures that responses are grounded in the most applicable and current information. The multi-agent approach further strengthens this advantage. For example, the predictor agent formulates distinct responses for each document, guaranteeing comprehensive consideration of diverse perspectives and factual details. The subsequent38801 10344V 1evaluation by the judge agent, which assesses the relevance and quality of each document- query-initial response triplet, ensures that the most relevant information is prioritized. Finally, the final-predictor agent generates a response using the top-ranked documents, resulting in responses that are both accurate and tailored to the specific context of the inquiry.

[0205] The modularity' of embodiments provides for flexibility' and extensibility’ while not requiring retraining or fine-tuning existing large language models. Both the retrieval and evaluation criteria can be tailored to meet the specific needs of individual domains or use cases. This customization enables users to optimize the system for specialized information retrieval and response evaluation scenarios w ithout retraining or fine-tuning a large language model.VII. EXAMPLE COMPUTING SYSTEM

[0206] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 12 in computer system 1200. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0207] The subsystems shown in FIG. 12 are interconnected via a system bus 1224. Additional subsystems such as a printer 1208, keyboard 1216, storage device(s) 1218, monitor 1222 (e.g., a display screen, such as an LED), which is coupled to display adapter 1212, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 1202, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 1214 (e.g., USB, FireWire®). For example, I / O port 1214 or external interface 1220 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 1200 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 1224 allows the central processor 1206 to communicate with each subsystem and to control the execution of a plurality' of instructions from system memory' 1204 or the storage device(s) 1218 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 1204 and / or the storage device(s) 1218 may embody a computer readable medium. Another subsystem is a data collection device 1210, such as a camera, microphone, accelerometer, and the like. Any of the39801 10344V 1data mentioned herein can be output from one component to another component and / or can be output to a user.

[0208] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 1220, by an internal interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 10, 200, 500, 1,000, 10,000, 50,000, 100,000, 500,00, or one million communication messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0209] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components.

[0210] A computer system can include a plurality of the components or subsystems, e.g., connected together by external interface or by an internal interface. In some embodiments, computer systems, subsystems, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a serv er can each include multiple systems, subsystems, or components.

[0211] It should be understood that any of the embodiments of the present disclosure can be implemented in the form of control logic using hardware (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software with a generally programmable processor in a modular or integrated manner. As used herein a processor includes a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other40801 10344V 1ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0212] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission, suitable media include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk), flash memory, and the like. The computer readable medium may be any combination of such storage or transmission devices.

[0213] Such programs may also be encoded and transmitted using earner signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium according to an embodiment of the present disclosure may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0214] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term “real-time” may refer to computing operations or processes that are completed within a certain time constraint. The time constraint may be 1 minute, 1 hour, 1 day, or 7 days. Thus, embodiments involve computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective steps or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be41801 10344V 1optional. Additionally, and of the steps of any of the methods can be performed with modules, circuits, or other means for performing these steps.

[0215] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure. However, other embodiments of the disclosure may involve specific embodiments relating to each individual aspect, or specific combinations of these individual aspects. The above description of exemplary embodiments of the disclosure has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above. The embodiments were chosen and described in order to best explain the principles of the disclosure and its practical applications to thereby enable others skilled in the art to best utilize the disclosure in various embodiments and with various modifications as are suited to the particular use contemplated.

[0216] The above description is illustrative and is not restrictive. Many variations of the disclosure will become apparent to those skilled in the art upon review of the disclosure. The scope of the disclosure should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0217] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the disclosure.

[0218] A recitation of “a”, "an" or “the’' is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or.” and not an “exclusive or” unless specifically indicated to the contrary.

[0219] All patents, patent applications, publications, and descriptions mentioned above are herein incorporated by reference in their entirety for all purposes. None is admitted to be prior art.42801 10344V 1

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for generating a response to a query, the method comprising: obtaining the query; retrieving a plurality’ of candidate documents from one or more data sources based on the query; for each candidate document of the plurality of candidate documents: generating, by a first language model agent, an initial response to the query using the respective candidate document, thereby producing a set of documentquery-response triplets; for each document-query-response triplet of the set of document-query- response triplets, evaluating, by a second language model agent, a respective relevance indicator of the initial response based on the respective candidate document; selecting a subset of candidate documents based on the respective relevance indicator for each candidate document of the plurality of candidate documents; and generating, by a third language model agent, the response to the query using the subset of candidate documents.

2. The method of claim 1, wherein evaluating the respective relevance indicator of the initial response comprises: providing the respective relevance indicator, the respective candidate document, and the query from the document-query-response triplet to a machine learning model in the second language model agent; and outputting, by the second language model agent, the relevance indicator for the document-query-response triplet.

3. The method of claim 2, wherein the respective relevance indicator is based on a difference between log probabilities that the respective candidate document supports or does not support the initial response for the query.

4. The method of claim 1, wherein selecting the subset of candidate documents comprises:43801 10344 V lselecting the subset of candidate documents based on the respective relevance indicators.

5. The method of claim 4, further comprising: wherein the subset of candidate documents is selected based on an adaptive threshold, the adaptive threshold being determined dynamically for each query based on a statistical analysis of the respective relevance indicators for the plurality of candidate documents.

6. The method of claim 5, wherein the adaptive threshold is calculated as a function of a mean and a standard deviation of the respective relevance indicators for the plurality of candidate documents.

7. The method of claim 1, further comprising: ordering the subset of candidate documents according to their respective relevance indicators prior to generating the response, such that candidate documents of the subset of candidate documents with higher respective relevance indicators are positioned earlier in a context provided to the third language model agent.

8. The method of claim 7, wherein the ordering is performed in descending order of the respective relevance indicators.

9. The method of claim 1, wherein the third language model agent is instantiated using a same pre-trained language model as the first language model agent and the second language model agent.

10. The method of claim 1, wherein obtaining the query comprises: receiving the query from a user device, wherein the query includes text input by a user of the user device.

11. The method of claim 1, wherein retrieving the plurality of candidate documents comprises: determining a query' embedding for the query; comparing the query' embedding to a plurality' of document embeddings in the one or more data sources; and44801 10344 V lobtaining the plurality of candidate documents based on the comparing, where each candidate document of the plurality of candidate documents is stored in association with a document embedding.

12. The method of claim 1, wherein each candidate document of the plurality of candidate documents includes datasets, articles, and / or descriptions, and wherein the query includes a question.

13. The method of claim 1, further comprising: generating a prompt comprising the query and the subset of candidate documents.

14. The method of claim 13, wherein generating the response comprises: inputting the prompt into the third language model agent; generating, by the third language model agent, the response based on the prompt; and obtaining the response from the third language model agent.

15. The method of claim 1, wherein generating, for each candidate document of the plurality of candidate documents, the initial response to the query comprises: for each candidate document of the plurality of candidate documents: generating an initial prompt based on the query and the candidate document; providing the initial prompt to the first language model agent; determining, by the first language model agent, the initial response based on the initial prompt; and generating the document-query-response triplet comprising the candidate document, the query, and the initial response.

16. The method of claim 1 , further comprising: providing the response subsequent to obtaining the query, wherein the response is provided to a device.

17. The method of claim 1, wherein the respective relevance indicator is a binary flag or a numerical value.45801 10344V 118. The method of claim 1, wherein the first language model agent, the second language model agent, and the third language model agent include machine learning models.

19. The method of claim 1 , wherein the initial response is a text output generated by a machine learning model in the first language model agent.

20. A system comprising: a processor; and a non-transitory computer readable medium comprising code, executable by the processor for performing any of the above methods.46801 10344V 1

Images