System and method for generating prompt-based queries for diverse searches

Abstract

An exemplary method for prompt-based query generation is provided. The method includes receiving, by a computing device, at least two prompts related to a search task to be performed against a corpus of documents relevant to the task. The method includes applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a synthetically generated query and a document from the corpus of documents. The method includes training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to take an input query related to the search task and predict output documents to be retrieved from the corpus of documents. The method includes providing, by the computing device, the trained document retrieval model.

Description

[Technical Field]

[0001] Cross-referencing / incorporation by reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 376,508, filed on 21 September 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Natural language processing tasks such as question answering and claim verification involve retrieving knowledge from massive sets of documents containing millions or even billions of candidates. Training machine learning models to perform such tasks requires large datasets of annotated query document pairs for each task. [Overview of the project]

[0003] There are many diverse and unique search tasks targeting different scenarios. For example, different question-answering (QA) tasks often have distinct distributions of queries. Also, for example, search systems are generally trained on question-answering data, but search tasks are not limited to question-answering tasks. For example, a task may include finding entities mentioned in a query, or finding evidence to support or refute a given statement. Unlike other search systems, neural search systems are generally trained on question-answering data. Because training a neural search system for each task requires a considerable number of annotated query document pairs, neural search systems may not function well for many tasks due to insufficient data. Also, for example, dual encoder models generally rely on more data than their cross-attention counterparts, typically requiring tens of thousands of labeled examples for the in-batch softmax loss function to be effective. Thus, there are technical problems in training neural search systems for various tasks for which there are few labeled examples.

[0004] As described herein, the prompt-based query generation, filtering, and retriever training system PROMPTAGATOR, also known as PROMPTAGATOR, leverages the capabilities of large-scale language models (LLMs) with the efficiency of standard dual encoders. PROMPTAGATOR can create task-specific dual encoders by generating large amounts of task-specific synthetic data. PROMPTAGATOR can achieve significant improvements in search performance (e.g., without rerankers) by using only 2 to 8 examples in its prompts. PROMPTAGATOR also greatly reduces the need for highly designed search architectures.

[0005] In one embodiment, a method for computer implementation for prompt-based query generation is provided. The method includes a computing device receiving at least two prompts related to a search task to be performed against a corpus of documents related to the task. The method also includes applying a large language model to generate a synthetic training dataset containing a plurality of query document pairs based on the at least two prompts and the document corpus, wherein each query document pair contains a document from the document corpus that was synthesized. The method further includes training a document retrieval model to take input queries related to the search task from the plurality of query document pairs from the synthetic training dataset and predict output documents to be retrieved from the document corpus. The method further includes providing the trained document retrieval model by the computing device.

[0006] In a second embodiment, a computing device for prompt-based query generation is provided. The computing device includes one or more processors and data storage. The data storage stores computer executable instructions that, when executed by one or more processors, cause the computing device to perform a function. The function includes the computing device receiving at least two prompts relating to a search task to be executed against a corpus of documents relating to the task; applying a large language model to generate a synthetic training dataset containing a plurality of query document pairs based on the at least two prompts and the corpus of documents, each query document pair containing a query generated by synthesis and a document from the corpus of documents; training a document search model to take input queries relating to the search task and predict output documents retrieved from the corpus of documents using the plurality of query document pairs from the synthetic training dataset; and providing the trained document search model by the computing device.

[0007] In a third aspect, a computer program for prompt-based query generation is provided. The computer program, when executed by a computer, includes instructions that cause the computer to perform a function. The function includes: receiving by a computing device at least two prompts relating to a search task to be performed on a corpus of documents relating to the task; applying a large language model to generate a synthetic training dataset containing a plurality of query document pairs based on the at least two prompts and the corpus of documents, each query document pair containing a query generated by synthesis and a document from the corpus of documents; training a document search model to take input queries relating to the search task and predict output documents to be retrieved from the corpus of documents using the plurality of query document pairs from the synthetic training dataset; and providing the trained document search model by the computing device.

[0008] In a fourth aspect, a product for prompt-based query generation is provided. The product includes one or more computer-readable media containing computer-readable instructions that cause the computing device to perform a function when executed by one or more processors of the computing device. The function includes: the computing device receiving at least two prompts relating to a search task to be performed against a corpus of documents relating to the task; applying a large language model to generate a synthetic training dataset containing a plurality of query document pairs based on the at least two prompts and the corpus of documents, each query document pair containing a query generated by synthesis and a document from the corpus of documents; training a document search model to take input queries relating to the search task and predict output documents to be retrieved from the corpus of documents using the plurality of query document pairs from the synthetic training dataset; and providing the trained document search model to the computing device.

[0009] In a fifth aspect, a system for prompt-based query generation is provided. The system includes means for receiving by a computing device at least two prompts relating to a search task performed on a corpus of documents relating to the task; means for applying a large language model to generate a synthetic training dataset containing a plurality of query document pairs based on the at least two prompts and the corpus of documents, wherein each query document pair contains a query generated by synthesis and a document from the corpus of documents; means for training a document search model to obtain input queries relating to the search task and predict output documents to be retrieved from the corpus of documents using the plurality of query document pairs from the synthetic training dataset; and means for providing the trained document search model by a computing device.

[0010] A sixth aspect provides a method for a computer implementation that applies a trained document retrieval model. The method includes a computing device receiving an input query related to a retrieval task performed on a corpus of documents related to the task. The method also includes the trained document retrieval model predicting documents from the document corpus, wherein the documents are highly relevant to the input query, and the document retrieval model has been trained on multiple query document pairs from a synthetic training dataset, which is generated by a large-scale language model based on at least two prompts related to the retrieval task. The method further includes the computing device providing the predicted documents in response to the input query.

[0011] In a seventh aspect, a computing device for applying a trained document retrieval model is provided. The computing device includes one or more processors and data storage. The data storage stores computer-executable instructions that, when executed by one or more processors, cause the computing device to perform functions. The functions include receiving, by the computing device, an input query related to a search task to be performed on a corpus of documents related to a task, predicting, by the trained document retrieval model, a document from the corpus of documents, where the document has a high relevance to the input query, and the document retrieval model is trained with a plurality of query-document pairs from a synthetic training dataset, and the synthetic training dataset is generated by a large language model based on at least two prompts related to the search task, predicting, and providing, by the computing device, the predicted document in response to the input query.

[0012] In an eighth aspect, a computer program for applying a trained document retrieval model is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions. The functions include receiving, by a computing device, an input query related to a search task to be performed on a corpus of documents related to a task, predicting, by the trained document retrieval model, a document from the corpus of documents, where the document has a high relevance to the input query, and the document retrieval model is trained with a plurality of query-document pairs from a synthetic training dataset, and the synthetic training dataset is generated by a large language model based on at least two prompts related to the search task, predicting, and providing, by the computing device, the predicted document in response to the input query.

[0013] In a ninth aspect, a product for applying a trained document search model is provided. The product includes one or more computer-readable media storing computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include receiving, by the computing device, an input query related to a search task to be performed on a corpus of documents related to a task; predicting, by the trained document search model, a document from the corpus of documents, wherein the document has a high relevance to the input query, and the document search model is trained with a plurality of query-document pairs from a synthetic training dataset, and the synthetic training dataset is generated by a large language model based on at least two prompts related to the search task; and providing, by the computing device, the predicted document in response to the input query.

[0014] In a tenth aspect, a system for applying a trained document search model is provided. The system includes means for receiving, by a computing device, an input query related to a search task to be performed on a corpus of documents related to a task; means for predicting, by the trained document search model, a document from the corpus of documents, wherein the document has a high relevance to the input query, and the document search model is trained with a plurality of query-document pairs from a synthetic training dataset, and the synthetic training dataset is generated by a large language model based on at least two prompts related to the search task; and means for providing, by the computing device, the predicted document in response to the input query.

[0015] The above summary is merely illustrative and is not intended to be limiting in any way. In addition to the above exemplary aspects, embodiments, and features, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description, and the accompanying drawings. [Brief explanation of the drawing]

[0016] [Figure 1] This figure shows an overview of a document retrieval model training method according to an exemplary embodiment. [Figure 2] This figure shows an exemplary query generation model and an exemplary document retrieval model according to exemplary embodiments. [Figure 3] Exemplary prompt templates for various datasets are shown using exemplary embodiments. [Figure 4] Exemplary embodiments illustrate comparative examples of different retriever frameworks. [Figure 5] Exemplary embodiments illustrate the differences between retriever frameworks and comparative examples of retriever frameworks having different re-rankers. [Figure 6A] This is a bar graph illustrating the effect of round-trip filtering in an exemplary embodiment. [Figure 6B] A graph showing an example of human annotation in an exemplary embodiment is shown. [Figure 6C] The results of an exemplary ablation of a query generation model by an exemplary embodiment are shown. [Figure 7] This illustrates the effects of different fine-tuned language network (FLAN) versions using exemplary embodiments. [Figure 8] This illustrates the effect of selected examples on the model output using exemplary embodiments. [Figure 9A] This example illustrates the exemplary top first word distribution for queries generated from different models within an Argumentation Analysis (ArguAna) dataset. [Figure 9B] This example illustrates the exemplary top first word distribution for queries generated from different models within the ArguAna dataset. [Figure 9C]This example illustrates the exemplary top first word distribution for queries generated from different models within the ArguAna dataset. [Figure 9D] This example illustrates the exemplary top first word distribution for queries generated from different models within the ArguAna dataset. [Figure 10A] This example illustrates exemplary fusion shots and zero-shot generated queries randomly sampled from various datasets. [Figure 10B] This example illustrates exemplary fusion shots and zero-shot generated queries randomly sampled from various datasets. [Figure 10C] This example illustrates exemplary fusion shots and zero-shot generated queries randomly sampled from various datasets. [Figure 11] An exemplary embodiment shows a table with the average query length of various datasets. [Figure 12] This figure shows the training and inference phases of a machine learning model according to an exemplary embodiment. [Figure 13] This document illustrates a distributed computing architecture by exemplary embodiment. [Figure 14] This is a block diagram of a computing device according to an exemplary embodiment. [Figure 15] This illustrates a network of computing clusters deployed as a cloud-based server system, according to an exemplary embodiment. [Figure 16] This is a flowchart of the method according to an exemplary embodiment. [Figure 17] This is a flowchart of another method according to an exemplary embodiment. [Modes for carrying out the invention]

[0017] This application relates, in one aspect, to solving the problem of data scarcity while maintaining the efficiency of a small dual encoder by leveraging the capabilities of a large language model (LLM). To address the data scarcity problem, PROMPTAGATOR combines prompts and a large language model as a query generator without fine tuning. By using a large language model, PROMPTAGATOR can generate appropriate queries from natural questions without training data. PROMPTAGATOR can use a few prompts that enable fusion shot retrieval, and significant improvements can be made by inputting these new examples within the prompts, using only 2 to 8 examples for each task. PROMPTAGATOR can amplify the capabilities of a few examples by creating task-specific prompts instead of directly training the dual encoder with a few examples. A large amount of training data, including query and document pairs, can be generated. After query generation, the task-specific noisy pair data can be used to train the task-specific dual encoder. A second iteration for training can be performed by filtering the noisy pair data to generate a clean dataset and training the task-specific dual encoder using the clean dataset.

[0018] Neural search models, such as dual encoders, can search large sets of documents containing millions to billions of passages. However, Benchmarking Information Retrieval (BEIR) has shown that neural retrievers can struggle to perform well on diverse search tasks where dedicated training data is lacking. To address this problem, several approaches focus on transferring knowledge from high-resource question-answer (QA) datasets and propose architectures with good inductive bias, such as models that enable fine-grained, token-level interactions (e.g., Contextualized Late interaction over BERT (ColBERT) and SParse Lexical AnD Expansion (SPLADE)) associated with higher inference costs.

[0019] Data augmentation through synthetic query generation generally involves question generators trained on high-resource QA datasets, which often fail to generalize well to new retrieval tasks. Some approaches leverage synthetic query generation on the target corpus to mitigate the problem of insufficient data. While effective for improving performance on QA retrieval tasks, one limitation of such approaches is that, because the query generation model is trained on a question-answer dataset, the distribution of generated questions may not match the true query distribution of the target task.

[0020] The Fusion Shot LLM query generator described herein can produce good queries without fine-tuning the model. In fact, the data generated by synthesis may be powerful enough to reduce and / or eliminate the use of annotated query document pairs from traditional high-resource datasets such as Natural Questions.

[0021] To ensure the quality of the generated data, we will describe a filtering strategy that uses only the generated data. This filtering strategy can remove ambiguous, general, and low-quality questions, significantly improving search performance. While an LLM is used to generate the training data, note that the search model during inference can be any standard-sized retriever.

[0022] Existing approaches to using LLMs do not apply to task-specific fusion shot adaptations and often involve high inference costs. For example, some approaches involve using GPT-3 with dual encoders, but their embedding dimension is 12,000, which can result in an exorbitant search index footprint and inference costs for many applications. Other existing approaches involve prompting the LLM for question generation, but this approach does not use task-specific fusion shot prompts for rapid task adaptation. This approach also primarily focuses on models that re-rank top searches from existing retrievers rather than directly adapting the underlying retrievers, thereby enabling efficient searching of millions or billions of documents.

[0023] The approach described herein involves considering differences across search tasks (e.g., search intent and query distribution) and proposes a fusion shot search evaluation for a dataset (e.g., the BEIR dataset). As explained, PROMPTAGATOR is a simplified model for fusion shot search that prompts an LLM to generate synthetic task-specific training data. Thus, neural retrievers and rerankers can be trained on only a small number of supervised examples. As illustrated, PROMPTAGATOR with 2 to 8 examples can produce retrievers significantly better than existing models trained on MS MARCO or NQ with over 500,000 human-annotated examples, utilizing a more costly architecture. For example, PROMPTAGATOR may outperform ColBERT v2 and SPLADE v2 on several search tasks that can be tested, while reranking can boost results by another 5 points on standard search evaluation metrics.

[0024] System Overview Figure 1 shows an overview 100 of a document retrieval model training method according to an exemplary embodiment. Some embodiments include receiving at least two prompts (e.g., a first prompt 110 and a second prompt 115) related to a retrieval task performed against a corpus of documents 105 related to the task. Some embodiments include applying LLM 120 to generate a synthetic training dataset 130 containing a plurality of query document pairs 135 (e.g., generation 125) based on the at least two prompts (e.g., a first prompt 110 and a second prompt 115) and the corpus of documents 105. Each of the plurality of query document pairs 135 may include a query generated by synthesis and a document from the corpus of documents 105. Some embodiments include training a document retrieval model 145 (e.g., training 140) to take input queries associated with the retrieval task from the plurality of query document pairs 135 from the synthetic training dataset 130 and predict output documents to be retrieved from the corpus of documents 105. Additional details regarding each of these steps are provided below.

[0025] task This specification describes a fusion shot search approach for a variety of search tasks, each task associated with a short description and a few annotated examples to clearly illustrate the search intent. As described herein, for each new search task, a fusion shot example may be input into a large-scale language model (e.g., LLM120), such as a fine-tuned language network (FLAN). The FLAN is an LLM that has not been trained on any document search task or query generation task from documents. The FLAN may be prompted to perform query generation from documents (e.g., generation125). Generally speaking, the fusion shot examples ensure that the specific search intent of a particular task is captured. Using this query generator large-scale language model (LLM)120, a large number of relevant queries may be synthesized for any given document, resulting in rich data (e.g., multiple query document pairs135 from a synthetic training dataset130) for training a retriever (e.g., a document search model145), including a highly efficient dual encoder model.

[0026] Generally speaking, different search tasks may have different search intents, or in other words, different definitions of "relevance." For example, consider the task of searching for documents from Wikipedia. The task could be a first task of searching for entities mentioned in the query, or a second task of finding evidence to support or refute a given statement. Determining which documents are relevant to the query can vary greatly from task to task, even if the tasks share the same domain. Furthermore, different tasks may have distinct distributions of queries, even if their search intents are similar. For example, some queries may be long, constructive questions, while others may be short financial questions. The search domain can range from general web pages to Wikipedia articles, Quora questions, or papers on specific diseases.

[0027] For illustrative purposes, we describe a Fusion Shot search configuration for the BEIR benchmark. Given a large corpus (e.g., document corpus 105), a search model (e.g., document search model 145) may be configured to find documents highly relevant to a given query q, according to a predefined concept of relevance. Formally, the search task may be formulated as follows: T = {D, Q, I} (Equation 1)

[0028] Here, D={d1,d2,…,d n} is a large corpus of documents for searching (e.g., Corpus 105 of Documents), Q is the query distribution, and I is the underlying search intent for the task. Depending on the task, D may be any set of documents such as the web or Wikipedia. Search Q also varies depending on the task, e.g., a short keyword search query, a question, a discussion, etc. If I(q,d)=1, this indicates that the search intent of q is satisfied by document d. For example, in a question that responds to the task, I QA (q,d)=1 indicates that document d responds to query q. For the same (q,d) pair, the relevance can be either 1 or 0, depending on the search intent. For example, some argument retrieval tasks may attempt to find supporting arguments, while others may attempt to find counterarguments.

[0029] Generally, Target Search Corpus D T (For example, a corpus of documents for a specific task, 105, may be given, but the number of annotated query document pairs for a new task may be limited.) Existing approaches involve retrieving a new corpus D T The focus is on adapting to the query Q. T Or intention I T The divergence of the terms is not considered. As described herein, the search intent can be expressed in a short description and very few examples.

[0030] Fusion Shot Settings Humans are generally capable of understanding retrieval tasks by reading short prompts and examining a small number of examples. Therefore, it may be natural to determine whether a small number of examples (e.g., 8 or fewer) is sufficient for a machine learning model to learn a task-specific retriever. In this regard, the FuShot retrieval evaluation may be built upon the BEIR heterogeneous retrieval benchmark.

[0031] BEIR includes 18 information retrieval datasets across nine domains, including biomedical, finance, news, Twitter, Wikipedia, StackExchange, Quora, science, and a variety of other topics. These datasets cover a diverse range of search intents, such as QA search (questions to documents), duplicate question discovery (questions to questions), and fact-checking (claims to documents). The original BEIR evaluation uses a zero-shot setup, where no queries are used for training, or query document pairs from the evaluation dataset may be used.

[0032] In some embodiments, as described herein, BEIR may be modified to a fusion shot setting by randomly selecting several (e.g., 2 to 8) examples of relevant query documents within the domain as task-specific training. Generally, it is possible to obtain this number of examples. Examples may be sampled from a development set if one is available. For BEIR tasks that only have a test set, samples from the test data may be used.

[0033] PROMPTAGATOR model PROMPTAGATOR can be configured to transform a small number of examples into a larger number of examples by prompting an LLM (e.g., LLM120) to generate more data (e.g., a synthetic training dataset 130) rather than directly training a retriever (document retrieval model 145) using the examples. In some embodiments, PROMPTAGATOR may include three components: prompt-based query generation, consistency filtering, and retriever training. During prompt-based query generation, D T Task-specific prompts can be combined with a large-scale language model to generate relevant queries for all documents within the dataset. Next, a filtering step may clean the generated data based on round-trip consistency. In some embodiments, a retriever trained solely on synthetic data (e.g., multiple query document pairs 135 from a synthetic training dataset 130) may be used to filter the synthetic data. Finally, a retriever (e.g., a dual encoder) and a cross-attention reranker may be trained based on the filtered data.

[0034] FIG. 2 is a diagram 200 showing an exemplary query generation model and an exemplary document search model according to an exemplary embodiment. As described herein, PROMPTAGATOR combines a prompt with a large language model (e.g., LLM205) that has been instruction-tuned to form query generation with far fewer restrictions compared to other models. It may rely on a very small number of examples (e.g., 8 or less) to train a retriever (e.g., dual encoder 210) and may have better performance than a custom-designed model. In some embodiments, the cross-attention reranker 215 may be a list-wise model based on T5. For example, it may receive a list of documents from synthetic data 220 when a query is given as input based on a query-document pairing. Also, each query-document pair may be represented as "Query:{q} Document:{d}", and the pair may be input into the encoder of the cross-attention reranker 215 (e.g., T5 model). In some embodiments, a projection layer may be applied to the output encoding of the first token, and the output may be used as a ranking score. Also, for example, the model may be optimized using softmax cross-entropy loss with respect to a ranking list composed of positive (q,d + ) pairs and sampled negative (q,d - ) pairs (e.g., 31 sampled negative pairs). Different from monoT5, a point-wise reranker trained to generate relevance labels using an encoder-decoder model, the cross-attention reranker 215 may be a list-wise reranker and can directly optimize ranking performance.

[0035] Prompt-based query generation In some embodiments, task-specific few-shot examples 225 may be input into a large language model (e.g., LLM205). The LLM205 may be prompted to perform query generation 230 from documents. More precisely, k few-shot examples 225 are {(qi d i )} k Therefore, each example is a query (q i ~Q T ) and target task T(I T (q i d i Documents related to that query by )=1) (d i ∈D T )

[0036] Following FLAN, the command prompt can be applied to LLM205 using the following string prefixes. e doc (d i )|e query (q1)|...|e doc (d k )|e query (q k )|e doc (d) (Formula 2)

[0037] Here, "|" indicates a separator token, e doc (d) and e query (q) is the task-specific document and query description, respectively, and d indicates a new document presented during inference. In some embodiments, LLM205 is e doc (d) = "Argument:{d}" and e query = You may be prompted to generate a counter-argument based on "Counter Argument:{q}".

[0038] Figure 3 shows exemplary prompt templates 300 for various datasets according to exemplary embodiments. A complete set of exemplary descriptions used in prompts is shown in Figure 3. The datasets include Argumentation Analysis (ArguAna), Financial Question Answering (FiQA), natural multi-hop Question Answering (HotpotQA), entity search over the database-pedia (DBpedia) knowledge base (DBpedia-Entity), NutritionFacts Corpus (NFCorpus), Touch e 2020, Text REtrieval Conference data for Covid (TREC-Covid), Scientific Claims (SciFact), Scientific Documentations (SciDocs), and Fact Extraction and Verification (FEVER).

[0039] Referring again to Figure 2, LLM205 is,

number

number

number

[0040] In some embodiments, such a prompt is D TThis process was performed on documents from 235, amplifying information from a small number of examples (e.g., task-specific fusion shot example 225) into a large synthetic dataset (e.g., synthetic data 220), and then synthesizing...

number

[0041] In some embodiments, LLM205 may be an FLAN model, and the query generator is p FLAN (q|d) may be denoted as (q|d). The FLAN may be trained on a set of tasks described by instructions and generally achieve high zero-shot / fu-shot performance on unseen tasks. Some embodiments involve using 137 billion parameter checkpoints. Up to eight examples may be used during prompt engineering, and this number may be further reduced if it exceeds the input length limit of the FLAN. Also, for example, individual queries and documents in the examples may be manually truncated if they are longer than the threshold length. In another example, up to one million documents may be randomly sampled from each corpus, and eight questions may be generated for each document using sampling-based decoding with a temperature parameter of 0.7.

[0042] Data generated by round-trip filtering In some embodiments, round-trip filtering 240 may be applied to improve the quality of the synthetic dataset 220. Generally speaking, the synthetic query generated from passage d

number

number

number

number

[0043] Round-trip filtering 240 can be effective for generating synthetic questions for QA tasks. However, existing techniques typically rely on question-answering models with reverse filtering. Since not all search tasks are similar to question-answering, such existing approaches may not always be sufficient. Instead, in training step 250, training may be applied to an initial retriever (e.g., a small dual encoder) from unfiltered synthetic data (e.g., synthetic data 220). The trained initial retriever (e.g., a trained small dual encoder) can then be used to filter the synthetic data. In general, such an approach works well across different search intents observed in BEIR. More precisely, synthetic query document pairs

number

number

number

[0044] Round-trip filtering as latent variable modeling In some embodiments, round-trip filtering 240 may include displaying queries as estimated latent variables. Each query q is a latent variable, and there is a distribution p(d|q,θ) for which documents are retrieved for that query. * Consider a hypothetical graphical model whose observed variables follow ). Here, θ * This represents the parameters of the hypothetical "optimal" retriever that can select the "best" document for the query. To generate synthetic data, the query is a posterior distribution, p(q|d,θ). * ) may be sampled from, which may be given as follows according to Bayes' rule.

number

[0045] Here, p(q|θ) * ) is the prior distribution across the queries. In some aspects, p(q|θ) * It can be assumed that p(q|θ) is a "non-informative prior distribution" that is uniform over all q. In terms of notation, p(q|θ) * ) may be denoted as p(q). θ is a parameter of the optimal retriever. * If θ is known, the formula in Equation 3 can be calculated directly. However, *θ is generally an unknown variable, and therefore the variable θ can be estimated using expectation maximization (EM). Generally, the round-trip filtering algorithms described herein are some versions of EM. EM and other latent variable learning methods may be used to impute missing data.

[0046] In EM, the variable θ can be estimated by approximately maximizing the marginal likelihood of the observed variable, as described by the following relationship.

number

[0047] For example, in EM, an initial estimation of p(q|d) may be performed for all documents d. In some embodiments, the FLAN query generator performs an initial estimation of p FLAN It can be used as (q|d). The M step of EM may involve a calculation described by the following relation:

number

[0048] This is paired with a query generated by LLM (e.g., FLAN) as described herein. T From the document, the initial retriever

number

number

number

[0049] As shown by Equation 6,

number

[0050] In some embodiments,

number

number

number

[0051] Such EM-like procedures may be repeated until convergence occurs, but generally, a single rounding of filtering can provide sufficient quality improvement. In some embodiments, training of the final retriever model may be seen as another M step.

[0052] FuShot PROMPTAGATOR Retriever As described herein, the data generated by synthesis (e.g., synthetic data 220) is task-specific for tasks where fine-tuning within a domain may be difficult due to insufficient data. neural It is configured to allow training of the retriever. In some embodiments, the search architecture of the dual encoder 210 may be used in conjunction with a fine-tuning recipe for pre-training.

[0053] For example, pre-training 245 may involve pre-training of a retriever (e.g., dual encoder 210). In some embodiments, pre-training may be performed on C4 using a cropping task independent of the Contriever, where two random crops from the same document may be considered artificial positive (query, document) pairs, and training may involve cross-entropy loss for random negatives in a batch in training step 250. Subsequently, the dual encoder 210 uses the random negatives in the batch to generate prompt-based queries from

number

[0054] Some embodiments may include a modified PROMPTAGATOR, referred to herein as PROMPTAGATOR++. Generally, this may be a reranker (e.g., a cross-attention reranker 215) trained on the same synthetic data (e.g., in the synthetic dataset 220) generated from a prompt-based QGen. The QGen may be configured to refine the retrieved candidates using the slower but more accurate cross-attention reranker 215. The cross-attention reranker 215 may also be trained using cross-entropy loss (e.g., with 31 negatives 255 sampled from the top 200 passages retrieved by the PROMPTAGATOR retriever), which approximates the inference time distribution (e.g., reranking the top 200 from the retriever).

[0055] Zero Shot PROMPTAGATOR In some embodiments, prompt-based query generation may be configured to be performed in a zero-shot manner, and the zero-shot prompt 260 may be applied regardless of the target task. For example, the zero-shot prompt 260 may be "Read {d} passage and generate a query," where {d} is the text of the document. The zero-shot PROMPTAGATOR and zero-shot PROMPTAGATOR++ may be configured by training a retriever (e.g., document retrieval model 145, dual encoder 210) and a reranker (e.g., cross-attention reranker 215) with such data. These models may serve as a baseline to illustrate the benefits of adapting the zero-shot prompt to the target task.

[0056] Comparison with existing approaches Figure 4 shows 400 comparative examples of different retriever frameworks by exemplary embodiments. For example, the PROMPTAGATOR configuration can be compared to several other approaches. Some aspects of the PROMPTAGATOR approach are simpler than those of the comparative approaches. For example, the dual encoder 210 may not rely on hard negative mining, distillation from cross-attention teachers, and / or token-level search. Also, the reranker, such as the cross-attention reranker 215 (e.g., having 125 million parameters), may be configured to be smaller than the rerankers in other approaches. In some embodiments, as described herein, simpler and smaller architectures may be designed to achieve high performance results when trained with fusion-adapted synthetic data. Compared to the Inquisitive Parrots for Search (InPars) and Unsupervised Pass-sage Re-ranking (UPR) approaches, the PROMPTAGATOR approach employs task-specific fusion-adaptation. In contrast, the prompts in InPars and UPR are task-independent and therefore have the same limitations as the previous query generation approaches. Another difference between the approaches is that existing approaches focus on reranking, while the PROMPTAGATOR approach allows for fusion learning for both reranking and retrieval.

[0057] evaluation As described herein, the PROMPTAGATOR approach can be evaluated using the BEIR benchmark. Ablation studies and qualitative analyses may also be performed, for example. In some embodiments, the FLAN training set may overlap with two BEIR datasets, NQ and Quora. Also, for example, some existing approaches use this dataset for fully supervised learning and therefore do not report fu-shot or zero-shot results on MS MARCO. Thus, evaluation may not be based on the MS MARCO, NQ, and Quora datasets. Instead, evaluation may be performed using the normalized impairment cumulative gain (nDCG@10) over 10 items, which is a search evaluation metric for BEIR.

[0058] For query generation in PROMPTAGATOR, the query may be sampled from FLAN at a given temperature (e.g., 0.7). In some embodiments, better results with respect to MS MARCO may be obtained by setting the filtering threshold K to 1 for round-trip filtering. Therefore, the filtering threshold K may be set to 1 for all BEIR datasets. In some embodiments, the dual encoder may be implemented based on a Generalizable T5 Retriever (GTR). To ensure efficiency, a T5-based encoder architecture consisting of 110 million parameters may be utilized. For the PROMPTAGATOR++ re-ranking model, an encoder checkpoint of T5-based version 1.1 with 125 million parameters may be initialized. In some embodiments, the top 200 candidates retrieved from the PROMPTAGATOR dual encoder may be re-ranked at inference time.

[0059] In some embodiments, the hyperparameters may include a batch size of 6k. However, since some BEIR corpora contain only a few thousand documents, multiple related documents may appear in the same batch. This can, in some cases, negatively interact with the in-batch softmax loss. Therefore, the dataset may be divided into groups based on corpus size, such as small groups (corpus size < 50k), medium groups (corpus size 50k-500k), and large groups (corpus size > 500k). For dual encoder training, a batch size of 128 may be used for small datasets, and a batch size of 6k may be used for others. Also, for example, 5k steps may be fine-tuned for large datasets, and 1k for others. For ranking models, a batch size of 64 may be used for all datasets, with large datasets being fine-tuned to 20k steps and others to 5k steps.

[0060] result Figure 5 shows 500 comparative examples of retriever frameworks with different retriever frameworks and rerankers, according to exemplary embodiments. For example, experimental results for such models are shown. As illustrated, the zero-shot PROMPTAGATOR can serve as a strong baseline, favorably compared to other search baselines trained on O(100k) examples from MS MARCO. Also, for example, the fu-shot PROMPTAGATOR can significantly improve upon the zero-shot PROMPTAGATOR, increasing the average nDCG@10 by more than 2 points. This evaluation highlights the impact of fu-shot learning. For example, the fu-shot PROMPTAGATOR can outperform strong baselines such as GenQ and GPL, which also use query generation to augment the training data, as well as ColBERT v2 and SPLADE v2, which depend on token-level interaction architectures and distillation recipes, despite having simple training procedures and model architectures.

[0061] As illustrated, the reranker PROMPTAGATOR++ can boost performance by another 5 points compared to nDCG@10. As a result, it can provide a significant improvement over UPR, which uses an instruction-tuned LLM similar to T0 instructions and FLAN. It can also outperform monoT5-3B, which has been previously shown to achieve the previous state-of-the-art reranking performance in BEIR. Existing reranker approaches generally use a large 3 billion parameter model for better generalization, but PROMPTAGATOR++ uses a smaller (e.g., 125 million) reranker.

[0062] Comparing the FuShot PROMPTAGATOR to the baseline, significant improvements can be observed in Webis-Touch e2020 (Touch e), followed by ArguAna (arg). The goal of Webis-Touch e2020 is to search for documents on controversial topics, such as "Should felons who have completed their sentences be given the right to vote?" The goal of ArguAna is to find counterarguments to the input argument, which may be only a few sentences long. Both tasks differ generally from the traditional QA search data used by other models, which is occupied by factoid questions. However, the FuShot PROMPTAGATOR can be well adapted to this task in a small number of examples.

[0063] Effects of round-trip filtering Figure 6A is a bar graph 600A showing the effect of round-trip filtering in an exemplary embodiment. Figure 6A shows the difference in quality of the Fusion Shot PROMPTAGATOR between filtered and unfiltered models. Generally, filtering can improve performance for 8 out of 11 datasets, resulting in an average improvement of 2.5 points, demonstrating the effectiveness of the filtering strategies described herein. Filtering appears to negatively impact the model quality of NFCorpus and SciFact, which may be due to the relatively small size of these datasets relative to the generated queries and may indicate retrieval overfitting.

[0064] Generated queries vs. human-annotated queries Figure 6B is a graph showing a human-annotated example according to an exemplary embodiment. In Figure 6B, an 8-shot PROMPTAGATOR is evaluated on MS MARCO and compared to a dual encoder trained on supervised data from MS MARCO. For evaluation purposes, no additional components have been added to maintain a simpler comparison. MS MARCO can be chosen because it has sufficient labeled data for this task, and neither FLAN nor the models described herein have been trained on MS MARCO examples. The results show that the 8 examples and LLM can replace a large portion of the supervised examples.

[0065] Compare with other query generation approaches Figure 6C shows the result of an exemplary ablation of a query generation model using an exemplary embodiment, 600C. Figure 6C compares the zero-shot PROMPTAGATOR with two other query generation approaches: GenQ, a conventional system using a T5 query generation model trained on MS MARCO, and natural questions-QGen (NQ-QGen), a T5 QGen model fine-tuned on NQ. Several advantages of the zero-shot PROMPTAGATOR are shown, for example, by significant difference over both baseline models. In general, NQ-QGen uses the same filtering, dual encoder training, batch size, and training steps as PROMPTAGATOR, thereby ensuring a fair comparison of the query generators. Result 600C suggests that the contributing factor to PROMPTAGATOR may be better queries by prompting the LLM, and not necessarily due to a specific training recipe or specific hyperparameters.

[0066] Comparison of Fusion Shot and Zero Shot or higher Referring again to the 500 comparative examples of different retriever frameworks and retriever frameworks with rerankers in Figure 5, the FuShot PROMPTAGATOR appears to generally outperform the ZeroShot PROMPTAGATOR, with the exception of Climate-FEVER. The Climate-FEVER dataset annotates query document pairs using one of three tags: "support," "refute," or "not enough information." However, BEIR generally considers these three annotations relevant, which can lead to undesirable results. Using query document pairs annotated with "not enough information" in a prompt can negatively impact the generation quality. Using such pairs in a FuShot prompt can negatively impact query generation. In some evaluations, the FuShot prompt from FEVER may be used because the two datasets share the same corpus and similar search intent. Based on such modified annotated examples, the FuShot PROMPTAGATOR appears to perform better than the ZeroShot PROMPTAGATOR.

[0067] Impact of FLAN Version FLAN was trained on a set of datasets that partially overlap with BEIR. Specifically, FLAN includes Natural Questions (NQ) and Quora. However, FLAN is not trained on query document pairs from NQ or Quora. Additional ablation experiments may be designed to determine whether the inclusion of this data could bias the results regarding the final search evaluation. For example, based on the FLAN recipe, an additional LLM could be trained by excluding both the NQ and Quora datasets.

[0068] Figure 7 shows the effect of different FLAN versions according to an exemplary embodiment. The results are divided into two tables, 700a and 700b. The last column 705 of table 700b shows the average of the respective rows from tables 700a and 700b. For example, the average value in the last column 705. 48 .5 is the average of the values ​​in the first row of tables 700a and 700b. Similarly, the average value of 47.0 in the last column 705 is the average of the values ​​in the second row of tables 700a and 700b.

[0069] Figure 8 shows the effect of selected examples on the model output according to an exemplary embodiment. For example, Figure 8 shows the effect and / or improvement of using eight examples to fine-tune an existing retrieval model (e.g., GTR). Accuracy may decrease slightly, but overall performance is better than the existing retriever. The results are divided into two tables, 800A and 800B. The last column 805 of table 800B shows the average of the respective rows in tables 800A and 800B. For example, the average value of 40.4 in the last column 805 is the average of the values ​​in the first row of tables 800A and 800B. Similarly, the average value of 38.7 in the last column 805 is the average of the values ​​in the second row of tables 800A and 800B.

[0070] qualitative analysis Some of the advantages of the FuShot PROMPTAGATOR can be explained based on the distribution of queries generated by different query generation methods for ArguAna. For example, for each distribution, a histogram of the first word of each query can be shown.

[0071] Figures 9A–9D show exemplary first-word distributions for queries generated from different models in the ArguAna dataset by exemplary embodiments. Figure 9A shows the first-word distribution for Gold queries. Figure 9B shows the first-word distribution for FuShot PROMPTAGATOR, from which queries are generated. As shown, the distribution shown in Figure 9B is closer to the actual distribution shown in Figure 9A. Figure 9C shows the first-word distribution for NQ-QGen, from which queries are generated. NQ-QGen generates mostly questions, even though queries in this task are generally discussions rather than questions. Figure 9D shows that FuShot FLAN can generate a variety of queries even when there are only four examples in the prompt. Additional examples are shown in Figures 10A–10C.

[0072] Figures 10A–10C show exemplary fusion shot and zero shot generated queries randomly sampled from various datasets according to exemplary embodiments. Generally, fusion shot generated queries appear closer to the original queries, while zero shot queries appear almost like questions. For example, on the ArguAna dataset, fusion shot queries are generally longer and more claim-like. Also, zero shot queries are generally short, question-like queries. In the case of the HotpotQA dataset, even though both fusion shot and zero shot queries appear to generate question-like queries, fusion shot queries appear to generate multi-hop questions, while zero shots appear to generate single-hop questions.

[0073] Referring to Figure 10A, columns 10A-C1 show an exemplary paragraph, columns 10A-C2 show the query generated by the fusion shot process, columns 10A-C3 show the query generated by the zero shot process, and columns 10A-C4 show an analysis demonstrating the analysis on the ArguAna dataset. The queries generated by fusion shot are more statement-like and longer than the queries generated by zero shot.

[0074] Referring to Figure 10B, columns 10B-C1 show an exemplary paragraph, columns 10B-C2 show queries generated by the fusion shot process, columns 10B-C3 show queries generated by the zero shot process, and columns 10B-C4 show an analysis demonstrating the analysis on the Touch▲e▼-2020 dataset, where fusion shots generate more argumentative, argument-like queries, while zero shots generate potentially grammatically incorrect, random statements.

[0075] Referring to Figure 10C, columns 10C-C1 show an exemplary paragraph, columns 10C-C2 show queries generated by the fusion shot process, columns 10C-C3 show queries generated by the zero shot process, and columns 10C-C4 show an analysis demonstrating the analysis on the HotpotQA dataset, where fusion shot queries can be multi-hop questions, while zero shot does not generate multi-hop questions.

[0076] Direct use of fusion shot examples Referring again to Figure 8, the effect of directly using the fusion shot example can be analyzed by fine-tuning the task-specific model using a GTR dual encoder (e.g., 110 million) with the fusion shot example shown in Figure 3. As expected, it does not have a significant impact on the GTR base, and the average nDCG decreases from 40.4 to 38.7.

[0077] Query generation statistics Figure 11 shows Table 1100 with the average query lengths of various datasets, according to an exemplary embodiment. In Figure 11, the lengths of questions generated by different query generation systems are analyzed. NQ-QGen appears to generate shorter queries because its query generation model is fine-tuned on the NQ dataset, and the generated questions appear to have similar lengths to NQ questions. The average for NQ-QGen is shown as 9.6. However, ZeroShot PROMPTAGATOR appears to have a higher variance with respect to length compared to NQ-QGen. The average for ZeroShot PROMPTAGATOR is shown as 13.5. Also, for example, FuShot PROMPTAGATOR appears to have a significantly larger variance with respect to the length of the generated queries. The average for FuShot PROMPTAGATOR is shown as 17.8.

[0078] Computational usage and environmental impact In some embodiments, 137 billion FLANs based on LaMDA may be used. LaMDA can be pre-trained on a massive corpus consisting of 1.56 trillion words, costing 451 megawatt-hours (MWh) of energy and a carbon footprint of 25.2 tCO2e. PROMPTAGATOR generates 29.23 million queries * 2 prompts = 58.46 million queries, totaling 610 million words.

[0079] Comparison with existing models Some existing neural retrievers employ a dual-encoder architecture, encoding queries and documents separately into dense vectors and using Maximum Inner Product Search (MIPS) to retrieve documents. Existing approaches have primarily focused on the following aspects: developing better pre-training tasks, improving contrast negatives, and improving generalization across different domains.

[0080] While dual encoders enable fast searches, their frequency of appearance can be limited by the fact that their score is merely a dot product between the query vector and the document vector. Cross-attention rerankers are generally used to rerank retrieved candidates because they can explicitly model the interaction between queries and document tokens. Distilling cross-attention models into dual encoders has been effective in closing the gap between dense retrievers and cross-attention models.

[0081] An alternative to bridge the gap between dense retrievers and cross-attention models is to allow some degree of fine-grained query-document interaction in the retriever. For example, queries and documents can be represented and retrieved using multiple vectors instead of a single vector. The ColBER, COIL, and SPLADE approaches use token-level interaction between queries and documents. These models cannot directly use the MIPS algorithm because they do not simply model the dot product. Therefore, these models typically have much higher inference / service costs compared to dual encoders.

[0082] In general, prompted LLMs can be used for query generation to improve search reranking. For example, UPR uses prompted LLMs to directly rerank passages. InPars uses fusion shot prompts with GPT-3 to generate synthetic data for training a T5-based reranker. We have tested InPars on multiple search datasets, and it is based on task-independent prompts built from MS MARCO and does not involve task-specific fusion shot learning. InPars also focuses solely on reranking, while PROMPTAGATOR's approach addresses full-scale search. Pre-trained LLMs have significantly advanced fusion shot learning based on prompting strategies such as in-context learning and command prompts. Some approaches involve fine-tuning the LLM specifically for fusion shot learning, while others do not. As described herein, PROMPTAGATOR's approach uses the LLM as a fusion shot data generator.

[0083] technical improvements The techniques described herein enable fusion shot retrieval, and models can achieve significant improvements using as few as 2 to 8 examples (referred to herein as prompts) per task. Generally, very few prompts can train an LLM to automatically generate synthetic data, which can then be used to train a document retrieval model. Existing models attempt to generate training data directly. The term “prompt” as used herein may be general in the case of zero shots and task-specific in the case of fusion shots. Prompts are combined with large language models to generate queries for a target task using a corpus of documents.

[0084] PROMPTAGATOR can achieve good performance with dual encoders without requiring a highly designed search architecture. For example, a dedicated architecture is not necessary, and existing large-scale language models can be used.

[0085] The techniques described herein can generate queries for searching about a wide variety of target tasks.

[0086] The search can be performed without a reranker.

[0087] Prompting using instruction-tuned large-scale language models can generally address the major limitations of other models that focus on question-answering datasets.

[0088] The techniques described herein demonstrate that good query generation can significantly improve the performance of dual encoders, and as a result, PROMPTAGATOR is remarkably more efficient compared to other LLM applications for searching.

[0089] The PROMPTAGATOR-based dual encoder performs better than ColBERTv2 and SPLADE v2 for 11 search tasks.

[0090] PROMPTAGATOR can manipulate LLM to generate text tailored to the target task, thus further improving search performance.

[0091] Unlike existing fusion shot search approaches, which are fine-tuning based and often require thousands of labeled examples, the technique described herein generates labeled examples synthetically using a small number of task-specific prompts.

[0092] Zero-shot PROMPTAGATOR brings about more changes with respect to length queries.

[0093] A comparison of the distribution of the first words of queries generated by different query generation methods shows that the distribution for synthetic queries generated by FuShot PROMPTAGATOR is much closer to the distribution of actual queries generated by humans.

[0094] The model reduces computational usage and environmental impact. For example, the energy cost and carbon footprint for a pre-trained model were 451 MWh and 26 tCO2e, respectively. In some embodiments, the additional instruction tuning gradient steps for fine-tuning the FLAN may be less than 2% of the number of pre-training steps, and therefore the estimated additional energy cost can be relatively small.

[0095] While FLAN is used herein for illustrative purposes only, LLM may be any model, including but not limited to variations of T0 or PaLM.

[0096] Exemplary Uses The model can be deployed on a server.

[0097] The model can be tailored to specific types of tasks, knowledge domains, datasets, organizations, and so on.

[0098] In some embodiments, the model may be applied to search for documents supporting arguments and counterarguments. For example, the query might be "The company should pay its employees at least X," and the documents in the query document pair might include all posts on social networking sites that support or oppose the query.

[0099] In some embodiments, the model can be applied to perform semantic searches within a database.

[0100] Search tasks can be performed in different settings such as finance, healthcare, education, and law. Many such settings involve data with protected information, and organizations may prefer not to share protected data. Models may be provided to organizations (e.g., via application programming interfaces (APIs), as software as a service (SaaS), machine learning as a service (MLaaS), etc.), and organizations can enter a few prompts to train the model and apply the trained model, while all operations are performed and the data remains inside the organization's firewall.

[0101] Search tasks may include searching for documents from a question, searching for questions from a question, searching for documents from an argument, searching for documents that support an argument, or searching for documents that support a counter-argument.

[0102] In some embodiments, the task may be to retrieve information from an email system.

[0103] As used herein, the term “document” generally refers to any information retrieved from a corpus of documents in response to a query. The information retrieved may include documents, collections of documents, images, parts of images, videos, sound clips, parts of text from one or more documents in the corpus of documents, summaries and / or excerpts based on parts of text from one or more documents in the corpus of documents. Furthermore, the term “corpus of documents” generally refers to documents relevant to a particular task. For example, a query might be a legal query related to a particular client’s litigation issue, and the corpus of documents would correspond to client-specific documents, or more generally, litigation-specific documents. Another example is a query related to a medical question concerning a particular disease, and the corpus of documents would correspond to documents related to the disease or the disease in question. Generally, the user may indicate and / or select a relevant corpus of documents. A document corpus may include documents stored in one or more databases (e.g., mobile, desktop, cloud, etc.), email folders, posts to social networking sites, collections of web pages, scientific databases, images in image libraries, videos in video libraries, audio, music libraries, etc. Generally, a model can be applied to any collection of data from which information can be retrieved, provided that the information is structured to be searchable. This may include text documents, images, videos, audio files, etc. Document retrieval, for example, may also include speech-to-text conversion.

[0104] In some embodiments, the query may relate to the issue being searched for. For example, a query to an email folder might state, "Find information regarding a dispute over baggage claim," and the retrieved documents may be one or more emails exchanged as part of the dispute process. In some embodiments, the documents may be summaries outlining the history of the dispute, including relevant dates and issues. Such documents may be prepared based on a combination of sentiment analysis, natural language processing (NLP), and the like.

[0105] These and other exemplary uses are intended within the scope of this disclosure.

[0106] Training machine learning models to generate inference / prediction Figure 12 shows Figure 1200 illustrating training phases 1202 and inference phases 1204 of a trained machine learning model(s) 1232 according to an exemplary embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about the training data (and the patterns within it). The resulting trained machine learning algorithms are sometimes referred to as trained machine learning models. For example, Figure 12 shows training phase 1202, in which one or more machine learning algorithms(s) 1220 are trained on training data 1210 to become a trained machine learning model(s) 1232. Then, during inference phase 1204, the trained machine learning model(s) 1232 may receive input data 1230 and one or more inference / prediction requests 1240 (perhaps as part of input data 1230) and respond by providing one or more inferences and / or predictions 1250 as output.

[0107] Therefore, a trained machine learning model(s) 1232 may include one or more models of one or more machine learning algorithms 1220. The machine learning algorithms 1220 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, large-scale language models (LLMs), neural search systems, appropriate statistical machine learning algorithms, and / or heuristic machine learning systems as described herein). The machine learning algorithms 1220 may be supervised or unsupervised, and may implement any appropriate combination of online and offline learning.

[0108] In some examples, machine learning algorithms 1220 and / or trained machine learning models 1232 can be accelerated using on-device coprocessors such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs). Such on-device coprocessors can be used to accelerate machine learning algorithms 1220 and / or trained machine learning models 1232. In some examples, trained machine learning models 1232 can be trained to provide inference on a particular computing device, reside, run, and / or otherwise perform inference on a particular computing device.

[0109] During training phase 1202, machine learning algorithms 1220 may be trained by providing at least training data 1210 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning includes providing some (or all) of the training data 1210 to machine learning algorithms 1220, and allowing machine learning algorithms 1220 to determine one or more output inferences based on the provided portion (or all) of the training data 1210. Supervised learning includes providing some (or all) of the training data 1210 to machine learning algorithms 1220, allowing machine learning algorithms 1220 to determine one or more output inferences based on the provided portion of the training data 1210, where the output inferences are either accepted or corrected based on the correct results related to the training data 1210. In some examples, supervised learning of machine learning algorithm(s) 1220 may depend on a set of rules and / or labels about the training input, and the set of rules and / or labels may be used to correct the inference of machine learning algorithm(s) 1220.

[0110] Semi-supervised learning involves having correct results for some, but not all, of the training data 1210. During semi-supervised learning, supervised learning is used for the portion of the training data 1210 for which correct results are obtained, and unsupervised learning is used for the portion of the training data 1210 for which correct results are not obtained. Reinforcement learning involves a machine learning algorithm(s) 1220 receiving a reward signal for prior inferences, which may be numerical. During reinforcement learning, the machine learning algorithm(s) 1220 may output inferences and receive a reward signal in response, and the machine learning algorithm(s) 1220 may be configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm(s) 1220 and / or the trained machine learning model(s) 1232 may be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0111] In some examples, the machine learning algorithm(s) 1220 and / or the pre-trained machine learning model(s) 1232 may utilize transfer learning techniques. For example, the transfer learning technique may involve the pre-trained machine learning model(s) 1232 being pre-trained on a set of data and then further trained using training data 1210. More specifically, the machine learning algorithm(s) 1220 may be pre-trained on data from one or more computing devices and the resulting pre-trained machine learning model provided to computing device CD1, which is intended to run the pre-trained machine learning model during the inference phase 1204. Then, during the training phase 1202, the pre-trained machine learning model may be further trained using training data 1210, which may be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1220 and / or the pre-trained machine learning model using training data 1210 from the data of CD1 may be performed using either supervised or unsupervised learning. Training stage 1202 can be completed when machine learning algorithms 1220 and / or pre-trained machine learning models are trained on at least training data 1210. The resulting machine learning models can be used as at least one of the trained machine learning models 1232. Training data 1210 may consist of multiple query document pairs, each query document pair containing a corpus of synthesized queries and documents.

[0112] In particular, once training phase 1202 is complete, the trained machine learning model(s) 1232 may be provided to the computing device if they are not already on the computing device. Inference phase 1204 may begin after the trained machine learning model(s) 1232 have been provided to computing device CD1.

[0113] During the inference phase 1204, the trained machine learning model(s) 1232 may receive input data 1230, generate and output one or more corresponding inferences and / or predictions 1250 about the input data 1230. Thus, the input data 1230 may be used as input to the trained machine learning model(s) 1232 to provide the corresponding inferences and / or predictions 1250 to kernel and non-kernel components. For example, the trained machine learning model(s) 1232 may generate inferences and / or predictions 1250 in response to one or more inference / prediction requests 1240. In some examples, the trained machine learning model(s) 1232 may be run by part of other software. For example, the trained machine learning model(s) 1232 may be run by an inference or prediction daemon so that it is readily available to provide inferences and / or predictions on demand. The input data 1230 may include data from computing device CD1 running a trained machine learning model(s) 1232, and / or input data from one or more computing devices other than CD1. For example, the input data 1230 may include at least two prompts related to a search task performed on a corpus of documents related to the task. Alternatively, for example, the input data 1230 may include input queries related to a search task performed on a corpus of documents related to the task. Other types of input data are similarly possible.

[0114] The inferences(s) and / or predictions(s)1250 may include output documents, output query document pairs, numerical values ​​(e.g., search scores), and / or other output data produced by the trained machine learning model(s)1232 that operates on the input data1230 (and training data1210). In some examples, the trained machine learning model(s)1232 uses the output inferences(s) and / or predictions(s)1250 as input feedback. 1260 It can be used as such. A trained machine learning model (or multiple models) 1232 can also rely on past inferences as input to generate new inferences.

[0115] Large-scale language models and / or document retrieval models can be examples of machine learning algorithms 1220. After training, a trained version of a neural network can be an example of a trained machine learning model 1232. In this approach, one or more inference / prediction requests 1240 can be requests responding to an input query, and the corresponding examples of inference and / or predictions 1250 can be documents predicted from a corpus of documents responding to the input query.

[0116] In some examples, a single computing device CD_SOLO might contain a trained version of a document retrieval model. The computing device CD_SOLO could then receive requests in response to input queries and use the trained version of the neural network to predict documents from a corpus of documents that respond to the input queries.

[0117] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide predicted documents. For example, the first computing device CD_CLI may generate a request in response to an input query and send it to the second computing device CD_SRV. CD_SRV may then use a trained version of a neural network to predict a document from a corpus of documents that respond to the input query and respond to a request from CD_CLI for the predicted document. Upon receiving a response to the request, CD_CLI may then provide the requested response to the input query (e.g., using a user interface and / or display, print copy, electronic communication, etc.).

[0118] Examples of data networks Figure 13 shows a distributed computing architecture 1300 according to an exemplary embodiment. The distributed computing architecture 1300 includes server devices 1308, 1310 configured to communicate with programmable devices 1304a, 1304b, 1304c, 1304d, and 1304e via a network 1306. The network 1306 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public internet, or any other type of network configured to provide a communication path between networked computing devices. The network 1306 may also correspond to one or more combinations of LANs, WANs, corporate intranets, and / or the public internet.

[0119] Although Figure 13 shows only five programmable devices, the distributed application architecture may serve tens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 1304a, 1304b, 1304c, 1304d, 1304e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, desktop computer, wearable computing device, head-mountable device (HMD), network terminal, or mobile computing device. In some examples, such as those shown by programmable devices 1304a, 1304b, 1304c, and 1304e, the programmable devices may be directly connected to network 1306. In other examples, such as those shown by programmable device 1304d, the programmable devices may be indirectly connected to network 1306 via an associated computing device, such as programmable device 1304c. In this example, programmable device 1304c may function as an associated computing device for passing electronic communications between programmable device 1304d and network 1306. In other examples, such as that shown by the programmable device 1304e, the computing device may be part of and / or inside a means of transport such as a car, truck, bus, boat or ship, or airplane. In other examples not shown in Figure 13, the programmable device may be connected to the network 1306 both directly and indirectly.

[0120] Server devices 1308 and 1310 may be configured to perform one or more services requested by programmable devices 1304a to 1304e. For example, server devices 1308 and / or 1310 may provide content to programmable devices 1304a to 1304e. Content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. Content may include compressed content and / or uncompressed content. Content may be encrypted and / or decrypted. Other types of content are also possible.

[0121] As another example, server devices 1308 and / or 1310 can provide programmable devices 1304a-1304e with access to software for databases, search, computation, graphics, audio, video, World Wide Web / Internet access, and / or other functions. Many other examples of server devices are similarly possible.

[0122] Computing device architecture Figure 14 is a block diagram of an exemplary computing device 1400 according to an exemplary embodiment. In particular, the computing device 1400 shown in Figure 14 may be configured to perform prompt-based query generation and / or at least one function for method 1600 and / or method 1700.

[0123] The computing device 1400 may include a user interface module 1401, a network communication module 1402, one or more processors 1403, data storage 1404, one or more cameras 1412, one or more sensors 1414, and a power supply system 1416, all of which may be linked to one another via a system bus, network, or other connection mechanism 1405.

[0124] The user interface module 1401 may be operable to transmit data to and / or receive data from external user input / output devices. For example, the user interface module 1401 may be configured to transmit data to and / or receive data from user input devices such as touchscreens, computer mice, keyboards, keypads, touchpads, trackballs, joysticks, speech recognition modules, and / or other similar devices. The user interface module 1401 may also be configured to provide output to user display devices such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices that are currently known or will be developed in the future. The user interface module 1401 may also be configured to generate audible output using devices such as speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. The user interface module 1401 may further comprise one or more haptic devices capable of generating tactile outputs, such as vibration and / or other outputs detectable by touch and / or physical contact with the computing device 1400. In some examples, the user interface module 1401 may be used to provide a graphical user interface (GUI) for utilizing the computing device 1400, such as the graphical user interface of a mobile phone device. In some examples, the user interface module 1401 may be used to provide user-adjustable controls for receiving threshold distances.

[0125] The network communication module 1402 may include one or more devices providing one or more wireless interfaces 1407 and / or one or more wired interfaces 1408 that can be configured to communicate over a network. The wireless interfaces 1407 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth® transceivers, Zigbee® transceivers, Wi-Fi® transceivers, WiMAX® transceivers, LTE® transceivers, and / or other types of wireless transceivers that can be configured to communicate over a wireless network. The wired interfaces 1408 may include one or more wired transmitters, receivers, and / or transceivers, such as Ethernet transceivers, Universal Serial Bus (USB) transceivers, or similar transceivers that can be configured to communicate over twisted pair wire, coaxial cable, fiber optic link, or a similar physical connection to a wired network.

[0126] In some examples, the network communication module 1402 can be configured to provide reliable, secure, and / or authenticated communication. For each communication described herein, information to facilitate reliable communication (e.g., guaranteed message delivery) may be provided, perhaps as part of the message header and / or footer (e.g., packet / message ordering information, encapsulation header and / or footer, size / time information, and transmit verification information such as cyclic redundancy check (CRC) and / or parity check values). The communication may be protected (e.g., encoded or encrypted) and / or decrypted / decoded using one or more encryption protocols and / or algorithms, including but not limited to Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Edelman (RSA) algorithm, Diffie-Hellman algorithm, Secure Sockets Protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). Other encryption protocols and / or algorithms may be used, in addition to or in the same manner as those enumerated herein, to protect communications (and subsequently decrypt / decode them).

[0127] One or more processors 1403 may include one or more general-purpose processors and / or one or more dedicated processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.). One or more processors 1403 may be configured to execute computer-readable instructions 1406 and / or other instructions described herein that are contained in the data storage 1404.

[0128] The data storage 1404 may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1403. The one or more computer-readable storage media may include volatile and / or non-volatile storage components such as optical storage, magnetic storage, organic storage, or other memory or disk storage, which can be integrated, in whole or in part, with at least one of the one or more processors 1403. In some examples, the data storage 1404 may be implemented using a single physical device (e.g., one optical storage unit, magnetic storage unit, organic storage unit, or other memory or disk storage unit), while in other examples, the data storage 1404 may be implemented using two or more physical devices.

[0129] The data storage 1404 may include computer-readable instructions 1406 and possibly additional data. In some examples, the data storage 1404 may include storage necessary to perform at least some of the methods, scenarios, and techniques described herein, and / or at least some of the functions of the devices and networks described herein. In some examples, the data storage 1404 may include storage for a trained neural network model 1410 (e.g., a large language model, a dual encoder, or other model of a trained neural network). In particular in these examples, the computer-readable instructions 1406 may include instructions that, when executed by one or more processors 1403, enable the computing device 1400 to provide some or all of the functions of the trained neural network model 1410.

[0130] In some examples, the computing device 1400 may include one or more cameras 1412. The cameras 1412 may include one or more image capture devices, such as still cameras and / or video cameras, equipped to capture light and record the captured light in one or more images. That is, the cameras 1412 can produce one or more images of the captured light. The one or more images may be one or more still images and / or images used in video imaging. The cameras 1412 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or light of one or more other frequencies.

[0131] In some examples, the computing device 1400 may include one or more sensors 1414. The sensors 1414 may be configured to measure the state within the computing device 1400 and / or the state in the environment surrounding the computing device 1400, and to provide data about these states.For example, the sensor 1414 is (i) a sensor for acquiring data about the computing device 1400, including but not limited to a thermometer for measuring the temperature of the computing device 1400, a battery sensor for measuring the power of one or more batteries of the power supply system 1416, and / or other sensors for measuring the status of the computing device 1400; (ii) an identification sensor for identifying other objects and / or devices, including but not limited to a radio frequency identification (RFID) reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g., quick response (QR) code) reader, and laser tracker, the identification sensor may be configured to read identifiers such as RFID tags, barcodes, QR codes (registered trademarks), and / or other devices and / or objects configured to read and provide identification information; (iii) a tilt sensor, gyroscope, accelerometer, (iv) Sensors that measure the position and / or motion of the computing device 1400, including but not limited to Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; environmental sensors that acquire data indicating the environment of the computing device 1400, including but not limited to infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, motion sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (v) force sensors that measure one or more forces acting around the computing device 1400 (e.g., inertial force and / or G force), including but not limited to one or more sensors that measure force, torque, ground force, and friction in one or more dimensions, and / or zero moment points (ZMPs) and / or ZMP sensors that identify the location of ZMPs. Many other examples of sensors 1414 are similarly possible.

[0132] The power system 1416 may include one or more batteries 1418 and / or one or more external power interfaces 1420 for providing power to the computing device 1400. Each of the one or more batteries 1418, when electrically coupled to the computing device 1400, can function as a source of stored power for the computing device 1400. One or more batteries 1418 of the power system 1416 may be configured to be portable. Some or all of the one or more batteries 1418 may be easily removable from the computing device 1400. In other examples, some or all of the one or more batteries 1418 may be inside the computing device 1400 and therefore not easily removable from the computing device 1400. Some or all of the one or more batteries 1418 may be rechargeable. For example, rechargeable batteries may be recharged via a wired connection between the batteries and other power sources, such as by one or more power sources located outside the computing device 1400 and connected to the computing device 1400 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1418 may be non-rechargeable batteries.

[0133] One or more external power interfaces 1420 of the power system 1416 may include one or more wired power interfaces, such as USB cables and / or power cords, that enable wired power connections to one or more external power sources of the computing device 1400. One or more external power interfaces 1420 may include one or more wireless power interfaces, such as Qi wireless chargers, that enable wireless power connections to one or more external power sources via Qi wireless chargers. Once a power connection to an external power source is established using one or more external power interfaces 1420, the computing device 1400 can draw power from the external power source through the established power connection. In some examples, the power system 1416 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0134] Cloud-based servers Figure 15 shows a cloud-based server system according to an exemplary embodiment. In Figure 15, the functions of the prompt-based query generation model, the large-scale language model, and / or the document retrieval model may be distributed among the compute clusters 1509a, 1509b, and 1509c. Compute cluster 1509a may include one or more compute devices 1500a, a cluster storage array 1510a, and a cluster router 1511a connected by a local cluster network 1512a. Similarly, compute cluster 1509b may include one or more compute devices 1500b, a cluster storage array 1510b, and a cluster router 1511b connected by a local cluster network 1512b. Similarly, compute cluster 1509c may include one or more compute devices 1500c, a cluster storage array 1510c, and a cluster router 1511c connected by a local cluster network 1512c.

[0135] In some embodiments, computing clusters 1509a, 1509b, and 1509c may be a single computing device located in a single computing center. In other embodiments, computing clusters 1509a, 1509b, and 1509c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers at diverse geographical locations. For example, Figure 15 shows computing clusters 1509a, 1509b, and 1509c, each located in a different physical location.

[0136] In some embodiments, data and services in computing clusters 1509a, 1509b, and 1509c may be stored on a non-transient, tangible computer-readable medium (or computer-readable storage medium) and encoded as computer-readable information accessible by other computing devices. In some embodiments, computing clusters 1509a, 1509b, and 1509c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located in one or more diverse geographical locations.

[0137] In some embodiments, each of the computing clusters 1509a, 1509b, and 1509c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on the one or more computing tasks assigned to each computing cluster.

[0138] In computing cluster 1509a, for example, computing device 1500a may be configured to perform various computing tasks such as prompt-based query generation, large-scale language modeling, and / or document retrieval modeling. In one embodiment, various functions of the neural network and / or computing devices may be distributed among one or more of computing devices 1500a, 1500b, and 1500c. Computing devices 1500b and 1500c in each computing cluster 1509b and 1509c may be configured similarly to computing device 1500a in computing cluster 1509a. On the other hand, in some embodiments, computing devices 1500a, 1500b, and 1500c may be configured to perform different functions.

[0139] In some embodiments, computing tasks and stored data associated with various computing tasks of prompt-based query generation, large-scale language models, and / or document retrieval models may be distributed across computing devices 1500a, 1500b, and 1500c, at least in part, based on the processing requirements of neural networks and / or computing devices, the processing power of computing devices 1500a, 1500b, and 1500c, the latency of network links between computing devices within each computing cluster and between the computing clusters themselves, and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design objectives of the overall system architecture.

[0140] The cluster storage arrays 1510a, 1510b, and 1510c of computing clusters 1509a, 1509b, and 1509c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers may also be configured, either alone or in combination with their respective computing devices, to manage backups or redundant copies of data stored in the cluster storage arrays, protecting against disk drive or other cluster storage array failures and / or network failures that would prevent one or more computing devices from accessing one or more cluster storage arrays.

[0141] Just as the functionality of various computing tasks such as prompt-based query generation, large-scale language models, and / or document retrieval models can be distributed across computing devices 1500a, 1500b, and 1500c in computing clusters 1509a, 1509b, and 1509c, various active and / or backup portions of these components can be distributed across cluster storage arrays 1510a, 1510b, and 1510c. For example, some cluster storage arrays may be configured to store the first layer of a neural network and / or some portion of the data on the computing devices, while other cluster storage arrays may store the second layer of a neural network and / or other portions of the data on the computing devices. Also, for example, some cluster storage arrays may be configured to store the data for the encoder of a neural network, while other cluster storage arrays may store the data for the decoder of a neural network. Additionally, some cluster storage arrays may be configured to store backup versions of the data stored in other cluster storage arrays.

[0142] Cluster routers 1511a, 1511b, and 1511c within computing clusters 1509a, 1509b, and 1509c may include network equipment configured to provide internal and external communications to the computing clusters. For example, cluster router 1511a within computing cluster 1509a may include one or more internet switching and routing devices configured to provide (i) local area network communications between computing device 1500a and cluster storage array 1510a via local cluster network 1512a, and (ii) wide area network communications between computing cluster 1509a and computing clusters 1509b and 1509c via wide area network link 1513a to network 1306. Cluster routers 1511b and 1511c may include network equipment similar to that of cluster router 1511a, and cluster routers 1511b and 1511c may perform similar operations for computing clusters 1509b and 1509c as cluster router 1511a does for computing cluster 1509a. c It can perform networking functions for that purpose.

[0143] In some embodiments, the configuration of cluster routers 1511a, 1511b, and 1511c may be based at least in part on the data communication requirements of computing devices and cluster storage arrays, the data communication capabilities of network equipment within cluster routers 1511a, 1511b, and 1511c, the latency and throughput of local cluster networks 1512a, 1512b, and 1512c, the latency, throughput, and cost of wide area network links 1513a, 1513b, and 1513c, and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria of the mitigation system architecture.

[0144] Exemplary method of operation Figure 16 is a flowchart of Method 1600 according to an exemplary embodiment. Method 1600 may be performed by a computing device such as computing device 1400. Method 1600 for prompt-based query generation can be started in block 1610, and the method includes the computing device receiving at least two prompts related to a search task to be performed against a corpus of documents related to the task.

[0145] In block 1620, the method includes applying a large language model to generate a synthetic training dataset containing multiple query document pairs based on at least two prompt and document corpora, wherein each query document pair contains a document from the synthetically generated query and document corpus.

[0146] In block 1630, the method includes obtaining input queries relevant to a search task from multiple query document pairs from a synthetic training dataset, and training a document search model to predict output documents to be retrieved from a corpus of documents.

[0147] In block 1640, the method includes providing a document retrieval model trained by a computing device.

[0148] Some embodiments include determining a search score for each query document pair by a document search model, wherein the search score indicates the relevance of the documents in the query document pair to the query.

[0149] In some embodiments, the document retrieval model may be a dual encoder model, which includes a query encoder for encoding queries and a document encoder for encoding documents.

[0150] In some embodiments, the dual encoder model may also be a joint embedding model, where the query embedding of a given query is within a threshold distance of the document embedding of a given document, when the given document is highly relevant to the given query.

[0151] In some embodiments, the query document pairs may contain noisy data. Such embodiments include filtering the query document pairs to remove the noisy data. Such embodiments may include fine-tuning the document retrieval model based on the filtered query document pairs. In some embodiments, fine-tuning the document retrieval model may be based on a standard softmax loss with random negatives in the batch.

[0152] In some embodiments, filtering includes one or more of length filtering, prompt filtering, or round-trip filtering.

[0153] In some embodiments, the large-scale language model may be a fine-tuned language network (FLAN).

[0154] In some embodiments, for each document in the corpus, queries may be sampled based on a distribution that includes an adjustable temperature hyperparameter. Such embodiments include maintaining the temperature hyperparameter below a temperature threshold in order to generate a diverse array of queries.

[0155] In some embodiments, the search task includes one or more of the following: searching from a question to a document, searching from a question to a question, searching from an argument to a document, searching for documents that support an argument, or searching for documents that support a counterargument.

[0156] Some embodiments include pre-training a document retrieval model on unsupervised general domain data.

[0157] In some embodiments, the retrieval tasks and document corpora may be maintained behind the organization's firewall. Such embodiments include providing the organization with large-scale language models and document retrieval models. Receiving, applying, and training may be performed behind the organization's firewall.

[0158] In some embodiments, at least two prompts are fewer than eight prompts.

[0159] Figure 17 is a flowchart of Method 1700 according to an exemplary embodiment. Method 1700, which applies a trained document retrieval model, may be executed by a computing device such as computing device 1400. Method 1700 can begin in block 1710, and the method includes the computing device receiving an input query related to a retrieval task to be executed against a corpus of documents related to the task.

[0160] In block 1720, the method includes predicting documents from a corpus of documents using a trained document retrieval model, wherein the documents are highly relevant to the input query, the document retrieval model is trained on multiple query document pairs from a synthetic training dataset, the synthetic training dataset is generated by a large language model based on at least two prompts related to the retrieval task.

[0161] In block 1730, the method includes providing a predicted document in response to an input query by a computing device.

[0162] In some embodiments, the document retrieval model may be a dual encoder model, which includes a query encoder for encoding queries and a document encoder for encoding documents.

[0163] In some embodiments, the large-scale language model may be a fine-tuned language network (FLAN).

[0164] In some embodiments, the search task includes one or more of the following: searching from a question to a document, searching from a question to a question, searching from an argument to a document, searching for documents that support an argument, or searching for documents that support a counterargument.

[0165] In some embodiments, the input query may be voice input, and the search task may be performed by an intelligent voice assistant implemented on the computer.

[0166] As described herein, PROMPTAGATOR is an approach to fusion shot retrieval. This allows for the training of task-specific retrievers and rerankers with only a small number of annotated examples. Fusion shot examples amplified by prompt-based LLM query generation simplify the complexity of training neural retrievers for new tasks, leading to significant performance improvements. PROMPTAGATOR can be seen as distillation of LLMs into standard-sized dual encoders via prompt-based query generation. While the distillation process is computationally expensive, it drastically reduces the cost of inference.

[0167] This disclosure should not be limited to the specific embodiments described herein, which are intended to illustrate a variety of forms. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope thereof. In addition to those enumerated herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the above description. Such modifications and variations shall be included in the appended claims.

[0168] The detailed description above, with reference to the accompanying drawings, illustrates various features and functions of the disclosed systems, devices, and methods. In the drawings, similar symbols typically identify similar components unless otherwise indicated by context. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to limit the scope. Other embodiments may be utilized and other modifications may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily apparent that the aspects of this disclosure can be designed in arrangements, substitutions, combinations, separations, and a wide variety of different configurations, as generally described herein and shown in the drawings, all of which are expressly intended herein.

[0169] With respect to any or all of the ladder diagrams, scenarios, and flowcharts in the figures, and as discussed herein, each step, block, and / or communication may represent the processing and / or transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be performed out of the illustrated or discussed order, including substantially simultaneous or reversed order, depending on the functions involved. Furthermore, any of the ladder diagrams, scenarios, and flowcharts discussed herein may use more or fewer blocks and / or functions, and these ladder diagrams, scenarios, and flowcharts may be combined with each other, partially or entirely.

[0170] A block representing the processing of information may correspond to a circuit that can be configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, a block representing the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions that can be executed by a processor to perform a specific logical function or logical operation in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk, hard drive, or other storage medium.

[0171] Computer-readable media may also include non-temporary computer-readable media such as register memory, processor cache, and non-temporary computer-readable media that store data for short periods, such as random access memory (RAM). Computer-readable media may also include non-temporary computer-readable media that store program code and / or data for longer periods, such as secondary or persistent long-term storage devices such as read-only memory (ROM), optical or magnetic disks, and compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage systems. Computer-readable media can be thought of, for example, as computer-readable storage media or tangible storage devices.

[0172] Furthermore, a block representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules within the same physical device. However, other information transmissions may occur between software and / or hardware modules in different physical devices.

[0173] Various aspects and embodiments are disclosed herein, but other aspects and embodiments will be obvious to those skilled in the art. The various aspects and embodiments disclosed herein are provided for illustrative purposes only and are not intended to limit, and the true scope is indicated by the following claims.

Claims

1. A computer-implemented method for prompt-based query generation, The computing device receives at least two prompts related to a search task performed on a corpus of documents related to the task, Applying a large-scale language model to generate a synthetic training dataset containing multiple query document pairs based on at least two prompts and a corpus of documents, wherein the multiple query document pairs contain noisy data, and each query document pair contains a synthetically generated query and documents from the corpus of documents. In order to remove the noisy data, the following steps are taken: filtering the multiple query document pairs, The document retrieval model is trained to obtain input queries related to the retrieval task from the multiple query document pairs from the synthetic training dataset, and to output a retrieval score indicating the relevance between the input queries and documents in the corpus of documents, which is used to predict output documents to be retrieved from the corpus of documents. The computing device provides the trained document search model, A method of implementation in a computer, including

2. A computer-implemented method according to claim 1, further comprising determining a search score for each query document pair using the document search model, wherein the search score indicates the relevance of the documents in the query document pair to the query.

3. The method implemented in a computer according to claim 1, wherein the document retrieval model is a dual encoder model including a query encoder for encoding queries and a document encoder for encoding documents.

4. A method implemented in a computer according to claim 3, wherein the dual encoder model is a joint embedding model, and when a given document has a high relevance to a given query, the query embedding of the given query is within the threshold distance range of the document embedding of the given document.

5. The method implemented on a computer according to claim 1, further comprising fine-tuning the document retrieval model based on the filtered query document pairs.

6. The method for fine-tuning the document search model, as described in claim 5, is based on a standard softmax loss using random negatives in a batch, and is implemented in a computer.

7. The filtering method implemented in a computer according to claim 1, wherein the filtering includes one or more of length filtering, prompt filtering, or round-trip filtering.

8. The method for implementing the large-scale language model in a computer according to claim 1, wherein the large-scale language model is a fine-tuned language network (FLAN).

9. For each document in the corpus, queries are sampled based on a distribution that includes adjustable temperature hyperparameters, and the method implemented on the computer is: A computer-implemented method according to claim 1, further comprising maintaining the temperature hyperparameter below a temperature threshold in order to generate a diverse array of queries.

10. The computer-implemented method according to claim 1, wherein the search task includes one or more of the following: searching from a question to a document, searching from a question to a question, searching from an argument to a document, searching for documents supporting an argument, or searching for documents supporting a counterargument.

11. The computer-implemented method according to claim 1, further comprising pre-training the document retrieval model with unsupervised general domain data.

12. The search task and the corpus of documents are maintained behind the organization's firewall and implemented on the computer in the following manner: Further includes providing the aforementioned large-scale language model and the aforementioned document retrieval model to the aforementioned organization, The method implemented on a computer according to claim 1, wherein the receiving, applying, and training are performed inside the firewall of the organization.

13. The method implemented in a computer according to claim 1, wherein the at least two prompts are fewer than eight prompts.

14. A computer-implemented method for applying a trained document search model, The computing device receives input queries related to a search task, which are then executed against a corpus of documents related to the task. Predicting output documents retrieved from the document corpus based on a search score output by the trained document retrieval model, which indicates the relevance between the input query and documents in the document corpus, wherein the output documents are highly relevant to the input query, the document retrieval model is trained on multiple query document pairs from a synthetic training dataset, each query document pair from the synthetic training dataset includes a document from the document corpus and a query generated by a large-scale language model based on at least two prompts related to the search task and the document corpus, and the multiple query document pairs include noisy data, which are filtered to remove the noisy data. The computing device provides the predicted output document in response to the input query, A method of implementation in a computer, including

15. The method implemented in a computer according to claim 14, wherein the document retrieval model is a dual encoder model including a query encoder for encoding queries and a document encoder for encoding documents.

16. The method implemented in a computer according to claim 14, wherein the large-scale language model is a fine-tuned language network (FLAN).

17. The computer-implemented method according to claim 14, wherein the search task includes one or more of the following: searching from a question to a document, searching from a question to a question, searching from an argument to a document, searching for documents supporting an argument, or searching for documents supporting a counterargument.

18. The computer-implemented method according to claim 14, wherein the input query is voice input, and the search task is performed by an intelligent voice assistant implemented on the computer.

19. The computing device determines the request to respond to the input query, The process involves transmitting the request from the computing device to a second computing device, wherein the second computing device transmits a trained version of the document retrieval model. The computing device further includes, after sending the request, receiving the predicted output document from the second computing device, A method implemented in a computer according to claim 14, wherein providing the predicted output document includes providing the predicted output document upon receipt from the second computing device.

20. A computing device, One or more processors, A computing device comprising: data storage, which stores computer-executable instructions that, when executed by one or more processors, cause the computing device to perform a function including a method implemented in a computer according to any one of claims 1 to 13.

21. A computing device, One or more processors, A computing device comprising: data storage, which stores computer-executable instructions that, when executed by one or more processors, cause the computing device to perform a function including a method implemented in a computer according to any one of claims 14 to 18.

22. The computing device according to claim 20, wherein the computing device is a mobile device.

23. The computing device according to claim 21, wherein the computing device is a mobile device.

24. A computer program that, when executed by a computer, includes instructions causing the computer to perform steps according to any one of claims 1 to 13.

25. A computer program that, when executed by a computer, includes instructions causing the computer to perform steps according to any one of claims 14 to 18.

26. One or more computer-readable storage media storing computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform a function including a method implemented in a computer according to any one of claims 1 to 13.

27. ​​One or more computer-readable storage media storing computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform a function including a method implemented in a computer according to any one of claims 14 to 18.

28. A computing system, A computing system comprising means for carrying out the computer-implemented method described in any one of claims 1 to 13.

29. A computing system, A computing system comprising means for carrying out the computer-implemented method described in any one of claims 14 to 19.

Images