Systems and methods for prompt-based query generation for diverse searches

Abstract

An exemplary method for prompt-based query generation is provided. The method includes receiving, by a computing device, at least two prompts related to a search task to be performed against a corpus of documents relevant to the task. The method includes applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a synthetically generated query and a document from the corpus of documents. The method includes training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to take an input query related to the search task and predict output documents to be retrieved from the corpus of documents. The method includes providing, by the computing device, the trained document retrieval model.

Description

[Technical Field]

[0001] CROSS-REFERENCE / INCORPORATION BY REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 376,508, filed September 21, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Natural language processing tasks such as question answering and assertion verification involve retrieving knowledge from large collections of documents containing millions or even billions of candidates. Training machine learning models to perform such tasks requires large datasets of annotated query-document pairs for each task. Summary of the Invention

[0003] There are many diverse and unique search tasks targeting different scenarios. For example, different question-answering (QA) tasks often have distinct query distributions. Also, for example, search systems are typically trained on question-answering data, but search tasks are not limited to question-answering tasks. For example, tasks may include searching for entities mentioned in a query or finding evidence supporting or contradicting a given statement. Unlike other search systems, neural search systems are typically trained on question-answering data. Because training a neural search system for each task requires a significant number of annotated query-document pairs, neural search systems may not perform well for many tasks due to lack of data. Also, for example, dual-encoder models generally rely on more data than their cross-attention counterparts, and an in-batch softmax loss function typically requires tens of thousands of labeled examples to be effective. Therefore, there are technical challenges in training neural search systems for various tasks with few labeled examples.

[0004] As described herein, prompt-based query generation, filtering, and retriever training, also referred to as the PROMPTAGATOR system or PROMPTAGATOR, can harness the power of large language models (LLMs) with the efficiency of standard dual encoders. PROMPTAGATOR can create task-specific dual encoders by generating large amounts of task-specific synthetic data. PROMPTAGATOR can achieve significant improvements in retrieval (e.g., without reranking) by using as few as 2-8 examples in prompts. PROMPTAGATOR also significantly reduces the need for highly engineered search architectures.

[0005] In one aspect, a computer-implemented method for prompt-based query generation is provided. The method includes receiving, by a computing device, at least two prompts related to a search task to be performed against a corpus of documents related to the task. The method also includes applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a synthetically generated query and a document from the corpus of documents. The method further includes training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to take an input query related to the search task and predict output documents to be retrieved from the corpus of documents. The method further includes providing, by the computing device, the trained document retrieval model.

[0006] In a second aspect, a computing device for prompt-based query generation is provided. The computing device includes one or more processors and a data storage. The data storage has stored therein computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform functions. The functions include receiving, by the computing device, at least two prompts related to a search task to be performed on a corpus of documents related to the task; applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a document from the synthetically generated query and document corpus; training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to predict output documents retrieved from the corpus of documents, taking an input query related to the search task; and providing, by the computing device, the trained document retrieval model.

[0007] In a third aspect, a computer program for prompt-based query generation is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions including receiving, by a computing device, at least two prompts related to a search task to be performed on a corpus of documents related to the task; applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a document from the synthetically generated query and document corpus; training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to predict output documents retrieved from the corpus of documents, taking an input query related to the search task; and providing, by the computing device, the trained document retrieval model.

[0008] In a fourth aspect, an article of manufacture for prompt-based query generation is provided. The article of manufacture includes one or more computer-readable media having stored thereon computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include receiving, by the computing device, at least two prompts related to a search task to be performed on a corpus of documents related to the task; applying, based on the at least two prompts and the corpus of documents, a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs, each query-document pair including a document from the synthetically generated query and document corpus; training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to predict output documents retrieved from the corpus of documents, taking an input query related to the search task; and providing, by the computing device, the trained document retrieval model.

[0009] In a fifth aspect, a system for prompt-based query generation is provided, including: means for receiving, by a computing device, at least two prompts related to a search task to be performed against a corpus of documents related to the task; means for applying a large-scale language model to generate a synthetic training dataset including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a document from the synthetically generated query and document corpus; means for training a document retrieval model with the plurality of query-document pairs from the synthetic training dataset to predict output documents to be retrieved from the corpus of documents, taking an input query related to the search task; and means for providing, by the computing device, the trained document retrieval model.

[0010] In a sixth aspect, a computer-implemented method for applying a trained document retrieval model is provided. The method includes receiving, by a computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task. The method also includes predicting, by the trained document retrieval model, documents from the corpus of documents that have high relevance to the input query, the documents having high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task. The method additionally includes providing, by the computing device, the predicted documents in response to the input query.

[0011] In a seventh aspect, a computing device for applying a trained document retrieval model is provided. The computing device includes one or more processors and a data storage. The data storage has stored therein computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform functions. The functions include receiving, by the computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task; predicting, by the trained document retrieval model, documents from the corpus of documents that have high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task; and providing, by the computing device, the predicted documents in response to the input query.

[0012] In an eighth aspect, a computer program for applying a trained document retrieval model is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions including receiving, by a computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task; predicting, by the trained document retrieval model, documents from the corpus of documents that have high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task; and providing, by the computing device, the predicted documents in response to the input query.

[0013] In a ninth aspect, an article of manufacture for applying a trained document retrieval model is provided. The article of manufacture includes one or more computer-readable media having stored thereon computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include receiving, by the computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task; predicting, by the trained document retrieval model, documents from the corpus of documents that have high relevance to the input query, the document retrieval model being trained with multiple query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task; and providing, by the computing device, the predicted documents in response to the input query.

[0014] In a tenth aspect, a system for applying a trained document retrieval model is provided, the system including: means for receiving, by a computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task; means for predicting, by the trained document retrieval model, documents from the corpus of documents, the documents having high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task; and means for providing, by the computing device, the predicted documents in response to the input query.

[0015] The above summary is illustrative only and is not intended to be in any way limiting. In addition to the exemplary aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the figures and the following detailed description, and accompanying drawings. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an overview of aspects of training a document retrieval model, according to an exemplary embodiment. [Figure 2] 1 illustrates an exemplary query generation model and an exemplary document retrieval model, according to an exemplary embodiment. [Figure 3] 10 illustrates exemplary prompt templates for various data sets, according to an exemplary embodiment. [Figure 4] 10 illustrates a comparison example of different retriever frameworks, according to an example embodiment. [Figure 5] 10 illustrates a comparative example of a retriever framework with different retriever frameworks and rerankers, according to an exemplary embodiment. [Figure 6A] 10 is a bar graph illustrating the impact of round-trip filtering, according to an example embodiment. [Figure 6B] 10 is an illustration of a graph showing examples of human annotations, in accordance with an illustrative embodiment; [Figure 6C] 10 illustrates exemplary ablation results for a query generation model, according to an exemplary embodiment. [Figure 7] 1 illustrates the impact of different fine-tuned language net (FLAN) versions, in accordance with an example embodiment. [Figure 8] 10 illustrates the impact of selected examples on model output, in accordance with an illustrative embodiment; [Figure 9A] 1 illustrates exemplary top first word distributions for queries generated from different models in the Argumentation Analysis (ArguAna) dataset, according to an exemplary embodiment. [Figure 9B] 10 illustrates exemplary top first word distributions for queries generated from different models in the ArguAna dataset, according to an exemplary embodiment. [Figure 9C]10 illustrates exemplary top first word distributions for queries generated from different models in the ArguAna dataset, according to an exemplary embodiment. [Figure 9D] 10 illustrates exemplary top first word distributions for queries generated from different models in the ArguAna dataset, according to an exemplary embodiment. [Figure 10A] 10 illustrates exemplary few-shot and zero-shot generated queries randomly sampled from various datasets, according to an example embodiment. [Figure 10B] 10 illustrates exemplary few-shot and zero-shot generated queries randomly sampled from various datasets, according to an example embodiment. [Figure 10C] 10 illustrates exemplary few-shot and zero-shot generated queries randomly sampled from various datasets, according to an example embodiment. [Figure 11] 1 illustrates a table with average query lengths for various datasets, according to an example embodiment. [Figure 12] FIG. 1 illustrates the training and inference stages of a machine learning model, according to an example embodiment. [Figure 13] 1 illustrates a distributed computing architecture in accordance with an exemplary embodiment. [Figure 14] FIG. 1 is a block diagram of a computing device in accordance with an exemplary embodiment. [Figure 15] 1 illustrates a network of computing clusters arranged as a cloud-based server system, according to an example embodiment. [Figure 16] 1 is a flowchart of a method according to an example embodiment. [Figure 17] 10 is a flowchart of another method according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] In one aspect, this application relates to solving the problem of data scarcity while maintaining the efficiency of small-scale dual encoders by leveraging the power of large-scale language models (LLMs). To address the data scarcity problem, PROMPTAGATOR combines prompts and large-scale language models as a query generator without fine-tuning. The use of large-scale language models allows PROMPTAGATOR to generate appropriate queries from natural questions without training data. PROMPTAGATOR can use a few prompts that enable short-shot search, using only 2-8 examples for each task, and inputting these new examples within the prompts provides significant improvement. Instead of directly training the dual encoder using a few examples, PROMPTAGATOR can amplify the power of these few examples by creating task-specific prompts. A large amount of training data including query and document pairs can be generated. After query generation, task-specific noisy paired data can be used to train a task-specific dual encoder. A second iteration for training can be performed by filtering the noisy paired data to generate a clean dataset and training the task-specific dual encoder using the clean dataset.

[0018] Neural retrieval models, such as dual encoders, are capable of searching large document collections containing millions to billions of passages. However, Benchmarking Information Retrieval (BEIR) has shown that neural retrievers can struggle to perform well on diverse retrieval tasks where dedicated training data is lacking. To address this issue, some approaches focus on transferring knowledge from resource-rich question-answering (QA) datasets and propose architectures with favorable inductive bias, such as models that allow fine-grained token-level interactions (e.g., Contextualized Late Interaction over BERT (ColBERT) and Sparse Lexical AnD Expansion (SPLADE)), associated with higher inference costs.

[0019] Data augmentation through synthetic query generation typically involves question generators trained from high-resource QA datasets, which often fail to generalize well to new search tasks. Some approaches leverage synthetic question generation on target corpora to mitigate the data scarcity problem. While effective in improving performance on QA search tasks, one limitation of such approaches is that, because the question generation model is trained on a question-answering dataset, the distribution of generated questions may not match the true query distribution of the target task.

[0020] The few-shot LLM query generator described herein can produce good queries without fine-tuning the model. In fact, synthetically generated data may be powerful enough to reduce and / or eliminate the use of annotated query-document pairs from traditional high-resource datasets such as Natural Questions.

[0021] To ensure the quality of the generated data, we describe a filtering strategy that uses only generated data. The filtering strategy can eliminate ambiguous, general, and low-quality questions, significantly improving search performance. Note that although an LLM is used to generate the training data, the search model at inference time can be any standard-sized retriever.

[0022] Existing approaches using LLMs do not apply task-specific short-shot adaptation and often involve high inference costs. For example, some approaches involve using GPT-3 with dual encoders, but their embedding dimension is 12,000, which can make the search index footprint and inference cost prohibitive for many applications. Other existing approaches involve prompting LLMs for question generation, but this approach does not use task-specific short-shot prompts for rapid task adaptation. This approach also primarily focuses on models that rerank top results from existing retrievers rather than directly adapting the underlying retrievers, allowing for efficient searching of millions or billions of documents.

[0023] The approach described herein involves considering differences across search tasks (e.g., search intent and query distribution) and proposes a few-shot search evaluation on a dataset (e.g., the BEIR dataset). As described, PROMPTAGATOR is a simplified model for few-shot search that prompts LLMs to generate synthetic, task-specific training data. Thus, neural retrievers and rerankers can be trained based on only a small number of supervised examples. As illustrated, PROMPTAGATOR with 2-8 examples can produce retrievers that significantly outperform existing models trained on MS MARCO or NQ, which have over 500,000 human-annotated examples and utilize more costly architectures. For example, PROMPTAGATOR can outperform ColBERT v2 and SPLADE v2 on several search tasks that can be tested, while reranking can boost results by an additional 5 points on standard search evaluation metrics.

[0024] System Overview FIG. 1 illustrates an overview 100 of aspects of training a document retrieval model, according to an example embodiment. Some embodiments include receiving at least two prompts (e.g., a first prompt 110 and a second prompt 115) associated with a search task to be performed on a corpus of documents 105 relevant to the task. Some embodiments include applying an LLM 120 to generate (e.g., generate 125) a synthetic training dataset 130 including a plurality of query-document pairs 135 based on the at least two prompts (e.g., the first prompt 110 and the second prompt 115) and the corpus of documents 105. Each of the plurality of query-document pairs 135 may include a synthetically generated query and a document from the corpus of documents 105. Some embodiments include training (e.g., training 140) a document retrieval model 145 with the plurality of query-document pairs 135 from the synthetic training dataset 130 to take an input query associated with the search task and predict output documents to be retrieved from the corpus of documents 105. Additional details regarding each of these steps are provided below.

[0025] task This specification describes a short-shot search approach for a variety of search tasks, each associated with a short description and a small number of annotated examples to clearly indicate the search intent. As described herein, for each new search task, the short-shot examples may be input into a large-scale language model (e.g., LLM 120) such as a fine-tuned language net (FLAN). The FLAN is an LLM that has not been trained on any document retrieval or query generation tasks from documents. The FLAN may be prompted to perform query generation (e.g., generation 125) from documents. Generally, the short-shot examples ensure that the unique search intent of a particular task is captured. Using this query generator large-scale language model (LLM) 120, a large number of relevant queries for any document may be synthetically generated, providing abundant data (e.g., multiple query-document pairs 135 from a synthetic training dataset 130) for training a retriever (e.g., document retrieval model 145), including a highly efficient dual-encoder model.

[0026] Generally speaking, different search tasks may have different search intents, or in other words, different definitions of "relevance." For example, consider the task of retrieving documents from Wikipedia. The first task may be to search for entities mentioned in the query, or the second task may be to find evidence supporting or refuting a given statement. Determining which documents are relevant to a query may vary significantly from task to task, even if the tasks share the same domain. Furthermore, different tasks may have distinct distributions of queries, even if their search intents may be similar. For example, some queries may be long, structured questions, while others may be short, financial questions. The search domain may vary from general web pages to Wikipedia articles, Quora questions, or papers on specific diseases.

[0027] For illustrative purposes, we describe a few-shot search setting for the BEIR benchmark. Given a large corpus (e.g., corpus of documents 105), a search model (e.g., document retrieval model 145) can be configured to find documents that are relevant to a provided query q according to a predefined notion of relevance. Formally, the search task can be formulated as follows: T={D,Q,I} (Equation 1)

[0028] where D={d1,d2,…,d n} is a large corpus of documents for search (e.g., document corpus 105), Q is the query distribution, and I is the search intent underlying the task. Depending on the task, D may be any collection of documents, such as the Web or Wikipedia. The search Q also varies depending on the task, e.g., a short keyword search query, a question, a discussion, etc. If I(q,d)=1, this indicates that the search intent of q is satisfied by document d. For example, in a question answering task, I QA (q,d)=1 indicates that document d is responsive to query q. For the same (q,d) pair, relevance may be either 1 or 0 depending on the search intent. For example, some argument search tasks may seek to find supporting arguments, while other tasks may seek to find counterarguments.

[0029] In general, the target search corpus D T While a corpus of documents for a particular task (e.g., 105) may be given, the amount of annotated query-document pairs for a new task may be limited. Existing approaches train retrievers on a new corpus D T Although the focus is on adapting to the query Q T or intention I T As explained herein, search intent can be expressed with short descriptions and very few examples.

[0030] Future Shot Settings Humans are generally able to understand a search task by reading a short prompt and examining a small number of examples. Therefore, it may be natural to determine whether a small number of examples (e.g., eight or fewer) may be sufficient for a machine learning model to learn a task-specific retriever. In this regard, the Few Shots Search evaluation may build on the BEIR heterogeneous search benchmark.

[0031] BEIR includes 18 information retrieval datasets across nine domains, including biomedical, finance, news, Twitter, Wikipedia, StackExchange, Quora, science, and miscellaneous. These datasets cover a diverse range of search intents, including Q&A search (question to document), duplicate question detection (question to question), and fact-checking (claim to document). The original BEIR evaluation used a zero-shot setup, meaning that either no queries were used for training, or query-document pairs from the evaluation datasets could be used.

[0032] In some aspects, as described herein, BEIR may be modified to a few-shot setting by randomly taking a few (e.g., 2-8) examples of relevant query documents in the domain as task-specific supervision. Generally, it is possible to obtain this number of examples. Examples may be sampled from a development set, if one is available. For BEIR tasks that only have a test set, samples from the test data may be used.

[0033] PROMPTAGATOR model A PROMPTAGATOR may be configured to convert a small number of examples into a larger number of examples by prompting an LLM (e.g., LLM 120) to generate more data (e.g., synthetic training dataset 130) rather than directly training a retriever (document retrieval model 145) using examples. In some embodiments, a PROMPTAGATOR may include three components: prompt-based query generation, consistency filtering, and retriever training. During prompt-based query generation, D T The task-specific prompts may be combined with a large-scale language model to generate relevant queries for all documents in . A filtering step may then clean the generated data based on round-trip consistency. In some embodiments, a retriever trained only on synthetic data (e.g., multiple query-document pairs 135 from the synthetic training dataset 130) may be used to filter the synthetic data. Finally, a retriever (e.g., a dual encoder) and a cross-attention reranker may be trained based on the filtered data.

[0034] FIG. 2 is a diagram 200 illustrating an exemplary query generation model and an exemplary document retrieval model, according to an exemplary embodiment. As described herein, PROMPTAGATOR combines prompts with an instruction-tuned large-scale language model (e.g., LLM 205) to form query generation with far fewer constraints than other models. It may rely on a very small number of examples (e.g., eight or fewer) to train a retriever (e.g., dual encoder 210) and may outperform custom-designed models. In some embodiments, the cross-attention reranker 215 may be a list-wise model based on T5. For example, it may receive a list of documents from the synthetic data 220 given a query as input based on query-document pairing. Each query-document pair may be represented as "Query:{q} Document:{d}", and the pair may be input to the encoder of the cross-attention reranker 215 (e.g., T5 model). In some embodiments, a projection layer may be applied to output encoding of the first token, and the output may be used as a ranking score. For example, the model may generate a list of documents based on query-document pairings from the synthetic data 220 given a query as input. + ) pairs and sampled negatives (q,d - ) pairs (e.g., 31 sampled negative pairs) using softmax cross-entropy loss on a ranking list composed of these pairs. Unlike monoT5, which is a point-wise reranker trained to generate relevance labels using an encoder-decoder model, the cross-attention reranker 215 may be a list-wise reranker and can directly optimize ranking performance.

[0035] Prompt-Based Query Generation In some embodiments, task-specific short-shot examples 225 may be input to a large-scale language model (e.g., LLM 205). The LLM 205 may be prompted to perform query generation 230 from the documents. More precisely, the k short-shot examples 225 may be expressed as {(qi ,d i )} k Then, each example is a query (q i ~Q T ) and target task T(I T (q i ,d i )=1) documents related to that query (d i ∈D T )

[0036] Following FLAN, command prompts may be applied to the LLM205 using the following string prefixes: e doc (d i )|e query (q1)|...|e doc (d k )|e query (q k )|e doc (d) (Formula 2)

[0037] where '|' indicates a separator token, and e doc (d) and e query (q) are the task-specific document and the query description, respectively, and d denotes the new document presented during inference. In some embodiments, the LLM 205 doc (d) = "Argument: {d}" and e query = You may be prompted to generate a counterargument based on "Counter Argument:{q}".

[0038] Figure 3 shows example prompt templates 300 for various datasets, according to an example embodiment. A complete set of example descriptions used in the prompts is shown in Figure 3. The datasets include Argumentation Analysis (ArguAna), Financial Question Answering (FiQA), natural multi-hop Question Answering (HotpotQA), entity search over the database-pedia (DBpedia) knowledge base (DBpedia-Entity), NutritionFacts Corpus (NFCorpus), Touch▲e▼2020, Text REtrieval Conference data for Covid (TREC-Covid), Scientific Claims (SciFact), Scientific Documentation (SciDocs), and Fact Extraction and VERification (FEVER).

[0039] Referring again to Figure 2, the LLM205:

number

number

number

[0040] In some embodiments, such a prompt may include: T235 to amplify information from a small number of examples (e.g., task-specific short-shot examples 225) into a large-scale synthetic dataset (e.g., synthetic data 220).

number

[0041] In some embodiments, the LLM 205 may be a FLAN model and the query generator may be p FLAN (q|d). FLAN may be trained on a set of tasks described by instructions and generally achieves high zero-shot / few-shot performance for first-time tasks. Some embodiments include using 137 billion parameter checkpoints. During prompt engineering, a maximum of eight examples may be used, and that number may be further reduced if the input length limit of FLAN is exceeded. Also, for example, individual queries and documents in the examples may be manually truncated if they are longer than a threshold length. As another example, up to one million documents may be randomly sampled from each corpus, and eight questions per document may be generated using sampling-based decoding with a temperature parameter of 0.7.

[0042] Data generated by round-trip filtering In some embodiments, round-trip filtering 240 may be applied to improve the quality of the synthetic dataset 220. Generally speaking, a synthetic query generated from a passage d

number

number

number

number

[0043] Round-trip filtering 240 can be effective for generating synthetic questions for QA tasks. However, existing techniques typically rely on backward-filtering question-answering models. Because not all search tasks are similar to question answering, such existing approaches may not be sufficient. Instead, in the training step 250, training may be applied to an initial retriever (e.g., a small-scale dual encoder) from unfiltered synthetic data (e.g., synthetic data 220). The trained initial retriever (e.g., a trained small-scale dual encoder) can then be used to filter the synthetic data. In general, such an approach works well across the different search intents observed in BEIR. More precisely, synthetic query-document pairs

number

number

number

[0044] Round-trip filtering as latent variable modeling In some embodiments, round-trip filtering 240 may involve viewing queries as latent variables to be estimated. Each query q is a latent variable, and there is a distribution p(d|q,θ) over which documents are retrieved for that query. * Consider a hypothetical graphical model in which the observed variables follow θ * represents the parameters of a hypothetical "optimal" retriever that can select the "best" documents for a query. To generate synthetic data, the query is given a posterior distribution, p(q|d,θ * ), which may be given according to Bayes' rule as follows:

number

[0045] Here, p(q|θ * ) is a prior distribution over the query. In some aspects, p(q|θ * ) can be assumed to be an "uninformative prior" that is uniform over all q. In terms of notation, p(q|θ * ) may be denoted as p(q). The optimal retriever parameters θ * If θ is known, then the formula in Equation 3 can be calculated directly. *are generally unknown variables, and therefore the variables θ can be estimated using expectation-maximization (EM). Generally, the round-trip filtering algorithm described herein is a version of EM. EM and other latent variable learning methods may be used to impute missing data.

[0046] In EM, the variables θ can be estimated by approximately maximizing the marginal likelihood of the observed variables, as described by the following relation:

number

[0047] For example, in EM, an initial guess of p(q|d) may be made for every document d. In some embodiments, the FLAN query generator uses the initial guess p FLAN (q|d). The M-step of EM may involve a calculation described by the following relationship:

number

[0048] This is achieved by pairing queries generated by LLM (e.g., FLAN) with D T Document from the initial retriever

number

number

number

[0049] As shown by Equation 6,

number

[0050] In some embodiments,

number

number

number

[0051] Such an EM-like procedure may be repeated until convergence, but generally a single round of filtering may result in sufficient improvement in quality. In some aspects, training the final retriever model may be viewed as another M-step.

[0052] Fewshot Promptagator Retriever As described herein, synthetically generated data (e.g., synthetic data 220) may be used to generate task-specific training data for tasks that may be difficult to fine-tune in the domain due to lack of data. neural In some embodiments, the search architecture of the dual encoder 210 may be used in conjunction with a pre-training fine-tuning recipe.

[0053] For example, pre-training 245 may involve pre-training a retriever (e.g., dual encoder 210). In some embodiments, pre-training may be done on C4 using a cropping task independent of the contriever, where two random crops from the same document may be viewed as artificial positive (query, document) pairs, and training may involve cross-entropy loss on random negatives within the batch in training step 250. The dual encoder 210 then uses the random negatives within the batch to generate the positives from the prompt-based query generation.

number

[0054] Some embodiments may include a modified PROMPTAGATOR, referred to herein as PROMPTAGATOR++. Generally, this may be a reranker (e.g., cross-attention reranker 215) trained on the same synthetic data (e.g., in synthetic dataset 220) generated from prompt-based QGen. QGen may be configured to refine the retrieved candidates using the slower but more accurate cross-attention reranker 215. The cross-attention reranker 215 may be trained using a cross-entropy loss (e.g., with 31 negatives 255 sampled from the top 200 passages retrieved by the PROMPTAGATOR retriever), which approximates the inference time distribution (e.g., reranking the top 200 from the retriever).

[0055] Zero Shot Promptagarator In some embodiments, prompt-based query generation may be configured to occur in a zero-shot manner, and the zero-shot prompt 260 may be applied regardless of the target task. For example, the zero-shot prompt 260 may be "Read passage {d} and generate a query," where {d} indicates the text of the document. Zero-shot PROMPTAGATOR and Zero-shot PROMPTAGATOR++ may be constructed by training a retriever (e.g., document retrieval model 145, dual encoder 210) and a reranker (e.g., cross-attention reranker 215) on such data. These models may serve as a baseline for illustrating the benefits of adapting the zero-shot prompt to the target task.

[0056] Comparison with existing approaches FIG. 4 illustrates a comparative example 400 of different retriever frameworks, according to an exemplary embodiment. For example, the PROMPTAGATOR configuration can be compared to several other approaches. Some aspects of the PROMPTAGATOR approach are simpler than the comparison approaches. For example, the dual encoder 210 may not rely on hard negative mining, cross-attention supervised distillation, and / or token-level search. Also, a reranker, such as the cross-attention reranker 215 (e.g., with 125 million parameters), may be configured to be smaller than rerankers in other approaches. In some embodiments, as described herein, simpler and smaller architectures can be designed to achieve high performance results when trained with short-shot adapted synthetic data. Compared to the Inquisitive Parrots for Search (InPars) and Unsupervised Passage Re-ranking (UPR) approaches, the PROMPTAGATOR approach employs task-specific short-shot adaptation. In contrast, the prompts in InPars and UPR are task-independent and therefore suffer from the same limitations as previous query generation approaches. Another difference between the approaches is that existing approaches focus on re-ranking, whereas the PROMPTAGATOR approach enables few-shot learning for both re-ranking and search.

[0057] evaluation As described herein, the PROMPTAGATOR approach may be evaluated on the BEIR benchmark. For example, ablation studies and qualitative analyses may also be performed. In some embodiments, the FLAN training set may overlap with two BEIR datasets, NQ and Quora. For example, some existing approaches do not report few-shot or zero-shot results on MS MARCO because they use this dataset for fully supervised learning. Therefore, evaluation may not be based on the MS MARCO, NQ, and Quora datasets. Instead, evaluation may be performed on the normalized impairment cumulative gain for 10 items (nDCG@10), a search evaluation metric for BEIR.

[0058] For query generation in PROMPTAGATOR, questions may be sampled from FLAN at a given temperature (e.g., 0.7). In some embodiments, setting the filtering threshold K to 1 for round-trip filtering may yield better results for MS MARCO. Therefore, the filtering threshold K may be set to 1 for all BEIR datasets. In some embodiments, the dual encoder may be implemented based on the Generalizable T5 Retriever (GTR). To ensure efficiency, a T5-based encoder architecture consisting of 110 million parameters may be utilized. For the PROMPTAGATOR++ reranking model, a T5-based version 1.1 encoder checkpoint with 125 million parameters may be initialized. In some embodiments, the top 200 candidates retrieved from the PROMPTAGATOR dual encoder may be reranked during inference.

[0059] In some embodiments, hyperparameters may include a batch size of 6k. However, because some of the BEIR corpora contain only a few thousand documents, multiple related documents appear in the same batch. This can negatively interact with the in-batch softmax loss in some cases. Therefore, datasets may be divided into groups based on corpus size, such as a small group (corpus size < 50k), a medium group (corpus size 50k-500k), and a large group (corpus size > 500k). For dual-encoder training, a batch size of 128 may be used for small datasets, and a batch size of 6k may be used for others. Also, for example, 5k steps may be used for fine-tuning large datasets, and 1k steps for others. For ranking models, a batch size of 64 may be used for all datasets, and large datasets may be fine-tuned for 20k steps, and others for 5k steps.

[0060] result FIG. 5 illustrates a comparative example 500 of retriever frameworks with different retriever frameworks and rerankers, according to an exemplary embodiment. Experimental results for such models are shown. As shown, zero-shot PROMPTAGATOR can serve as a strong baseline, comparing favorably with other search baselines trained on O(100k) examples from MS MARCO. Furthermore, for example, few-shot PROMPTAGATOR can significantly improve over zero-shot PROMPTAGATOR, increasing the average nDCG@10 by more than two points. This evaluation highlights the impact of few-shot learning. For example, few-shot PROMPTAGATOR, despite having a simple training procedure and model architecture, can outperform strong baselines such as GenQ and GPL, which also use query generation to augment training data, and ColBERT v2 and SPLADE v2, which rely on a token-level interaction architecture and distillation recipe.

[0061] As shown, the reranker PROMPTAGATOR++ can boost performance by an additional 5 points over nDCG@10. This can result in a significant improvement over UPR, where the reranker uses T0 instructions, an instruction-tuned LLM similar to FLAN. It can also outperform monoT5-3B, which was previously shown to achieve state-of-the-art reranking performance in BEIR. While existing reranking approaches generally use large 3 billion parameter models for better generalization, PROMPTAGATOR++ uses a smaller (e.g., 125 million) reranker.

[0062] Comparing Fuchs PROMPTAGATOR to the baselines, significant improvements can be observed in Webis-Touch▲e▼2020 (Touch▲e▼), followed by ArguAna (arg). The goal of Webis-Touch▲e▼2020 is to retrieve documents about controversial topics, such as "Should felons who have completed their sentences be granted the right to vote?" The goal of ArguAna is to find counterarguments to the input argument, which may be several sentences long. Both tasks are generally different from traditional QA search data used by other models, which are dominated by factoid questions. However, Fuchs PROMPTAGATOR is able to successfully adapt to this task with a small number of examples.

[0063] The impact of round-trip filtering Figure 6A is a bar graph 600A illustrating the impact of round-trip filtering, according to an example embodiment. Figure 6A shows the quality difference for the Fushot Promptagarator with and without filtering. Overall, filtering can improve performance for 8 of the 11 datasets, resulting in an average improvement of 2.5 points, demonstrating the effectiveness of the filtering strategy described herein. Filtering appears to have a negative impact on model quality for NFCorpus and SciFact, which may be due to the relatively small size of these datasets with respect to the queries generated and may indicate overfitting of the retriever.

[0064] Generated queries vs. human-annotated queries FIG. 6B is a graph 600B showing human-annotated examples according to an exemplary embodiment. In FIG. 6B, the 8-shot PROMPTAGATOR is evaluated with MS MARCO and compared to a dual encoder trained with supervised data from MS MARCO. For evaluation purposes, no additional components are added to maintain a simpler comparison. MS MARCO may be chosen because there is sufficient labeled data for this task, and neither FLAN nor the models described herein are trained with MS MARCO examples. The results show that 8 examples and LLM can replace the majority of supervised examples.

[0065] Comparison with other query generation approaches FIG. 6C shows exemplary ablation results 600C for a query generation model according to an exemplary embodiment. FIG. 6C compares zero-shot PROMPTAGATOR with two other query generation approaches: GenQ, a conventional system using a T5 query generation model trained with MS MARCO, and natural questions-QGen (NQ-QGen), a T5 QGen model fine-tuned with NQ. Several advantages of zero-shot PROMPTAGATOR are demonstrated, including outperforming both baseline models by significant margins. In general, NQ-QGen uses the same filtering, dual-encoder training, batch size, and training steps as PROMPTAGATOR, thereby providing a fair comparison of query generators. Results 600C indicate that the contributing factors to PROMPTAGATOR may be better queries due to prompting LLMs, and not due to specific training recipes or specific hyperparameters.

[0066] Comparison of few shots and over zero shots Referring again to Figure 5, a comparative example 500 between different retriever frameworks and a retriever framework with a reranker, the short-shot PROMPTAGATOR appears to generally outperform the zero-shot PROMPTAGATOR, except for Climate-FEVER. The Climate-FEVER dataset annotates query-document pairs using one of three tags: "supports," "refutes," or "not enough information." However, BEIR generally considers these three annotations relevant, which may lead to undesirable results. Using query-document pairs annotated with "not enough information" in prompts may adversely affect generation quality. Using such pairs in short-shot prompts may adversely affect query generation. In some evaluations, the short-shot prompt from FEVER may be used because the two datasets share the same corpus and similar search intent. Based on such modified annotated examples, the short-shot PROMPTAGATOR appears to outperform the zero-shot PROMPTAGATOR.

[0067] FLAN version impact FLAN was trained on a set of datasets that overlaps somewhat with BEIR. Specifically, FLAN includes Natural Questions (NQ) and Quora. However, FLAN was not trained on query-document pairs from NQ or Quora. Additional ablation experiments can be designed to determine whether the inclusion of this data could bias the results for the final search evaluation. For example, based on the FLAN recipe, additional LLMs can be trained by excluding both the NQ and Quora datasets.

[0068] Figure 7 shows the impact of different FLAN versions, according to an example embodiment. The results are split into two tables 700a and 700b. The last column 705 of table 700b shows the average of the respective rows from tables 700a and 700b. For example, the average values ​​in the last column 705 48 The .5 is the average of the values ​​in the first row of tables 700a and 700b. Similarly, the average value of 47.0 in the last column 705 is the average of the values ​​in the second row of tables 700a and 700b.

[0069] FIG. 8 illustrates the impact of selected examples on model output, according to an exemplary embodiment. For example, FIG. 8 illustrates the impact and / or improvement of using eight examples to fine-tune an existing search model (e.g., GTR). While accuracy may be slightly reduced, overall performance is superior to existing retrievers. The results are split into two tables 800A and 800B. The last column 805 of table 800B shows the average of the rows in each of tables 800A and 800B. For example, the average value of 40.4 in the last column 805 is the average of the values ​​in the first row of tables 800A and 800B. Similarly, the average value of 38.7 in the last column 805 is the average of the values ​​in the second row of tables 800A and 800B.

[0070] qualitative analysis Some advantages of the Future Promptator can be illustrated based on the distribution of queries generated by different query generation methods for ArguAna. For example, for each distribution, a histogram of the first words of each query can be shown.

[0071] 9A-9D show exemplary first word distributions for queries generated from different models in the ArguAna dataset, according to an exemplary embodiment. FIG. 9A shows the distribution of first words for Gold queries. FIG. 9B shows the distribution of first words for the Future Promptator, from which queries are generated. As shown, the distribution shown in FIG. 9B is closer to the actual distribution shown in FIG. 9A. FIG. 9C shows the distribution of first words for the Future Promptator, from which queries are generated. NQ-QGen generates mostly questions, even though queries in this task are generally arguments rather than questions. FIG. 9D shows that Future FLAN can generate a wide variety of queries even when the prompt has only four examples. Additional examples are shown in FIGS. 10A-10C.

[0072] 10A-10C show exemplary few-shot and zero-shot generated queries randomly sampled from various datasets, according to an exemplary embodiment. Generally, few-shot generated queries appear close to the original query, while zero-shot queries appear almost to be questions. For example, in the ArguAna dataset, few-shot queries are generally longer and more assertion-like. Also, for example, zero-shot queries are generally short, question-like queries. For the HotpotQA dataset, even though both few-shot and zero-shot queries appear to generate question-like queries, few-shot queries appear to generate multi-hop questions, while zero-shot queries appear to generate single-hop questions.

[0073] Referring to Figure 10A, columns 10A-C1 show example paragraphs, columns 10A-C2 show queries generated by a few-shot process, columns 10A-C3 show queries generated by a zero-shot process, and columns 10A-C4 show an analysis on the ArguAna dataset showing that queries generated in few-shot are more statement-like and longer than queries generated in zero-shot.

[0074] Referring to FIG. 10B, columns 10B-C1 show example paragraphs, columns 10B-C2 show queries generated by a few-shot process, columns 10B-C3 show queries generated by a zero-shot process, and columns 10B-C4 show an analysis showing an analysis on the Touch▲e▼-2020 dataset, where few-shot generates more argumentative queries, while zero-shot generates random statements that may be grammatically incorrect.

[0075] Referring to FIG. 10C, columns 10C-C1 show example paragraphs, columns 10C-C2 show queries generated by a few-shot process, columns 10C-C3 show queries generated by a zero-shot process, and columns 10C-C4 show an analysis showing an analysis on the HotpotQA dataset, where few-shot queries can be multi-hop questions, while zero-shot queries do not generate multi-hop questions.

[0076] Direct use of the Fewshot example Referring again to Figure 8, the effect of using the few-shot examples directly can be analyzed by fine-tuning the task-specific model using the GTR dual encoder (e.g., 110 million) with the few-shot examples shown in Figure 3. As expected, it does not have a significant impact on the GTR base, with the average nDCG decreasing from 40.4 to 38.7.

[0077] Query Generation Statistics FIG. 11 shows a table 1100 with average query lengths for various datasets, according to an exemplary embodiment. In FIG. 11, the length of queries generated by different query generation systems is analyzed. NQ-QGen appears to generate shorter queries due to the fine-tuning of its query generation model on the NQ dataset, and the queries it generates appear to have similar lengths to NQ queries. The mean value 1115 for NQ-QGen is shown to be 9.6. However, Zero-Shot PROMPTAGATOR appears to have a higher variance in length compared to NQ-QGen. The mean value 1110 for Zero-Shot PROMPTAGATOR is shown to be 13.5. Also, for example, Few-Shot PROMPTAGATOR appears to have a significantly larger variance in the length of generated queries. The mean value 1105 for Few-Shot PROMPTAGATOR is shown to be 17.8.

[0078] Computational usage and environmental impact In some embodiments, a 137 billion FLAN based on LaMDA may be used. LaMDA may be pre-trained on a large corpus consisting of 1.56 trillion words, costing 451 megawatt-hours (MWh) of energy and a carbon footprint of 25.2 tCO2e. PROMPTAGATOR generates 29.23 million queries * 2 prompts = 58.46 million queries, totaling 610 million words.

[0079] Comparison with existing models Some existing neural retrievers employ a dual-encoder architecture, which encodes queries and documents separately into dense vectors and retrieves documents using maximum inner product search (MIPS). Existing approaches have mainly focused on the following aspects: developing better pre-training tasks, improving contrastive negativity, and improving generalization across different domains.

[0080] Although dual encoders enable fast retrieval, their expression may be limited by the fact that their score is merely a dot product between the query and document vectors. Cross-attention models are commonly used to re-rank retrieved candidates because cross-attention rerankers can explicitly model the interaction between query and document tokens. Distilling the cross-attention model into a dual encoder has been effective in closing the gap between dense retrievers and cross-attention models.

[0081] An alternative to bridge the gap between dense retrievers and cross-attention models is to allow some fine-grained query-document interaction in the retriever. For example, the query and document can be represented and retrieved as multiple vectors instead of a single vector. The ColBER, COIL, and SPLADE approaches use token-level interactions between queries and documents. These models cannot directly use the MIPS algorithm because they do not simply model the dot product. Therefore, these models typically have much higher inference / serving costs compared to dual encoders.

[0082] In general, prompted LLMs can be used in query generation to improve search reranking. For example, UPR uses prompted LLMs to directly rerank passages. InPars uses few-shot prompts with GPT-3 to generate synthetic data for training a T5-based reranker. While InPars has been tested on multiple search datasets, it is based on task-independent prompts built from MS MARCO and does not involve task-specific few-shot learning. InPars also focuses solely on reranking, while the PROMPTAGATOR approach addresses full-scale search. Pre-trained LLMs have significantly higher few-shot learning capabilities based on prompting strategies such as in-context learning and imperative prompts. Some approaches involve fine-tuning LLMs specifically for few-shot learning, while others do not. As described herein, the PROMPTAGATOR approach uses LLMs as a few-shot data generator.

[0083] technical improvements The techniques described herein enable few-shot retrieval, where models can achieve significant improvements using as few as two to eight examples (referred to herein as prompts) per task. Generally, very few prompts can train an LLM to automatically generate synthetic data, which can be used to train a document retrieval model. Existing models attempt to generate training data directly. As used herein, the term "prompt" can be general in the zero-shot case or task-specific in the few-shot case. The prompts are combined with a large-scale language model to create queries for the target task using a corpus of documents.

[0084] PROMPTAGATOR can achieve good performance with dual encoders without requiring a highly engineered search architecture. For example, no specialized architecture is required and existing large-scale language models can be used.

[0085] The techniques described herein can generate queries for searches on many different target tasks.

[0086] The search can be performed without a reranker.

[0087] Prompting with instruction-tuned large-scale language models can address a major limitation of other models that generally focus on question-answering datasets.

[0088] The techniques described herein demonstrate that good query generation can significantly improve the performance of dual encoders, resulting in PROMPTAGATOR being significantly more efficient compared to other LLM applications for search.

[0089] The PROMPTAGATOR-based dual encoder outperforms ColBERTv2 and SPLADE v2 on 11 retrieval tasks.

[0090] PROMPTAGATOR can manipulate LLMs to generate text tailored to the target task, thus further improving retrieval performance.

[0091] Unlike existing few-shot search approaches, which are fine-tuning based and often require thousands of labeled examples, the techniques described herein synthetically generate labeled examples using a small number of task-specific prompts.

[0092] The zero-shot PROMPTAGATOR produces more variation for length queries.

[0093] A comparison of the distributions of first words in queries generated by different query generation methods shows that the distribution for synthetic queries generated by the Future Promptator is much closer to the distribution of real queries generated by humans.

[0094] The model reduces computational usage and environmental impact. For example, the energy cost and carbon footprint for the pre-trained model were 451 MWh and 26 tCO2e, respectively. In some embodiments, the additional instruction tuning gradient steps to fine-tune FLAN may be less than 2% of the number of pre-training steps, so the estimated additional energy cost may be relatively small.

[0095] Although FLAN is used herein for exemplary purposes, the LLM may be any model, including but not limited to variations of TO or PaLM.

[0096] Exemplary Uses The model can be deployed on a server.

[0097] Models may be tailored to particular types of tasks, knowledge domains, datasets, organizations, etc.

[0098] In some embodiments, the model may be applied to search for documents that support arguments and counterarguments. For example, a query may be "Companies should pay their employees at least X," and the documents in a query-document pair may include all posts on a social networking site that support or oppose the query.

[0099] In some embodiments, the model may be applied to perform semantic searches within a database.

[0100] Search tasks may be performed in different settings, such as financial, medical, educational, legal, etc. Many such settings contain data with protected information, and organizations may prefer not to share protected data. Models may be provided to organizations (e.g., via an application programming interface (API), as software as a service (SaaS), machine learning as a service (MLaaS), etc.), and the organization can enter a few prompts to train the model and apply the trained model, but all operations are performed and the data remains inside the organization's firewall.

[0101] Search tasks may include retrieving documents from a question, retrieving questions from a question, retrieving documents from an argument, retrieving documents supporting an argument, or retrieving documents supporting a counterargument.

[0102] In some embodiments, the task may be retrieving information from an email system.

[0103] As used herein, the term "documents" generally refers to any information retrieved from a corpus of documents in response to a query. The retrieved information may include documents, collections of documents, images, portions of images, videos, sound clips, portions of text from one or more documents from the corpus of documents, summaries and / or excerpts based on portions of text from one or more documents from the corpus of documents, etc. Also, for example, the term "corpus of documents" generally refers to documents related to a particular task. For example, the query may be a legal query related to a particular client's litigation matter, and the corpus of documents corresponds to client-specific documents, or more generally, litigation-specific documents. As another example, the query may relate to a medical question related to a particular illness, and the corpus of documents corresponds to documents related to the illness or related illnesses. Generally, a user may indicate and / or select a relevant corpus of documents. A corpus of documents may include documents stored in one or more databases (e.g., mobile, desktop, cloud, etc.), email folders, posts on social networking sites, collections of web pages, scientific databases, images in an image library, videos in a video library, audio, music libraries, etc. In general, the model may be applied to any collection of data from which information can be retrieved, organized such that the information is searchable. This may include text documents, images, videos, audio files, etc. Also, for example, document retrieval may include speech-to-text conversion.

[0104] In some embodiments, the query may be related to the issue being searched. For example, a query to an email folder may state, "Find information regarding baggage claim disputes," and the documents searched may be one or more emails exchanged as part of the dispute process. In some embodiments, the documents may be summaries outlining the history of the dispute, including relevant dates and issues. Such documents may be prepared based on a combination of sentiment analysis, natural language processing (NLP), and the like.

[0105] These and other exemplary applications are contemplated within the scope of this disclosure.

[0106] Training a machine learning model to generate inferences / predictions FIG. 12 shows a diagram 1200 illustrating the training stage 1202 and inference stage 1204 of trained machine learning model(s) 1232, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms with an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions regarding (the patterns in) the training data. The resulting trained machine learning algorithms may be referred to as trained machine learning models. For example, FIG. 12 shows the training stage 1202, in which one or more machine learning algorithm(s) 1220 are trained with training data 1210 to become trained machine learning model(s) 1232. Then, during the inference stage 1204, the trained machine learning model(s) 1232 may receive input data 1230 and one or more inference / prediction requests 1240 (perhaps as part of the input data 1230) and, in response, provide one or more inferences and / or prediction(s) 1250 as output.

[0107] Thus, trained machine learning model(s) 1232 may include one or more models of one or more machine learning algorithm(s) 1220. The machine learning algorithm(s) 1220 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks described herein, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, large-scale language models (LLMs), neural search systems, suitable statistical machine learning algorithms, and / or heuristic machine learning systems). The machine learning algorithm(s) 1220 may be supervised or unsupervised and may implement any suitable combination of online and offline learning.

[0108] In some examples, the machine learning algorithm(s) 1220 and / or the trained machine learning model(s) 1232 may be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such on-device coprocessors may be used to accelerate the machine learning algorithm(s) 1220 and / or the trained machine learning model(s) 1232. In some examples, the trained machine learning model(s) 1232 may be trained to provide inferences on, reside on, execute, and / or otherwise perform inferences for a particular computing device.

[0109] During the training phase 1202, the machine learning algorithm(s) 1220 may be trained by providing at least the training data 1210 as training inputs using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing some (or all) of the training data 1210 to the machine learning algorithm(s) 1220, and the machine learning algorithm(s) 1220 determining one or more output inferences based on the provided portion (or all) of the training data 1210. Supervised learning involves providing some (or all) of the training data 1210 to the machine learning algorithm(s) 1220, and the machine learning algorithm(s) 1220 determining one or more output inferences based on the provided portion (or all) of the training data 1210, where the output inference(s) are either accepted or corrected based on the correct results associated with the training data 1210. In some examples, the supervised learning of the machine learning algorithm(s) 1220 may be governed by a set of rules and / or a set of labels for the training inputs, which may be used to correct the inferences of the machine learning algorithm(s) 1220.

[0110] Semi-supervised learning involves having correct results for some, but not all, of the training data 1210. During semi-supervised learning, supervised learning is used for portions of the training data 1210 that have correct results, and unsupervised learning is used for portions of the training data 1210 that do not have correct results. Reinforcement learning involves the machine learning algorithm(s) 1220 receiving a reward signal related to a prior inference, where the reward signal may be a numerical value. During reinforcement learning, the machine learning algorithm(s) 1220 can output an inference and receive a reward signal in response, where the machine learning algorithm(s) 1220 are configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm(s) 1220 and / or the trained machine learning model(s) 1232 may be trained using other machine learning techniques, including, but not limited to, incremental learning and curriculum learning.

[0111] In some examples, the machine learning algorithm(s) 1220 and / or the trained machine learning model(s) 1232 can use transfer learning techniques. For example, transfer learning techniques may include the trained machine learning model(s) 1232 being pre-trained on a set of data and additionally trained using training data 1210. More specifically, the machine learning algorithm(s) 1220 may be pre-trained on data from one or more computing devices and the resulting trained machine learning model provided to computing device CD1, which is intended to execute the trained machine learning model during inference stage 1204. Then, during training stage 1202, the pre-trained machine learning model may be additionally trained using training data 1210, which may be derived from kernel data and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 1220 and / or the pre-trained machine learning model using training data 1210 of CD1's data may be performed using either supervised learning or unsupervised learning. The training phase 1202 may be completed once the machine learning algorithm(s) 1220 and / or pre-trained machine learning model(s) have been trained with at least the training data 1210. The resulting trained machine learning model may be utilized as at least one of the trained machine learning model(s) 1232. The training data 1210 may be a plurality of query-document pairs, each including a document from a synthetically generated query and document corpus.

[0112] In particular, once the training phase 1202 is complete, the trained machine learning model(s) 1232 may be provided to the computing device if not already on the computing device. The inference phase 1204 may begin after the trained machine learning model(s) 1232 are provided to the computing device CD1.

[0113] During the inference stage 1204, the trained machine learning model(s) 1232 may receive input data 1230 and generate and output one or more corresponding inferences and / or prediction(s) 1250 regarding the input data 1230. Thus, the input data 1230 may be used as input to the trained machine learning model(s) 1232 to provide the corresponding inference(s) and / or prediction(s) 1250 to kernel and non-kernel components. For example, the trained machine learning model(s) 1232 may generate the inference(s) and / or prediction(s) 1250 in response to one or more inference / prediction requests 1240. In some examples, the trained machine learning model(s) 1232 may be executed by another piece of software. For example, the trained machine learning model(s) 1232 may be executed by an inference or prediction daemon so that they are readily available to provide inferences and / or predictions upon request. The input data 1230 may include data from the computing device CD1 running the trained machine learning model(s) 1232 and / or input data from one or more computing devices other than CD1. For example, the input data 1230 may include at least two prompts associated with a search task to be performed against a corpus of documents relevant to the task. Also, for example, the input data 1230 may include an input query associated with a search task to be performed against a corpus of documents relevant to the task. Other types of input data are possible as well.

[0114] The inference(s) and / or prediction(s) 1250 may include output documents, output query-document pairs, numerical values ​​(e.g., search scores), and / or other output data produced by the trained machine learning model(s) 1232 operating on the input data 1230 (and training data 1210). In some examples, the trained machine learning model(s) 1232 may generate the output inference(s) and / or prediction(s) 1250 based on the input feedback. 1260 The trained machine learning model(s) 1232 can also rely on past inferences as input to generate new inferences.

[0115] Large-scale language models and / or document retrieval models may be examples of machine learning algorithm(s) 1220. After training, a trained version of a neural network may be an example of trained machine learning model(s) 1232. In this approach, one or more examples of inference / prediction request(s) 1240 may be requests in response to an input query, and corresponding examples of inference and / or prediction(s) 1250 may be documents predicted from a corpus of documents in response to the input query.

[0116] In some examples, one computing device CD_SOLO may include a trained version of the document retrieval model, possibly after training. The computing device CD_SOLO may then receive a request to respond to an input query and use the trained version of the neural network to predict documents from a corpus of documents that respond to the input query.

[0117] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide the predicted documents; for example, a first computing device CD_CLI may generate and send a request responsive to an input query to a second computing device CD_SRV. CD_SRV may then use a trained version of the neural network to predict predicted documents from a corpus of documents responsive to the input query and respond to a request from CD_CLI for the predicted documents. Then, upon receiving a response to the request, CD_CLI may provide the requested response to the input query (e.g., using a user interface and / or display, a printed copy, electronic communication, etc.).

[0118] Data Network Example 13 illustrates a distributed computing architecture 1300, according to an example embodiment. The distributed computing architecture 1300 includes server devices 1308, 1310 configured to communicate with programmable devices 1304a, 1304b, 1304c, 1304d, and 1304e via a network 1306. The network 1306 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 1306 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public Internet.

[0119] While FIG. 13 shows only five programmable devices, the distributed application architecture may service tens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 1304a, 1304b, 1304c, 1304d, and 1304e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, a mobile computing device, etc. In some examples, as shown by programmable devices 1304a, 1304b, 1304c, and 1304e, the programmable devices may be directly connected to the network 1306. In other examples, as shown by programmable device 1304d, the programmable devices may be indirectly connected to the network 1306 through an associated computing device, such as programmable device 1304c. In this example, programmable device 1304c may serve as an associated computing device for passing electronic communications between programmable device 1304d and the network 1306. In other examples, such as shown by programmable device 1304e, the computing device may be part of and / or within a vehicle such as a car, truck, bus, boat or ship, airplane, etc. In other examples not shown in Figure 13, the programmable device may be connected both directly and indirectly to the network 1306.

[0120] The server devices 1308, 1310 may be configured to perform one or more services requested by the programmable devices 1304a-1304e. For example, the server devices 1308 and / or 1310 may provide content to the programmable devices 1304a-1304e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or decrypted. Other types of content are possible as well.

[0121] As another example, server devices 1308 and / or 1310 may provide programmable devices 1304a-1304e with access to software for database, search, computation, graphics, audio, video, World Wide Web / Internet usage, and / or other functions. Many other examples of server devices are possible as well.

[0122] Computing Device Architecture 14 is a block diagram of an exemplary computing device 1400, according to an exemplary embodiment. In particular, the computing device 1400 shown in FIG. 14 may be configured to perform at least one function for prompt-based query generation, and / or method 1600, and / or method 1700.

[0123] The computing device 1400 may include a user interface module 1401, a network communication module 1402, one or more processors 1403, data storage 1404, one or more camera(s) 1412, one or more sensors 1414, and a power system 1416, all of which may be linked to each other via a system bus, network, or other connection mechanism 1405.

[0124] The user interface module 1401 may be operable to transmit data to and / or receive data from external user input / output devices. For example, the user interface module 1401 may be configured to transmit data to and / or receive data from user input devices such as a touchscreen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 1401 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices now known or later developed. The user interface module 1401 may also be configured to generate audible output using devices such as speakers, speaker jacks, audio output ports, audio output devices, earphones, and / or other similar devices. User interface module 1401 may further comprise one or more haptic devices that may generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with computing device 1400. In some examples, user interface module 1401 may be used to provide a graphical user interface (GUI) for utilizing computing device 1400, such as, for example, the graphical user interface of a mobile phone device. In some examples, user interface module 1401 may be used to provide a user-adjustable control for receiving a threshold distance.

[0125] The network communication module 1402 may include one or more devices providing one or more wireless interface(s) 1407 and / or one or more wired interface(s) 1408 configurable to communicate over a network. The wireless interface(s) 1407 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee™ transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other type of wireless transceiver configurable to communicate over a wireless network. The wired interface(s) 1408 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate over twisted pair wire, coaxial cable, fiber optic link, or similar physical connection to a wired network.

[0126] In some examples, the network communication module 1402 can be configured to provide reliable, secure, and / or authenticated communications. For each communication described herein, information to facilitate reliable communications (e.g., guaranteed message delivery) may be provided, perhaps as part of the message header and / or footer (e.g., packet / message ordering information, encapsulation header and / or footer, size / time information, and transmission verification information such as a cyclic redundancy check (CRC) and / or parity check value). Communications may be protected (e.g., encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or the Digital Signature Algorithm (DSA). Other encryption protocols and / or algorithms may be used similar to or in addition to those listed herein to protect (and subsequently decrypt / decode) communications.

[0127] The one or more processors 1403 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.) The one or more processors 1403 may be configured to execute computer-readable instructions 1406 contained in data storage 1404 and / or other instructions described herein.

[0128] Data storage 1404 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 1403. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical storage, magnetic storage, organic storage, or other memory or disk storage, which may be integrated, in whole or in part, with at least one of the one or more processors 1403. In some examples, data storage 1404 may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 1404 may be implemented using two or more physical devices.

[0129] Data storage 1404 may include computer-readable instructions 1406 and possibly additional data. In some examples, data storage 1404 may include storage necessary to execute at least some of the methods, scenarios, and techniques described herein and / or at least some of the functionality of the devices and networks described herein. In some examples, data storage 1404 may include storage for trained neural network models 1410 (e.g., models of trained neural networks, such as large-scale language models, dual encoders, etc.). In particular, in these examples, computer-readable instructions 1406 may include instructions that, when executed by one or more processors 1403, enable computing device 1400 to provide some or all of the functionality of trained neural network models 1410.

[0130] In some examples, computing device 1400 may include one or more camera(s) 1412. Camera(s) 1412 may include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light into one or more images. That is, camera(s) 1412 may generate image(s) of the captured light. The one or more images may be one or more still images and / or one or more images utilized in video capture. Camera(s) 1412 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as light of one or more other frequencies.

[0131] In some examples, computing device 1400 may include one or more sensors 1414. Sensors 1414 may be configured to measure conditions within computing device 1400 and / or conditions in the environment of computing device 1400 and provide data regarding these conditions.For example, the sensors 1414 may be (i) sensors for obtaining data regarding the computing device 1400, such as, but not limited to, a thermometer for measuring the temperature of the computing device 1400, a battery sensor for measuring the power of one or more batteries of the power supply system 1416, and / or other sensors for measuring the condition of the computing device 1400; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., a quick response (QR) code) reader, and a laser tracker, which may be configured to read identifiers such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to read and provide at least identification information; (iii) tilt sensors, gyroscopes, accelerometers, The sensors 1414 may include one or more of: (i) sensors that measure the position and / or movement of the computing device 1400, such as, but not limited to, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; (ii) environmental sensors that acquire data indicative of the environment of the computing device 1400, such as, but not limited to, infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, movement sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (iii) force sensors that measure one or more forces (e.g., inertial forces and / or G-forces) acting about the computing device 1400, such as, but not limited to, one or more sensors that measure force, torque, ground force, friction in one or more dimensions, and / or a zero moment point (ZMP) sensor that identifies the location of the ZMP. Many other examples of sensors 1414 are possible as well.

[0132] The power supply system 1416 may include one or more batteries 1418 and / or one or more external power interfaces 1420 for providing power to the computing device 1400. When electrically coupled to the computing device 1400, each battery of the one or more batteries 1418 can serve as a source of stored power for the computing device 1400. The one or more batteries 1418 of the power supply system 1416 may be configured to be portable. Some or all of the one or more batteries 1418 may be easily removable from the computing device 1400. In other examples, some or all of the one or more batteries 1418 may be internal to the computing device 1400 and therefore may not be easily removable from the computing device 1400. Some or all of the one or more batteries 1418 may be rechargeable. For example, a rechargeable battery may be recharged via a wired connection between the battery and another power source, such as by one or more power sources external to the computing device 1400 and connected to the computing device 1400 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1418 may be non-rechargeable batteries.

[0133] The one or more external power interfaces 1420 of the power system 1416 may include one or more wired power interfaces, such as a USB cable and / or a power cord, that enable a wired power connection to one or more power sources external to the computing device 1400. The one or more external power interfaces 1420 may include one or more wireless power interfaces, such as a Qi wireless charger, that enable a wireless power connection to one or more external power sources, such as via a Qi wireless charger. Once a power connection to an external power source is established using the one or more external power interfaces 1420, the computing device 1400 can draw power from the external power source via the established power connection. In some examples, the power system 1416 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0134] Cloud-based Server FIG. 15 illustrates a cloud-based server system according to an exemplary embodiment. In FIG. 15, the functionality of the prompt-based query generation model, the large-scale language model, and / or the document retrieval model may be distributed among computing clusters 1509a, 1509b, and 1509c. Computing cluster 1509a may include one or more computing devices 1500a, a cluster storage array 1510a, and a cluster router 1511a connected by a local cluster network 1512a. Similarly, computing cluster 1509b may include one or more computing devices 1500b, a cluster storage array 1510b, and a cluster router 1511b connected by a local cluster network 1512b. Similarly, computing cluster 1509c may include one or more computing devices 1500c, a cluster storage array 1510c, and a cluster router 1511c connected by a local cluster network 1512c.

[0135] In some embodiments, computing cluster 1509a, 1509b, 1509c may be a single computing device residing in a single computing center. In other embodiments, computing cluster 1509a, 1509b, 1509c may include multiple computing devices within a single computing center, or even multiple computing devices located in multiple computing centers located in various geographic locations. For example, FIG. 15 illustrates each of computing clusters 1509a, 1509b, 1509c residing in different physical locations.

[0136] In some embodiments, the data and services in computing clusters 1509 a, 1509 b, 1509 c may be stored on a non-transitory, tangible computer-readable medium (or computer-readable storage medium) and encoded as computer-readable information accessible by other computing devices. In some embodiments, computing clusters 1509 a, 1509 b, 1509 c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located in one or more diverse geographic locations.

[0137] In some embodiments, each of computing clusters 1509a, 1509b, and 1509c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0138] In computing cluster 1509a, for example, computing device 1500a may be configured to perform various computing tasks such as prompt-based query generation, large-scale language models, and / or document retrieval models. In one embodiment, various functions of the neural network and / or computing devices may be distributed among one or more of computing devices 1500a, 1500b, and 1500c. Computing devices 1500b and 1500c in each computing cluster 1509b and 1509c may be configured similarly to computing device 1500a in computing cluster 1509a. Meanwhile, in some embodiments, computing devices 1500a, 1500b, and 1500c may be configured to perform different functions.

[0139] In some embodiments, computing tasks and stored data associated with various computing tasks of prompt-based query generation, large-scale language models, and / or document retrieval models may be distributed across computing devices 1500a, 1500b, and 1500c based at least in part on the processing requirements of the neural networks and / or computing devices, the processing power of computing devices 1500a, 1500b, and 1500c, the latency of network links between computing devices within each computing cluster and between the computing clusters themselves, and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0140] The cluster storage arrays 1510a, 1510b, 1510c of the computing clusters 1509a, 1509b, 1509c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in combination with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0141] Similar to how the functionality of various computing tasks such as prompt-based query generation, large-scale language models, and / or document retrieval models may be distributed across computing devices 1500a, 1500b, and 1500c of computing clusters 1509a, 1509b, and 1509c, various active and / or backup portions of these components may be distributed across cluster storage arrays 1510a, 1510b, and 1510c. For example, some cluster storage arrays may be configured to store some portions of the data for a first layer of a neural network and / or computing devices, while other cluster storage arrays may store other portions(s) of the data for a second layer of a neural network and / or computing devices. Also, for example, some cluster storage arrays may be configured to store data for an encoder of a neural network, while other cluster storage arrays may store data for a decoder of the neural network. Additionally, some cluster storage arrays may be configured to store backup versions of data stored in other cluster storage arrays.

[0142] Cluster routers 1511a, 1511b, and 1511c in computing clusters 1509a, 1509b, and 1509c may include network equipment configured to provide internal and external communications for the computing clusters. For example, cluster router 1511a in computing cluster 1509a may include one or more Internet switching and routing devices configured to provide (i) local area network communications between computing device 1500a and cluster storage array 1510a via local cluster network 1512a, and (ii) wide area network communications between computing cluster 1509a and computing clusters 1509b and 1509c via wide area network link 1513a to network 1306. Cluster routers 1511b and 1511c may include network equipment similar to cluster router 1511a, and cluster routers 1511b and 1511c perform the same functions for computing clusters 1509b and 1509c that cluster router 1511a performs for computing cluster 1509a. c It can perform networking functions for

[0143] In some embodiments, the configuration of the cluster routers 1511a, 1511b, 1511c may be based at least in part on the data communication requirements of the computing devices and cluster storage arrays, the data communication capabilities of the network equipment within the cluster routers 1511a, 1511b, 1511c, the latency and throughput of the local cluster networks 1512a, 1512b, 1512c, the latency, throughput, and cost of the wide area network links 1513a, 1513b, 1513c, and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria of the mitigation system architecture.

[0144] Exemplary Methods of Operation 16 is a flowchart of a method 1600 according to an example embodiment. The method 1600 may be performed by a computing device, such as the computing device 1400. The method 1600 for prompt-based query generation may begin at block 1610, where the method includes receiving, by the computing device, at least two prompts related to a search task to be performed against a corpus of documents relevant to the task.

[0145] At block 1620, the method includes applying a large-scale language model to generate a synthetic training data set including a plurality of query-document pairs based on at least two prompts and a corpus of documents, each query-document pair including a document from the synthetically generated query and document corpus.

[0146] At block 1630, the method includes training a document retrieval model to obtain an input query related to a search task with a plurality of query-document pairs from a synthetic training dataset and to predict output documents to be retrieved from a corpus of documents.

[0147] At block 1640, the method includes providing, by the computing device, the trained document retrieval model.

[0148] Some embodiments include determining, by a document retrieval model, a search score for each query document pair, the search score indicating the relevance of the document of the query document pair to the query of the query document pair.

[0149] In some embodiments, the document retrieval model may be a dual encoder model that includes a query encoder that encodes the query and a document encoder that encodes the document.

[0150] In some embodiments, the dual encoder model may be a joint embedding model, where the query embedding of a given query is within a threshold distance of the document embedding of a given document if the given document has high relevance to the given query.

[0151] In some embodiments, the plurality of query document pairs may include noisy data. Such embodiments may include filtering the plurality of query document pairs to remove the noisy data. Such embodiments may include fine-tuning the document retrieval model based on the filtered plurality of query document pairs. In some embodiments, fine-tuning the document retrieval model may be based on a standard softmax loss with random negatives within a batch.

[0152] In some embodiments, the filtering includes one or more of length filtering, prompt filtering, or round-trip filtering.

[0153] In some embodiments, the large-scale language model may be a fine-tuned language net (FLAN).

[0154] In some embodiments, for each document in the corpus, queries may be sampled based on a distribution that includes an adjustable temperature hyperparameter. Such embodiments include maintaining the temperature hyperparameter below a temperature threshold to generate a diverse array of queries.

[0155] In some embodiments, the search task includes one or more of a question-to-document search, a question-to-question search, an argument-to-document search, an argument support document search, or a rebuttal support document search.

[0156] Some embodiments involve pre-training a document retrieval model on unsupervised general domain data.

[0157] In some embodiments, the search task and document corpus may be maintained behind an organization's firewall. Such embodiments include providing the large-scale language model and document retrieval model to the organization. The receiving, applying, and training may be performed behind the organization's firewall.

[0158] In some embodiments, at least two prompts is less than eight prompts.

[0159] 17 is a flowchart of a method 1700 according to an example embodiment. The method 1700 of applying a trained document retrieval model may be performed by a computing device, such as computing device 1400. The method 1700 may begin at block 1710, where the method includes receiving, by the computing device, an input query related to a search task to be performed against a corpus of documents relevant to the task.

[0160] At block 1720, the method includes predicting, by a trained document retrieval model, a document from a corpus of documents that has high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task.

[0161] At block 1730, the method includes providing, by the computing device, predicted documents in response to the input query.

[0162] In some embodiments, the document retrieval model may be a dual encoder model that includes a query encoder that encodes the query and a document encoder that encodes the document.

[0163] In some embodiments, the large-scale language model may be a fine-tuned language net (FLAN).

[0164] In some embodiments, the search task includes one or more of a question-to-document search, a question-to-question search, an argument-to-document search, an argument support document search, or a rebuttal support document search.

[0165] In some embodiments, the input query may be a voice input and the search task may be performed by an intelligent voice assistant implemented on a computer.

[0166] As described herein, PROMPTAGATOR is an approach to short-shot search that enables training task-specific retrievers and rerankers with only a small number of annotated examples. The short-shot examples amplified by prompt-based LLM query generation simplify the complexity of training neural retrievers for new tasks, leading to significant performance gains. PROMPTAGATOR can be viewed as a distillation of LLMs into a standard-sized dual encoder with prompt-based query generation. While the distillation process is computationally expensive, it significantly reduces the cost of inference.

[0167] The present disclosure should not be limited in terms of the specific embodiments described in this application, which are intended as illustrations of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope thereof. Functionally equivalent methods and apparatuses within the scope of the present disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the above description. Such modifications and variations are intended to be included within the scope of the appended claims.

[0168] The foregoing detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the figures, like symbols generally identify like elements unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0169] With respect to any or all of the ladder diagrams, scenarios, and flowcharts in the figures and as discussed herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages may be performed out of the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or functions may be used in any of the ladder diagrams, scenarios, and flowcharts discussed herein, and these ladder diagrams, scenarios, and flowcharts may be combined with each other, either in part or in whole.

[0170] The blocks representing processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the blocks representing processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement specific logical functions or operations in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk, or hard drive, or other storage medium.

[0171] Computer-readable media may also include non-transitory computer-readable media, such as register memory, processor cache, and non-transitory computer-readable media that store data for a short period of time, such as random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a longer period of time, such as secondary or permanent long-term storage devices, such as read-only memory (ROM), optical or magnetic disks, and compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, to be a computer-readable storage medium or a tangible storage device.

[0172] Additionally, blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules within the same physical device, although other information transfers may be between software and / or hardware modules in different physical devices.

[0173] While various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are offered by way of example and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

1. 1. A computer-implemented method for prompt-based query generation, comprising: receiving, by a computing device, at least two prompts related to a search task to be performed on a corpus of documents relevant to the task; applying a large-scale language model to generate a synthetic training data set including a plurality of query-document pairs based on the at least two prompts and the corpus of documents, each query-document pair including a synthetically generated query and a document from the corpus of documents; training a document retrieval model to take an input query related to the retrieval task with the plurality of query-document pairs from the synthetic training dataset and to predict output documents to be retrieved from the corpus of documents; providing, by the computing device, the trained document retrieval model; A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, further comprising determining, by the document retrieval model, a search score for each query-document pair, the search score indicating a relevance of the document of the query-document pair to the query of the query-document pair.

3. The computer-implemented method of claim 1 , wherein the document retrieval model is a dual-encoder model that includes a query encoder that encodes a query and a document encoder that encodes a document.

4. 4. The computer-implemented method of claim 3, wherein the dual-encoder model is a joint embedding model, and when a given document has high relevance to a given query, the query embedding of the given query is within a threshold distance of the document embedding of the given document.

5. The plurality of query-document pairs includes noisy data, and the computer-implemented method further comprises: The computer-implemented method of claim 1 , further comprising filtering the plurality of query-document pairs to remove the noisy data.

6. The computer-implemented method of claim 5 , further comprising fine-tuning the document retrieval model based on the filtered plurality of query-document pairs.

7. 7. The computer-implemented method of claim 6, wherein the fine-tuning of the document retrieval model is based on a standard softmax loss with random negatives within a batch.

8. The computer-implemented method of claim 5 , wherein the filtering includes one or more of length filtering, prompt filtering, or round-trip filtering.

9. The computer-implemented method of claim 1 , wherein the large-scale language model is a Fine-Tuned Language Net (FLAN).

10. For each document in the corpus, queries are sampled based on a distribution that includes an adjustable temperature hyperparameter, and the computer-implemented method comprises: The computer-implemented method of claim 1 , further comprising maintaining the temperature hyperparameter below a temperature threshold to generate a diverse array of queries.

11. 10. The computer-implemented method of claim 1, wherein the search task comprises one or more of a question-to-document search, a question-to-question search, an argument-to-document search, an argument support document search, or a rebuttal support document search.

12. The computer-implemented method of claim 1 , further comprising pre-training the document retrieval model on unsupervised general domain data.

13. The search task and the corpus of documents are maintained inside an organization's firewall, and the computer-implemented method comprises: providing the large-scale language model and the document retrieval model to the organization; The computer-implemented method of claim 1 , wherein the receiving, applying, and training are performed inside the organization's firewall.

14. The computer-implemented method of claim 1 , wherein the at least two prompts is less than eight prompts.

15. 1. A computer-implemented method of applying a trained document retrieval model, comprising: receiving, by a computing device, an input query related to a search task to be performed against a corpus of documents related to the task; predicting documents from the corpus of documents with the trained document retrieval model, the documents having high relevance to the input query, the document retrieval model being trained with a plurality of query-document pairs from a synthetic training dataset, the synthetic training dataset being generated by a large-scale language model based on at least two prompts related to the search task; providing, by the computing device, the predicted documents in response to the input query; A computer-implemented method comprising:

16. 16. The computer-implemented method of claim 15, wherein the document retrieval model is a dual-encoder model that includes a query encoder that encodes a query and a document encoder that encodes a document.

17. 16. The computer-implemented method of claim 15, wherein the large-scale language model is a Fine-Tuned Language Net (FLAN).

18. 16. The computer-implemented method of claim 15, wherein the search task includes one or more of a question-to-document search, a question-to-question search, an argument-to-document search, an argument support document search, or a rebuttal support document search.

19. 16. The computer-implemented method of claim 15, wherein the input query is a voice input and the search task is performed by a computer-implemented intelligent voice assistant.

20. determining, by the computing device, a request responsive to the input query; sending the request from the computing device to a second computing device, the second computing device including a trained version of the document retrieval model; after sending the request, the computing device receiving the predicted document from the second computing device; 16. The computer-implemented method of claim 15, wherein providing the predicted document comprises providing the predicted document upon receiving it from the second computing device.

21. 1. A computing device comprising: one or more processors; a data storage having stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform functions including the computer-implemented method of any one of claims 1 to 20.

22. The computing device of claim 21 , wherein the computing device is a mobile device.

23. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps according to the method of any one of claims 1 to 20.

24. 21. An article of manufacture comprising one or more non-transitory computer-readable media having stored thereon computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions comprising the computer-implemented method of any one of claims 1 to 20.

25. 1. A computing system comprising: A computing system comprising means for performing the computer-implemented method of any one of claims 1 to 20.

Images