A method for constructing an intelligent search engine based on a large language model

Abstract

The invention discloses a method for constructing an intelligent search engine based on a large language model, comprising: step 1: original query; step 2: manual setting of filters; step 3: based on the original query and filter setting input by the user, the retrieval rewriting module is responsible for converting it into a professional retrieval statement, and the modules involved in the retrieval rewriting process include: dynamic few-shot examples based on NER, professional terminology recommendation, retrieval main sentence rewriting based on LLM, retrieval main sentence verification, filter automatic planning module, retrieval sentence conversion module and retrieval main sentence merge filter clause; step 4: using a retrieval engine to retrieve relevant documents; step 5: document reordering; step 6: document summary and indexing. The invention optimizes the query rewriting module and the reordering module, and the accuracy of retrieval rewriting and the rationality of recall document reordering are significantly improved.

Description

Technical Field

[0001] The invention relates to a search engine construction method, and specifically to an intelligent search engine construction method based on a large language model. Background Art

[0002] The query rewriting task refers to converting the original query input by the user into a search statement that is easier for the search engine / database to understand. The present invention mainly considers the query rewriting task of the search engine. Most of the existing technical solutions based on large language models directly use prompts to call LLM to generate rewritten statements. For example, HyDE directly answers the user's query with LLM, and inputs the answer content as the result of query rewriting into the search engine, which can recall more relevant documents. Few-shot prompting is also a commonly used technology for enhancing the rewriting effect, but the examples used are usually fixed. In order to further improve the effect, a dynamic few-shot solution is often used in engineering, that is, the examples used in the prompt are dynamically obtained based on the relevance to the current input.

[0003] The document re-ranking task is to re-rank the list of documents roughly recalled by the retrieval engine according to their relevance to the question, so as to improve the accuracy of the large language model answer under the RAG framework. The most widely used re-ranker is based on the supervised fine-tuned BERT model. However, in actual business scenarios, we often cannot obtain enough annotated corpus to train the BERT re-ranker; in addition, if the model trained on other general corpus is directly used, satisfactory business results may not be achieved. With the rapid development of large language models, researchers have begun to try to use large language models as re-rankers in zero-shot scenarios. For example, the query likelihood (QLM) generated when decoding the large language model is used as the relevance score between the document and the question, and the relative order of all documents is obtained in the pointwise way. A large language model + listwise prompt solution is proposed to obtain document ranking, and an attempt is made to obtain the topk most relevant ordered documents in a sliding window manner combined with bubble sort.

[0004] In the query rewriting task, the existing solutions usually do not consider the automatic planning of filters, so it is difficult to achieve complex rewriting requirements such as impact factors and journal partitions. The present invention proposes a filter automatic planning and retrieval statement conversion framework to solve this problem. In addition, in the use of few-shot prompts, the existing solutions only consider the subject content similarity between queries, without considering the similarity of NER skeletons. The present invention proposes a dynamic few-shot prompt word strategy based on NER, which can recall few-shot examples with more reference value.

[0005] In the document re-ranking task based on a large language model, for pointwise ranking, existing solutions only use a single relevance score. Even if the LLM capability is strong, direct scoring in a zero-shot scenario will still result in many errors. Accordingly, the present invention proposes a multi-element scoring system solution, which first performs a coarse division on each document and then performs a fine division, thereby improving the accuracy of document ranking.

[0006] For listwise sorting tasks based on LLM, the existing solutions based on sliding window and bubble sort can only obtain the topk most relevant ordered documents, but cannot obtain the order of all documents; in addition, when topk is greater than the maximum number of documents m that can be sorted at one time by LLM, this solution cannot be used. Accordingly, the present invention proposes a solution based on a large language model and a fast sorting algorithm, which can obtain the order of all documents at a lower cost; and for the scenario where only the topk most relevant ordered documents need to be taken, this paper also proposes a solution combined with a partial fast sorting algorithm, which can also complete the sorting task when topk>m. Summary of the invention

[0007] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a method for constructing an intelligent search engine based on a large language model, optimize the query rewriting module and the reranking module, and significantly improve the accuracy of retrieval rewriting and the rationality of recalled document reranking.

[0008] The technical solution adopted by the present invention is: a method for constructing an intelligent search engine based on a large language model, comprising the following steps:

[0009] Step 1: Original Query (Input 1)

[0010] The original query refers to the search statement described by the user in natural language.

[0011] Step 2: Manual filter settings (input 2)

[0012] In addition to the original query, users can also set filters to constrain the recalled documents to obtain retrieval results that better meet their needs.

[0013] Step 3: Search rewrite

[0014] Based on the original query and filter settings input by the user, the retrieval rewriting module is responsible for converting it into a professional retrieval statement. The various modules involved in the retrieval rewriting process include: dynamic few-shot examples based on NER, professional terminology recommendation, LLM-based retrieval main sentence rewriting, retrieval main sentence verification, filter automatic planning module, retrieval sentence conversion module and retrieval main sentence merge filter clause.

[0015] Step 4: Use search engines to retrieve relevant documents

[0016] After obtaining the final search rewrite statement, enter it into the search engine to obtain a list of related documents.

[0017] Step 5: Reorder the documents

[0018] After recalling the relevant documents, these relevant documents need to be reordered to obtain the top k documents in a more reasonable order.

[0019] Step 6: Document Summarization and Indexing

[0020] Finally, based on the user's original query and the re-ranked document list, the present invention uses a large language model to summarize the document content, answer user questions, and identify the reference source after a specific answer segment, so as to facilitate the user to perform factual verification and improve the credibility of the answer content.

[0021] Furthermore, in step 2, by using a time filter, the user can specify the publication time range of the recalled documents; by using a publisher filter, the user can limit the return to only documents from a specified publisher; by using an impact factor filter, the user can specify the impact factor range of the recalled documents; by using a document type filter, the user can specify the type of the recalled document (formal article, commentary), etc.

[0022] Furthermore, the dynamic few-shot example based on NER in step 3 is specifically:

[0023] The few-shot example technology of the large language model refers to providing several examples related to the task in the prompt, thereby improving the success rate of LLM in completing the task. The dynamic few-shot technology means that the examples in the prompt are not fixed, but similar examples are queried from the example library based on the current input. Such examples are more referenceable and can further improve the accuracy of task execution.

[0024] During the launch of the intelligent search engine, the system will gradually accumulate real original user queries and search rewriting results, which can be used as "experience" after manual proofreading, that is, the source of dynamic few-shot sample libraries. However, if we directly use the original user query to search the sample library, we can recall examples with similar subject content, but it is difficult to recall examples with similar structures, and the latter are often more valuable for reference for search rewriting tasks.

[0025] A solution that can recall examples with similar subject content and similar structure. The specific process is mainly divided into two steps: named entity recognition and hybrid retrieval;

[0026] Named Entity Recognition (NER):

[0027] The original query entered by the user first passes through the named entity recognition module. The entity types include phenotype <phenotype>, Time Range<time_range> , Publisher Name<publisher_name> 、Name<person_name> 、Drug name<drug_name> After extracting the entity types and their positions contained in the original query, the entities are replaced with entity type identifiers to obtain the "NER skeleton" of the original query statement.

[0028] Hybrid search:

[0029] After obtaining the NER skeleton of the original query, we use the original query and the NER skeleton for hybrid retrieval to obtain rewritten examples with similar topics or structures. The retrieval method uses a hybrid retrieval method of BM25+dense embedding, where BM25 is used to measure literal similarity and dense embedding is used to evaluate semantic similarity.

[0030] The example library search engine connected to the hybrid search module uses the Elastic Search framework to build an index. Each record includes the original query string, the original query embedding, the NER skeleton string, and the NER skeleton embedding. The embedding model is based on BERT and the contrastive learning training based on the InfoNCE loss function. It is worth noting that when building a search index for the NER skeleton string, it is necessary to add special string fragments representing entity types to the tokenizer vocabulary, such as " <phenotype>”、"<time_range> ", etc., to avoid incorrectly splitting the tag during word segmentation.

[0031] Furthermore, in step 3, for the named entity recognition module, the present invention will adopt different construction strategies at different stages. In the cold start stage of the system, we adopt unsupervised NER based on a large language model; after the system has accumulated enough real user queries as training corpus, it will switch to supervised NER based on BERT.

[0032] 1) Unsupervised NER based on large language model:

[0033] When the amount of historical queries is insufficient, this paper adopts an unsupervised NER solution based on a large language model, which can be further divided into a single-stage or two-stage strategy. Different large language models are suitable for different strategies due to their different instruction alignment capabilities, and you can choose one based on the effect.

[0034] The single-stage strategy is to let the large language model directly output the recognized entity through the prompt instruction. The prompt contains the NER task description, the description of each entity type, and the definition of the output format. For the output format, this paper designs two output formats based on XML tags and JSON. The output format based on XML tags means that the large language model inserts XML start tags and end tags on both sides of the recognized entity;

[0035] By parsing the location and content of the XML tags, the recognized entity type, mention, and location can be obtained.

[0036] By parsing the json string, the mention and type of the entity can be obtained; unlike the xml output, in order to further obtain the location of the entity, it is necessary to use regular expressions or pattern matching algorithms to match the recognized mention in the input sentence. This paper uses the AC automaton multi-mode matching algorithm for mention matching.

[0037] The two-stage strategy means that the large language model is first asked to output the entity types contained in the query (stage one), and then for each entity type, the large language model is asked to extract entities of the specified type in the query (stage two). The purpose of this is to split a task into multiple subtasks, thereby reducing the difficulty and avoiding the problem of poor NER effect caused by too many entity types and a weak large language model when using a single-stage strategy.

[0038] 2) Supervised NER based on BERT:

[0039] In order to save computing resources and improve the speed and accuracy of entity recognition, when the historical query volume is sufficient, this article will train a supervised NER model based on BERT. The training corpus is annotated and manually verified using a large language model. The annotation form uses BIO (B-begin, I-inside, O-outside) sequence annotation, and the model architecture uses the BERT+CRF layer architecture.

[0040] Furthermore, the professional terminology recommendation in step 3 is specifically as follows:

[0041] In addition to including relevant documents, some search engines also use terms in the professional term library to mark the keywords and subject terms of the document. Users can directly use professional terms to perform keyword / subject term searches when searching.

[0042] Specifically, when an original query is input, the present invention does not directly use the query to recall relevant professional terms, but first uses a large language model (LLM) to summarize the keyword list by itself. If the LLM has some knowledge of the specified professional terminology library, it can also be directly asked to try to output relevant professional terms as a keyword list in the prompt word, but the professional terms output by the LLM are not necessarily completely consistent with the terms in the standard library. Afterwards, use the keyword list as input to search in the self-built Elastic Search term retrieval engine, and recall the topk related term list. Note that the writing of the recalled professional terms is completely correct at this time, which can be used as a reference source for the retrieval main sentence rewriting module to assist in more accurate retrieval rewriting.

[0043] Furthermore, the LLM-based rewriting of the main search sentence in step 3 is specifically as follows:

[0044] After obtaining a few-shot example similar to the original query and a recommended list of professional terms, the present invention constructs a prompt and uses a large language model to complete the rewriting task of the retrieval main sentence, wherein the prompt includes a rewriting task description, retrieval engine related information, rewriting principles, few-shot examples, a recommended list of professional terms, the user's original query, etc. After obtaining the output of the LLM, it is parsed to obtain the rewritten retrieval main sentence.

[0045] Furthermore, the retrieval of the main sentence verification in step 3 is specifically as follows:

[0046] The main search sentence rewritten by LLM may have grammatical errors, too many conditional restrictions, and other problems, which will lead to subsequent search errors or empty search results. Therefore, the present invention adopts a multi-round "rewrite-verify" mechanism. After obtaining the rewritten results of the main search sentence in each round, try to use it for retrieval. If the search engine reports an error, the error message of the search engine is used as a prompt and fed back to the large language model, requiring it to modify the existing rewritten sentence. Through this strategy, the accuracy of the rewritten sentence and the success rate of the search execution can be improved.

[0047] Furthermore, in step 3, the filter automatic planning module, the search sentence conversion module and the search main sentence merge the filter clause specifically as follows:

[0048] User queries often contain retrieval requirements that are difficult to process, but the retrieval library connected to the system may not directly support retrieval by setting the label (Field) of the search statement.

[0049] Based on this, this paper proposes a filter automatic planning and retrieval statement conversion framework, and introduces the filter tool as a supplement to the main rewriting process to achieve more accurate retrieval rewriting.

[0050] Specifically, in the "automatic filter planning" module, in order to improve parallelism and rewriting speed, we did not use a "greedy" agent such as ReAct, but first planned the required filter list, and then determined the parameters of each filter in parallel and called the filter to generate the corresponding search clause.

[0051] In the "Filter Planning" module, we use the large language model and prompt word engineering to determine the required filter list. The prompt used should include the task description, original query, instructions for using all filters, etc. For each selected filter, the "Filter Parameter Determination" module further determines the filter parameters through the large language model and prompt word engineering; then the filter is called to generate the corresponding search clause, and different filters have different generation logics.

[0052] In addition to automatically planning the required filters, users can also directly select filters and set parameters in the system, and the system will also call the filters to generate search clauses for them. Finally, after merging the search clauses generated by all filters with the search main clause, the final search statement rewriting result R can be obtained.

[0053] Furthermore, in step 5, during the cold start phase of the intelligent search engine, that is, when insufficient annotated corpus is available, we cannot use a re-ranker trained on corpus. In addition, if a re-ranker trained on other corpora is used directly, poor business results may result because the domain and language style of the training corpus may not be consistent with the actual application scenario.

[0054] Based on this, this paper proposes two zero-shot reranking schemes based on large language models, hoping to utilize the powerful general capabilities of large language models to better complete the reranking task in zero-shot scenarios without training data, and gradually accumulate training corpus for the subsequent construction of supervised reranking models.

[0055] According to different technical routes, it can be divided into Pointwise reordering and Listwise reordering. Pointwise reordering refers to inputting a query and a document, calculating the relevance score between the document and the query through a scoring model, and then reordering the documents according to the score of each document. Listwise reordering refers to inputting a query and multiple documents, and directly determining the order of each document through a sorting model. Below we will explain the two solutions in turn.

[0056] 1. Pointwise reordering based on large language model:

[0057] Large language models directly give document relevance scores in zero-shot scenarios, which often result in inaccurate scores. In order to reduce the deviation from the true score, this paper proposes a multi-element scoring framework that gives each document a score tuple / vector instead of just a single score. The length of the score tuple and the content of the examination can be flexibly set according to different business scenarios.

[0058] Taking a 3D tuple as an example, when sorting documents based on the score tuple, r1, r2, and r3 will be compared in sequence, and the criteria are:

[0059] d i >d j If and only if any of the following occurs where d i >d j means that document i (more relevant) is ranked before document j, r n (i) represents the r of the i-th document n Score (n=1,2,3).

[0060] The present invention designs a 3-dimensional score tuple (r1, r2, r3), where the first 2-dimensional scores are obtained by letting the large language model answer true or false questions or multiple-choice questions. Different answers will be mapped to different scores, the purpose of which is to preliminarily divide documents into different levels. As shown in the following table, r1 is the first-level score, which is used to evaluate whether the document content is biased or toxic; r2 is the second-level score, which is used to preliminarily divide the relevance of the document.

[0061]

[0062] Example of a multi-level scoring system

[0063] The last level score is a floating point number between 0 and 1, which represents a more refined relevance score. Its function is to further sort the documents that have been classified into different levels (based on r1 and r2). For scenarios where the probability distribution of the output token of the large language model cannot be obtained, we will use the black box score s black ; Otherwise, we will use the white-box score s white .

[0064] For black box scoring black The present invention uses prompt to directly let the large language model evaluate the relevance between the query and the document, and outputs a relevance score between 0 and 1 in the form of natural language. At the same time, it requires step-by-step explanation of the reasons for the score (CoT, chain of thought). Explaining the reasons helps to obtain a more accurate relevance score.

[0065] For white-box scoring white If the decoding process of the large language model can be controlled, this paper will use the query likelihood score (QLM) as the document relevance score; otherwise, this paper will directly use the p(True) score as the document relevance score.

[0066] 2. Listwise reordering based on large language model:

[0067] In addition to using the Pointwise method to assign a score to each document in turn and then sorting based on the score, you can also use the Listwise method to let the large language model directly evaluate the relative order between multiple documents. However, when there are a large number of documents to be sorted, it is often impossible to complete the sorting using a one-time input and output. One reason is that it is limited by the context window size of the large language model, and the other is that when sorting too many documents at one time, the large language model cannot make a good judgment, and the effect is much lower than when only sorting a small number of documents.

[0068] Accordingly, this paper proposes a solution based on the quicksort algorithm and the large language model ListwisePrompt, which can quickly obtain the relative order among all documents in the scenario where the large language model can only sort some documents at a time.

[0069] The Quick Sort algorithm is a sorting algorithm based on pairwise comparison. It adopts the divide-and-conquer strategy with a time complexity of O(nlogn), where n is the number of elements to be sorted. In this paper, the "pairwise comparison" in the algorithm is changed to "batch comparison of multiple elements", and the recursion can be terminated in advance when the length of the divide-and-conquer subarray is less than m, where m is the maximum number of elements that the large language model can sort at one time.

[0070] In addition, in addition to re-ranking all the recalled documents, the more common scenario we encounter is to only obtain the list of the top k (<n) most relevant documents from n documents, and these top k documents are sorted according to the relevance from high to low. In this scenario, this paper is achieved by making corresponding adjustments to the partial quicksort algorithm.

[0071] The difference between the llm_partial_qsort algorithm and the llm_qsort algorithm is not significant. The difference is only that the recursive divide-and-conquer of the right interval is not always called.

[0072] Advantages of the present invention:

[0073] First, the development of large language models has brought about profound changes to the search engine field. By introducing large language models in the retrieval processes such as retrieval rewriting, document re-ranking, and retrieval result summarization, the intelligence level of search engines has been greatly improved, thereby reducing the threshold for users to write professional retrieval statements and shortening the time for users to read search results.

[0074] Second, in the retrieval rewriting module:

[0075] 1. A filter automatic planning and retrieval statement conversion framework is proposed as a supplement to the main rewriting process, which can effectively achieve high-difficulty rewriting in complex scenarios.

[0076] 2. A dynamic few-shot prompting strategy based on named entity recognition is proposed, which effectively improves the referenceability of the recalled examples, thereby improving the accuracy of retrieval statement rewriting;

[0077] Third, in the document re-ranking module, the present invention proposes two zero-shot document re-ranking schemes based on large language models, which can effectively solve the problem that supervised models cannot be used due to insufficient corpus in the cold start stage:

[0078] 1. A pointwise re-ranking scheme based on a large language model is proposed. The document relevance is decomposed into multiple layers of scores using a multi-level scoring framework, making the final document ranking more accurate and reasonable.

[0079] 2. Combining the quick sort algorithm / partial quick sort algorithm, a listwise reordering scheme based on a large language model is proposed to solve the problem that the large language model cannot give a complete sorting at one time through the listwise prompt. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 A flow chart of the method implemented by the present invention;

[0081] Figure 2 This is a dynamic few-shot flow chart based on NER of the present invention;

[0082] Figure 3 Recommended flow chart for the technical terminology of the present invention;

[0083] Figure 4 Generate a flow chart for filter planning and search clauses of the present invention;

[0084] Figure 5 This is the Pointwise and Listwise reordering flow chart of the present invention;

[0085] Figure 6 It is a flow chart of the quick sorting algorithm llm_qsort based on LLM of the present invention;

[0086] Figure 7 It is the flow chart of interval partitioning algorithm llm_partition of the present invention;

[0087] Figure 8 This is the flow chart of the partial quick sorting algorithm llm_partial_qsort based on LLM of the present invention. DETAILED DESCRIPTION

[0088] The present invention will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0089] like Figure 1 As shown, a method for constructing an intelligent search engine based on a large language model includes the following steps:

[0090] Step 1: Original Query (Input 1)

[0091] Original query refers to the search statement described by the user in natural language, such as "research progress of type 1 diabetes in the past three years", "tumor-related clinical trial research in JAMA in the past five years", etc.

[0092] Step 2: Manual filter settings (input 2)

[0093] In addition to the original query, users can also constrain the recalled documents by setting filters to obtain search results that better meet their needs. For example, using the time filter, users can specify the publication time range of the recalled documents; using the publisher filter, users can limit the return to only documents from the specified publisher; using the impact factor filter, users can specify the impact factor range of the recalled documents; using the document type filter, users can specify the type of recalled documents (formal articles, current affairs comments), etc.

[0094] Step 3: Search rewrite

[0095] Based on the original query and filter settings entered by the user, the search rewriting module is responsible for converting them into professional search statements. The following table shows relevant examples of PubMed search statement rewriting. Only by using the rewritten search statements can accurate searches and recall relevant documents in the search engine.

[0096]

[0097]

[0098] PubMed search rephrasing example

[0099] The various modules involved in the search rewriting process are described below.

[0100] 1. Dynamic few-shot example based on NER

[0101] The few-shot example technology of the large language model refers to providing several examples related to the task in the prompt, thereby improving the success rate of LLM in completing the task. The dynamic few-shot technology means that the examples in the prompt are not fixed, but similar examples are queried from the example library based on the current input. Such examples are more referenceable and can further improve the accuracy of task execution.

[0102] During the launch of the intelligent search engine, the system will gradually accumulate real original user queries and search rewriting results, which can be used as "experience" after manual proofreading, that is, the source of dynamic few-shot sample libraries. However, if we directly use the original user query to search the sample library, we can recall examples with similar subject content, but it is difficult to recall examples with similar structures, and the latter are often more valuable for reference for search rewriting tasks.

[0103] Based on this, this paper proposes a solution that can recall examples with similar subject content and similar structure. The specific process is as follows Figure 2 As shown, it is mainly divided into two steps: named entity recognition and hybrid retrieval, which are explained in detail as follows.

[0104] (1) Named Entity Recognition:

[0105] The original query entered by the user first passes through the named entity recognition module. The entity types include phenotype <phenotype>, Time Range<time_range> , Publisher Name<publisher_name> 、Name<person_name> 、Drug name<drug_name> After extracting the entity types and their positions contained in the original query, the entities are replaced with entity type identifiers to obtain the "NER skeleton" of the original query statement, as shown in the following table.

[0106]

[0107] NER skeleton example

[0108] For the named entity recognition module, the present invention will adopt different construction strategies at different stages. In the cold start stage of the system, we use unsupervised NER based on a large language model; after the system has accumulated enough real user queries as training corpus, it will switch to supervised NER based on BERT.

[0109] 1) Unsupervised NER based on large language model:

[0110] When the amount of historical queries is insufficient, this paper adopts an unsupervised NER solution based on a large language model, which can be further divided into a single-stage or two-stage strategy. Different large language models are suitable for different strategies due to their different instruction alignment capabilities, and you can choose one based on the effect.

[0111] The single-stage strategy is to let the large language model directly output the recognized entity through the prompt instruction. The prompt contains the NER task description, the description of each entity type, and the definition of the output format. For the output format, this paper designs two output formats based on XML tags and JSON. The output format based on XML tags means that the large language model inserts XML start tags and end tags on both sides of the recognized entity. Taking "Research progress of type 1 diabetes in the past three years" as an example, the expected output of the large model is:

[0112] <phenotype> Type 1 diabetes< / phenotype> <time_range> In the past 3 years< / time_range> Research progress

[0113] By parsing the position and content of the XML tag, the recognized entity type, mention, and position can be obtained. Similarly, based on the JSON output format, the expected output of the large model is:

[0114] [{"mention":"Type 1 diabetes","entity_type":"phenotype"},{"mention":"Recent 3 years","entity_type":"time_range"}]

[0115] By parsing the json string, the mention and type of the entity can be obtained; unlike the xml output, in order to further obtain the location of the entity, it is necessary to use regular expressions or pattern matching algorithms to match the recognized mention in the input sentence. This paper uses the AC automaton multi-mode matching algorithm for mention matching.

[0116] The two-stage strategy means that the large language model is first asked to output the entity types contained in the query (stage one), and then for each entity type, the large language model is asked to extract entities of the specified type in the query (stage two). The purpose of this is to split a task into multiple subtasks, thereby reducing the difficulty and avoiding the problem of poor NER effect caused by too many entity types and a weak large language model when using a single-stage strategy.

[0117] 2) Supervised NER based on BERT:

[0118] In order to save computing resources and improve the speed and accuracy of entity recognition, when the historical query volume is sufficient, this paper will train a supervised NER model based on BERT. The training corpus is annotated and manually verified using a large language model. The annotation form uses BIO (B-begin, I-inside, O-outside) sequence annotation. The model architecture uses the BERT+CRF layer architecture. The loss function L is defined as follows:

[0119] L=-lnP(y|x)

[0120]

[0121] Where x=(x1x2…x T ) represents the input token sequence (original search sentence), y=(y1y2…y T ) represents the correct entity BIO tag sequence, T is the sequence length, Represents the y at the i-th position of the model output i The probability of the label, A∈R k*k is a trainable probability transfer matrix, k is the number of output label types, Indicates the current position output y i Label, next position output y i+1 The "transition" probability of the label. S(x,y) represents the score of outputting sequence y given sequence x, P(y|x) represents the probability of outputting sequence y given sequence x, and y' represents all possible label sequences except sequence y. During training, ∑ y′ e s(x,y′) It is calculated by the dynamic programming forward algorithm. By minimizing the loss function L, the probability of outputting the y sequence is maximized, thereby realizing network parameter training.

[0122] (2) Hybrid search:

[0123] After obtaining the NER skeleton of the original query, the original query and the NER skeleton are used for mixed retrieval to obtain rewritten examples with similar themes or structures. For example, if "research progress of type 1 diabetes in the past 3 years" is entered as the original query, examples with similar themes such as "treatment plans for type 1 diabetes" and "genetic factors of type 1 diabetes" can be recalled, as well as examples with similar structures such as "research progress of Marfan syndrome in the past 5 years". The retrieval method uses a mixed retrieval method of BM25+denseembedding, where BM25 is used to measure literal similarity and dense embedding is used to evaluate semantic similarity.

[0124] The example library search engine connected to the hybrid search module uses the Elastic Search framework to build an index. Each record includes the original query string, the original query embedding, the NER skeleton string, and the NER skeleton embedding. The embedding model is based on BERT and the contrastive learning training based on the InfoNCE loss function. It is worth noting that when building a search index for the NER skeleton string, it is necessary to add special string fragments representing entity types to the tokenizer vocabulary, such as " <phenotype>”、"<time_range> ", etc., to avoid incorrectly splitting the tag during word segmentation.

[0125] 2. Recommended professional terms

[0126] In addition to collecting relevant documents, some search engines also use terms in the professional term library to mark the keywords and subject terms of the documents. Users can directly use professional terms to search for keywords / subject terms when searching. For example, PubMed uses the MeSH term library as the subject term library for its collected documents. For this type of search engine, the present invention adopts the following method: Figure 3 The process shown makes recommendations for professional terminology.

[0127] Specifically, when an original query is input, the present invention does not directly use the query to recall relevant professional terms, but first uses a large language model (LLM) to summarize the keyword list by itself. If the LLM has some knowledge of the specified professional terminology library, it can also be directly asked to try to output relevant professional terms as a keyword list in the prompt word, but the professional terms output by the LLM are not necessarily completely consistent with the terms in the standard library. Afterwards, use the keyword list as input to search in the self-built Elastic Search term retrieval engine, and recall the topk related term list. Note that the writing of the recalled professional terms is completely correct at this time, which can be used as a reference source for the retrieval main sentence rewriting module to assist in more accurate retrieval rewriting.

[0128] 3. Rewriting the main retrieval clause based on LLM

[0129] After obtaining a few-shot example similar to the original query and a recommended list of professional terms, the present invention constructs a prompt and uses a large language model to complete the rewriting task of the retrieval main sentence, wherein the prompt includes a rewriting task description, retrieval engine related information (such as the retrieval engine name, restricted fields), rewriting principles, few-shot examples, a recommended list of professional terms, the user's original query, etc. After obtaining the output of the LLM, it is parsed to obtain the rewritten retrieval main sentence.

[0130] 4. Retrieval of main sentence verification

[0131] The main search sentence rewritten by LLM may have grammatical errors, too many conditional restrictions, and other problems, which will lead to subsequent search errors or empty search results. Therefore, the present invention adopts a multi-round "rewrite-verify" mechanism. After obtaining the rewritten results of the main search sentence in each round, try to use it for retrieval. If the search engine reports an error, the error message of the search engine is used as a prompt and fed back to the large language model, requiring it to modify the existing rewritten sentence. Through this strategy, the accuracy of the rewritten sentence and the success rate of the search execution can be improved.

[0132] 5. Automatic filter planning module, search sentence conversion module and search main sentence merge filter clause

[0133] User queries often include difficult-to-handle search requirements, such as limiting the impact factor range of returned documents, limiting the SCI classification of documents, limiting the source of search results to official government announcements, etc. However, the search library connected to the system may not directly support searching by setting labels for search statements. For example, PubMed does not support the "impact factor" label, and the Google search engine does not support setting the result type of official government announcements.

[0134] Based on this, this paper proposes a filter automatic planning and retrieval statement conversion framework, and introduces the filter tool as a supplement to the main rewriting process to achieve more accurate retrieval rewriting.

[0135] Specifically, in the "automatic filter planning" module, in order to improve parallelism and rewriting speed, we did not use a "greedy" agent such as ReAct, but first planned the required filter list, and then determined the parameters of each filter in parallel and called the filter to generate the corresponding search clause.

[0136] The overall framework is as follows Figure 4 As shown in the figure, in the "Filter Planning" module, we determine the required filter list [F1, F2, F3] through the large language model and prompt word engineering. The prompt used should contain the task description, the original query, the instructions for using all filters, etc. For each selected filter F, the parameters of the filter are further determined in the "Filter Parameter Determination" module through the large language model and prompt word engineering; then the filter is called to generate the corresponding search clause, and different filters have different generation logics.

[0137] In addition to automatically planning the required filters, users can also directly select filters and set parameters in the system, and the system will also call the filters to generate search clauses for them. Finally, after merging the search clauses generated by all filters with the search main clause, the final search statement rewriting result R can be obtained.

[0138] Step 4: Use search engines to retrieve relevant documents

[0139] After obtaining the final search rewrite statement, enter it into the search engine to obtain a list of related documents.

[0140] Step 5: Reorder the documents

[0141] After recalling the relevant documents, these relevant documents need to be reordered to obtain the top k documents in a more reasonable order.

[0142] During the cold start phase of the intelligent search engine, when there is not enough annotated corpus, we cannot use the re-ranker model trained based on the corpus. In addition, if we directly use the re-ranker model trained with other corpora, it may lead to poor business results because the domain and language style of the training corpus may not be consistent with the actual application scenario.

[0143] Based on this, this paper proposes two zero-shot reranking schemes based on large language models, hoping to utilize the powerful general capabilities of large language models to better complete the reranking task in zero-shot scenarios without training data, and gradually accumulate training corpus for the subsequent construction of supervised reranking models.

[0144] According to different technical routes, it can be divided into Pointwise reordering and Listwise reordering. Figure 5 As shown in the figure, Pointwise reordering means inputting a query and a document, calculating the relevance score between the document and the query through a scoring model, and then reordering the documents according to the score of each document. Listwise reordering means inputting a query and multiple documents, and directly determining the order of each document through a sorting model. Below we will explain the two solutions in turn.

[0145] 1. Pointwise Reordering Based on Large Language Model

[0146] Large language models directly give document relevance scores in zero-shot scenarios, which often result in inaccurate scores. In order to reduce the deviation from the true score, this paper proposes a multi-element scoring framework that gives each document a score tuple / vector instead of just a single score. The length of the score tuple and the content of the examination can be flexibly set according to different business scenarios.

[0147] Taking a 3D tuple as an example, when sorting documents based on the score tuple, r1, r2, and r3 will be compared in sequence, and the criteria are:

[0148] d i >d j If and only if any of the following occurs where d i >d j means that document i (more relevant) is ranked before document j, r n (i) represents the r of the i-th document n Score (n=1,2,3).

[0149] The present invention designs a 3-dimensional score tuple (r1, r2, r3), where the first 2-dimensional scores are obtained by letting the large language model answer true or false questions or multiple-choice questions. Different answers will be mapped to different scores, the purpose of which is to preliminarily divide documents into different levels. As shown in the following table, r1 is the first-level score, which is used to evaluate whether the document content is biased or toxic; r2 is the second-level score, which is used to preliminarily divide the relevance of the document.

[0150]

[0151] Example of a multi-level scoring system

[0152] The last level score is a floating point number between 0 and 1, which represents a more refined relevance score. Its function is to further sort the documents that have been classified into different levels (based on r1 and r2). For scenarios where the probability distribution of the output token of the large language model cannot be obtained, we will use the black box score s black ; Otherwise, we will use the white-box score s white .

[0153] For black box scoring black The present invention uses prompt to directly let the large language model evaluate the relevance between the query and the document, and outputs a relevance score between 0 and 1 in the form of natural language. At the same time, it requires step-by-step explanation of the reasons for the score (CoT, chain of thought). Explaining the reasons helps to obtain a more accurate relevance score.

[0154] For white-box scoring white If the decoding process of the large language model can be controlled, this paper will use the query likelihood score (QLM) as the document relevance score; otherwise, this paper will directly use the p(True) score as the document relevance score.

[0155] The principle of query likelihood score is to ask LLM to generate a relevant question based on the document content through prompt. When decoding, the log likelihood probability of each token of the decoded query is calculated and the length is normalized as the document score. The prompt used in this article is: "Please generate a question that can be answered based on the document content. The document content is: {doc}". Query likelihood score S QLM is defined as follows:

[0156]

[0157] Among them, |q| represents the number of query tokens, q t Indicates the tth token of the query, q <t represents the 1st to t-1th token of the query, d represents the document, P(q t |d,q <t , prompt) is the generation probability of the tth token during decoding.

[0158] The principle of p(True) scoring is to ask LLM to determine whether the document content is relevant to the query through prompt, and limit the model to only output "true" or "false" in the prompt, and do not add any other explanations or descriptions. The probability that the first token output by LLM is "true" is directly taken as the relevance score of the document, that is:

[0159] p(True)=P(q1=′true′|d,q <1 ,prompt)=P(q1=′true′|d,prompt)

[0160] The prompt used in this article is: "{doc}\nBased on the content of the above document, can you answer the following question: {query}? Please directly output true or false without adding any other explanations or instructions."

[0161] 2. Listwise Reranking Based on Large Language Model

[0162] In addition to using the Pointwise method to assign a score to each document in turn and then sorting based on the score, you can also use the Listwise method to let the large language model directly evaluate the relative order between multiple documents. However, when there are a large number of documents to be sorted, it is often impossible to complete the sorting using a one-time input and output. One reason is that it is limited by the context window size of the large language model, and the other is that when sorting too many documents at one time, the large language model cannot make a good judgment, and the effect is much lower than when only sorting a small number of documents.

[0163] Based on this, this paper proposes a solution based on the quick sort algorithm and the Listwise Prompt of the large language model. In the scenario where only a part of the documents can be sorted at a time using the large language model, the relative order of all documents can be quickly obtained. The prompt used in this paper is: "The following lists {n_docs} documents, each document uses [x] as an identifier (x is a number): \n[1]\n{docs[1]}\n[2]\n{docs[2]}... Please sort the above documents according to the relevance of the document content to the user's question. The more relevant documents are ranked first. The output format is: [x1]>[x2]>...".

[0164] The Quick Sort algorithm is a sorting algorithm based on pairwise comparison. It adopts the divide-and-conquer strategy and has a time complexity of O(nlogn), where n is the number of elements to be sorted. In this paper, the "pairwise comparison" in the algorithm is changed to "batch comparison of multiple elements", and the recursion can be terminated in advance when the length of the divide-and-conquer sub-array is less than m, where m is the maximum number of elements that the large language model can sort at one time.

[0165] The algorithm flow chart is as Figure 6 and Figure 7 .

[0166] Among them, the llm_inplace_sort function means that after using the LLM to obtain the order of the input document list, the array A is sorted in place; the llm_leq_pivot function means that after using the LLM to obtain the order between (the input document list + the pivot document), the list of subscripts of the elements before the pivot document is returned.

[0167] In addition, in addition to re-ranking all the recalled documents, the more common scenario we encounter is that we only need to obtain the top k (<n) most relevant document lists from n documents, and these top k documents are sorted in descending order of relevance. In this scenario, this paper is implemented by making corresponding adjustments to the partial quicksort algorithm.

[0168] The algorithm flow chart is as Figure 8 .

[0169] It can be seen that the difference between the llm_partial_qsort algorithm and the llm_qsort algorithm is not significant. The difference is only that the recursive divide-and-conquer of the right interval is not always called.

[0170] Step 6: Document summarization and indexing

[0171] Finally, based on the user's original query and the re-ranked document list, the present invention uses the large language model to summarize the document content, answer the user's questions, and mark the citation source after specific answer segments, which is convenient for users to conduct factual verification and improve the credibility of the answer content.< / phenotype> < / phenotype> < / phenotype> < / phenotype>

Claims

1. A method for constructing an intelligent search engine based on a large language model, characterized in that: The following steps are involved: Step 1: Original Query The original query refers to the search statement described by the user in natural language; Step 2: Manual filter settings In addition to the original query, users can also set filters to constrain the recalled documents to obtain retrieval results that better meet their needs; Step 3: Search rewrite Based on the original query and filter settings input by the user, the search rewriting process is responsible for converting it into a professional search statement. The modules involved in the search rewriting process include: dynamic few-shot example module based on NER, professional term recommendation module, search main sentence rewriting module based on LLM, search main sentence verification module, filter automatic planning module, search statement conversion module and search main sentence merge filter clause module; Step 4: Use search engines to retrieve relevant documents After obtaining the final search rewriting statement, input it into the search engine to obtain a list of related documents; Step 5: Reorder the documents After recalling the relevant documents, these relevant documents need to be reordered to obtain the topk documents in a more reasonable order; Step 6: Document Summarization and Indexing Finally, based on the user's original query and the re-ranked document list, a large language model is used to summarize the document content and answer the user's question. The reference source is marked after a specific answer segment to facilitate the user's factual verification and improve the credibility of the answer content. The dynamic few-shot example module based on NER in step 3 is specifically: The few-shot example technology of the large language model refers to providing several examples related to the task in the prompt, thereby improving the success rate of LLM in completing the task; the dynamic few-shot technology means that the examples in the prompt are not fixed, but similar examples are queried from the example library based on the current input. Such examples are more referenceable and can further improve the accuracy of task execution; During the launch of the intelligent search engine, the system will gradually accumulate the results of real original user queries and search rewrites, which can be used as "experience" after manual proofreading, that is, the source of dynamic few-shot sample libraries; The specific method is divided into two steps: named entity recognition and hybrid retrieval; Named Entity Recognition: The original query entered by the user first passes through the named entity recognition module. The entity types include phenotype <phenotype> , Time Range<time_range> , Publisher Name<publisher_name> 、Name<person_name> 、Drug name<drug_name> , after extracting the entity types and their positions contained in the original query, the entities are replaced with entity type identifiers to obtain the "NER skeleton" of the original query statement;< / phenotype> Hybrid search: After obtaining the NER skeleton of the original query, the original query and the NER skeleton are used for hybrid retrieval to obtain rewritten examples with similar topics or structures. The retrieval method uses a hybrid retrieval method of BM25+dense embedding, where BM25 is used to measure literal similarity and dense embedding is used to evaluate semantic similarity. The sample library retrieval engine connected to the above hybrid search uses the Elastic Search framework to build an index. Each record includes the original query string, original query embedding, NER skeleton string, and NER skeleton embedding. The embedding model is based on BERT and contrastive learning training based on the InfoNCE loss function. When building a retrieval index for the NER skeleton string, it is necessary to add special string fragments representing entity types to the tokenizer's vocabulary.

2. The method for constructing an intelligent search engine based on a large language model according to claim 1, characterized in that: In step 2, by using a time filter, the user can specify the publication time range of the recalled documents; by using a publisher filter, the user can limit the return to only documents from a specified publisher; by using a document impact factor filter, the user can specify the impact factor range of the recalled documents; by using a document type filter, the user can specify the type of the recalled documents.

3. The method for constructing an intelligent search engine based on a large language model according to claim 2, characterized in that: In step 3, different construction strategies will be adopted for named entity recognition at different stages; in the cold start stage of the system, unsupervised NER based on a large language model will be adopted; after the system has accumulated enough real user queries as training corpus, it will switch to supervised NER based on BERT; 1) Unsupervised NER based on large language model: When the historical query volume is insufficient, an unsupervised NER solution based on a large language model is used, which is further divided into a single-stage or two-stage strategy. Different large language models are suitable for different strategies due to their different instruction alignment capabilities, and you can choose one based on the effect. The single-stage strategy is to let the large language model directly output the recognized entity through the prompt instruction. The prompt contains the NER task description, the description of each entity type, and the definition of the output format. For the output format, two output formats based on XML tags and JSON are designed. The output format based on XML tags means that the large language model inserts XML start tags and end tags on both sides of the recognized entity. By parsing the position and content of the XML tag, the recognized entity type, mention, and position can be obtained; By parsing the json string, the mention and type of the entity can be obtained. Unlike the xml output, in order to further obtain the location of the entity, a regular expression or pattern matching algorithm is required to match the recognized mention in the input sentence. The AC automaton multi-mode matching algorithm is used for mention matching. The two-stage strategy means that the large language model is first used to output the entity types contained in the query statement, and then for each entity type, the large language model is used to extract entities of the specified type in the query statement. 2) Supervised NER based on BERT: In order to save computing resources and improve the speed and accuracy of entity recognition, when the historical query volume is sufficient, a supervised NER model based on BERT will be trained. The training corpus is annotated and manually verified using a large language model. The annotation form uses BIO sequence annotation, and the model architecture uses the BERT+CRF layer architecture.

4. The method for constructing an intelligent search engine based on a large language model according to claim 3, characterized in that: The professional terminology recommendation module in step 3 is specifically: In addition to indexing relevant documents, some search engines also use terms in the professional term library to mark the keywords and subject terms of documents. Users can directly use professional terms to search for keywords / subject terms when searching; An original query is input, and the relevant professional terms are not directly used to recall the query. Instead, LLM is used to summarize the keyword list by itself. If LLM has some knowledge of the specified professional terminology library, it can be directly asked to try to output relevant professional terms as a keyword list in the prompt word, but the professional terms output by LLM are not necessarily completely consistent with the terms in the standard library. After that, the keyword list is used as input to search in the self-built Elastic Search term retrieval engine to recall the topk related term list. Note that the writing of the recalled professional terms is completely correct at this time, which will be used as a reference source for the retrieval main sentence rewriting module to assist in more accurate retrieval rewriting.

5. The method for constructing an intelligent search engine based on a large language model according to claim 4, characterized in that: The LLM-based retrieval main sentence rewriting module in step 3 is specifically: After obtaining a few-shot examples similar to the original query and a list of recommended professional terms, a prompt is constructed to use a large language model to complete the task of rewriting the retrieval main sentence. The prompt includes the rewriting task description, retrieval engine-related information, rewriting principles, few-shot examples, a list of recommended professional terms, and the user's original query. After obtaining the output of the LLM, it is parsed to obtain the rewritten retrieval main sentence.

6. The method for constructing an intelligent search engine based on a large language model according to claim 5, characterized in that: The main sentence verification module in step 3 is specifically retrieved as follows: A multi-round "rewrite-verify" mechanism is adopted. After obtaining the rewriting results of the main search sentence in each round, an attempt is made to use it for retrieval. If the search engine reports an error, the error message of the search engine is used as a prompt and fed back to the large language model, requiring it to modify the existing rewritten sentence.

7. The method for constructing an intelligent search engine based on a large language model according to claim 6, characterized in that: The filter automatic planning module, the search sentence conversion module and the search main sentence merge filter clause module in step 3 are specifically: User queries often contain difficult-to-process search requirements, but the search database that the system connects to may not directly support searching by setting search statement tags; Using a filter automatic planning and search statement conversion framework, introducing filter tools as a supplement to the main rewriting process can achieve more accurate search rewriting; In the "Automatic filter planning" module, in order to improve parallelism and rewriting speed, the required filter list is first planned overall, and then the parameters of each filter are determined in parallel, and the filter is called to generate the corresponding search clause; In the "Automatic filter planning" module, the required filter list is determined through the large language model and prompt word engineering. The prompt used should include the task description, the original query, and the instructions for using all filters. For each selected filter, the filter parameters are further determined through the large language model and prompt word engineering. Then, the filter is called to generate the corresponding search clause. Different filters have different generation logics. In addition to automatically planning the required filters, users can also directly select filters and set parameters in the system. The system will also call the filters to generate search clauses for them. Finally, after merging the search clauses generated by all filters with the main search clause, the final search statement rewriting result R can be obtained.

8. The method for constructing an intelligent search engine based on a large language model according to claim 7, characterized in that: According to different technical routes, step 5 is divided into Pointwise reordering and Listwise reordering. Pointwise reordering refers to inputting a query and a document, calculating the relevance score between the document and the query through a scoring model, and then reordering the documents according to the score of each document. Listwise reordering refers to inputting a query and multiple documents, and directly determining the order of each document through a sorting model.

1. Pointwise reordering based on large language model: In zero-shot scenarios, large language models directly give document relevance scores, which often result in inaccurate scores. To reduce the deviation from the actual score, a multivariate scoring framework is used to assign a score tuple / vector to each document instead of just a single score. The length of the score tuple and the content of the examination can be flexibly set according to different business scenarios. For 3D tuples, when sorting documents based on the score tuple, r1, r2, and r3 are compared in turn, and the criteria are: d i >d j If and only if any of the following occurs where d i >d j Indicates that document i is ranked before document j, r n (i) represents the r of the i-th document n score, n = 1, 2, 3; A 3-dimensional score tuple (r1, r2, r3), where the first two dimensions are obtained by asking a large language model to answer true or false questions or multiple-choice questions. Different answers are mapped to different scores, with the purpose of preliminarily dividing documents into different levels; r1 is the first-level score, which is used to evaluate whether the document content is biased or toxic; r2 is the second-level score, which is used to preliminarily divide the relevance of the document; The score of the last level is a floating point number between 0 and 1, which represents a more refined relevance score. Its function is to further sort the documents that have been classified into each level. For scenarios where the probability distribution of the output token of the large language model cannot be obtained, the black box score s is used. black Otherwise, the white-box score s will be used white ; For black box scoring black , use prompt to directly let the large language model evaluate the relevance between the query and the document, output a relevance score between 0 and 1 in the form of natural language, and require step-by-step explanation of the reason for the score. Explaining the reason helps to get a more accurate relevance score; For white-box scoring white ,If the decoding process of the large language model can be controlled, the query likelihood score will be used as the ,relevance score of the document; otherwise, the p(True) score will be directly used as the ,relevance score of the document; 2. Listwise reordering based on large language model: In addition to using the Pointwise method to assign a score to each document in turn and then sorting based on the score, you can also use the Listwise method to let the large language model directly evaluate the relative order between multiple documents; however, when there are a large number of documents to be sorted, it is often impossible to complete the sorting using a one-time input and output. One reason is that it is limited by the context window size of the large language model, and the other is that when too many documents are sorted at one time, the large language model cannot make a good judgment, and the effect is much lower than when only a small number of documents are sorted; A solution based on the quick sort algorithm and the large language model Listwise Prompt is used to quickly obtain the relative order of all documents in a scenario where only a portion of the documents can be sorted at a time using the large language model. The quick sort algorithm is a sorting algorithm based on pairwise comparison, which adopts a divide-and-conquer strategy and has a time complexity of O(nlogn), where n is the number of elements to be sorted. The "pairwise comparison" in the algorithm is changed to "multi-element batch comparison", and the recursion can be terminated early when the length of the divide-and-conquer subarray is less than m, where m is the maximum number of elements that can be sorted at a time by the large language model. In addition to re-sorting all recalled documents, a more common scenario is to obtain a list of the top k most relevant documents from n documents, and these top k documents are sorted from high to low according to relevance; in this scenario, this is achieved by making corresponding adjustments to some quick sort algorithms.

Images