Knowledge base retrieval method and system applied to large language model

Abstract

The invention relates to the technical field of large language models, and discloses a knowledge base retrieval method and system applied to a large language model. According to the method, multiple independent retrieval engines are made through different retrieval steps and different execution strategies corresponding to the retrieval steps, the retrieval results corresponding to the data query statements are retrieved through the independent retrieval engines, and therefore the retrieval results are subjected to data merging, the prompt templates corresponding to the data query statements are obtained, and the user experience is improved. A retrieval process is refined into a plurality of different steps, a corresponding execution strategy is formulated for each step, a plurality of independent retrieval engines are further constructed, retrieval results matched with problems are retrieved by using various independent retrieval engines, automatic generation of the independent retrieval engines is realized, and the retrieval efficiency is improved. And the problem of information omission caused by a single retrieval strategy is solved through diversified retrieval strategies, so that the accuracy of model query is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a knowledge base retrieval method and system applied to large language models. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress in the field of natural language processing and are widely used in multiple fields such as intelligent customer service, content creation, knowledge retrieval, etc. These models have demonstrated excellent text generation, understanding and question-answering capabilities through deep learning of large-scale public data sets.

[0003] However, large language models mainly rely on massive public data sets for training, which often makes them incapable of processing specific enterprise private data due to lack of targeted knowledge. At the same time, the knowledge base of large language models is fixed after training and cannot reflect changes in the real world in real time. Moreover, the training goal of large language models is to predict the next most likely token rather than strictly reasoning based on facts. Finally, large language models also face the challenge of poor interpretability. Due to the high complexity of its internal structure and algorithm, the specific basis of the model output is often difficult to trace and explain, which makes it difficult for users to understand and trust the output results of the model. The above problems make large language models have certain limitations after inputting questions (queries), so that they cannot output query results in a targeted manner. In order to overcome these limitations, researchers have explored integrating embedding technology into the text retrieval process, by converting text into high-dimensional vectors and storing them in vector databases, and using advanced similarity algorithms to retrieve text fragments that best match the query.

[0004] At present, although this method of retrieving text fragments has improved the query efficiency of large models, in practical applications, different retrieval strategies have their own advantages. Relying solely on a single retrieval strategy still has limitations, especially when facing complex and changeable query scenarios. It is easy to miss information closely related to the query question, thereby reducing the accuracy of the model query results. Summary of the invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] In view of the above-mentioned shortcomings of the prior art, the present application discloses a knowledge base retrieval method and system applied to a large language model to overcome the problem of information omission caused by a single retrieval strategy.

[0007] The present application discloses a knowledge base retrieval method applied to a large language model, comprising: pre-setting a retrieval strategy material library, and formulating multiple independent retrieval engines based on the retrieval strategy material library, wherein the retrieval strategy material library includes one or more retrieval steps, and multiple execution strategies corresponding to each of the retrieval steps; obtaining a data query statement; parsing the data query statement to obtain a parsing result, and determining multiple target engines from each of the independent retrieval engines based on the parsing result, and determining a target integration strategy from the retrieval strategy material library; executing each of the target engines in parallel according to the data query statement to obtain the retrieval results output by each of the target engines, and screening and integrating each of the retrieval results through the target integration strategy to obtain a prompt template as an input to the large language model, wherein the target engine is used to perform knowledge base retrieval according to the data query statement.

[0008] In one embodiment of the present application, an independent search engine is developed in the following manner: determining a target step from each of the search steps, and determining a target strategy from the execution strategies corresponding to the target steps; arranging the target steps in a preset execution order to form an independent search engine consisting of the target steps and the target strategies.

[0009] In one embodiment of the present application, the target step includes a retriever execution step, wherein the target strategy corresponding to the retriever execution step includes at least two of the following: executing a vector similarity retriever, wherein the vector similarity retriever includes performing vector similarity calculation on data nodes in a preset knowledge base according to the data query statement to obtain a similarity index, and determining a retrieval result from the data node according to the similarity index, wherein the similarity index includes at least one of cosine similarity, Euclidean distance, and data dot product; executing a keyword frequency retriever, wherein the keyword frequency retriever includes obtaining text keywords in the data query statement and obtaining text documents in a preset knowledge base; obtaining the total number of documents by counting the number of text documents, and obtaining the number of key documents by counting the number of text documents containing the text keywords; obtaining the text by using a preset inverse document frequency formula based on the total number of documents and the number of key documents. The inverse document frequency weight corresponding to the keyword, the inverse document frequency formula is provided with an adjustment parameter so that the inverse document frequency weight is not equal to 0; the occurrence frequency of the text keyword in the text document is calculated according to the inverse document frequency weight to obtain the text similarity between the text keyword and the text document, and the retrieval result is determined from the text document according to the text similarity; a relational database retriever is executed, wherein the relational database retriever includes generating a database retrieval statement according to the data query statement, and searching from a preset relational database through the database retrieval statement to obtain the retrieval result; a graph database retriever is executed, wherein the graph database retriever includes searching from a preset graph database according to the data query statement to obtain the retrieval result; a search engine retriever is executed, wherein the search engine retriever includes inputting the data query statement into a preset search engine so that the preset search engine outputs the retrieval result.

[0010] In one embodiment of the present application, the target step also includes a query statement expansion step, and the execution order of the query statement expansion step is before the retriever execution step, wherein the target strategy corresponding to the query statement expansion step includes: a hypothetical document strategy, including inputting a preset document prompt word and the data query statement into a preset large language model, so as to guide the large language model to output a hypothetical document corresponding to the data query statement through the document prompt word, and updating the data query statement according to the hypothetical document, wherein the document prompt word includes an application environment corresponding to the data query statement; a multi-query strategy, including obtaining a question prompt word, and inputting the question prompt word and the data query statement into a preset large language model, so as to guide the large language model to output multiple sub-questions corresponding to the data query statement through the question prompt word, merging each of the sub-questions and the data query statement to obtain a sub-question list, and updating the data query statement according to the sub-question list.

[0011] In one embodiment of the present application, the target step also includes a node post-processing step, and the execution order of the node post-processing step is after the retriever execution step, wherein the target strategy corresponding to the node post-processing step includes at least two of the following: a deduplication strategy, including performing hash value calculations on the multiple search results output by the retriever execution step, and deduplicating each of the search results according to the comparison results between the hash values; a re-ranking strategy, including re-ranking the multiple search results output by the retriever execution step through a Rerank model; a K-means strategy, including calculating the multiple search results output by the retriever execution step using a K-means algorithm, determining abnormal nodes from each of the search results according to the calculated difference values, and deleting the abnormal nodes from each of the search results; an average value strategy, including scoring the multiple search results output by the retriever execution step, obtaining node scores corresponding to each of the search results, calculating the node scores corresponding to each of the search results, obtaining a score average, determining abnormal nodes from each of the search results according to the comparison results between the score average and each of the node scores, and deleting the abnormal nodes from each of the search results.

[0012] In one embodiment of the present application, the target integration strategy integrates the search results in at least one of the following ways: an Extend strategy, including obtaining a prompt template by splicing multiple search results; a reciprocal sorting fusion strategy, including fusing multiple search results using a preset reciprocal sorting fusion algorithm to obtain a prompt template; an Intersection merge strategy, including determining a node intersection based on multiple search results, and merging the node intersection to obtain a prompt template.

[0013] In one embodiment of the present application, before integrating each of the search results, the target integration strategy filters each of the search results in at least one of the following ways: deduplicating the search results in the prompt template according to the hash values ​​corresponding to each search result in the prompt template; reordering each search result in the prompt template through a Rerank model; calculating each search result in the prompt template using a K-means algorithm to filter each search result in the prompt template according to the calculation result; scoring each search result in the prompt template, and filtering each search result in the prompt template according to the scoring result.

[0014] In one embodiment of the present application, after formulating multiple independent search engines according to the search strategy material library, the method also includes: obtaining query statement samples and obtaining sample labels corresponding to the query statement samples; executing each of the independent search engines according to the query statement samples to obtain sample search results corresponding to each of the independent search engines; performing similarity calculations on each of the sample search results according to the sample labels to determine the search scores corresponding to each of the independent search engines; and filtering each of the independent search engines according to the search scores.

[0015] In one embodiment of the present application, multiple target engines are determined from each of the independent search engines based on the analysis results, and a target integration strategy is determined from a search strategy material library, including: obtaining a training sample set, wherein the training sample set includes multiple query statement types and type samples corresponding to each of the query statement types; obtaining search scores corresponding to different query statement types of each of the independent search engines based on the training sample set, so as to determine multiple type engines corresponding to each of the query statement types from each of the independent search engines; extracting the current statement type corresponding to the data query statement from the analysis results; determining multiple target engines from each of the independent search engines based on the matching results between the current statement type and each of the query statement types, and matching them from the search strategy material library based on the current statement type to obtain a target integration strategy.

[0016] The present application discloses a knowledge base retrieval system applied to a large language model, comprising: a setting module, used to set a retrieval strategy material library, and formulate multiple independent retrieval engines according to the retrieval strategy material library, wherein the retrieval strategy material library includes one or more retrieval steps, and multiple execution strategies corresponding to each of the retrieval steps; an acquisition module, used to acquire a data query statement; a determination module, used to parse the data query statement to obtain a parsing result, and determine multiple target engines from each of the independent retrieval engines according to the parsing result, and determine a target integration strategy from the retrieval strategy material library; a retrieval module, used to execute each of the target engines in parallel according to the data query statement, obtain the retrieval results output by each of the target engines, and filter and integrate each of the retrieval results through the target integration strategy to obtain a prompt template as an input to the large language model, wherein the target engine is used to perform knowledge base retrieval according to the data query statement.

[0017] Beneficial effects of this application:

[0018] Through the search strategy material library, a variety of independent search engines are formulated, and the data query statements are parsed to obtain the parsing results, so as to determine the target engine and the target integration strategy from the independent search engine and the search strategy material library respectively through the parsing results, so as to execute each target engine in parallel, and integrate each search result into a prompt template through the target integration strategy as the input of the large language model. In this way, by refining the search process into multiple different steps and formulating corresponding execution strategies for each step, multiple independent search engines are constructed, and the search results matching the search questions are retrieved by various independent search engines, which not only realizes the automatic generation of independent search engines, but also overcomes the problem of missing information caused by a single search strategy through diversified search strategies, thereby improving the accuracy of model queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a knowledge base retrieval method applied to a large language model in an embodiment of the present application;

[0020] Figure 2 is a flowchart of another knowledge base retrieval method applied to a large language model in an embodiment of the present application;

[0021] Figure 3 It is a flowchart of an independent search engine execution method for an independent search engine in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of a hypothetical method for executing a document policy in an embodiment of the present application;

[0023] Figure 5It is a flowchart of a method for executing a multi-query strategy in an embodiment of the present application;

[0024] Figure 6 It is a structural diagram of a knowledge base retrieval system applied to a large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.

[0026] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0027] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0028] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0029] Unless otherwise stated, the term "plurality" means two or more.

[0030] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0031] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0032] Combination Figure 1 As shown, the embodiment of the present disclosure provides a knowledge base retrieval method applied to a large language model, including:

[0033] Step S101, pre-setting a search strategy material library, and formulating multiple independent search engines according to the search strategy material library;

[0034] The search strategy material library includes one or more search steps, and a plurality of execution strategies corresponding to each search step;

[0035] Step S102, obtaining a data query statement;

[0036] Step S103, by parsing the data query statement, obtaining the parsing result, determining multiple target engines from each independent search engine according to the parsing result, and determining the target integration strategy from the search strategy material library;

[0037] Step S104, executing each target engine in parallel according to the data query statement, obtaining the search results output by each target engine respectively, and filtering and integrating each search result through the target integration strategy to obtain a prompt template as the input of the large language model;

[0038] Among them, the target engine is used to search the knowledge base according to the data query statement.

[0039] The knowledge base retrieval method applied to the large language model provided by the embodiment of the present disclosure is adopted, and multiple independent retrieval engines are formulated through the retrieval strategy material library, and the data query statement is parsed to obtain the parsing result, so as to determine the target engine and the target integration strategy from the independent retrieval engine and the retrieval strategy material library respectively through the parsing result, so as to execute each target engine in parallel, and integrate each retrieval result into a prompt template through the target integration strategy, so as to serve as the input of the large language model. In this way, by refining the retrieval process into multiple different steps and formulating corresponding execution strategies for each step, multiple independent retrieval engines are constructed, and the retrieval results matching the retrieval questions are retrieved by various independent retrieval engines, which not only realizes the automatic generation of independent retrieval engines, but also overcomes the information omission problem caused by a single retrieval strategy through diversified retrieval strategies, thereby improving the accuracy of model query.

[0040] Optionally, an independent search engine is developed in the following manner: determine a target step from each search step, and determine a target strategy from an execution strategy corresponding to the target step; arrange the target steps in a preset execution order to form an independent search engine consisting of the target steps and the target strategy.

[0041] In some embodiments, the target steps are arranged in a preset execution order, including a query statement expansion step (Query Transform), a retriever execution step (Retriever), and a node post-processing step (Node Post-process).

[0042] Combination Figure 2 As shown, the embodiment of the present disclosure provides a knowledge base retrieval method applied to a large language model, including:

[0043] Step S201, obtaining a data query statement (Query);

[0044] Among them, data query statements belong to natural language;

[0045] Step S202, inputting the data query statements into a plurality of independent search engines (Single Query Engines) respectively, so as to retrieve search results corresponding to the data query statements from a preset data pool through the independent search engines;

[0046] Among them, the independent search engines all use different independent search engines to ensure that the data query statements are parsed and retrieved from different angles and dimensions;

[0047] Among them, the search results are composed of similar nodes (Nodes);

[0048] Step S203, performing a merge operation (MergeOperate) according to the search results outputted by each independent search engine, to obtain a prompt template;

[0049] Step S204, post-processing the prompt template to further optimize and screen;

[0050] Step S205: input the data query statement and the prompt template into the large language model to obtain the search results output by the large language model.

[0051] Combination Figure 3 As shown, the embodiment of the present disclosure provides an independent search engine execution method for an independent search engine, including:

[0052] Step S301, obtaining an independent search engine;

[0053] Step S302, determining whether the independent search engine has a query statement expansion step, if yes, jump to step S303, if no, jump to step S304;

[0054] Step S303, executing the target strategy corresponding to the query statement expansion step;

[0055] The query statement expansion step is used to perform text expansion on the data query statement and update the data query statement according to the text expansion result.

[0056] Step S304, executing the target strategy corresponding to the retriever execution step;

[0057] The retrieval step is used to calculate the similarity of the data nodes in the preset data pool according to the data query statement to obtain the data similarity, and retrieve the data nodes according to the data similarity to obtain the retrieval result corresponding to the data query statement;

[0058] Step S305, executing the target strategy corresponding to the node post-processing step;

[0059] The node post-processing step is used to perform node processing on the search results output by the retriever execution step, and the node processing includes node integration and / or node filtering.

[0060] Optionally, the target step also includes a query statement expansion step, and the query statement expansion step is executed before the retriever execution step.

[0061] In some embodiments, by performing text expansion on the data query statement, the original query can be optimized and expanded to improve the query effect of the retrieval result.

[0062] Optionally, the target strategy corresponding to the query statement expansion step includes: a hypothetical document strategy, including inputting preset document prompt words and data query statements into a preset large language model, so as to guide the large language model to output a hypothetical document corresponding to the data query statement through the document prompt words, and updating the data query statement according to the hypothetical document, wherein the document prompt words include the application environment corresponding to the data query statement.

[0063] In some embodiments, the hypothetical document strategy adopts the retrieval enhancement technology HyDE (Hypothetical Document Embeddings), which uses large language models (LLMs) to generate hypothetical relevant documents for a given query, and then uses these hypothetical documents as new queries, embedding and searching for neighboring documents through a dense retriever, thereby improving the retrieval effect of the query by generating hypothetical documents.

[0064] In some embodiments, the hypothetical document strategy includes: utilizing the text generation capabilities of large language models (such as GPT-3, BERT, etc.) to generate hypothetical related documents for a given query, and attempting to capture information related to the query through these hypothetical documents; embedding the generated hypothetical documents through a dense retriever to generate an embedded representation of the document, so as to convert the document into a vector form and facilitate similarity comparison in the vector space; using the embedded representation of the hypothetical document, searching for actual documents adjacent to these hypothetical documents in the embedded space, and treating these adjacent documents as documents related to the original query; by using the hypothetical document as a new query, the HyDE method can expand the semantic information of the original query, thereby improving the accuracy of the retrieval. The hypothetical document contains more relevant information, allowing the retriever to more accurately match related documents.

[0065] In some embodiments, the document prompt words include fields such as role field, skill field, action field, constraint field and query field, and the prompt words carried by each field are shown in Table 1.

[0066] Table 1

[0067]

[0068] Combination Figure 4 As shown, the embodiment of the present disclosure provides a method for executing a hypothetical document policy, including:

[0069] Step S401, obtaining a prompt word template consisting of document prompt words;

[0070] Step S402, combining the data query statement and the prompt word template;

[0071] Step S403, calling the large model according to the data query statement and the prompt word template to generate a hypothetical document;

[0072] Step S404: update the data query statement according to the hypothetical document.

[0073] Optionally, the target strategy corresponding to the query statement expansion step includes: a multi-query strategy, including obtaining question prompt words, and inputting the question prompt words and the data query statement into a preset large language model, so as to guide the large language model to output multiple sub-questions corresponding to the data query statement through the question prompt words, merge each sub-question and the data query statement to obtain a sub-question list, and update the data query statement according to the sub-question list.

[0074] In some embodiments, a multi-query strategy (Multi-Question) generates multiple similar sub-questions for an original query through a large language model, and uses these sub-questions and the original query for retrieval.

[0075] Combination Figure 5 As shown, the embodiment of the present disclosure provides a method for executing a multi-query strategy, including:

[0076] Step S501, obtaining a prompt word template consisting of question prompt words;

[0077] Step S502, combining the data query statement and the question prompt word;

[0078] Step S503, calling the large model according to the data query statement and the question prompt words to generate multiple sub-questions;

[0079] Step S504: merge each sub-question and the data query statement to obtain a sub-question list, and update the data query statement according to the sub-question list.

[0080] Optionally, the target step includes a retriever execution step.

[0081] In some embodiments, the main function of the retriever execution step is to retrieve relevant documents based on the data query statement input by the user or the data query statement after executing the expansion step.

[0082] Optionally, the target strategy corresponding to the retriever execution step includes: executing a keyword frequency retriever, wherein the keyword frequency retriever includes obtaining text keywords in a data query statement and obtaining text documents in a preset knowledge base; obtaining the total number of documents by counting the number of text documents, and obtaining the number of key documents by counting the number of text documents containing text keywords; obtaining an inverse document frequency weight corresponding to the text keyword by calculating based on the total number of documents and the number of key documents using a preset inverse document frequency formula, wherein the inverse document frequency formula is provided with an adjustment parameter so that the inverse document frequency weight is not equal to 0; calculating the frequency of occurrence of the text keyword in the text document based on the inverse document frequency weight to obtain the text similarity between the text keyword and the text document, and determining the retrieval result from the text document based on the text similarity.

[0083] In some embodiments, the inverse document frequency formula without setting the adjustment parameter is as follows:

[0084]

[0085] In formula (1), q i is the i-th keyword in the data query statement, IDF(q i ) is the inverse document frequency weight corresponding to the i-th keyword, N is the total number of documents, n(q i ) is the number of key documents corresponding to the i-th keyword.

[0086] In some embodiments, when the ratio of the number of key documents to the total number of documents is 1:2, the result of the inverse document frequency weight is 0, which leads to the data similarity between the text keyword and the text document being 0. Therefore, an adjustment parameter is added to the inverse document frequency formula, and the obtained inverse document frequency formula is as follows:

[0087]

[0088] In formula (2), α is an adjustment parameter, and the adjustment parameter is set to 1.

[0089] In some embodiments, the data similarity between the text keyword and the text document is calculated based on the BM25 score function, where the BM25 score formula is as follows:

[0090]

[0091] In formula (3), score(D,Q) is the data similarity between text keywords and text documents, D is the data query statement, Q is the text document, and f(q i ,D) is the occurrence frequency of the ith keyword in the text document, |D| is the text length of the text document, avgdl is the average text length of the text documents in the preset data pool, k1 and b are both preset parameters, where parameter k1∈[1.2,2.0], parameter b=0.75.

[0092] Optionally, the target strategy corresponding to the retriever execution step includes: executing a vector similarity retriever, wherein the vector similarity retriever includes performing vector similarity calculations on data nodes in a preset knowledge base according to a data query statement to obtain a similarity index, and determining retrieval results from the data nodes based on the similarity index, wherein the similarity index includes at least one of cosine similarity, Euclidean distance, and data dot product.

[0093] In some embodiments, the vector similarity retriever includes: performing cosine similarity calculation based on the text vector corresponding to the data query statement and the text vectors between data nodes to obtain data similarity.

[0094] In some embodiments, the vector similarity retriever includes: performing Euclidean distance calculation based on the text vector corresponding to the data query statement and the text vectors between data nodes to obtain data similarity.

[0095] In some embodiments, the vector similarity retriever includes: performing dot product calculation based on the text vector corresponding to the data query statement and the text vectors between data nodes to obtain data similarity.

[0096] Optionally, the target strategy corresponding to the retriever execution step includes: executing a relational database retriever, wherein the relational database retriever includes generating a database search statement according to a data query statement, and searching from a preset relational database through the database search statement to obtain a search result.

[0097] Optionally, the target strategy corresponding to the retriever execution step includes: executing a graph database retriever, wherein the graph database retriever includes searching from a preset graph database according to a data query statement to obtain a search result.

[0098] Optionally, the target strategy corresponding to the retriever execution step includes: executing a search engine retriever, wherein the search engine retriever includes inputting a data query statement into a preset search engine so that the preset search engine outputs a search result.

[0099] Optionally, the target step further includes a node post-processing step, and the execution order of the node post-processing step is after the retriever execution step.

[0100] In some embodiments, the main purpose of the node post-processing step is to further process and optimize the search results obtained by retrieval to improve the quality, relevance and diversity of the final output results.

[0101] Optionally, the target strategy corresponding to the node post-processing step includes: a deduplication strategy, including respectively calculating hash values ​​according to multiple search results output by the retriever execution step, and deduplicating each search result according to the comparison results between the hash values.

[0102] Optionally, the target strategy corresponding to the node post-processing step includes: a re-ranking strategy, including re-ranking multiple retrieval results output by the retriever execution step through a Rerank model.

[0103] Optionally, the target strategy corresponding to the node post-processing step includes: a K-means strategy, including using the K-means algorithm to calculate the multiple retrieval results output by the retriever execution step, and determining abnormal nodes from each retrieval result based on the calculated difference value, and deleting the abnormal nodes from each retrieval result.

[0104] In some embodiments, before executing the K-means strategy, duplicate search results are first removed using a deduplication strategy, and then the search results are clustered into two categories using the K-means algorithm. The average scores corresponding to the two categories are calculated, and the category with the larger average score is selected.

[0105] Optionally, the target strategy corresponding to the node post-processing step includes: an average value strategy, including scoring the multiple search results output by the retriever execution step respectively to obtain the node scores corresponding to each search result, and calculating according to the node scores corresponding to each search result to obtain the score average value, determining abnormal nodes from each search result according to the comparison results between the score average value and each node score, and deleting abnormal nodes from each search result.

[0106] Optionally, the retrieval step includes a result integration step, and the target integration strategy is any execution strategy corresponding to the result integration step.

[0107] Optionally, the target integration strategy integrates the search results in the following manner: Extend strategy, including obtaining a prompt template by splicing multiple search results.

[0108] In some embodiments, the search results corresponding to each independent search engine are put into a list for splicing, and then deduplication is performed to obtain a prompt template.

[0109] Optionally, the target integration strategy integrates the search results in the following ways: a reciprocal rank fusion strategy, including fusing multiple search results using a preset reciprocal rank fusion algorithm (RRF, Reciprocal Rank Fusion) to obtain a prompt template;

[0110] In some embodiments, the search results corresponding to each independent search engine are fused through a reciprocal ranking fusion algorithm, where the working principle of the reciprocal ranking fusion algorithm is to obtain search results from multiple methods, assign a reciprocal ranking score to each document in the results, and then combine these scores to create a new ranking. The reciprocal ranking fusion algorithm can take into account the position of the item in the original ranking and give higher importance to items ranked higher in multiple lists, which helps to improve the overall quality and reliability of the final ranking.

[0111] Optionally, the target integration strategy integrates the search results in the following manner: Intersection merging strategy, including determining a node intersection according to multiple search results, and merging the node intersection to obtain a prompt template.

[0112] Optionally, before integrating the search results, the target integration strategy screens the search results in the following manner: performing deduplication processing on the search results in the prompt template according to the hash values ​​corresponding to the search results in the prompt template.

[0113] Optionally, before integrating the search results, the target integration strategy screens the search results in the following manner: after integrating the search results into a prompt template corresponding to the data query statement, the method further includes: re-ranking the search results in the prompt template through a Rerank model.

[0114] Optionally, before integrating the search results, the target integration strategy filters the search results in the following manner: after integrating the search results into a prompt template corresponding to the data query statement, the method also includes: using a K-means algorithm to calculate the search results in the prompt template to filter the search results in the prompt template according to the calculation results.

[0115] Optionally, before integrating the search results, the target integration strategy screens the search results in the following manner: scoring the search results in the prompt template, and filtering the search results in the prompt template according to the scoring results.

[0116] Optionally, after formulating multiple independent search engines according to the search strategy material library, the method also includes: obtaining query statement samples and obtaining sample labels corresponding to the query statement samples; executing each independent search engine according to the query statement samples to obtain sample search results corresponding to each independent search engine; performing similarity calculations on each sample search result according to the sample labels to determine the search scores corresponding to each independent search engine; and filtering each independent search engine according to the search scores.

[0117] In some embodiments, a traversal is performed according to the search strategy material library to obtain all independent search engines that can be combined; 1,000 query statements are extracted from the historical input data of the target big data as query statement samples, and each independent search engine is executed for each query statement sample to determine the sample search results corresponding to each independent search engine; the search score corresponding to each independent search engine is determined by similarity calculation, user scoring, etc.; all search scores corresponding to the same independent search engine are calculated to obtain an average search score; and 3 to 5 independent search engines with the highest average search scores are selected as qualified strategies.

[0118] Optionally, multiple target engines are determined from each independent search engine based on the analysis results, and a target integration strategy is determined from the search strategy material library, including: obtaining a training sample set, wherein the training sample set includes multiple query statement types and type samples corresponding to each query statement type; obtaining the search scores corresponding to each independent search engine in different query statement types based on the training sample set, so as to determine multiple type engines corresponding to each query statement type from each independent search engine; extracting the current statement type corresponding to the data query statement from the analysis results; determining multiple target engines from each independent search engine based on the matching results between the current statement type and each query statement type, and matching them from the search strategy material library based on the current statement type to obtain a target integration strategy.

[0119] In some embodiments, the current statement type includes scenarios requiring a large number of search results, scenarios focusing on the order of search results, scenarios requiring precise search results, etc.

[0120] In some embodiments, different execution strategies are adapted to different types of query statements, as follows: the multi-query strategy is particularly effective in processing complex queries or requiring more extensive contextual information; the cosine similarity strategy is suitable for high-dimensional spatial data, and in natural language processing tasks such as text classification and document similarity calculation, when the data is represented as a bag-of-words model, the cosine similarity strategy has outstanding effects; the node post-processing step is crucial to improving the overall performance of the RAG (Retrieval-Augmented Generation) system; the deduplication strategy and the rearrangement strategy are suitable for combined retrieval and can improve the accuracy of query combinations; the K-means strategy is suitable for scenarios with multiple scores close to 1 and multiple scores close to 0; the average strategy is suitable for scenarios with multiple scores close to 1 and multiple scores close to 0, or scenarios with relatively dispersed score distribution.

[0121] In some embodiments, based on the different applicable scenarios for each execution strategy, the applicable scenarios are applied to the query statement type; the qualified strategies corresponding to the query statement type are determined through multiple statement samples corresponding to the same query statement type; when the current statement type corresponding to the data query statement is obtained, the qualified strategies corresponding to the data query statement are determined through the current statement type, so as to retrieve the retrieval results according to the qualified strategies.

[0122] In some embodiments, a current statement type corresponding to a data query statement is obtained; a target integration strategy is determined from an execution strategy corresponding to a result integration step according to the current statement type; and search results corresponding to each independent search engine are merged through the target integration strategy to obtain a prompt template.

[0123] In some embodiments, the Extend strategy is applicable to scenarios that require a large number of search results; the inverse sorting fusion algorithm is applicable to scenarios that focus on the order of search results; and the Intersection merging strategy is applicable to scenarios that require precise search results.

[0124] Combination Figure 6 As shown, the embodiment of the present disclosure provides a knowledge base retrieval system applied to a large language model, including a setting module 601, an acquisition module 602, a determination module 603 and a retrieval module 604.

[0125] The setting module 601 is used to set a search strategy material library and formulate multiple independent search engines according to the search strategy material library, wherein the search strategy material library includes one or more search steps and multiple execution strategies corresponding to each search step;

[0126] The acquisition module 602 is used to acquire data query statements;

[0127] The determination module 603 is used to parse the data query statement to obtain the parsing result, determine multiple target engines from each independent search engine according to the parsing result, and determine the target integration strategy from the search strategy material library;

[0128] The retrieval module 604 is used to execute each target engine in parallel according to the data query statement, obtain the retrieval results output by each target engine, and filter and integrate each retrieval result through the target integration strategy to obtain a prompt template as the input of the large language model, wherein the target engine is used to perform knowledge base retrieval according to the data query statement

[0129] The knowledge base retrieval system applied to the large language model provided by the embodiment of the present disclosure is adopted. A plurality of independent retrieval engines are formulated through the retrieval strategy material library, and the data query statements are parsed to obtain the parsing results, so as to determine the target engine and the target integration strategy from the independent retrieval engine and the retrieval strategy material library respectively through the parsing results, thereby executing each target engine in parallel, and integrating each retrieval result into a prompt template through the target integration strategy, so as to serve as the input of the large language model. In this way, by refining the retrieval process into a plurality of different steps, and formulating a corresponding execution strategy for each step, and then constructing a plurality of independent retrieval engines, and using various independent retrieval engines to retrieve the retrieval results matching the question, not only the automatic generation of independent retrieval engines is realized, but also the problem of information omission caused by a single retrieval strategy is overcome through diversified retrieval strategies, thereby improving the accuracy of model query.

[0130] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation can be changed. Parts and sub-samples of some embodiments may be included in or replace parts and sub-samples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0132] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0133] The flowchart and block diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A knowledge base retrieval method applied to a large language model, characterized in that: include: Pre-setting a search strategy material library, and formulating multiple independent search engines based on the search strategy material library, wherein the search strategy material library includes one or more search steps, and multiple execution strategies corresponding to each of the search steps; Get data query statement; By parsing the data query statement, a parsing result is obtained, and a plurality of target engines are determined from each of the independent search engines according to the parsing result, and a target integration strategy is determined from a search strategy material library; Execute each of the target engines in parallel according to the data query statement to obtain the retrieval results output by each of the target engines respectively, and filter and integrate each of the retrieval results through the target integration strategy to obtain a prompt template as an input of the large language model, wherein the target engine is used to perform knowledge base retrieval according to the data query statement.

2. The method according to claim 1, characterized in that: Create an independent search engine by: Determine a target step from each of the retrieval steps, and determine a target strategy from the execution strategies corresponding to the target steps; The target steps are arranged in a preset execution order to form an independent search engine composed of the target steps and the target strategy.

3. The method according to claim 2, characterized in that The target step includes a retriever execution step, wherein the target strategy corresponding to the retriever execution step includes at least one of the following: Executing a vector similarity retriever, wherein the vector similarity retriever includes performing vector similarity calculation on data nodes in a preset knowledge base according to the data query statement to obtain a similarity index, and determining a search result from the data nodes according to the similarity index, wherein the similarity index includes at least one of cosine similarity, Euclidean distance, and data dot product; Executing a keyword frequency retriever, wherein the keyword frequency retriever includes obtaining text keywords in the data query statement and obtaining text documents in a preset knowledge base; obtaining a total number of documents by counting the number of text documents, and obtaining the number of key documents by counting the number of text documents containing the text keywords; obtaining an inverse document frequency weight corresponding to the text keyword by calculating according to the total number of documents and the number of key documents using a preset inverse document frequency formula, wherein the inverse document frequency formula is provided with an adjustment parameter so that the inverse document frequency weight is not equal to 0; calculating the frequency of occurrence of the text keyword in the text document according to the inverse document frequency weight to obtain a text similarity between the text keyword and the text document, and determining a search result from the text document according to the text similarity; Executing a relational database searcher, wherein the relational database searcher includes generating a database search statement according to the data query statement, and searching from a preset relational database through the database search statement to obtain a search result; Executing a graph database retriever, wherein the graph database retriever includes searching from a preset graph database according to the data query statement to obtain a search result; Execute a search engine retriever, wherein the search engine retriever includes inputting the data query statement into a preset search engine, so that the preset search engine outputs a search result.

4. The method according to claim 3, characterized in that The target step further includes a query statement expansion step, and the query statement expansion step is executed before the retriever execution step, wherein the target strategy corresponding to the query statement expansion step includes at least one of the following: A hypothetical document strategy includes inputting a preset document prompt word and the data query statement into a preset large language model, so as to guide the large language model to output a hypothetical document corresponding to the data query statement through the document prompt word, and updating the data query statement according to the hypothetical document, wherein the document prompt word includes an application environment corresponding to the data query statement; A multi-query strategy includes obtaining question prompt words, and inputting the question prompt words and the data query statement into a preset large language model, so as to guide the large language model to output multiple sub-questions corresponding to the data query statement through the question prompt words, merge each of the sub-questions and the data query statement to obtain a sub-question list, and update the data query statement according to the sub-question list.

5. The method according to claim 3, characterized in that: The target step further includes a node post-processing step, and the execution order of the node post-processing step is located after the retriever execution step, wherein the target strategy corresponding to the node post-processing step includes at least one of the following: A deduplication strategy, comprising respectively calculating hash values ​​according to the plurality of search results outputted by the retriever execution step, and deduplicating each of the search results according to the comparison results between the hash values; A re-ranking strategy, comprising re-ranking the multiple search results output by the retriever execution step through a Rerank model; A K-means strategy includes calculating the multiple search results output by the retriever execution step using a K-means algorithm, determining an abnormal node from each of the search results according to the calculated difference value, and deleting the abnormal node from each of the search results; The average value strategy includes scoring the multiple search results output by the retriever execution step respectively to obtain the node scores corresponding to each of the search results, and calculating according to the node scores corresponding to each of the search results to obtain the score average value, determining abnormal nodes from each of the search results according to the comparison results between the score average value and each of the node scores, and deleting the abnormal nodes from each of the search results.

6. The method according to claim 1, characterized in that The target integration strategy integrates the search results in at least one of the following ways: The Extend strategy includes obtaining a prompt template by splicing multiple search results; A reciprocal sorting fusion strategy includes fusing multiple search results using a preset reciprocal sorting fusion algorithm to obtain a prompt template; The Intersection merging strategy includes determining the node intersection according to multiple search results, and merging the node intersection to obtain a prompt template.

7. The method according to claim 6, characterized in that Before integrating the search results, the target integration strategy screens the search results in at least one of the following ways: Deduplication processing is performed on the search results in the prompt template according to the hash values ​​corresponding to the search results in the prompt template; Reranking the search results in the prompt template by using a Rerank model; Calculating each search result in the prompt template using a K-means algorithm, so as to filter each search result in the prompt template according to the calculation result; Scoring each search result in the prompt template, and filtering each search result in the prompt template according to the scoring result.

8. The method according to any one of claims 1 to 7, characterized in that: After formulating a plurality of independent search engines according to the search strategy material library, the method further comprises: Obtain a query statement sample and obtain a sample label corresponding to the query statement sample; Execute each of the independent search engines according to the query statement sample to obtain sample search results corresponding to each of the independent search engines; Calculating the similarity of each of the sample search results according to the sample labels to determine the search scores corresponding to each of the independent search engines; Each of the independent search engines is filtered according to the search score.

9. The method according to claim 8, characterized in that Determining a plurality of target engines from each of the independent search engines according to the analysis results, and determining a target integration strategy from a search strategy material library, including: Acquire a training sample set, wherein the training sample set includes multiple query statement types and type samples corresponding to each query statement type; Obtaining the search scores of each of the independent search engines corresponding to different query statement types according to the training sample set, so as to determine a plurality of type engines corresponding to each query statement type from each of the independent search engines; Extracting the current statement type corresponding to the data query statement from the parsing result; According to the matching results between the current statement type and each of the query statement types, a plurality of target engines are determined from each of the independent search engines, and matching is performed from a search strategy material library according to the current statement type to obtain a target integration strategy.

10. A knowledge base retrieval system applied to a large language model, characterized in that: include: A setting module, used to set a search strategy material library, and formulate multiple independent search engines according to the search strategy material library, wherein the search strategy material library includes one or more search steps, and multiple execution strategies corresponding to each of the search steps; The acquisition module is used to obtain data query statements; A determination module, used to parse the data query statement to obtain a parsing result, determine a plurality of target engines from each of the independent search engines according to the parsing result, and determine a target integration strategy from a search strategy material library; A retrieval module is used to execute each of the target engines in parallel according to the data query statement, obtain the retrieval results output by each of the target engines, and filter and integrate each of the retrieval results through the target integration strategy to obtain a prompt template as an input to the large language model, wherein the target engine is used to perform knowledge base retrieval according to the data query statement.

Images