Multi-language model combined active retrieval enhancement method and system

Abstract

The invention discloses a multi-language model combined active retrieval enhancement method and system, and relates to the technical field of data retrieval, and the method comprises the following steps: obtaining a plurality of to-be-retrieved questions; building corresponding prompt templates for the plurality of black-box large language models, and inputting the to-be-retrieved question and the prompt templates into the corresponding black-box large language models to obtain analysis results of different black-box large language models on the to-be-retrieved question; when the multiple analysis results are different, the corresponding black box large language model is asked through the prompt template, a prompt of a certain analysis result error reason is obtained, the prompt and the prompt template are spliced, and a final document is obtained; and re-inputting the final document into the corresponding black box large language model, and performing answer retrieval. According to the method, an active retrieval strategy is adopted, retrieval is carried out only when the model needs knowledge, and the expenditure caused by unnecessary retrieval and the influence of low-quality retrieval results on a final generation result are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and in particular to an active retrieval enhancement method and system combining multiple language models. Background Art

[0002] In the field of natural language processing (NLP), retrieval-augmented generation (RAG) technology cleverly combines the power of information retrieval and neural network generation models. By introducing relevant external information in the generation process, it can achieve accurate, diverse and contextual text generation based on large-scale databases.

[0003] When the concept of retrieval enhancement first emerged, most people preferred to fine-tune the generative model to improve the model accuracy, such as Med-PaLM. However, due to the high cost of fine-tuning due to the large number of model parameters, the concept of black-box RAG was created, which treats the generator as a black box (freezing the parameters of the large model) and expands it without fine-tuning. It has been successful in knowledge-intensive tasks and has received widespread attention.

[0004] In the existing technology, researchers fine-tune the retriever based on the "generation habits" of the current retrieval enhancement model (whether the large model will answer the correct answer based on the retrieved documents), and search all questions through the fine-tuned retriever. This method of searching all questions will reduce the accuracy of the retrieval, and due to the complex content of the external knowledge base, the retrieved knowledge may not be very relevant to the question, which will mislead the generator to generate wrong answers, resulting in unnecessary overhead and low-quality results during retrieval. Summary of the invention

[0005] The purpose of the present invention is to provide an active retrieval enhancement method and system combined with a multi-language model to address the deficiencies of the above-mentioned prior art, so as to solve the problem that a lot of knowledge in the prior art can be obtained from a large model rather than retrieved, and when the external knowledge base is very large, the time overhead caused by the retrieval may be huge, and the retrieved results are complicated and have low relevance, resulting in unnecessary overhead and low-quality results during retrieval.

[0006] The present invention specifically provides the following technical solutions:

[0007] An active retrieval enhancement method combining multi-language models includes the following steps:

[0008] Get the questions to be searched;

[0009] Construct corresponding prompt templates for multiple different types of black box large language models, and input the to-be-searched questions and the corresponding prompt templates into the corresponding black box large language models to obtain multiple analysis results of the to-be-searched questions by different black box large language models; wherein the prompt template actively performs a self-check on whether the black box large language model needs knowledge according to the to-be-searched questions;

[0010] When the analysis results are different, the corresponding black box large language model is questioned through the prompt template to obtain a prompt of the reason why a certain analysis result is wrong, and the prompt is spliced ​​with the prompt template to obtain the final document;

[0011] The final document and the question to be retrieved are re-input into the corresponding black box large language model to perform answer retrieval.

[0012] Preferably, the plurality of black box large language models include the Qianfan large model ERNIE-3.5 and the generative pre-trained model GPT-3.5-Turbo-0613, wherein ERNIE-3.5 covers a number of Chinese-related knowledge, and GPT-3.5-Turbo-0613 contains a number of English and English scene-related knowledge.

[0013] Preferably, obtaining a prompt of the reason for an error in a certain analysis result and splicing the prompt with a prompt template to obtain a final document includes:

[0014] Based on the prompt, the black box large language model gives the reasons why the specific options in the rhetorical question are wrong;

[0015] Processing the error and removing the error from the prompt template;

[0016] The retrieved knowledge and the reasons given after the rhetorical question are spliced ​​together as prompts to obtain the final document.

[0017] Preferably, the method further comprises:

[0018] The BM25 retrieval method is used to search in the knowledge base to obtain knowledge with a prompt similarity higher than a threshold value obtained by the reverse question, and the knowledge with a similarity higher than the threshold value is used as the expanded prior knowledge input into the black box large language model.

[0019] The present invention provides an active retrieval enhancement system combining multiple language models, comprising:

[0020] A collection module, used to obtain questions to be searched;

[0021] The analysis module is used to construct corresponding prompt templates for multiple different types of black box large language models, and input the to-be-searched questions and the corresponding prompt templates into the corresponding black box large language models to obtain multiple analysis results of the different black box large language models for the search questions; wherein the prompt template actively performs a self-check on whether the black box large language model needs knowledge according to the to-be-searched questions;

[0022] A self-checking splicing module is used to, when the analysis results are different, to ask the corresponding black box large language model through a prompt template to obtain a prompt of the reason why a certain analysis result is wrong, and to splice the prompt with the prompt template to obtain a final document;

[0023] The output module is used to re-input the final document and the question to be retrieved into the corresponding black box large language model to perform answer retrieval.

[0024] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the active retrieval enhancement method combined with a multi-language model.

[0025] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the active retrieval enhancement method combined with multiple language models are implemented.

[0026] Compared with the prior art, the present invention has the following significant advantages:

[0027] The present invention adopts a plurality of different types of black box large language models, and sets corresponding prompt templates by combining the advantages of each black box large language model. According to the question to be retrieved, the corresponding language model is actively self-checked through the corresponding prompt template, so as to realize retrieval only when the corresponding black box large language model needs additional knowledge, thereby reducing the overhead caused by unnecessary retrieval and the influence of low-quality retrieval results on the final generated results. At the same time, when the analysis results are different, the corresponding black box large language model is questioned in reverse, and the prompts and prompt templates obtained by the reverse questioning are used as the final document for re-retrieval, so that the timing and content of the retrieval can be adjusted according to whether the answer of the black box large language model is correct or not, thereby improving the relevance of the retrieval knowledge to the question, and at the same time improving the retrieval efficiency of the question and the quality of the generated answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is an overall flow chart of an active retrieval enhancement method combined with a multi-language model;

[0029] Figure 2 This is the prompt template diagram used by ERNIE-3.5 in the present invention;

[0030] Figure 3 This is the prompt template image used by GPT-3.5-turbo in the present invention;

[0031] Figure 4 This is the template image used by GPT-3.5-turbo when the answer verification fails in the present invention;

[0032] Figure 5 A prompt template diagram for outputting a final answer in a standardized manner in the present invention;

[0033] Figure 6 This is a format diagram of the preprocessed data set in the present invention. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] Based on the existing technology, an idea is proposed: Can we design a quick and simple retrieval enhancement process, actively decide whether to use retrieval means in the process, and how to better obtain texts that are more relevant to the question from the large model itself to improve the final output effect. The assumption for defining the timing of retrieval is: only when the results output by the model may be "unreliable" will relevant knowledge be retrieved to avoid unnecessary and inappropriate retrieval. Experiments also show that unnecessary retrieval may affect the results. As for what kind of knowledge can best improve the accuracy of the results, it is believed that a prompt template can be designed to let the model "self-check" and the model itself answer what kind of relevant knowledge is lacking, and then actively retrieve this kind of relevant knowledge to obtain external knowledge that is more helpful for generating results. Therefore, a multi-model combined active retrieval enhancement framework is proposed. In this work, a new retrieval enhancement language model framework is proposed, in which the generative models are regarded as black boxes. The entire retrieval framework is regarded as a plug-and-play module that can be adjusted at any time. When a specific question is given, the framework will combine the answers of the two major mainstream language models, GPT-3.5-trubo and ERNIE-3.5, to evaluate whether it is necessary to retrieve relevant knowledge. The retrieval part is also different from the traditional method. The model "self-reflects" on the domain knowledge that it may lack in response to the problem, and then uses the keywords of this type of domain knowledge as query conditions to search the external knowledge base, so that the retrieved knowledge is relevant to the field, thereby achieving better retrieval results.

[0036] The proposed retrieval enhancement process framework is applicable to most existing large models and does not require additional fine-tuning. Since the current performance of GPT-3.5 is relatively superior and balanced in all aspects, the performance of GPT-3.5 under the traditional retrieval enhancement method is compared. The test set uses five sub-datasets in the open source dataset MMLU, which is used to test the model's ability to understand the world and solve problems. The five subsets involve five different fields, including objective knowledge, law, ethics, history, world geography, etc. The experimental data shows that the performance of the retrieval enhancement process framework is advantageous or competitive, which proves the effectiveness and generalizability of the proposed method.

[0037] like Figure 1 As shown, the present invention provides a multi-language model combined active retrieval enhancement method to be described below, which specifically includes the following steps:

[0038] Step S1: Obtain the question to be retrieved.

[0039] Step S2: construct corresponding prompt templates for multiple different types of black box large language models, and input the questions to be retrieved and the corresponding prompt templates into the corresponding black box large language models to obtain multiple analysis results of the questions to be retrieved by different black box large language models; wherein the prompt template actively performs self-check on whether the black box large language model needs knowledge according to the questions to be retrieved.

[0040] Multiple black-box large language models include the Qianfan large model ERNIE-3.5 and the generative pre-trained model GPT-3.5-Turbo-0613. ERNIE-3.5 covers a number of Chinese-related knowledge, and the pre-training data sets include Chinese Wikipedia, Baidu Encyclopedia and other data sets. It can be found that ERNIE-3.5 is mainly used in scenes related to Chinese. GPT-3.5-Turbo-0613 contains a number of English and English scene-related knowledge. GPT-3.5-Turbo is the fourth in a series of language models designed by OPENAI. The pre-training data sets include English Wikipedia and blogs.

[0041] And in the "Large Language Model Comprehensive Performance Evaluation Report", it is clearly pointed out that Wenxin Yiyan has outstanding semantic understanding ability, especially better Chinese comprehension ability, better understanding of Chinese culture, strong timeliness, and subtle grasp of content security. And the reasons for choosing these two models are: first, because the accuracy of the two models on many data sets has reached SOTA, and second, it is believed that the capabilities of the two models can just form a kind of "complementarity" of Chinese and foreign knowledge. In actual application, there will definitely be situations where the problem involves both Chinese and foreign knowledge. In the traditional mode, due to the limitation of the data set, the accuracy of the answer given by a single model is definitely limited. In the process, the same question will be asked to the two models separately, and the answers of the two will be analyzed.

[0042] Obtain the knowledge that the black-box large language model lacks for the problem, and use retrieval methods to retrieve the text required by the black-box large language model.

[0043] This step is the specific process of using the active retrieval strategy, specifically:

[0044] The fundamental reason for proposing this strategy is that low-quality external knowledge obtained through unnecessary retrieval may affect the accuracy of the model generation results, causing it to give wrong answers. In addition, retrieving content from a huge external knowledge base will incur a huge time overhead. The larger the number of questions, the greater the time overhead. In the work, the retrieval system is selectively called according to the answers of the two models ERNIE-3.5 and GPT-3.5-Turbo-0613, and no retrieval is performed on every question. Accordingly, with this dynamic retrieval method, the gap with the traditional method becomes more obvious when facing a larger knowledge base and a larger number of questions. Inspired by Zhengbao et al., the content of the retrieval is very different from the traditional mode of retrieving relevant knowledge for questions.

[0045] The traditional retrieval enhancement method is to directly search for the question, obtain relevant knowledge from the external knowledge base, and then send it to the generation model to obtain the answer after splicing it with the question. The work of this invention is to introduce prompt learning, by designing a set of prompt templates. The purpose of designing prompt templates is to let the large model simulate the human thinking process, thereby stimulating the potential ability of the large model, so that the model can first give an analysis and understanding of the problem and point out the problems with the answer. In this way, the relevant knowledge that the model lacks for the problem can be found, and then the retrieval means can be used to retrieve the text that is most helpful to the model, thereby maximizing the retrieval efficiency and accuracy.

[0046] Step S3: When multiple analysis results are different, the corresponding black box large language model is questioned through the prompt template to obtain a prompt of the reason why a certain analysis result is wrong, and the prompt is spliced ​​with the prompt template to obtain the final document, otherwise the final answer is obtained directly.

[0047] The decision on whether to ask a counter-question is based on the results of the dual-model (two black-box large language models) verification. If the results are different, then ask a counter-question. If they are the same, then directly output the answer. The counter-question means asking the large model why option "x" (in the analysis result) is wrong for this question, so as to obtain more knowledge related to the question.

[0048] Among them, judging whether a question needs to be asked in reverse by comparing multiple analysis results and obtaining a prompt when asking in reverse, includes the following steps:

[0049] Determine whether the analysis results of different black box large language models for the question are the same. If different, enter the counter-questioning phase.

[0050] In the questioning phase, the (GPT-3.5-Turbo-0613) black box language model is questioned using the hint template constructed manually in advance. The black box language model generates a reason for thinking that the option is wrong based on the hint constructed by the hint template. This reason is also considered as a kind of external knowledge and is used as the final hint.

[0051] When asking questions, you can get the reason why a certain analysis result is wrong, and combine the prompt with the prompt template to get the final document, including:

[0052] Based on the prompts, the black-box large language model gives the reasons why the specific options in the rhetorical question are wrong.

[0053] Handle the error and remove it from the prompt template.

[0054] The retrieved knowledge and the reasons given after the rhetorical question are spliced ​​together as prompts to obtain the final document.

[0055] The method also includes: using the BM25 retrieval method to search in the knowledge base, obtaining knowledge whose prompt similarity with the obtained question is higher than a threshold, and using the knowledge whose similarity is higher than the threshold as expanded prior knowledge input into the black box large language model.

[0056] This step is the process of combining prompt learning to give full play to the advantages of retrieval enhancement, specifically:

[0057] Carefully constructed prompt templates are a key tool for improving the performance of large language models. Not only can they significantly stimulate the inherent potential of the model, but also, as verified by a series of experiments, these templates play a vital role in enhancing the model's ability to integrate external information. The improvement of this ability is directly reflected in the generation efficiency of the model, achieving a qualitative leap. In the present invention, special attention is paid to the design of prompt templates so that they can guide the model to "self-reflect" and identify the knowledge gaps faced when answering specific questions. This self-identification mechanism is crucial because it provides a clear direction for the model to guide it to retrieve the precise information fragments it needs from a wide range of external knowledge sources. In this way, the present invention can effectively solve the knowledge limitations that the model may encounter when facing specific problems, while ensuring that the retrieved information is accurate and reasonable. In addition, the experiments of the present invention also revealed an interesting phenomenon:

[0058] When generating answers, the model often provides multiple candidate answers, which may include the correct answer. However, due to the so-called "hallucination effect", the model may sometimes mistakenly regard the wrong answer as correct. To address this problem, a dual-model verification system is introduced, which improves the accuracy of the answers through cross-validation. In addition, the prompt template also includes a feedback mechanism that allows the model to take a second look at the generated answers, giving it a chance to correct the possible wrong answers generated initially. The prompt template is divided into four parts, each of which targets a specific stage or requirement in the model generation process, such as Figure 2 to Figure 5 shown.

[0059] exist Figure 4 In the prompt part shown, GPT-3.5-turbo will clearly give the reason why a specific option is wrong. It is found that if a certain option is clearly pointed out as wrong, it will mislead the model to exclude the option. If the option is correct, it will cause a loss of accuracy. Therefore, the reasons given by GPT-3.5-turbo are processed, and sentences such as The answer is not A / B / C / D. are removed from the prompt template to avoid affecting the final judgment of the model. Figure 5 The prompt part shown concatenates the retrieved knowledge and the reasons given by the model after "reflection" as prompts, and then re-inputs them into the big model together with the questions to obtain the final output result.

[0060] Step S4: re-input the final document and the question to be retrieved into the corresponding black box large language model to perform answer retrieval.

[0061] The prior knowledge obtained above is integrated into the manually constructed prompt template, and finally questions are asked to the black box large language model to obtain the final result.

[0062] The following is verified by experiment:

[0063] Testing of the method. Without any samples, the method of the present invention shows improved performance compared with various large-scale models. This result not only verifies the feasibility of the method of the present invention, but also demonstrates its wide applicability.

[0064] Dataset: MMLU (Massive Multi-Task Language Understanding) is a new benchmark that aims to measure the knowledge acquired during pre-training by evaluating models only in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to the way humans are evaluated. The benchmark covers 57 subjects in fields such as STEM, humanities, and social sciences. Its difficulty ranges from beginner to advanced professional level, and it tests both world knowledge and problem-solving ability. Due to issues such as the use of API overhead by black-box models, the experiments of this invention select five sub-datasets, which cover five different fields, such as: entity knowledge, law, ethics, history, and geography. Preprocess the dataset into a more substantial form, such as Figure 6 shown.

[0065] Experimental setup:

[0066] The baselines of the present invention are selected in three groups. The first group is advanced large language models, such as LLAMA2-34B (5-shot), gpt-text-davinci-003, etc. The second group of baselines consists of retrieval-enhanced language models, and only Atlas (5-shot) is selected in this group. In view of Atlas's simultaneous training strategy for retrievers and language models, it is regarded as a retrieval-enhanced white-box language model. The third group of baselines are the top two on the zero-sample accuracy list of the MMLU dataset, GAL-120B and GPT-NeoX-20B.

[0067] The research in this paper focuses on applying the carefully designed framework to two models, ERNIE-3.5 and GPT-3.5-Turbo-0613.

[0068] In this process, the external knowledge base wiki.train.raw is used as an information source. The first step of the operation is to perform a search judgment. Once it is determined that a search operation needs to be performed, the preset prompt template will be triggered and a second question will be asked to obtain the explanation after the model's self-reflection. Subsequently, the BM25 algorithm is used to retrieve these explanations in the external knowledge base, and the search results are integrated with the original question and the first explanation, and passed to the model as input, ultimately obtaining an accurate and detailed answer.

[0069] The experimental results are shown in Table 1. The experimental results are all from the official list of the MMLU dataset. The first three baseline models, including LLAMa2, gpt-text-davinci-003, and Atlas, are 5-shot tests, and the Atlas model proposed by Meta is also a retrieval enhancement model. The last two baseline models, including GAL and GPT-NeoX, are zero-shot tests and are the top two in the official list of the MMLU dataset.

[0070] The experimental results of the present invention and the data of each large model based on the MMLU official list, through comparative analysis, it can be clearly seen that the performance of the model of the present invention is significantly better than other large models.

[0071] In the test of zero-sample learning, the strategy of the present invention showed particularly outstanding improvement. Even in the test of a small number of samples, the model of the present invention achieved obvious performance enhancement.

[0072] Table 1 Experimental results of mainstream large models and our work on the dataset

[0073] Models Parameter Shot Accuracy / % LLAMa2 34B 5 62.6 gpt-text-davinci-003 175B 5 64.8 Atlas 11B 5 47.9 ChatGPT - 0 53.2 GAL 120B 0 52.6 GPT-NeoX 20B 0 28.6 Ours - 0 66.1

[0074] Ablation experiment:

[0075] The ablation experiment results are shown in Table 2.

[0076] Table 2 Ablation experiment results

[0077]

[0078] The innovation of the present invention is:

[0079] First, by combining the retrieval enhancement method of prompt learning and designing clever prompt templates, the model can self-reflect on the knowledge needed to solve the problem, thereby guiding the retrieval system to obtain the most relevant external information, further improving the efficiency and accuracy of the retrieval. Secondly, by adopting a dual-model verification system, in the answer verification stage, the answers of the two models are used to verify each other, which effectively reduces the errors that may be caused by a single model and improves the reliability of the answer. Finally, the process framework of the present invention effectively integrates the retrieval knowledge, and when combining the retrieved knowledge with the model answer, the reasons given by GPT-3.5-Turbo are processed to avoid misleading the model to exclude the correct option and improve the accuracy of the answer selection.

[0080] During the ablation experiment, it was observed that when it came to the world knowledge dataset, the two major models, ERNIE-3.5 and Gpt-turbo-3.5-0613, both showed unexpectedly low accuracy, which was even lower than the level of random guessing. After in-depth analysis of the reasons, it was found that too much retrieval knowledge led to incorrect retrieval content, which in turn had a negative impact on the final accuracy of the model. In addition, the experimental data of the present invention also shows that the retrieval resource overhead for each question is huge, and if this step is performed for each question, it will undoubtedly bring huge time costs. However, it can be found from Table 3 that the method of the present invention requires a significant reduction in the number of retrievals required, and in the process, a higher accuracy rate is also achieved.

[0081] Table 3 Number of topics that need retrieval enhancement

[0082]

[0083] Experimental summary:

[0084] The present invention verifies the effectiveness of the proposed framework in zero-shot learning scenarios through experiments, but there are still some limitations. For example, for certain knowledge in specific fields, the external knowledge base may not provide sufficient information support. In addition, the self-reflection ability of the model still needs to be further improved to produce more accurate and comprehensive explanatory text.

[0085] Ablation experiments further verified the effectiveness of the proposed framework, which not only successfully reduced the retrieval time but also improved the accuracy. In the test on the world knowledge dataset, the framework of the present invention significantly outperformed the results of the retrieval enhancement ERNIE-3.5 and GPT-3.5-Turbo models, proving the effectiveness of combining the advantages of multiple models and active retrieval strategies.

[0086] Based on the above method, the present invention provides an active retrieval enhancement system combining multiple language models, including: a collection module, an analysis module, a self-checking splicing module and an output module.

[0087] Among them, the collection module is used to obtain the questions to be retrieved; the analysis module is used to construct corresponding prompt templates for multiple different types of black box large language models, and compare the questions to be retrieved with the corresponding black box large language models to obtain multiple analysis results of different black box large language models for the retrieval questions; the prompt template actively self-checks whether the black box large language model needs knowledge based on the questions to be retrieved; the self-check splicing module is used to, when the multiple analysis results are different, ask the corresponding black box large language model through the prompt template to obtain a prompt about the reason why a certain analysis result is wrong, and splice the prompt with the prompt template to obtain the final document; the output module is used to re-input the final document and the questions to be retrieved into the corresponding black box large language model for answer retrieval.

[0088] The present invention also provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of an active retrieval enhancement method combining multiple language models.

[0089] In accordance with the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or with any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0090] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of an active retrieval enhancement method combining multiple language models are implemented.

[0091] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0092] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. For technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. An active retrieval enhancement method combining multi-language models, characterized in that: The steps include: Get the questions to be searched; Construct corresponding prompt templates for multiple different types of black box large language models, and input the to-be-searched questions and the corresponding prompt templates into the corresponding black box large language models to obtain multiple analysis results of the different black box large language models for the search questions; wherein the prompt template actively performs a self-check on the black box large language model to determine whether knowledge is required according to the to-be-searched questions; When the analysis results are different, the corresponding black box large language model is questioned through the prompt template to obtain a prompt of the reason why a certain analysis result is wrong, and the prompt is spliced ​​with the prompt template to obtain the final document; The final document and the question to be retrieved are re-input into the corresponding black box large language model to perform answer retrieval.

2. The active retrieval enhancement method combining multiple language models as claimed in claim 1, characterized in that: The multiple black box large language models include the Qianfan large model ERNIE-3.5 and the generative pre-trained model GPT-3.5-Turbo-0613, among which ERNIE-3.5 covers a number of Chinese-related knowledge, and GPT-3.5-Turbo-0613 contains a number of English and English scene-related knowledge.

3. The active retrieval enhancement method combining multiple language models as claimed in claim 1, characterized in that: The prompt of the reason for the error of a certain analysis result is obtained, and the prompt is spliced ​​with the prompt template to obtain the final document, including: Based on the prompt, the black box large language model gives the reasons why the specific options in the rhetorical question are wrong; Processing the error and removing the error from the prompt template; The retrieved knowledge and the reasons given after the rhetorical question are spliced ​​together as prompts to obtain the final document.

4. The active retrieval enhancement method combining multiple language models as claimed in claim 1, characterized in that: The method further comprises: The BM25 retrieval method is used to search in the knowledge base to obtain knowledge with a prompt similarity higher than a threshold value obtained by the reverse question, and the knowledge with a similarity higher than the threshold value is used as the expanded prior knowledge input into the black box large language model.

5. An active retrieval enhancement system combining multi-language models, characterized in that: include: A collection module, used to obtain questions to be searched; The analysis module is used to construct corresponding prompt templates for multiple different types of black box large language models, and input the to-be-searched questions and the corresponding prompt templates into the corresponding black box large language models to obtain multiple analysis results of the different black box large language models for the search questions; wherein the prompt template actively performs a self-check on whether the black box large language model needs knowledge according to the to-be-searched questions; A self-checking splicing module is used to, when the analysis results are different, to ask the corresponding black box large language model through a prompt template to obtain a prompt of the reason why a certain analysis result is wrong, and to splice the prompt with the prompt template to obtain a final document; The output module is used to re-input the final document and the question to be retrieved into the corresponding black box large language model to perform answer retrieval.

6. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the active retrieval enhancement method combined with a multi-language model as claimed in any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the active retrieval enhancement method combining multiple language models according to any one of claims 1 to 4 are implemented.

Images