Large language model question and answer method, device and equipment based on retrieval enhancement and medium

Abstract

The invention discloses a retrieval enhancement-based large language model question and answer method, apparatus and device, and a medium. The method comprises the steps of obtaining a target question input by a user; retrieving in a pre-constructed knowledge base according to the target question to obtain an initial retrieval result; according to a first preset prompt word, calling a large language model to determine a first correlation score between the initial retrieval result and the target question, and filtering the initial retrieval result based on the first correlation score to obtain a filtered retrieval result; and according to a second preset prompt word, based on the target question and the filtered retrieval result, calling a large language model to generate a target answer. According to the method, the introspection type retrieval strategy and the correlation evaluation mechanism are introduced by means of the large language model in the retrieval stage, introduction of useless information is reduced, the pertinence and the accuracy of retrieval are improved, therefore, consumption of computing resources is reduced, the accuracy of content generated by the large language model is improved, the RAG technical potential is brought into full play, and the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a large language model question answering method, device, equipment and medium based on retrieval enhancement. Background Art

[0002] With the rapid development of large language models (LLMs) technology, they have demonstrated powerful capabilities in dealing with a wide range of topics. However, the knowledge base of LLMs is inevitably limited by the public data used in their training as of a specific point in time, which leads to the model generating inaccurate answers or failing to cover professional knowledge when dealing with new knowledge, private information, or deep business data. In order to break through this bottleneck, the Retrieval-Augmented Generation (RAG) model is introduced as a key strategy to improve the performance of LLMs. The core of the RAG model lies in its fusion of retrieval and generation capabilities. The model allows business-related data, professional knowledge, etc. to be integrated into an extensible knowledge base as a supplement to the external knowledge of LLMs. This process does not require retraining of LLMs. Only new knowledge needs to be updated to the knowledge base to achieve instant knowledge expansion of the model.

[0003] In the existing implementation of RAG technology, a significant drawback is the problem of over-retrieval. The original intention of the RAG model was to provide richer and more accurate information support for complex natural language processing tasks by combining the capabilities of retrieval and generation. However, in practical applications, this indiscriminate broad retrieval strategy often becomes a double-edged sword. When the model tries to capture all possible information related to the input question, it will inevitably introduce a large amount of content that is not directly related to the core topic or even completely deviates from the topic. This redundant and irrelevant information not only increases the consumption of computing resources, but may also become interference items, affecting the model's extraction and integration of key information, ultimately resulting in a decrease in the quality of the generated results, or even misleading users. Summary of the invention

[0004] The embodiments of the present invention provide a large language model question answering method, apparatus, device and medium based on retrieval enhancement, so as to reduce the introduction of useless information in the retrieval stage, improve the pertinence and accuracy of the retrieval, thereby avoiding the problem of excessive retrieval of the RAG model, saving the use of resources, and improving the quality of the generated results.

[0005] In a first aspect, an embodiment of the present invention provides a large language model question answering method based on retrieval enhancement, the method comprising:

[0006] Get the target question input by the user;

[0007] Searching the pre-constructed knowledge base according to the target question to obtain initial search results;

[0008] According to the first preset prompt word, calling the large language model to determine a first relevance score between the initial search result and the target question, and filtering the initial search result based on the first relevance score to obtain a filtered search result;

[0009] According to the second preset prompt word, based on the target question and the filtered retrieval results, a large language model is called to generate a target answer.

[0010] Optionally, after calling the large language model to generate a target answer based on the target question and the filtered search results according to the second preset prompt word, the method further includes:

[0011] According to a third preset prompt word, calling a large language model to determine a second relevance score between the target answer and the filtered search result;

[0012] Whether the target answer needs to be regenerated is determined according to the second relevance score.

[0013] Optionally, after determining whether the target answer needs to be regenerated according to the second relevance score, the method further includes:

[0014] If not, calling the large language model to determine the answer score of the target answer to the target question according to the fourth preset prompt word;

[0015] It is determined whether the target answer can be output according to the answer score.

[0016] Optionally, after determining whether the target answer can be output according to the answer score, the method further includes:

[0017] If not, then according to the fifth preset prompt word, call the large language model to reconstruct the target question to obtain an optimized question;

[0018] The search in the pre-built knowledge base is performed according to the optimized question return.

[0019] Optionally, after filtering the initial search results based on the first relevance score to obtain filtered search results, the method further includes:

[0020] If the filtered search result is empty, then according to the fifth preset prompt word, the large language model is called to reconstruct the target question to obtain an optimized question;

[0021] The search in the pre-built knowledge base is performed according to the optimized question return.

[0022] Optionally, searching in a pre-constructed knowledge base according to the target question to obtain initial search results includes:

[0023] The target problem is converted into a problem vector based on an embedding method; the content in the pre-constructed knowledge base is represented in the form of a vector;

[0024] Based on a vector matching method, the initial search result is obtained by searching from the pre-constructed knowledge base according to the question vector.

[0025] In a second aspect, an embodiment of the present invention further provides a large language model question answering device based on retrieval enhancement, the device comprising:

[0026] A question acquisition module is used to obtain the target question input by the user;

[0027] A document retrieval module is used to search the pre-constructed knowledge base according to the target question to obtain initial retrieval results;

[0028] A document filtering module, configured to call a large language model to determine a first relevance score between the initial search result and the target question according to a first preset prompt word, and filter the initial search result based on the first relevance score to obtain a filtered search result;

[0029] The answer generation module is used to call the large language model to generate a target answer based on the second preset prompt word, the target question and the filtered retrieval result.

[0030] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0031] one or more processors;

[0032] A memory for storing one or more programs;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the retrieval-enhanced large language model question answering method provided by any embodiment of the present invention.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the retrieval-enhanced large language model question-answering method provided by any embodiment of the present invention.

[0035] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program, and when the program is executed by a processor, it implements the retrieval-enhanced large language model question answering method provided by any embodiment of the present invention.

[0036] An embodiment of the present invention provides a large language model question-answering method based on retrieval enhancement, which first obtains a target question input by a user, then searches a pre-constructed knowledge base based on the target question to obtain an initial retrieval result, and then calls a large language model to determine a first relevance score between the initial retrieval result and the target question based on a first preset prompt word, and filters the initial retrieval result based on the first relevance score to obtain a filtered retrieval result, and then calls a large language model to generate a target answer based on the target question and the filtered retrieval result according to a second preset prompt word. The large language model question-answering method based on retrieval enhancement provided by an embodiment of the present invention reduces the introduction of useless information and improves the pertinence and accuracy of retrieval by introducing an introspective retrieval strategy and a relevance evaluation mechanism with the help of a large language model in the retrieval stage, thereby reducing the consumption of computing resources and improving the accuracy of the content generated by the large language model, thereby giving full play to the potential of RAG technology and improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of a large language model question answering method based on retrieval enhancement provided in Example 1 of the present invention;

[0038] Figure 2 A schematic diagram of the overall processing logic of the large language model question answering method based on retrieval enhancement provided in the first embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of a large language model question answering device based on retrieval enhancement provided in the second embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0042] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0043] Embodiment 1

[0044] Figure 1 This is a flowchart of the large language model question answering method based on retrieval enhancement provided in the first embodiment of the present invention. This embodiment can be applied to the case where a large language model is used to answer user questions. The method can be executed by the large language model question answering device based on retrieval enhancement provided in the embodiment of the present invention, which can be implemented in hardware and / or software and can generally be integrated into a computer device. Figure 1 As shown, the specific steps include:

[0045] S11. Obtain the target question input by the user.

[0046] S12. Search the pre-constructed knowledge base according to the target question to obtain initial search results.

[0047] S13. According to a first preset prompt word, a large language model is called to determine a first relevance score between the initial search result and the target question, and the initial search result is filtered based on the first relevance score to obtain a filtered search result.

[0048] S14. According to the second preset prompt word, based on the target question and the filtered search results, call the large language model to generate a target answer.

[0049] Specifically, the target question input by the user, that is, the original question that the user wants to query, can be received through the front-end interactive interface. Then, based on similarity search, documents related to the target question can be retrieved from the pre-constructed knowledge base to obtain the initial search results. At the same time, the target question and the initial search results can be formed into a first dictionary {"documents":documents,"question":question} for subsequent use. The retrieved information fragments can be used as valuable context resources and provided to the large language model for answer generation, thereby expanding the knowledge scope of the large language model and significantly enhancing the comprehensiveness and timeliness of the answers.

[0050] Wherein, optionally, the searching in the pre-constructed knowledge base according to the target question to obtain the initial search result includes: converting the target question into a question vector based on an embedding method; representing the content in the pre-constructed knowledge base in the form of a vector; and retrieving the initial search result from the pre-constructed knowledge base according to the question vector based on a vector matching method. Specifically, the embedding method based on the Transformer structure can convert text into vectors of fixed length, and these vectors can capture the semantic and grammatical information in the text, so that similar texts have similar representations in the vector space. The document content in the pre-constructed knowledge base can be obtained in advance, and these document contents can be converted into document vectors parse_documents through the embedding method to form a vector database. After obtaining the target question, the target question can also be converted into a question vector question through the embedding method. Then, the question vector can be used to retrieve relevant documents based on the vector matching method to obtain the initial search result. By converting to a vectorized representation, the retrieval process can be made more efficient and accurate, thereby facilitating the rapid location of highly relevant document content. The retrieval process can be performed by a document retrieval module, and the document retrieval module can retrieve relevant document content from the vector database according to the input question.

[0051] After obtaining the initial search results, the initial search results and the target question (specifically, the first dictionary), as well as the first preset prompt word, can be input into the large language model for question and answer, so as to evaluate the first relevance score between each document in the initial search results and the target question. The first relevance score can be yes or no, that is, [binary_score: "yes" or "no"], and the first preset prompt word can be used to define the scorer's instructions to explain how to evaluate the relevance of the document to the question. Exemplarily, the first preset prompt word Prompt = "Your role is a scorer responsible for evaluating the relevance of the document to the user's question. \n1. Your job is to determine whether the retrieved document is relevant to the question raised by the user. \n2. If the document contains keywords that are directly related to the user's question, this can be used as an indicator of relevance. In addition to keywords, it is also necessary to consider whether the document content matches the user's question semantically. \n3. Carefully read each retrieved document and analyze whether the document content contains keywords or semantics related to the user's question. \n4. If the document does contain keywords or semantics related to the user's question, give a "yes" 5. If the document is irrelevant to the user's question or the relevance is not obvious, a "no" score is given. "Then the documents whose first relevance score is judged as "yes" can be added to the filtered document list filtered_docs, and irrelevant documents (i.e., documents whose first relevance score is judged as "no") can be skipped, thereby filtering the initial search results and obtaining the filtered search results. At the same time, the target question and the filtered search results can be formed into a second dictionary {"documents":filtered_docs,"question":question} for subsequent use. The filtering process can be performed by a document filtering module, which can filter documents according to the relevance between each input document and the input question.

[0052] After obtaining the filtered retrieval results, the filtered retrieval results and the target question (specifically, the second dictionary), as well as the second preset prompt word, can be input into the large language model for question and answer to generate the target answer generation_content. Among them, the second preset prompt word can be used to define prompt information to guide the large language model to generate answers based on the question and the document. The obtained target answer can be directly output to the user or further evaluated. The answer generation process can be executed by the answer generation module, which can use the large language model to generate the corresponding answer based on the input content.

[0053] On the basis of the above technical solution, optionally, after calling the large language model to generate the target answer based on the target question and the filtered retrieval results according to the second preset prompt word, it also includes: calling the large language model to determine the second correlation score between the target answer and the filtered retrieval results according to a third preset prompt word; and judging whether the target answer needs to be regenerated according to the second correlation score.

[0054] Specifically, after obtaining the target answer, the filtered retrieval results, the target question and the target answer can first be formed into a third dictionary {"documents":filtered_docs,"question":question,"generation":generation_content} for subsequent use. Each time a target answer is obtained, the filtered retrieval results and the target answer (specifically, the third dictionary can be used directly), as well as the third preset prompt word, can be input into the large language model for question and answer, so as to evaluate the second relevance score between the target answer and the filtered retrieval results. The second relevance score can be yes or no, that is, [binary_score: "yes" or "no"], which can be used to evaluate whether the generated answer is based on facts in the retrieved documents or supported by these facts. The third preset prompt word can be used to define instructions for the scorer to explain how to evaluate the relevance of the answer to the document. Exemplarily, the third preset prompt word Prompt = "Your role is a rater responsible for evaluating the answers generated by the large language model. \n1. Your evaluation will be based on whether the answer is associated with a set of retrieved facts. \n2. You need to review a set of retrieved facts that will serve as the basis for evaluating the answer. \n3. Carefully read the answer generated by the large language model and compare it with the retrieved facts. \n4. Determine whether the answer is based on these facts or supported by these facts. \n5. If the answer is indeed based on the retrieved facts or supported by these facts, give a "yes" score. \n6. If the answer is irrelevant to the retrieved facts or is not supported by the facts, a score of "No" is given. "Then it can be determined whether the target answer needs to be regenerated based on the second relevance score. Specifically, when the second relevance score is determined to be "No", it can be determined that the target answer needs to be regenerated, and the question and answer can be re-conducted based on the filtered search results, the target question and the second preset prompt word to generate a new target answer. Then the corresponding second relevance score can be evaluated again, and the process can be further repeated until the second relevance score is "Yes". At this time, the latest target answer can be directly output to the user, or further evaluation can be performed.

[0055] Further optionally, after determining whether the target answer needs to be regenerated based on the second relevance score, the method further includes: if not, calling a large language model to determine the answer score of the target answer to the target question based on a fourth preset prompt word; and determining whether the target answer can be output based on the answer score.

[0056] Specifically, when it is determined according to the second relevance score that the target answer does not need to be regenerated, it is possible to further evaluate whether the latest target answer successfully solves the target problem. Specifically, the target answer and the target question (specifically, the third dictionary can be used directly), as well as the fourth preset prompt word, can be input into the large language model for question and answer, so as to evaluate the answer score of the target answer for the target question. The answer score can be yes or no, that is, [binary_score: "yes" or "no"], which can be used to evaluate whether the generated answer can solve the user's problem. The fourth preset prompt word can be used to define the scorer's instructions to explain how to evaluate whether the answer successfully solves the problem. Exemplarily, the fourth preset prompt word Prompt = "Your role is a scorer, responsible for evaluating whether the answer generated by the large language model solves the problem.\n1. Your job is to determine whether the answer generated by the large language model successfully solves the problem.\n2. Read the answer carefully and compare it with the question to determine whether the answer directly provides a solution or explanation to the problem.\n3. Carefully read each answer generated by the large language model, compare the answer with the original question, and check whether the answer meets the requirements of the question.\n4. If the answer solves the problem clearly and accurately, give a "yes" score.\n5. If the answer fails to solve the problem or is irrelevant to the question, give a "no" score." Then, it can be determined whether the target answer can be output according to the answer score. Specifically, when the answer score is judged to be "yes", it can be determined that the target answer can be output and the target answer can be output to the user. When the answer score is judged to be "no", it is determined that the target answer cannot be output. At this time, the target answer can be regenerated, and a prompt can be given to the user, etc.

[0057] Traditional RAG technology often faces the severe test of output consistency. Ideally, the RAG model should be able to accurately extract relevant information from the retrieved knowledge and generate output content that is closely consistent with these facts. However, the reality is often more complicated. The complexity and uncertainty of large language models in processing natural language, as well as the diversity and potential inaccuracy of knowledge sources, all pose challenges to the consistency of output content. The model may generate content that is not completely consistent with the original knowledge due to misunderstanding of contextual information, interference from its own biases, or limitations of training data. This inconsistency not only weakens the credibility of the RAG model, but may also mislead users, affecting its effectiveness and value in practical applications. However, this method can introduce an introspective content generation strategy with the help of a large language model by adding the second relevance score and the answer score, evaluate whether the generated content strictly follows the reference knowledge to answer, and iterate multiple times to improve the answer quality, thereby strengthening the model's understanding of facts in the generation stage and ensuring the consistency and accuracy of the generated content.

[0058] Further optionally, after determining whether the target answer can be output according to the answer score, the method further includes: if not, calling a large language model to reconstruct the target question according to a fifth preset prompt word to obtain an optimized question; and returning to perform a search in the pre-constructed knowledge base according to the optimized question.

[0059] Specifically, when it is determined that the target answer cannot be output according to the answer score, the retrieval of the vector database can be optimized by question reconstruction. Specifically, the target question and the fifth preset prompt word can be input into the large language model for question and answer to generate an optimized question better_question. Among them, the fifth preset prompt word can be used to define prompt information to guide the large language model to reconstruct the target question. Exemplarily, the fifth preset prompt word Prompt = "You are an expert in question reconstruction, and you can convert the user's question into an optimized query for vector database retrieval to improve the search efficiency and accuracy of the large language model. \nYou can try to infer its underlying semantic intent based on the input question." Then, according to the optimized question, the search in the pre-constructed knowledge base can be returned, and then the above process can be re-executed based on the new initial search results. When it is determined again that the latest target answer cannot be output, the optimized question can be reconstructed again, and the process is repeated until it is determined that the latest target answer can be output. The reconstruction process can be performed by the question conversion module, and the question conversion module can reconstruct the input question. After each reconstruction is completed, a new state object {"documents":documents,"question":better_question} containing the optimized question can be returned to the document retrieval module.

[0060] On the basis of the above technical solution, optionally, after filtering the initial search results based on the first relevance score to obtain the filtered search results, it also includes: if the filtered search results are empty, calling the large language model to reconstruct the target question according to the fifth preset prompt word to obtain an optimized question; and returning to execute the search in the pre-constructed knowledge base according to the optimized question.

[0061] Specifically, after obtaining the filtered retrieval results, it is also possible to first determine whether the filtered retrieval results are empty based on the corresponding second dictionary. If it is empty, the target question can be directly reconstructed without executing the subsequent answer generation process for the target question. If the filtered retrieval result obtained based on the optimized question is empty, the optimized question is directly reconstructed again, and the subsequent answer generation process for the optimized question is no longer executed. After each reconstruction is completed, the search in the pre-constructed knowledge base can be returned based on the reconstructed optimized question, and then the above process can be re-executed based on the new initial retrieval results. When it is determined again that the latest filtered retrieval result is empty, the optimized question can be directly reconstructed again, and the process is repeated until it is determined that the latest filtered retrieval result is not empty. The direct reconstruction process can also be performed by the above-mentioned question conversion module.

[0062] When the method is used for knowledge question answering based on RAG, it can be used directly with simple configuration, and has a wider applicability than the traditional RAG model. The implementation method is simple, the logic is clear, and the adjustment and change are fast, so it can be well applied. Before applying the method, a workflow can be constructed first. The workflow includes a work module, a process edge, and a conditional branch. The workflow is consistent with the method processing logic. Among them, the work module may include the above-mentioned document retrieval module (Retrieve_documents), document filtering module (Filter_relevant_documents), answer generation module (Generate_answer_from_llm), and question transformation module (Transform_question), etc. The process edge may include executing the document filtering module ([Retrieve_documents]->[Filter_relevant_documents]) after the document retrieval module runs, and returning to the document retrieval module ([Transform_question]->[Retrieve_documents]) after the question transformation module runs, etc., to support iterative improvement of problems. The conditional branch may include a decide_question_transformation_or_content_generation branch: if the result of the document filtering module is "transform_question", the question transformation module is executed, and if the result is "generate_answer_from_llm", the answer generation module is executed. The branch code is as follows:

[0063]

[0064] The conditional branch may also include a decide_question_transformation_or_content_generation_or_retrieve_documents branch: if the result of the answer generation module is "not supported", the answer generation module is executed; if the result is "useful", the workflow is terminated; if the result is "not useful", the question transformation module is executed. The branch code is as follows:

[0065]

[0066]

[0067] Correspondingly, in the above method, each time after the filtered retrieval result is obtained, the decide_question_transformation_or_content_generation branch is executed to determine whether the next step should be to directly reconstruct the question to seek more relevant documents or to start generating answers. If the filtered retrieval result is empty, "transform_question" is returned, otherwise "generate_answer_from_llm" is returned. Each time after the target answer is generated, the decide_question_transformation_or_content_generation_or_retrieve_documents branch is executed to evaluate whether the generated answer is based on the retrieved documents and whether the answer successfully solves the user's question. If the target answer needs to be regenerated according to the second relevance score (such as the second relevance score is no), "notsupported" is returned. If the target answer can be output according to the answer score (such as the answer score is yes), "useful" is returned, and the workflow is completed. If the target answer cannot be output according to the answer score (such as the answer score is no), "not useful" is returned. The execution condition of the answer generation module is: the decide_question_transformation_or_content_generation branch returns "generate_answer_from_llm", or the decide_question_transformation_or_content_generation_or_retrieve_documents branch returns "not supported". The execution condition of the question transformation module is: the decide_question_transformation_or_content_generation_or_retrieve_documents branch returns "notuseful", or the decide_question_transformation_or_content_generation branch returns "transform_question". The overall processing logic diagram is as follows Figure 2 shown.

[0068] The technical solution provided by the embodiment of the present invention first obtains the target question input by the user, then searches the pre-constructed knowledge base according to the target question to obtain the initial search results, and then calls the large language model to determine the first relevance score between the initial search results and the target question according to the first preset prompt word, and filters the initial search results based on the first relevance score to obtain the filtered search results, and then calls the large language model to generate the target answer based on the target question and the filtered search results according to the second preset prompt word. By introducing the introspective search strategy and relevance evaluation mechanism with the help of the large language model in the search stage, the introduction of useless information is reduced, the relevance and accuracy of the search are improved, thereby reducing the consumption of computing resources, and improving the accuracy of the content generated by the large language model, thereby giving full play to the potential of RAG technology and improving the user experience.

[0069] Embodiment 2

[0070] Figure 3 This is a schematic diagram of the structure of the large language model question answering device based on retrieval enhancement provided in the second embodiment of the present invention. The device can be implemented by hardware and / or software, and can generally be integrated into a computer device to execute the large language model question answering method based on retrieval enhancement provided in any embodiment of the present invention. Figure 3 As shown, the device comprises:

[0071] The question acquisition module 31 is used to acquire the target question input by the user;

[0072] A document retrieval module 32, used to search the pre-constructed knowledge base according to the target question to obtain an initial retrieval result;

[0073] A document filtering module 33, configured to call a large language model to determine a first relevance score between the initial search result and the target question according to a first preset prompt word, and filter the initial search result based on the first relevance score to obtain a filtered search result;

[0074] The answer generation module 34 is used to call the large language model to generate a target answer based on the second preset prompt word, the target question and the filtered search results.

[0075] The technical solution provided by the embodiment of the present invention first obtains the target question input by the user, then searches the pre-constructed knowledge base according to the target question to obtain the initial search results, and then calls the large language model to determine the first relevance score between the initial search results and the target question according to the first preset prompt word, and filters the initial search results based on the first relevance score to obtain the filtered search results, and then calls the large language model to generate the target answer based on the target question and the filtered search results according to the second preset prompt word. By introducing the introspective search strategy and relevance evaluation mechanism with the help of the large language model in the search stage, the introduction of useless information is reduced, the relevance and accuracy of the search are improved, thereby reducing the consumption of computing resources, and improving the accuracy of the content generated by the large language model, thereby giving full play to the potential of RAG technology and improving the user experience.

[0076] On the basis of the above technical solution, optionally, the large language model question answering device based on retrieval enhancement further includes:

[0077] a relevance evaluation module, configured to, after calling the large language model to generate a target answer based on the target question and the filtered search results according to the second preset prompt word, call the large language model to determine a second relevance score between the target answer and the filtered search results according to a third preset prompt word;

[0078] The content support evaluation module is used to determine whether the target answer needs to be regenerated according to the second relevance score.

[0079] On the basis of the above technical solution, optionally, the large language model question answering device based on retrieval enhancement further includes:

[0080] an answer evaluation module, configured to, after determining whether the target answer needs to be regenerated according to the second relevance score, if not, call a large language model to determine the answer score of the target answer to the target question according to a fourth preset prompt word;

[0081] The problem solving evaluation module is used to determine whether the target answer can be output according to the solution score.

[0082] On the basis of the above technical solution, optionally, the large language model question answering device based on retrieval enhancement further includes:

[0083] A first question reconstruction module is used for, after determining whether the target answer can be output according to the answer score, if not, calling the large language model to reconstruct the target question according to a fifth preset prompt word to obtain an optimized question;

[0084] The first re-search module is used to return and perform a search in the pre-built knowledge base according to the optimized question.

[0085] On the basis of the above technical solution, optionally, the large language model question answering device based on retrieval enhancement further includes:

[0086] A second question reconstruction module is used for, after filtering the initial search results based on the first relevance score to obtain the filtered search results, if the filtered search results are empty, to reconstruct the target question by calling the large language model according to a fifth preset prompt word to obtain an optimized question;

[0087] The second re-retrieval module is used to return and perform a search in the pre-constructed knowledge base according to the optimized question.

[0088] On the basis of the above technical solution, optionally, the document retrieval module 32 includes:

[0089] A vector conversion unit, used to convert the target problem into a problem vector based on an embedding method; the content in the pre-constructed knowledge base is represented in the form of a vector;

[0090] A vector matching unit is used to retrieve the initial search result from the pre-constructed knowledge base according to the question vector based on a vector matching method.

[0091] The retrieval-enhanced large language model question-answering device provided in an embodiment of the present invention can execute the retrieval-enhanced large language model question-answering method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] It is worth noting that in the above-mentioned embodiment of the retrieval-enhanced large language model question-answering device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0093] Embodiment 3

[0094] Figure 4 The schematic diagram of the structure of the computer device provided for the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the implementation mode of the present invention. Figure 4 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 4As shown, the computer device includes a processor 41, a memory 42, an input device 43 and an output device 44; the number of processors 41 in the computer device can be one or more. Figure 4 Taking a processor 41 as an example, the processor 41, memory 42, input device 43 and output device 44 in the computer device can be connected through a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0095] The memory 42, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the large language model question-answering method based on retrieval enhancement in the embodiment of the present invention (for example, the question acquisition module 31, the document retrieval module 32, the document filtering module 33 and the answer generation module 34 in the large language model question-answering device based on retrieval enhancement). The processor 41 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 42, that is, realizing the above-mentioned large language model question-answering method based on retrieval enhancement.

[0096] The memory 42 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The input device 43 can be used to obtain the target question input by the user, and generate key signal input related to the user setting and function control of the computer device, etc. The output device 44 can be used to return the generated target answer to the user, etc.

[0098] Embodiment 4

[0099] Embodiment 4 of the present invention further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, is used to execute a large language model question answering method based on retrieval enhancement, the method comprising:

[0100] Get the target question input by the user;

[0101] Searching the pre-constructed knowledge base according to the target question to obtain initial search results;

[0102] According to the first preset prompt word, calling the large language model to determine a first relevance score between the initial search result and the target question, and filtering the initial search result based on the first relevance score to obtain a filtered search result;

[0103] According to the second preset prompt word, based on the target question and the filtered retrieval results, a large language model is called to generate a target answer.

[0104] The storage medium may be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system, which is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that may be executed by one or more processors.

[0105] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention is not limited to the method operations described above, and the computer executable instructions can also execute related operations in the retrieval-enhanced large language model question answering method provided by any embodiment of the present invention.

[0106] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0108] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0109] Embodiment 5

[0110] Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code, instruction), which can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the retrieval-enhanced large language model question-answering method provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0111] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A large language model question answering method based on retrieval enhancement, characterized in that: include: Get the target question input by the user; Searching the pre-constructed knowledge base according to the target question to obtain initial search results; According to the first preset prompt word, calling the large language model to determine a first relevance score between the initial search result and the target question, and filtering the initial search result based on the first relevance score to obtain a filtered search result; According to the second preset prompt word, based on the target question and the filtered retrieval results, a large language model is called to generate a target answer.

2. The large language model question answering method based on retrieval enhancement according to claim 1, characterized in that: After calling the large language model to generate the target answer based on the target question and the filtered search results according to the second preset prompt word, the method further includes: According to a third preset prompt word, calling a large language model to determine a second relevance score between the target answer and the filtered search result; Whether the target answer needs to be regenerated is determined according to the second relevance score.

3. The large language model question answering method based on retrieval enhancement according to claim 2 is characterized in that: After determining whether the target answer needs to be regenerated according to the second relevance score, the method further includes: If not, calling the large language model to determine the answer score of the target answer to the target question according to the fourth preset prompt word; It is determined whether the target answer can be output according to the answer score.

4. The large language model question answering method based on retrieval enhancement according to claim 3 is characterized in that: After determining whether the target answer can be output according to the answer score, the method further includes: If not, then according to the fifth preset prompt word, call the large language model to reconstruct the target question to obtain an optimized question; The search in the pre-built knowledge base is performed according to the optimized question return.

5. The large language model question answering method based on retrieval enhancement according to claim 1, characterized in that: After filtering the initial search results based on the first relevance score to obtain filtered search results, the method further includes: If the filtered search result is empty, then according to the fifth preset prompt word, the large language model is called to reconstruct the target question to obtain an optimized question; The search in the pre-built knowledge base is performed according to the optimized question return.

6. The large language model question answering method based on retrieval enhancement according to claim 1, characterized in that: The searching in the pre-constructed knowledge base according to the target question to obtain initial search results includes: The target problem is converted into a problem vector based on an embedding method; the content in the pre-constructed knowledge base is represented in the form of a vector; Based on a vector matching method, the initial search result is obtained by searching from the pre-constructed knowledge base according to the question vector.

7. A large language model question answering device based on retrieval enhancement, characterized in that: include: A question acquisition module is used to obtain the target question input by the user; A document retrieval module is used to search the pre-constructed knowledge base according to the target question to obtain initial retrieval results; A document filtering module, configured to call a large language model to determine a first relevance score between the initial search result and the target question according to a first preset prompt word, and filter the initial search result based on the first relevance score to obtain a filtered search result; The answer generation module is used to call the large language model to generate a target answer based on the second preset prompt word, the target question and the filtered retrieval result.

8. A computer device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the retrieval-enhanced large language model question answering method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the retrieval-enhanced large language model question answering method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the retrieval-enhanced large language model question answering method as described in any one of claims 1 to 6.

Images