Open domain question answering method, device, equipment and storage medium

Abstract

The present invention provides an open domain question answering method, device, equipment and storage medium, belonging to the field of natural language processing technology, including: obtaining a question to be queried, retrieving relevant documents of the question to be queried in a knowledge base; inputting the question to be queried and the relevant documents into a pre-trained question answering model to generate candidate answers for the question to be queried; verifying the correctness of the candidate answer, if the candidate answer fails to pass the verification, generating feedback about the missing content in the candidate answer based on the retrieved relevant documents and the question to be queried; retrieving relevant documents of the question to be queried in the knowledge base based on the feedback about the missing content in the candidate answer; iterating the process of generating candidate answers, verifying and re-retrieval until the candidate answer passes the verification. This method can improve the accuracy of the large language model's answers to complex questions.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and specifically relates to an open domain question answering method, device, equipment and storage medium. Background Art

[0002] Open-domain question answering is a challenging task in the field of natural language processing. It has two major properties: open-domain and knowledge-intensive. Open-domain question answering tasks are crucial to the development of applications such as intelligent customer service, chatbots, and question-answering systems.

[0003] With the rapid development of pre-trained Large Language Models (LLMs), LLMs can be used to handle open-domain question-answering tasks by inputting questions into the LLM and generating answers using the knowledge learned by the LLM during training. However, for this type of implicit internal parameterized knowledge, LLM cannot be effectively utilized under question-answering-based prompts, so that LLM exhibits "hallucination" phenomenon and cannot obtain the correct answer. At present, the "hallucination" phenomenon of LLM is mainly alleviated by the retrieval-enhanced generation method, that is, the question is used as a query to retrieve documents related to the question from an external knowledge source through a retriever, and then the LLM is used as a reader, the retrieved documents and questions are input to the LLM and the answer is output.

[0004] However, for complex questions in the open domain question answering field (multi-hop question answering problems), existing search engines are unable to retrieve all the information to answer the question at one time. They often only obtain partial or preliminary information to answer the question. As a result, when LLM faces complex multi-hop questions, it cannot give correct answers due to insufficient information. Therefore, how to improve the accuracy of LLM's answers to complex questions has become a problem to be solved. Summary of the invention

[0005] In order to solve the problem in the prior art that when LLM faces complex multi-hop questions, it cannot correctly answer them due to insufficient information, the present invention provides an open domain question answering method, apparatus, device and storage medium.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An open domain question answering method, comprising:

[0008] Obtain the question to be queried, and retrieve relevant documents of the question to be queried in the knowledge base;

[0009] Input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question;

[0010] Verify the correctness of the candidate answer. If the candidate answer fails verification, generate feedback about the missing content in the candidate answer based on the retrieved relevant documents and the query question;

[0011] Retrieve relevant documents from the knowledge base based on the feedback of missing content in the candidate answers;

[0012] The process of iteratively generating candidate answers, verifying, and re-retrieval until the candidate answers pass verification.

[0013] Optionally, before retrieving relevant documents of the query question in the knowledge base, the following is further included:

[0014] A dense retriever based on a Bert dual-tower structure is used as a retriever, and the retriever includes a document encoder;

[0015] All documents in the Wikipedia corpus are encoded using a document encoder to obtain document vectors, which are then loaded into the vector database FASSI to build a knowledge base.

[0016] Optionally, the retriever further includes a query encoder, which retrieves relevant documents of the query question in the knowledge base, including:

[0017] Encode the question to be queried through a query encoder to obtain a question vector of the question to be queried;

[0018] Calculate the similarity score between the question vector and all document vectors in the vector database FASSI, and sort the documents according to the similarity score;

[0019] The documents with the top K similarity scores are selected as relevant documents for the query question, where K is an integer greater than or equal to 1.

[0020] Optionally, verify the correctness of the candidate answer, including:

[0021] The retrieved relevant documents, query questions and candidate answers are input into the evidence extraction model based on the large language model to extract the evidence fragments supporting the candidate answers;

[0022] Input the evidence fragment and the query question into the pre-trained question-answering model to generate the verification answer;

[0023] The consistency of the verification answer and the candidate answer is determined by a complete match. If the verification answer is consistent with the candidate answer, the candidate answer passes the verification. If the verification answer is inconsistent with the candidate answer, the candidate answer fails the verification.

[0024] Optionally, it also includes:

[0025] When the query question and related documents are input into the pre-trained question-answering model, answer samples are input so that the question-answering model generates candidate answers to the query question according to the answer samples.

[0026] Optionally, generate feedback about missing content in the candidate answers based on the retrieved relevant documents and the query question, including:

[0027] The retrieved relevant documents and query questions are input into a feedback generation model based on a large language model, and feedback about the missing content in the candidate answers is generated through the feedback generation model.

[0028] Optionally, the method further includes: when verifying the correctness of the candidate answer, if the candidate answer passes the verification, outputting the candidate answer.

[0029] The present invention also provides an open domain question answering device, comprising:

[0030] The first retrieval module is used to obtain the question to be queried and retrieve relevant documents of the question to be queried in the knowledge base;

[0031] A generation module is used to input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question;

[0032] A verification module is used to verify the correctness of the candidate answer. If the candidate answer fails the verification, feedback about the missing content in the candidate answer is generated based on the retrieved relevant documents and the query question;

[0033] The second retrieval module is used to retrieve relevant documents of the query question in the knowledge base again based on the feedback of the missing content in the candidate answers;

[0034] The iteration module is used to iteratively generate candidate answers, verify and retrieve again until the candidate answers pass the verification.

[0035] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned open domain question answering method when executing the program.

[0036] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the open domain question answering method is implemented.

[0037] The open domain question answering method provided by the present invention has the following beneficial effects:

[0038] After generating candidate answers to the questions to be queried through the pre-trained question-answering model, the present invention does not directly use the candidate answers as the final answers, but verifies the accuracy of the candidate answers. When the verification fails, feedback on the missing content in the candidate answers is generated based on the retrieved related documents and the questions to be queried, and based on the feedback, the relevant documents of the questions to be queried are retrieved again in the knowledge base to generate candidate answers. When the accuracy of the candidate answers is not high, the same content will still be retrieved based on the same query question, and the missing content cannot be supplemented. Therefore, continuously generating feedback on the missing content in the candidate answers for iterative retrieval will make the retrieved related documents more comprehensive and accurate, thereby making the prompt content received by the question-answering model richer and generating more accurate answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic flow chart of an open domain question answering method provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a search process provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of an open domain question answering framework provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of an open domain question answering device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0045] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0046] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0047] In view of the problem that the large language model in the prior art has low accuracy in answering questions, the present invention proposes an open domain question answering method, such as Figure 1 As shown, the method comprises the following steps:

[0048] S1: Get the question to be queried and retrieve relevant documents of the question to be queried in the knowledge base.

[0049] In this invention, a dense passage retrieval (DPR) based on Bert is used as a retrieval device. The DPR retrieval device adopts a dual-tower structure, which consists of a query encoder and a and document encoder The DPR retriever used in the present invention is a pre-trained model and does not require additional training.

[0050] In the embodiment of the present invention, in order to save the time cost spent on retrieving relevant documents, a local offline knowledge base is constructed through the Wikipedia corpus to achieve document retrieval.

[0051] Figure 2 A schematic diagram of a retrieval process provided by an embodiment of the present invention.

[0052] For example, Figure 2 As shown, each document in the Wikipedia corpus is All passed through the document encoder Encode and get the vector representation of the document , and then load each document vector into the dense vector database FASSI to form a document vector set , build a local offline knowledge base.

[0053] Further, using the query question As a query, through the query encoder Encode and get the vector representation of the query Then All document vectors in the FASSI vector database Calculate the cosine similarity score and sort it according to the similarity score. Finally, put the sorted documents as the related documents obtained in the current search.

[0054] In the above embodiment, by constructing a local offline knowledge base, each time a relevant document is retrieved, there is no need to frequently encode the corpus documents, and the retrieval is directly performed from the FASSI library according to the cosine similarity, thereby saving retrieval time.

[0055] S2: Input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question.

[0056] The present invention uses a large language model as a question-answering model To generate candidate answers, the question-answering model can be a gpt-3.5-turbo, a gpt-4 model, and a llama2 model. The embodiment of the present invention does not specifically limit this. It should be understood that the question-answering model used in the present invention is a pre-trained question-answering model and does not need to be trained again.

[0057] Furthermore, the retrieved relevant documents and the query question are input into the question-answering model. , prompting the question-answering model to generate a candidate answer A to answer the query question.

[0058] Optionally, when inputting the retrieved relevant documents and the question to be queried into the question-answering mode, N (for example, 3) demonstration samples of answers may also be input as prompts to obtain a more accurate answer and output format.

[0059] S3: Verify the correctness of the candidate answer. If the candidate answer fails the verification, generate feedback about the missing content in the candidate answer based on the retrieved relevant documents and the query question.

[0060] It should be understood that when a question-answering model faces complex questions, it is often difficult to obtain an accurate answer in one go. Therefore, in order to ensure the accuracy of the answer, the candidate answers are verified.

[0061] In the present invention, verifying candidate answers consists of two stages, namely, the evidence extraction stage and the evidence-based question answering stage.

[0062] Specifically, in the evidence extraction stage, the retrieved relevant documents D and the query questions are and candidate answers Enter the evidence extraction model based on the large language model , suggesting evidence extraction model Extract relevant documents D from the retrieved documents to support the candidate answer Fragments of evidence .

[0063] The purpose of the evidence extraction phase is to simplify the retrieved relevant documents and filter out valid information from them. The formula is as follows:

[0064] ;

[0065] in, , They represent instructions and demonstration examples related to evidence extraction respectively.

[0066] In the evidence-based question-answering stage, the extracted evidence fragment E and the query question are As input to the question answering model , prompting the question-answering model to generate a verification answer to the question , the formula is as follows:

[0067] ;

[0068] Then, by judging the candidate answers and check the answer The consistency of the candidate answer is used to verify the correctness of the answer. and check the answer If they are consistent, the verification is successful and the current candidate answer is the final answer and is output. If they are inconsistent, the verification is unsuccessful and the confidence of the current candidate answer is low. Additional retrieval is required to supplement the information before retrieval. The process can be expressed by the following formula:

[0069] ;

[0070] in, Indicates the verification result.

[0071] Optionally, the present invention uses complete matching to determine candidate answers and check the answer Consistency is a strict way to judge consistency. You can also use powerful large language models such as gpt-4 to judge consistency, which is more robust.

[0072] In the embodiment of the present invention, when the verification model fails to verify the candidate answer, it indicates that the answer is less correct and an additional search is needed to supplement the missing content in the candidate answer. As a query, only the same documents can be retrieved, which cannot supplement the missing content. Therefore, it is necessary to construct a new query that is highly relevant to the missing content.

[0073] The present invention uses the large language model to understand the current missing content and generate feedback about the missing content. Specifically, the retrieved relevant documents and the initial query question are As input to the feedback generation model based on the large language model , prompting the feedback generation model to generate feedback about the missing content and using the generated feedback as a new query , the formula is as follows:

[0074] ;

[0075] in, , They represent instructions and demonstration examples related to feedback generation of missing content, respectively.

[0076] Optionally, the feedback generation model may adopt gpt-3.5-turbo, gpt-4 or llama2, which is not specifically limited in the embodiment of the present invention.

[0077] In addition, the present invention designs two different feedback forms, one is the form of questions corresponding to the missing content, and the other is the form of the missing content itself. Technical personnel in this field can choose one according to different actual scenarios.

[0078] S4: Based on the feedback of missing content in the candidate answers, relevant documents of the query question are retrieved again in the knowledge base.

[0079] That is, based on the new query generated in S3 Retrieve relevant documents of the query question in the knowledge base again.

[0080] S5: Iterate the process of generating candidate answers, verifying and re-retrieval until the candidate answer passes verification or the maximum number of iterations is reached.

[0081] The iterative process is as follows: input the question to be queried and all related documents previously retrieved into the pre-trained question-answering model to generate candidate answers to the question to be queried, and then verify the correctness of the candidate answers. If the verification fails, continue to input all related documents previously retrieved and the question to be queried into the feedback generation model based on the large language model to generate feedback about the missing content in the candidate answers, and search again based on the feedback about the missing content in the candidate answers until the candidate answers pass the verification or the maximum number of iterations (for example, 3 times) is reached, and the final answer is output.

[0082] In the above embodiment, the maximum number of iterations is set to terminate the iteration process in time to prevent the phenomenon of infinite iteration. When the maximum number of iterations is reached, the candidate answer obtained in the last iteration is used as the final answer. In the iterative retrieval enhancement generation framework proposed in the present invention, the number of iterations can be flexibly controlled and unnecessary noise information caused by redundant retrieval can be avoided.

[0083] Figure 3 A schematic diagram of an open domain question answering framework provided by an embodiment of the present invention.

[0084] For example, Figure 3 As shown in Figure 2, open domain question answering is divided into four stages:

[0085] 1. Retrieval stage: Input question Q to search the Wikipedia corpus, and select the top k similarity rankings as relevant documents .

[0086] 2. Question-answering stage: Input question Q and related documents into the question-answering model In the and .

[0087] 3. Verification phase: Question Q and retrieved relevant documents and candidate answers or Enter the evidence extraction model based on the large language model , extract evidence and obtain evidence fragments ; Question Q and evidence fragment Enter the Question Answering Model , obtain the verification answer; then verify the consistency between the verification answer and the candidate answer.

[0088] 4. Feedback generation stage: When the verification answer and the candidate answer are inconsistent, the question Q and related documents are sent to the Input Feedback Generation Model Based on Large Language Model , generate feedback of missing content as a new query, search again in the Wikipedia corpus based on the new query, and take the top k similarity rankings as relevant documents The supplementary documents are used again, and the question-answering stage and the verification stage are carried out again until the verification is passed or the retrieval reaches the maximum number of iterations, and the final result is output as the answer.

[0089] Based on the same inventive concept, the embodiment of the present invention also provides an open domain question answering device, such as Figure 4 As shown, the device comprises:

[0090] The first retrieval module 41 is used to obtain a question to be queried and retrieve documents related to the question to be queried in the knowledge base.

[0091] The generation module 42 is used to input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question.

[0092] The verification module 43 is used to verify the correctness of the candidate answer. If the candidate answer fails the verification, feedback about the missing content in the candidate answer is generated based on the retrieved relevant documents and the query question.

[0093] The second retrieval module 44 is used to retrieve documents related to the query question in the knowledge base again based on the feedback of the missing content in the candidate answers.

[0094] The iteration module 45 is used to iteratively generate candidate answers, verify and search again until the candidate answers pass the verification.

[0095] Each module in the above-mentioned open domain question-answering device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0096] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the open domain question answering method embodiment. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0097] Furthermore, the present invention also provides a non-temporary computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, a non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the open domain question answering method embodiment can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0098] It should be understood by those skilled in the art that embodiments of the present invention may provide methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0102] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the invention has been described in detail in this specification and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belongs to the protection scope of the present invention.

Claims

1. An open domain question answering method, characterized in that: include: Obtain the question to be queried, and retrieve relevant documents of the question to be queried in the knowledge base; Input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question; Verify the correctness of the candidate answer. If the candidate answer fails verification, generate feedback about the missing content in the candidate answer based on the retrieved relevant documents and the query question; Retrieve relevant documents from the knowledge base based on the feedback of missing content in the candidate answers; Iterate the process of generating candidate answers, verifying and re-retrieval until the candidate answers pass verification; The retriever also includes a query encoder that retrieves relevant documents for the query in the knowledge base, including: Encode the question to be queried through a query encoder to obtain a question vector of the question to be queried; Calculate the similarity score between the question vector and all document vectors in the vector database FASSI, and sort the documents according to the similarity score; Select the top K documents with the highest similarity scores as the relevant documents for the query, where K is an integer greater than or equal to 1; Verify the correctness of candidate answers, including: The retrieved relevant documents, query questions and candidate answers are input into the evidence extraction model based on the large language model to extract the evidence fragments supporting the candidate answers; Input the evidence fragment and the query question into the pre-trained question-answering model to generate the verification answer; The consistency between the verification answer and the candidate answer is determined by a complete match. If the verification answer is consistent with the candidate answer, the candidate answer passes the verification. If the verification answer is inconsistent with the candidate answer, the candidate answer fails the verification. Generates feedback about missing content in candidate answers based on the retrieved relevant documents and the query, including: The retrieved relevant documents and query questions are input into a feedback generation model based on a large language model, and feedback about the missing content in the candidate answers is generated through the feedback generation model.

2. The open domain question answering method according to claim 1, characterized in that: Before searching the knowledge base for relevant documents for the query, it also includes: A dense retriever based on a Bert dual-tower structure is used as a retriever, and the retriever includes a document encoder; All documents in the Wikipedia corpus are encoded using a document encoder to obtain document vectors, which are then loaded into the vector database FASSI to build a knowledge base.

3. The open domain question answering method according to claim 1, characterized in that: Also includes: When the query question and related documents are input into the pre-trained question-answering model, answer samples are input so that the question-answering model generates candidate answers to the query question according to the answer samples.

4. The open domain question answering method according to claim 1, characterized in that: Also includes: When verifying the correctness of a candidate answer, if the candidate answer passes the verification, the candidate answer is output.

5. A device for implementing the open domain question answering method according to claim 1, characterized in that: include: The first retrieval module is used to obtain the question to be queried and retrieve relevant documents of the question to be queried in the knowledge base; A generation module is used to input the query question and related documents into the pre-trained question-answering model to generate candidate answers to the query question; A verification module is used to verify the correctness of the candidate answer. If the candidate answer fails the verification, feedback about the missing content in the candidate answer is generated based on the retrieved relevant documents and the query question; The second retrieval module is used to retrieve relevant documents of the query question in the knowledge base again based on the feedback of the missing content in the candidate answers; The iteration module is used to iteratively generate candidate answers, verify and retrieve again until the candidate answers pass the verification.

6. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the open domain question answering method according to any one of claims 1 to 4 when executing the program.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the open domain question answering method according to any one of claims 1 to 4 is implemented.

Images